본문 바로가기
프로그래밍/hadoop

Hiveserver2 Thread의 지속 적인 증가 현상 해결

by W.C. 2016. 3. 10.
728x90

HDP-2.3.4.0-3485 버젼을 사용 중인데...

특이하게 종종 Hiveserver2의 task Thread가 증가 하는 경우가 발생한다.

결국에는 Thread가 계속 증가하다가 Hadoop의 성능이 현저하게 떨어지고 결국에는 뻗어 버리고 만다.


처음에는 원인을 몰랐다.


ps -ef|grep hiveserver


명령후 나오는 pid를 가지고


/proc/{pid}/task/


폴더 안에 들어가 보면 해당 process의 thread 수를 알 수가 있다.


이를 통해서 hiveserver2를 통해서 생성되는 thread 수를 파악 했고


for ((;;)) ; do ls |wc -l ; sleep 5 ;done


을 통해 쿼리 수행 시 thread 수를 계속적으로 모니터링 한 결과 지속 적인 증가가 이루어 지는 걸 알았다.


왜? 증가 하는 것일까?


Hive 소스를 들여다 보는 것도 시간과 노력 그리고 실력이 있어야 가능한데 한참을 들여다 보다가


좀더 단서가 필요했다. 그냥은 도저히..


그래서 jstack을 이용하여 더 상세하게 원인을 파악해 보았다.


jstack 해당 pid 


을 하면 과련 jvm dump를 볼 수 있다.


하지만 가끔 볼 수 없다고 나오는데 그럴 때 해당 process의 소유자 계정으로 봐야 한다.


hiveserver의 경우


sudo -u hive jstack 해당 pid


로 한 결과!


  1. Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
  2. "IPC Client (1445384588) connection to lognodem.hadoop.com/10.10.82.223:8020 from anonymous" #7175 daemon prio=5 os_prio=0 tid=0x00007f9bc546d800 nid=0x447b in Object.wait() [0x00007f9b8d494000]
  3. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  4. at java.lang.Object.wait(Native Method)
  5. at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
  6. - locked <0x0000000745d59d28> (a org.apache.hadoop.ipc.Client$Connection)
  7. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
  8. "IPC Parameter Sending Thread #178" #7173 daemon prio=5 os_prio=0 tid=0x00007f9bc4edc800 nid=0x3b1b waiting on condition [0x00007f9b89959000]
  9. java.lang.Thread.State: TIMED_WAITING (parking)
  10. at sun.misc.Unsafe.park(Native Method)
  11. - parking to wait for <0x000000064b8bf9f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
  12. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  13. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
  14. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
  15. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
  16. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  17. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  18. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  19. at java.lang.Thread.run(Thread.java:745)
  20. "LogFDsCachecleanInActiveFDsTimer" #6810 daemon prio=5 os_prio=0 tid=0x00007f9bc41cc800 nid=0x1c0 in Object.wait() [0x00007f9b87e40000]
  21. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  22. at java.lang.Object.wait(Native Method)
  23. at java.util.TimerThread.mainLoop(Timer.java:552)
  24. - locked <0x000000075726d230> (a java.util.TaskQueue)
  25. at java.util.TimerThread.run(Timer.java:505)
  26. "LogFDsCacheFlushTimer" #6809 daemon prio=5 os_prio=0 tid=0x00007f9bc4227800 nid=0x1bf in Object.wait() [0x00007f9b8a161000]
  27. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  28. at java.lang.Object.wait(Native Method)
  29. at java.util.TimerThread.mainLoop(Timer.java:552)
  30. - locked <0x000000075726d548> (a java.util.TaskQueue)
  31. at java.util.TimerThread.run(Timer.java:505)
  32. "LogFDsCachecleanInActiveFDsTimer" #6800 daemon prio=5 os_prio=0 tid=0x00007f9bc45a0800 nid=0x1b3 in Object.wait() [0x00007f9b88446000]
  33. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  34. at java.lang.Object.wait(Native Method)
  35. at java.util.TimerThread.mainLoop(Timer.java:552)
  36. - locked <0x000000075726df98> (a java.util.TaskQueue)
  37. at java.util.TimerThread.run(Timer.java:505)
  38. "LogFDsCacheFlushTimer" #6799 daemon prio=5 os_prio=0 tid=0x00007f9bc49a9000 nid=0x1b2 in Object.wait() [0x00007f9b8c787000]
  39. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  40. at java.lang.Object.wait(Native Method)
  41. at java.util.TimerThread.mainLoop(Timer.java:552)
  42. - locked <0x000000075726e2b0> (a java.util.TaskQueue)
  43. at java.util.TimerThread.run(Timer.java:505)
  44. "LogFDsCachecleanInActiveFDsTimer" #6788 daemon prio=5 os_prio=0 tid=0x00007f9bc459e000 nid=0x177 in Object.wait() [0x00007f9b907c7000]
  45. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  46. at java.lang.Object.wait(Native Method)
  47. at java.util.TimerThread.mainLoop(Timer.java:552)
  48. - locked <0x0000000757286c88> (a java.util.TaskQueue)
  49. at java.util.TimerThread.run(Timer.java:505)
  50. "LogFDsCacheFlushTimer" #6787 daemon prio=5 os_prio=0 tid=0x00007f9bc422a800 nid=0x176 in Object.wait() [0x00007f9b89757000]
  51. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  52. at java.lang.Object.wait(Native Method)
  53. at java.util.TimerThread.mainLoop(Timer.java:552)
  54. - locked <0x0000000757286fa0> (a java.util.TaskQueue)
  55. at java.util.TimerThread.run(Timer.java:505)
  56. "LogFDsCachecleanInActiveFDsTimer" #6778 daemon prio=5 os_prio=0 tid=0x00007f9bc4834000 nid=0x16b in Object.wait() [0x00007f9b88749000]
  57. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  58. at java.lang.Object.wait(Native Method)
  59. at java.util.TimerThread.mainLoop(Timer.java:552)
  60. - locked <0x00000007572141a0> (a java.util.TaskQueue)
  61. at java.util.TimerThread.run(Timer.java:505)
  62. "LogFDsCacheFlushTimer" #6777 daemon prio=5 os_prio=0 tid=0x00007f9bc4171800 nid=0x16a in Object.wait() [0x00007f9b8e0a0000]
  63. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  64. at java.lang.Object.wait(Native Method)
  65. at java.util.TimerThread.mainLoop(Timer.java:552)
  66. - locked <0x0000000757213420> (a java.util.TaskQueue)
  67. at java.util.TimerThread.run(Timer.java:505)
  68. "LogFDsCachecleanInActiveFDsTimer" #6765 daemon prio=5 os_prio=0 tid=0x00007f9bc546b000 nid=0x153 in Object.wait() [0x00007f9b8894b000]
  69. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  70. at java.lang.Object.wait(Native Method)
  71. at java.util.TimerThread.mainLoop(Timer.java:552)
  72. - locked <0x0000000757288190> (a java.util.TaskQueue)
  73. at java.util.TimerThread.run(Timer.java:505)
  74. "LogFDsCacheFlushTimer" #6764 daemon prio=5 os_prio=0 tid=0x00007f9bc4c2f800 nid=0x152 in Object.wait() [0x00007f9b88a4c000]
  75. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  76. at java.lang.Object.wait(Native Method)
  77. at java.util.TimerThread.mainLoop(Timer.java:552)
  78. - locked <0x00000007572884a8> (a java.util.TaskQueue)
  79. at java.util.TimerThread.run(Timer.java:505)
  80. "LogFDsCachecleanInActiveFDsTimer" #6755 daemon prio=5 os_prio=0 tid=0x00007f9bc422b000 nid=0x147 in Object.wait() [0x00007f9b8ab6b000]
  81. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  82. at java.lang.Object.wait(Native Method)
  83. at java.util.TimerThread.mainLoop(Timer.java:552)
  84. - locked <0x0000000757288ef8> (a java.util.TaskQueue)
  85. at java.util.TimerThread.run(Timer.java:505)
  86. "LogFDsCacheFlushTimer" #6754 daemon prio=5 os_prio=0 tid=0x00007f9bc429d000 nid=0x146 in Object.wait() [0x00007f9b97a37000]
  87. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  88. at java.lang.Object.wait(Native Method)
  89. at java.util.TimerThread.mainLoop(Timer.java:552)
  90. - locked <0x0000000757289210> (a java.util.TaskQueue)
  91. at java.util.TimerThread.run(Timer.java:505)
  92. "LogFDsCachecleanInActiveFDsTimer" #6744 daemon prio=5 os_prio=0 tid=0x00007f9bc5175800 nid=0x13d in Object.wait() [0x00007f9b89f5f000]
  93. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  94. at java.lang.Object.wait(Native Method)
  95. at java.util.TimerThread.mainLoop(Timer.java:552)
  96. - locked <0x000000075728a268> (a java.util.TaskQueue)
  97. at java.util.TimerThread.run(Timer.java:505)
  98. "LogFDsCacheFlushTimer" #6743 daemon prio=5 os_prio=0 tid=0x00007f9bc4b19000 nid=0x13c in Object.wait() [0x00007f9b9b370000]
  99. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  100. at java.lang.Object.wait(Native Method)
  101. at java.util.TimerThread.mainLoop(Timer.java:552)
  102. - locked <0x000000075728a580> (a java.util.TaskQueue)
  103. at java.util.TimerThread.run(Timer.java:505)
  104. "LogFDsCachecleanInActiveFDsTimer" #6739 daemon prio=5 os_prio=0 tid=0x00007f9bc5173800 nid=0x138 in Object.wait() [0x00007f9b8a666000]
  105. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  106. at java.lang.Object.wait(Native Method)
  107. at java.util.TimerThread.mainLoop(Timer.java:552)
  108. - locked <0x000000075728b550> (a java.util.TaskQueue)
  109. at java.util.TimerThread.run(Timer.java:505)
  110. "LogFDsCacheFlushTimer" #6738 daemon prio=5 os_prio=0 tid=0x00007f9bc4c42800 nid=0x137 in Object.wait() [0x00007f9b91ada000]
  111. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  112. at java.lang.Object.wait(Native Method)
  113. at java.util.TimerThread.mainLoop(Timer.java:552)
  114. - locked <0x000000075728b868> (a java.util.TaskQueue)
  115. at java.util.TimerThread.run(Timer.java:505)
  116. "LogFDsCachecleanInActiveFDsTimer" #6734 daemon prio=5 os_prio=0 tid=0x00007f9bc5153800 nid=0x133 in Object.wait() [0x00007f9b89151000]
  117. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  118. at java.lang.Object.wait(Native Method)
  119. at java.util.TimerThread.mainLoop(Timer.java:552)
  120. - locked <0x000000075728c838> (a java.util.TaskQueue)
  121. at java.util.TimerThread.run(Timer.java:505)
  122. "LogFDsCacheFlushTimer" #6733 daemon prio=5 os_prio=0 tid=0x00007f9bc449c000 nid=0x132 in Object.wait() [0x00007f9b89252000]
  123. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  124. at java.lang.Object.wait(Native Method)
  125. at java.util.TimerThread.mainLoop(Timer.java:552)
  126. - locked <0x000000075728cb50> (a java.util.TaskQueue)
  127. at java.util.TimerThread.run(Timer.java:505)
  128. "LogFDsCachecleanInActiveFDsTimer" #6727 daemon prio=5 os_prio=0 tid=0x00007f9bc449a000 nid=0x7ffe in Object.wait() [0x00007f9b8a767000]
  129. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  130. at java.lang.Object.wait(Native Method)
  131. at java.util.TimerThread.mainLoop(Timer.java:552)
  132. - locked <0x000000075728d3f8> (a java.util.TaskQueue)
  133. at java.util.TimerThread.run(Timer.java:505)
  134. "LogFDsCacheFlushTimer" #6726 daemon prio=5 os_prio=0 tid=0x00007f9bc4d10000 nid=0x7ffd in Object.wait() [0x00007f9b8ca8a000]
  135. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  136. at java.lang.Object.wait(Native Method)
  137. at java.util.TimerThread.mainLoop(Timer.java:552)
  138. - locked <0x000000075728d710> (a java.util.TaskQueue)
  139. at java.util.TimerThread.run(Timer.java:505)
  140. "LogFDsCachecleanInActiveFDsTimer" #6717 daemon prio=5 os_prio=0 tid=0x00007f9bc4496000 nid=0x7fc2 in Object.wait() [0x00007f9b89555000]
  141. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  142. at java.lang.Object.wait(Native Method)
  143. at java.util.TimerThread.mainLoop(Timer.java:552)
  144. - locked <0x0000000651f9dbe0> (a java.util.TaskQueue)
  145. at java.util.TimerThread.run(Timer.java:505)
  146. "LogFDsCacheFlushTimer" #6716 daemon prio=5 os_prio=0 tid=0x00007f9bc4172000 nid=0x7fc1 in Object.wait() [0x00007f9b8a868000]
  147. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  148. at java.lang.Object.wait(Native Method)
  149. at java.util.TimerThread.mainLoop(Timer.java:552)
  150. - locked <0x0000000651f9def8> (a java.util.TaskQueue)
  151. at java.util.TimerThread.run(Timer.java:505)
  152. "LogFDsCachecleanInActiveFDsTimer" #6704 daemon prio=5 os_prio=0 tid=0x00007f9bc4281800 nid=0x7fb1 in Object.wait() [0x00007f9b8eaaa000]
  153. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  154. at java.lang.Object.wait(Native Method)
  155. at java.util.TimerThread.mainLoop(Timer.java:552)
  156. - locked <0x0000000651f66878> (a java.util.TaskQueue)
  157. at java.util.TimerThread.run(Timer.java:505)
  158. "LogFDsCacheFlushTimer" #6703 daemon prio=5 os_prio=0 tid=0x00007f9bc413c000 nid=0x7fb0 in Object.wait() [0x00007f9b8be7e000]
  159. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  160. at java.lang.Object.wait(Native Method)
  161. at java.util.TimerThread.mainLoop(Timer.java:552)
  162. - locked <0x0000000651f5a4a8> (a java.util.TaskQueue)
  163. at java.util.TimerThread.run(Timer.java:505)
  164. "LogFDsCachecleanInActiveFDsTimer" #6694 daemon prio=5 os_prio=0 tid=0x00007f9bc496b800 nid=0x7f9b in Object.wait() [0x00007f9b89858000]
  165. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  166. at java.lang.Object.wait(Native Method)
  167. at java.util.TimerThread.mainLoop(Timer.java:552)
  168. - locked <0x0000000651f59b28> (a java.util.TaskQueue)
  169. at java.util.TimerThread.run(Timer.java:505)
  170. "LogFDsCacheFlushTimer" #6693 daemon prio=5 os_prio=0 tid=0x00007f9bc44d5000 nid=0x7f9a in Object.wait() [0x00007f9b8ac6c000]
  171. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  172. at java.lang.Object.wait(Native Method)
  173. at java.util.TimerThread.mainLoop(Timer.java:552)
  174. - locked <0x0000000651f59810> (a java.util.TaskQueue)
  175. at java.util.TimerThread.run(Timer.java:505)
  176. "LogFDsCachecleanInActiveFDsTimer" #6680 daemon prio=5 os_prio=0 tid=0x00007f9bc4aee800 nid=0x7d41 in Object.wait() [0x00007f9b8c383000]
  177. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  178. at java.lang.Object.wait(Native Method)
  179. at java.util.TimerThread.mainLoop(Timer.java:552)
  180. - locked <0x0000000651cc2d80> (a java.util.TaskQueue)
  181. at java.util.TimerThread.run(Timer.java:505)
  182. "LogFDsCacheFlushTimer" #6679 daemon prio=5 os_prio=0 tid=0x00007f9bc477d800 nid=0x7d40 in Object.wait() [0x00007f9b8c484000]
  183. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  184. at java.lang.Object.wait(Native Method)
  185. at java.util.TimerThread.mainLoop(Timer.java:552)
  186. - locked <0x0000000651cc2828> (a java.util.TaskQueue)
  187. at java.util.TimerThread.run(Timer.java:505)
  188. "LogFDsCachecleanInActiveFDsTimer" #6670 daemon prio=5 os_prio=0 tid=0x00007f9bc4296000 nid=0x7d36 in Object.wait() [0x00007f9b89c5c000]
  189. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  190. at java.lang.Object.wait(Native Method)
  191. at java.util.TimerThread.mainLoop(Timer.java:552)
  192. - locked <0x0000000651519540> (a java.util.TaskQueue)
  193. at java.util.TimerThread.run(Timer.java:505)
  194. "LogFDsCacheFlushTimer" #6669 daemon prio=5 os_prio=0 tid=0x00007f9bc49a2000 nid=0x7d35 in Object.wait() [0x00007f9b89d5d000]
  195. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  196. at java.lang.Object.wait(Native Method)
  197. at java.util.TimerThread.mainLoop(Timer.java:552)
  198. - locked <0x0000000651546b68> (a java.util.TaskQueue)
  199. at java.util.TimerThread.run(Timer.java:505)
  200. "LogFDsCachecleanInActiveFDsTimer" #6658 daemon prio=5 os_prio=0 tid=0x00007f9bc465d800 nid=0x7d29 in Object.wait() [0x00007f9b8af6f000]
  201. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  202. at java.lang.Object.wait(Native Method)
  203. at java.util.TimerThread.mainLoop(Timer.java:552)
  204. - locked <0x0000000651b3ede0> (a java.util.TaskQueue)
  205. at java.util.TimerThread.run(Timer.java:505)
  206. "LogFDsCacheFlushTimer" #6657 daemon prio=5 os_prio=0 tid=0x00007f9bc4150000 nid=0x7d28 in Object.wait() [0x00007f9b8bd7d000]
  207. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  208. at java.lang.Object.wait(Native Method)
  209. at java.util.TimerThread.mainLoop(Timer.java:552)
  210. - locked <0x0000000651b3f0f8> (a java.util.TaskQueue)
  211. at java.util.TimerThread.run(Timer.java:505)
  212. "LogFDsCachecleanInActiveFDsTimer" #6648 daemon prio=5 os_prio=0 tid=0x00007f9bc4cd9800 nid=0x7ced in Object.wait() [0x00007f9b92cec000]
  213. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  214. at java.lang.Object.wait(Native Method)
  215. at java.util.TimerThread.mainLoop(Timer.java:552)
  216. - locked <0x000000065157fba0> (a java.util.TaskQueue)
  217. at java.util.TimerThread.run(Timer.java:505)
  218. "LogFDsCacheFlushTimer" #6647 daemon prio=5 os_prio=0 tid=0x00007f9bc4157000 nid=0x7cec in Object.wait() [0x00007f9b8b171000]
  219. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  220. at java.lang.Object.wait(Native Method)
  221. at java.util.TimerThread.mainLoop(Timer.java:552)
  222. - locked <0x0000000650e40660> (a java.util.TaskQueue)
  223. at java.util.TimerThread.run(Timer.java:505)
  224. "LogFDsCachecleanInActiveFDsTimer" #6635 daemon prio=5 os_prio=0 tid=0x00007f9bc4c37800 nid=0x7cda in Object.wait() [0x00007f9b8a262000]
  225. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  226. at java.lang.Object.wait(Native Method)
  227. at java.util.TimerThread.mainLoop(Timer.java:552)
  228. - locked <0x0000000650edd280> (a java.util.TaskQueue)
  229. at java.util.TimerThread.run(Timer.java:505)
  230. "LogFDsCacheFlushTimer" #6634 daemon prio=5 os_prio=0 tid=0x00007f9bc49a5800 nid=0x7cd9 in Object.wait() [0x00007f9b8a363000]
  231. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  232. at java.lang.Object.wait(Native Method)
  233. at java.util.TimerThread.mainLoop(Timer.java:552)
  234. - locked <0x0000000650edd598> (a java.util.TaskQueue)
  235. at java.util.TimerThread.run(Timer.java:505)
  236. "LogFDsCachecleanInActiveFDsTimer" #6625 daemon prio=5 os_prio=0 tid=0x00007f9bc413a800 nid=0x7cca in Object.wait() [0x00007f9b8b878000]
  237. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  238. at java.lang.Object.wait(Native Method)
  239. at java.util.TimerThread.mainLoop(Timer.java:552)
  240. - locked <0x0000000650f10c88> (a java.util.TaskQueue)
  241. at java.util.TimerThread.run(Timer.java:505)
  242. "LogFDsCacheFlushTimer" #6624 daemon prio=5 os_prio=0 tid=0x00007f9bc472d800 nid=0x7cc9 in Object.wait() [0x00007f9b8d393000]
  243. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  244. at java.lang.Object.wait(Native Method)
  245. at java.util.TimerThread.mainLoop(Timer.java:552)
  246. - locked <0x0000000650f21828> (a java.util.TaskQueue)
  247. at java.util.TimerThread.run(Timer.java:505)
  248. "LogFDsCachecleanInActiveFDsTimer" #6613 daemon prio=5 os_prio=0 tid=0x00007f9bc4d13800 nid=0x7cbd in Object.wait() [0x00007f9b8e1a1000]
  249. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  250. at java.lang.Object.wait(Native Method)
  251. at java.util.TimerThread.mainLoop(Timer.java:552)
  252. - locked <0x0000000650f5e5a0> (a java.util.TaskQueue)
  253. at java.util.TimerThread.run(Timer.java:505)
  254. "LogFDsCacheFlushTimer" #6612 daemon prio=5 os_prio=0 tid=0x00007f9bc4dfe000 nid=0x7cbc in Object.wait() [0x00007f9b8bf7f000]
  255. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  256. at java.lang.Object.wait(Native Method)
  257. at java.util.TimerThread.mainLoop(Timer.java:552)
  258. - locked <0x0000000650f23030> (a java.util.TaskQueue)
  259. at java.util.TimerThread.run(Timer.java:505)
  260. "LogFDsCachecleanInActiveFDsTimer" #6608 daemon prio=5 os_prio=0 tid=0x00007f9bc4667000 nid=0x7cb7 in Object.wait() [0x00007f9b8a969000]
  261. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  262. at java.lang.Object.wait(Native Method)
  263. at java.util.TimerThread.mainLoop(Timer.java:552)
  264. - locked <0x000000065192ebc8> (a java.util.TaskQueue)
  265. at java.util.TimerThread.run(Timer.java:505)
  266. "LogFDsCacheFlushTimer" #6607 daemon prio=5 os_prio=0 tid=0x00007f9bc4666800 nid=0x7cb6 in Object.wait() [0x00007f9b8aa6a000]
  267. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  268. at java.lang.Object.wait(Native Method)
  269. at java.util.TimerThread.mainLoop(Timer.java:552)
  270. - locked <0x000000065192eee0> (a java.util.TaskQueue)
  271. at java.util.TimerThread.run(Timer.java:505)
  272. "LogFDsCachecleanInActiveFDsTimer" #6598 daemon prio=5 os_prio=0 tid=0x00007f9bc429a000 nid=0x7cac in Object.wait() [0x00007f9b8ba7a000]
  273. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  274. at java.lang.Object.wait(Native Method)
  275. at java.util.TimerThread.mainLoop(Timer.java:552)
  276. - locked <0x00000006519153a0> (a java.util.TaskQueue)
  277. at java.util.TimerThread.run(Timer.java:505)
  278. "LogFDsCacheFlushTimer" #6597 daemon prio=5 os_prio=0 tid=0x00007f9bc4295000 nid=0x7cab in Object.wait() [0x00007f9b91ede000]
  279. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  280. at java.lang.Object.wait(Native Method)
  281. at java.util.TimerThread.mainLoop(Timer.java:552)
  282. - locked <0x000000065191af28> (a java.util.TaskQueue)
  283. at java.util.TimerThread.run(Timer.java:505)
  284. "LogFDsCachecleanInActiveFDsTimer" #6585 daemon prio=5 os_prio=0 tid=0x00007f9bc4148000 nid=0x7c9d in Object.wait() [0x00007f9b8ad6d000]
  285. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  286. at java.lang.Object.wait(Native Method)
  287. at java.util.TimerThread.mainLoop(Timer.java:552)
  288. - locked <0x0000000650f652f0> (a java.util.TaskQueue)
  289. at java.util.TimerThread.run(Timer.java:505)
  290. "LogFDsCacheFlushTimer" #6584 daemon prio=5 os_prio=0 tid=0x00007f9bc4afb800 nid=0x7c9c in Object.wait() [0x00007f9b8ae6e000]
  291. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  292. at java.lang.Object.wait(Native Method)
  293. at java.util.TimerThread.mainLoop(Timer.java:552)
  294. - locked <0x0000000650f71978> (a java.util.TaskQueue)
  295. at java.util.TimerThread.run(Timer.java:505)
  296. "LogFDsCachecleanInActiveFDsTimer" #6575 daemon prio=5 os_prio=0 tid=0x00007f9bc427f800 nid=0x7c91 in Object.wait() [0x00007f9b8c585000]
  297. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  298. at java.lang.Object.wait(Native Method)
  299. at java.util.TimerThread.mainLoop(Timer.java:552)
  300. - locked <0x000000065186c610> (a java.util.TaskQueue)
  301. at java.util.TimerThread.run(Timer.java:505)
  302. "LogFDsCacheFlushTimer" #6574 daemon prio=5 os_prio=0 tid=0x00007f9bc4157800 nid=0x7c90 in Object.wait() [0x00007f9b8d191000]
  303. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  304. at java.lang.Object.wait(Native Method)
  305. at java.util.TimerThread.mainLoop(Timer.java:552)
  306. - locked <0x00000006509b0950> (a java.util.TaskQueue)
  307. at java.util.TimerThread.run(Timer.java:505)
  308. "LogFDsCachecleanInActiveFDsTimer" #6563 daemon prio=5 os_prio=0 tid=0x00007f9bc4155000 nid=0x7c55 in Object.wait() [0x00007f9b90ece000]
  309. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  310. at java.lang.Object.wait(Native Method)
  311. at java.util.TimerThread.mainLoop(Timer.java:552)
  312. - locked <0x0000000650a81930> (a java.util.TaskQueue)
  313. at java.util.TimerThread.run(Timer.java:505)
  314. "LogFDsCacheFlushTimer" #6562 daemon prio=5 os_prio=0 tid=0x00007f9bc4524800 nid=0x7c54 in Object.wait() [0x00007f9b8d898000]
  315. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  316. at java.lang.Object.wait(Native Method)
  317. at java.util.TimerThread.mainLoop(Timer.java:552)
  318. - locked <0x0000000650a81a38> (a java.util.TaskQueue)
  319. at java.util.TimerThread.run(Timer.java:505)
  320. "LogFDsCachecleanInActiveFDsTimer" #6558 daemon prio=5 os_prio=0 tid=0x00007f9bc4d11800 nid=0x7c4e in Object.wait() [0x00007f9b926e6000]
  321. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  322. at java.lang.Object.wait(Native Method)
  323. at java.util.TimerThread.mainLoop(Timer.java:552)
  324. - locked <0x00000006514872a8> (a java.util.TaskQueue)
  325. at java.util.TimerThread.run(Timer.java:505)
  326. "LogFDsCacheFlushTimer" #6557 daemon prio=5 os_prio=0 tid=0x00007f9bc43ec000 nid=0x7c4d in Object.wait() [0x00007f9b8cb8b000]
  327. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  328. at java.lang.Object.wait(Native Method)
  329. at java.util.TimerThread.mainLoop(Timer.java:552)
  330. - locked <0x0000000651486d68> (a java.util.TaskQueue)
  331. at java.util.TimerThread.run(Timer.java:505)
  332. "LogFDsCachecleanInActiveFDsTimer" #6548 daemon prio=5 os_prio=0 tid=0x00007f9bc4229000 nid=0x7c37 in Object.wait() [0x00007f9b8b575000]
  333. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  334. at java.lang.Object.wait(Native Method)
  335. at java.util.TimerThread.mainLoop(Timer.java:552)
  336. - locked <0x0000000650a82590> (a java.util.TaskQueue)
  337. at java.util.TimerThread.run(Timer.java:505)
  338. "LogFDsCacheFlushTimer" #6547 daemon prio=5 os_prio=0 tid=0x00007f9bc4228800 nid=0x7c36 in Object.wait() [0x00007f9b8b676000]
  339. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  340. at java.lang.Object.wait(Native Method)
  341. at java.util.TimerThread.mainLoop(Timer.java:552)
  342. - locked <0x0000000650aa7030> (a java.util.TaskQueue)
  343. at java.util.TimerThread.run(Timer.java:505)
  344. "LogFDsCachecleanInActiveFDsTimer" #6534 daemon prio=5 os_prio=0 tid=0x00007f9bc495d000 nid=0x79e9 in Object.wait() [0x00007f9b8b979000]
  345. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  346. at java.lang.Object.wait(Native Method)
  347. at java.util.TimerThread.mainLoop(Timer.java:552)
  348. - locked <0x0000000650aa7cf8> (a java.util.TaskQueue)
  349. at java.util.TimerThread.run(Timer.java:505)
  350. "LogFDsCacheFlushTimer" #6533 daemon prio=5 os_prio=0 tid=0x00007f9bc472a800 nid=0x79e8 in Object.wait() [0x00007f9b8ce8e000]
  351. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  352. at java.lang.Object.wait(Native Method)
  353. at java.util.TimerThread.mainLoop(Timer.java:552)
  354. - locked <0x0000000650aa7e00> (a java.util.TaskQueue)
  355. at java.util.TimerThread.run(Timer.java:505)
  356. "LogFDsCachecleanInActiveFDsTimer" #6524 daemon prio=5 os_prio=0 tid=0x00007f9bc4141000 nid=0x79dd in Object.wait() [0x00007f9b8f1b1000]
  357. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  358. at java.lang.Object.wait(Native Method)
  359. at java.util.TimerThread.mainLoop(Timer.java:552)
  360. - locked <0x00000006513f4768> (a java.util.TaskQueue)
  361. at java.util.TimerThread.run(Timer.java:505)
  362. "LogFDsCacheFlushTimer" #6523 daemon prio=5 os_prio=0 tid=0x00007f9bc54b3800 nid=0x79dc in Object.wait() [0x00007f9b904c4000]
  363. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  364. at java.lang.Object.wait(Native Method)
  365. at java.util.TimerThread.mainLoop(Timer.java:552)
  366. - locked <0x00000006513fe7b8> (a java.util.TaskQueue)
  367. at java.util.TimerThread.run(Timer.java:505)
  368. "LogFDsCachecleanInActiveFDsTimer" #6507 daemon prio=5 os_prio=0 tid=0x00007f9bc4aff800 nid=0x799a in Object.wait() [0x00007f9b8bb7b000]
  369. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  370. at java.lang.Object.wait(Native Method)
  371. at java.util.TimerThread.mainLoop(Timer.java:552)
  372. - locked <0x0000000650f71e90> (a java.util.TaskQueue)
  373. at java.util.TimerThread.run(Timer.java:505)
  374. "LogFDsCacheFlushTimer" #6506 daemon prio=5 os_prio=0 tid=0x00007f9bc429b000 nid=0x7999 in Object.wait() [0x00007f9b8bc7c000]
  375. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  376. at java.lang.Object.wait(Native Method)
  377. at java.util.TimerThread.mainLoop(Timer.java:552)
  378. - locked <0x0000000650f8ca40> (a java.util.TaskQueue)
  379. at java.util.TimerThread.run(Timer.java:505)
  380. "LogFDsCachecleanInActiveFDsTimer" #6497 daemon prio=5 os_prio=0 tid=0x00007f9bc5167800 nid=0x798e in Object.wait() [0x00007f9b8c181000]
  381. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  382. at java.lang.Object.wait(Native Method)
  383. at java.util.TimerThread.mainLoop(Timer.java:552)
  384. - locked <0x0000000650f8cf38> (a java.util.TaskQueue)
  385. at java.util.TimerThread.run(Timer.java:505)
  386. "LogFDsCacheFlushTimer" #6496 daemon prio=5 os_prio=0 tid=0x00007f9bc5167000 nid=0x798d in Object.wait() [0x00007f9b8c282000]
  387. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  388. at java.lang.Object.wait(Native Method)
  389. at java.util.TimerThread.mainLoop(Timer.java:552)
  390. - locked <0x0000000650f72570> (a java.util.TaskQueue)
  391. at java.util.TimerThread.run(Timer.java:505)
  392. "LogFDsCachecleanInActiveFDsTimer" #6485 daemon prio=5 os_prio=0 tid=0x00007f9bc5360800 nid=0x797b in Object.wait() [0x00007f9b8e8a8000]
  393. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  394. at java.lang.Object.wait(Native Method)
  395. at java.util.TimerThread.mainLoop(Timer.java:552)
  396. - locked <0x00000006509b1f68> (a java.util.TaskQueue)
  397. at java.util.TimerThread.run(Timer.java:505)
  398. "LogFDsCacheFlushTimer" #6484 daemon prio=5 os_prio=0 tid=0x00007f9bc4c45800 nid=0x797a in Object.wait() [0x00007f9b8febe000]
  399. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  400. at java.lang.Object.wait(Native Method)
  401. at java.util.TimerThread.mainLoop(Timer.java:552)
  402. - locked <0x00000006509b1d80> (a java.util.TaskQueue)
  403. at java.util.TimerThread.run(Timer.java:505)
  404. "LogFDsCachecleanInActiveFDsTimer" #6475 daemon prio=5 os_prio=0 tid=0x00007f9bc5531000 nid=0x796f in Object.wait() [0x00007f9b8fdbd000]
  405. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  406. at java.lang.Object.wait(Native Method)
  407. at java.util.TimerThread.mainLoop(Timer.java:552)
  408. - locked <0x00000006509ec3c8> (a java.util.TaskQueue)
  409. at java.util.TimerThread.run(Timer.java:505)
  410. "LogFDsCacheFlushTimer" #6474 daemon prio=5 os_prio=0 tid=0x00007f9bc4528800 nid=0x796e in Object.wait() [0x00007f9b8f2b2000]
  411. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  412. at java.lang.Object.wait(Native Method)
  413. at java.util.TimerThread.mainLoop(Timer.java:552)
  414. - locked <0x00000006509ec560> (a java.util.TaskQueue)
  415. at java.util.TimerThread.run(Timer.java:505)
  416. "LogFDsCachecleanInActiveFDsTimer" #6463 daemon prio=5 os_prio=0 tid=0x00007f9bc44c5800 nid=0x7963 in Object.wait() [0x00007f9b922e2000]
  417. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  418. at java.lang.Object.wait(Native Method)
  419. at java.util.TimerThread.mainLoop(Timer.java:552)
  420. - locked <0x00000006509ddb10> (a java.util.TaskQueue)
  421. at java.util.TimerThread.run(Timer.java:505)
  422. "LogFDsCacheFlushTimer" #6462 daemon prio=5 os_prio=0 tid=0x00007f9bc431d800 nid=0x7962 in Object.wait() [0x00007f9b8dd9d000]
  423. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  424. at java.lang.Object.wait(Native Method)
  425. at java.util.TimerThread.mainLoop(Timer.java:552)
  426. - locked <0x00000006509b2700> (a java.util.TaskQueue)
  427. at java.util.TimerThread.run(Timer.java:505)
  428. "LogFDsCachecleanInActiveFDsTimer" #6457 daemon prio=5 os_prio=0 tid=0x00007f9bc4159000 nid=0x7956 in Object.wait() [0x00007f9b8c888000]
  429. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  430. at java.lang.Object.wait(Native Method)
  431. at java.util.TimerThread.mainLoop(Timer.java:552)
  432. - locked <0x00000006509a6980> (a java.util.TaskQueue)
  433. at java.util.TimerThread.run(Timer.java:505)
  434. "LogFDsCacheFlushTimer" #6456 daemon prio=5 os_prio=0 tid=0x00007f9bc4c3f800 nid=0x7955 in Object.wait() [0x00007f9b8c989000]
  435. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  436. at java.lang.Object.wait(Native Method)
  437. at java.util.TimerThread.mainLoop(Timer.java:552)
  438. - locked <0x00000006509a6c98> (a java.util.TaskQueue)
  439. at java.util.TimerThread.run(Timer.java:505)
  440. "LogFDsCachecleanInActiveFDsTimer" #6447 daemon prio=5 os_prio=0 tid=0x00007f9bc43ea800 nid=0x7948 in Object.wait() [0x00007f9b8cc8c000]
  441. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  442. at java.lang.Object.wait(Native Method)
  443. at java.util.TimerThread.mainLoop(Timer.java:552)
  444. - locked <0x00000006506dc628> (a java.util.TaskQueue)
  445. at java.util.TimerThread.run(Timer.java:505)
  446. "LogFDsCacheFlushTimer" #6446 daemon prio=5 os_prio=0 tid=0x00007f9bc42c6000 nid=0x7947 in Object.wait() [0x00007f9b8cd8d000]
  447. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  448. at java.lang.Object.wait(Native Method)
  449. at java.util.TimerThread.mainLoop(Timer.java:552)
  450. - locked <0x00000006509a8c10> (a java.util.TaskQueue)
  451. at java.util.TimerThread.run(Timer.java:505)
  452. "LogFDsCachecleanInActiveFDsTimer" #6435 daemon prio=5 os_prio=0 tid=0x00007f9bc52e7000 nid=0x793c in Object.wait() [0x00007f9b8d999000]
  453. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  454. at java.lang.Object.wait(Native Method)
  455. at java.util.TimerThread.mainLoop(Timer.java:552)
  456. - locked <0x00000006509a7d68> (a java.util.TaskQueue)
  457. at java.util.TimerThread.run(Timer.java:505)
  458. "LogFDsCacheFlushTimer" #6434 daemon prio=5 os_prio=0 tid=0x00007f9bc4755000 nid=0x793b in Object.wait() [0x00007f9b94908000]
  459. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  460. at java.lang.Object.wait(Native Method)
  461. at java.util.TimerThread.mainLoop(Timer.java:552)
  462. - locked <0x00000006509a8080> (a java.util.TaskQueue)
  463. at java.util.TimerThread.run(Timer.java:505)
  464. "LogFDsCachecleanInActiveFDsTimer" #6429 daemon prio=5 os_prio=0 tid=0x00007f9bc42ca000 nid=0x7904 in Object.wait() [0x00007f9b8cf8f000]
  465. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  466. at java.lang.Object.wait(Native Method)
  467. at java.util.TimerThread.mainLoop(Timer.java:552)
  468. - locked <0x00000006506c6ec8> (a java.util.TaskQueue)
  469. at java.util.TimerThread.run(Timer.java:505)
  470. "LogFDsCacheFlushTimer" #6428 daemon prio=5 os_prio=0 tid=0x00007f9bc42c9800 nid=0x7903 in Object.wait() [0x00007f9b8d090000]
  471. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  472. at java.lang.Object.wait(Native Method)
  473. at java.util.TimerThread.mainLoop(Timer.java:552)
  474. - locked <0x00000006506c71e0> (a java.util.TaskQueue)
  475. at java.util.TimerThread.run(Timer.java:505)
  476. "LogFDsCachecleanInActiveFDsTimer" #6419 daemon prio=5 os_prio=0 tid=0x00007f9bc472c800 nid=0x78f6 in Object.wait() [0x00007f9b8d595000]
  477. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  478. at java.lang.Object.wait(Native Method)
  479. at java.util.TimerThread.mainLoop(Timer.java:552)
  480. - locked <0x00000006506c7b90> (a java.util.TaskQueue)
  481. at java.util.TimerThread.run(Timer.java:505)
  482. "LogFDsCacheFlushTimer" #6418 daemon prio=5 os_prio=0 tid=0x00007f9bc472b800 nid=0x78f5 in Object.wait() [0x00007f9b8d696000]
  483. java.lang.Thread.State: TIMED_WAITING (on object monitor)
  484. at java.lang.Object.wait(Native Method)
  485. at java.util.TimerThread.mainLoop(Timer.java:552)
  486. - locked <0x00000006506cf0c8> (a java.util.TaskQueue)
  487. at java.util.TimerThread.run(Timer.java:505)


이렇게 

LogFDsCacheFlushTimer와 LogFDsCachecleanInActiveFDsTimer


가 무한 생성되고 TIMED_WAITING이 걸려 있다.


결구 모르겠다


하고 Hortonworks 포럼에 이 사실을 올렸다.


결과 빠른 답변


https://issues.apache.org/jira/browse/TEZ-3017

https://issues.apache.org/jira/browse/HIVE-12766


위와 같은 이슈에 등록 되어 있고 hdp 2.4 version에서 고쳐서 나온다고 한다.


그전에 해결 방법은 


YARN=> Config=> Advanced 에서


yarn.timeline-service.enabled 에 체크를 해제 해주면 된다.


일단 증상은 해결된 것으로 보인다.


하지만 timeline service를 사용 못해서 tez log 를 확인 하는데 약간의 어려움은 있다.





저작자표시 비영리 동일조건

'프로그래밍 > hadoop' 카테고리의 다른 글

Ambari metrics collector ERROR  (0) 2018.11.15
DataNode Java Heap Warning  (0) 2016.08.24
JounalNode 오류 관련 해프닝  (1) 2016.07.12
HDP 2.0에서 HDP 2.4로 업그레이드 삽질  (0) 2016.05.27
PostgreSQL shell command(ambari)  (0) 2016.04.11

관련글

티스토리툴바