Hiveserver2 Thread의 지속 적인 증가 현상 해결

HDP-2.3.4.0-3485 버젼을 사용 중인데...

특이하게 종종 Hiveserver2의 task Thread가 증가 하는 경우가 발생한다.

결국에는 Thread가 계속 증가하다가 Hadoop의 성능이 현저하게 떨어지고 결국에는 뻗어 버리고 만다.

처음에는 원인을 몰랐다.

ps -ef|grep hiveserver

명령후 나오는 pid를 가지고

/proc/{pid}/task/

폴더 안에 들어가 보면 해당 process의 thread 수를 알 수가 있다.

이를 통해서 hiveserver2를 통해서 생성되는 thread 수를 파악 했고

for ((;;)) ; do ls |wc -l ; sleep 5 ;done

을 통해 쿼리 수행 시 thread 수를 계속적으로 모니터링 한 결과 지속 적인 증가가 이루어 지는 걸 알았다.

왜? 증가 하는 것일까?

Hive 소스를 들여다 보는 것도 시간과 노력 그리고 실력이 있어야 가능한데 한참을 들여다 보다가

좀더 단서가 필요했다. 그냥은 도저히..

그래서 jstack을 이용하여 더 상세하게 원인을 파악해 보았다.

jstack 해당 pid 

을 하면 과련 jvm dump를 볼 수 있다.

하지만 가끔 볼 수 없다고 나오는데 그럴 때 해당 process의 소유자 계정으로 봐야 한다.

hiveserver의 경우

sudo -u hive jstack 해당 pid

로 한 결과!

1. Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode):
2. "IPC Client (1445384588) connection to lognodem.hadoop.com/10.10.82.223:8020 from anonymous" #7175 daemon prio=5 os_prio=0 tid=0x00007f9bc546d800 nid=0x447b in Object.wait() [0x00007f9b8d494000]
3. java.lang.Thread.State: TIMED_WAITING (on object monitor)
4. at java.lang.Object.wait(Native Method)
5. at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
6. - locked <0x0000000745d59d28> (a org.apache.hadoop.ipc.Client$Connection)
7. at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)
8. "IPC Parameter Sending Thread #178" #7173 daemon prio=5 os_prio=0 tid=0x00007f9bc4edc800 nid=0x3b1b waiting on condition [0x00007f9b89959000]
9. java.lang.Thread.State: TIMED_WAITING (parking)
10. at sun.misc.Unsafe.park(Native Method)
11. - parking to wait for <0x000000064b8bf9f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
12. at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
13. at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
14. at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
15. at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
16. at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
17. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
18. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
19. at java.lang.Thread.run(Thread.java:745)
20. "LogFDsCachecleanInActiveFDsTimer" #6810 daemon prio=5 os_prio=0 tid=0x00007f9bc41cc800 nid=0x1c0 in Object.wait() [0x00007f9b87e40000]
21. java.lang.Thread.State: TIMED_WAITING (on object monitor)
22. at java.lang.Object.wait(Native Method)
23. at java.util.TimerThread.mainLoop(Timer.java:552)
24. - locked <0x000000075726d230> (a java.util.TaskQueue)
25. at java.util.TimerThread.run(Timer.java:505)
26. "LogFDsCacheFlushTimer" #6809 daemon prio=5 os_prio=0 tid=0x00007f9bc4227800 nid=0x1bf in Object.wait() [0x00007f9b8a161000]
27. java.lang.Thread.State: TIMED_WAITING (on object monitor)
28. at java.lang.Object.wait(Native Method)
29. at java.util.TimerThread.mainLoop(Timer.java:552)
30. - locked <0x000000075726d548> (a java.util.TaskQueue)
31. at java.util.TimerThread.run(Timer.java:505)
32. "LogFDsCachecleanInActiveFDsTimer" #6800 daemon prio=5 os_prio=0 tid=0x00007f9bc45a0800 nid=0x1b3 in Object.wait() [0x00007f9b88446000]
33. java.lang.Thread.State: TIMED_WAITING (on object monitor)
34. at java.lang.Object.wait(Native Method)
35. at java.util.TimerThread.mainLoop(Timer.java:552)
36. - locked <0x000000075726df98> (a java.util.TaskQueue)
37. at java.util.TimerThread.run(Timer.java:505)
38. "LogFDsCacheFlushTimer" #6799 daemon prio=5 os_prio=0 tid=0x00007f9bc49a9000 nid=0x1b2 in Object.wait() [0x00007f9b8c787000]
39. java.lang.Thread.State: TIMED_WAITING (on object monitor)
40. at java.lang.Object.wait(Native Method)
41. at java.util.TimerThread.mainLoop(Timer.java:552)
42. - locked <0x000000075726e2b0> (a java.util.TaskQueue)
43. at java.util.TimerThread.run(Timer.java:505)
44. "LogFDsCachecleanInActiveFDsTimer" #6788 daemon prio=5 os_prio=0 tid=0x00007f9bc459e000 nid=0x177 in Object.wait() [0x00007f9b907c7000]
45. java.lang.Thread.State: TIMED_WAITING (on object monitor)
46. at java.lang.Object.wait(Native Method)
47. at java.util.TimerThread.mainLoop(Timer.java:552)
48. - locked <0x0000000757286c88> (a java.util.TaskQueue)
49. at java.util.TimerThread.run(Timer.java:505)
50. "LogFDsCacheFlushTimer" #6787 daemon prio=5 os_prio=0 tid=0x00007f9bc422a800 nid=0x176 in Object.wait() [0x00007f9b89757000]
51. java.lang.Thread.State: TIMED_WAITING (on object monitor)
52. at java.lang.Object.wait(Native Method)
53. at java.util.TimerThread.mainLoop(Timer.java:552)
54. - locked <0x0000000757286fa0> (a java.util.TaskQueue)
55. at java.util.TimerThread.run(Timer.java:505)
56. "LogFDsCachecleanInActiveFDsTimer" #6778 daemon prio=5 os_prio=0 tid=0x00007f9bc4834000 nid=0x16b in Object.wait() [0x00007f9b88749000]
57. java.lang.Thread.State: TIMED_WAITING (on object monitor)
58. at java.lang.Object.wait(Native Method)
59. at java.util.TimerThread.mainLoop(Timer.java:552)
60. - locked <0x00000007572141a0> (a java.util.TaskQueue)
61. at java.util.TimerThread.run(Timer.java:505)
62. "LogFDsCacheFlushTimer" #6777 daemon prio=5 os_prio=0 tid=0x00007f9bc4171800 nid=0x16a in Object.wait() [0x00007f9b8e0a0000]
63. java.lang.Thread.State: TIMED_WAITING (on object monitor)
64. at java.lang.Object.wait(Native Method)
65. at java.util.TimerThread.mainLoop(Timer.java:552)
66. - locked <0x0000000757213420> (a java.util.TaskQueue)
67. at java.util.TimerThread.run(Timer.java:505)
68. "LogFDsCachecleanInActiveFDsTimer" #6765 daemon prio=5 os_prio=0 tid=0x00007f9bc546b000 nid=0x153 in Object.wait() [0x00007f9b8894b000]
69. java.lang.Thread.State: TIMED_WAITING (on object monitor)
70. at java.lang.Object.wait(Native Method)
71. at java.util.TimerThread.mainLoop(Timer.java:552)
72. - locked <0x0000000757288190> (a java.util.TaskQueue)
73. at java.util.TimerThread.run(Timer.java:505)
74. "LogFDsCacheFlushTimer" #6764 daemon prio=5 os_prio=0 tid=0x00007f9bc4c2f800 nid=0x152 in Object.wait() [0x00007f9b88a4c000]
75. java.lang.Thread.State: TIMED_WAITING (on object monitor)
76. at java.lang.Object.wait(Native Method)
77. at java.util.TimerThread.mainLoop(Timer.java:552)
78. - locked <0x00000007572884a8> (a java.util.TaskQueue)
79. at java.util.TimerThread.run(Timer.java:505)
80. "LogFDsCachecleanInActiveFDsTimer" #6755 daemon prio=5 os_prio=0 tid=0x00007f9bc422b000 nid=0x147 in Object.wait() [0x00007f9b8ab6b000]
81. java.lang.Thread.State: TIMED_WAITING (on object monitor)
82. at java.lang.Object.wait(Native Method)
83. at java.util.TimerThread.mainLoop(Timer.java:552)
84. - locked <0x0000000757288ef8> (a java.util.TaskQueue)
85. at java.util.TimerThread.run(Timer.java:505)
86. "LogFDsCacheFlushTimer" #6754 daemon prio=5 os_prio=0 tid=0x00007f9bc429d000 nid=0x146 in Object.wait() [0x00007f9b97a37000]
87. java.lang.Thread.State: TIMED_WAITING (on object monitor)
88. at java.lang.Object.wait(Native Method)
89. at java.util.TimerThread.mainLoop(Timer.java:552)
90. - locked <0x0000000757289210> (a java.util.TaskQueue)
91. at java.util.TimerThread.run(Timer.java:505)
92. "LogFDsCachecleanInActiveFDsTimer" #6744 daemon prio=5 os_prio=0 tid=0x00007f9bc5175800 nid=0x13d in Object.wait() [0x00007f9b89f5f000]
93. java.lang.Thread.State: TIMED_WAITING (on object monitor)
94. at java.lang.Object.wait(Native Method)
95. at java.util.TimerThread.mainLoop(Timer.java:552)
96. - locked <0x000000075728a268> (a java.util.TaskQueue)
97. at java.util.TimerThread.run(Timer.java:505)
98. "LogFDsCacheFlushTimer" #6743 daemon prio=5 os_prio=0 tid=0x00007f9bc4b19000 nid=0x13c in Object.wait() [0x00007f9b9b370000]
99. java.lang.Thread.State: TIMED_WAITING (on object monitor)
100. at java.lang.Object.wait(Native Method)
101. at java.util.TimerThread.mainLoop(Timer.java:552)
102. - locked <0x000000075728a580> (a java.util.TaskQueue)
103. at java.util.TimerThread.run(Timer.java:505)
104. "LogFDsCachecleanInActiveFDsTimer" #6739 daemon prio=5 os_prio=0 tid=0x00007f9bc5173800 nid=0x138 in Object.wait() [0x00007f9b8a666000]
105. java.lang.Thread.State: TIMED_WAITING (on object monitor)
106. at java.lang.Object.wait(Native Method)
107. at java.util.TimerThread.mainLoop(Timer.java:552)
108. - locked <0x000000075728b550> (a java.util.TaskQueue)
109. at java.util.TimerThread.run(Timer.java:505)
110. "LogFDsCacheFlushTimer" #6738 daemon prio=5 os_prio=0 tid=0x00007f9bc4c42800 nid=0x137 in Object.wait() [0x00007f9b91ada000]
111. java.lang.Thread.State: TIMED_WAITING (on object monitor)
112. at java.lang.Object.wait(Native Method)
113. at java.util.TimerThread.mainLoop(Timer.java:552)
114. - locked <0x000000075728b868> (a java.util.TaskQueue)
115. at java.util.TimerThread.run(Timer.java:505)
116. "LogFDsCachecleanInActiveFDsTimer" #6734 daemon prio=5 os_prio=0 tid=0x00007f9bc5153800 nid=0x133 in Object.wait() [0x00007f9b89151000]
117. java.lang.Thread.State: TIMED_WAITING (on object monitor)
118. at java.lang.Object.wait(Native Method)
119. at java.util.TimerThread.mainLoop(Timer.java:552)
120. - locked <0x000000075728c838> (a java.util.TaskQueue)
121. at java.util.TimerThread.run(Timer.java:505)
122. "LogFDsCacheFlushTimer" #6733 daemon prio=5 os_prio=0 tid=0x00007f9bc449c000 nid=0x132 in Object.wait() [0x00007f9b89252000]
123. java.lang.Thread.State: TIMED_WAITING (on object monitor)
124. at java.lang.Object.wait(Native Method)
125. at java.util.TimerThread.mainLoop(Timer.java:552)
126. - locked <0x000000075728cb50> (a java.util.TaskQueue)
127. at java.util.TimerThread.run(Timer.java:505)
128. "LogFDsCachecleanInActiveFDsTimer" #6727 daemon prio=5 os_prio=0 tid=0x00007f9bc449a000 nid=0x7ffe in Object.wait() [0x00007f9b8a767000]
129. java.lang.Thread.State: TIMED_WAITING (on object monitor)
130. at java.lang.Object.wait(Native Method)
131. at java.util.TimerThread.mainLoop(Timer.java:552)
132. - locked <0x000000075728d3f8> (a java.util.TaskQueue)
133. at java.util.TimerThread.run(Timer.java:505)
134. "LogFDsCacheFlushTimer" #6726 daemon prio=5 os_prio=0 tid=0x00007f9bc4d10000 nid=0x7ffd in Object.wait() [0x00007f9b8ca8a000]
135. java.lang.Thread.State: TIMED_WAITING (on object monitor)
136. at java.lang.Object.wait(Native Method)
137. at java.util.TimerThread.mainLoop(Timer.java:552)
138. - locked <0x000000075728d710> (a java.util.TaskQueue)
139. at java.util.TimerThread.run(Timer.java:505)
140. "LogFDsCachecleanInActiveFDsTimer" #6717 daemon prio=5 os_prio=0 tid=0x00007f9bc4496000 nid=0x7fc2 in Object.wait() [0x00007f9b89555000]
141. java.lang.Thread.State: TIMED_WAITING (on object monitor)
142. at java.lang.Object.wait(Native Method)
143. at java.util.TimerThread.mainLoop(Timer.java:552)
144. - locked <0x0000000651f9dbe0> (a java.util.TaskQueue)
145. at java.util.TimerThread.run(Timer.java:505)
146. "LogFDsCacheFlushTimer" #6716 daemon prio=5 os_prio=0 tid=0x00007f9bc4172000 nid=0x7fc1 in Object.wait() [0x00007f9b8a868000]
147. java.lang.Thread.State: TIMED_WAITING (on object monitor)
148. at java.lang.Object.wait(Native Method)
149. at java.util.TimerThread.mainLoop(Timer.java:552)
150. - locked <0x0000000651f9def8> (a java.util.TaskQueue)
151. at java.util.TimerThread.run(Timer.java:505)
152. "LogFDsCachecleanInActiveFDsTimer" #6704 daemon prio=5 os_prio=0 tid=0x00007f9bc4281800 nid=0x7fb1 in Object.wait() [0x00007f9b8eaaa000]
153. java.lang.Thread.State: TIMED_WAITING (on object monitor)
154. at java.lang.Object.wait(Native Method)
155. at java.util.TimerThread.mainLoop(Timer.java:552)
156. - locked <0x0000000651f66878> (a java.util.TaskQueue)
157. at java.util.TimerThread.run(Timer.java:505)
158. "LogFDsCacheFlushTimer" #6703 daemon prio=5 os_prio=0 tid=0x00007f9bc413c000 nid=0x7fb0 in Object.wait() [0x00007f9b8be7e000]
159. java.lang.Thread.State: TIMED_WAITING (on object monitor)
160. at java.lang.Object.wait(Native Method)
161. at java.util.TimerThread.mainLoop(Timer.java:552)
162. - locked <0x0000000651f5a4a8> (a java.util.TaskQueue)
163. at java.util.TimerThread.run(Timer.java:505)
164. "LogFDsCachecleanInActiveFDsTimer" #6694 daemon prio=5 os_prio=0 tid=0x00007f9bc496b800 nid=0x7f9b in Object.wait() [0x00007f9b89858000]
165. java.lang.Thread.State: TIMED_WAITING (on object monitor)
166. at java.lang.Object.wait(Native Method)
167. at java.util.TimerThread.mainLoop(Timer.java:552)
168. - locked <0x0000000651f59b28> (a java.util.TaskQueue)
169. at java.util.TimerThread.run(Timer.java:505)
170. "LogFDsCacheFlushTimer" #6693 daemon prio=5 os_prio=0 tid=0x00007f9bc44d5000 nid=0x7f9a in Object.wait() [0x00007f9b8ac6c000]
171. java.lang.Thread.State: TIMED_WAITING (on object monitor)
172. at java.lang.Object.wait(Native Method)
173. at java.util.TimerThread.mainLoop(Timer.java:552)
174. - locked <0x0000000651f59810> (a java.util.TaskQueue)
175. at java.util.TimerThread.run(Timer.java:505)
176. "LogFDsCachecleanInActiveFDsTimer" #6680 daemon prio=5 os_prio=0 tid=0x00007f9bc4aee800 nid=0x7d41 in Object.wait() [0x00007f9b8c383000]
177. java.lang.Thread.State: TIMED_WAITING (on object monitor)
178. at java.lang.Object.wait(Native Method)
179. at java.util.TimerThread.mainLoop(Timer.java:552)
180. - locked <0x0000000651cc2d80> (a java.util.TaskQueue)
181. at java.util.TimerThread.run(Timer.java:505)
182. "LogFDsCacheFlushTimer" #6679 daemon prio=5 os_prio=0 tid=0x00007f9bc477d800 nid=0x7d40 in Object.wait() [0x00007f9b8c484000]
183. java.lang.Thread.State: TIMED_WAITING (on object monitor)
184. at java.lang.Object.wait(Native Method)
185. at java.util.TimerThread.mainLoop(Timer.java:552)
186. - locked <0x0000000651cc2828> (a java.util.TaskQueue)
187. at java.util.TimerThread.run(Timer.java:505)
188. "LogFDsCachecleanInActiveFDsTimer" #6670 daemon prio=5 os_prio=0 tid=0x00007f9bc4296000 nid=0x7d36 in Object.wait() [0x00007f9b89c5c000]
189. java.lang.Thread.State: TIMED_WAITING (on object monitor)
190. at java.lang.Object.wait(Native Method)
191. at java.util.TimerThread.mainLoop(Timer.java:552)
192. - locked <0x0000000651519540> (a java.util.TaskQueue)
193. at java.util.TimerThread.run(Timer.java:505)
194. "LogFDsCacheFlushTimer" #6669 daemon prio=5 os_prio=0 tid=0x00007f9bc49a2000 nid=0x7d35 in Object.wait() [0x00007f9b89d5d000]
195. java.lang.Thread.State: TIMED_WAITING (on object monitor)
196. at java.lang.Object.wait(Native Method)
197. at java.util.TimerThread.mainLoop(Timer.java:552)
198. - locked <0x0000000651546b68> (a java.util.TaskQueue)
199. at java.util.TimerThread.run(Timer.java:505)
200. "LogFDsCachecleanInActiveFDsTimer" #6658 daemon prio=5 os_prio=0 tid=0x00007f9bc465d800 nid=0x7d29 in Object.wait() [0x00007f9b8af6f000]
201. java.lang.Thread.State: TIMED_WAITING (on object monitor)
202. at java.lang.Object.wait(Native Method)
203. at java.util.TimerThread.mainLoop(Timer.java:552)
204. - locked <0x0000000651b3ede0> (a java.util.TaskQueue)
205. at java.util.TimerThread.run(Timer.java:505)
206. "LogFDsCacheFlushTimer" #6657 daemon prio=5 os_prio=0 tid=0x00007f9bc4150000 nid=0x7d28 in Object.wait() [0x00007f9b8bd7d000]
207. java.lang.Thread.State: TIMED_WAITING (on object monitor)
208. at java.lang.Object.wait(Native Method)
209. at java.util.TimerThread.mainLoop(Timer.java:552)
210. - locked <0x0000000651b3f0f8> (a java.util.TaskQueue)
211. at java.util.TimerThread.run(Timer.java:505)
212. "LogFDsCachecleanInActiveFDsTimer" #6648 daemon prio=5 os_prio=0 tid=0x00007f9bc4cd9800 nid=0x7ced in Object.wait() [0x00007f9b92cec000]
213. java.lang.Thread.State: TIMED_WAITING (on object monitor)
214. at java.lang.Object.wait(Native Method)
215. at java.util.TimerThread.mainLoop(Timer.java:552)
216. - locked <0x000000065157fba0> (a java.util.TaskQueue)
217. at java.util.TimerThread.run(Timer.java:505)
218. "LogFDsCacheFlushTimer" #6647 daemon prio=5 os_prio=0 tid=0x00007f9bc4157000 nid=0x7cec in Object.wait() [0x00007f9b8b171000]
219. java.lang.Thread.State: TIMED_WAITING (on object monitor)
220. at java.lang.Object.wait(Native Method)
221. at java.util.TimerThread.mainLoop(Timer.java:552)
222. - locked <0x0000000650e40660> (a java.util.TaskQueue)
223. at java.util.TimerThread.run(Timer.java:505)
224. "LogFDsCachecleanInActiveFDsTimer" #6635 daemon prio=5 os_prio=0 tid=0x00007f9bc4c37800 nid=0x7cda in Object.wait() [0x00007f9b8a262000]
225. java.lang.Thread.State: TIMED_WAITING (on object monitor)
226. at java.lang.Object.wait(Native Method)
227. at java.util.TimerThread.mainLoop(Timer.java:552)
228. - locked <0x0000000650edd280> (a java.util.TaskQueue)
229. at java.util.TimerThread.run(Timer.java:505)
230. "LogFDsCacheFlushTimer" #6634 daemon prio=5 os_prio=0 tid=0x00007f9bc49a5800 nid=0x7cd9 in Object.wait() [0x00007f9b8a363000]
231. java.lang.Thread.State: TIMED_WAITING (on object monitor)
232. at java.lang.Object.wait(Native Method)
233. at java.util.TimerThread.mainLoop(Timer.java:552)
234. - locked <0x0000000650edd598> (a java.util.TaskQueue)
235. at java.util.TimerThread.run(Timer.java:505)
236. "LogFDsCachecleanInActiveFDsTimer" #6625 daemon prio=5 os_prio=0 tid=0x00007f9bc413a800 nid=0x7cca in Object.wait() [0x00007f9b8b878000]
237. java.lang.Thread.State: TIMED_WAITING (on object monitor)
238. at java.lang.Object.wait(Native Method)
239. at java.util.TimerThread.mainLoop(Timer.java:552)
240. - locked <0x0000000650f10c88> (a java.util.TaskQueue)
241. at java.util.TimerThread.run(Timer.java:505)
242. "LogFDsCacheFlushTimer" #6624 daemon prio=5 os_prio=0 tid=0x00007f9bc472d800 nid=0x7cc9 in Object.wait() [0x00007f9b8d393000]
243. java.lang.Thread.State: TIMED_WAITING (on object monitor)
244. at java.lang.Object.wait(Native Method)
245. at java.util.TimerThread.mainLoop(Timer.java:552)
246. - locked <0x0000000650f21828> (a java.util.TaskQueue)
247. at java.util.TimerThread.run(Timer.java:505)
248. "LogFDsCachecleanInActiveFDsTimer" #6613 daemon prio=5 os_prio=0 tid=0x00007f9bc4d13800 nid=0x7cbd in Object.wait() [0x00007f9b8e1a1000]
249. java.lang.Thread.State: TIMED_WAITING (on object monitor)
250. at java.lang.Object.wait(Native Method)
251. at java.util.TimerThread.mainLoop(Timer.java:552)
252. - locked <0x0000000650f5e5a0> (a java.util.TaskQueue)
253. at java.util.TimerThread.run(Timer.java:505)
254. "LogFDsCacheFlushTimer" #6612 daemon prio=5 os_prio=0 tid=0x00007f9bc4dfe000 nid=0x7cbc in Object.wait() [0x00007f9b8bf7f000]
255. java.lang.Thread.State: TIMED_WAITING (on object monitor)
256. at java.lang.Object.wait(Native Method)
257. at java.util.TimerThread.mainLoop(Timer.java:552)
258. - locked <0x0000000650f23030> (a java.util.TaskQueue)
259. at java.util.TimerThread.run(Timer.java:505)
260. "LogFDsCachecleanInActiveFDsTimer" #6608 daemon prio=5 os_prio=0 tid=0x00007f9bc4667000 nid=0x7cb7 in Object.wait() [0x00007f9b8a969000]
261. java.lang.Thread.State: TIMED_WAITING (on object monitor)
262. at java.lang.Object.wait(Native Method)
263. at java.util.TimerThread.mainLoop(Timer.java:552)
264. - locked <0x000000065192ebc8> (a java.util.TaskQueue)
265. at java.util.TimerThread.run(Timer.java:505)
266. "LogFDsCacheFlushTimer" #6607 daemon prio=5 os_prio=0 tid=0x00007f9bc4666800 nid=0x7cb6 in Object.wait() [0x00007f9b8aa6a000]
267. java.lang.Thread.State: TIMED_WAITING (on object monitor)
268. at java.lang.Object.wait(Native Method)
269. at java.util.TimerThread.mainLoop(Timer.java:552)
270. - locked <0x000000065192eee0> (a java.util.TaskQueue)
271. at java.util.TimerThread.run(Timer.java:505)
272. "LogFDsCachecleanInActiveFDsTimer" #6598 daemon prio=5 os_prio=0 tid=0x00007f9bc429a000 nid=0x7cac in Object.wait() [0x00007f9b8ba7a000]
273. java.lang.Thread.State: TIMED_WAITING (on object monitor)
274. at java.lang.Object.wait(Native Method)
275. at java.util.TimerThread.mainLoop(Timer.java:552)
276. - locked <0x00000006519153a0> (a java.util.TaskQueue)
277. at java.util.TimerThread.run(Timer.java:505)
278. "LogFDsCacheFlushTimer" #6597 daemon prio=5 os_prio=0 tid=0x00007f9bc4295000 nid=0x7cab in Object.wait() [0x00007f9b91ede000]
279. java.lang.Thread.State: TIMED_WAITING (on object monitor)
280. at java.lang.Object.wait(Native Method)
281. at java.util.TimerThread.mainLoop(Timer.java:552)
282. - locked <0x000000065191af28> (a java.util.TaskQueue)
283. at java.util.TimerThread.run(Timer.java:505)
284. "LogFDsCachecleanInActiveFDsTimer" #6585 daemon prio=5 os_prio=0 tid=0x00007f9bc4148000 nid=0x7c9d in Object.wait() [0x00007f9b8ad6d000]
285. java.lang.Thread.State: TIMED_WAITING (on object monitor)
286. at java.lang.Object.wait(Native Method)
287. at java.util.TimerThread.mainLoop(Timer.java:552)
288. - locked <0x0000000650f652f0> (a java.util.TaskQueue)
289. at java.util.TimerThread.run(Timer.java:505)
290. "LogFDsCacheFlushTimer" #6584 daemon prio=5 os_prio=0 tid=0x00007f9bc4afb800 nid=0x7c9c in Object.wait() [0x00007f9b8ae6e000]
291. java.lang.Thread.State: TIMED_WAITING (on object monitor)
292. at java.lang.Object.wait(Native Method)
293. at java.util.TimerThread.mainLoop(Timer.java:552)
294. - locked <0x0000000650f71978> (a java.util.TaskQueue)
295. at java.util.TimerThread.run(Timer.java:505)
296. "LogFDsCachecleanInActiveFDsTimer" #6575 daemon prio=5 os_prio=0 tid=0x00007f9bc427f800 nid=0x7c91 in Object.wait() [0x00007f9b8c585000]
297. java.lang.Thread.State: TIMED_WAITING (on object monitor)
298. at java.lang.Object.wait(Native Method)
299. at java.util.TimerThread.mainLoop(Timer.java:552)
300. - locked <0x000000065186c610> (a java.util.TaskQueue)
301. at java.util.TimerThread.run(Timer.java:505)
302. "LogFDsCacheFlushTimer" #6574 daemon prio=5 os_prio=0 tid=0x00007f9bc4157800 nid=0x7c90 in Object.wait() [0x00007f9b8d191000]
303. java.lang.Thread.State: TIMED_WAITING (on object monitor)
304. at java.lang.Object.wait(Native Method)
305. at java.util.TimerThread.mainLoop(Timer.java:552)
306. - locked <0x00000006509b0950> (a java.util.TaskQueue)
307. at java.util.TimerThread.run(Timer.java:505)
308. "LogFDsCachecleanInActiveFDsTimer" #6563 daemon prio=5 os_prio=0 tid=0x00007f9bc4155000 nid=0x7c55 in Object.wait() [0x00007f9b90ece000]
309. java.lang.Thread.State: TIMED_WAITING (on object monitor)
310. at java.lang.Object.wait(Native Method)
311. at java.util.TimerThread.mainLoop(Timer.java:552)
312. - locked <0x0000000650a81930> (a java.util.TaskQueue)
313. at java.util.TimerThread.run(Timer.java:505)
314. "LogFDsCacheFlushTimer" #6562 daemon prio=5 os_prio=0 tid=0x00007f9bc4524800 nid=0x7c54 in Object.wait() [0x00007f9b8d898000]
315. java.lang.Thread.State: TIMED_WAITING (on object monitor)
316. at java.lang.Object.wait(Native Method)
317. at java.util.TimerThread.mainLoop(Timer.java:552)
318. - locked <0x0000000650a81a38> (a java.util.TaskQueue)
319. at java.util.TimerThread.run(Timer.java:505)
320. "LogFDsCachecleanInActiveFDsTimer" #6558 daemon prio=5 os_prio=0 tid=0x00007f9bc4d11800 nid=0x7c4e in Object.wait() [0x00007f9b926e6000]
321. java.lang.Thread.State: TIMED_WAITING (on object monitor)
322. at java.lang.Object.wait(Native Method)
323. at java.util.TimerThread.mainLoop(Timer.java:552)
324. - locked <0x00000006514872a8> (a java.util.TaskQueue)
325. at java.util.TimerThread.run(Timer.java:505)
326. "LogFDsCacheFlushTimer" #6557 daemon prio=5 os_prio=0 tid=0x00007f9bc43ec000 nid=0x7c4d in Object.wait() [0x00007f9b8cb8b000]
327. java.lang.Thread.State: TIMED_WAITING (on object monitor)
328. at java.lang.Object.wait(Native Method)
329. at java.util.TimerThread.mainLoop(Timer.java:552)
330. - locked <0x0000000651486d68> (a java.util.TaskQueue)
331. at java.util.TimerThread.run(Timer.java:505)
332. "LogFDsCachecleanInActiveFDsTimer" #6548 daemon prio=5 os_prio=0 tid=0x00007f9bc4229000 nid=0x7c37 in Object.wait() [0x00007f9b8b575000]
333. java.lang.Thread.State: TIMED_WAITING (on object monitor)
334. at java.lang.Object.wait(Native Method)
335. at java.util.TimerThread.mainLoop(Timer.java:552)
336. - locked <0x0000000650a82590> (a java.util.TaskQueue)
337. at java.util.TimerThread.run(Timer.java:505)
338. "LogFDsCacheFlushTimer" #6547 daemon prio=5 os_prio=0 tid=0x00007f9bc4228800 nid=0x7c36 in Object.wait() [0x00007f9b8b676000]
339. java.lang.Thread.State: TIMED_WAITING (on object monitor)
340. at java.lang.Object.wait(Native Method)
341. at java.util.TimerThread.mainLoop(Timer.java:552)
342. - locked <0x0000000650aa7030> (a java.util.TaskQueue)
343. at java.util.TimerThread.run(Timer.java:505)
344. "LogFDsCachecleanInActiveFDsTimer" #6534 daemon prio=5 os_prio=0 tid=0x00007f9bc495d000 nid=0x79e9 in Object.wait() [0x00007f9b8b979000]
345. java.lang.Thread.State: TIMED_WAITING (on object monitor)
346. at java.lang.Object.wait(Native Method)
347. at java.util.TimerThread.mainLoop(Timer.java:552)
348. - locked <0x0000000650aa7cf8> (a java.util.TaskQueue)
349. at java.util.TimerThread.run(Timer.java:505)
350. "LogFDsCacheFlushTimer" #6533 daemon prio=5 os_prio=0 tid=0x00007f9bc472a800 nid=0x79e8 in Object.wait() [0x00007f9b8ce8e000]
351. java.lang.Thread.State: TIMED_WAITING (on object monitor)
352. at java.lang.Object.wait(Native Method)
353. at java.util.TimerThread.mainLoop(Timer.java:552)
354. - locked <0x0000000650aa7e00> (a java.util.TaskQueue)
355. at java.util.TimerThread.run(Timer.java:505)
356. "LogFDsCachecleanInActiveFDsTimer" #6524 daemon prio=5 os_prio=0 tid=0x00007f9bc4141000 nid=0x79dd in Object.wait() [0x00007f9b8f1b1000]
357. java.lang.Thread.State: TIMED_WAITING (on object monitor)
358. at java.lang.Object.wait(Native Method)
359. at java.util.TimerThread.mainLoop(Timer.java:552)
360. - locked <0x00000006513f4768> (a java.util.TaskQueue)
361. at java.util.TimerThread.run(Timer.java:505)
362. "LogFDsCacheFlushTimer" #6523 daemon prio=5 os_prio=0 tid=0x00007f9bc54b3800 nid=0x79dc in Object.wait() [0x00007f9b904c4000]
363. java.lang.Thread.State: TIMED_WAITING (on object monitor)
364. at java.lang.Object.wait(Native Method)
365. at java.util.TimerThread.mainLoop(Timer.java:552)
366. - locked <0x00000006513fe7b8> (a java.util.TaskQueue)
367. at java.util.TimerThread.run(Timer.java:505)
368. "LogFDsCachecleanInActiveFDsTimer" #6507 daemon prio=5 os_prio=0 tid=0x00007f9bc4aff800 nid=0x799a in Object.wait() [0x00007f9b8bb7b000]
369. java.lang.Thread.State: TIMED_WAITING (on object monitor)
370. at java.lang.Object.wait(Native Method)
371. at java.util.TimerThread.mainLoop(Timer.java:552)
372. - locked <0x0000000650f71e90> (a java.util.TaskQueue)
373. at java.util.TimerThread.run(Timer.java:505)
374. "LogFDsCacheFlushTimer" #6506 daemon prio=5 os_prio=0 tid=0x00007f9bc429b000 nid=0x7999 in Object.wait() [0x00007f9b8bc7c000]
375. java.lang.Thread.State: TIMED_WAITING (on object monitor)
376. at java.lang.Object.wait(Native Method)
377. at java.util.TimerThread.mainLoop(Timer.java:552)
378. - locked <0x0000000650f8ca40> (a java.util.TaskQueue)
379. at java.util.TimerThread.run(Timer.java:505)
380. "LogFDsCachecleanInActiveFDsTimer" #6497 daemon prio=5 os_prio=0 tid=0x00007f9bc5167800 nid=0x798e in Object.wait() [0x00007f9b8c181000]
381. java.lang.Thread.State: TIMED_WAITING (on object monitor)
382. at java.lang.Object.wait(Native Method)
383. at java.util.TimerThread.mainLoop(Timer.java:552)
384. - locked <0x0000000650f8cf38> (a java.util.TaskQueue)
385. at java.util.TimerThread.run(Timer.java:505)
386. "LogFDsCacheFlushTimer" #6496 daemon prio=5 os_prio=0 tid=0x00007f9bc5167000 nid=0x798d in Object.wait() [0x00007f9b8c282000]
387. java.lang.Thread.State: TIMED_WAITING (on object monitor)
388. at java.lang.Object.wait(Native Method)
389. at java.util.TimerThread.mainLoop(Timer.java:552)
390. - locked <0x0000000650f72570> (a java.util.TaskQueue)
391. at java.util.TimerThread.run(Timer.java:505)
392. "LogFDsCachecleanInActiveFDsTimer" #6485 daemon prio=5 os_prio=0 tid=0x00007f9bc5360800 nid=0x797b in Object.wait() [0x00007f9b8e8a8000]
393. java.lang.Thread.State: TIMED_WAITING (on object monitor)
394. at java.lang.Object.wait(Native Method)
395. at java.util.TimerThread.mainLoop(Timer.java:552)
396. - locked <0x00000006509b1f68> (a java.util.TaskQueue)
397. at java.util.TimerThread.run(Timer.java:505)
398. "LogFDsCacheFlushTimer" #6484 daemon prio=5 os_prio=0 tid=0x00007f9bc4c45800 nid=0x797a in Object.wait() [0x00007f9b8febe000]
399. java.lang.Thread.State: TIMED_WAITING (on object monitor)
400. at java.lang.Object.wait(Native Method)
401. at java.util.TimerThread.mainLoop(Timer.java:552)
402. - locked <0x00000006509b1d80> (a java.util.TaskQueue)
403. at java.util.TimerThread.run(Timer.java:505)
404. "LogFDsCachecleanInActiveFDsTimer" #6475 daemon prio=5 os_prio=0 tid=0x00007f9bc5531000 nid=0x796f in Object.wait() [0x00007f9b8fdbd000]
405. java.lang.Thread.State: TIMED_WAITING (on object monitor)
406. at java.lang.Object.wait(Native Method)
407. at java.util.TimerThread.mainLoop(Timer.java:552)
408. - locked <0x00000006509ec3c8> (a java.util.TaskQueue)
409. at java.util.TimerThread.run(Timer.java:505)
410. "LogFDsCacheFlushTimer" #6474 daemon prio=5 os_prio=0 tid=0x00007f9bc4528800 nid=0x796e in Object.wait() [0x00007f9b8f2b2000]
411. java.lang.Thread.State: TIMED_WAITING (on object monitor)
412. at java.lang.Object.wait(Native Method)
413. at java.util.TimerThread.mainLoop(Timer.java:552)
414. - locked <0x00000006509ec560> (a java.util.TaskQueue)
415. at java.util.TimerThread.run(Timer.java:505)
416. "LogFDsCachecleanInActiveFDsTimer" #6463 daemon prio=5 os_prio=0 tid=0x00007f9bc44c5800 nid=0x7963 in Object.wait() [0x00007f9b922e2000]
417. java.lang.Thread.State: TIMED_WAITING (on object monitor)
418. at java.lang.Object.wait(Native Method)
419. at java.util.TimerThread.mainLoop(Timer.java:552)
420. - locked <0x00000006509ddb10> (a java.util.TaskQueue)
421. at java.util.TimerThread.run(Timer.java:505)
422. "LogFDsCacheFlushTimer" #6462 daemon prio=5 os_prio=0 tid=0x00007f9bc431d800 nid=0x7962 in Object.wait() [0x00007f9b8dd9d000]
423. java.lang.Thread.State: TIMED_WAITING (on object monitor)
424. at java.lang.Object.wait(Native Method)
425. at java.util.TimerThread.mainLoop(Timer.java:552)
426. - locked <0x00000006509b2700> (a java.util.TaskQueue)
427. at java.util.TimerThread.run(Timer.java:505)
428. "LogFDsCachecleanInActiveFDsTimer" #6457 daemon prio=5 os_prio=0 tid=0x00007f9bc4159000 nid=0x7956 in Object.wait() [0x00007f9b8c888000]
429. java.lang.Thread.State: TIMED_WAITING (on object monitor)
430. at java.lang.Object.wait(Native Method)
431. at java.util.TimerThread.mainLoop(Timer.java:552)
432. - locked <0x00000006509a6980> (a java.util.TaskQueue)
433. at java.util.TimerThread.run(Timer.java:505)
434. "LogFDsCacheFlushTimer" #6456 daemon prio=5 os_prio=0 tid=0x00007f9bc4c3f800 nid=0x7955 in Object.wait() [0x00007f9b8c989000]
435. java.lang.Thread.State: TIMED_WAITING (on object monitor)
436. at java.lang.Object.wait(Native Method)
437. at java.util.TimerThread.mainLoop(Timer.java:552)
438. - locked <0x00000006509a6c98> (a java.util.TaskQueue)
439. at java.util.TimerThread.run(Timer.java:505)
440. "LogFDsCachecleanInActiveFDsTimer" #6447 daemon prio=5 os_prio=0 tid=0x00007f9bc43ea800 nid=0x7948 in Object.wait() [0x00007f9b8cc8c000]
441. java.lang.Thread.State: TIMED_WAITING (on object monitor)
442. at java.lang.Object.wait(Native Method)
443. at java.util.TimerThread.mainLoop(Timer.java:552)
444. - locked <0x00000006506dc628> (a java.util.TaskQueue)
445. at java.util.TimerThread.run(Timer.java:505)
446. "LogFDsCacheFlushTimer" #6446 daemon prio=5 os_prio=0 tid=0x00007f9bc42c6000 nid=0x7947 in Object.wait() [0x00007f9b8cd8d000]
447. java.lang.Thread.State: TIMED_WAITING (on object monitor)
448. at java.lang.Object.wait(Native Method)
449. at java.util.TimerThread.mainLoop(Timer.java:552)
450. - locked <0x00000006509a8c10> (a java.util.TaskQueue)
451. at java.util.TimerThread.run(Timer.java:505)
452. "LogFDsCachecleanInActiveFDsTimer" #6435 daemon prio=5 os_prio=0 tid=0x00007f9bc52e7000 nid=0x793c in Object.wait() [0x00007f9b8d999000]
453. java.lang.Thread.State: TIMED_WAITING (on object monitor)
454. at java.lang.Object.wait(Native Method)
455. at java.util.TimerThread.mainLoop(Timer.java:552)
456. - locked <0x00000006509a7d68> (a java.util.TaskQueue)
457. at java.util.TimerThread.run(Timer.java:505)
458. "LogFDsCacheFlushTimer" #6434 daemon prio=5 os_prio=0 tid=0x00007f9bc4755000 nid=0x793b in Object.wait() [0x00007f9b94908000]
459. java.lang.Thread.State: TIMED_WAITING (on object monitor)
460. at java.lang.Object.wait(Native Method)
461. at java.util.TimerThread.mainLoop(Timer.java:552)
462. - locked <0x00000006509a8080> (a java.util.TaskQueue)
463. at java.util.TimerThread.run(Timer.java:505)
464. "LogFDsCachecleanInActiveFDsTimer" #6429 daemon prio=5 os_prio=0 tid=0x00007f9bc42ca000 nid=0x7904 in Object.wait() [0x00007f9b8cf8f000]
465. java.lang.Thread.State: TIMED_WAITING (on object monitor)
466. at java.lang.Object.wait(Native Method)
467. at java.util.TimerThread.mainLoop(Timer.java:552)
468. - locked <0x00000006506c6ec8> (a java.util.TaskQueue)
469. at java.util.TimerThread.run(Timer.java:505)
470. "LogFDsCacheFlushTimer" #6428 daemon prio=5 os_prio=0 tid=0x00007f9bc42c9800 nid=0x7903 in Object.wait() [0x00007f9b8d090000]
471. java.lang.Thread.State: TIMED_WAITING (on object monitor)
472. at java.lang.Object.wait(Native Method)
473. at java.util.TimerThread.mainLoop(Timer.java:552)
474. - locked <0x00000006506c71e0> (a java.util.TaskQueue)
475. at java.util.TimerThread.run(Timer.java:505)
476. "LogFDsCachecleanInActiveFDsTimer" #6419 daemon prio=5 os_prio=0 tid=0x00007f9bc472c800 nid=0x78f6 in Object.wait() [0x00007f9b8d595000]
477. java.lang.Thread.State: TIMED_WAITING (on object monitor)
478. at java.lang.Object.wait(Native Method)
479. at java.util.TimerThread.mainLoop(Timer.java:552)
480. - locked <0x00000006506c7b90> (a java.util.TaskQueue)
481. at java.util.TimerThread.run(Timer.java:505)
482. "LogFDsCacheFlushTimer" #6418 daemon prio=5 os_prio=0 tid=0x00007f9bc472b800 nid=0x78f5 in Object.wait() [0x00007f9b8d696000]
483. java.lang.Thread.State: TIMED_WAITING (on object monitor)
484. at java.lang.Object.wait(Native Method)
485. at java.util.TimerThread.mainLoop(Timer.java:552)
486. - locked <0x00000006506cf0c8> (a java.util.TaskQueue)
487. at java.util.TimerThread.run(Timer.java:505)

이렇게 

LogFDsCacheFlushTimer와 LogFDsCachecleanInActiveFDsTimer

가 무한 생성되고 TIMED_WAITING이 걸려 있다.

결구 모르겠다

하고 Hortonworks 포럼에 이 사실을 올렸다.

결과 빠른 답변

https://issues.apache.org/jira/browse/TEZ-3017

https://issues.apache.org/jira/browse/HIVE-12766

위와 같은 이슈에 등록 되어 있고 hdp 2.4 version에서 고쳐서 나온다고 한다.

그전에 해결 방법은 

YARN=> Config=> Advanced 에서

yarn.timeline-service.enabled 에 체크를 해제 해주면 된다.

일단 증상은 해결된 것으로 보인다.

하지만 timeline service를 사용 못해서 tez log 를 확인 하는데 약간의 어려움은 있다.