HDP Hiveserver2 JAVA heap Error

HDP <HDP-2.3.2.0-2950> 버젼을 운용 중 발생 중인 문제점..

Hiveserver2에서 3주 정도 지나면 Java Heap Memory Error를 뱉는다..

여기에 설치된 Hive 버젼은 1.2.1.2.3

에러 메시지는 다음과 같다.

Java.lang.OutOfMemoryError: Java heap space 2016-03-04 10:45:23,530 WARN [HiveServer2-Handler-Pool: Thread-675783]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error fetching results: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:83) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy19.fetchResults(Unknown Source) at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:454) at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyPrimitiveClass(LazyFactory.java:121) at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObject(LazyFactory.java:177) at org.apache.hadoop.hive.serde2.lazy.LazyArray.uncheckedGetElement(LazyArray.java:185) at org.apache.hadoop.hive.serde2.lazy.LazyArray.getList(LazyArray.java:226) at org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyListObjectInspector.getList(LazyListObjectInspector.java:96) at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:298) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:198) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:184) at org.apache.hadoop.hive.serde2.SerDeUtils.toThriftPayload(SerDeUtils.java:180) at org.apache.hadoop.hive.ql.exec.FetchFormatter$ThriftFormatter.convert(FetchFormatter.java:61) at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:90) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:425) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:417) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672) at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347) at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:221) at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:685) at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy19.fetchResults(Unknown Source) at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:454) at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)

신기 한 것은 한번의 주기 작업 시 두번의 쿼리를 시행하게 되는데

첫번째 쿼리는 동작하고 두 번째 쿼리에서 실행이 안된다는 것이다....

몇 일을 이 문제를 해결하기 위해 고민하다가 결국 HortonWork Forum에 글을 올렸다.

하지만 돌아 오는 대답은... 현재 이 버젼에 대해서 Memory leak을 못찾겠다는 것이다... 흠...

Support 팀에게 문의를 하라고 하는데..

현재 여력이 안되어 소스를 직접 들여다 보기도 애매한 상황이다...

결국은 버젼 업데이트가 필요 할 것으로 보이고 그렇게 조치될 것 같은 이 불안한 상황이...