Hi Team,
I have a weird error threw out in the S3 logs ( store index log ) ,it is displaying “run out the memory” for CliPeon. However, I don’t see that error displaying in the logs anywhere in Overlord,Middle Manager, Historical, Tranquility server, Coordinator…
Also, I am checking all resources (RAM, CPU …) all the nodes seem normal to me .
I am using stream ingestion ( from Kinesis Stream) viva Tranquility Server with HTTP POST, it sometimes the ingestion success sometimes don’t.
Thanks
Tom
019-01-15T01:08:19,501 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[interface org.apache.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [org.apache.druid.server.log.NoopRequestLoggerProvider@7767bd4e]
2019-01-15T01:08:19,504 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up. Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:
- Error in custom provider, java.lang.OutOfMemoryError
at org.apache.druid.guice.DruidProcessingModule.getMergeBufferPool(DruidProcessingModule.java:131) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.DruidProcessingModule)
at org.apache.druid.guice.DruidProcessingModule.getMergeBufferPool(DruidProcessingModule.java:131) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.DruidProcessingModule)
while locating org.apache.druid.collections.BlockingPool<java.nio.ByteBuffer> annotated with @org.apache.druid.guice.annotations.Merging()
for the 4th parameter of org.apache.druid.query.groupby.strategy.GroupByStrategyV2.(GroupByStrategyV2.java:108)
while locating org.apache.druid.query.groupby.strategy.GroupByStrategyV2
for the 3rd parameter of org.apache.druid.query.groupby.strategy.GroupByStrategySelector.(GroupByStrategySelector.java:44)
while locating org.apache.druid.query.groupby.strategy.GroupByStrategySelector
for the 1st parameter of org.apache.druid.query.groupby.GroupByQueryQueryToolChest.(GroupByQueryQueryToolChest.java:106)
at org.apache.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:101) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryRunnerFactoryModule)
while locating org.apache.druid.query.groupby.GroupByQueryQueryToolChest
while locating org.apache.druid.query.QueryToolChest annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=78, type=MAPBINDER, keyType=java.lang.Class<? extends org.apache.druid.query.Query>)
at org.apache.druid.guice.DruidBinders.queryToolChestBinder(DruidBinders.java:45) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryRunnerFactoryModule -> com.google.inject.multibindings.MapBinder$RealMapBinder)
while locating java.util.Map<java.lang.Class<? extends org.apache.druid.query.Query>, org.apache.druid.query.QueryToolChest>
for the 1st parameter of org.apache.druid.query.MapQueryToolChestWarehouse.(MapQueryToolChestWarehouse.java:36)
while locating org.apache.druid.query.MapQueryToolChestWarehouse
while locating org.apache.druid.query.QueryToolChestWarehouse
for the 1st parameter of org.apache.druid.server.QueryLifecycleFactory.(QueryLifecycleFactory.java:52)
at org.apache.druid.server.QueryLifecycleFactory.class(QueryLifecycleFactory.java:52)
while locating org.apache.druid.server.QueryLifecycleFactory
for the 1st parameter of org.apache.druid.server.QueryResource.(QueryResource.java:115)
at org.apache.druid.server.QueryResource.class(QueryResource.java:80)
while locating org.apache.druid.server.QueryResource
1 error
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:132) ~[druid-api-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:107) [druid-services-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:348) [druid-services-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.cli.Main.main(Main.java:118) [druid-services-0.13.0-incubating.jar:0.13.0-incubating]
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method) ~[?:1.8.0_191]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:127) ~[?:1.8.0_191]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_191]
at org.apache.druid.offheap.OffheapBufferGenerator.get(OffheapBufferGenerator.java:52) ~[dru