Ingest Stream data sometimes working sometimes failed

Hi Team,

I have a weird error threw out in the S3 logs ( store index log ) ,it is displaying “run out the memory” for CliPeon. However, I don’t see that error displaying in the logs anywhere in Overlord,Middle Manager, Historical, Tranquility server, Coordinator…

Also, I am checking all resources (RAM, CPU …) all the nodes seem normal to me .

I am using stream ingestion ( from Kinesis Stream) viva Tranquility Server with HTTP POST, it sometimes the ingestion success sometimes don’t.

Thanks

Tom

019-01-15T01:08:19,501 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[interface org.apache.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [org.apache.druid.server.log.NoopRequestLoggerProvider@7767bd4e]

2019-01-15T01:08:19,504 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up. Failing.

com.google.inject.ProvisionException: Unable to provision, see the following errors:

  1. Error in custom provider, java.lang.OutOfMemoryError

at org.apache.druid.guice.DruidProcessingModule.getMergeBufferPool(DruidProcessingModule.java:131) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.DruidProcessingModule)

at org.apache.druid.guice.DruidProcessingModule.getMergeBufferPool(DruidProcessingModule.java:131) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.DruidProcessingModule)

while locating org.apache.druid.collections.BlockingPool<java.nio.ByteBuffer> annotated with @org.apache.druid.guice.annotations.Merging()

for the 4th parameter of org.apache.druid.query.groupby.strategy.GroupByStrategyV2.(GroupByStrategyV2.java:108)

while locating org.apache.druid.query.groupby.strategy.GroupByStrategyV2

for the 3rd parameter of org.apache.druid.query.groupby.strategy.GroupByStrategySelector.(GroupByStrategySelector.java:44)

while locating org.apache.druid.query.groupby.strategy.GroupByStrategySelector

for the 1st parameter of org.apache.druid.query.groupby.GroupByQueryQueryToolChest.(GroupByQueryQueryToolChest.java:106)

at org.apache.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:101) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryRunnerFactoryModule)

while locating org.apache.druid.query.groupby.GroupByQueryQueryToolChest

while locating org.apache.druid.query.QueryToolChest annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=78, type=MAPBINDER, keyType=java.lang.Class<? extends org.apache.druid.query.Query>)

at org.apache.druid.guice.DruidBinders.queryToolChestBinder(DruidBinders.java:45) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryRunnerFactoryModule -> com.google.inject.multibindings.MapBinder$RealMapBinder)

while locating java.util.Map<java.lang.Class<? extends org.apache.druid.query.Query>, org.apache.druid.query.QueryToolChest>

for the 1st parameter of org.apache.druid.query.MapQueryToolChestWarehouse.(MapQueryToolChestWarehouse.java:36)

while locating org.apache.druid.query.MapQueryToolChestWarehouse

while locating org.apache.druid.query.QueryToolChestWarehouse

for the 1st parameter of org.apache.druid.server.QueryLifecycleFactory.(QueryLifecycleFactory.java:52)

at org.apache.druid.server.QueryLifecycleFactory.class(QueryLifecycleFactory.java:52)

while locating org.apache.druid.server.QueryLifecycleFactory

for the 1st parameter of org.apache.druid.server.QueryResource.(QueryResource.java:115)

at org.apache.druid.server.QueryResource.class(QueryResource.java:80)

while locating org.apache.druid.server.QueryResource

1 error

at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]

at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:132) ~[druid-api-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:107) [druid-services-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.cli.CliPeon.run(CliPeon.java:348) [druid-services-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.cli.Main.main(Main.java:118) [druid-services-0.13.0-incubating.jar:0.13.0-incubating]

Caused by: java.lang.OutOfMemoryError

at sun.misc.Unsafe.allocateMemory(Native Method) ~[?:1.8.0_191]

at java.nio.DirectByteBuffer.(DirectByteBuffer.java:127) ~[?:1.8.0_191]

at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_191]

at org.apache.druid.offheap.OffheapBufferGenerator.get(OffheapBufferGenerator.java:52) ~[dru

Tom,
Other link similar to your issue. It might help you.

https://groups.google.com/forum/#!topic/druid-user/0yda77Y3Va0

Thanks,

–siva

Tom,
Another link – https://groups.google.com/forum/#!topic/druid-user/eKLjJw8G6Dw – talks about modifying MM runtime properties to get rid of OOM error.

Thanks,

–siva