[Configuration parameters] io.druid.cli.CliPeon - Error when starting up. Failing

Hi,

I am trying to test Druid and to test its functionalities on a small sample of data. I followed the quickstart guide to load data from imply.io (http://imply.io/docs/latest/quickstart.html). Everything is running fine on my Macbook. I can load the test data and perform query with it.

However, when I run the same configuration on a Unix machine (Red Hat Entreprise Linux), I am encountering problems. I get a “java.lang.OutOfMemoryError” (see below). I tried to tweak memory settings but I could not figure out a working configuration. Any input?

Here is the stacktrace:

2015-12-21T08:22:28,798 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.global.http.] as [io.druid.guice.http.DruidHttpClientConfig@23ee75c5]

2015-12-21T08:22:29,119 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method) ~[?:1.8.0_05]

at java.lang.Thread.start(Unknown Source) ~[?:1.8.0_05]

at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) ~[?:1.8.0_05]

at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:1.8.0_05]

at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39) ~[netty-3.10.4.Final.jar:?]

at com.metamx.http.client.HttpClientInit.createBootstrap(HttpClientInit.java:133) ~[http-client-1.0.4.jar:?]

at com.metamx.http.client.HttpClientInit.createClient(HttpClientInit.java:85) ~[http-client-1.0.4.jar:?]

at io.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:114) ~[druid-server-0.8.2.jar:0.8.2]

at io.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:84) ~[druid-server-0.8.2.jar:0.8.2]

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]

at io.druid.guice.PolyBind$ConfiggedProvider.get(PolyBind.java:178) ~[druid-api-0.3.13.jar:0.8.2]

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) ~[druid-api-0.3.13.jar:0.8.2]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:153) ~[druid-api-0.3.13.jar:0.8.2]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.8.2.jar:0.8.2]

at io.druid.cli.CliPeon.run(CliPeon.java:211) [druid-services-0.8.2.jar:0.8.2]

at io.druid.cli.Main.main(Main.java:91) [druid-services-0.8.2.jar:0.8.2]

By playing around, I now get this error which doesn’t say much about which process need extra memory:

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,631,367,680], memoryNeeded[51,539,607,552] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[47] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:75)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)

while locating io.druid.query.groupby.GroupByQueryEngine

for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:76)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:131)

while locating io.druid.indexing.common.TaskToolboxFactory

for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)

while locating io.druid.server.QueryResource

Hi Gael,

By playing around, I now get this error which doesn’t say much about which process need extra memory:

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,631,367,680], memoryNeeded[51,539,607,552] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[47] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:75)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)

while locating io.druid.query.groupby.GroupByQueryEngine

for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:76)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:131)

while locating io.druid.indexing.common.TaskToolboxFactory

for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)

while locating io.druid.server.QueryResource

In your jvm settings or runtime.properties Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,631,367,680], memoryNeeded[51,539,607,552] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[47] + 1 )

The problem is not that I cannot do the math but which conf file should I configure out of the 6?

  • broker
  • _common
  • coordinator
  • historical
  • middleManager
  • overlord
    There are no config that has numThreads=47 neither any of the memory size specified and no attemps at setting -XX:MaxDirectMemorySize changes anything for any of the conf file.

Hi Gael,
I had seen a similar issue and you might want to look at the javaOpts property over here http://druid.io/docs/latest/configuration/indexing-service.html#middlemanager-configs to set the direct memory for individual peons in JVM.

For the number of threads and buffer size I think these worked for me

druid.indexer.fork.property.druid.processing.buffer.sizeBytes

druid.indexer.fork.property.druid.processing.numThreads

I fixed it in the middle manager configs. Hope this helps.

Thanks,

Ram

Hi Gael, you shouldn’t need to do math. The numbers you need to set are printed in that error message. Can you share an example of a config where you have problems and the error message you see when you start things up?

Hi,I meet the same issue,Could you give some details about config?
Thanks