[Configuration parameters] io.druid.cli.CliPeon - Error when starting up. Failing

Apache Druid Q&A Google Groups
#1

Hi,

I am trying to test Druid and to test its functionalities on a small sample of data. I followed the quickstart guide to load data from imply.io (http://imply.io/docs/latest/quickstart.html). Everything is running fine on my Macbook. I can load the test data and perform query with it.

However, when I run the same configuration on a Unix machine (Red Hat Entreprise Linux), I am encountering problems. I get a “java.lang.OutOfMemoryError” (see below). I tried to tweak memory settings but I could not figure out a working configuration. Any input?

Here is the stacktrace:

2015-12-21T08:22:28,798 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.global.http.] as [io.druid.guice.http.DruidHttpClientConfig@23ee75c5]

2015-12-21T08:22:29,119 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method) ~[?:1.8.0_05]

at java.lang.Thread.start(Unknown Source) ~[?:1.8.0_05]

at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) ~[?:1.8.0_05]

at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:1.8.0_05]

at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80) ~[netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39) ~[netty-3.10.4.Final.jar:?]

at com.metamx.http.client.HttpClientInit.createBootstrap(HttpClientInit.java:133) ~[http-client-1.0.4.jar:?]

at com.metamx.http.client.HttpClientInit.createClient(HttpClientInit.java:85) ~[http-client-1.0.4.jar:?]

at io.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:114) ~[druid-server-0.8.2.jar:0.8.2]

at io.druid.guice.http.HttpClientModule$HttpClientProvider.get(HttpClientModule.java:84) ~[druid-server-0.8.2.jar:0.8.2]

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]

at io.druid.guice.PolyBind$ConfiggedProvider.get(PolyBind.java:178) ~[druid-api-0.3.13.jar:0.8.2]

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at com.google.inject.Scopes$1$1.get(Scopes.java:65) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) ~[druid-api-0.3.13.jar:0.8.2]

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001) ~[guice-4.0-beta.jar:?]

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:153) ~[druid-api-0.3.13.jar:0.8.2]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.8.2.jar:0.8.2]

at io.druid.cli.CliPeon.run(CliPeon.java:211) [druid-services-0.8.2.jar:0.8.2]

at io.druid.cli.Main.main(Main.java:91) [druid-services-0.8.2.jar:0.8.2]

#2

By playing around, I now get this error which doesn’t say much about which process need extra memory:

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,631,367,680], memoryNeeded[51,539,607,552] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[47] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:75)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)

while locating io.druid.query.groupby.GroupByQueryEngine

for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:76)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:131)

while locating io.druid.indexing.common.TaskToolboxFactory

for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)

while locating io.druid.server.QueryResource

#3

Hi Gael,

By playing around, I now get this error which doesn’t say much about which process need extra memory:

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,631,367,680], memoryNeeded[51,539,607,552] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[47] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:75)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)

while locating io.druid.query.groupby.GroupByQueryEngine

for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:76)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:131)

while locating io.druid.indexing.common.TaskToolboxFactory

for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)

while locating io.druid.server.QueryResource

In your jvm settings or runtime.properties Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,631,367,680], memoryNeeded[51,539,607,552] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[47] + 1 )

#4

The problem is not that I cannot do the math but which conf file should I configure out of the 6?

  • broker
  • _common
  • coordinator
  • historical
  • middleManager
  • overlord
    There are no config that has numThreads=47 neither any of the memory size specified and no attemps at setting -XX:MaxDirectMemorySize changes anything for any of the conf file.
#5

Hi Gael,
I had seen a similar issue and you might want to look at the javaOpts property over here http://druid.io/docs/latest/configuration/indexing-service.html#middlemanager-configs to set the direct memory for individual peons in JVM.

For the number of threads and buffer size I think these worked for me

druid.indexer.fork.property.druid.processing.buffer.sizeBytes

druid.indexer.fork.property.druid.processing.numThreads

I fixed it in the middle manager configs. Hope this helps.

Thanks,

Ram

#6

Hi Gael, you shouldn’t need to do math. The numbers you need to set are printed in that error message. Can you share an example of a config where you have problems and the error message you see when you start things up?

#7

Hi,I meet the same issue,Could you give some details about config?
Thanks