"Not enough direct memory" while launching Peon.

Hi,

When the middle manger launching Peon.

There are always some exception.

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[1,763,704,832], memoryNeeded[2,147,483,648] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[1] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 4 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:34)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 1 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:90)

while locating io.druid.server.QueryResource

I have changed “druid.indexer.runner.javaOpts” configure value, add set “MaxDirectMemorySize” to 3g(druid.indexer.runner.javaOpts="-server -Xmx3g -XX:MaxDirectMemorySize=3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"), but it doesn’t work.

I’m so confused about it.

Hi, what version of Druid is this?

Strange that the config is not getting picked up. What happens if you set “druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912” on your middle manager runtime.properties?

Hi, Fangjin:

The version of druid is"0.8.1-rc2", if I changed the config as you said, it can work now.

Hi, what version of Druid is this?

Hi,

When the middle manger launching Peon.

There are always some exception.

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[1,763,704,832], memoryNeeded[2,147,483,648] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[1] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 4 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:34)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 1 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:90)

while locating io.druid.server.QueryResource

I have changed “druid.indexer.runner.javaOpts” configure value, add set “MaxDirectMemorySize” to 3g(druid.indexer.runner.javaOpts="-server -Xmx3g -XX:MaxDirectMemorySize=3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"), but it doesn’t work.

I’m so confused about it.

Strange that the config is not getting picked up. What happens if you set “druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912” on your middle manager runtime.properties?

But it seems change “druid.indexer.runner.javaOpts” without effect

在 2015年11月2日星期一 UTC+8上午3:00:04,Fangjin Yang写道:

Does setting the property in the MM configs work?

The other config setting of MM is working except “druid.indexer.runner.javaOpts”.
Do you have some advice how to check this setting(druid.indexer.runner.javaOpts) changes?

Thanks

在 2015年11月2日星期一 UTC+8上午7:53:23,Fangjin Yang写道:

Hi, Fangjin:

It seems that it can work when the quotes removed.

Does setting the property in the MM configs work?

Hi, Fangjin:

The version of druid is"0.8.1-rc2", if I changed the config as you said, it can work now.

But it seems change “druid.indexer.runner.javaOpts” without effect

在 2015年11月2日星期一 UTC+8上午3:00:04,Fangjin Yang写道:

Hi, what version of Druid is this?

Strange that the config is not getting picked up. What happens if you set “druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912” on your middle manager runtime.properties?

Hi,

When the middle manger launching Peon.

There are always some exception.

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[1,763,704,832], memoryNeeded[2,147,483,648] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[1] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 4 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:34)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 1 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:90)

while locating io.druid.server.QueryResource

I have changed “druid.indexer.runner.javaOpts” configure value, add set “MaxDirectMemorySize” to 3g(druid.indexer.runner.javaOpts="-server -Xmx3g -XX:MaxDirectMemorySize=3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"), but it doesn’t work.

I’m so confused about it.

You received this message because you are subscribed to the Google Groups “Druid User” group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d373a8ac-9d68-43e0-9664-67fd49d61ed8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

(Change druid.indexer.runner.javaOpts="-server -Xmx3g -XX:MaxDirectMemorySize=3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" to druid.indexer.runner.javaOpts=-server -Xmx3g -XX:MaxDirectMemorySize=3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps)

在 2015年11月2日星期一 UTC+8上午7:53:23,Fangjin Yang写道: