CliPeon failed to start

When I setup druid to a better cluster for HDFS integration, all peon tasks failed. MiddleManager server logged the following errors 8 times. I actually specified numThreads smaller than 31 so I am not sure what the error message really means. Similar druid properties allowed me to ingest data in a cluster with less resources.
I am using Druid-0.7.0 version.

2015-05-04T20:40:23,301 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.
com.google.inject.ProvisionException: Guice provision errors:

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,556,918,784], memoryNeeded[34,359,738,368] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[31] + 1 )
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:76)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)
    while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:77)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)
    while locating io.druid.query.groupby.GroupByQueryRunnerFactory
    while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)
    at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)
    while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)
    while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
    at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)
    while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:76)
    at io.druid.cli.CliPeon$1.configure(CliPeon.java:129)
    while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.(ThreadPoolTaskRunner.java:71)
    at io.druid.cli.CliPeon$1.configure(CliPeon.java:155)
    while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
    while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:89)
    while locating io.druid.server.QueryResource

  2. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[28,556,918,784], memoryNeeded[34,359,738,368] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[31] + 1 )
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:76)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)
    while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 2 at io.druid.query.groupby.GroupByQueryQueryToolChest.(GroupByQueryQueryToolChest.java:99)
    at io.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:72)
    while locating io.druid.query.groupby.GroupByQueryQueryToolChest
    for parameter 3 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:77)

Repeated 8 times.

Here is the middleManager’s runtime.properties:

druid.host=r80an10
druid.port=8088
druid.service=druid/poc/middlemanager

We enable using the local query cache here

druid.broker.cache.useCache=true
druid.broker.cache.populateCache=true

druid.broker.http.numConnections=20
druid.broker.http.readTimeout=PT5M

druid.processing.buffer.sizeBytes=1000000000
druid.processing.numThreads=10

druid.server.http.numThreads=15

druid.indexer.logs.type=file
druid.indexer.logs.directory=/druidpoc/task
druid.indexer.fork.property.druid.indexer.logs.type=file
druid.indexer.fork.property.druid.indexer.logs.directory=/druidpoc/task/peon

Resources for peons

druid.indexer.runner.javaOpts=-server -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Peon properties

druid.indexer.fork.property.druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=136870912
druid.indexer.fork.property.druid.processing.numThreads=3
druid.indexer.fork.property.druid.segmentCache.locations=[{“path”: “/druidpoc/persistent/zk_druid”, “maxSize”: 0}]
druid.indexer.task.baseDir=/druidpoc/
druid.indexer.task.baseTaskDir=/druidpoc/persistent/task/

druid.realtime.chathandler.type=announce

druid.indexer.fork.property.druid.server.http.numThreads=30
druid.indexer.fork.property.druid.storage.type=local
druid.indexer.fork.property.druid.storage.directory=/druidpoc/task/peon/log
druid.indexer.fork.property.druid.indexer.task.chathandler.type=announce

druid.worker.capacity=3

After discussion over ICR with Fangjin, the issue is resolved. When automating the start up script, the path for middleManager should be an upper case M. Putting the lower one caused middleManager’s runtime.properties file not picked up.

Thanks a lot Fangjin.