Problem in Batch ingestion - Couldnot start historical node

Hi ,
I am trying to do batch ingestion with help of this link http://druid.io/docs/latest/tutorials/tutorial-loading-batch-data.html. But i could not start historical node . while i start the server , following 5 errors occured.

  1. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[238,616,576], memoryNeeded[4,294,967,296] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[3] + 1 )
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:75)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)
    while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)
    while locating io.druid.query.groupby.GroupByQueryRunnerFactory
    while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)
    at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)
    while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)
    while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
    at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)
    while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 1 at io.druid.server.coordination.ServerManager.(ServerManager.java:107)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:77)
    while locating io.druid.server.coordination.ServerManager
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:79)
    while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)
    while locating io.druid.server.QueryResource

  2. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[238,616,576], memoryNeeded[4,294,967,296] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[3] + 1 )
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.(GroupByQueryEngine.java:75)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)
    while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 2 at io.druid.query.groupby.GroupByQueryQueryToolChest.(GroupByQueryQueryToolChest.java:113)
    at io.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:72)
    while locating io.druid.query.groupby.GroupByQueryQueryToolChest
    for parameter 3 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)
    while locating io.druid.query.groupby.GroupByQueryRunnerFactory
    while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)
    at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)
    while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)
    while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
    at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)
    while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 1 at io.druid.server.coordination.ServerManager.(ServerManager.java:107)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:77)
    while locating io.druid.server.coordination.ServerManager
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:79)
    while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)
    while locating io.druid.server.QueryResource

  3. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[238,616,576], memoryNeeded[4,294,967,296] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[3] + 1 )
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 3 at io.druid.query.groupby.GroupByQueryQueryToolChest.(GroupByQueryQueryToolChest.java:113)
    at io.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:72)
    while locating io.druid.query.groupby.GroupByQueryQueryToolChest
    for parameter 3 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)
    while locating io.druid.query.groupby.GroupByQueryRunnerFactory
    while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)
    at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)
    while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)
    while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
    at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)
    while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 1 at io.druid.server.coordination.ServerManager.(ServerManager.java:107)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:77)
    while locating io.druid.server.coordination.ServerManager
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:79)
    while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)
    while locating io.druid.server.QueryResource

  4. Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[238,616,576], memoryNeeded[4,294,967,296] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[3] + 1 )
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)
    while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 4 at io.druid.query.groupby.GroupByQueryRunnerFactory.(GroupByQueryRunnerFactory.java:79)
    at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)
    while locating io.druid.query.groupby.GroupByQueryRunnerFactory
    while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)
    at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)
    while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.(DefaultQueryRunnerFactoryConglomerate.java:34)
    while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
    at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)
    while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 1 at io.druid.server.coordination.ServerManager.(ServerManager.java:107)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:77)
    while locating io.druid.server.coordination.ServerManager
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:79)
    while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)
    while locating io.druid.server.QueryResource

  5. druid.segmentCache.locations - may not be empty
    at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:131)
    at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:131)
    while locating com.google.common.base.Supplier<io.druid.segment.loading.SegmentLoaderConfig>
    at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:132)
    while locating io.druid.segment.loading.SegmentLoaderConfig
    for parameter 1 at io.druid.segment.loading.SegmentLoaderLocalCacheManager.(SegmentLoaderLocalCacheManager.java:57)
    while locating io.druid.segment.loading.SegmentLoaderLocalCacheManager
    at io.druid.guice.LocalDataStorageDruidModule.configure(LocalDataStorageDruidModule.java:49)
    while locating io.druid.segment.loading.SegmentLoader
    for parameter 0 at io.druid.server.coordination.ServerManager.(ServerManager.java:107)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:77)
    while locating io.druid.server.coordination.ServerManager
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:79)
    while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)
    while locating io.druid.server.QueryResource

5 errors
How to resolve that???

I am starting the historical node server with command "java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common/common.runtime.properties:config/historical/runtime.properties:lib/* io.druid.cli.Main server historical

" .

i ve attached runtime.props for historical node.

Thanks in advance.

runtime.properties (1.51 KB)

as the logs message said you need to adjust the MaxDirectMemorySize. try -XX:MaxDirectMemorySize=10240g as options of your Druid JVM java opts

Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.

If you are getting started wtih Druid for the first time, you should probably try:

http://druid.io/docs/0.9.0-rc1/tutorials/quickstart.html

Much easier quickstart. It is based off of http://imply.io/docs/latest/quickstart , which will be even easier