Historical and broker node not starting up

I was getting the below error when trying to start historical and broker nodes:

2018-10-11T05:11:08,077 INFO [main] io.druid.guice.ConfigProvider - Unable to build instance of class[class io.druid.query.DruidProcessingConfig]
2018-10-11T05:11:08,078 ERROR [main] io.druid.cli.CliBroker - Error when starting up.  Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error in custom provider, java.lang.NumberFormatException: For input string: "20000000000"
  at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
  at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
  while locating io.druid.query.DruidProcessingConfig
    for the 1st parameter of io.druid.guice.DruidProcessingModule.getProcessingExecutorService(DruidProcessingModule.java:96)
  at io.druid.guice.DruidProcessingModule.getProcessingExecutorService(DruidProcessingModule.java:96) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingModule)
  while locating java.util.concurrent.ExecutorService annotated with @io.druid.guice.annotations.Processing()
    for the 1st parameter of io.druid.query.IntervalChunkingQueryRunnerDecorator.<init>(IntervalChunkingQueryRunnerDecorator.java:38)
  while locating io.druid.query.IntervalChunkingQueryRunnerDecorator
    for the 1st parameter of io.druid.query.timeseries.TimeseriesQueryQueryToolChest.<init>(TimeseriesQueryQueryToolChest.java:81)
  at io.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:101) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.QueryRunnerFactoryModule)
  while locating io.druid.query.timeseries.TimeseriesQueryQueryToolChest
  while locating io.druid.query.QueryToolChest annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=72, type=MAPBINDER, keyType=java.lang.Class<? extends io.druid.query.Query>)
  at io.druid.guice.DruidBinders.queryToolChestBinder(DruidBinders.java:45) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.QueryRunnerFactoryModule -> com.google.inject.multibindings.MapBinder$RealMapBinder)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryToolChest>
    for the 1st parameter of io.druid.query.MapQueryToolChestWarehouse.<init>(MapQueryToolChestWarehouse.java:36)
  while locating io.druid.query.MapQueryToolChestWarehouse
  while locating io.druid.query.QueryToolChestWarehouse
    for the 1st parameter of io.druid.server.QueryLifecycleFactory.<init>(QueryLifecycleFactory.java:52)
  at io.druid.server.QueryLifecycleFactory.class(QueryLifecycleFactory.java:52)
  while locating io.druid.server.QueryLifecycleFactory
    for the 1st parameter of io.druid.server.BrokerQueryResource.<init>(BrokerQueryResource.java:68)
  at io.druid.cli.CliBroker$1.configure(CliBroker.java:117) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliBroker$1)
  while locating io.druid.server.BrokerQueryResource
Caused by: java.lang.NumberFormatException: For input string: "20000000000"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:583)
        at java.lang.Integer.valueOf(Integer.java:766)
        at org.skife.config.DefaultCoercibles$16.coerce(DefaultCoercibles.java:255)
        at org.skife.config.DefaultCoercibles$16.coerce(DefaultCoercibles.java:253)
        at org.skife.config.Bully.coerce(Bully.java:164)
        at org.skife.config.Bully.coerce(Bully.java:81)
        at org.skife.config.ConfigurationObjectFactory.buildSimple(ConfigurationObjectFactory.java:173)
        at org.skife.config.ConfigurationObjectFactory.internalBuild(ConfigurationObjectFactory.java:76)
        at org.skife.config.ConfigurationObjectFactory.buildWithReplacements(ConfigurationObjectFactory.java:50)
        at io.druid.guice.ConfigProvider.get(ConfigProvider.java:85)
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapt



I thought may be I am getting the error because the number 20000000000 is larger than Integer.MAX_VALUE. So I removed one zero and was able to start both the nodes.
However, when I submit the task for ingestion I get the same error again:

[io.druid.server.initialization.ServerConfig@893d5e45]
2018-10-11T05:58:27,833 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

  1. Error in custom provider, java.lang.NumberFormatException: For input string: “2560000000”
    at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
    at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
    while locating io.druid.query.DruidProcessingConfig
    for the 1st parameter of io.druid.guice.DruidProcessingModule.getProcessingExecutorService(DruidProcessingModule.java:96)
    at io.druid.guice.DruidProcessingModule.getProcessingExecutorService(DruidProcessingModule.java:96) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingModule)
    while locating java.util.concurrent.ExecutorService annotated with @io.druid.guice.annotations.Processing()
    for the 12th parameter of io.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:108)
    at io.druid.cli.CliPeon$1.configure(CliPeon.java:165) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliPeon$1)
    while locating io.druid.indexing.common.TaskToolboxFactory
    for the 1st parameter of io.druid.indexing.overlord.ThreadPoolTaskRunner.(ThreadPoolTaskRunner.java:100)
    at io.druid.cli.CliPeon$1.configure(CliPeon.java:192) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliPeon$1)
    while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
    while locating io.druid.indexing.overlord.TaskRunner
    for the 4th parameter of io.druid.indexing.worker.executor.ExecutorLifecycle.(ExecutorLifecycle.java:78)
    at io.druid.cli.CliPeon$1.configure(CliPeon.java:182) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliPeon$1)
    while locating io.druid.indexing.worker.executor.ExecutorLifecycle
    Caused by: java.lang.NumberFormatException: For input string: “2560000000”
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:583)
    at java.lang.Integer.valueOf(Integer.java:766)
    at org.skife.config.DefaultCoercibles$16.coerce(DefaultCoercibles.java:255)
  1. Historical Node

Runtime.properties:

druid.service=druid/historical

druid.port=8083

HTTP server threads

druid.server.http.numThreads=40

Processing threads and buffers

druid.processing.buffer.sizeBytes=20000000000

druid.processing.numThreads=7

#below number is 50 million

druid.query.groupBy.maxIntermediateRows=50000000

#below number is 500 million

druid.query.groupBy.maxResults=500000000

Segment storage

Segments assigned to a Historical node are first stored on the local file system (in a disk cache) and then served by the Historical node. These locations define where that local cache resides. This value cannot be NULL or EMPTY.

druid.segmentCache.locations=[{“path”:“var/druid/segment-cache”,“maxSize”:300000000000}]

#Below number is 5TB

#The maximum number of bytes-worth of segments that the node wants assigned to it. This is not a limit that Historical nodes actually enforces, just a value published to the Coordinator node so it can plan accordingly.

druid.server.maxSize=5000000000000

JVM.config:

-server

-Xms8g

-Xmx10g

-XX:MaxDirectMemorySize=150g

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=var/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

2. Broker Node:


1. Broker

**jvm.config**

-server

-Xms20g

-Xmx20g

-XX:MaxDirectMemorySize=150g

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=var/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

**Runtime.properties**

druid.service=druid/broker

druid.port=8082

# HTTP server threads

druid.broker.http.numConnections=20

#max(10, (Number of cores * 17) / 16 + 2) + 30

druid.server.http.numThreads=40

# Processing threads and buffers

#below number is 20GB

druid.processing.buffer.sizeBytes=20000000000

druid.processing.numThreads=7

# Query cache (we use a small local cache)

druid.broker.cache.useCache=true

druid.broker.cache.populateCache=true

#below number is 50 million

druid.query.groupBy.maxIntermediateRows=50000000

#below number is 500 million

druid.query.groupBy.maxResults=500000000

#druid.cache.type=local

#druid.cache.sizeInBytes        The maximum size of the cache in bytes on heap. None (unlimited)

#druid.cache.sizeInBytes=10000000

Changed the value of
druid.indexer.fork.property.druid.processing.buffer.sizeBytes and brought it below the Integer.MAX_VALUE and everything works now :slight_smile:

Glad you got it working!

The Integer.MAX_VALUE restriction exists since the processing buffers are Java ByteBuffers which can’t be bigger than that, and you’re correct, and the size needs to be configured on broker, historical, and middleManager/peon.

Thanks,

Jon