Number format exception in historical node if we configure druid.processing.buffer.sizeBytes=3000000000

  1. Error in custom provider, java.lang.NumberFormatException: For input string: “3073741824”
    at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
    at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
    while locating io.druid.query.DruidProcessingConfig
    while locating io.druid.segment.column.ColumnConfig
    for the 3rd parameter of io.druid.segment.IndexIO.(IndexIO.java:101)
    while locating io.druid.segment.IndexIO
    for the 1st parameter of io.druid.segment.loading.SegmentLoaderLocalCacheManager.(SegmentLoaderLocalCacheManager.java:68)
    while locating io.druid.segment.loading.SegmentLoaderLocalCacheManager
    at io.druid.guice.LocalDataStorageDruidModule.configure(LocalDataStorageDruidModule.java:54) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.LocalDataStorageDruidModule)
    while locating io.druid.segment.loading.SegmentLoader
    for the 1st parameter of io.druid.server.SegmentManager.(SegmentManager.java:100)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:94) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliHistorical$1)
    while locating io.druid.server.SegmentManager
    for the 8th parameter of io.druid.server.coordination.ServerManager.(ServerManager.java:98)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:93) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliHistorical$1)
    while locating io.druid.server.coordination.ServerManager
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:96) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliHistorical$1)
    while locating io.druid.query.QuerySegmentWalker
    for the 2nd parameter of io.druid.server.QueryLifecycleFactory.(QueryLifecycleFactory.java:52)
    at io.druid.server.QueryLifecycleFactory.class(QueryLifecycleFactory.java:52)
    while locating io.druid.server.QueryLifecycleFactory
    for the 1st parameter of io.druid.server.QueryResource.(QueryResource.java:113)
    at io.druid.server.QueryResource.class(QueryResource.java:78)
    while locating io.druid.server.QueryResource
    Caused by: java.lang.NumberFormatException: For input string: “3073741824”
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:583)
    at java.lang.Integer.valueOf(Integer.java:766)
    at org.skife.config.DefaultCoercibles$16.coerce(DefaultCoercibles.java:255)
    at org.skife.config.DefaultCoercibles$16.coerce(DefaultCoercibles.java:253)
    at org.skife.config.Bully.coerce(Bully.java:164)
    at org.skife.config.Bully.coerce(Bully.java:81)
    at org.skife.config.ConfigurationObjectFactory.buildSimple(ConfigurationObjectFactory.java:173)
    at org.skife.config.ConfigurationObjectFactory.internalBuild(ConfigurationObjectFactory.java:76)
    at org.skife.config.ConfigurationObjectFactory.buildWithReplacements(ConfigurationObjectFactory.java:50)
    at io.druid.guice.ConfigProvider.get(ConfigProvider.java:85)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)

I see and Integer.parseInt from the stack trace. Set the size to no more than 2147483648 and try?

Can you please tell me in brief why it is restricted to 2147483648 .

My server is having the capacity of 8 cores and 60 gb .

no of threads we can configure only 7.as per the document it is noofcores-1.

as per this formula druid.processing.buffer.sizeBytes * (druid.processing.numMergeBuffers + druid.processing.numThreads + 1)

Its occuping only 20 GB. We are unable to uitilize the remaining 40Gb.

How to increase the performance and utilization of all resources.

It simply has to do with the largest number that an “Integer” can store… see here - https://stackoverflow.com/a/15005226

Btw, did you try what I suggested? Did it work?

If you want to get hands on and test it, run this in java.

System.out.println(Integer.MAX_VALUE);

The max recommended value for druid.processing.buffer.sizeBytes from the docs is 1 GB. One buffer is allocated for each processing thread. A size between 500MB and 1GB is a reasonable choice for general use, See. Try lowering the value the property druid.processing.buffer.sizeBytes.

Thanks,

Sashi

K.Thank you Sir.