Number format exception in historical node if we configure druid.processing.buffer.sizeBytes=3000000000

TROUBLESHOOTING Google Groups
#1
  1. Error in custom provider, java.lang.NumberFormatException: For input string: “3073741824”
    at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
    at io.druid.guice.ConfigProvider.bind(ConfigProvider.java:46) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.DruidProcessingConfigModule)
    while locating io.druid.query.DruidProcessingConfig
    while locating io.druid.segment.column.ColumnConfig
    for the 3rd parameter of io.druid.segment.IndexIO.(IndexIO.java:101)
    while locating io.druid.segment.IndexIO
    for the 1st parameter of io.druid.segment.loading.SegmentLoaderLocalCacheManager.(SegmentLoaderLocalCacheManager.java:68)
    while locating io.druid.segment.loading.SegmentLoaderLocalCacheManager
    at io.druid.guice.LocalDataStorageDruidModule.configure(LocalDataStorageDruidModule.java:54) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.LocalDataStorageDruidModule)
    while locating io.druid.segment.loading.SegmentLoader
    for the 1st parameter of io.druid.server.SegmentManager.(SegmentManager.java:100)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:94) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliHistorical$1)
    while locating io.druid.server.SegmentManager
    for the 8th parameter of io.druid.server.coordination.ServerManager.(ServerManager.java:98)
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:93) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliHistorical$1)
    while locating io.druid.server.coordination.ServerManager
    at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:96) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliHistorical$1)
    while locating io.druid.query.QuerySegmentWalker
    for the 2nd parameter of io.druid.server.QueryLifecycleFactory.(QueryLifecycleFactory.java:52)
    at io.druid.server.QueryLifecycleFactory.class(QueryLifecycleFactory.java:52)
    while locating io.druid.server.QueryLifecycleFactory
    for the 1st parameter of io.druid.server.QueryResource.(QueryResource.java:113)
    at io.druid.server.QueryResource.class(QueryResource.java:78)
    while locating io.druid.server.QueryResource
    Caused by: java.lang.NumberFormatException: For input string: “3073741824”
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:583)
    at java.lang.Integer.valueOf(Integer.java:766)
    at org.skife.config.DefaultCoercibles$16.coerce(DefaultCoercibles.java:255)
    at org.skife.config.DefaultCoercibles$16.coerce(DefaultCoercibles.java:253)
    at org.skife.config.Bully.coerce(Bully.java:164)
    at org.skife.config.Bully.coerce(Bully.java:81)
    at org.skife.config.ConfigurationObjectFactory.buildSimple(ConfigurationObjectFactory.java:173)
    at org.skife.config.ConfigurationObjectFactory.internalBuild(ConfigurationObjectFactory.java:76)
    at org.skife.config.ConfigurationObjectFactory.buildWithReplacements(ConfigurationObjectFactory.java:50)
    at io.druid.guice.ConfigProvider.get(ConfigProvider.java:85)
    at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
    at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
#2

I see and Integer.parseInt from the stack trace. Set the size to no more than 2147483648 and try?

#3

Can you please tell me in brief why it is restricted to 2147483648 .

My server is having the capacity of 8 cores and 60 gb .

no of threads we can configure only 7.as per the document it is noofcores-1.

as per this formula druid.processing.buffer.sizeBytes * (druid.processing.numMergeBuffers + druid.processing.numThreads + 1)

Its occuping only 20 GB. We are unable to uitilize the remaining 40Gb.

How to increase the performance and utilization of all resources.

#4

It simply has to do with the largest number that an “Integer” can store… see here - https://stackoverflow.com/a/15005226

Btw, did you try what I suggested? Did it work?

#5

If you want to get hands on and test it, run this in java.

System.out.println(Integer.MAX_VALUE);
#6

The max recommended value for druid.processing.buffer.sizeBytes from the docs is 1 GB. One buffer is allocated for each processing thread. A size between 500MB and 1GB is a reasonable choice for general use, See. Try lowering the value the property druid.processing.buffer.sizeBytes.

Thanks,

Sashi

#7

K.Thank you Sir.