Runtime property values in config file are not picked up by druid


I received following error for real-time tasks.

Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, druid.processing.numThreads, or druid.processing.numMergeBuffers: maxDirectMemory[1,860,698,112], memoryNeeded[6,442,450,944] = druid.processing.buffer.sizeBytes[1,073,741,824] * (druid.processing.numMergeBuffers[2] + druid.processing.numThreads[3] + 1)


So either of the above configurations can be adjusted to overcome above error. However the file for middle manager doesn’t have any of these properties or they are named slightly differently.



Number of tasks per middleManager


Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager


HTTP server threads


Processing threads and buffers on Peons

Hadoop indexing



So we don’t have druid.processing.numThreads but rather have Is this supposed to be same?

No matter what i set any of the following properties to, i get the same error. Where exactly are the following properties to be changed if not in under config/middlemanager?

*druid.processing.buffer.sizeBytes, druid.processing.numThreads, druid.processing.numMergeBuffers*



My bad.These properties (druid.processing.buffer.sizeBytes, druid.processing.numThreads, druid.processing.numMergeBuffers) should be in the for the overlord . I got confused due to similar naming for the other properties.



The properties can be passed to the peons by setting them in druid.indexer.runner.javaOpts
Not sure why you noticed the change in overlord propagated to tasks, maybe you are using the local task runner instead of a remote one.

Hi Nishant,

I was facing another issue where tasks were not getting created and i suspected that it has something to do with running tasks on middle manager. In order to debug i was using “local” mode for tasks.

After i set druid.processing.buffer.sizeBytes, druid.processing.numThreads, druid.processing.numMergeBuffers in overlord’s properties the issue i was facing due to low memory was resovled.

But just to understand this better, druid.indexer.runner.javaOpts is applicable only when using “remote” and not “local” mode?