Runtime property values in config file are not picked up by druid

Apache Druid Q&A Google Groups
#1

Hi,

I received following error for real-time tasks.

Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, druid.processing.numThreads, or druid.processing.numMergeBuffers: maxDirectMemory[1,860,698,112], memoryNeeded[6,442,450,944] = druid.processing.buffer.sizeBytes[1,073,741,824] * (druid.processing.numMergeBuffers[2] + druid.processing.numThreads[3] + 1)

``

So either of the above configurations can be adjusted to overcome above error. However the runtime.properties file for middle manager doesn’t have any of these properties or they are named slightly differently.

druid.service=druid/middleManager

druid.plaintextPort=8091

Number of tasks per middleManager

druid.worker.capacity=3

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

druid.indexer.task.baseTaskDir=var/druid/task

HTTP server threads

druid.server.http.numThreads=25

Processing threads and buffers on Peons

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912

druid.indexer.fork.property.druid.processing.numThreads=2

Hadoop indexing

druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

``

So we don’t have druid.processing.numThreads but rather have druid.indexer.fork.property.druid.processing.numThreads. Is this supposed to be same?

No matter what i set any of the following properties to, i get the same error. Where exactly are the following properties to be changed if not in runtime.properties under config/middlemanager?

*druid.processing.buffer.sizeBytes, druid.processing.numThreads, druid.processing.numMergeBuffers*

Thanks,

Prathamesh

#2

My bad.These properties (druid.processing.buffer.sizeBytes, druid.processing.numThreads, druid.processing.numMergeBuffers) should be in the runtime.properties for the overlord . I got confused due to similar naming for the other properties.

Thanks,

Prathamesh

#3

The properties can be passed to the peons by setting them in druid.indexer.runner.javaOpts
Not sure why you noticed the change in overlord propagated to tasks, maybe you are using the local task runner instead of a remote one.

#4

Hi Nishant,

I was facing another issue where tasks were not getting created and i suspected that it has something to do with running tasks on middle manager. In order to debug i was using “local” mode for tasks.

After i set druid.processing.buffer.sizeBytes, druid.processing.numThreads, druid.processing.numMergeBuffers in overlord’s properties the issue i was facing due to low memory was resovled.

But just to understand this better, druid.indexer.runner.javaOpts is applicable only when using “remote” and not “local” mode?

Thanks,

Prathamesh