Middle manager Configuration

Hi,

When looking at middle manager configuration I see 2 attributes:

Druid.worker.capacity
Maximum number of tasks the MiddleManager can accept.
Number of available processors - 1

and

druid.processing.numThreads
The number of processing threads to have available for parallel processing of segments.
Number of cores - 1 (or 1)

In the above explanation both number of processors and no. of cores point to same configuration right ? Means if mine is a 8 core machine both the values would be 7 by default ?

So if I have 8 core machine and i configure default Druid.worker.capacity there would be 7 parallel tasks running in my middle manager and if default provide peon configuration druid.processing.numThreads

then each of my peon will try to spawn 7 threads to complete my task. This somehow doesn’t make sense to me.

Ideally we would need to split the configuration for these to 2 threads to match the total no. of process right ?

Regards,

Akhilesh

Hi Akhilesh,

‘druid.processing.numThreads’ is for query processing and they are not used for data ingestion.

Please note that stream ingestion tasks, e.g., Kafka/Kinesis index tasks, are capable of querying stream data while ingesting.

‘druid.processing.numThreads’ is to set how many threads you will use for query processing.

The ideal configuration would depend on your workload, but ‘2’ might be a good start.

Jihoon