Broker numThreads and numMergeBuffers

Hi everyone,

I was looking through the sizing guide and it says:

"On the Broker, the amount of direct memory needed depends on how many merge buffers (used for merging GroupBys) are configured. The Broker does not generally need processing threads or processing buffers, as query results are merged on-heap in the HTTP connection threads instead.

  • druid.processing.buffer.sizeBytes can be set to 500MB.
  • druid.processing.numThreads: set this to 1 (the minimum allowed)
  • druid.processing.numMergeBuffers: set this to the same value as on Historicals or a bit higher"
    However, in our use case, almost all our queries are groupByV2, which I think means they need a thread and merge buffer, so in this case we should be setting numThreads and numMergeBuffers both to something like (cpu cores - 1) ?

Can anyone that knows for sure confirm either way?

Thank you!

Michael

As default druid.processing.numThreads = 1 is good idea. This means 1 thread per CPU core

I don’t think that’s right. The documentation for numThreads says:
“The number of processing threads to have available for parallel processing of segments. Our rule of thumb is num_cores - 1, which means that even under heavy load there will still be one core available to do background tasks like talking with ZooKeeper and pulling down segments. If only one core is available, this property defaults to the value 1.”

Can anyone answer my original question that knows for sure?