I need help in understanding Coordinator's formula for deciding number of auto compaction tasks. Coordinator process documentation in this link - https://druid.apache.org/docs/latest/design/coordinator.html, under Compacting Segments section mentions the formula for maximum number of compaction tasks - min(sum of worker capacity * slotRatio, maxSlots).
1. Worker capacity is 'Maximum number of tasks the MiddleManager can accept', which is Number of CPU - 1. We have 4 middle managers, each with 8 CPUs. So worker capacity of a single middle manager is 7. Sum of worker capacity should be 7*4 = 28. Is this right?
2. Maxslots = Is it the maximum number of slots available when Coordinator wants to submit compaction task? For example, even though middle manager can accept 7 tasks, currently only 3 are occupied, which means 4 are available. For 4 middle managers each having 3 slots occupied, max slots should be 4*4 = 16. Is this correct?
3. What is slot ratio? I searched the documentation but unfortunately couldn't find anything that defines what this is.
We added auto compaction configuration on Druid 0.16.0 version for a data source. When we have 7 middle managers, each with 8 CPUs, we observed 4 compaction tasks at a time. When we have 4 middle managers, each with 8 CPUs, we observed 2 compaction tasks at a time.
Can someone please explain how the formula is playing in our case? Please let me know if you need additional information from me.
Thank you for your time.