Peons don’t seem to be using more than a single core.
I am setting the following in middleManager config:
And i know this is being picked up because I get the MaxDirectMemory error if I intentionally set it low enough.
However when checking cpu utilization via ‘top’, the process hovers around 100% when I expect 300%.
Can someone please assist?
Thanks very much
Ingestion always uses just one core per peon/task. The druid.processing.numThreads only controls how many threads are used for querying. So if you want to scale out ingestion, the way to do it is to use multiple tasks.
But then I am limited by the segmentGranularity, correct? If I am aggregating to ‘day’ then there can only be one task per day. Each day is currently taking 60minutes to Index, are you saying there is no way to speed this up?