Worker capacity not fulfilled after splitting data node

Hey,

I split data node into 2 separate nodes : middleManager and historical. MiddleManager(m4.2xlarge) having druid.worker.capacity=8 property configured.

But after I’ve done that, even though Overlord console shows “worker capacity = 8” I cannot submit more than 2 tasks in parallel.

Could it be caused by the fact that Overlord runs on a weak m4.large machine with 2 virtual cores ? Because from configuration perspective it all seems to be set correctly.

What type of indexing tasks are you submitting to the middle manager? Are the other tasks that you submitted sitting in the pending queue? Indexing tasks are assigned to an availability group and tasks in the same availability group are prevented from running concurrently on the same worker (to support proper replication) even if there’s excess capacity. Could this be what you’re seeing?

I have this setup (i’m using modified cluster from implydata distribution) :

master m4.large (overlord, coordinator, derby, zk)

middleManager m4.2xlarge (worker capacity = 8)

broker-1 m4.large

broker-2 m4.large

historical-1 m4.large

historical-2 m4.large

I use the default tier by both historical nodes to avoid single point of failure. All workers/peons reside on a single middleManager node and I’m submitting 8 hadoop indexing tasks in parallel :

I index say 365 days with DAY granularity, each indexing thread is indexing 365/8 portion…

but at the same time there are only 2 “Running Tasks” and there are always 0 “Pending Tasks”. It’s like the overlord does’t accept more tasks than 2, not even to the Pending queue.

I switched from implydata Quickstart distribution where I could index 8 tasks in parallel using the same setup.

Odd, any interesting logs on the overlord or middle manager?

Hi David,

it was my mistake I did on the client application. I used a default Future ExecutionContext which defaults to #Runtime.availableProcessors number of threads… so

I was issuing as many indexing tasks in parallel as many virtual cores my machine had. Cheers !