Indexing tasks are in pending state even though worker's used capacity is below available capacity

I have a druid setup in which indexing service is running in remote mode i.e

‘druid.indexer.runner.type=remote’

And also i have set the max capacity of worker=15 i.e

druid.worker.capacity=15
still i observe that only 4 indexing task are in running state at any given time and overlord’s web interface show that used capacity of worker is 4 and worker capacity is 15.

What could be the reason for all other task to be added to pending queue instead of running queue??

And also what happens to data that i push to indexing service while my task was in waiting state?Does that data get lost?

I am using tranquility for ingesting data.

Thanks

Rohit

Hi Rohit, the three usual reasons for pending tasks are (a) no capacity on workers; (b) not enough workers to satisfy redundancy requirements; © some other task has a lock that the pending task needs. You said that the overlord console shows that you have capacity, so it’s probably not (a). Could it be (b)- do you have redundancy in tranquility set higher than the number of available workers?

Data pushed while tasks are pending will be buffered on the client side for the windowPeriod, then dropped. If you have a copy of the data in batch storage like S3 or HDFS, you could re-ingest it using batch indexing.

Gian,

Redundancy has been set at 1 and i have exactly 1 worker running.

And had been case © those pending task should have been shown under the heading “Waiting Tasks-Tasks waiting on locks” on web interface instead of under “Pending Tasks-Tasks waiting to be assigned to a worker”.

Is there any other reason for pending tasks??

Thanks

Rohit

Not really that I can think of. Can you attach a screenshot of your overlord console, and your overlord logs? There might be something interesting in one of those.

Any updates on this one ? well, my taska arent in pending, but running mode, and also … they are outputing logs regularly real time, just that the task has been running for 2 hours now, for ingesting hardly any 1000 records ??

Hey Sumit,

Can you post your task spec and task logs?

Hi Gian, I’m facing similar issue about worker utilisation. I set druid.worker.capacity=10, but I only see 2 running workers. Previously, I had 1 task in pending state, though it’s not there in the screen shot. Why is task got into pending while it’s not even reaching the capacity?

Overlord log: https://pastebin.com/Xnq9PH9a.