[druid-user] Ingestion tasks in pending state


I am running kafka based ingestion. All of a sudden the tasks have started entering into PENDING state and from coordinator-overlord logs I could see below as the reason:

coordinator-overlord.log:146278:2021-04-15T14:18:51,062 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: ***************
coordinator-overlord.log:146279:2021-04-15T14:18:51,062 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Added pending task ***************
coordinator-overlord.log:146281:2021-04-15T14:18:51,062 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [] because: [task is not in knownTaskIds[[, ***************, ***************]]]

Could you provide some pointers to debug this?


This is just a suspicion, but have you checked that your metadata database is up and running, and working OK?
Any other errors in other logs (eg, about metadata, but could be anything else)?

It should be something to do with druid worker capacity. Check the configuration in middlemanager node with the number of tasks running concurrently