0.12.3-Tasks are automatically failed with duration -1


We have been trying supervisors in our 0.12.3 version of druid.

We have ingestion tasks being submitted to overlord node through supervisors, however most of the tasks are automatically submitted with task duration -1 and are failing with no logs. We have a shared HDFS setup but i don’t see any logs for the tasks. I was not able to track the tasks and check the reason for them failing.

I’ve tried restarting supervisors but that was not helpful.

Note: Queue insertion time column on the overlord console has a different value when the task is running and has a different value when it is completed. Is this expected behavior ?

Thanks !

I would check the overlord and middle manager logs, searching for the ID of a failing task, and see if you find any error or exception logs.

You could also try temporarily setting log storage to “local” on your middle managers, and check the local filesystems on your middle managers to see if any task logs show up there (perhaps they can’t be written to HDFS for some reason).