Re: [druid-user] Surprising index_parallel batch ingestion behavior

Apache Druid Q&A Google Groups
#1

It could be due to timeChunkLock , the other task may not be getting the lock as other running task holds it. This can be confirmed by looking at overlord logs.

If that’s the case, You can try segment locking - when the segment lock is used, a task locks individual segments instead of the entire time chunk. As a result, two or more tasks can create segments for the same time chunk of the same data source simultaneously if they are reading different segments.

To enable segment locking, you may need to set forceTimeChunkLock to false.

Please note -
The segment locking is still experimental. It could have unknown bugs which potentially lead to incorrect query results.

Regards,
Vaibhav

#2

Druid takes lock on segment, but if you have the scenario where more than one index_parallel tasks try to take lock for same window one of them will eventually fail and will print saying cant take lock. But pending tasks is not something related to it. I would suggest to check your worker capacity and overlord setting something might be missing.