Kafka indexing service with task replicas

We’re migrating our Druid cluster from Tranquility-based indexing to Kafka service indexing. It’s working great for us so far, but we are trying to understand some of the possible failure modes.

We run two replicas of each task for redundancy. Due to the way Tranquility works, in our prior system both tasks would always contain the exact same data. It did not matter which task pushed the segment.

With Kafka indexing, it seems that there is a possibility that one of the replicas could be underperforming and lag the other task. If that straggler task were to push the segment before the other, then it looks like all of the tasks for the next interval are reset to the earlier offset and we lose some forward progress. Actually, it’s possible for data that was previously available in queries to become unavailable, which could confuse our customers. Is this just a limitation of the current approach, or is there some facility to address this that I am missing?

This grew out of a discussion we had about how to monitor the ingestion lag. Essentially, “if one task is up to date, but another is lagging, are we OK or NOT OK?”

Thanks for any thoughts!

Max

I think I found the code that prevents this from happening, and it’s a quite elegant solution.

https://github.com/druid-io/druid/blob/master/extensions-core/kafka-indexing-service/src/main/java/io/druid/indexing/kafka/supervisor/KafkaSupervisor.java#L1158

If I understand it correctly, the up to date segment will push right away and the lagging segment will try to read up to the same offsets and then push. Most likely the first segment wins, and the slow task is killed.