Druid Kafka indexing service problems

Hello,

Lately started experiencing issues with Kafka indexing service.

Issues:

  • Hourly task keeps running forever.
  • After first hour segments not merged together and not uploaded to HDFS deep storage anymore.
    Otherwise new events seem to be coming in for the same task and task just keeps making new segments. Old segments are also served from same task.

Is there a way how to move/merge those segments manually to deep storage/MySQL metadata storage?

There’s no good way to manually move segments to deep storage from an index task. If things used to work and stopped working, my first inclination would be to make sure that HDFS/historical nodes have capacity for the new segments. If you post your logs for the indexing task, coordinator, and overlord, there might be some hints in there as to what is going on.