Can druid talk to multiple kafka clusters?
We are planning to switch to new kafka cluster for few topics and thus updating runtime properties.
But does druid will be able read through the topic data from the right cluster and maintain the offset??
Are you using the Kafka indexing service, Tranquility Kafka, or something else?
Assuming you are using Kafka indexing service (which is the recommended ingestion option): yes, different supervisors can talk to different Kafka clusters. If you are moving a topic to a new Kafka cluster, to avoid duplicates or missed data, you should:
first point the data producers to the new Kafka cluster so that it doesn’t write the same events to both clusters
wait for the tasks to read to the end of the old topic
suspend the supervisor, and then update the spec to point to the new Kafka brokers and set useEarliestOffset: true
Reset the supervisor, which will cause it to clear the offsets from the old topic and reset them to the beginning of the stream
Resume the supervisor
Hope this helps. Let me know if you’re using a different setup.
Thanks David I was looking for the exact thing that you mentioned.i will do cut over in few weeks,will come back to you.