Multiple Kafka cluster

Can druid talk to multiple kafka clusters?
We are planning to switch to new kafka cluster for few topics and thus updating runtime properties.
But does druid will be able read through the topic data from the right cluster and maintain the offset??

Chitra

Hi Chitra,

Are you using the Kafka indexing service, Tranquility Kafka, or something else?

Assuming you are using Kafka indexing service (which is the recommended ingestion option): yes, different supervisors can talk to different Kafka clusters. If you are moving a topic to a new Kafka cluster, to avoid duplicates or missed data, you should:

  1. first point the data producers to the new Kafka cluster so that it doesn’t write the same events to both clusters

  2. wait for the tasks to read to the end of the old topic

  3. suspend the supervisor, and then update the spec to point to the new Kafka brokers and set useEarliestOffset: true

  4. Reset the supervisor, which will cause it to clear the offsets from the old topic and reset them to the beginning of the stream

  5. Resume the supervisor

Hope this helps. Let me know if you’re using a different setup.

David

Thanks David I was looking for the exact thing that you mentioned.i will do cut over in few weeks,will come back to you.
Thank you!