I’m a bit new to Kafka so sorry if this is a non-question for most people.
We are planning on moving our druid ingestion over to the Kafka Indexing Service using a managed Kafka provider.
But say we want to change providers in the future. Am I correct in assuming we would mirror the Kafka cluster over to the new provider, update the supervisor with the new bootstrap servers, then update the bootstrap servers of the producing application?
Druid also keeps track of the kafka offsets it has ingested,
In addition to changing the bootstrap servers, you will also need to reset the the offsets maintained in druid.
The steps IMO would be -
- Move producers to the new kafka cluster
- Wait for kafka indexing service to ingest all events from existing cluster
- Suspend the supervisor, this would make sure all running tasks handover the events
- Change the kafka supervisor spec to point to new kafka cluster
- Reset the supervisor, set useEarliestOffset=true, resume ingestion
Details on how to suspend/resume/reset supervisor are here -
Ahh okay, it was resetting the offset that had me concerned.
In this plan, there would be no need to use MirrorMaker to mirror the old cluster to the new cluster then?
Yes no need to mirror in this case, as we are ingesting all the data in firat cluster and then moving on to the other one.