Best Practice on Migrating Coordinator/Zookeeper

I am trying to migrate an 0.9.0 druid cluster off of a degraded zookeeper instance. I am moving the whole cluster to a new provider and am not sure what complications might arise by just swapping the zookeeper connection string and restarting. I know zk is used not only for service discovery but segment announcement afaik.

We use tranquility to index from kafka so I am not sure how swapping zookeeper will work in this case. Will announced segments get ‘stuck’, should i disable the hourly workers, let the jobs queue up, then restart the middle managers once they are idle?

Cluster specs:

1 ZK

2 Coordinators

4 historicals

4 middle managers (new provider) & tranq

2 Brokers


Pere, if you shut down your cluster, update the Zk config string, and bring it back up, things should just work.

Thanks Fangjin,

Is there any good way to keep the indexers up through the process? I am trying to not have my realtime indexers fail out and lose data if at all possible. If I do a rolling indexer restart then some of the index jobs will have the wrong zk and the segment would not be read/loaded by the other historicals no? Tranquility unfortunately doesn’t re-create indexed jobs afaik when all the middleManagers are reset. I would lose potentially an hour of data upon reset.