i need to migrate my zookeeper ensemble.
My initial thought is to just recreate a new ensemble and kill the old one.
How will druid react in that case? The nodes will essentially suddenly see zookeeper being completely empty.
Whats the recommendation?
ZooKeeper is used for locking and for awareness of the running cluster but not for long-term configuration. So, replacing the entire ensemble should be fine, as long as you do it while Druid services are shut down. Note that if you also use this cluster for kafka consumer state, you’ll be resetting your kafka offsets, which will likely cause some data inconsistency.
It’s possible to migrate the ensemble online, too. You’d need to replace the ZooKeeper servers one by one, making sure that for each replacement you update the configs on the other ZooKeeper servers and on the Druid servers. This process is somewhat labor intensive if you haven’t automated it.
if i replace one by one, can i maintain 100% uptime?
That could be very dangerous. It might be possible to double-up your zk nodes in each region, then slowly take down the old nodes while reconfiguring the cluster as you go.
Doing this with 100% uptime is not a fast process but should be able to be done.
What did mean by “double-up your zk nodes in each region” ??