What if Druid loses zookeeper or it's meta database?

So I was asked some questions today regarding what will happen to Druid if it lost all of its zookeepers.

I was also asked what would happen if it lost it's meta database which in my case is mysql.

I said that Druid would be in deep trouble if either should happen, but I wanted to run the question by the Druid users on the forum

This also made me wonder what steps I would take in the future should I need to perform some maintenance on the metadata base environment.

Thoughts please?

If the metadata db can be recovered then the loss is minimal, only
related to changes to segments or tasks that were changing after the
last backup. You'll have to reprocess whatever happened during that
timeframe. If the DB is gone-gone, then that's a pretty bad scenario
as the cluster will think everything should be blank, and when a new
blank DB is brought up it will probably unload all the historical data
and kill all the tasks that shouldn't be running (because it thinks
there should be no tasks).

If the zookeeper cluster is torn down and brought back up, all the
druid specific things should recover as they are all ephemeral as long
as the same ip can be used for the new zk. (if you are using
Tranquility I don't know if that is the case). During the bring-up
process the brokers may or may not get a bit confused as to the
correct state of the cluster, but it should recover once all
historicals have announced themselves properly.

While ZK is down or the DB is unavailable, overlord tasks will not be
able to complete and will be in a backoff loop. The coordinator will
probably not be able to establish leadership and will not be able to
coordinate the cluster.

The good news is that any data already loaded *should* be able to be
queried during a zk outage because druid caches the last known state
of the cluster. If your brokers or historicals restart you are out of
luck until zk comes back up though.