Druid + tranquility Overlord high availability

I’m looking for recommendations for high availability for Overlorad with tranquility + indexing service setup.

I understand multiple overlord nodes can run at the same time and get elected as leader if the previous leader was died. However, the new leader will mark all the running task as failed. All the new messages pushed (service.apply(messages)) by Tranquility will be stuck at NoHostAvailable exception even though all the middle managers are available to take new tasks.

To make indexing continue after a new overlord take leadership or the old one restarted, I have do remove /tranquility dir in ZooKeeper so that new tasks will be up and running, this will lose some data but we think it’s better than having all the realtime ingestion stuck as we can always backfill with batch ingestion.

My questions is that for any reason why Overlord cannot simply restore state such that any new Overlord leader should be able to continue the running tasks? Currently it seems like overlord single node failure will cause ingestion to stop.

Thanks
Shuai

Hey Shuai,

Are you running tasks directly on your overlord? If so then you can get high availability by:

When 0.8.3 is released you can also set druid.indexer.task.restoreTasksOnRestart=true and that will allow you to restart nodes any time you want and not have tasks fail.

Hey Gian, thanks for the response

I’m running all middle manager nodes remotely. Currently when I kill the Overlord server, seems like pushing is still ok for a while without overlord. If that true that tranquility does not need talk to overlord once it knows which worker to talk to? As soon as an other overlord takes leadership, all the indexing tasks were killed. And tranquility will report a lot NoHostAvailable exceptions until next segment period starts.

That’s a awesome news we can do druid.indexer.task.restoreTasksOnRestart=true! I’m currently running 0.8.3 RC3, is this feature included here? This will make my life sooo much easier if it works.

Also, is the restoreTasksOnRestart an overlord config or a middleManager config? Would love it to be both but it’s good enough to be just overlord, middle manager can live with replications

One more question, sorry about multiple posts :slight_smile:

Reading through this: https://github.com/druid-io/druid/pull/1881

“make it possible to restart tasks on the same hardware”, does this mean the indexing service only be restorable if the same overlord was restarted? For HA, at least two Overlord nodes fight for leadership, it won’t be too helpful if the restore can only be done on the original overlord. Please let me if this understanding is not correct (hopefully it’s not :))

Hey Shuai,

Yes, that functionality is in 0.8.3-rc3.

Have you set druid.indexer.storage.type=storage? If not, try doing that, and that should allow two Overlords to share information about running tasks.

Just to be clear, when you have a fully HA overlord setup (runner=remote, storage=metadata, & two overlords) then you should be able to restart the overlords whenever you want and not have tasks fail.

And if you’re using 0.8.3 with druid.indexer.task.restoreTasksOnRestart=true, you should also be able to restart middleManagers whenever you want.

Gian, thanks so much! druid.indexer.storage.type=storage is all I need :), everything works great now!