Upgrading Overlord and Middle Manager / restarting index tasks

Hi - say we have an application using Tranquility to send data to a Druid Indexing Service task. While this is running, we find we need to restart the Overlord and/or Middle Manager processes to deploy an update. This could be a new Druid version, and update to some important config values, etc. We found this document:


Our use case would fall under the Indexing Service Without Autoscaling. Is this really the only way to do it? i.e. send disable POST to Middle Manager and then you have to wait until the current indexing tasks are done (could be an hour or a day depending on settings) so that the indexing tasks for the next time period get scheduled on a different Middle Manager, then you can roll the disabled MM?

What if an index task stops halfway through its time period? Does the index task store any kind of intermediate state on disk that can be recovered/continued if the index task starts up again?



Yeah, that’s the only way to do a rolling update without downtime and without autoscaling. The tasks don’t currently have the ability to restore state after being restarted.

Let’s say instead of using the Indexing Service, we were using a standalone Realtime node, and instead of using Tranquility we were using Kafka. The docs [1] state “Standalone real-time nodes can be updated one at a time in a rolling fashion.” What happens when I stop a Realtime node and then restart it? The Realtime node was likely building up a new segment file so it had some intermediate state. What happens to that state? Is it flushed to disk and then recovered when the Realtime node restarts?

[1] http://druid.io/docs/0.8.0/operations/rolling-updates.html

Realtime nodes reading from Kafka will periodically flush data to disk, and commit their Kafka offsets to ZooKeeper when they flush to disk. When you stop a node, anything that wasn’t flushed is thrown away. On next startup, it will load up all the data from disk, load up committed offsets from ZooKeeper, and resume ingesting.

Great, thanks for all the info Gian! Does the Realtime node use a shutdown hook to flush to disk, so if we send it a SIGTERM there will be a clean shutdown?

No matter how you shut down the realtime node, it won’t flush in-memory data to disk. It will always discard in-memory data. But, usually this is OK, since when it next boots up it will resume from offsets matching what it has actually persisted to disk. So the discarded data would be reingested from Kafka.

If you need things flushed to disk more often to minimize the amount of reingesting needed on restart, you can set the intermediatePersistPeriod shorter.

If we have 3 tranquility apps (configured with 1 partition, 1 replica) to consume kafka topic, which consists of 7 partitions, when one of them is killed, the rest two tranquilites will rebalance to consume all 7 partitions.
Will these two tranquilities start from the exact offsets to consume the topic without missing any messages?

在 2015年9月4日星期五 UTC+8上午6:15:35,Gian Merlino写道:

Hey Xuehui,

Tranquility does not have a builtin Kafka consumer, so if you are consuming from Kafka into Tranquility, you probably are using some third-party code to do that. So the answer to your question depends on how that third-party code works.

If it’s something using the high-level Kafka consumer, and you are doing manual committing only when the Tranquility futures actually resolve, then you would not lose any messages (but you might get some duplicates).