I am running 4 Realtime nodes that are ingesting data from Kafka using Firehose (linear sharding, each node handling 6 partitions). I want to safely kill one Realtime node such that there is no data loss i.e. all the data that was read by that node from Kafka should be persisted and it should commit Kafka offsets before handing over the partitions to other nodes. What are the possible options to ensure that there is no data loss?
You could use a TimedShutoffFirehose (see here: http://druid.io/docs/latest/ingestion/firehose.html) wrapped around the Kafka firehose to force it to shut down at a particular time (you could just set it to right now). After the firehose shuts down, wait for the realtime node to handoff all of its data and then you can safely kill it.