Tranquility Replication

I tried searching for my questions and found related information, but not my exact questions…apologies if this is a repeat.

I have a few questions around replication with Tranquility. My understanding is with replication set to two, Tranquility will create two tasks with identical partition numbers and events go to both replicas.

a) Are the tasks guaranteed to be on separate MiddleManagers?

b) If only one MiddleManager is active (below the replication level), will writes succeed?

c) When the segment goes to deep storage, are two copies stored?

d) What happens if a replica goes down and comes back up?

Hey Andre, responses inline.

I tried searching for my questions and found related information, but not my exact questions…apologies if this is a repeat.

I have a few questions around replication with Tranquility. My understanding is with replication set to two, Tranquility will create two tasks with identical partition numbers and events go to both replicas.

a) Are the tasks guaranteed to be on separate MiddleManagers?

Yes.

b) If only one MiddleManager is active (below the replication level), will writes succeed?

That depends. Tranquility doesn’t like tasks sitting indefinitely in pending state, so if you have a replication factor of 2 and only one middle manager, one of the tasks will stay in pending (since it’s not allowed to run on the same node as the other task) and Tranquility will wait until that task has started running before sending messages to any of the replicas. If on the other hand the middle manager dies while running, Tranquility will wait until the task is marked as failed (timeout set by druid.indexer.runner.taskCleanupTimeout) and then will drop that replica and push events to the live task.

c) When the segment goes to deep storage, are two copies stored?

Yes, both replicas will push their segments to deep storage. The last one pushed (the one with the higher version timestamp) is the one that will be served by Druid historical nodes.

d) What happens if a replica goes down and comes back up?

Similar to b), when a replica goes down, Tranquility will keep retrying until the replica either comes back (at which point it will send all the messages that it buffered to all the replicas) or the middle manager timeout elapses and the replica is marked as failed (at which point it will drop that replica and push the buffered messages to the live replicas).

Perfect. Thanks for the reply.