Will a risen real-time node override history data by handing-off debt?


If i have 2 real-time nodes and first of them stops. Second will hand-off 1 hour of data to historical. When i restart first, will it override the historical data by handing-off the debt?

Thanks, Vadim.

Hey Vadim,

If the realtime nodes are supposed to be “replicas” of each other (they have the same shardSpec) then what happens depends on the deep storage segment pusher implementation. The behavior is not strictly defined, potentially the older version would stay or potentially the newer version would override.

If the realtime nodes are not “replicas” (they have different shardSpecs) then they will not override each other.

One other note. This funkiness with realtime node “replicas” is one of the reasons we usually recommend running with tranquility if you need realtime replication. It is a lot better at keeping the replicas in sync. It’s not perfect, but it tries really hard.

If you need perfect, batch ingestion is the way to go, potentially with a lambda architecture (hybrid batch/realtime).

In the future we intend to offer higher fidelity streaming ingestion options.