I have noticed recently that when my Kafka Consumers using tranquility-core consistently fail to send to a handful of middleManagers tasks say (18 out of 60), there is a complete data loss(Out of 25 million per minute only less than 10K is being pushed). The 18 tasks reside on two middleManagers and when I removed the middle Managers the data flow was back to normal. If I had set the replication to 2 then I would have had another set of segments that could potentially save me from this data loss.
Is this the intended behaviour?
Is there a configuration I am missing that could help me circumvent this to an extent?
Or Is this an improvement that can be thought of that could mark down these nodes and not send data at all?
Would like to hear your thoughts on this
I am running on Druid 0.8.2
Replication : 1