Recently I found our druid realtime node(standalone) forgot to delete some local segments which have been pushed to deep storage. After quick scanning codes, it seems realtime node listens to ZNodes and will hold local segments until got notified that some historical nodes have load the exactly segment.
What I hit seems my realtime node was in big GC trouble, then lost ZK connection for a while, then missed the ZNodes notification, then keeps the local segments forever. Correct me if I’m wrong please, thanks!
PS: I’m using druid 0.8.1, 3 nodes ZK cluster.