I am using a druid sink with apache flink and using tranquility to send data to druid. How can I support exactly once semantics with a druid sink ? Druid does not support two phase commits. Is there any trick to support exactly once with checkpointing in flink ?
Druid provides exactly-once delivery guarantees when data is ingested from Apache Kafka using Kafka Indexing Service.
How about flink , is there anything I can do on the flink side to guarantee exactly once. I have kafka as my flink source and the sink is druid, is there anything I can do to guarantee exactly once. Like replace an entire segment when there is drop over the wire or when the jvm crashes before checkpointing is complete.
In that scenario the simplest thing is to do kafka -> flink -> kafka -> druid.