Mimic messageTime in Indexing Serivce?

Hey,

Is there some way to do this?

I passed in a custom timekeeper to my DruidBeams with a different time and messages are still getting thrown out.

How would I properly tell Tranquility what “now” is? or is this totally out of scope for what Tranquility is going for?

Thanks,

+1, seems tranquility does not support backfill and rely on replication factor for availability. Very curious to know if anything can be done easier than running standalone index jobs.

Hey Nicholas,

You can provide a custom Timekeeper to Tranquility that changes its definition of what “now” is. This should work but may be a bit awkward to set up. Generally Tranquility (and Druid realtime ingestion) are intended for current data and not for backfills. We do intend to make streaming backfills easier in the future.

Gian,

I am using a custom timekeeper at the moment and it looks like its working pretty well. The purpose of this was to implement some checkpointing with Kafka 9 and saving offsets manually just in case a consumer goes down it can pick back up from where it left off.

But this leads me to another Druid specific question: When this does happen, and tranquility sends a beam with an older timestamp for a task to ingest that older data, will the persisted segment be overwritten or merged?

Thanks again,

Tranquility actually has some code in it to prevent re-opening old segments. It doesn’t currently have any way to add data to a time range that has already been handed off. So, in that case it’d just drop the old messages. This will change in a future version.