How does Kafka Indexing Service handle late events?

Hi,

So, according to this article: https://imply.io/post/exactly-once-streaming-ingestion, the Kafka Indexing Service (KIS) can be used to ingest data with arbitrary old timestamps. But suppose there is a delay upstream due to which some events occur in Kafka with a timestamp corresponding to which data has already been indexed and handed off to historicals. How does KIS handle this scenario? Will these events be appended in some way to the segment in deep storage? Or will the entire segment be re-indexed?

Thanks

I’m not an expert by any means, but my understanding is a new ‘segment’ within the ‘time-window’ will be created.

I.e. if your granularity is 15-minutes, and say you already have 10 segments created (in a 15 minute window) (in cold-storage/historical), then when the old data arrives, you will have 11 segments (or more…depending on I guess how much arrives late, and how ‘closely’ that data arrives in ‘real-time’).

(Also assume your are streaming the data in).

See this post: (I asked essentially same question not too long ago):

https://groups.google.com/forum/#!topic/druid-user/H_YMiD_Wh3E