appendToExisting equivalent of kafka supervisor config

Hi guys,

For static firehose usage, we have this flag “appendToExisting” in the ioConfig which gives us some options on whether to replace the data or merge it (as described here

Do we have something similar to work with kafka supervisors?

Hi Karthik:

Kafka indexing will not append to existing segments. if an event is received late, a new segment will be created for it regardless of when it was received. If there are too many small segments created due to late arrival data, you probably want to run compaction tasks to optimize them .

Hope this helps.

Makes sense. Thanks for confirming.