Delta Ingestion with overrides


We are facing one problem while using druid and not able to get the solution for the same.

Use Case:

We have a batch ingestion pipeline and have delayed events coming in and want to override the data in current segment rows with latest data.

What we tried:

We tried using delta ingestion and is working fine but problem with data ingestion that its not overriding the value means for metric its aggregating the values and for dimension it is treating it as different record.So creating a new row in segment.

Also we thought while making merge.on true on co-ordinator will merge the old and new segment but as we saw it ignore the old segment.

Desired behaviour:

As per our use case we wanted that if in delayed event timestamp and a list of dimension is same just override the old rows with new one

Any lead will help :slight_smile:

Any lead on this, can we reindex multiple data sources with sort and unique functionlity?