We are facing one problem while using druid and not able to get the solution for the same.
We have a batch ingestion pipeline and have delayed events coming in and want to override the data in current segment rows with latest data.
What we tried:
We tried using delta ingestion and is working fine but problem with data ingestion that its not overriding the value means for metric its aggregating the values and for dimension it is treating it as different record.So creating a new row in segment.
Also we thought while making merge.on true on co-ordinator will merge the old and new segment but as we saw it ignore the old segment.
As per our use case we wanted that if in delayed event timestamp and a list of dimension is same just override the old rows with new one
Any lead will help