We are submitting one index_parallel ingestion task per month and so for 14 months, we submit 14 ingestion for tasks with multiple files per month per ingestion task. We do reingestion every day for 14 month time period. For better consistency of reporting, we would like the swapping the old segments with new segments for 14 months at the same time. How can we achieve this?
Druid will automatically take care of it, As you reingest the data the newer version of the segment will be loaded and the older one will be unloaded from the historical.
Please read this for more details on updating the exiting data in druid :
Thanks and Regards,
Thanks for the reply. I have read that. I would like to know more at what level the atomic swapping happens?
Let’s say an ingestion task created 10 new segments. The atomic swapping is done at segment level OR atomic swapping is done for all the segments produced in the ingestion_task?
I am looking for case where I have 14 ingestion_tasks with each producing 10 new segments. I want to atomic swap for 14 ingestion_tasks * 10 new segments per tasks = 140 segment swap in an atomic manner. How can we do it ?