I’m trying to figure out how to migrate data. Lets say I want to add a new dimension. segmentGranularity is set to day.
Lets assume the current time is “2015-10-14T12:10:00Z”
I’ll stop the current realtime task
Run the batch ingestion task with interval “2014/2015-10-14T12:10:00Z”
Start the realtime task again
After few minutes I notice that all the events from “2015-10-14T00:00:00Z/2015-10-14T12:10:00Z” are gone. I guess this is because the realtime index task takes priority because the segmentGranularity is set to day.
I can see two options how to solve it:
Step by step data migration: start sending events with the new dimension, then do the batch ingestion a day later and after that switch the app to use the new dimension.
Reduce the segmentGranularity to an hour. This way I lose maximally an hour worth of events, but these can be ingested through the batch ingestion task an hour later.
Obviously the best option is the (1) approach. However, sometimes it would be better to use a quicker route. Is it possible without setting the segementGranularity to