I have been working on a way to ingest data through Google Dataflow and Tranquility and everything seems to work like a charm. Since Dataflow can be used both for batching and streaming,
I would like to be able to set the interval I’m working on to be able to ingest data from a past interval but using realtime tasks. Is there a way to create a realtime tasks “in the past” for reindexation purposes (like reindexing the same data on the same datasource but with a different granularity)?
I’ve seen the interval set in tranquilities source files but is there a clean way to do this? I’m also wondering if I can use realtime tasks for reindexation like this and if it’s not going to be problematique when merging with previously ingested data.