IndexTask Incremental Publishing and forceGuaranteedRollup

On a nightly basic, we run an IndexTask to reindex data for the previous day.

We’ve noticed that the existing segments are dropped on the cluster while the indexing task is running which is not the behavior that we are hoping for. Ideally, the new segments would be built in the background and only be made available once the task is complete. That way, clients don’t see data temporarily disappear.

From reading through the IndexTask, it seems that the segments get dropped due to the “incremental publishing” part of the task. Whenever maxRowsInSegment or maxRowsInAppenderator is reached, the current segment is incrementally published which then drops other segments for the interval. It looks like this behavior was added in

I noticed that there is a forceGuaranteedRollup flag in the tuning config that appears to modify this behavior and not incrementally publish the segments which appears to be the behavior that we’re after. However, I don’t see any documentation on this flag so I’m unsure if it should be used and if so, what the correct way to use it is

Is forceGuaranteedRollup a solution to this issue? If so, what is the correct way to use it?