Impact of compaction on druid queries

Hi Team,

If compaction is in progress for a DAY of data, will there be an impact on the response returned for Druid queries (For example, if compaction is in progress, will the response for druid query miss out on certain rows returned in the response)?



Hi Vinay,

There should be no impact on queries during compaction, it happens essentially offline in the background and creates a new set of segments. The segments being compacted will be served by the historical nodes until the compaction is complete, at which point they are replaced with the new segments. Note that if rollup is enabled, the amount of rollup might be increased after compaction, resulting in fewer total physical rows, but I don’t think that is what you are asking.

If historical is involved, wont it affect queries? Aren’t the broker requests served by historical? I assumed the compaction takes at least one threat out of the thread pool allocated to historical.

Compaction essentially involves segment creating, so it’s as good as an ingestion process. When compaction is completed, the existing segments are replaced by the newly compacted segments. Historical is not involved in this.

Historical would continue to serve the segments requested by the queries and hence there won’t be any slowdown in the queries due to compaction.

Thank you. That makes it clear.