[druid-user] Re: Clarification about Segment Search Policy in Compaction Configuration

Hi Mihir,
I’m looking at Coordinator Process · 2020.11 and I see this note that you’re referring to. The description on Configuration reference · 2020.11 of inputSegmentSizeBytes makes me think that the “lot of small segments” is indeed a vague statement. I believe that the number of small segments itself doesn’t matter. What matters is when the number of segments multiplied by the segment size then exceeds the inputSegmentSizeBytes. The limitation is the RAM needed to open and compact multiple segments at once. You could add RAM and try increasing inputSegmentSizeBytes, or alternatively, decrease the number of compaction subtasks so less memory is needed at once. Did you try to run your compaction task? What happened?
Thanks,
Matt

Hi Mihir :slight_smile:

There’s a bit of sys table SQL I added to this post… it may help profile your intervals?

Note that it doesn’t reference this added bit of information!

AFIK the coordinator is issuing compaction tasks to the overlord - they are just a different tag of compact task. Kind of like automated maintenance.