[druid-user] Druid auto-compaction behaviour

Apache Druid Q&A Google Groups
#1

Hello,

I have a hourly append ingestion that can add hourly segments from now to up to 30 days ago (720 hours)

I’d need the auto-compaction to start compacting all the last 30 days. Instead, druid creates 25 compaction tasks (I guess because there are 25 available peons) for the last 25 hours and stops, skipping the rest of segments.

Is there any way to configure the auto-compaction in a smarter way so that it creates compaction tasks for all the last 30 days hours (where needed)?

#2

Possibly inputSizeInBytes needs to be set higher to be able to handle the amount of data in older days. Can you send your compaction config (the json)?

#3

Tried to increase from 400MB to 1GB and no changes. Each auto-compaction task took exactly 1h of segments.
Meanwhile, found the druid.coordinator.period.indexingPeriod property and increased the frequence from 30 minutes to 5 minutes.