I’ve set up druid dynamic compaction config on a realtime datasource but it seems this is even failing to launch a task.
If helps, I have long running indexer 5h with an intermediateHandoffPeriod of 5mins.
Couldn’t find anything anywhere on the below… any ideas pls?
are you running the overlord with the “local” mode by any chance (https://druid.apache.org/docs/latest/design/overlord.html)?
The auto compaction doesn’t work if the overlord is running in the local mode.
BTW, intermediateHandoffPeriod is to trigger handoff when your stream data is too sparse. In general, you don’t have to set it but can use just the default (which is infinite).
Thanks Jihoon. It was indeed running with “local” mode. Changed it to remote since but facing other issue acquiring the locks on the segment as the indexers are long running. If I set 100 priority on the compaction tasks then the indexer fail…so struggling to find a compromise
Oh yeah, before 0.16, compaction and indexing tasks cannot be run at the same time for the same time chunk. I would suggest to set a higher priority for indexing tasks and set ‘skipOffsetFromLate’ for auto compaction, so that they are not likely to run for the same time chunk at the same time.
In 0.16.0, a new compaction type, i.e., minor compaction, was introduced. The minor compaction allows you to run compaction and indexing tasks for the same time chunk at the same time. You can enable minor compaction by setting ‘forceTimeChunkLock’ false. Please note that this minor compaction is still experimental and see https://github.com/apache/incubator-druid/issues/7491 and https://druid.apache.org/docs/latest/ingestion/tasks.html for more details.