It is possible to enable automate compactin without manual intervention using a spec file?

Hello,

I’m writing this issue because I’ve been doing different tests to automate the druid segments automatically when creating a data source using the spec file.

Introducing a partitionSpec in the ingestion spec doesn’t seem to create automatic compaction, as the data sources show “Not enabled” in the compaction field. When checking the data source payload, the introduced partitionSpec can’t be found

Is there any way to enable compaction without manual intervention? I have read that it is possible using curl, but our idea is to make it as automatic as possible.

Thanks,
Fernando.

Hi Fernando,

You can enable automatic compaction by clicking on the blue pencil icon next to the “Not enabled” text.
This opens up a configuration dialog in which you can specify the partitioning strategy and target segment size, as well as other tuning options if you want.

Once you submit the configuration, autocompaction jobs will run automatically whenever there are a sufficient amount of non compacted data, if compute capacity is available.

The autocompaction tasks show up in the list of ingestion tasks with a prefix like coordinator-issued_compact.

1 Like

Dear Hellman,

I think I didn’t express myself well in the question, sorry for my mistake. I meant is it possible to enable compaction without using the druid UI? My idea was to add those compaction fields when I create a data source using a spec file, but when I add those fields to the file, it doesn’t work.

Thanks Hellmar, your answer is exactly right, but I’d like to add that there is no way to set autocompaction for a data source using the ingestion spec.

One could use API calls in order to reduce the manual intervention needed.

Thanks Vieth,

That’s the answer I was expecting, but I wanted to know if I’m missing something to read.

Best regards,
Fernando.