segmentGranularity ignored for compaction task

Hi,

I
had multiple segments created per day for my datasource and wanted to compact the same at MONTH segment granularity. I tried submitted the following compaction task to the overlord.

{
“type”: “compact”,
“dataSource”: “xReports”,
“interval”: “2019-01-01/2019-12-31”,
“segmentGranularity”: “MONTH”,
“keepSegmentGranularity” : false
}

``

I
had data for 2 months (from May to June). After compaction i saw that only one segment was created for all the data and the MONTH granularity not respected. I had tried earlier with MONTH granularity and not specifying the keepSegmentGranularity parameter (which is believe is deprecated or to be deprecated soon)and didn't see new segments getting created.

However
on specifying keepSegmentGranularity as false compaction worked although not how i wanted.The compaction task seems to be ignoring the provided segment granularity and created single segment for the entire year. I saw following specification for compaction in the overlord log for the compact task. It doesn’t indicate the supplied granularity

2019-07-02T11:10:03,792 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {
  "type" : "compact",
  "id" : "compact_xReports_2019-07-02T11:09:59.155Z",
  "resource" : {
    "availabilityGroup" : "compact_xReports_2019-07-02T11:09:59.155Z",
    "requiredCapacity" : 1
  },
  "dataSource" : "xReports",
  "interval" : "2019-01-01T00:00:00.000Z/2019-12-31T00:00:00.000Z",
  "segments" : null,
  "keepSegmentGranularity" : false,
  "targetCompactionSizeBytes" : null,
  "tuningConfig" : null,
  "context" : { },
  "groupId" : "compact_xReports_2019-07-02T11:09:59.155Z",
  "dimensionsSpec" : null
}

``

Why doesn’t the compaction task use the segment granularity provided? Is this something that was broken for druid 0.13.0-incubating and is fixed now?

Thanks,

Prathamesh

It appears that support for segmentGranularity in compaction task was added in Druid 0.14.0-incubating.

You may want to lookup https://github.com/apache/incubator-druid/issues/7988

Hi Venkat,

The above issue you mentioned is for auto-compaction.

I was using older version with where on has to submit compaction tasks. the manual compaction task also supports segmentGranularity. However I found by looking at the commit that this feature was added in the version 0.14.0-incubating.

I was able to achieve what i was trying to do by making used of the option keepSegmentGranularity set to false. When using this one has to ensure to choose right interval as all segments in the specified interval would be compacted. So if one want to compact multiple segments of a DAY into single segment for the DAY and do this for entire month’s worth of data, compaction task would have to be submitted for each day.

Thanks,

Prathamesh