Compaction task for specific interval

Hi all,
I’m trying to run compaction task for 2019-07-06 date, it has 2.5G segment and I would like to split it

to segments of 500MB.
I tried to post this job:

{

“type”: “compact”,

“dataSource”: “events”,

“interval”: “2019-07-06/2019-07-07”,

“targetCompactionSizeBytes”: 536870912,

“inputSegmentSizeBytes”: 4194304000

}

``

When I tried to get the compactions jobs which are now running I got the following response:
Some questions:
1. Why the time interval is not specified?
2. Why the compaction task doesn’t have a unique Id? how can I post multiple tasks with different ids?

{

“compactionConfigs”: [

{

“dataSource”: “events”,

“keepSegmentGranularity”: true,

“taskPriority”: 25,

“inputSegmentSizeBytes”: 4194304000,

“targetCompactionSizeBytes”: 536870912,

“maxRowsPerSegment”: null,

“maxNumSegmentsToCompact”: 150,

“skipOffsetFromLatest”: “P1D”,

“tuningConfig”: null,

“taskContext”: null

}

],

“compactionTaskSlotRatio”: 0.1,

“maxCompactionTaskSlots”: 2147483647

}

``

Hi Alon:

Where did you get the response output?

When I submit a compaction task just like yours, the payload would be like


{

"type": "compact",

"id": "compact_DATASOURCE_2019-07-11T21:34:14.320Z",

"resource": {

"availabilityGroup": "compact_DATASOURCE_2019-07-11T21:34:14.320Z",

"requiredCapacity": 1

},

"dataSource": "DATASOURCE",

"interval": "2015-01-01T00:00:00.000Z/2016-12-18T00:00:00.000Z",

"segments": null,

"dimensionsSpec": null,

"metricsSpec": null,

"segmentGranularity": null,

"targetCompactionSizeBytes": 536870912,

"tuningConfig": null,

"context": {},

"groupId": "compact_DATASOURCE_2019-07-11T21:34:14.320Z"

}

Hi Ming,
thank you for your reply

Just for clarification My master node have the coordinator and the overlord dockerized.
When I submit the task , I POST it to http://[master-ip]:8081/druid/coordinator/v1/config/compaction
and the response return empty

{

“type”: “compact”,

“dataSource”: “events”,

“interval”: “2019-07-06/2019-07-07”,

“targetCompactionSizeBytes”: 536870912,

“inputSegmentSizeBytes”: 4194304000

}

``

When I GET this request http://master-ip:8081/druid/coordinator/v1/config/compaction

I got this response

{

“compactionConfigs”: [

   {

       "dataSource": "events",

       "keepSegmentGranularity": true,

       "taskPriority": 25,

       "inputSegmentSizeBytes": 4194304000,

       "targetCompactionSizeBytes": 536870912,

       "maxRowsPerSegment": null,

       "maxNumSegmentsToCompact": 150,

       "skipOffsetFromLatest": "P1D",

       "tuningConfig": null,

       "taskContext": null

   }

],

“compactionTaskSlotRatio”: 0.1,

“maxCompactionTaskSlots”: 2147483647

}

``

Hi Ming,
thank you for your reply

Just for clarification My master node have the coordinator and the overlord dockerized.
When I submit the task , I POST it to http://[master-ip]:8081/druid/coordinator/v1/config/compaction
and the response return empty

{

“type”: “compact”,

“dataSource”: “events”,

“interval”: “2019-07-06/2019-07-07”,

“targetCompactionSizeBytes”: 536870912,

“inputSegmentSizeBytes”: 4194304000

}

``

When I GET this request http://master-ip:8081/druid/coordinator/v1/config/compaction

I got this response

{

“compactionConfigs”: [

   {

       "dataSource": "events",

       "keepSegmentGranularity": true,

       "taskPriority": 25,

       "inputSegmentSizeBytes": 4194304000,

       "targetCompactionSizeBytes": 536870912,

       "maxRowsPerSegment": null,

       "maxNumSegmentsToCompact": 150,

       "skipOffsetFromLatest": "P1D",

       "tuningConfig": null,

       "taskContext": null

   }

],

“compactionTaskSlotRatio”: 0.1,

“maxCompactionTaskSlots”: 2147483647

}

``

Cool. At least it’s not returning any errors. So does the compaction task finish successfully? You can find the payload information from the running task logs, or from router UI…