Documentation about Compaction

Hi,

I’ve been working on Druid’s compaction capabilities for the last few days and I’ve found the documentation quite confusing.

The problems I’ve seen so far:

  • UI shows a parameter called “skipOffsetFromLatest” but I cannot see it in this doc.
  • Docs mentions ioConfig is required but UI doesn’t send it while submitting compaction task. No dimensionsSpec and metricsSpec either.
  • The docs suggest that segmentGranularity can be changed but I’m not able to see such an option in the UI. If I manually prepare a compaction spec and submit it (with segmentGranularity included), I’m able to submit it but I cannot see it when I send a GET request to the API. (/coordinator/v1/config/compaction)
  • Inside the manually prepared compaction spec, I’ve passed a partitionsSpec and I’m not able to see that either.
    This is the spec I’m sending:

{
“type”: “compact”,
“dataSource”: “my_datasource”,
“segmentGranularity”: “WEEK”,
“skipOffsetFromLatest”: “P3D”,
“ioConfig”: {
“type”: “compact”,
“inputSpec”: {
“type”: “interval”,
“interval”: “2020-01-01/2020-04-18”
}
},
“tuningConfig”: {
“type”: “index_parallel”,
“maxNumConcurrentSubTasks”: 2,
“partitionsSpec”: {
“type”: “hashed”,
“numShards”: 1
}
}
}

And this is what I get from the API:

$ http http://localhost:8888/druid/coordinator/v1/config/compaction/my_datasource
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 265
Content-Type: application/json
Server: Jetty(9.4.12.v20180830)
Vary: Accept-Encoding, User-Agent

{
“dataSource”: “my_datasource”,
“inputSegmentSizeBytes”: 419430400,
“maxRowsPerSegment”: null,
“skipOffsetFromLatest”: “P3D”,
“taskContext”: null,
“taskPriority”: 25,
“tuningConfig”: {
“indexSpec”: null,
“maxBytesInMemory”: null,
“maxNumConcurrentSubTasks”: 2,
“maxPendingPersists”: null,
“maxRowsInMemory”: null,
“maxTotalRows”: null,
“pushTimeout”: null,
“splitHintSpec”: null,
“type”: “index_parallel”
}
}

Also, what I’ve understood from my reading through the docs is that the coordinator submits this task to a worker (to whom is not mentioned by the docs but I’m assuming it’s owned by middleManagers) but I’m not able to see a task for compaction nor any logs about it.

What am I missing? Is the docs really wrong?