How to enable auto compaction?

Hi, I can follow the tutorial to get a compaction task working. But failed to get automatic compaction works. Would anyone share ae example what API to use and a sample payload ?

Many thanks,
Christine

Hi Christine,

See http://druid.io/docs/latest/operations/api-reference.html

Under POST /druid/coordinator/v1/config/compaction

Also linked from there is this: http://druid.io/docs/latest/configuration/index.html#compaction-dynamic-configuration

So POST something like

{“dataSource”:“wikipedia”,“keepSegmentGranularity”:true,“taskPriority”:25,“inputSegmentSizeBytes”:419430400,“targetCompactionSizeBytes”:419430400,“maxRowsPerSegment”:null,“maxNumSegmentsToCompact”:150,“skipOffsetFromLatest”:“P1D”,“tuningConfig”:null,“taskContext”:null}

to /druid/coordinator/v1/config/compaction

Hope this helps.

First thank you so much for your response. I did as you suggested, as if I don’t see any compaction.

  1. issue curl command to POST the request, I don’t see any response on the console;

  2. curl -X GET http://:8090/druid/coordinator/v1/config/compaction, don’t see any response;

  3. On overlord console, I don’t see a compaction task running (maybe auto compaction won’t show on the console)

  4. In overlord log (set debug mode), I do see something like this: doesn’t see much error message except 404 code.

HttpChannelOverHttp@24e0d1d7{r=1,c=false,a=IDLE,uri=//:8090/druid/coordinator/v1/conf ig/compaction,age=1}
handle //:8090/druid/coordinator/v1/config/compaction

3651 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,847 HttpChannelState.handling() -
handling HttpChannelState@20df4193{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false}

3652 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,847 HttpChannel.handle() -
HttpChannelOverHttp@24e0d1d7{r=1,c=false,a=DISPATCHED,uri=//:8090/druid/coordinator/v1/config/compaction,age=1} action DISPATCH

3653 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,847 Server.handle() - REQUEST POST
/druid/coordinator/v1/config/compaction on HttpChannelOverHttp@24e0d1d7{r=1,c=false,a=DISPATCHED,
uri=//:8090/druid/coordinator/v1/config/compaction,age=1}

3654 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,847 GzipHandler.handle() -
org.eclipse.jetty.server.handler.gzip.GzipHandler@d2e6b0b handle Request(POST
//:8090
/druid/coordinator/v1/config/compaction)@c80b337 in null

3655 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,848 ContextHandler.doScope() - scope
null>>/druid/coordinator/v1/config/compaction @ o.e.j.s.ServletContextHandler@eac3a26{/,[jar:file:/app/apache-druid-0.13.0-incubating/lib/druid-server-0.13.0-incubating.jar!/static,
jar:file:/app/apache-druid-0.13.0-incubating/lib/druid-indexing-service-0.13.0-incu
bating.jar!/indexer_static],AVAILABLE}

3656 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,848 ContextHandler.doScope() -
context=||/druid/coordinator/v1/config/compaction @
o.e.j.s.ServletContextHandler@eac3a26{/,[jar:file:/app/apache-druid-0.13.0-incubating/lib/druid-server-0.13.0-incubating.jar!/static,
jar:file:/app/apache-druid-0.13.0-incubating/lib/druid-indexing-service-0.13.0-incuba
ting.jar!/indexer_static],AVAILABLE}

3657 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,848 session.doScope() -
sessionHandler=org.eclipse.jetty.server.session.SessionHandler280078161==dftMaxIdleSec=-1

3658 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,848 session.doScope() - session=null

3659 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,848 ServletHandler.doScope() -
servlet |/druid/coordinator/v1/config/compaction|null -> default@5c13d641==org.eclipse.jetty.servlet.DefaultServlet,jsp=null,order=-1,inst=true

3660 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,848 ServletHandler.doHandle() -
chain=org.apache.druid.server.security.SecuritySanityCheckFilter-7f5538a1->org.apache.druid.server.security.AuthenticationWrappingFilter-31c628e7->org.apache.druid.server.security.AllowOptionsResourceFilter-7d3fb0ef->org.apache.druid.server.security.PreResponseAuthoriza
tionCheckFilter-64db4967->org.apache.druid.server.http.RedirectFilter-24a2e565->com.google.inject.servlet.GuiceFilter-3e36b7a0->default@5c13d641==org.eclipse.jetty.servlet.DefaultServlet,jsp=null,order=-1,inst=true

3661 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,849 ServletHandler.doFilter() - call
filter org.apache.druid.server.security.SecuritySanityCheckFilter-7f5538a1

3662 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,849 ServletHandler.doFilter() - call
filter org.apache.druid.server.security.AuthenticationWrappingFilter-31c628e7

3663 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,849 ServletHandler.doFilter() - call
filter org.apache.druid.server.security.AllowOptionsResourceFilter-7d3fb0ef

3664 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,849 ServletHandler.doFilter() - call
filter
org.apache.druid.server.security.PreResponseAuthorizationCheckFilter-64db4967

3665 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,849 ServletHandler.doFilter() - call
filter org.apache.druid.server.http.RedirectFilter-24a2e565

3666 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,849 ServletHandler.doFilter() - call
filter com.google.inject.servlet.GuiceFilter-3e36b7a0

3667 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,852 ContainerResponse.logException()

  • Mapped exception to response: 404 (Not Found)

3668 1543[^[[36mDEBUG^[[m] 2019-04-08 12:53:03,853 session.doScope() -
FinalSession=null old_session_manager=null
this=org.eclipse.jetty.server.session.SessionHandler280078161==dftMaxIdleSec=-1

3669 [^[[36mDEBUG^[[m] 2019-04-08 12:53:03,853 Server.handle() - handled=true
async=false committed=false on
HttpChannelOverHttp@24e0d1d7{r=1,c=false,a=DISPATCHED,uri=//ulv101
143.fg.rbc.com:8090/druid/coordinator/v1/config/compaction,age=7}

Is it possible you are using an older version of Druid? Auto-compaction is a relatively recently added feature.

Hi, Gian
Thanks for your response. I am using the latest Druid
0.13.0-incubating release. Not sure what I missed (maybe some setting
on coordinator/overlord). Did you have the auto-compaction working?
If, so could you share your configuration?

Hi Christine,

Can you please share what post for the compaction task?

It does work in 0.13.

Thanks & Rgds
Venkat

    Hi, Gian
    Thanks for your response. I am using the latest Druid
    0.13.0-incubating release. Not sure what I missed (maybe some setting
    on coordinator/overlord). Did you have the auto-compaction working?
    If, so could you share your configuration?

Hi, Venkat

Here is what I did:

curl -XPOST -H’Content-Type: application/json’ -d @compact.json http://:8090/druid/coordinator/v1/config/compaction

compact.json contains: (I also tried a payload to leave turningConfig empty).

{

“dataSource”:“dataSource”,

“keepSegmentGranularity”:true,

“taskPriority”:25,

“inputSegmentSizeBytes”:419430400,

“targetCompactionSizeBytes”:419430400,

“maxRowsPerSegment”:null,

“maxNumSegmentsToCompact”:150,

“skipOffsetFromLatest”:“P1D”,

“tuningConfig”: {

"type": "index",

"targetPartitionSize" : 5000000,

"maxRowsInMemory" : 25000,

"forceExtendableShardSpecs" : true

},

“taskContext”:null

}

Hi Christine,

Can you please try replacing the port number from 8090 to 8081? 8090 is generally overlord and 8081 is co-ordniator.

Thanks & Rgds

Venkat

Hi, Venkat

What a dumb mistake. Thank you so much for your help.
I re-did this
curl -XPOST -H’Content-Type: application/json’ -d @d_compact.json http://:8081/druid/coordinator/v1/config/compaction

And I can get the compaction configuration via the get endpoint:

curl -X GET http://:8081/druid/coordinator/v1/config/compaction

{“compactionConfigs”:[{“dataSource”:“mts”,“keepSegmentGranularity”:true,“taskPriority”:25,“inputSegmentSizeBytes”:419430400,“targetCompactionSizeBytes”:419430400,“maxNumSegmentsToCompact”:50,“skipOffsetFromLatest”:“P1D”,“tuningConfig”:{“maxRowsInMemory”:75000,“maxTotalRows”:20000000,“indexSpec”:{“bitmap”:{“type”:“concise”},“dimensionCompression”:“lz4”,“metricCompression”:“lz4”,“longEncoding”:“longs”},“maxPendingPersists”:0,“publishTimeout”:0,“type”:“index”},“taskContext”:null}],“compactionTaskSlotRatio”:0.1,“maxCompactionTaskSlots”:2147483647}

But I still don’t see any changes of the segments for the datasource. For example, I see 40 segments were created for 2019-03-31, it stays the same. I would expect that at some point it will get compacted.

If I run a static compaction task, then I do see 40 segments got compact into 1.

Are there any configuration I need to change in the auto compaction setting? How to monitor that the compaction tasks are running?

Many thanks,

Christine