Coordinator dynamic properties - killDataSourceWhitelist

Hi,
I want to delete old version segments from deep storage after ingesting newer segments (with updated schema and logic). Here is what I did:

in coordinator properties:

druid.coordinator.kill.on = true

druid.coordinator.kill.period = PT1H

druid.coordinator.kill.durationToRetain = P1D

druid.coordinator.kill.maxSegments = 1000

After that I posted dynamic properties to coordinator’s endpoint - http://<COORDINATOR_IP>:/druid/coordinator/v1/config

{

“killDataSourceWhitelist”: [“summaries_hourly”, “details_hourly”]

}

This returns “200 OK” with blank values for KillDataSourceWhitelist

{“millisToWaitBeforeDeleting”:900000,“mergeBytesLimit”:524288000,“mergeSegmentsLimit”:100,“maxSegmentsToMove”:5,“replicantLifetime”:15,“replicationThrottleLimit”:10,“balancerComputeThreads”:1,“emitBalancingStats”:true,“killDataSourceWhitelist”:,“killAllDataSources”:false}

I see that there are lots of segments marked as “used=false” in druid_segments table in metadata storage but still deep storage files are not deleted. Am I missing something here?

Thanks,

-Suresh

Are you sure you did a “POST” and not “GET” while sending the request? How did you send the request?

Also, if you are using druid version less than 0.10.1, then I would POST whole configuration i.e.

{“millisToWaitBeforeDeleting”:900000,“mergeBytesLimit”:524288000,“mergeSegmentsLimit”:100,“maxSegmentsToMove”:5,“replicantLifetime”:15,“replicationThrottleLimit”:10,“balancerComputeThreads”:1,“emitBalancingStats”:true,“killDataSourceWhitelist”:[“summaries_hourly”, “details_hourly”],“killAllDataSources”:false}

because in older versions of druid, coordinator dynamic config partial update would make everything else to default to zeros.

hth,

Himanshu

Hi Himanshu,
Yes. I am using curl to POST configuration to coordinator endpoint. Currently, I’m using v9.2 and that may be the reason partial configuration posting was defaulting every configuration. So, other than this dynamic configuration in Coordinator, anything else is required?

Thanks,

-Suresh

All the configuration for setting up auto-kill is mentioned in the docs at http://druid.io/docs/0.9.2/configuration/coordinator.html .

Currently it looks like not working because “KillDataSourceWhitelist” is not configured unless “curl http://<COORDINATOR_IP>:/druid/coordinator/v1/config” says otherwise.

I can’t think why POST to set the property wouldn’t work, I have done it many times. Just to be sure can you give us the actual curl command you used (add ‘-vv’ flag to it too) and output printed?