killTask Side effects on Historical Segment-Cache

Hey Everyone,

I have been scheduling(cron job) killTask for the window of 1 month, but I observed a side-effect of this - historical deletes all the segments of that datasource for which the killTask is fired and starts re-downloading all the left over segments.

Is this the normal behavior or am I going wrong somewhere ? Is it because permanent deletion also includes disable datasource command ?

Also, I would like to know if the same situation persists if I use dynamic delete/drop segments provided by coordinator ?

Cheers,

Akul

Hi
After task is done, hands-off will begin.
So if you kill the task, segments are removed.
I think you wanna have data(segments) retention period. I that right???

Hi Josh,

Yep correct. And I also want to delete the data(segments) from the deep storage as well.

Hi

So you use ‘Segment Destroying Tasks’ query in (http://druid.io/docs/latest/ingestion/tasks.html)?

As my cluster, upper query is available and I used to use upper query, retention rule in coordinator web ui and coordinator option for data retention.

Hey Josh,

I’m aware about the possibilities. My exact question here is:

I have been scheduling(cron job) killTask for the window of 1 month, but I observed a side-effect of this - historical deletes all the segments of that datasource for which the killTask is fired and starts re-downloading all the left over segments.

Is this the normal behavior or am I going wrong somewhere ? Is it because permanent deletion also includes disable datasource command ?

Also, I would like to know if the same situation persists if I use dynamic delete/drop segments provided by coordinator ?