Retention Rules conflicting?

Sorry there is a lot in here, trying to understand druid…

Hi all, doing a lot of experimenting with Druid, trying to understand what can and cannot do with it.

I have data-source, with a load-forever rule.

on the consolidated UI, I clicked Drop-Data.

At which point it does seem like data has/is beeing dropped, but the segment count keeps going up and down (no data is being loaded/streamed in on this datasource atm), and sometimes the row count was at 0, then it showed up again as non zero…this happened a few times.

I’m guessing the load-forever rule is running…

I cannot see if a ‘drop-data’ task is still running?

This data source eventually disappeared from consolidated UI, so I am guessing the drop-data finally completed. (I am not trying to figure out how to ‘reload’ the data from cold-storage)

I wasn’t expect the datasource to vanish…I would expect it show up but with no segments, calling:

/druid/coordinator/v1/datasources?full

does not show it either? How would I create a ‘kill’ task if for datasource that doesn’t exist? I was led to believe I needed to drop the data, then I could create kill task for the datasource, but if the datasource no longer exists?

(I did delete the kafka-indexing task for datasource)…perhaps this combination of deleting the indexing task and dropping the data also 'drops datasource from exsistance?)

Hi Daniel,

Druid doesn’t keep dataSource state, but only segment state. The coordinator basically shows only ‘published’ segments which are set to true for ‘used’ in metadata store. This is why the coordinator API doesn’t return the dataSource if it has no published segments. But, still, you should be able to kill those dropped segments because they do exist in the metadata store. Note that the kill task can delete only unpublished segments (‘used’ = false). Since it checks unpublished segments, the kill task could handled the dropped dataSource properly.

At which point it does seem like data has/is beeing dropped, but the segment count keeps going up and down (no data is being loaded/streamed in on this datasource atm), and sometimes the row count was at 0, then it showed up again as non zero…this happened a few times.

The UI is backed by the system table which runs on brokers and is periodically synced. So, it may take a few seconds until the change is actually applied to the system table.

However, if you saw something goes back and forth, it might be a bug. Would you give me some more details?

Jihoon

In reference to:

. The coordinator basically shows only ‘published’ segments which are set to true for ‘used’ in metadata store

Indeed I see:

/druid/coordinator/v1/metadata/datasources?includeDisabled

shows the datasource with the dropped data.

May natural question now is…how can I ‘re-enable’ a datasource, clearly It is currently not possible via the UI, but I’m guessing i can upload a new load rule via the coordinator api, something like:

POST: /druid/coordinator/v1/rules/{Disabled_dataSourceName}{
“type” : “loadForever”
}

or if I only want to laod a specific data range:
{
“type” : “loadByInterval”,
“interval”: “2012-01-01/2013-01-01”,
}

``


etc..

In reference to:

At which point it does seem like data has/is beeing dropped, but the segment count keeps going up and down (no data is being loaded/streamed in on this datasource atm), and sometimes the row count was at 0, then it showed up again as non zero…this happened a few times.

The UI is backed by the system table which runs on brokers and is periodically synced. So, it may take a few seconds until the change is actually applied to the system table.

However, if you saw something goes back and forth, it might be a bug. Would you give me some more details?

well I’m off or a 1 week vacation starting tomorrow, will likely not have time to ‘retrace my steps’. From what i remember…

I had data source with 44k segments, with the default cluster rule in place, the datasource had an associated kafka-indexing-task configured.

No new data was being sent to kafka topic.

I clicked drop-data in the UI.

…nothing appeared to change for a bit…and i didn’t see any ‘task’ or anything like that saying it was dropping the data.

So I think…I changed the retention rule to a new one (I was trying to define a drop rule and maybe a load 1m period rule), to a none default, but I think it was also load-forever.

I think shortly after this I did notice the segment/row count had dropped, during this window I did notice the rows count fluctuated up a few times, which made me thinkg I needed to somehow delete the load rules

i.e. in summary I created new rules while the ‘drop data’ action was running, which I think led to the up/down behavior of row counts displayed in the UI (presumably the segment count may have fluctuated up/down as well).

eventually I noticed the segment count/row count dropping.

Thanks for the details.

LoadRules and dropRules are a bit different. DropRules are about defining what segments should be unpublished, while loadRules are about defining how many replicas should be loaded for “published” segments.

So, they are not like completely opposite.

For your question, since loadRules only consider published segments, they don’t reload unpublished segments automatically. You can do it by updating metadata store manually or using reload by interval API in future releases (https://github.com/apache/incubator-druid/pull/7490 was recently merged into master and not released yet).

Regarding segment/row count fluctuation, I’ll check the code around it.

Thanks!

Jihoon