Segments appearing in historical default tier even though datasource retention policy put it in custom tier

I have a datasource that I have set to be a custom tier that I call ‘cold_tier’; which I have set up in the retention policy. What I am noticing is that there are segments of that datasource that are residing in the default tier historical node.

When I look at a specific segment that is stored on the _default_tier, I see that it is also replicated on the cold_tier as well… as a matter of fact, I have set this datasource to have a replication factor of 2 and is stored on 2 cold_tier historical nodes and 1 _default_tier node (that is a replication of factor 3 not 2).

Here is my questions:

  1. Why are the segments going to _default_tier, when the retention policy states for it to goto cold_tier?
  2. The replication factor is set to 2 for this datasource, but it seems be ignoring that because I am assuming that is due to the _default_tier?
  3. How do I have my datasource not store data on the _default_tier?

Relates to Apache Druid 0.20

Can you send over a copy of your load rules?

Rachel,

I assume you mean retention policy?

I have attached screen shots of the retention policy along with screenshots of the services and and query I ran to verify which host the data resides on. Let me know if I am sending you the wrong things.

I think what is happening here is that the loadForever rule, which is not seen, but is still invoked is happening. I haven’t had a chance to play around with it, but I think you will need to add rules to make sure the cold data is removed from the default tier.

@Rachel_Pedreschi,

I assume you are talking about what I have circled in the picture below?

I see the cluster _default retention rules being reachable from every datasource retention policy, but I assume that it is a global apply i.e. there is only one _default retention, not a _default retention per datasource.

Also, follow up question. If I remove the _default_tier nodes, will the data replicate to the cold_tier at this point?