[druid-user] Want to load data into two different historicals of different tier

Hi All,

I want to load data into two different historicals.
For Example: I have 2 historical nodes with named default_tier and cold_tier.
I want to load 10 days of data into historicals, in that first 3 days of data into first historical( i.e default_tier) and rest 7 days of data into second historical( i.e cold_tier).
For this we have come up with the rule as below:


But the problem is when ingestion is running, whatever the new data(future data) which is getting added is loaded into default tier historical, but the older data is not getting off loaded from default_tier historical and loaded into cold_tier historical.

Is there any way to dynamically off load the data or do we need to re trigger the rule for some interval??

Regards,
Vinay

Hi Vinay,

The Coordinator prioritizes loading segments over dropping segments. Could you please share your Coordinator log? Or could you look for messages such as org.apache.druid.server.coordinator.rules.LoadRule - Throttling replication for segment or org.apache.druid.server.coordinator.rules.LoadRule - Loading in progress, skipping drop until loading is complete?

Best,

Mark

Hi Mark,
Yes, I am able to find the messages in the log and have attached the log.
But now this is the load rules I have applied similar to above with HOUR concept.

In the logs, i am able to see that loading has been priortised over drop.
Segements are getting off loaded in cold_tier but not getting loaded back with the applicable segments based on the rules.
How we can off load and load the segments with equal priority?
Once the segment has been offloaded from the default tier by loadByPeriod rule, is that segment available to load for the cold tier?
Also, the log says “org.apache.druid.server.coordinator.rules.LoadRule”,“message”:"Loading in progress, skipping drop until loading is complete" , when this loading will be completed, as we will be having continuous ingestion of data?
Regards
Vinay