We have a test cluster that was set up with Druid 0.9.0 with hot and cold tier along with load rules. We had plenty of data loaded in both tiers.
Today I upgraded to Druid 0.9.1.1. All cluster nodes started nicely, but the coordinator console showed no data in the cold tier and only very few data in hot tier. I waited for data to load but noticed that the data-volume for the hot tier went up and down and up and down again.
Looking at the log file of one of the historical nodes, I see that the same segment gets loaded and dropped over and over again.
I attached a log fragment to this post, having grepped the historical log-file for one single segment.
A couple of weeks ago I had already deployed Druid 0.9.1.rc1 onto the same test-cluster and did not have any problems. This however was at a time when we didn’t have tiering in place yet.
BTW: when upgrading to Druid 0.9.1.1, is it necessary to delete and recreate the metastore database? (I didn’t do this)
druid_091_loading-droping-issue.txt (13.6 KB)