I had initially setup a druid cluster with 2 historical nodes with 30gb memory each. 2 middle manager nodes, one node with coordinator and overlord running, 1 broker node.
After successfully running it for 3-4weeks, I saw that my tasks were staying in the running state even after the window period. I then happened to add one more historical node with same configuration, this resulted in my tasks working fine again.
What this meant was all the data ingested to druid is going to memory and I will have to keep on adding historical nodes.
Is there a way to flush some of the data from memory to deep storage and it should get loaded into memory whenever a query is fired against that set of data?