Flush data from Historical Node Memory to Deep Storage

I had initially setup a druid cluster with 2 historical nodes with 30gb memory each. 2 middle manager nodes, one node with coordinator and overlord running, 1 broker node.

After successfully running it for 3-4weeks, I saw that my tasks were staying in the running state even after the window period. I then happened to add one more historical node with same configuration, this resulted in my tasks working fine again.

What this meant was all the data ingested to druid is going to memory and I will have to keep on adding historical nodes.

Is there a way to flush some of the data from memory to deep storage and it should get loaded into memory whenever a query is fired against that set of data?

Hi abhishek.

Check your segment retention policy(segment load rule)

Check you historical node configuration - druid.server.maxSize

Thanks, my bottleneck indeed was druid.server.maxSize, as I was assuming that data will be moved to historical storage and the node disk will be freed up.
I am now increasing the capacity of cluster based on druid.server.maxSize.

However, I would like to know if there is a way to not keep old data on the disk , so that I dont have to keep adding disks/nodes and don’t have to keep data forever and may only be pulled on to the disk if queried.