Estimate max memory used by Historical

Hello I’m looking for a formula that can give the estimate memory (Heap+DirectMemory) needed for my Historical pod.

I’ve found that formula :

But the current memory used is really higher than that (around x2) …

I’ve also found this additional formula for segment decompression:

Thus, there is additional direct memory overhead of (64KiB * number of columns read per segment * number of segments read) when reading segments. 

My question, is there any formula that can give a roughly estimation of the needed RAM ?

Unfortunately, the answer is no, not beyond what you’ve already referenced. You saw in the Basic Cluster Tuning doc that:

Please note that this document provides general guidelines and rules-of-thumb: these are not absolute, universal rules for cluster tuning, and this introductory guide is not an exhaustive description of all Druid tuning properties, which are described in the configuration reference.

I came across this old discussion about HISTORICAL CLUSTER RAM REQUIREMENT ESTIMATION, and it includes a couple of calculations. Perhaps it can provide some further guidance.

Hi OlveBZH,

The formula mentioned in your link is the rule of thumb that we use. There is a 24GB limit set that we don’t usually cross.

You can increase it to the max available. Just make sure you set xms and xmx to this same upper limit.

If you are using lookups, you can try to optimize them.