Estimate max memory used by Historical

Hello I’m looking for a formula that can give the estimate memory (Heap+DirectMemory) needed for my Historical pod.

I’ve found that formula :

But the current memory used is really higher than that (around x2) …

I’ve also found this additional formula for segment decompression:

Thus, there is additional direct memory overhead of (64KiB * number of columns read per segment * number of segments read) when reading segments. 

My question, is there any formula that can give a roughly estimation of the needed RAM ?

Unfortunately, the answer is no, not beyond what you’ve already referenced. You saw in the Basic Cluster Tuning doc that:

Please note that this document provides general guidelines and rules-of-thumb: these are not absolute, universal rules for cluster tuning, and this introductory guide is not an exhaustive description of all Druid tuning properties, which are described in the configuration reference.

I came across this old discussion about HISTORICAL CLUSTER RAM REQUIREMENT ESTIMATION, and it includes a couple of calculations. Perhaps it can provide some further guidance.

Hi OlveBZH,

The formula mentioned in your link is the rule of thumb that we use. There is a 24GB limit set that we don’t usually cross.

You can increase it to the max available. Just make sure you set xms and xmx to this same upper limit.

If you are using lookups, you can try to optimize them.

Also bare in mind that Druid historicals use mmap when they read segment files, meaning that any free operating system memory will be used up as well to speed up things. I have heard of some people having tonnes of free system memory to reduce the risk of evictions … So you may also need to consider the size of the local segment cache, and think about how much OS memory will be available to cache things. E.g. if you have 8 threads available, and a query will take up all 8, think about how big each segment will be and how much OS memory there will be available to mmap it.

In general the RAM requirement formula is
Direct memory= ( processing.numThread + no of merge buffer +1) x buffer size + 2 * lookup size(incase lookup is configured to be off heap)
Heap Size = 0.5 * number of cores + 2* lookup size(for heap lookup which is default conf) + Cache size + (64kb * no of col accessed in queries * concurrent segment scan)

The memory requirement for segment decompression (64kb * no of col * concurrency) is very small. At any given point in time, one core can scan only one segment. if you consider this way assume in ur query u r aggregating say 10 columns and you have 32 core machines, then this memory is 32 * 10 * 64 ~ 20mb. I often ignore this unless if we have a special scenario where queries are significantly accessing a large no of columns. Even in the document, this is not considered during memory calculation.

Another important aspect while sizing the RAM is the availability of memory for the Operating System. Druid makes use of the memory mapping technique to read data from the segment. The process of loading the segment is offloaded to OS and OS does its best job loading the data in an efficient and faster manner. Often it’s a good practice to keep 10 to 15 % of actively queried segment size, free for the Operating System. If you have high concurrency keep more free memory for OS, this will help the segments to reside in memory and gain better performance.

Memory mapping also takes some heap but I am not certain at this point how much memory it will take. However, this value is really really tiny that I often ignore.

Summary -
Total memory required = Heap + Direct memory + Free memory for OS

1 Like