Historical memory configuration and caching

Hi Druid team,

I have been trying to optimize historical nodes configuration, checked out lots of docs / group posts, but still haven’t found the answers to all the questions that I have. Could you please help?

The questions are the following:

  1. Druid uses memory-mapped files technique to map the segments into memory on historical nodes. The question is, are those segments mapped outside of allocated to the java process memory, or inside it? I know the formula is:

memory_for_segments = total_memory - heap_size - (processing.buffer.sizeBytes * (processing.numThreads+1)) - JVM overhead (~1G).

Does total_memory here mean all available RAM on the machine? If yes, that means that Druid maps segments outside of the java process, which means we need to find a tradeoff between (directmemory + heap) and free memory.

  1. What is a healthy memory for segments / node maxSize ratio (memory / disk ratio) for a historical node?

  2. As for caching, if we use local cache, what is the feasible cache size per node, and where is this cache physically stored - in heap or direct memory? If it’s in direct memory, then the direct memory setting should be increased by the size of cache, shouldn’t it?

Kind regards,

Oleg

Answers Inline. Hopefully they help a bit.

Hi Druid team,

I have been trying to optimize historical nodes configuration, checked out lots of docs / group posts, but still haven’t found the answers to all the questions that I have. Could you please help?

The questions are the following:

  1. Druid uses memory-mapped files technique to map the segments into memory on historical nodes. The question is, are those segments mapped outside of allocated to the java process memory, or inside it? I know the formula is:

memory_for_segments = total_memory - heap_size - (processing.buffer.sizeBytes * (processing.numThreads+1)) - JVM overhead (~1G).

Does total_memory here mean all available RAM on the machine? If yes, that means that Druid maps segments outside of the java process, which means we need to find a tradeoff between (directmemory + heap) and free memory.

Technically total memory is the memory available to the cgroup (if running on linux) which for default cases is going to be all the system memory. In general you are going to have JVM Heap + JVM Direct memory + Page Cache (mmap’d segments) filling up your allowed memory space.

  1. What is a healthy memory for segments / node maxSize ratio (memory / disk ratio) for a historical node?

memory / disk / cpu balance completely depends on what kind of data you are hosting and what kind of queries you do.

Example: if you have a bunch of complex queries (hyper-uniques) it is going to drastically change your access patterns (probably need more cpu) compared to simply doing ‘count’ queries with dimension filters.

  1. As for caching, if we use local cache, what is the feasible cache size per node, and where is this cache physically stored - in heap or direct memory? If it’s in direct memory, then the direct memory setting should be increased by the size of cache, shouldn’t it?

Local cache counts against JVM heap memory. And jvm memory tuning is another one of those areas that varies quite a bit based on your workload. If you find people issue the same queries over and over, giving more memory to cache might be good. If you find people asking unique queries, then a smaller cache with more room for processing power will probably work better.

Thank you very much! Now it’s clear. Probably it makes sense to state the part about total memory and page cache in docs too?

Thanks again!

Hi Oleg, this is a great opportunity to contribute to the Druid docs :slight_smile:

https://github.com/druid-io/druid/tree/master/docs/content