Confusion doc

According to the doc there are 2 things

  • memory_for_segments = total_memory - heap - direct_memory - jvm_overhead

  • all segments are memory mapped before they can be queried

And there is the JVM configuration to limit the off heap usage using --xx:MaxDirectMemorySize

Let us say that I

*132G total_memory

  • 8G for historical

  • 20G for direct memory

  • 8G for jam overhead

Does this imply that the JVM will have 96G for memory_for_segments and what that even mean ? My off heap usage has been limited by --xx:MaxDirectMemorySize. How does druid use the 96 GB ( OS Cache ? )

My query is, whether segments are stored in the off heap governed by max direct_memory in any way or just memory mapped ? Is it being treated as a cache for most recent segments ?

Segments are just memory mapped in the OS cache and not governed by max direct memory setting.
direct memory is used to allocate bytebuffers used for query processing.

Thanks. That makes sense.

Would there thus make any sense for us to give more than ( processing.buffer.sizeBytes * (processing.numThreads+1)) - JVM overhead (~1G) to direct memory, apart from the frequency with which a GC happens on it ( with the 100ms delay ). It seems more reasonable to allow for more memory for segments than for direct memory to have a higher memory/max_size ratio and limit io.

Thank yo

Giving as much memory to the memory mapping of segments makes a lot of sense, if, as you said, it would improve the memory/max_size ratio closer to 1:1.
I did not understand what you wrote about GC. The garbage collector is not responsible for direct memory, only for heap memory.

The question whether it makes sense to give more than the prescribed size to direct memory is a good one. I think originally, direct memory was only used for processing buffers but new bits and pieces get added to Druid constantly. I believe that depending on which features you use, further space is required in direct mem. For instance, if you use lookups along with offheap caching, this goes into direct mem or if you use the groupby engine v2, then further space in direct mem is needed for the merge buffers. But I never know whether there are further internal structures in Druid that make use of direct mem.

I don’t know of a good solid way to measure whether the direct mem setting chosen is good.
For the size of the processing buffer, I believe if you pick a value that is too low then Druid needs to make several sequential passes over the data, so you would see higher segment scan times if the buffer size was too small.