Will the historical node load all the segments on its disk cache into memory

Hi,
I want to know whether the historical node will load all the segments on its disk into memory, and always maintain them into memory until it unannounce them.

The historical node’s memory capacity is smaller than its segments’ disk capacity, will it be not enough capacity to load the segment into memory and cause “out of memory” exception?

Historical nodes will memory-map all segments that are currently announced, although they won’t necessarily be resident in memory at all times. If you don’t have enough memory for that, then the OS will page them in and out as needed. So, it’s totally fine to have more segments loaded than you have memory.

Thanks,Gian. Is it for “mapDB” implementing the page swap in and out?
I have followed this doc(http://druid.io/docs/0.8.0/configuration/production-cluster.html) to chose ec2 type and JVM setting for historical node , do you some advices of JVM setting for supporting 460GB disk capacity for historical segment storage.

Could you help to give advice for JVM setting when historical node have

在 2015年7月20日星期一 UTC+8上午1:34:53,Gian Merlino写道:

The code for interacting with segments is actually unique to Druid. There is some MapDB stuff in Druid, but it’s for temporary files used during indexing.

If you’re using aws, the configs listed on that doc should be a good starting point; they suggest r3.8xlarge for historical nodes, which have 640GB of disk and 244GB of ram. That instance type could be run entirely in-memory (if you set maxSize < 200GB or so) or could be run partially in memory (if you set maxSize larger than that).