JVM VIRT size of druid historical node growns with segment loading, dynamic libraries of JVM includes .smooth files?

Hello,

I have an OOM (Heap space) problem when starting historical node Running on a docker container using OpenJDK:8 image

The historical node starts loading data from deep storage, I have set enough RAM to allow the heap, I have also tuned many other parameters.

But the system shows a memory behaviour that i´m not sure if I am missunderstanding:
I´m using a testing datasource in a simple configuration, datasource contains aprox 50Gb and replication factor is 1.
When I execute top I find there are 12Gb + 50 Gb (62.071g) in VIRT memory. I didnt expected this and this size growns linear with my data size .
VIRT size increases linear with segments loaded size while the system is loading segments from deep storage, until the system crashes.
My Swap memory is disabled and Heap size of JVM is much more lower than that, RAM is also lower 4Gb
TOP shows the following info (see VIRT):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28 root 20 0 62.071g 3.277g 846708 S 357.0 1.3 14:25.56 java

I have generated the JVM dump and I see that the dynamic libraries section contains all my disk files, not only libraries, including my druid segments (.smoosh files).

The question is:

the JVM is loading as library all data (.smoosh file) and this is the size that the JVM is showing in VIRT?

this could be the reason of the OOM crash because of some library data (i.e. file path) is in heap???

Is it normal that dynamic library dump contains all the disk content? could somebody produce a heap OOM and see if its dump contains that?

heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit

Narrow klass base: 0x0000000000000000, Narrow klass shift: 3

Compressed class space size: 1073741824 Address: 0x0000000100000000

Heap:

par new generation total 235968K, used 222996K [0x00000000c0000000, 0x00000000d0000000, 0x00000000d0000000)

eden space 209792K, 96% used [0x00000000c0000000, 0x00000000cc678190, 0x00000000ccce0000)

from space 26176K, 75% used [0x00000000ccce0000, 0x00000000ce02cf90, 0x00000000ce670000)

to space 26176K, 0% used [0x00000000ce670000, 0x00000000ce670000, 0x00000000d0000000)

concurrent mark-sweep generation total 786432K, used 786429K [0x00000000d0000000, 0x0000000100000000, 0x0000000100000000)

Metaspace used 63001K, capacity 64096K, committed 64572K, reserved 1105920K

class space used 8150K, capacity 8419K, committed 8544K, reserved 1048576K

Dynamic libraries:

00400000-00401000 r-xp 00000000 08:49 1056714 /usr/local/openjdk-8/bin/java

00600000-00601000 r–p 00000000 08:49 1056714 /usr/local/openjdk-8/bin/java

00601000-00602000 rw-p 00001000 08:49 1056714 /usr/local/openjdk-8/bin/java

016b0000-016d1000 rw-p 00000000 00:00 0 [heap]

c0000000-100858000 rw-p 00000000 00:00 0

100858000-140000000 —p 00000000 00:00 0

thousand of segment files and anything in disk is shown in dynamic libraries section!!! :

7fbd79573000-7fbd79810000 r–s 00000000 08:11 169346167 /mnt/mesos/sandbox/segmentcache_0_16_0/xxxx_2/2019-09-03T00:00:00.000Z_2019-09-04T00:00:00.000Z/2019-10-31T20:42:09.922Z/65/00000.smoosh

7fbd79810000-7fbd79ab3000 r–s 00000000 08:11 169346162 /mnt/mesos/sandbox/segmentcache_0_16_0/xxxx_2/2019-09-03T00:00:00.000Z_2019-09-04T00:00:00.000Z/2019-10-31T20:42:09.922Z/66/00000.smoosh

7fbd79ab3000-7fbd79ac8000 r–s 00000000 08:11 169346157 /mnt/mesos/sandbox/segmentcache_0_16_0/xxxx_2/2019-09-03T00:00:00.000Z_2019-09-04T00:00:00.000Z/2019-10-31T20:42:09.922Z/63/00000.smoosh

7fbd79ac8000-7fbd79add000 r–s 00000000 08:11 169346154 /mnt/mesos/sandbox/segmentcache_0_16_0/xxxx_2/2019-09-03T00:00:00.000Z_2019-09-04T00:00:00.000Z/2019-10-31T20:42:09.922Z/64/00000.smoosh

7fbd79add000-7fbd79bb4000 r–s 00000000 08:11 169346147 /mnt/mesos/sandbox/segmentcache_0_16_0/xxxx_2/2019-09-03T00:00:00.000Z_2019-09-04T00:00:00.000Z/2019-10-31T20:42:09.922Z/61/00000.smoosh

also .so and .jar libraries are shown.

7fd07016f000-7fd070199000 r-xp 00000000 08:49 1057149 /usr/local/openjdk-8/jre/lib/amd64/libjava.so

7fd070199000-7fd070398000 —p 0002a000 08:49 1057149 /usr/local/openjdk-8/jre/lib/amd64/libjava.so

7fd070398000-7fd070399000 r–p 00029000 08:49 1057149 /usr/local/openjdk-8/jre/lib/amd64/libjava.so

7fd070399000-7fd07039b000 rw-p 0002a000 08:49 1057149 /usr/local/openjdk-8/jre/lib/amd64/libjava.so

7fd07039b000-7fd0703a9000 r-xp 00000000 08:49 1057169 /usr/local/openjdk-8/jre/lib/amd64/libverify.so

7fd0703a9000-7fd0705a8000 —p 0000e000 08:49 1057169 /usr/local/openjdk-8/jre/lib/amd64/libverify.so

7fd0705a8000-7fd0705aa000 r–p 0000d000 08:49 1057169 /usr/local/openjdk-8/jre/lib/amd64/libverify.so

7fd0705aa000-7fd0705ab000 rw-p 0000f000 08:49 1057169 /usr/local/openjdk-8/jre/lib/amd64/libverify.so

Long story short:

Virtual address space is indeed used for all segments.

Physical pages are handled through the kernel’s tunings.

JVM controls the Heap and Direct memory buffer sizes. You can limit these at startup with command line options.

See https://stackoverflow.com/a/14347298/665049 for some more info on the differences.

Kernel OOM killing shouldn’t be coming from virtual address space unless you have some really odd tunings.

OOME jvm exception could be coming from a number of places. You can try limiting some of the pool sizes (like maybe limit processing threads to 1). Also different queries cause differing heap pressure, so if you are using the newer groupBy query implementation or the older one can make a difference, and topN will have an even different heap signature.

check the following sysctl:
vm.max_map_count = 1280000\

relative to the number of segmants you have

Not being able to mmap() a file LOOKS like an OOM to the JVM

I had this problem a month or so ago

ALSO, make sure you have compaction turned ON

I went from 65000+ segments to 7000ish

Thanks Charles, there is no querys running, it is not caused by groupby or topn querys, buffers etc…

My heap size, total memory, direct memory, are adjusted, I have set bigger values to test it and always get the same result.

It is only loading segments, and the only strange thing I have seen is all my disk data into the dynamic library section of the core dump.

All core dumps I have seen before have been always only libraries dll,.so. jars… Never have seen data files there.

So it is normal that virt size show in top command is as big as your loaded segments? Can confirm that?

Many thanks

Thanks Larry, I will try this.

So it is normal that virt size show in top command is as big as your loaded segments? Can confirm that?

Yes that is normal. All the segments are memory mapped, which allocates virtual address space.

Larry’s observations are very good about some oddities of the JVM.

Thanks!