Realtime Node uses more memory than Xmx

Hi everybody,

I run my realtime node like this:

java -XX:+UseG1GC -Xmn17812239k -Xloggc:/var/log/druid/realtime/gc.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -server -Xmx45g -Xms45g -Djava.io.tmpdir=/opt/rb/var/druid/realtime -XX:OnOutOfMemoryError=/opt/rb/var/sv/druid_realtime/kill -Dlog4j.configuration=file:/tmp/druid/realtime/log4j.properties -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djute.maxbuffer=8388608 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDetails -Djava.rmi.server.hostname=192.168.11.108 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9082 -cp /opt/rb/var/druid/app/druid-services.jar:/opt/rb/var/druid/app/postgresql.jar:/tmp/druid/realtime io.druid.cli.Main server realtime

``

I limited the Xmx and Xms to 45g, but I can see how the realtime process consumes more memory than 45g

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

6399 druid 20 0 89.8g 57g 4.9g S 4.6 91.7 2549:41 java

I have seen that realtime use offHeap memory too, but you can set the offHeap memory size using this property: druid.processing.buffer.sizeBytes , and default value is a 1g. I think that my realtime should use 45g Heap + 1g OffHeap = 46g, but at the time I can see how it uses 57g.

Someone has any idea about why my realtime consumes more than 46g???

Thanks,

Andres Gomez

Realtime nodes use offheap memory for two things, processing (limited by that setting you mentioned) and spilled incremental indexes (not strictly limited, but they are limited in practice by the max throughput of your realtime node). The spilled incremental indexes are disk-backed memory mapped files. You can expect them to take roughly 2–3x the size of the final segments that the realtime node generates.

Regards Gian,

Andres Gomez

Andres, the processing buffer is used for intermediate results and computations. The size is also for every processing thread. Think of the intermediate chunks that get created as part of real-time processing mini-segments. On a historical node, the memory that is not used for heap, intermediate processing, and jvm overhead is used to page segments into memory. The same concept applies for real-time nodes, although real-time processing does not have a concept of a server.maxSize.

You can think of the memory used by a realtime node as roughly:

  • On heap is mostly the incremental index. This is going to be (maxRowsInMemory * (maxPendingPersists + 1)) * (factor based on your row size)

  • Off heap processing buffers will be druid.processing.buffer.sizeBytes * druid.processing.numThreads

  • Off heap spilled indexes should be roughly 2–3x of the size of the final segment generated by your realtime nodes, which in turn is based on how much data you are feeding in to each node. We usually suggest targeting 300MB–800MB segments per realtime worker.

We have our current production config posted on: http://druid.io/docs/latest/Production-Cluster-Configuration.html

It’s basically 3g heap, 512MB processing buffers, 2 processing threads, 75000 – 150000 maxRowsInMemory, 0 maxPendingPersists. With that we usually expect to need 6–7GB of ram per peon.

yeah… but I remember that the standalone realtime nodes had a bug when you use N shadSpecs on the same node, when the realtime tries merge the intermediate-segments to build final segment.

Thanks for roughly memory used by a realtime, I try to calculate my memory used and check with my limit! :slight_smile:

Regards,

Andres

Hi FJ,

ohh ok!! Then the realtime is a total memory of:

  • 45g Xmx

  • 1g default value (druid.processing.buffer.sizeBytes) * 7 (druid.processing.numThreads) because my computer has 8 cores = 7g

Total memory ~ 52g, more or less???

Regards,

Andrés

Hi Andres, ~52g will be used for query processing (this does not include the memory required for segments). ~1G or so will probably be JVM overhead. I think it is important to understand the distinction between the memory used for processing versus the memory used for segments. You may want to read: http://druid.io/docs/0.7.0/Performance-FAQ.html