How does Druid manage Garbage Collection? a quick overview

Druid uses 3 primary memory layers. On-Heap (JVM, joins, lookups), Off-heap (merges, etc) and mmap (segment cache).

Druid uses G1GC for the JVM collector. In the log files, you’ll see things like “Allocation Failed” and “Full GC” collection operations. Allocation failures are concurrent collections (they don’t pause the JVM) are normal and expected. They should be very very short and could be very frequent. FULL GC on the other hand are stop-the-world/JVM collections and can have a significant impact on performance.

Garbage collections can be a problem in any system if configured wrong. GC tuning is definitely a dark art.

1 Like

Do you know if there are any metrics emitted that allow you to measure the GC collections?