Historical cannot load segments

Hi,

I restarted the server where my druid instance is running today and i faced an issue.
On startup, the Historical process failed to load all my segments.
It’s failed with error :

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2097152 bytes for AllocateHeap
# Possible reasons:
#   The system is out of physical RAM or swap space
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.cpp:46), pid=177934, tid=178238
#
# JRE version: OpenJDK Runtime Environment Corretto-11.0.14.10.1 (11.0.14.1+10) (build 11.0.14.1+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.14.10.1 (11.0.14.1+10-LTS, mixed mode, tiered, g1 gc, linux-amd64)

When running, we can clearly see the Historical java process memory rising up to 50% but not fulling it. The swap is somehow is not rising, but the virtual memory is exploding up to 8To before the crash.

For the record, the server who run this as 800Go of RAM and 100Go of swap.
The Druid version is ‘apache-druid-25.0.0’.
I have 400247 segments for about 13.4To

# du -sh /data-druid-1/* /data-druid-2/* /data-druid-3/*
4.3T    /data-druid-1/druid
33G     /data-druid-1/swapfile
4.5T    /data-druid-2/druid
33G     /data-druid-2/swapfile
4.5T    /data-druid-3/druid
33G     /data-druid-3/swapfile

The Historical jvm config is :

historical# cat jvm.config 
-server
-Xms64g
-Xmx64g
-XX:MaxDirectMemorySize=64g
-XX:+ExitOnOutOfMemoryError
-XX:+UseG1GC
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

I tried to every possible solutions proposed in the error

The ulimits on my server is ulimited and i increased the nofile.

# cat /etc/security/limits.conf
root             soft nofile 5000000
root             hard nofile 5000000

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3090403
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3090403
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

For the sysctl, i increased the max_map_count as seen in an early post.

# sysctl -a
...
vm.max_map_count = 262144
...

I came to my limits, do someone have an idea of what could i do.
Thanks in advance for your help. If you need any other information tell me.

Hi @Theo_Eymard ,
Is this a single server deployment? If so, what are the memory settings for other processes on it? Could the overall memory of the server be exhausted?

Could you also paste more of the historical log prior to the failure and take a look at the coordinator log, to see if there are any issues there.

While I don’t know if this is adding to the issue, at 13.4 TB with 400247 segments, your average segment size is about 34 MB, you’ll probably benefit from some compaction to achieve better segment size optimization. You should target 10x to 20x that size for segments.

can you do a free -g on your machine before starting druid? Are you using control groups in linux? The error basically means that the JVM is not able to allocate heap. If you have 800GB RAM (I got that correct?) then this should not happen unless something like a control group is limiting the available RAM for druid

Hello,

@Sergio_Ferragut, Yes it is a single server deployment,but when i launch druid, the overall memory is not exhausted.
@Vijay_Narayanan1 We dont use cgroups, but docker is running on the server.
Here is the free -g.

# free -g

              total        used        free      shared  buff/cache   available
Mem:            754           4         167           0         582         745
Swap:           103           0         103

Yesterday, I ran netdata while trying to start druid.
Here is a screenshot :

We can see my multiple tries, with the java process rising the memory up to 40Gb.
But i don’t understand why the virtual memory rise up to 8Gb while the real memory isn’t full.

Here is more logs :

# ./bin/start-micro-quickstart 
[Thu Jan 12 10:57:35 2023] Starting Apache Druid.
[Thu Jan 12 10:57:35 2023] Open http://localhost:8888/ or http://logs-analytics2:8888/ in your browser to access the web console.
[Thu Jan 12 10:57:35 2023] Or, if you have enabled TLS, use https on port 9088.
[Thu Jan 12 10:57:35 2023] Starting services with log directory [/opt/baremetal/apache-druid-25.0.0/log].
[Thu Jan 12 10:57:35 2023] Running command[coordinator-overlord]: bin/run-druid coordinator-overlord conf/druid/single-server/micro-quickstart
[Thu Jan 12 10:57:35 2023] Running command[zk]: bin/run-zk conf
[Thu Jan 12 10:57:35 2023] Running command[router]: bin/run-druid router conf/druid/single-server/micro-quickstart
[Thu Jan 12 10:57:35 2023] Running command[middleManager]: bin/run-druid middleManager conf/druid/single-server/micro-quickstart
[Thu Jan 12 10:57:35 2023] Running command[broker]: bin/run-druid broker conf/druid/single-server/micro-quickstart
[Thu Jan 12 10:57:35 2023] Running command[historical]: bin/run-druid historical conf/druid/single-server/micro-quickstart
[Thu Jan 12 10:59:03 2023] Command[historical] exited (pid = 485312, exited = 1)
[Thu Jan 12 10:59:03 2023] Command[historical] failed, see its logfile for more details
# less historical.log
2023-01-12T10:55:58,550 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final
...
2023-01-12T10:56:02,187 WARN [main] org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is inefficient for short content, break even is size 23
2023-01-12T10:56:02,310 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[209,715,200]
2023-01-12T10:56:02,421 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[209,715,200]
2023-01-12T10:56:02,585 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[209,715,200]
2023-01-12T10:56:02,691 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[209,715,200]
2023-01-12T10:56:02,878 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
...
2023-01-12T10:56:03,048 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-01-12T10:56:03,048 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=var/tmp
2023-01-12T10:56:03,048 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2023-01-12T10:56:03,048 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2023-01-12T10:56:03,048 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2023-01-12T10:56:03,049 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.4.0-136-generic
2023-01-12T10:56:03,049 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=user
2023-01-12T10:56:03,049 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/user
2023-01-12T10:56:03,049 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/baremetal/apache-druid-25.0.0
2023-01-12T10:56:03,049 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=64585MB
2023-01-12T10:56:03,049 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=65536MB
2023-01-12T10:56:03,049 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=65536MB
...
2023-01-12T10:59:14,658 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment cache file [30024/400247][/data-druid-2/druid/var/druid/segment-cache/info_dir/datasource_2022-03-26T21:00:00.000Z_2022-03-26T22:00:00.000Z_2022-03-26T21:00:56.450Z_3].
2023-01-12T10:59:14,659 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment cache file [30025/400247][/data-druid-2/druid/var/druid/segment-cache/info_dir/datasource_2022_2022-01-17T21:00:00.000Z_2022-01-17T22:00:00.000Z_2022-01-17T21:00:29.177Z_4].
2023-01-12T10:59:14,659 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment cache file [30026/400247][/data-druid-2/druid/var/druid/segment-cache/info_dir/datasource_2022_2022-03-21T20:00:00.000Z_2022-03-21T21:00:00.000Z_2022-03-21T20:01:06.412Z_5].

# less historical.stdout.log

Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml                                                                                   
WARNING: An illegal reflective access operation has occurred                                                                                                                                                       
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.Str
ing,byte[],int,int,java.security.ProtectionDomain)                                                                                                                                                                 
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1                                                                                                
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                                                              
WARNING: All illegal access operations will be denied in a future release                                                                                                                                          
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml                                                                                   
WARNING: An illegal reflective access operation has occurred                                                                                                                                                       
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.Str
ing,byte[],int,int,java.security.ProtectionDomain)                                                                                                                                                                 
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1                                                                                                
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                                                              
WARNING: All illegal access operations will be denied in a future release                                                                                                                                          
#                                                                                                                                                                                                                  
# There is insufficient memory for the Java Runtime Environment to continue.                                                                                                                                       
# Native memory allocation (malloc) failed to allocate 2097152 bytes for AllocateHeap                                                                                                                              
# An error report file with more information is saved as:                                                                                                                                                          
# /opt/baremetal/apache-druid-25.0.0/hs_err_pid177934.log                                                                                                                                                          
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml                                                                                   
WARNING: An illegal reflective access operation has occurred                                                                                                                                                       
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.Str
ing,byte[],int,int,java.security.ProtectionDomain)                                                                                                                                                                 
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1                                                                                                
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                                                              
WARNING: All illegal access operations will be denied in a future release                                                                                                                                          
#                                                                                                                                                                                                                  
# There is insufficient memory for the Java Runtime Environment to continue.                                                                                                                                       
# Native memory allocation (malloc) failed to allocate 1048576 bytes for AllocateHeap                                                                                                                              
# An error report file with more information is saved as:                                                                                                                                                          
# /opt/baremetal/apache-druid-25.0.0/hs_err_pid184973.log                                                                                                                                                          
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml                                                                                   
WARNING: An illegal reflective access operation has occurred                                                                                                                                                       
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.Str
ing,byte[],int,int,java.security.ProtectionDomain)                                                                                                                                                                 
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1                                                                                                
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                                                              
WARNING: All illegal access operations will be denied in a future release                                                                                                                                          
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml                                                                                   
WARNING: An illegal reflective access operation has occurred                                                                                                                                                       
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.Str
ing,byte[],int,int,java.security.ProtectionDomain)                                                                                                                                                                 
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1                                                                                                
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                                                              
WARNING: All illegal access operations will be denied in a future release                                                                                                                                          
#                                                                                                                                                                                                                  
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for AllocateHeap
# An error report file with more information is saved as:
# /opt/baremetal/apache-druid-25.0.0/hs_err_pid191941.log
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2097152 bytes for AllocateHeap
# An error report file with more information is saved as:
# /opt/baremetal/apache-druid-25.0.0/hs_err_pid193769.log
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2097152 bytes for AllocateHeap
# An error report file with more information is saved as:
# /opt/baremetal/apache-druid-25.0.0/hs_err_pid483394.log
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1048576 bytes for AllocateHeap
# An error report file with more information is saved as:
# /opt/baremetal/apache-druid-25.0.0/hs_err_pid485312.log
Running [historical], logging to [/opt/baremetal/apache-druid-25.0.0/bin/../log/historical.log] if no changes made to log4j2.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/baremetal/apache-druid-25.0.0/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
# less /opt/baremetal/apache-druid-25.0.0/hs_err_pid485312.log

---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
uname:Linux 5.4.0-136-generic #153-Ubuntu SMP Thu Nov 24 15:56:58 UTC 2022 x86_64
OS uptime: 1 days 0:31 hours
libc:glibc 2.31 NPTL 2.31 
rlimit (soft/hard): STACK 8192k/infinity , CORE 0k/infinity , NPROC 3090403/3090403 , NOFILE 1048576/1048576 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , ME
MLOCK 65536k/65536k
load average:8.72 4.69 2.54

/proc/meminfo:
MemTotal:       791233612 kB
MemFree:        130857236 kB
MemAvailable:   733427176 kB
Buffers:        11542884 kB
Cached:         582135476 kB
SwapCached:         5432 kB
Active:         487025124 kB
Inactive:       149825016 kB
Active(anon):   42850468 kB
Inactive(anon):   316932 kB
Active(file):   444174656 kB
Inactive(file): 149508084 kB
Unevictable:       18472 kB
Mlocked:           18472 kB
SwapTotal:      109051888 kB
SwapFree:       108999408 kB
Dirty:             92740 kB
Writeback:             0 kB
AnonPages:      43188528 kB
Mapped:         302055884 kB
Shmem:              5596 kB
KReclaimable:   13873844 kB
Slab:           14626972 kB
SReclaimable:   13873844 kB
SUnreclaim:       753128 kB
KernelStack:       35584 kB
PageTables:      5578104 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    503431700 kB
Committed_AS:   85977148 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      495028 kB
VmallocChunk:          0 kB
Percpu:            52416 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:    1208
HugePages_Free:     1208
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2473984 kB
DirectMap4k:     8468168 kB
DirectMap2M:    500785152 kB
DirectMap1G:    296747008 kB

/sys/kernel/mm/transparent_hugepage/enabled:
always [madvise] never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):
always defer defer+madvise [madvise] never

Process Memory:
Virtual Size: 9264841928K (peak: 9264841928K)
Resident Set Size: 341875060K (peak: 341875060K) (anon: 40023600K, file: 301851460K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 160832K (may have wrapped)

/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
6180806
/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
262144
/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
4194304

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-39
cpu_memory_nodes: 0-1
active_processor_count: 40
cpu_quota: -1
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -2
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 477539278848
memory_max_usage_in_bytes: 773000908800

Steal ticks since vm start: 0
Steal ticks percentage since vm start:  0.000

CPU:total 40 (initial active 40) (10 cores per cpu, 2 threads per core) family 6 model 85 stepping 7 microcode 0x5003302, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx, evex, fma
CPU Model and flags from /proc/cpuinfo:
model name      : Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Online cpus:
0-39
Offline cpus:

BIOS frequency limitation:
<Not Available>
Frequency switch latency (ns):
0
Available cpu frequencies:
<Not Available>
Current governor:
powersave
Core performance/turbo boost:
<Not Available>

Memory: 4k page, physical 791233612k(130856732k free), swap 109051888k(108999408k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.14.1+10-LTS) for linux-amd64 JRE (11.0.14.1+10-LTS), built on Feb  7 2022 23:51:33 by "jenkins" with gcc 7.3.1 20180303 (Red Hat 7.3.1-5)

Hope this informations is enough, if you need more tell me.

How are you starting the docker container? It seems to me that the amount of RAM assigned to the container is not enough

Is the free -g output above from within the container?

Druid is not running in a container, but docker is install and run other containers .
The free -g has been run in the server directly.

for 400k segments why have you set vm.max_map_count=262144? You will need as many memory maps as the segments. Can you set vm.max_map_count = 1000000 and try?

You where right thanks a lot @Vijay_Narayanan1.
I tried it yesterday, but i didn’t increase enough or i didn’t refresh sysctl i don’t know.

However, thanks for your advice @Sergio_Ferragut and @Vijay_Narayanan1.

Were a still new to druid, do you have some advice for segment compaction ?

Thanks for your help.

cool. Since you are on 25.0 you should consider multi dimensional range partitioning on the most common grouping keys in your queries. You can ingest with dynamic in the realtime task and use autocompaction to compact into range.