java.lang.OutOfMemoryError: Java heap space error while submitting a large dataset for indexing

Apache Druid Q&A Google Groups
#1

Hi All,

I have druid installed in my single node EC2 instance . It is M5d12x large AWS EC2 instance.

I have ingested around 0.521 GB of data to my historical node so far each of like around ~55MB of files .

But now my use case is to ingest the data of 665 GB in to it , which has ~ 73 million records in it, with around 400 columns(dimensions in it )

**I have tried following things so far :- **

  • in my index task I have set below things , but still no luck :-

“targetPartitionSize” : 0,

“maxRowsInMemory” : 0,

“maxCacheCapacityBytes” : 0,

“maxFetchCapacityBytes” : 0,

“prefetchTriggerBytes” : 0,

  • I have increased the default heap size of historical and other services from 256m to 700g , but still no luck.
    -Xms256m

-Xmx700g

  • This is the current jvm.config of my historical node :
    -server

-Xms256m

-Xmx700g

-XX:MaxDirectMemorySize=700g

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=var/heapdump.bin

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=var/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

  • and the results of the command Jmap – heap PID_of_historcial_node

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/dev/shm/temp/java

Java HotSpot™ 64-Bit Server VM warning: Insufficient space for shared memory file:

24203

Try using the -Djava.io.tmpdir= option to select an alternate temp location.

Attaching to process ID 21802, please wait…

Debugger attached successfully.

Server compiler detected.

JVM version is 25.191-b12

using thread-local object allocation.

Parallel GC with 33 thread(s)

Heap Configuration:

MinHeapFreeRatio = 0

MaxHeapFreeRatio = 100

MaxHeapSize = 751619276800 (716800.0MB)

NewSize = 89128960 (85.0MB)

MaxNewSize = 250539409408 (238933.0MB)

OldSize = 179306496 (171.0MB)

NewRatio = 2

SurvivorRatio = 8

MetaspaceSize = 21807104 (20.796875MB)

CompressedClassSpaceSize = 1073741824 (1024.0MB)

MaxMetaspaceSize = 17592186044415 MB

G1HeapRegionSize = 0 (0.0MB)

Heap Usage:

PS Young Generation

Eden Space:

capacity = 224919552 (214.5MB)

used = 70385832 (67.12516021728516MB)

free = 154533720 (147.37483978271484MB)

31.293780987079327% used

From Space:

capacity = 14680064 (14.0MB)

used = 14655016 (13.976112365722656MB)

free = 25048 (0.02388763427734375MB)

99.82937404087612% used

To Space:

capacity = 23592960 (22.5MB)

used = 0 (0.0MB)

free = 23592960 (22.5MB)

0.0% used

PS Old Generation

capacity = 179306496 (171.0MB)

used = 35727064 (34.071983337402344MB)

free = 143579432 (136.92801666259766MB)

19.925136454621253% used

15365 interned Strings occupying 1636512 bytes.

Can someone give some tips how to achieve this ? and if the historical node is the issue here or not ! is there anything else I can try out ?

Thanks,

Anoosha

#2

Hi Anusha,

  1. Have you been trying to ingest it via Kafka Indexing Service?

  2. Please attach the logs that you are getting while ingesting the data.

  3. Also, please attach the runtime properties & jvm configuration of both historical & middle manager.

Looking it at first, seems like some issue with JVM configuration. Please share the above details & accordingly we’ll able to help you better.

Thanks!

#3

Any reason you are running all services on a single node? That aside, check the jvm and runtime properties of broker, coordinator and middlemanager. Assume you have sufficient memory to host all the services (tweaking #of threads and etc)

#4

From the Jmap output, the location for shared memory file /dev/shm/temp/java is running out of space. You might want to either increase it or move it to a location with bigger size.

Rommel Garcia