When I do the following config changes in druid Historicals service, what I expect is to ingest data of 60 million rows but instead it is throwing java.lang.OutOfMemoryError: Java heap space.
- -XX:MaxDirectMemorySize=10240g
- Step 2druid.processing.buffer.sizeBytes=500MiB
- druid.processing.numMergeBuffers=8
Things I've tried
- Tried with the above set of configuration changes in the druid broker service and the task will remain in the state of PENDING infinitely
Logs
**2022-08-10T12:46:13,651 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in DETERMINE_PARTITIONS.** **java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space** ** at org.apache.druid.data.input.impl.InputEntityIteratingReader.lambda$read$0(InputEntityIteratingReader.java:81) ~[druid-core-0.22.1.jar:0.22.1]** ** at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:84) ~[druid-core-0.22.1.jar:0.22.1]** ** at org.apache.druid.java.util.common.parsers.CloseableIterator$2.(CloseableIterator.java:69) ~[druid-core-0.22.1.jar:0.22.1]** ** at org.apache.druid.java.util.common.parsers.CloseableIterator.flatMap(CloseableIterator.java:67) ~[druid-core-0.22.1.jar:0.22.1]** ** at org.apache.druid.data.input.impl.InputEntityIteratingReader.createIterator(InputEntityIteratingReader.java:103) ~[druid-core-0.22.1.jar:0.22.1]** ** at org.apache.druid.data.input.impl.InputEntityIteratingReader.read(InputEntityIteratingReader.java:74) ~[druid-core-0.22.1.jar:0.22.1]** ** at org.apache.druid.segment.transform.TransformingInputSourceReader.read(TransformingInputSourceReader.java:43) ~[druid-processing-0.22.1.jar:0.22.1]**Seeking an expertise here as this issue has become a blocker. Kindly note that we are able to ingest data of 20 million rows with the above configuration without any hurdles
What more configuration to be included to be able to process 60 million rows of data
Thanks,
Keerthi Kumar N