Druid Java heap space error

Hi,

I have had a couple of successful attempts in ingesting our data into Druid. When I tried to ingest a new set of data into a new data source I started getting out of memory error.

Need some help in figuring out the cause of this error.

A few differences in the new data set compared with my previous successful attempts are:

  • Increasing the data file size to be ~1.5 GB from 20 MB.

  • Including list of bigint element (i.e. one column is a list of bigint) in my data set.

Attached a small input data set to get the picture of how my input data looks like.

My task file looks like this:

{

“type”:“index”,

“spec”:{

  "dataSchema":{

     "dataSource":"BTGeo_Impression",

     "parser":{

        "type":"string",

        "parseSpec":{

           "format":"tsv",

           "timestampSpec":{

              "column":"timestamp",

              "format":"posix"

           },

           "columns":[

              "timestamp",

              "sspId",

              "adspaceId",

              "targetId",

              "regionId",

              "metroId",

              "referrerUrl",

              "pageUrl",

              "TotalImpression",

              "SoldImpression",

              "winningPrice"

           ],

           "delimiter":"\t",

           "dimensionsSpec":{

              "dimensions":[

                 "sspId",

                 "adspaceId",

                 "targetId",

                 "regionId",

                 "metroId",

                 "pageUrl"

              ],

              "dimensionExclusions":[

              ],

              "spatialDimensions":[

              ]

           }

        }

     },

     "metricsSpec":[

        {

           "type":"longSum",

           "name":"TotalImpressions",

           "fieldName":"TotalImpression"

        },

        {

           "type":"longSum",

           "name":"SoldImpressions",

           "fieldName":"SoldImpression"

        },

        {

           "type":"doubleSum",

           "name":"GrossRevenue",

           "fieldName":"winningPrice"

        }

     ],

     "granularitySpec":{

        "type":"uniform",

        "segmentGranularity":"DAY",

        "queryGranularity":"NONE",

        "intervals":[

           "2015-11-09/2015-11-10"

        ]

     }

  },

  "ioConfig":{

     "type":"index",

     "firehose":{

        "type":"local",

        "baseDir":"examples/indexing/",

        "filter":"test2.tsv"

     }

  },

  "tuningConfig":{

     "type":"index",

     "targetPartitionSize":0,

     "rowFlushBoundary":0

  }

}

}

Below is the error log.

2015-11-11T00:46:49,962 INFO [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Running task: index_BTGeoImpression_2015-11-11T00:43:53.105Z
2015-11-11T00:46:49,963 INFO [task-runner-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_BTGeoImpression_2015-11-11T00:43:53.105Z]: LockListAction{}
2015-11-11T00:46:49,978 INFO [task-runner-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_BTGeoImpression_2015-11-11T00:43:53.105Z] to overlord[http://localhost:8090/druid/indexer/v1/action]: LockListAction{}
2015-11-11T00:46:50,002 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [test2.tsv] in and beneath [/home/druid/druid-0.8.1/examples/indexing]
2015-11-11T00:46:50,018 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Found files: [/home/druid/druid-0.8.1/examples/indexing/test2.tsv]
2015-11-11T00:47:09,495 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Determining partitions for interval[2015-11-09T00:00:00.000Z/2015-11-10T00:00:00.000Z] with targetPartitionSize[5000000]
2015-11-11T00:47:09,529 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [test2.tsv] in and beneath [/home/druid/druid-0.8.1/examples/indexing]
2015-11-11T00:47:09,530 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Found files: [/home/druid/druid-0.8.1/examples/indexing/test2.tsv]
2015-11-11T00:47:32,709 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Estimated approximately [944,410.544130] rows of data.
2015-11-11T00:47:32,709 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Will require [1] shard(s).
2015-11-11T00:47:32,711 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [test2.tsv] in and beneath [/home/druid/druid-0.8.1/examples/indexing]
2015-11-11T00:47:32,711 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Found files: [/home/druid/druid-0.8.1/examples/indexing/test2.tsv]
2015-11-11T00:49:52,592 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while running task[IndexTask{id=index_BTGeoImpression_2015-11-11T00:43:53.105Z, type=index, dataSource=BTGeoImpression}]
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[?:1.7.0_80]
	at java.lang.String.<init>(String.java:203) ~[?:1.7.0_80]
	at java.lang.String.substring(String.java:1913) ~[?:1.7.0_80]
	at java.lang.String.subSequence(String.java:1946) ~[?:1.7.0_80]
	at com.google.common.base.Splitter$SplittingIterator.computeNext(Splitter.java:612) ~[guava-16.0.1.jar:?]
	at com.google.common.base.Splitter$SplittingIterator.computeNext(Splitter.java:522) ~[guava-16.0.1.jar:?]
	at com.google.common.base.AbstractIterator.tryToComputeNext(AbstractIterator.java:64) ~[guava-16.0.1.jar:?]
	at com.google.common.base.AbstractIterator.hasNext(AbstractIterator.java:59) ~[guava-16.0.1.jar:?]
	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43) ~[guava-16.0.1.jar:?]
	at com.metamx.common.collect.Utils.zipMapPartial(Utils.java:88) ~[java-util-0.27.0.jar:?]
	at com.metamx.common.parsers.DelimitedParser.parse(DelimitedParser.java:142) ~[java-util-0.27.0.jar:?]
	at io.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:86) ~[druid-api-0.3.9.jar:0.3.9]
	at io.druid.data.input.impl.StringInputRowParser.parse(StringInputRowParser.java:91) ~[druid-api-0.3.9.jar:0.3.9]
	at io.druid.data.input.impl.FileIteratingFirehose.nextRow(FileIteratingFirehose.java:54) ~[druid-api-0.3.9.jar:0.3.9]
	at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:364) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:205) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-indexing-service-0.8.1.jar:0.8.1]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-indexing-service-0.8.1.jar:0.8.1]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80]
2015-11-11T00:49:52,601 ERROR [main] io.druid.cli.CliPeon - Error when starting up.  Failing.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:162) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	at io.druid.cli.CliPeon.run(CliPeon.java:225) [druid-services-0.8.1.jar:0.8.1]
	at io.druid.cli.Main.main(Main.java:91) [druid-services-0.8.1.jar:0.8.1]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:159) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	... 2 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[?:1.7.0_80]
	at java.lang.String.<init>(String.java:203) ~[?:1.7.0_80]
	at java.lang.String.substring(String.java:1913) ~[?:1.7.0_80]
	at java.lang.String.subSequence(String.java:1946) ~[?:1.7.0_80]
	at com.google.common.base.Splitter$SplittingIterator.computeNext(Splitter.java:612) ~[guava-16.0.1.jar:?]
	at com.google.common.base.Splitter$SplittingIterator.computeNext(Splitter.java:522) ~[guava-16.0.1.jar:?]
	at com.google.common.base.AbstractIterator.tryToComputeNext(AbstractIterator.java:64) ~[guava-16.0.1.jar:?]
	at com.google.common.base.AbstractIterator.hasNext(AbstractIterator.java:59) ~[guava-16.0.1.jar:?]
	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43) ~[guava-16.0.1.jar:?]
	at com.metamx.common.collect.Utils.zipMapPartial(Utils.java:88) ~[java-util-0.27.0.jar:?]
	at com.metamx.common.parsers.DelimitedParser.parse(DelimitedParser.java:142) ~[java-util-0.27.0.jar:?]
	at io.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:86) ~[druid-api-0.3.9.jar:0.3.9]
	at io.druid.data.input.impl.StringInputRowParser.parse(StringInputRowParser.java:91) ~[druid-api-0.3.9.jar:0.3.9]
	at io.druid.data.input.impl.FileIteratingFirehose.nextRow(FileIteratingFirehose.java:54) ~[druid-api-0.3.9.jar:0.3.9]
	at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:364) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:205) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) ~[druid-indexing-service-0.8.1.jar:0.8.1]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[?:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.7.0_80]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.7.0_80]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_80]
2015-11-11T00:49:52,603 INFO [Thread-59] io.druid.cli.CliPeon - Running shutdown hook
2015-11-11 00:49:52,631 FATAL Unable to register shutdown hook because JVM is shutting down.


BTGeoTestData.tsv (2.04 KB)

Hi Gokila,

Maybe you can try increasing the heap size for your realtime service with the JVM -Xms flags, e.g.:

http://druid.io/docs/latest/configuration/production-cluster.html

Another thing you could try is tweaking the heap usage of the realtime via maxRowsInMemory and maxPendingPersists config:

http://druid.io/docs/latest/ingestion/realtime-ingestion.html

  • Jon