Heap Space Error

Hi,
I was trying to ingest a huge file of size 500 mb approx. consisting of 1300000 rows of json data. I am running indexing task using overload service with heap space of size 6gb but it still giving me Out of Memory Exception. Please if anyone could help me with this issue, i would really appreciate that. Thanks in advance.

PS: please find the attached log file.

log.txt (9.81 KB)

Hi Manvendra,

It’s actually the peon spawned by overlord(if you are running overlord in local mode) that is doing the indexing job. What is the heap space used by your peon?

It’s likely your peon is using 256M default JVM heap space.

I believe you can configure the heap space using this property

druid.indexer.runner.javaOpts
-X Java options to run the peon in its own JVM.More info(http://druid.io/docs/latest/Indexing-Service-Config.html)

Thanks for your reply, increasing from 4gb (initially) to 6 gb, did work but i think it is too much of heap space for 500 MB data, along with this when indexing task succeeded it indexed only one row.
log :

015-06-18T15:24:56,144 INFO [task-runner-0] io.druid.indexing.common.index.YeOldePlumberSchool - Spilling index[0] with rows[1] to: /tmp/persistent/task/index_elastic_data_2015-06-18T15:23:26.891Z/work/elastic_data_2012-12-12T00:00:00.000Z_2012-12-13T00:00:00.000Z_2015-06-18T15:23:26.892Z_0/elastic_data_2012-12-12T00:00:00.000Z_2012-12-13T00:00:00.000Z_2015-06-18T15:23:26.892Z/spill0

2015-06-18T13:19:14,190 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Task[index_elastic_data_2015-06-18T13:17:39.265Z] interval[2012-12-12T00:00:00.000Z/2012-12-12T00:05:00.000Z] partition[0] took in 1 rows (1 processed, 0 unparseable, 0 thrown away) and output 1 rows

2015-06-18T13:19:26,643 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_elastic_data_2015-06-18T13:17:39.265Z",
  "status" : "SUCCESS",
  "duration" : 73648
}
2015-06-18T13:19:26,651 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.stop()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@59a2388e].
2015-06-18T13:19:26,651 INFO [main] io.druid.server.coordination.AbstractDataSegmentAnnouncer - Stopping class io.druid.server.coordination.BatchDataSegmentAnnouncer with config[io.druid.server.initialization.ZkPathsConfig@58d3f4be]
2015-06-18T13:19:26,651 INFO [main] io.druid.curator.announcement.Announcer - unannouncing [/druid/announcements/[172.16.13.182:8100](http://172.16.13.182:8100/)]
2015-06-18T13:19:26,889 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop()] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@2b736fee].
2015-06-18T13:19:26,898 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name='[172.16.13.182:8100](http://172.16.13.182:8100/)', host='[172.16.13.182:8100](http://172.16.13.182:8100/)', maxSize=0, tier='_default_tier', type='indexer-executor', priority='0'}]
2015-06-18T13:19:26,910 INFO [main] org.eclipse.jetty.server.ServerConnector - Stopped ServerConnector@68ef01a5{HTTP/1.1}{[0.0.0.0:8100](http://0.0.0.0:8100/)}
2015-06-18T13:19:26,915 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@1e01b133{/,null,UNAVAILABLE}
2015-06-18T13:19:26,924 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@101bdd1c].
2015-06-18T13:19:26,925 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.client.ServerInventoryView.stop() throws java.io.IOException] on object[io.druid.client.BatchServerInventoryView@31b7d869].
2015-06-18T13:19:26,925 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@724aefc3].
2015-06-18T13:19:26,925 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@6e3ecf5c].
2015-06-18T13:19:26,931 INFO [main] io.druid.curator.CuratorModule - Stopping Curator
2015-06-18T13:19:26,938 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down
2015-06-18T13:19:26,938 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x14e063b60700010 closed
2015-06-18T13:19:26,940 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.http.client.NettyHttpClient.stop()] on object[com.metamx.http.client.NettyHttpClient@58fa5769].
2015-06-18T13:19:26,971 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.metrics.MonitorScheduler.stop()] on object[com.metamx.metrics.MonitorScheduler@3456558].
2015-06-18T13:19:26,971 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@516462cc].
Heap
 garbage-first heap   total 5200896K, used 3766927K [0x0000000640000000, 0x0000000640109eb8, 0x00000007c0000000)
  region size 1024K, 166 young (169984K), 49 survivors (50176K)
 Metaspace       used 42763K, capacity 43236K, committed 43468K, reserved 1087488K
  class space    used 5449K, capacity 5559K, committed 5580K, reserved 1048576K

Is your JSON file one large array of objects or is it newline-delimited individual objects? Druid takes the latter, from looking at the stack trace, it’s running out of memory when it is trying to expand the size of the backing array while reading in a line from the file. My best guess as to why that could be is because it is treating the 500MB file as a single line.

–Eric

Thanks for your reply Eric and it did help as i wasn’t aware of that. But when i started indexing that file it took around 2 hours just for 200 rows out of 1300000 rows and index task takes complete sets of rows but process only small number of rows and throws rest of the every time. I have attached last of the log files if that could be of any use.

log.txt (124 KB)

Hi Manvendra, can you try running this same task with 0.8.0-rc1? There was a bug that was addressed around the indexing method used to ingest data for the index task that can lead to some of the same symptoms are you hitting.

http://static.druid.io/artifacts/releases/druid-0.8.0-rc1-bin.tar.gz