Prevent Middle Manager from filling up the disk space "Spill failed"

Hello,

What configuration variable setting will prevent the Middle Manager disk space from filling up? This will cause the indexing tasks to fail.

java.io.IOException: Spill failed
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1533) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1442) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1988) [hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774) [hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) [hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) [hadoop-mapreduce-client-common-2.3.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_72-internal]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_72-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72-internal]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72-internal]
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/spill0.out
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) ~[hadoop-common-2.3.0.jar:?]
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) ~[hadoop-common-2.3.0.jar:?]
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) ~[hadoop-common-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MROutputFiles.getSpillFileForWrite(MROutputFiles.java:146) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1564) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:853) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]

``

These are my current configs:

druid.service=druid/middleManager

druid.host=HOST_IP_ADDRESS

druid.port=8091

Number of tasks per middleManager

druid.worker.capacity=3

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

druid.indexer.task.baseTaskDir=var/druid/task

HTTP server threads

druid.server.http.numThreads=25

Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numThreads=2

Hadoop indexing

druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

druid.indexer.task.defaultHadoopCoordinates=[“org.apache.hadoop:hadoop-client:2.3.0”]

``

-server

-Xms64m

-Xmx64m

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=var/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

``

The middle manager that is failing runs in an m4.xlarge. The problem folders are:

/tmp/hadoop-root @ 14GB

DRUID_PATH/var/tmp @ 15GB

Thank you for any help in advance.

What types of tasks are you normally running? You can try to run less tasks per middle manager, or if you are running a lot of realtime tasks, the setting most related is handoff. If you are running out of disk space, you should try creating smaller segments and handing them off faster.

Hello Fangjin

For example, a single indexing task attempts to process 40 million events distributed across 31 CSV files on S3; each one around 200MB in size. That would be a day’s worth of data, and I have a need to load at least a year. There is orchestration around the middle managers, so I can spin up dozens of them immediately, but should I consider the EMR solution instead?

Hey Carlos,

Try adding this to your middleManager’s java startup line:

-Dhadoop.hadoop.tmp.dir=var/hadoop-tmp

That should stop it from storing things in /tmp. If after that, var is still not big enough, you might need more disk space on your instance.

Btw, if you have access to a hadoop cluster you’ll probably find that’s a more scalable and reliable way of loading a lot of data in batch in production. The local-mode batch tasks are mostly intended to make peoples’ lives easier in simple development environments.