Error on hadoop batch ingestion

Hi, list.

I did a hadoop batch ingestion on Druid but I getting this error:

2016-11-16T20:00:24,544 INFO [localfetcher#17] org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - fetcher#17 - MergeManager returned Status.WAIT ...
2016-11-16T20:00:24,545 INFO [localfetcher#17] org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - fetcher#17 - MergeManager returned Status.WAIT ...
2016-11-16T20:00:24,545 INFO [localfetcher#17] org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - fetcher#17 - MergeManager returned Status.WAIT ...
2016-11-16T20:00:25,921 WARN [Thread-53] org.apache.hadoop.mapred.LocalJobRunner - job_local633977191_0001
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) [hadoop-mapreduce-client-common-2.3.0.jar:?]
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_101]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_91.out
	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) ~[hadoop-common-2.3.0.jar:?]
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) ~[hadoop-common-2.3.0.jar:?]
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) ~[hadoop-common-2.3.0.jar:?]
	at org.apache.hadoop.mapred.MROutputFiles.getInputFileForWrite(MROutputFiles.java:206) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.merge(MergeManagerImpl.java:450) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:94) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
2016-11-16T20:00:26,213 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Job job_local633977191_0001 failed with state FAILED due to: NA
2016-11-16T20:00:26,287 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 33

Does anyone know why?

thanks

Hi,
It seems the problem is that when Hadoop is trying to spill to disk, your nodes don’t have enough disk space to hold all the intermediate output of the mappers. Make sure you have enough capacity on the disk. you can set the directory location on local disk by setting the property - mapreduce.cluster.local.dir