Exception in retry loop java.io.IOException: Mkdirs failed to create file

I am trying to do the hadoop ingestion via docker tutorial but getting the following error:

2020-10-14T22:12:29,622 ERROR [pool-29-thread-1] io.druid.indexer.JobHelper - Exception in retry loop java.io.IOException: Mkdirs failed to create file:/druid/segments/datasource/20200626T000000.000Z_20200627T000000.000Z/2020-10-14T22_11_31.450Z (exists=false, cwd=file:/Users/lapfx/druid/druid-0.12.2-SNAPSHOT) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455) ~[hadoop-common-2.7.3.jar:?] at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) ~[hadoop-common-2.7.3.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) ~[hadoop-common-2.7.3.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:872) ~[hadoop-common-2.7.3.jar:?] at io.druid.indexer.JobHelper$4.push(JobHelper.java:417) [druid-indexing-hadoop-0.12.2-SNAPSHOT.jar:0.12.2-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_242] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_242] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_242] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_242] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) [hadoop-common-2.7.3.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) [hadoop-common-2.7.3.jar:?] at com.sun.proxy.$Proxy203.push(Unknown Source) [?:?] at io.druid.indexer.JobHelper.serializeOutIndex(JobHelper.java:434) [druid-indexing-hadoop-0.12.2-SNAPSHOT.jar:0.12.2-SNAPSHOT] at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:750) [druid-indexing-hadoop-0.12.2-SNAPSHOT.jar:0.12.2-SNAPSHOT] at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:500) [druid-indexing-hadoop-0.12.2-SNAPSHOT.jar:0.12.2-SNAPSHOT] at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) [hadoop-mapreduce-client-core-2.7.3.jar:?] at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) [hadoop-mapreduce-client-core-2.7.3.jar:?] at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) [hadoop-mapreduce-client-core-2.7.3.jar:?] at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319) [hadoop-mapreduce-client-common-2.7.3.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

Are there any errors in the Hadoop side? Also, do any ingestion tasks work? How have you configured Druid to store segments?

yes, I am getting this error on ingestion tasks which input data from the docker instances. I also followed https://druid.apache.org/docs/0.15.0-incubating/tutorials/tutorial-batch-hadoop.html tutorial on wikipedia data but still get the same error:
2020-10-15T16:40:38,995 ERROR [pool-32-thread-1] io.druid.indexer.JobHelper - Exception in retry loop java.io.IOException: Mkdirs failed to create file:/druid/segments/wikipedia/20150912T000000.000Z_20150913T000000.000Z/2020-10-15T16_39_20.127Z (exists=false, cwd=file:/Users/lapfx/druidoct/druid-0.12.2-SNAPSHOT) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455) ~[hadoop-common-2.7.3.jar:?] at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) ~[hadoop-common-2.7.3.jar:?] at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) ~[hadoop-common-2.7.3.jar:?]

common.runtime has the following properties set:

For HDFS:

druid.storage.type=hdfs
druid.storage.storageDirectory=/druid/segments

For HDFS:

druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=/druid/indexing-logs# For HDFS:

druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=/druid/indexing-logs

Ingestion spec is in the attachments.

wikipedia-index-hadoop.json (2.38 KB)

Any particular reason you have chosen to store the druid segments in HDFS? It is a more common practice to store them on a faster system, like a traditional filesystem. I would guess here that your druid user doesn’t have the right permissions to write the HDFS or there is a issue on that side. I’d recommend either storing your segments on a linux filesystem, or look at your hadoop instance to make sure you have that set up properly with the correct permissions, etc.

https://druid.apache.org/docs/latest/ingestion/faq.html#how-do-i-get-hdfs-to-work
https://druid.apache.org/docs/latest/development/extensions-core/hdfs.html

Sorry, I just realized that you had Deep Storage configured for HDFS. That is fine, you can ignore my previous note, but I still think that is where the issue might be.