Does anyone know how hadoop configuration files required for using hdfs as deep storage can be passed to druid processes?
I am able to use hdfs as deep storage when running the cluster locally. However i am using the docker-druid image for running a cluster in cloud. I have included the hadoop files in the classpath by adding them in a folder which is on classpath as shown below: (i add the hadoop files to docker image at /usr/local/druid/lib/
io.druid.cli.Main server coordinator
However this doesn’t see to work. I see following errors on middle manager:
2019-02-01T07:00:01,088 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution
java.io.IOException: Mkdirs failed to create /druid/indexing-logs (exists=false, cwd=file:/var/lib/druid)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455) ~[?:?]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) ~[?:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) ~[?:?]
I guess its considering that hdfs is running locally. Is there any other way to pass hdfs configuration when using hdfs as deep storage?