Passing Hadoop configuration in supervisor.conf


Does anyone know how hadoop configuration files required for using hdfs as deep storage can be passed to druid processes?

I am able to use hdfs as deep storage when running the cluster locally. However i am using the docker-druid image for running a cluster in cloud. I have included the hadoop files in the classpath by adding them in a folder which is on classpath as shown below: (i add the hadoop files to docker image at /usr/local/druid/lib/

-cp /usr/local/druid/lib/*
io.druid.cli.Main server coordinator


However this doesn’t see to work. I see following errors on middle manager:

2019-02-01T07:00:01,088 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution Mkdirs failed to create /druid/indexing-logs (exists=false, cwd=file:/var/lib/druid)

at org.apache.hadoop.fs.ChecksumFileSystem.create( ~[?:?]

at org.apache.hadoop.fs.ChecksumFileSystem.create( ~[?:?]

at org.apache.hadoop.fs.FileSystem.create( ~[?:?]

I guess its considering that hdfs is running locally. Is there any other way to pass hdfs configuration when using hdfs as deep storage?



Hi Prathamesh,
There is a section in the druid docs here:

Look through HDFS section and verify you have done those configuration steps.

Hope that helps.