Passing Hadoop configuration in supervisor.conf

Hi,

Does anyone know how hadoop configuration files required for using hdfs as deep storage can be passed to druid processes?

I am able to use hdfs as deep storage when running the cluster locally. However i am using the docker-druid image for running a cluster in cloud. I have included the hadoop files in the classpath by adding them in a folder which is on classpath as shown below: (i add the hadoop files to docker image at /usr/local/druid/lib/

-cp /usr/local/druid/lib/*
io.druid.cli.Main server coordinator

``

However this doesn’t see to work. I see following errors on middle manager:

2019-02-01T07:00:01,088 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution

java.io.IOException: Mkdirs failed to create /druid/indexing-logs (exists=false, cwd=file:/var/lib/druid)

at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455) ~[?:?]

at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) ~[?:?]

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) ~[?:?]

I guess its considering that hdfs is running locally. Is there any other way to pass hdfs configuration when using hdfs as deep storage?

Thanks,

Prathamesh

Hi Prathamesh,
There is a section in the druid docs here:

http://druid.io/docs/latest/tutorials/cluster.html

Look through HDFS section and verify you have done those configuration steps.

Hope that helps.

Regards,

Robert