Example <hadoop_config_path> values

Question 1: Can anyone provide example values of the “<hadoop_config_path>” value mentioned on the following pages?

From http://druid.io/docs/latest/tutorials/tutorial-loading-batch-data.html

– Quote Start –

To start the Indexing Service:

java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/overlord:lib/*:<hadoop_config_path> io.druid.cli.Main server overlord

``

– Quote End –

From http://druid.io/docs/latest/ingestion/batch-ingestion.html

– Quote Start –

The HadoopDruidIndexer can be run like so:

java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:<hadoop_config_path> io.druid.cli.Main index hadoop <spec_file>

``

– Quote End –

Question 2: Is the “<hadoop_config_path>” value just the folder that contains the “core-site.xml” and “hdfs-site.xml” files?

Question 3: How does the Hadoop Config path get set with the Druid Indexing Service?

http://druid.io/docs/latest/ingestion/batch-ingestion.html

curl -X 'POST' -H 'Content-Type:application/json' -d @example_index_hadoop_task.json localhost:8090/druid/indexer/v1/task

``

Question 4: Does the indexing service (localhost:8090/druid/indexer/v1/task) just inherit the classpath from the Druid Overlord (Default port for 8090)?

For question 1,2:

Yes, the hadoop_config_path is a location where core-site.xml, hdfs-site.xml etc. are present.

You can add the original directory containing those files to the classpath, or I believe you could also copy those .xml files into config/_common with the rest of the druid config.

For question 3,4:

That indexing service endpoint is part of the overlord, I believe you would need to add a directory containing the hadoop config to your overlord classpath (i.e., it is inherited).

  • Jon

Thanks for the response.

I am tried your suggestion about putting my Hadoop configuration into the druid/config/_common/ directory, and you were correct, my core-site.xml and hdfs-site.xml configuration are picked up as expected.