Error when indexing using Hortonworks YARN cluster


When I try to launch an index task I get

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
	at ~[druid-indexing-hadoop-]
	at io.druid.indexer.JobHelper.runJobs( ~[druid-indexing-hadoop-]
	at ~[druid-indexing-hadoop-]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask( ~[druid-indexing-service-]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_101]
	at sun.reflect.NativeMethodAccessorImpl.invoke( ~[?:1.8.0_101]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke( ~[?:1.8.0_101]
	at java.lang.reflect.Method.invoke( ~[?:1.8.0_101]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader( ~[druid-indexing-service-]

The hadoop version is 2.7.1

I’ve tried all the combination regarding hadoop config file and I don’t find a way to includes this in the classpath
I join the mapred-site.xml and core-site.xml

Could you help me to find out how to configure the index task


mapred-site.xml (6.81 KB)

core-site.xml (3.3 KB)

aggregation-index-static.json (2.92 KB)

Hey Richard,

Did you build Druid against 2.7.1 or did you adjust the hadoop-dependencies directory? If so you need to include hadoop-aws too; that was split out into its own package. It contains the NativeS3FileSystem.

Thanks it works !!

just to help other people that may want to run druid with hortonworks you need to add to your core-site.xml

mapreduce.job.classloader true

to prevent a package conflict