pull-deps -h gives an error

I’m trying to build a dependency directory for druid based on hadoop 2.6.0

According to the latest instructions that should look something like this:

java -Ddruid.extensions.hadoopDependenciesDir=/usr/local/druid/hadoop/hadoop_druid_dependencies -classpath “/usr/local/bin/druid/lib/*” io.druid.cli.Main tools pull-deps --hadoop-coordinate org.apache.hadoop:hadoop-client:2.6.0

But when I run this I get the error:

Dec 28, 2015 6:45:28 PM org.hibernate.validator.internal.util.Version
INFO: HV000001: Hibernate Validator 5.1.3.Final
2015-12-28T18:45:28,650 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=, defaultVersion=‘0.8.2’, localRepository=’/home/ec2-user/.m2/repository’, remoteRepositories=[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]
ERROR!!!
Found unexpected parameters: [–hadoop-coordinate, org.apache.hadoop:hadoop-client:2.4.0]

It would seem this does not do what I thought.

I don’t see a:
/usr/local/druid/hadoop/hadoop_druid_dependencies

directory being created. :frowning:

I’m just trying to get around the jackson xml problem.

So far I’ve tried:

recompiling the source from master. (latest Druid does not compile with the older jackson I need).

following the pull-deps documentation.

Still no closer to getting hadoop batch loading working. :frowning:

I have a hadoop 2.6.0 cluster that appears to be using jackson 2.2.3

Tried building a fat jar and launching with that. Now I get:

2015-12-28_21:34:22.80010 2015-12-28T21:34:22,798 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties

2015-12-28_21:34:22.80173 2015-12-28T21:34:22,801 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties

2015-12-28_21:34:22.83593 2015-12-28T21:34:22,835 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final

2015-12-28_21:34:23.33894 2015-12-28T21:34:23,338 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-s3-extensions, io.druid.extensions:druid-histogram, io.druid.extensions:mysql-metadata-storage], defaultVersion=‘0.1-SNAPSHOT’, localRepository=’/home/ec2-user/.m2/repository’, remoteRepositories=[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]

2015-12-28_21:34:23.48761 2015-12-28T21:34:23,487 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-s3-extensions] for class[io.druid.cli.CliCommandCreator]

2015-12-28_21:34:24.18585 2015-12-28T21:34:24,182 ERROR [main] io.druid.initialization.Initialization - Unable to resolve artifacts for [io.druid.extensions:druid-s3-extensions:jar:0.1-SNAPSHOT (runtime) -> < [ (https://repo1.maven.org/maven2/, releases+snapshots), (https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local, releases+snapshots)]].

2015-12-28_21:34:24.18589 java.lang.NullPointerException

2015-12-28_21:34:24.18590 at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:361) ~[druid_build-assembly-0.1-SNAPSHOT.jar:0.1-SNAPSHOT]

2015-12-28_21:34:24.18590 at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289) ~[druid_build-assembly-0.1-SNAPSHOT.jar:0.1-SNAPSHOT]

2015-12-28_21:34:24.18591 at io.druid.initialization.Initialization.getClassLoaderForCoordinates(Initialization.java:253) [druid_build-assembly-0.1-SNAPSHOT.jar:0.1-SNAPSHOT]

2015-12-28_21:34:24.18591 at io.druid.initialization.Initialization.getFromExtensions(Initialization.java:153) [druid_build-assembly-0.1-SNAPSHOT.jar:0.1-SNAPSHOT]

2015-12-28_21:34:24.18591 at io.druid.cli.Main.main(Main.java:76) [druid_build-assembly-0.1-SNAPSHOT.jar:0.1-SNAPSHOT]

2015-12-28_21:34:24.18645 Exception in thread “main” java.lang.NullPointerException

2015-12-28_21:34:24.18654 at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:361)

2015-12-28_21:34:24.18659 at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289)

2015-12-28_21:34:24.18666 at io.druid.initialization.Initialization.getClassLoaderForCoordinates(Initialization.java:253)

2015-12-28_21:34:24.18673 at io.druid.initialization.Initialization.getFromExtensions(Initialization.java:153)

2015-12-28_21:34:24.18678 at io.druid.cli.Main.main(Main.java:76)

For all the stuff I’ve tried I just appear to be trading one set of errors for another.

Figured out my error with using the sbt compiled fat jar.

I needed to comment out the extensions in:

config/_common/common.runtime.properties

because the fat jar already contains them.

Hi DashV, if you don’t use pull dependencies and instead include the hadoop configurations and hadoop jars in the classpath, does it work then?

Dunno.

Managed to get it working with the fat jar and I ran out of time to explore other options. We’ve run several jobs with the fat jar now. So I will most likely stick with this as the solution for the near term.