Druid 0.8.3 - How to download druid dependencies using proxy

Hello,

I am using druid 0.8.3 with the core extensions ( kafka-eight and mysql-metadata-storage ). I have deployed druid on the machine which doesn’t have direct connection to remote repositories ( upstream proxy). Sometimes, druid tries to install maven dependencies and fails to start.

From the different blog posts and druid docs, it seems that druid tar contains all the required jars that druid and its core extensions need. But, in this case it tries to download jars and fails.

My question is regarding the possibility of using proxy to download jars. Is there any possibility that applying proxy setting to start command would let druid download required jars?

I tried using java -Dhttp.proxyHost= -Dhttp.proxyPort= -classpath lib/*:config/_common:config/broker io.druid.cli.Main server broker command but druid is still not able to download the missing artifacts.
If this is not the case, then what are the workarounds ? use of pull-deps to pull the dependencies and re-bundle druid will solve the issue ?

Any suggestions on this would be the great help.

Thanks,
Jvalant

sample exception:

INFO: HV000001: Hibernate Validator 5.1.3.Final

2016-09-17T03:33:39,994 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-kafka-eight, io.druid.exten

sions:mysql-metadata-storage], defaultVersion=‘0.8.3’, localRepository=‘extensions-repo’, remoteRepositories=}]

2016-09-17T03:33:40,770 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.cli.CliCommandCreator]

2016-09-17T03:33:58,644 ERROR [main] io.druid.initialization.Initialization - Unable to resolve artifacts for [io.druid.extensions:druid-kafka-eight:jar:0.8.3 (runtime) -> < ].

org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at io.druid.extensions:druid-kafka-eight:jar:0.8.3 -> org.apache.kafka:kafka_2.10:jar:0.8.2.1 -> org.apache.zookeeper:zookeeper:jar:3.4.6 -> jline:jline:jar:0.9.94 -> junit:junit

:jar:4.11 -> org.hamcrest:hamcrest-core:jar:1.3

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:380) ~[aether-impl-0.9.0.M2.jar:?]

at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289) ~[tesla-aether-0.0.5.jar:0.0.5]

at io.druid.initialization.Initialization.getClassLoaderForCoordinates(Initialization.java:254) [druid-server-0.8.3.jar:0.8.3]

at io.druid.initialization.Initialization.getFromExtensions(Initialization.java:154) [druid-server-0.8.3.jar:0.8.3]

at io.druid.cli.Main.main(Main.java:84) [druid-services-0.8.3.jar:0.8.3]

Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at io.druid.extensions:druid-kafka-eight:jar:0.8.3 -> org.apache.kafka:kafka_2.10:jar:0.8.2.1 -> org.apache.zookeeper:zookeeper:jar:3.4.6 -> jline:jline:jar:0.9.94 ->

junit:junit:jar:4.11 -> org.hamcrest:hamcrest-core:jar:1.3

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:292) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:342) ~[aether-impl-0.9.0.M2.jar:?]

… 4 more

Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.hamcrest:hamcrest-core:jar:1.3

at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:335) ~[maven-aether-provider-3.1.1.jar:3.1.1]

at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217) ~[maven-aether-provider-3.1.1.jar:3.1.1]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:342) ~[aether-impl-0.9.0.M2.jar:?]

… 4 more

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.hamcrest:hamcrest-core:pom:1.3 from/to central (http://repo.maven.apache.org/maven2): /usr/local/druid/extensions-repo/org/hamcrest/hamcrest-core/1.3/aether-64c62218-8c

31-4b93-9fa5-da92c1dc099b-hamcrest-core-1.3.pom-in-progress (No such file or directory)

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239) ~[aether-impl-0.9.0.M2.jar:?]

at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:320) ~[maven-aether-provider-3.1.1.jar:3.1.1]

at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217) ~[maven-aether-provider-3.1.1.jar:3.1.1]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:342) ~[aether-impl-0.9.0.M2.jar:?]

… 4 more

Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.hamcrest:hamcrest-core:pom:1.3 from/to central (http://repo.maven.apache.org/maven2): /usr/local/druid/extensions-repo/org/hamcrest/hamcrest-core/1.3/aether-64c62218-8c31-4

b93-9fa5-da92c1dc099b-hamcrest-core-1.3.pom-in-progress (No such file or directory)

at io.tesla.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:830) ~[aether-connector-okhttp-0.0.9.jar:0.0.9]

at io.tesla.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:824) ~[aether-connector-okhttp-0.0.9.jar:0.0.9]

at io.tesla.aether.connector.AetherRepositoryConnector$GetTask.flush(AetherRepositoryConnector.java:619) ~[aether-connector-okhttp-0.0.9.jar:0.0.9]

at io.tesla.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:238) ~[aether-connector-okhttp-0.0.9.jar:0.0.9]

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:535) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:436) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239) ~[aether-impl-0.9.0.M2.jar:?]

at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:320) ~[maven-aether-provider-3.1.1.jar:3.1.1]

at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217) ~[maven-aether-provider-3.1.1.jar:3.1.1]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573) ~[aether-impl-0.9.0.M2.jar:?]

at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)

Hi All,

Any pointers on this would be greatly appreciated. Addition to above question, the other question in my mind was about the druid tar. So, does druid 0.8.3 tar comes with all the necessary jars that it needs for druid and core extensions ?

Thanks,
Jvalant

Druid bundles all the core extensions in the tar ball except from mysql, they are in the “extensions” directory. Read this - http://druid.io/docs/latest/operations/including-extensions.html for how to download and include extensions. Use pull-deps to download the extensions and place them in the extensions folder. I would not recommend downloading extensions at runtime.

Hi All,

Any pointers on this would be greatly appreciated. Addition to above question, the other question in my mind was about the druid tar. So, does druid 0.8.3 tar comes with all the necessary jars that it needs for druid and core extensions ?

Thanks,
Jvalant

Just realized that you are using Druid 0.8.3, please ignore what I said earlier that applies only to 0.9 versions. For 0.8.3 you can follow this - http://druid.io/docs/0.8.3/operations/including-extensions.html

Druid bundles all the core extensions in the tar ball except from mysql, they are in the “extensions” directory. Read this - http://druid.io/docs/latest/operations/including-extensions.html for how to download and include extensions. Use pull-deps to download the extensions and place them in the extensions folder. I would not recommend downloading extensions at runtime.

Hi All,

Any pointers on this would be greatly appreciated. Addition to above question, the other question in my mind was about the druid tar. So, does druid 0.8.3 tar comes with all the necessary jars that it needs for druid and core extensions ?

Thanks,
Jvalant

Hi Parag,

Really appreciate you recommendation. So, if I use pull-deps tool, does it downloads all the required jar? I mean is it guaranteed that it won’t fail to start if my machine won’t have internet connectivity?

Can I start my druid processes providing proxy settings to download jars ?

Hoping for your response.

Thanks,
Jvalant

Hey Jvalant,

This is all much simpler in 0.9.x, so I recommend upgrading to that if possible. But if you are stuck on 0.8.x, you can download extension dependencies with a command like:

java \

-cp “lib/*” \

-Ddruid.extensions.coordinates=’[“io.druid.extensions:druid-hdfs-storage”,“io.druid.extensions:postgresql-metadata-storage”]’ \

-Ddruid.extensions.localRepository=“extensions-repo” \

io.druid.cli.Main tools pull-deps

Hi Gian,
Thanks for the suggestion. Is it possible that I provide proxy settings while starting druid processes?
Ex.
**java **-Dhttp.proxyHost= -Dhttp.proxyPort= -classpath lib/*:config/_common:config/broker io.druid.cli.Main server broker

Please let me know if there is any possibility to pass proxy settings to druid processes to download required dependencies at run time.

Thanks,
Jvalant

I’m not sure if that’ll work; I haven’t ever tried using a proxy to download dependencies with 0.8.x. Usually I stuck with running pull-deps in advance.