How to install a community extension in Druid off a maven repo?

Has anyone had any luck with attempting at installing druid’s graphite-emitter community extension?

root@2469696db2d9:/opt/druid# java -cp {DRUID_PATH}conf/druid/{ROLE}:${DRUID_PATH}/lib/* io.druid.cli.Main tools pull-deps --no-default-remote-repositories -c “io.druid.extensions:graphite-emitter:0.9.2” -r http://central.maven.org/maven2/

Feb 01, 2017 11:09:05 PM org.hibernate.validator.internal.util.Version

INFO: HV000001: Hibernate Validator 5.1.3.Final

2017-02-01T23:09:06,009 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=‘extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, hadoopContainerDruidClasspath=‘null’, loadList=null}]

2017-02-01T23:09:06,013 INFO [main] io.druid.initialization.Initialization - Loading extension [graphite-emitter] for class [io.druid.cli.CliCommandCreator]

2017-02-01T23:09:06,267 INFO [main] io.druid.cli.PullDependencies - Root extension directory [%s] already exists, skip creating

2017-02-01T23:09:06,267 INFO [main] io.druid.cli.PullDependencies - Root extension directory [%s] already exists, skip creating

2017-02-01T23:09:06,267 INFO [main] io.druid.cli.PullDependencies - Start pull-deps with local repository [/root/.m2/repository] and remote repositories [[http://central.maven.org/maven2/]]

2017-02-01T23:09:06,267 INFO [main] io.druid.cli.PullDependencies - Start downloading dependencies for extension coordinates: [[io.druid.extensions:graphite-emitter:0.9.2]]

2017-02-01T23:09:06,270 INFO [main] io.druid.cli.PullDependencies - Directory [/opt/druid/extensions/graphite-emitter] already exists, skipping creating a directory

2017-02-01T23:09:06,276 INFO [main] io.druid.cli.PullDependencies - Start downloading extension [io.druid.extensions:graphite-emitter:jar:0.9.2]

2017-02-01T23:09:06,806 ERROR [main] io.druid.cli.PullDependencies - Unable to resolve artifacts for [io.druid.extensions:graphite-emitter:jar:0.9.2 (runtime) -> < [ (http://central.maven.org/maven2/, releases+snapshots)]].

org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact io.druid.extensions:graphite-emitter:jar:0.9.2 in (http://central.maven.org/maven2/)

at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384) ~[aether-impl-0.9.0.M2.jar:?]

at io.tesla.aether.internal.DefaultTeslaAether.resolveArtifacts(DefaultTeslaAether.java:289) ~[tesla-aether-0.0.5.jar:0.0.5]

at io.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:350) [druid-services-0.9.2.jar:0.9.2]

hi the group id is

io.druid.extensions.contrib

not

That worked, Slim! Thanks man!

Hey, Slim - another question for ya brother?

I’m getting a bunch of these errors:

io.druid.emitter.graphite.GraphiteEmitter - Already connected

Here is my configuration across my nodes:

99 druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

100 druid.emitter=graphite

101 druid.emitter.graphite.batchSize=1

114 # Graphite emitters:

115 druid.emitter.graphite.hostname=druid-graphite-environment.demandbase.com

116 druid.emitter.graphite.port=2004

117 druid.emitter.graphite.flushPeriod=6

118 druid.emitter.graphite.eventConverter={“type”:“all”, “namespacePrefix”: “.druid.environment”}

119

I’ve now got the Graphite Emitter “emitting” metrics, but I don’t see them in Graphite nor do I see

can you use druid.environment as prefix instead of .druid.evironment ?

Yessir, lemme try that right quick. Thanks for the prompt response :slight_smile:

Slim, same exception:

2/3/2017 8:35:48 AM2017-02-03T16:35:48,137 ERROR [GraphiteEmitter-1] io.druid.emitter.graphite.GraphiteEmitter - Already connected

2/3/2017 8:35:48 AMjava.lang.IllegalStateException: Already connected

2/3/2017 8:35:48 AM	at com.codahale.metrics.graphite.PickledGraphite.connect(PickledGraphite.java:175) ~[metrics-graphite-3.1.2.jar:3.1.2]

2/3/2017 8:35:48 AM	at io.druid.emitter.graphite.GraphiteEmitter$ConsumerRunnable.run(GraphiteEmitter.java:177) [graphite-emitter-0.9.2.jar:0.9.2]

2/3/2017 8:35:48 AM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

2/3/2017 8:35:48 AM	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_121]

2/3/2017 8:35:48 AM	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]

2/3/2017 8:35:48 AM	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_121]

2/3/2017 8:35:48 AM	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

2/3/2017 8:35:48 AM	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

2/3/2017 8:35:48 AM	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

The beginning of the exception:

2/3/2017 8:37:27 AMFeb 3, 2017 16:37:27 +0000 [21 28] com.newrelic INFO: Using RUM version 1016 for application "druid-broker-develop"

2/3/2017 8:37:27 AMFeb 3, 2017 16:37:27 +0000 [21 28] com.newrelic INFO: Real user monitoring is enabled with auto instrumentation for application "druid-broker-develop"

2/3/2017 8:38:22 AM2017-02-03T16:38:22,629 ERROR [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - Broken pipe (Write failed)

2/3/2017 8:38:22 AMjava.net.SocketException: Broken pipe (Write failed)

2/3/2017 8:38:22 AM	at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.net.SocketOutputStream.write(SocketOutputStream.java:143) ~[?:1.8.0_121]

2/3/2017 8:38:22 AM	at com.codahale.metrics.graphite.PickledGraphite.writeMetrics(PickledGraphite.java:262) ~[metrics-graphite-3.1.2.jar:3.1.2]

2/3/2017 8:38:22 AM	at com.codahale.metrics.graphite.PickledGraphite.send(PickledGraphite.java:215) ~[metrics-graphite-3.1.2.jar:3.1.2]

2/3/2017 8:38:22 AM	at io.druid.emitter.graphite.GraphiteEmitter$ConsumerRunnable.run(GraphiteEmitter.java:165) [graphite-emitter-0.9.2.jar:0.9.2]

2/3/2017 8:38:22 AM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

2/3/2017 8:38:22 AM	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

2/3/2017 8:38:22 AM2017-02-03T16:38:22,634 ERROR [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - Already connected

2/3/2017 8:38:22 AMjava.lang.IllegalStateException: Already connected

2/3/2017 8:38:22 AM	at com.codahale.metrics.graphite.PickledGraphite.connect(PickledGraphite.java:175) ~[metrics-graphite-3.1.2.jar:3.1.2]

2/3/2017 8:38:22 AM	at io.druid.emitter.graphite.GraphiteEmitter$ConsumerRunnable.run(GraphiteEmitter.java:177) [graphite-emitter-0.9.2.jar:0.9.2]

2/3/2017 8:38:22 AM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

Current config:

druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"]

druid.emitter=graphite

druid.emitter.graphite.batchSize=1

# Graphite emitters:

druid.emitter.graphite.hostname=druid-graphite-develop.demandbase.com

druid.emitter.graphite.port=2004

druid.emitter.graphite.flushPeriod=6

druid.emitter.graphite.eventConverter={"type":"all", "namespacePrefix": "druid.d

evelop"}

Seems like it is a IO exception make sure the port/hostname is correct.

can you telnet from the host to that host/port ?

Right now I connect through an AWS load balancer over TCP to port 2004 which routes to TCP port 2004. Is this valid for Pickle?

From container:

root@08f6ea2bcd84:/opt/druid# telnet druid-graphite-develop.demandbase.com 2004

Trying 10.9.2.60…

Connected to druid-graphite-develop.demandbase.com.

Escape character is ‘^]’.

Hey, slim - one more question - it seems that only my coordinator is reporting. Is this due to the way I convert the emitter?

Hey the flush period is in Ms can you set that to something less aggressive like 60000 (e.g. one minute)?

same for batch size ~ hundreds .

Thanks slim!

Hi, Slim -

Question for you - It seems like running graphite in a container with Rancher, and Druid sending emitters to Graphite takes up quite a bit of space. Have you seen this behavior? I get disk full errors off other services that run on the same host as Graphite.

well if you are dumping all (e.g. Without filter) the druid metrics toward one graphite node i am sure you will crash it 100%. Druid sends lot of high cardinality metrics

I recommend to filter to some selective metrics depending on what you want to monitor. You can for instance drop all the high cardinality dimensions like QueryId.