Not being able to resolve hostnames in Graphite Emitter with Druid 0.9.2

Hi

I’m trying to setup graphite emitter with Druid 0.9.2 and I’ve stumbled across a problem which I can’t make sense of.

Configuration:

druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”, “com.metamx.metrics.SysMonitor”]
druid.request.logging.type=emitter
druid.emitter=graphite
druid.emitter.logging.logLevel=debug
druid.emitter.graphite.hostname=“carbon.hostedgraphite.com
druid.emitter.graphite.port=2004
druid.emitter.graphite.eventConverter={“type”:“all”, “namespacePrefix”: “druid.poc”, “ignoreHostname”:false, “ignoreServiceName”:false}

``

The error is:

2017-02-10T12:23:10,251 INFO [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - trying to connect to graphite server
2017-02-10T12:23:10,257 ERROR [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - “carbon.hostedgraphite.com
java.net.UnknownHostException: “carbon.hostedgraphite.com
at com.codahale.metrics.graphite.PickledGraphite.connect(PickledGraphite.java:182) ~[metrics-graphite-3.1.2.jar:3.1.2]
at io.druid.emitter.graphite.GraphiteEmitter$ConsumerRunnable.run(GraphiteEmitter.java:145) [graphite-emitter-0.9.1.1.jar:0.9.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

``

After a bit of digging in metrics-graphite-3.2 and java.net.inetAddress it seems the problem is due to name server lookups not working through Druid while dig as well as a stand alone Java program can do it[1].

I added a small patch to metrics-graphite-3.2 branch to see if this is the case, patch looks like:

diff --git a/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java b/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java
index 6917dd7…1d386ea 100644
— a/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java
+++ b/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java
@@ -12,6 +12,7 @@ import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.net.InetSocketAddress;
+import java.net.InetAddress;
import java.net.Socket;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
@@ -176,7 +177,12 @@ public class PickledGraphite implements GraphiteSender {
if (address == null) {
address = new InetSocketAddress(hostname, port);
}

Hi Sam Thanks for the report, i will try to look at this ASAP.

What is your OS ?

Hi

I’ve tested this with Ubuntu 16.04.{1,2} LTS

Thanks,
Sam

Hi

After comparing DNS queries between the Resolve.java and plugin I pinpointed the problem. The configuration was wrong. It shouldn’t be quoted hostname. I could see the quoted hostname being part of the lookup query which of course wouldn’t work.