Emitting metrics to dogstatsd

Hello
I am trying to emit metrics to dogstatsd using statsd emitter. I was successful in installing datadog on the machine, install statsd-emitter via pull-deps.

I see the following logs when I start my broker node

2019-03-27T18:38:44,909 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [statsd-emitter] for class [interface org.apache.druid.cli.CliCommandCreator]
2019-03-27T18:38:44,919 INFO [main] org.apache.druid.initialization.Initialization - added URL[file:/usr/local/lib/druid/extensions/statsd-emitter/statsd-emitter-0.12.3.jar] for extension[statsd-emitter]
2019-03-27T18:38:44,920 INFO [main] org.apache.druid.initialization.Initialization - added URL[file:/usr/local/lib/druid/extensions/statsd-emitter/java-statsd-client-3.0.1.jar] for extension[statsd-emitter]
2019-03-27T18:38:45,148 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [statsd-emitter] for class [interface org.apache.druid.initialization.DruidModule]
2019-03-27T18:38:46,085 INFO [main] org.apache.druid.cli.CliBroker - * druid.emitter: “statsd”
2019-03-27T18:38:46,085 INFO [main] org.apache.druid.cli.CliBroker - * druid.extensions.loadList: [“statsd-emitter”, “druid-s3-extensions”, “postgresql-metadata-storage”]
2019-03-27T18:39:53,831 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=’/usr/local/lib/druid/extensions’, useExtensionClassloaderFirst=false, hadoopDependenciesDir=’/usr/local/lib/druid/hadoop-dependencies’, hadoopContainerDruidClasspath=‘null’, addExtensionsToHadoopContainer=false, loadList=[statsd-emitter, druid-s3-extensions, postgresql-metadata-storage]}]
2019-03-27T18:39:53,834 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [statsd-emitter] for class [interface org.apache.druid.cli.CliCommandCreator]
2019-03-27T18:39:53,845 INFO [main] org.apache.druid.initialization.Initialization - added URL[file:/usr/local/lib/druid/extensions/statsd-emitter/statsd-emitter-0.12.3.jar] for extension[statsd-emitter]
2019-03-27T18:39:53,845 INFO [main] org.apache.druid.initialization.Initialization - added URL[file:/usr/local/lib/druid/extensions/statsd-emitter/java-statsd-client-3.0.1.jar] for extension[statsd-emitter]
2019-03-27T18:39:54,042 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [statsd-emitter] for class [interface org.apache.druid.initialization.DruidModule]
2019-03-27T18:39:54,732 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping druid.emitter.statsd.prefix property: one of it’s prefixes is also used as a property key. Prefix: druid
2019-03-27T18:39:54,732 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping druid.emitter.statsd.dogstatsd property: one of it’s prefixes is also used as a property key. Prefix: druid
2019-03-27T18:39:54,733 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping druid.emitter.statsd.dogstatsdConstantTags property: one of it’s prefixes is also used as a property key. Prefix: druid
2019-03-27T18:39:54,733 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping druid.emitter.statsd.hostname property: one of it’s prefixes is also used as a property key. Prefix: druid
2019-03-27T18:39:54,734 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping druid.emitter.statsd.port property: one of it’s prefixes is also used as a property key. Prefix: druid
2019-03-27T18:39:54,940 INFO [main] org.apache.druid.cli.CliBroker - * druid.emitter: “statsd”
2019-03-27T18:39:54,940 INFO [main] org.apache.druid.cli.CliBroker - * druid.emitter.statsd.dogstatsd: true
2019-03-27T18:39:54,940 INFO [main] org.apache.druid.cli.CliBroker - * druid.emitter.statsd.hostname: “localhost”
2019-03-27T18:39:54,941 INFO [main] org.apache.druid.cli.CliBroker - * druid.emitter.statsd.port: “8125”
2019-03-27T18:39:54,941 INFO [main] org.apache.druid.cli.CliBroker - * druid.emitter.statsd.prefix: druid.metric
2019-03-27T18:39:54,941 INFO [main] org.apache.druid.cli.CliBroker - * druid.extensions.loadList: [“statsd-emitter”, “druid-s3-extensions”, “postgresql-metadata-storage”]

``

I looked at the code for StatsD Emitter, and tried to grep for logs of the emitter, but I could not find any.

My configuration is below

druid.emitter=“statsd”
druid.emitter.statsd.hostname=“localhost”
druid.emitter.statsd.port=“8125”
druid.emitter.statsd.prefix=druid.metric
druid.emitter.statsd.dogstatsd=true

``

There are two questions I have

  1. I am not sure what the below means

Skipping druid.emitter.statsd.prefix property: one of it’s prefixes is also used as a property key. Prefix: druid

I tried looking for in the code and just found the following comment

// Clash is possible between properties, which are used to configure different objects: e. g.
// druid.emitter=parametrized is used to configure Emitter class, and druid.emitter.parametrized.xxx=yyy is used
// to configure ParametrizedUriEmitterConfig object. So skipping xxx=yyy key-value pair when configuring Emitter
// doesn't make any difference. That is why we just log this situation, instead of throwing an exception.
  1. Why am I not seeing data in datadog? How do I verify if emitter is working or not.

Thanks

Hi Monil,

I have seen that error when I had an extra space in one of the emitter configs. Can you try searching the broker log for io.druid.server.emitter.EmitterModule - Underlying emitter for ServiceEmitter: This should reveal which emitter Druid is using.

Best,

Caroline

Hi Caroline,

This is what I see
2019-03-27T20:50:40,269 INFO [main] org.apache.druid.server.emitter.EmitterModule - Underlying emitter for ServiceEmitter: org.apache.druid.java.util.emitter.core.NoopEmitter@10641c09

``

It seems it is defaulting to NoopEmitter which means that the StatsDEmitter is not loading. Am I right?

If yes, I am not sure how to debug the issue. any pointers will be great.

Thanks

What version of Druid are you using? dogstatsd-specific support (including the dogstatsd=true flag you’re setting) is new in Druid 0.14 which is currently unreleased (though there are RCs available).

That said you should be able to at least test it with 0.13, you’ll just end up with lots of metrics with different names instead of tags.

Hi Monil,

If you are seeing the Noop emitter the statsd-emitter isn’t getting loaded. This is coming down the pipeline at some point:

https://github.com/apache/incubator-druid/pull/7328

but in the meantime check for config errors. Try removing the quotes in

druid.emitter=“statsd”
druid.emitter.statsd.hostname=“localhost”
druid.emitter.statsd.port=“8125”

Best,

Caroline

I am running on 0.13.0.

The silent failure makes sense. Thanks for the link. I tried removing the quotes but that did not seem to work.

How can I find out where the load is failing? My guess would be connecting to the localhost dogstatd client. Is there an easy way or a followed way to debug?

David,
Does the 0.12.X version of statsd work with 0.13.0 version of druid? I added some debug logs to the statsd on 0.13.0 tag. I was able to see the binding work, but I was not able to see logs from following function. I even hardcoded the config in the code to make sure it is not my config that is messing up the bootup

static final StatsDEmitter of(StatsDEmitterConfig config, ObjectMapper mapper)

``

Link of the above function is here

Also, how do I download 0.14.0-rc3?

Bumping… Hoping someone notices this thread.
Here is how my directory looks like after pull deps

-rw-rw-r-- 1 druid druid 8978 Apr 10 16:47 java-statsd-client-3.0.1.jar

-rw-rw-r-- 1 druid druid 15234 Apr 10 16:47 statsd-emitter-0.12.3.jar

My druid versions is 0.13.0

The config is as follows

druid.emitter=statsd

druid.emitter.logging.logLevel=debug

druid.emitter.statsd.hostname=localhost

druid.emitter.statsd.port=8125

druid.emitter.statsd.prefix=druid-environment

druid.emitter.statsd.includeHost=true

druid.monitoring.monitors=[“org.apache.druid.java.util.metrics.JvmMonitor”]

I think there is something failing silently, but I am not sure what or why.

Any pointers will be helpful.

Druid 0.12.X version is non-apache version which uses io.druid for package names, and 0.13.0 is apache version which uses org.apache.druid naming
mixing extensions from these versions is not supposed to work smoothly. If you are still trying that.

In case you have updated the druid version and statsd-emitter both to 0.13.0 version, make sure the mertics you are looking for are defined in the default list here https://github.com/apache/incubator-druid/blob/0.13.0-incubating/extensions-contrib/statsd-emitter/src/main/resources/defaultMetricDimensions.json otherwise you need to define your metrics mapping and configure that.

I am actually using mixed versions i.e. 0.12.3 for statsd and 0.13.0 for druid. I was unable to find 0.13.0 version for statsd. Here is where I was looking
https://mvnrepository.com/artifact/org.apache.druid.extensions?p=2

Is it located in a different place?

Yes, check out https://mvnrepository.com/artifact/io.druid.extensions?p=2

Thank you. This helped. I was not using the incompatible version.

Correction, I was using incompatible version of statsd. i.e. using 0.12.3 statsd with 0.13.0 druid.