Druid now support 3 types of metrics emitter(noop, logging, http), but we want to customize our own emitter, is there an easy way to do that?
Be more specific, we want to send metrics to OpenTSDB, and OpenTSDB support feeding data with http request, but it seems the structure of druid’s http request is different with OpenTSDB’s.
Any suggestion to do this?
Have you considered emitting metrics about druid to druid?
Druid metrics contain a lot of information about what exactly is occurring at any given time. If you emit them to OpenTSDB, you may have to drop some of this data. Alternatively, as Charles mentioned, you can just emit this metrics data to another Druid cluster.
were you able to figure out how to emit druid metrics into opentsdb endpoint?
We currently have a big setup of opentsdb for monitoring and not looking to spin up another druid cluster just to store metrics.
Please let me know.
Not sure if Kurt will have something more to add, but in general, you can add emitters to Druid as extensions. You could check out extensions-contrib/graphite-emitter/ in the Druid repo for an example of how to write an extension.
The main thing to be aware of if you implement an emitter is that “emit” should not block or throw exceptions, as it gets called in-line with tasks like query processing. Because we really want to avoid blocking other Druid work, and we also don’t want unbounded memory use, emitters usually drop events that are generated while their send queue is full.
Actually, looking at the graphite emitter, IMO it would be better if it dropped messages on queue full immediately rather than blocking up to 1 sec before dropping. Filed this issue for that: https://github.com/druid-io/druid/issues/2868
We did have druid metrics stored in our opentsdb, by setting up a simple http server which receives all the druid metrics and convert to opentsdb’s format then send to opentsdb with http API.
Is this something you’d care to share?