What's the format of the JSON metrics and alerts being sent via HTTP POST metrics?

We send metrics with following settings.

druid.emitter=http
druid.emitter.logging.loggerClass=HttpPostEmitter

We receive them as a JSON with an array of objects. Most of them belong to the feed ‘metrics’, but some contain information about errors and, as far as I know, they belong to the feed ‘alerts’. We would like to report them to our Sentry but we are unsure about the scheme of those objects. I’ve seen some progress in the document: https://docs.google.com/spreadsheets/d/15XxGrGv2ggdt4SnCoIsckqUNJBZ9ludyhC9hNQa3l-M/edit#gid=0 but currently it describes only metrics. That would be great to see it documented what data may appear in each feed.

Thanks in advance.

Hi Michal,

I was hoping to rework Druid metrics with this pull request:

and document what the schema looks like, as well as what all the metrics are, and their expected values

If you are patient, you can wait for those changes to be merged.

In terms of your question around alerts, we do have two feeds, a metric feed for system level metrics, and an alert feed for more concerning exceptions. The alert events are designed to be schemaless, as in different alerts from different subsystems may have different dimensions.

All of the alerts share these same properties:

timestamp

service

host

class

exceptionType

exceptionMessage

exceptionStackTrace

What we do is emit the alerts to Kafka, and we have separate service that reads from Kafka and sends out emails about the alerts.

Let me know if you’d like more explanation.

– FJ

Hi Fangjin,

We’re still using 0.6.160 version of druid and we didn’t get exceptionType key in alerts at all. I looked at the source library and it seems to be creating it for over two years?

Are we missing something?

W dniu środa, 15 kwietnia 2015 19:38:38 UTC+2 użytkownik Fangjin Yang napisał:

Alert event will contain…

feed (= “alerts”)

timestamp

service

host

severity

description

data

You can see more details at https://github.com/metamx/emitter/blob/master/src/main/java/com/metamx/emitter/service/AlertEvent.java#L128

– Himanshu

We should probably create a documentation page on how we emit alerts.