Send droppedCount from Tranquility to Graphite?

Hello all,

We are using Druid and Tranquility Kafka to ingest data. Window time is an issue for us, and we want to monitor how often and how many events are dropped due to this. With real-time nodes, there is a metric for “ingest/events/thrownAway”, but since Tranquility drops out-of-window messages on the client side, this will always be 0 when using Tranquility Kafka. However, the Tranquility logs contain the necessary information, e.g:

2016-09-28 11:42:20,284 INFO c.m.tranquility.kafka.KafkaConsumer: Flushed {FOO={receivedCount=51, sentCount=51, droppedCount=0, unparseableCount=0}, BAR={receivedCount=50, sentCount=50, droppedCount=0, unparseableCount=0}} pending messages in 0ms and committed offsets in 41ms

These metrics are exactly what we need, but we want to make them available in Graphite. So my questions is: does Tranquility expose these metrics in a way that is programmatically accessible, such as JMX or an API? If not, what would be the preferred way to go about adding this feature to Tranquility?

If you would use the latest version of tranquility i.e tranquilizer then it has a mechanism to receive callback for each event sent to druid.
The callback has the status of the event hence you can count the dropped events easily.

Please refer to https://github.com/druid-io/tranquility/blob/master/core/src/test/java/com/metamx/tranquility/example/JavaExample.java

I don’t remember the syntax of older tranquility but there also it used to return the “Future” as response and finding the difference between the number of events sent and number of future received you can find the total events that were dropped.

Hey John,

These aren’t exposed through JMX. When you’re using the Tranquility Core API directly, you can register a callback like Rohit pointed out. But when you’re using Tranquility Server or Kafka, that’s not an option. IMO the best way to add that feature would be to add an option for Tranquility to emit messages using an Emitter like Druid does. It could use the same emitter package, so you’d have options like “logging”, “http”, or any custom emitter. Ideally this would be set up through runtime properties, the same as Druid does it.

Btw, if you’re using Tranquility Kafka, you could also give the new Kafka indexing service a try and see if that meets your needs: http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html. It doesn’t require a separate service, doesn’t drop messages, it offers exactly-once guarantees, and it emits metrics through Druid’s normal metrics emitters.

Hi Gian,

Thanks for the reply. I suspected that this might be the case. If I decided to solve by adding emitters to Tranquility I will submit a PR when it’s done.

I’m keeping a close eye on the Kafka indexing service, but I want to give it some more time to mature before committing to it in our production environment. Also, we are big users of topic patterns which AFAIK are not implemented in the indexing service. But the long-term goal is for sure to transition to the indexing service.

hi Gian, how can I configure to use http emitter in trqnauility?

在 2016年9月30日星期五 UTC+8上午3:04:53,Gian Merlino写道:

I don’t believe tranq has an http emitter and it needs to be added to the code base

Yes, that’s true.