Counters/logs of tranquility when it drops messages

I am using tranquility to ingest messages in Druid.
I have observed that sometime tranquility drops certain messages which may be because of messages belonging to timeinterval outside of window period or may be because of some other reason.

But i don’t see any logs when tranquility drops these messages,is it possible to enable logs for such events??

Also is it possible to know which messages were dropped like it may print those messages in logs or somewhere?

Thanks

Rohit

These metrics are included on the Druid side. If you enable druid.emitter=logging and look at the task logs of the realtime tasks, you should see metrics displayed about the number of events ingested, dropped, with exceptions, etc.

BTW, this proposal may be interesting to you:

https://groups.google.com/forum/#!searchin/druid-development/gian$20windowperiod/druid-development/kHgHTgqKFlQ/fXvtsNxWzlMJ

These metrics would be present on druid side only if messages are rejected by druid but as far as i understand,
tranquility set the rejection policy as none and does all the rejection of events (falling outside the window period) on the client side.Is that correct?

Hi Rohit,
tranquility uses serverTime rejectionPolicy instead of none.

Sorry, for the false info, just verified tranquility uses none rejectionPolicy.
@Gian will be able to give more pointers here.

This information is important for us to take druid into production as otherwise we would be in complete dark regarding the messages being dropped.
Any help would be appreciated.

Hi Rohit, I’ve pinged Gian about your question.

Hi Rohit,

You can get the number of dropped messages for a specific call to “propagate” by looking at the value of the resolved future from “propagate”. It will be the number of messages actually sent through to Druid, so if it’s less than the number of messages you passed to “propagate”, then you can figure out how many messages were dropped by subtracting.

If you want to see the messages that were dropped, the simplest way is just to log out the messages from calls to “propagate” where its return value indicates that it dropped some messages. This will also potentially log out some messages that were not dropped, however, since it’s possible for tranquility to drop part of a batch while sending the rest. If you want to only see the dropped messages, you can provide a beamDecorateFn to the DruidBeams builder. This wraps the innermost beam in the beam stack (see the ASCII art on ClusteredBeam.scala) and so it should be all-or-nothing in terms of whether the batch of messages it sees made it through or not.