Messages dropped by druid (v0.9.0)

Apache Druid Q&A Google Groups
#1

Hi Folks,

I am having issues with message being dropped when the kill the middle managers & restart them:

com.metamx.tranquility.tranquilizer.MessageDroppedException: Message dropped

at com.twitter.finagle.NoStacktrace(Unknown Source)

I don’t see the rest of the stacktrace but only the message above.

  1. I verified that the messages are within the window period (PT9M)

  2. Middle Managers are all running fine after the restart

However, I do see that the messages start processing again from the beginning of next hour.

Has anyone encountered issues like this ? Any workaround?

Middle Manager config:

  1. I have about 40 middle managers (with 3 workers each)

  2. Each with 64Gi memory (8 cores of CPU)

Ingest rate to achieve is 200-250 million events/minute