I am having issues with message being dropped when the kill the middle managers & restart them:
com.metamx.tranquility.tranquilizer.MessageDroppedException: Message dropped
at com.twitter.finagle.NoStacktrace(Unknown Source)
I don’t see the rest of the stacktrace but only the message above.
I verified that the messages are within the window period (PT9M)
Middle Managers are all running fine after the restart
However, I do see that the messages start processing again from the beginning of next hour.
Has anyone encountered issues like this ? Any workaround?
Middle Manager config:
I have about 40 middle managers (with 3 workers each)
Each with 64Gi memory (8 cores of CPU)
Ingest rate to achieve is 200-250 million events/minute