We just setup a new system with some sample data injection. For some reason, tranquility keeps dropping every single message. We looked at the middleManager and overlord logs as tranquility logs show dropped message counts, but we didn’t see any messages at all.
We tried to clear /var/zookeeper to reset Zookeeper, we tried to clear the beams under zookeeper and we tried to clean the druid_tasks tables to reset things. Nothing worked so far.
Where can we start troubleshooting this?
tranq.txt (48.2 KB)
Looking at the indexing logs on middleManager, I think I found an issue. I keep getting these Error messages:
“Error in custom provider, com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.”
I attached a sample file. Has anyone run into a similar issue?
index_realtime_log.log (66.2 KB)
The root cause of the issue was in the index logs. I had to set the region as instructed here: http://druid.io/docs/latest/development/extensions-core/s3.html
Next issue was to set the -XX:MaxDirectMemorySize to the right value for peons. Once I did that, the issue resolved itself.