Repeating "Turns out we decided not to actually make beams for identifier" message from Tranquility

We just setup a new system with some sample data injection. For some reason, tranquility keeps dropping every single message. We looked at the middleManager and overlord logs as tranquility logs show dropped message counts, but we didn’t see any messages at all.

We tried to clear /var/zookeeper to reset Zookeeper, we tried to clear the beams under zookeeper and we tried to clean the druid_tasks tables to reset things. Nothing worked so far.

Where can we start troubleshooting this?

Thanks

tranq.txt (48.2 KB)

Looking at the indexing logs on middleManager, I think I found an issue. I keep getting these Error messages:

“Error in custom provider, com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.”

I attached a sample file. Has anyone run into a similar issue?

Thanks

index_realtime_log.log (66.2 KB)

The root cause of the issue was in the index logs. I had to set the region as instructed here: http://druid.io/docs/latest/development/extensions-core/s3.html

Next issue was to set the -XX:MaxDirectMemorySize to the right value for peons. Once I did that, the issue resolved itself.