What would cause this error in Tranquility?

We’re getting some error like this from Tranquility and it looks like we don’t get those events. I’m not sure how to debug this issue and what would cause this error.

We’re using Storm and tranquility with Kinesis. When I see this error it looks like there’s a connection issue with Zookeeper but I’m not sure if that’s the case because wouldn’t it fail for all of the events?

com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:test-sparrow-15-0004-0001, Dtab.base=[], Dtab.local=[]
	at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-06-17 15:01:26 c.m.e.c.LoggingEmitter [INFO] Event [{"feed":"alerts","timestamp":"2016-06-17T15:01:26.895Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to propagate events: druid:overlord/test-sparrow","data":{"exceptionType":"com.twitter.finagle.NoBrokersAvailableException","exceptionStackTrace":"com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:test-sparrow-15-0004-0001, Dtab.base=[], Dtab.local=[]\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n","timestamp":"2016-06-17T15:00:00.000Z","beams":"MergingPartitioningBeam(DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 0, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_0_0/test-sparrow-15-0000-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_0_1/test-sparrow-15-0000-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 1, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_1_0/test-sparrow-15-0001-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_1_1/test-sparrow-15-0001-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 2, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_2_0/test-sparrow-15-0002-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_2_1/test-sparrow-15-0002-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 3, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_3_0/test-sparrow-15-0003-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_3_1/test-sparrow-15-0003-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 4, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_4_0/test-sparrow-15-0004-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_4_1/test-sparrow-15-0004-0001]))","eventCount":2,"exceptionMessage":"No hosts are available for disco!firehose:druid:overlord:test-sparrow-15-0004-0001, Dtab.base=[], Dtab.local=[]"}}]
2016-06-17 15:01:29 c.m.t.b.ClusteredBeam [WARN] Emitting alert: [anomaly] Failed to propagate events: druid:overlord/test-sparrow
{
  "eventCount" : 25,
  "timestamp" : "2016-06-17T15:00:00.000Z",
  "beams" : "MergingPartitioningBeam(DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 0, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_0_0/test-sparrow-15-0000-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_0_1/test-sparrow-15-0000-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 1, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_1_0/test-sparrow-15-0001-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_1_1/test-sparrow-15-0001-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 2, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_2_0/test-sparrow-15-0002-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_2_1/test-sparrow-15-0002-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 3, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_3_0/test-sparrow-15-0003-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_3_1/test-sparrow-15-0003-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 4, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_4_0/test-sparrow-15-0004-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_4_1/test-sparrow-15-0004-0001]))"
}
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:test-sparrow-15-0004-0001, Dtab.base=[], Dtab.local=[]
	at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-06-17 15:01:29 c.m.e.c.LoggingEmitter [INFO] Event [{"feed":"alerts","timestamp":"2016-06-17T15:01:29.026Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to propagate events: druid:overlord/test-sparrow","data":{"exceptionType":"com.twitter.finagle.NoBrokersAvailableException","exceptionStackTrace":"com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:test-sparrow-15-0004-0001, Dtab.base=[], Dtab.local=[]\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n","timestamp":"2016-06-17T15:00:00.000Z","beams":"MergingPartitioningBeam(DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 0, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_0_0/test-sparrow-15-0000-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_0_1/test-sparrow-15-0000-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 1, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_1_0/test-sparrow-15-0001-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_1_1/test-sparrow-15-0001-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 2, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_2_0/test-sparrow-15-0002-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_2_1/test-sparrow-15-0002-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 3, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_3_0/test-sparrow-15-0003-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_3_1/test-sparrow-15-0003-0001]), DruidBeam(interval = 2016-06-17T15:00:00.000/2016-06-17T16:00:00.000, partition = 4, tasks = [index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_4_0/test-sparrow-15-0004-0000; index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_4_1/test-sparrow-15-0004-0001]))","eventCount":25,"exceptionMessage":"No hosts are available for disco!firehose:druid:overlord:test-sparrow-15-0004-0001, Dtab.base=[], Dtab.local=[]"}}]

Hey Noppanit,

This error usually happens when tasks can’t be found. In this case the task is “index_realtime_test-sparrow_2016-06-17T15:00:00.000Z_0_0”. Do you see that task running in your overlord console? (Maybe it’s stuck in pending or your workers are out of capacity?)