Druid Tranquility creating two beams for a given time frame: Emitting alert: [anomaly]


We are trying to connect Tranquility for indexing service and the setup has been very similar before (was working perfectly).

We recently had to restart our services.

The following error keep appearing on the logs:

**2018-08-13 07:23:04,552 [Hashed wheel timer #1] WARN  c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to propagate events: druid:overlord/pixel-feed
  "eventCount" : 1,
  "timestamp" : "2018-08-13T07:00:00.000Z",
  "beams" : "MergingPartitioningBeam(DruidBeam(interval = 2018-08-13T07:00:00.000Z/2018-08-13T08:00:00.000Z, partition = 0, tasks = [index_realtime_pixel-feed_2018-08-13T07:00:00.000Z_0_0/pixel-feed-007-0000-0000; index_realtime_pixel-feed_2018-08-13T07:00:00.000Z_0_1/pixel-feed-007-0000-0001]))"
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:pixel-feed-007-0000-0001, Dtab.base=[], Dtab.local=[]
        at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2018-08-13 07:23:04,552 [Hashed wheel timer #1] INFO  c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2018-08-13T07:23:04.552Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to propagate events: druid:overlord/pixel-feed","data":{"exceptionType":"com.twitter.finagle.NoBrokersAvailableException","exceptionStackTrace":"com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:pixel-feed-007-0000-0001, Dtab.base=[], Dtab.local=[]\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n","timestamp":"2018-08-13T07:00:00.000Z","beams":"MergingPartitioningBeam(DruidBeam(interval = 2018-08-13T07:00:00.000Z/2018-08-13T08:00:00.000Z, partition = 0, tasks = [index_realtime_pixel-feed_2018-08-13T07:00:00.000Z_0_0/pixel-feed-007-0000-0000; index_realtime_pixel-feed_2018-08-13T07:00:00.000Z_0_1/pixel-feed-007-0000-0001]))","eventCount":1,"exceptionMessage":"No hosts are available for disco!firehose:druid:overlord:pixel-feed-007-0000-0001, Dtab.base=[], Dtab.local=[]"}}]**

Tranquility is able to connect to Kafka as well as Overlord
However, it has been creating more than one beam for a given event time frame. - it was not happening before.

We can also see in the Overlord Console that the extra beam is in PENDING STATE. (i.e.: **index_realtime_pixel-feed_2018-08-13T07:00:00.000Z_0_1**)

Attaching Kafka.json file.


kafka.json (19.2 KB)

Multiple tasks are created by tranquility when you have more than 1 shard or have replication more than 1.

Above exception seem to indicate that something went wrong with the task and tranquility is not able to send events to that task.

Check overlord console and verify that you have enough resources to run the tasks. If yes, then look further in the task logs for more details about the failure.

Thanks Nishant.
That worked.

Reducing the task.replicants as 1 helped.