Tranquility error

We’re using Druid 0.9.0 and from time to time we would see this error.

T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 11, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0/sparrow-firehose-web-012-0011-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_1/sparrow-firehose-web-012-0011-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 12, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_0/sparrow-firehose-web-012-0012-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_1/sparrow-firehose-web-012-0012-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 13, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_0/sparrow-firehose-web-012-0013-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_1/sparrow-firehose-web-012-0013-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 14, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_0/sparrow-firehose-web-012-0014-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_1/sparrow-firehose-web-012-0014-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 15, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_0/sparrow-firehose-web-012-0015-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_1/sparrow-firehose-web-012-0015-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 16, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_0/sparrow-firehose-web-012-0016-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_1/sparrow-firehose-web-012-0016-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 17, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_0/sparrow-firehose-web-012-0017-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_1/sparrow-firehose-web-012-0017-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 18, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_0/sparrow-firehose-web-012-0018-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_1/sparrow-firehose-web-012-0018-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 19, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_0/sparrow-firehose-web-012-0019-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_1/sparrow-firehose-web-012-0019-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 20, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_0/sparrow-firehose-web-012-0020-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_1/sparrow-firehose-web-012-0020-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 21, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_0/sparrow-firehose-web-012-0021-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_1/sparrow-firehose-web-012-0021-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 22, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_0/sparrow-firehose-web-012-0022-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_1/sparrow-firehose-web-012-0022-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 23, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_0/sparrow-firehose-web-012-0023-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_1/sparrow-firehose-web-012-0023-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 24, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_0/sparrow-firehose-web-012-0024-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_1/sparrow-firehose-web-012-0024-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 25, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_0/sparrow-firehose-web-012-0025-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_1/sparrow-firehose-web-012-0025-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 26, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_0/sparrow-firehose-web-012-0026-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_1/sparrow-firehose-web-012-0026-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 27, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_0/sparrow-firehose-web-012-0027-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_1/sparrow-firehose-web-012-0027-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 28, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_0/sparrow-firehose-web-012-0028-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_1/sparrow-firehose-web-012-0028-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 29, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_0/sparrow-firehose-web-012-0029-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_1/sparrow-firehose-web-012-0029-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 30, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_0/sparrow-firehose-web-012-0030-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_1/sparrow-firehose-web-012-0030-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 31, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_0/sparrow-firehose-web-012-0031-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_1/sparrow-firehose-web-012-0031-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 32, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_0/sparrow-firehose-web-012-0032-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_1/sparrow-firehose-web-012-0032-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 33, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_0/sparrow-firehose-web-012-0033-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_1/sparrow-firehose-web-012-0033-0001]))","eventCount":112,"exceptionMessage":"exceeded 1.minutes+30.seconds to disco!firehose:druid:overlord:sparrow-firehose-web-012-0004-0000 while waiting for a response for the request, including retries (if applicable)"}}]
2017-09-01 13:08:51.757 c.m.t.b.ClusteredBeam [WARN] Emitting alert: [anomaly] Failed to propagate events: druid:overlord/sparrow-firehose-web
{
  "eventCount" : 80,
  "timestamp" : "2017-09-01T12:00:00.000Z",
  "beams" : "MergingPartitioningBeam(DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 0, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_0/sparrow-firehose-web-012-0000-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_1/sparrow-firehose-web-012-0000-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 1, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_0/sparrow-firehose-web-012-0001-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_1/sparrow-firehose-web-012-0001-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 2, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_0/sparrow-firehose-web-012-0002-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_1/sparrow-firehose-web-012-0002-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 3, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_0/sparrow-firehose-web-012-0003-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_1/sparrow-firehose-web-012-0003-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 4, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_0/sparrow-firehose-web-012-0004-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_1/sparrow-firehose-web-012-0004-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 5, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_0/sparrow-firehose-web-012-0005-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_1/sparrow-firehose-web-012-0005-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 6, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_0/sparrow-firehose-web-012-0006-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_1/sparrow-firehose-web-012-0006-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 7, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_0/sparrow-firehose-web-012-0007-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_1/sparrow-firehose-web-012-0007-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 8, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_0/sparrow-firehose-web-012-0008-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_1/sparrow-firehose-web-012-0008-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 9, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_0/sparrow-firehose-web-012-0009-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_1/sparrow-firehose-web-012-0009-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 10, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_0/sparrow-firehose-web-012-0010-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_1/sparrow-firehose-web-012-0010-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 11, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0/sparrow-firehose-web-012-0011-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_1/sparrow-firehose-web-012-0011-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 12, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_0/sparrow-firehose-web-012-0012-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_1/sparrow-firehose-web-012-0012-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 13, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_0/sparrow-firehose-web-012-0013-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_1/sparrow-firehose-web-012-0013-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 14, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_0/sparrow-firehose-web-012-0014-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_1/sparrow-firehose-web-012-0014-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 15, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_0/sparrow-firehose-web-012-0015-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_1/sparrow-firehose-web-012-0015-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 16, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_0/sparrow-firehose-web-012-0016-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_1/sparrow-firehose-web-012-0016-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 17, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_0/sparrow-firehose-web-012-0017-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_1/sparrow-firehose-web-012-0017-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 18, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_0/sparrow-firehose-web-012-0018-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_1/sparrow-firehose-web-012-0018-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 19, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_0/sparrow-firehose-web-012-0019-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_1/sparrow-firehose-web-012-0019-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 20, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_0/sparrow-firehose-web-012-0020-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_1/sparrow-firehose-web-012-0020-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 21, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_0/sparrow-firehose-web-012-0021-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_1/sparrow-firehose-web-012-0021-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 22, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_0/sparrow-firehose-web-012-0022-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_1/sparrow-firehose-web-012-0022-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 23, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_0/sparrow-firehose-web-012-0023-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_1/sparrow-firehose-web-012-0023-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 24, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_0/sparrow-firehose-web-012-0024-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_1/sparrow-firehose-web-012-0024-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 25, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_0/sparrow-firehose-web-012-0025-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_1/sparrow-firehose-web-012-0025-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 26, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_0/sparrow-firehose-web-012-0026-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_1/sparrow-firehose-web-012-0026-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 27, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_0/sparrow-firehose-web-012-0027-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_1/sparrow-firehose-web-012-0027-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 28, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_0/sparrow-firehose-web-012-0028-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_1/sparrow-firehose-web-012-0028-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 29, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_0/sparrow-firehose-web-012-0029-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_1/sparrow-firehose-web-012-0029-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 30, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_0/sparrow-firehose-web-012-0030-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_1/sparrow-firehose-web-012-0030-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 31, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_0/sparrow-firehose-web-012-0031-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_1/sparrow-firehose-web-012-0031-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 32, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_0/sparrow-firehose-web-012-0032-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_1/sparrow-firehose-web-012-0032-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 33, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_0/sparrow-firehose-web-012-0033-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_1/sparrow-firehose-web-012-0033-0001]))"
}
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!firehose:druid:overlord:sparrow-firehose-web-012-0004-0000 while waiting for a response for the request, including retries (if applicable)
        at com.twitter.finagle.NoStacktrace(Unknown Source) ~[?:?]
2017-09-01 13:08:51.761 c.m.e.c.LoggingEmitter [INFO] Event [{"feed":"alerts","timestamp":"2017-09-01T13:08:51.760Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to propagate events: druid:overlord/sparrow-firehose-web","data":{"exceptionType":"com.twitter.finagle.GlobalRequestTimeoutException","exceptionStackTrace":"com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!firehose:druid:overlord:sparrow-firehose-web-012-0004-0000 while waiting for a response for the request, including retries (if applicable)\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n","timestamp":"2017-09-01T12:00:00.000Z","beams":"MergingPartitioningBeam(DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 0, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_0/sparrow-firehose-web-012-0000-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_1/sparrow-firehose-web-012-0000-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 1, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_0/sparrow-firehose-web-012-0001-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_1/sparrow-firehose-web-012-0001-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 2, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_0/sparrow-firehose-web-012-0002-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_1/sparrow-firehose-web-012-0002-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 3, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_0/sparrow-firehose-web-012-0003-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_1/sparrow-firehose-web-012-0003-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 4, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_0/sparrow-firehose-web-012-0004-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_1/sparrow-firehose-web-012-0004-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 5, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_0/sparrow-firehose-web-012-0005-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_1/sparrow-firehose-web-012-0005-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 6, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_0/sparrow-firehose-web-012-0006-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_1/sparrow-firehose-web-012-0006-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 7, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_0/sparrow-firehose-web-012-0007-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_1/sparrow-firehose-web-012-0007-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 8, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_0/sparrow-firehose-web-012-0008-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_1/sparrow-firehose-web-012-0008-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 9, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_0/sparrow-firehose-web-012-0009-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_1/sparrow-firehose-web-012-0009-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 10, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_0/sparrow-firehose-web-012-0010-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_1/sparrow-firehose-web-012-0010-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 11, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0/sparrow-firehose-web-012-0011-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_1/sparrow-firehose-web-012-0011-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 12, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_0/sparrow-firehose-web-012-0012-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_1/sparrow-firehose-web-012-0012-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 13, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_0/sparrow-firehose-web-012-0013-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_1/sparrow-firehose-web-012-0013-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 14, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_0/sparrow-firehose-web-012-0014-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_1/sparrow-firehose-web-012-0014-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 15, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_0/sparrow-firehose-web-012-0015-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_1/sparrow-firehose-web-012-0015-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 16, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_0/sparrow-firehose-web-012-0016-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_1/sparrow-firehose-web-012-0016-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 17, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_0/sparrow-firehose-web-012-0017-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_1/sparrow-firehose-web-012-0017-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 18, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_0/sparrow-firehose-web-012-0018-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_1/sparrow-firehose-web-012-0018-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 19, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_0/sparrow-firehose-web-012-0019-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_1/sparrow-firehose-web-012-0019-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 20, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_0/sparrow-firehose-web-012-0020-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_1/sparrow-firehose-web-012-0020-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 21, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_0/sparrow-firehose-web-012-0021-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_1/sparrow-firehose-web-012-0021-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 22, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_0/sparrow-firehose-web-012-0022-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_1/sparrow-firehose-web-012-0022-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 23, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_0/sparrow-firehose-web-012-0023-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_1/sparrow-firehose-web-012-0023-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 24, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_0/sparrow-firehose-web-012-0024-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_1/sparrow-firehose-web-012-0024-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 25, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_0/sparrow-firehose-web-012-0025-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_1/sparrow-firehose-web-012-0025-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 26, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_0/sparrow-firehose-web-012-0026-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_1/sparrow-firehose-web-012-0026-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 27, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_0/sparrow-firehose-web-012-0027-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_1/sparrow-firehose-web-012-0027-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 28, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_0/sparrow-firehose-web-012-0028-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_1/sparrow-firehose-web-012-0028-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 29, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_0/sparrow-firehose-web-012-0029-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_1/sparrow-firehose-web-012-0029-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 30, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_0/sparrow-firehose-web-012-0030-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_1/sparrow-firehose-web-012-0030-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 31, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_0/sparrow-firehose-web-012-0031-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_1/sparrow-firehose-web-012-0031-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 32, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_0/sparrow-firehose-web-012-0032-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_1/sparrow-firehose-web-012-0032-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 33, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_0/sparrow-firehose-web-012-0033-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_1/sparrow-firehose-web-012-0033-0001]))","eventCount":80,"exceptionMessage":"exceeded 1.minutes+30.seconds to disco!firehose:druid:overlord:sparrow-firehose-web-012-0004-0000 while waiting for a response for the request, including retries (if applicable)"}}]
2017-09-01 13:08:51.815 c.m.t.b.ClusteredBeam [WARN] Emitting alert: [anomaly] Failed to propagate events: druid:overlord/sparrow-firehose-web
{
  "eventCount" : 72,
  "timestamp" : "2017-09-01T12:00:00.000Z",
  "beams" : "MergingPartitioningBeam(DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 0, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_0/sparrow-firehose-web-012-0000-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_1/sparrow-firehose-web-012-0000-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 1, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_0/sparrow-firehose-web-012-0001-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_1/sparrow-firehose-web-012-0001-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 2, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_0/sparrow-firehose-web-012-0002-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_1/sparrow-firehose-web-012-0002-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 3, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_0/sparrow-firehose-web-012-0003-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_1/sparrow-firehose-web-012-0003-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 4, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_0/sparrow-firehose-web-012-0004-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_1/sparrow-firehose-web-012-0004-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 5, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_0/sparrow-firehose-web-012-0005-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_1/sparrow-firehose-web-012-0005-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 6, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_0/sparrow-firehose-web-012-0006-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_1/sparrow-firehose-web-012-0006-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 7, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_0/sparrow-firehose-web-012-0007-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_1/sparrow-firehose-web-012-0007-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 8, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_0/sparrow-firehose-web-012-0008-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_1/sparrow-firehose-web-012-0008-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 9, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_0/sparrow-firehose-web-012-0009-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_1/sparrow-firehose-web-012-0009-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 10, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_0/sparrow-firehose-web-012-0010-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_1/sparrow-firehose-web-012-0010-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 11, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0/sparrow-firehose-web-012-0011-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_1/sparrow-firehose-web-012-0011-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 12, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_0/sparrow-firehose-web-012-0012-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_1/sparrow-firehose-web-012-0012-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 13, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_0/sparrow-firehose-web-012-0013-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_1/sparrow-firehose-web-012-0013-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 14, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_0/sparrow-firehose-web-012-0014-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_1/sparrow-firehose-web-012-0014-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 15, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_0/sparrow-firehose-web-012-0015-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_1/sparrow-firehose-web-012-0015-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 16, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_0/sparrow-firehose-web-012-0016-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_1/sparrow-firehose-web-012-0016-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 17, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_0/sparrow-firehose-web-012-0017-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_1/sparrow-firehose-web-012-0017-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 18, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_0/sparrow-firehose-web-012-0018-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_1/sparrow-firehose-web-012-0018-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 19, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_0/sparrow-firehose-web-012-0019-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_1/sparrow-firehose-web-012-0019-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 20, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_0/sparrow-firehose-web-012-0020-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_1/sparrow-firehose-web-012-0020-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 21, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_0/sparrow-firehose-web-012-0021-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_1/sparrow-firehose-web-012-0021-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 22, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_0/sparrow-firehose-web-012-0022-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_1/sparrow-firehose-web-012-0022-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 23, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_0/sparrow-firehose-web-012-0023-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_1/sparrow-firehose-web-012-0023-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 24, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_0/sparrow-firehose-web-012-0024-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_1/sparrow-firehose-web-012-0024-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 25, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_0/sparrow-firehose-web-012-0025-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_1/sparrow-firehose-web-012-0025-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 26, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_0/sparrow-firehose-web-012-0026-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_1/sparrow-firehose-web-012-0026-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 27, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_0/sparrow-firehose-web-012-0027-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_1/sparrow-firehose-web-012-0027-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 28, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_0/sparrow-firehose-web-012-0028-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_1/sparrow-firehose-web-012-0028-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 29, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_0/sparrow-firehose-web-012-0029-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_1/sparrow-firehose-web-012-0029-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 30, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_0/sparrow-firehose-web-012-0030-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_1/sparrow-firehose-web-012-0030-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 31, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_0/sparrow-firehose-web-012-0031-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_1/sparrow-firehose-web-012-0031-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 32, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_0/sparrow-firehose-web-012-0032-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_1/sparrow-firehose-web-012-0032-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 33, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_0/sparrow-firehose-web-012-0033-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_1/sparrow-firehose-web-012-0033-0001]))"
}
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!firehose:druid:overlord:sparrow-firehose-web-012-0004-0000 while waiting for a response for the request, including retries (if applicable)
        at com.twitter.finagle.NoStacktrace(Unknown Source) ~[?:?]
2017-09-01 13:08:51.818 c.m.e.c.LoggingEmitter [INFO] Event [{"feed":"alerts","timestamp":"2017-09-01T13:08:51.817Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to propagate events: druid:overlord/sparrow-firehose-web","data":{"exceptionType":"com.twitter.finagle.GlobalRequestTimeoutException","exceptionStackTrace":"com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!firehose:druid:overlord:sparrow-firehose-web-012-0004-0000 while waiting for a response for the request, including retries (if applicable)\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n","timestamp":"2017-09-01T12:00:00.000Z","beams":"MergingPartitioningBeam(DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 0, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_0/sparrow-firehose-web-012-0000-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_0_1/sparrow-firehose-web-012-0000-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 1, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_0/sparrow-firehose-web-012-0001-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_1_1/sparrow-firehose-web-012-0001-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 2, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_0/sparrow-firehose-web-012-0002-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_2_1/sparrow-firehose-web-012-0002-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 3, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_0/sparrow-firehose-web-012-0003-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_3_1/sparrow-firehose-web-012-0003-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 4, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_0/sparrow-firehose-web-012-0004-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_4_1/sparrow-firehose-web-012-0004-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 5, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_0/sparrow-firehose-web-012-0005-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_5_1/sparrow-firehose-web-012-0005-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 6, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_0/sparrow-firehose-web-012-0006-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_6_1/sparrow-firehose-web-012-0006-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 7, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_0/sparrow-firehose-web-012-0007-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_7_1/sparrow-firehose-web-012-0007-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 8, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_0/sparrow-firehose-web-012-0008-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_8_1/sparrow-firehose-web-012-0008-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 9, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_0/sparrow-firehose-web-012-0009-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_9_1/sparrow-firehose-web-012-0009-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 10, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_0/sparrow-firehose-web-012-0010-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_10_1/sparrow-firehose-web-012-0010-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 11, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0/sparrow-firehose-web-012-0011-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_1/sparrow-firehose-web-012-0011-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 12, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_0/sparrow-firehose-web-012-0012-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_12_1/sparrow-firehose-web-012-0012-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 13, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_0/sparrow-firehose-web-012-0013-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_13_1/sparrow-firehose-web-012-0013-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 14, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_0/sparrow-firehose-web-012-0014-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_14_1/sparrow-firehose-web-012-0014-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 15, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_0/sparrow-firehose-web-012-0015-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_15_1/sparrow-firehose-web-012-0015-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 16, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_0/sparrow-firehose-web-012-0016-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_16_1/sparrow-firehose-web-012-0016-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 17, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_0/sparrow-firehose-web-012-0017-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_17_1/sparrow-firehose-web-012-0017-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 18, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_0/sparrow-firehose-web-012-0018-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_18_1/sparrow-firehose-web-012-0018-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 19, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_0/sparrow-firehose-web-012-0019-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_19_1/sparrow-firehose-web-012-0019-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 20, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_0/sparrow-firehose-web-012-0020-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_20_1/sparrow-firehose-web-012-0020-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 21, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_0/sparrow-firehose-web-012-0021-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_21_1/sparrow-firehose-web-012-0021-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 22, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_0/sparrow-firehose-web-012-0022-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_22_1/sparrow-firehose-web-012-0022-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 23, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_0/sparrow-firehose-web-012-0023-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_23_1/sparrow-firehose-web-012-0023-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 24, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_0/sparrow-firehose-web-012-0024-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_24_1/sparrow-firehose-web-012-0024-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 25, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_0/sparrow-firehose-web-012-0025-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_25_1/sparrow-firehose-web-012-0025-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 26, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_0/sparrow-firehose-web-012-0026-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_26_1/sparrow-firehose-web-012-0026-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 27, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_0/sparrow-firehose-web-012-0027-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_27_1/sparrow-firehose-web-012-0027-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 28, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_0/sparrow-firehose-web-012-0028-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_28_1/sparrow-firehose-web-012-0028-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 29, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_0/sparrow-firehose-web-012-0029-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_29_1/sparrow-firehose-web-012-0029-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 30, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_0/sparrow-firehose-web-012-0030-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_30_1/sparrow-firehose-web-012-0030-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 31, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_0/sparrow-firehose-web-012-0031-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_31_1/sparrow-firehose-web-012-0031-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 32, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_0/sparrow-firehose-web-012-0032-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_32_1/sparrow-firehose-web-012-0032-0001]), DruidBeam(interval = 2017-09-01T12:00:00.000Z/2017-09-01T13:00:00.000Z, partition = 33, tasks = [index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_0/sparrow-firehose-web-012-0033-0000; index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_33_1/sparrow-firehose-web-012-0033-0001]))","eventCount":72,"exceptionMessage":"exceeded 1.minutes+30.seconds to disco!firehose:druid:overlord:sparrow-firehose-web-012-0004-0000 while waiting for a response for the request, including retries (if applicable)"}}]

``

However, I check that the task index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0 is running in coordinator and here’s the status I get

{

  • status:

    {

    • id: “index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0”,

    • status: “RUNNING”,

    • duration: -1

    },

  • task: “index_realtime_sparrow-firehose-web_2017-09-01T12:00:00.000Z_11_0”

}

``

We also have 162 slots and only 67 tasks running. Not sure if we can mitigate this error? Or what causes this type of error?

It’s hard to tell from that log what’s going on. Are you sure the task is “really” running? Like, does it have a worker assigned and does it have a log? It’s possible that it’s in RUNNING state but hasn’t been assigned to a worker yet. Or that it has been assigned but something is unhealthy with it.

You could also consider upgrading Druid – there have been a number of fixes in the indexing service since 0.9.0.