Druid is stopping ingested tranquility data

I already setup the farm of Druid , Tranquility Server… so far I could success fetch data from Kinesis stream , then http post to Druid via Tranquility server and working as expected.

After all indexed I could query data back…

However, at this time I don’t see data ingested to druid any more (Messages from Kinesis are comming nicely, I checked all Druid logs and services ( overlord, middleManager, broker …) , they all up and running without any exception , all resources (cpu, memory …) looks stable.

There is only exception I could suspect from Tranquility Server log regarding overlord ( but overlord host is ok).

I have no idea what is going or insight

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: 2019-01-11 18:04:28,757 [Hashed wheel timer #1] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to propagate events: druid:overlord/wtb_kinesis

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: {

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: “eventCount” : 1,

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: “timestamp” : “2019-01-10T16:00:00.000Z”,

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: “beams” : “MergingPartitioningBeam(DruidBeam(interval = 2019-01-10T16:00:00.000Z/2019-01-10T17:00:00.000Z, partition = 0, tasks = [index_realtime_wtb_kinesis_2019-01-10T16:00:00.000Z_0_0/wtb_kinesis-016-0000-0000]))”

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: }

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:wtb_kinesis-016-0000-0000, Dtab.base=[], Dtab.local=[]

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]

Jan 11 18:04:28 ip-172-31-41-203.us-west-2.compute.internal DruidTranquilityService[11446]: 2019-01-11 18:04:28,763 [Hashed wheel timer #1] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2019-01-11T18:04:28.763Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to propagate events: druid:overlord/wtb_kinesis”,“data”:{“exceptionType”:“com.twitter.finagle.NoBrokersAvailableException”,“exceptionStackTrace”:“com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!firehose:druid:overlord:wtb_kinesis-016-0000-0000, Dtab.base=, Dtab.local=\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n”,“timestamp”:“2019-01-10T16:00:00.000Z”,“beams”:“MergingPartitioningBeam(DruidBeam(interval = 2019-01-10T16:00:00.000Z/2019-01-10T17:00:00.000Z, partition = 0, tasks = [index_realtime_wtb_kinesis_2019-01-10T16:00:00.000Z_0_0/wtb_kinesis-016-0000-0000]))”,“eventCount”:1,“exceptionMessage”:“No hosts are available for disco!firehose:druid:overlord:wtb_kinesis-016-0000-0000, Dtab.base=, Dtab.local=”}}]

Thanks,

Tom