How to debug when tranquility receives data, but fails to send it to druid

I have indexing service started via

java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/overlord:lib/*: io.druid.cli.Main server overlord


I have the following server.json file

“dataSources” : [
“spec” : {
“dataSchema” : {
“dataSource” : “wikipedia”,
“parser” : {
“type” : “string”,
“parseSpec” : {
“format” : “json”,
“timestampSpec” : {
“column” : “timestamp”,
“format” : “auto”
“dimensionsSpec” : {
“dimensions”: [“page”,“language”,“user”,“unpatrolled”,“newPage”,“robot”,“anonymous”,“namespace”,“continent”,“country”,“region”,“city”],
“dimensionExclusions” : ,
“spatialDimensions” :
“metricsSpec” : [{
“type” : “count”,
“name” : “count”
}, {
“type” : “doubleSum”,
“name” : “added”,
“fieldName” : “added”
}, {
“type” : “doubleSum”,
“name” : “deleted”,
“fieldName” : “deleted”
}, {
“type” : “doubleSum”,
“name” : “delta”,
“fieldName” : “delta”
“granularitySpec” : {
“type” : “uniform”,
“segmentGranularity” : “DAY”,
“queryGranularity” : “NONE”
“tuningConfig” : {
“windowPeriod” : “PT10M”,
“type” : “realtime”,
“intermediatePersistPeriod” : “PT10M”,
“maxRowsInMemory” : “100000”
“properties” : {
“task.partitions” : “1”,
“task.replicants” : “1”
“properties” : {
“zookeeper.connect” : “localhost”,
“http.port” : “8200”,
“http.threads” : “8”


Then, I start the server using

bin/tranquility server -configFile conf/server.json


I perform post to, with content-type equals application/json

{“timestamp”: “2013-08-10T01:02:33Z”, “page”: “Selamat Pagi”, “language” : “en”, “user” : “catty”, “unpatrolled” : “true”, “newPage” : “true”, “robot”: “false”, “anonymous”: “false”, “namespace”:“article”, “continent”:“North America”, “country”:“United States”, “region”:“Bay Area”, “city”:“San Francisco”, “added”: 57, “deleted”: 200, “delta”: -143}


I get the the following respond



It seems that tranquility has received our data, but failed to send it to druid!

I try to run

curl -X POST ‘http://localhost:8084/druid/v2/?pretty’ -H ‘content-type: application/json’ -d @select.json


, but doesn’t get the output I inserted via tranquility.

Any idea why? How can I debug on the failure? Thanks.

Hey Yan,

It’s likely that the reason the data isn’t being accepted by Druid is that it is old data and falls outside of the window period. Try modifying the timestamp to be the current time in ISO8601 format. For more information, you can check out this documentation:

Hi David,

Thanks. Only at the end of the day, I realize Druid + tranquility don’t accept old data.

But, all the while when I was testing using Druid + Kafka console, it was able to accept old data (year 2013). May I know why it is so? Is it because Kafka console is dealing with Druid’s realtime node, and tranquility is dealing with Druid’s indexing node?

I re-test again. This time, I get the following outcome from tranquility

2016-03-15 02:04:21,296 [ClusteredBeam-ZkFuturePool-48ba4ff4-ec80-46a0-9fbf-6ce0058fb4fc] INFO c.m.tranquility.beam.ClusteredBeam - Merged beam already created for identifier[druid:overlord/wikipedia] timestamp[2016-03-15T00:00:00.000Z], with sufficient partitions (target = 1, actual = 1)
2016-03-15 02:04:21,300 [ClusteredBeam-ZkFuturePool-48ba4ff4-ec80-46a0-9fbf-6ce0058fb4fc] INFO c.m.tranquility.beam.ClusteredBeam - Adding beams for identifier[druid:overlord/wikipedia] timestamp[2016-03-15T00:00:00.000Z]: List(Map(interval -> 2016-03-15T00:00:00.000Z/2016-03-16T00:00:00.000Z, partition -> 0, tasks -> List(Map(id -> index_realtime_wikipedia_2016-03-15T00:00:00.000Z_0_0, firehoseId -> wikipedia-15-0000-0000)), timestamp -> 2016-03-15T00:00:00.000Z))
Mar 15, 2016 2:04:21 AM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp
INFO: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135
2016-03-15 02:04:21,472 [ClusteredBeam-ZkFuturePool-48ba4ff4-ec80-46a0-9fbf-6ce0058fb4fc] INFO - Updating instances for service[firehose:druid:overlord:wikipedia-15-0000-0000] to Set()
2016-03-15 02:04:21,529 [ClusteredBeam-ZkFuturePool-48ba4ff4-ec80-46a0-9fbf-6ce0058fb4fc] INFO c.m.t.finagle.FinagleRegistry - Created client for service: firehose:druid:overlord:wikipedia-15-0000-0000
2016-03-15 02:04:21,695 [ClusteredBeam-ZkFuturePool-48ba4ff4-ec80-46a0-9fbf-6ce0058fb4fc] INFO - Updating instances for service[druid:overlord] to Set(ServiceInstance{name=‘druid:overlord’, id=‘f7250bd2-27d0-422b-86ae-79edc7a13166’, address=‘ip-172-30-1-252.ap-southeast-1.compute.internal’, port=8090, sslPort=null, payload=null, registrationTimeUTC=1458007440232, serviceType=DYNAMIC, uriSpec=null})
2016-03-15 02:04:21,711 [ClusteredBeam-ZkFuturePool-48ba4ff4-ec80-46a0-9fbf-6ce0058fb4fc] INFO c.m.t.finagle.FinagleRegistry - Created client for service: druid:overlord
2016-03-15 02:04:21,924 [finagle/netty3-1] INFO c.m.tranquility.druid.TaskClient - Task index_realtime_wikipedia_2016-03-15T00:00:00.000Z_0_0 status changed from TaskRunning -> TaskNotFound

<… Hang for 1+ minutes …>

2016-03-15 02:05:53,429 [Hashed wheel timer #1] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to propagate events: druid:overlord/wikipedia
“eventCount” : 1,
“timestamp” : “2016-03-15T00:00:00.000Z”,
“beams” : “MergingPartitioningBeam(DruidBeam(interval = 2016-03-15T00:00:00.000Z/2016-03-16T00:00:00.000Z, partition = 0, tasks = [index_realtime_wikipedia_2016-03-15T00:00:00.000Z_0_0/wikipedia-15-0000-0000]))”
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for firehose:druid:overlord:wikipedia-15-0000-0000, Dtab.base=, Dtab.local=
at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-03-15 02:05:53,439 [Hashed wheel timer #1] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2016-03-15T02:05:53.435Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to propagate events: druid:overlord/wikipedia”,“data”:{“exceptionType”:“com.twitter.finagle.NoBrokersAvailableException”,“exceptionStackTrace”:“com.twitter.finagle.NoBrokersAvailableException: No hosts are available for firehose:druid:overlord:wikipedia-15-0000-0000, Dtab.base=, Dtab.local=\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n”,“timestamp”:“2016-03-15T00:00:00.000Z”,“beams”:“MergingPartitioningBeam(DruidBeam(interval = 2016-03-15T00:00:00.000Z/2016-03-16T00:00:00.000Z, partition = 0, tasks = [index_realtime_wikipedia_2016-03-15T00:00:00.000Z_0_0/wikipedia-15-0000-0000]))”,“eventCount”:1,“exceptionMessage”:“No hosts are available for firehose:druid:overlord:wikipedia-15-0000-0000, Dtab.base=, Dtab.local=”}}]


I had confirmed overlord is started.

java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/overlord:lib/*: io.druid.cli.Main server overlord


Any idea what goes wrong in between? I already have “coordinator”, “historical”, “broker”, “realtime”, “tranquility”, “kafka server”, “kafka console”, “indexing”, “middle manager” running.

I just realize you are working on Imply, which is a commercial product built around druid. I think I might give a try on Imply, as it seems it is much easier to setup than Druid. May I know is there any restriction to use Imply commercially?

Thank you very much!

Hey Choek,

The error you’re seeing is most likely due to Druid not being able to start up new indexing tasks. Perhaps you’re out of capacity or perhaps things aren’t configured right.

To make your life easier I would suggest taking at look either at the new Druid quickstart (for the not-yet-released 0.9.0, at, or the Imply quickstart that it is based on (, uses the current stable Druid 0.8.3).

The Imply distribution is 100% free software (Apache 2 license) and there is no restriction on using it commercially. We do offer paid support.

Hi Gian,

Thanks for recommending Imply. We manage to make a successful “load streaming data”, by following step-by-step guide in Imply.

We will now fully focus on evaluating Imply, due to its ease of use.