Posting Streaming data w/ Tranquility HTTP API faling

Hello,

I am currently trying to evaluate Druid.

I followed the tutorial for ingesting streaming data using http://druid-io.imply.io/docs/latest/tutorials/quickstart.html

With the following:

Druid 0.9.0 rc-3

Tranquility 0.7.4

When posting to Tranquility, I get a “Server error” message:

bin/generate-example-metrics | curl -XPOST -H’Content-Type: application/json’ --data-binary @- http://localhost:8200/v1/post/metrics

In the Tranquility console logs, I see the following:

2016-04-10 06:18:02,811 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/metrics] timestamp[2016-04-10T06:00:00.000Z] (target = 1, actual = 0)
2016-04-10 06:18:02,918 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): {
“type” : “index_realtime”,
“id” : “index_realtime_metrics_2016-04-10T06:00:00.000Z_0_0”,
“resource” : {
“availabilityGroup” : “metrics-06-0000”,
“requiredCapacity” : 1
},
“spec” : {
“dataSchema” : {
“dataSource” : “metrics”,
“parser” : {
“type” : “map”,
“parseSpec” : {
“format” : “json”,
“timestampSpec” : {
“column” : “timestamp”,
“format” : “auto”,
“missingValue” : null
},
“dimensionsSpec” : {
“dimensionExclusions” : [ “value”, “timestamp” ],
“spatialDimensions” :
}
}
},
“metricsSpec” : [ {
“type” : “count”,
“name” : “count”
}, {
“type” : “doubleSum”,
“name” : “value_sum”,
“fieldName” : “value”
}, {
“type” : “doubleMin”,
“name” : “value_min”,
“fieldName” : “value”
}, {
“type” : “doubleMax”,
“name” : “value_max”,
“fieldName” : “value”
} ],
“granularitySpec” : {
“type” : “uniform”,
“segmentGranularity” : “HOUR”,
“queryGranularity” : {
“type” : “none”
}
}
},
“ioConfig” : {
“type” : “realtime”,
“plumber” : null,
“firehose” : {
“type” : “clipped”,
“interval” : “2016-04-10T06:00:00.000Z/2016-04-10T07:00:00.000Z”,
“delegate” : {
“type” : “timed”,
“shutoffTime” : “2016-04-10T07:15:00.000Z”,
“delegate” : {
“type” : “receiver”,
“serviceName” : “firehose:druid:overlord:metrics-06-0000-0000”,
“bufferSize” : 100000
}
}
}
},
“tuningConfig” : {
“shardSpec” : {
“type” : “linear”,
“partitionNum” : 0
},
“rejectionPolicy” : {
“type” : “none”
},
“buildV9Directly” : false,
“maxPendingPersists” : 0,
“intermediatePersistPeriod” : “PT10M”,
“windowPeriod” : “PT10M”,
“type” : “realtime”,
“maxRowsInMemory” : “100000”
}
}
}
Apr 10, 2016 6:18:02 AM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp
INFO: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135
2016-04-10 06:18:03,183 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set(ServiceInstance{name=‘druid:overlord’, id=‘e83dcdb6-7352-47e7-a568-b39dbacf9008’, address=‘192.168.1.107’, port=8090, sslPort=null, payload=null, registrationTimeUTC=1460263365842, serviceType=DYNAMIC, uriSpec=null})
2016-04-10 06:18:03,269 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] INFO c.m.t.finagle.FinagleRegistry - Created client for service: disco!druid:overlord
2016-04-10 06:19:33,325 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/metrics
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!druid:overlord while waiting for a response for the request, including retries (if applicable)
at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-04-10 06:19:33,333 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/metrics
{ }
java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics] timestamp[2016-04-10T06:00:00.000Z]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!druid:overlord while waiting for a response for the request, including retries (if applicable)
at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-04-10 06:19:33,343 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2016-04-10T06:19:33.339Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to create merged beam: druid:overlord/metrics”,“data”:{“exceptionType”:“java.lang.IllegalStateException”,“exceptionMessage”:“Failed to save new beam for identifier[druid:overlord/metrics] timestamp[2016-04-10T06:00:00.000Z]”,“exceptionStackTrace”:“java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics] timestamp[2016-04-10T06:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!druid:overlord while waiting for a response for the request, including retries (if applicable)\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n”}}]
2016-04-10 06:19:33,344 [ClusteredBeam-ZkFuturePool-97b537c8-3182-4879-8931-3c974356e9f6] WARN c.m.t.tranquilizer.Tranquilizer - Failed to send 1 messages.
java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics] timestamp[2016-04-10T06:00:00.000Z]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!druid:overlord while waiting for a response for the request, including retries (if applicable)
at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]

``

I get the same exception when I try the quick start using the Imply release with the supervisor.

http://imply.io/docs/latest/quickstart

Hi,
Above exception means that tranquility is not able to locate druid overlord.

Do you have druid overlord running ?

Also, in case you are using custom service names, you need to make sure that service name of overlord (configured via druid.service property in druid) matches the “druid.selectors.indexing.serviceName” in the tranquility.

Hello Nishant,

Thank you for the reply.

The druid.service property of my overlord matches that configured in tranquility.

Here is my overlord configuration:

druid.service=druid/overlord
druid.port=8090

druid.indexer.queue.startDelay=PT30S

druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

``

Here is my Tranquility configuration:

{

“dataSources” : {

“metrics” : {

“spec” : {

“dataSchema” : {

“dataSource” : “metrics”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“timestampSpec” : {

“column” : “timestamp”,

“format” : “auto”

},

“dimensionsSpec” : {

“dimensions” : ,

“dimensionExclusions” : [

“timestamp”,

“value”

]

},

“format” : “json”

}

},

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “hour”,

“queryGranularity” : “none”

},

“metricsSpec” : [

{

“type” : “count”,

“name” : “count”

},

{

“name” : “value_sum”,

“type” : “doubleSum”,

“fieldName” : “value”

},

{

“fieldName” : “value”,

“name” : “value_min”,

“type” : “doubleMin”

},

{

“type” : “doubleMax”,

“name” : “value_max”,

“fieldName” : “value”

}

]

},

“ioConfig” : {

“type” : “realtime”

},

“tuningConfig” : {

“type” : “realtime”,

“maxRowsInMemory” : “100000”,

“intermediatePersistPeriod” : “PT10M”,

“windowPeriod” : “PT10M”

}

},

“properties” : {

“task.partitions” : “1”,

“task.replicants” : “1”

}

}

},

“properties” : {

“zookeeper.connect” : “master.example.com”,

“druid.discovery.curator.path” : “/druid/discovery”,

“druid.selectors.indexing.serviceName” : “druid/overlord”,

“http.port” : “8200”,

“http.threads” : “8”

}

}

``

I haven’t made any modification to the files handed in the quick start tutorial.

Hey Carlos,

This error “GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to disco!druid:overlord while waiting for a response” indicates that your tranquility server can’t contact your overlord at 192.168.1.107:8090. Are they running on the same machine? Is it possible that it can’t contact its own 192.168.1.107 IP address due to some routing issue or security policy? If that’s not it – try checking your overlord log for clues. It’s possible that it failed to start up for some reason.

Hello, the issue was caused by the antivirus in my machine, ESSET 32, which was blocking some of the internal traffic, thus preventing the services from talking to each other.

Thank you for your help.