Running Tranquility from Docker container

Hello,

Is there any requirements from environment that could prevent from running Tranquility client in Docker container?

My setup seems to run fine, but when trying to run in from within Docker I get this:

[2015-12-13 09:12:01,895] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to propagate events: overlord/clicks
{
“eventCount” : 1000,
“timestamp” : “2015-12-13T00:00:00.000Z”,
“beams” : “MergingPartitioningBeam(DruidBeam(interval = 2015-12-13T00:00:00.000Z/2015-12-14T00:00:00.000Z, partition = 0, tasks = [index_realtime_clicks_2015-12-13T00:00:00.000Z_0_0/clicks-13-0000-0000]))”
}
com.twitter.finagle.FailedFastException: Endpoint is marked down. For more details see: https://twitter.github.io/finagle/guide/FAQ.html#why-do-clients-see-com-twitter-finagle-failedfastexception-s
at com.twitter.finagle.NoStacktrace(Unknown Source)
[2015-12-13 09:12:01,895] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2015-12-13T09:12:01.895Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to propagate events: overlord/clicks”,“data”:{“exceptionType”:“com.twitter.finagle.FailedFastException”,“exceptionStackTrace”:“com.twitter.finagle.FailedFastException: Endpoint is marked down. For more details see: https://twitter.github.io/finagle/guide/FAQ.html#why-do-clients-see-com-twitter-finagle-failedfastexception-s\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n”,“timestamp”:“2015-12-13T00:00:00.000Z”,“beams”:“MergingPartitioningBeam(DruidBeam(interval = 2015-12-13T00:00:00.000Z/2015-12-14T00:00:00.000Z, partition = 0, tasks = [index_realtime_clicks_2015-12-13T00:00:00.000Z_0_0/clicks-13-0000-0000]))”,“eventCount”:1000,“exceptionMessage”:“Endpoint is marked down. For more details see: https://twitter.github.io/finagle/guide/FAQ.html#why-do-clients-see-com-twitter-finagle-failedfastexception-s”}}]

``

Thanks!

Michael

I suspect this is a configuration mismatch between tranquility and default names in Druid. What is your serviceName set to in Tranq?

I use:

DruidLocation.create(“overlord”, “druid:firehorse:clicks”)

``

I guess serviceName is “overlord”?

It’s

“druid:firehose:clicks”

``

not:

“druid:firehorse:clicks”

``

:slight_smile:

Hey Michael,

Perhaps try “druid/overlord”? The string there needs to match the “druid.service” in your overlord’s runtime.properties. If you haven’t provided one, the default is “druid/overlord”.

Hey Gian,

It’s set to “overlord” in runtime.properties. The strange thing is that the client works when running it as “java -jar …”, but when running the same command from within a Docker container it fails.

Thanks!

Michael

Are you running everything (tranquility & the overlord) in the same Docker container? If not then maybe you need to publish some additional ports of the container, or perhaps there’s another networking issue of some kind. The earliest exception you got from tranquility should have more information about what is wrong. (The “fail fast” exceptions occur after the first one and don’t really tell you that much)

No, the overlord is running outside of the Docker container and on another machine.
I’m pretty sure all the exceptions were like the one I’ve posted, but I’ll give it another try.

This is the first exception I get:

[2015-12-16 14:14:12,342] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=, defaultVersion=‘null’, localRepository=’/root/.m2/repository’, remoteRepositories=[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]
[2015-12-16 14:14:12,790] INFO c.metamx.emitter.core.LoggingEmitter - Start: started [true]
Dec 16, 2015 2:14:13 PM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp
INFO: Finagle version 6.25.0 (rev=78909170b7cc97044481274e297805d770465110) built at 20150423-134857
[2015-12-16 14:14:14,110] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[overlord] to Set(ServiceInstance{name=‘overlord’, id=‘867eb9bb-4503-4ad9-8bad-9ae716689d37’, address=‘druidmaster-20001-001-prod.eu1.appsflyer.com’, port=8090, sslPort=null, payload=null, registrationTimeUTC=1450244118315, serviceType=DYNAMIC, uriSpec=null})
[2015-12-16 14:14:14,309] INFO c.m.t.finagle.FinagleRegistry - Created client for service: overlord
[2015-12-16 14:14:14,826] INFO c.m.tranquility.beam.ClusteredBeam - Merged beam already created for identifier[overlord/clicks] timestamp[2015-12-16T00:00:00.000Z], with sufficient partitions (target = 1, actual = 1)
[2015-12-16 14:14:14,832] INFO c.m.tranquility.beam.ClusteredBeam - Adding beams for identifier[overlord/clicks] timestamp[2015-12-16T00:00:00.000Z]: List(Map(interval -> 2015-12-16T00:00:00.000Z/2015-12-17T00:00:00.000Z, partition -> 0, tasks -> List(Map(id -> index_realtime_clicks_2015-12-16T00:00:00.000Z_0_0, firehoseId -> clicks-16-0000-0000), Map(id -> index_realtime_clicks_2015-12-16T00:00:00.000Z_0_1, firehoseId -> clicks-16-0000-0001)), timestamp -> 2015-12-16T00:00:00.000Z))
[2015-12-16 14:14:14,873] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:firehose:clicks-16-0000-0000] to Set(ServiceInstance{name=‘druid:firehose:clicks-16-0000-0000’, id=‘3d563be7-57d8-4f0d-bcf7-7f5571abe3eb’, address=‘druiddata-20001-001-prod.eu1.appsflyer.com’, port=8100, sslPort=null, payload=null, registrationTimeUTC=1450244213006, serviceType=DYNAMIC, uriSpec=null})
[2015-12-16 14:14:14,879] INFO c.m.t.finagle.FinagleRegistry - Created client for service: druid:firehose:clicks-16-0000-0000
[2015-12-16 14:14:14,906] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:firehose:clicks-16-0000-0001] to Set(ServiceInstance{name=‘druid:firehose:clicks-16-0000-0001’, id=‘0d7bf7c4-dd85-4926-a945-b53a483f13f4’, address=‘druiddata-20002-001-prod.eu1.appsflyer.com’, port=8100, sslPort=null, payload=null, registrationTimeUTC=1450244212813, serviceType=DYNAMIC, uriSpec=null})
[2015-12-16 14:14:14,915] INFO c.m.t.finagle.FinagleRegistry - Created client for service: druid:firehose:clicks-16-0000-0001
[2015-12-16 14:15:18,672] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to propagate events: overlord/clicks
{
“eventCount” : 1000,
“timestamp” : “2015-12-16T00:00:00.000Z”,
“beams” : “MergingPartitioningBeam(DruidBeam(interval = 2015-12-16T00:00:00.000Z/2015-12-17T00:00:00.000Z, partition = 0, tasks = [index_realtime_clicks_2015-12-16T00:00:00.000Z_0_0/clicks-16-0000-0000; index_realtime_clicks_2015-12-16T00:00:00.000Z_0_1/clicks-16-0000-0001]))”
}
com.twitter.finagle.FailedFastException: Endpoint is marked down. For more details see: https://twitter.github.io/finagle/guide/FAQ.html#why-do-clients-see-com-twitter-finagle-failedfastexception-s
at com.twitter.finagle.NoStacktrace(Unknown Source)
[2015-12-16 14:15:18,708] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2015-12-16T14:15:18.679Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to propagate events: overlord/clicks”,“data”:{“exceptionType”:“com.twitter.finagle.FailedFastException”,“exceptionStackTrace”:“com.twitter.finagle.FailedFastException: Endpoint is marked down. For more details see: https://twitter.github.io/finagle/guide/FAQ.html#why-do-clients-see-com-twitter-finagle-failedfastexception-s\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n”,“timestamp”:“2015-12-16T00:00:00.000Z”,“beams”:“MergingPartitioningBeam(DruidBeam(interval = 2015-12-16T00:00:00.000Z/2015-12-17T00:00:00.000Z, partition = 0, tasks = [index_realtime_clicks_2015-12-16T00:00:00.000Z_0_0/clicks-16-0000-0000; index_realtime_clicks_2015-12-16T00:00:00.000Z_0_1/clicks-16-0000-0001]))”,“eventCount”:1000,“exceptionMessage”:“Endpoint is marked down. For more details see: https://twitter.github.io/finagle/guide/FAQ.html#why-do-clients-see-com-twitter-finagle-failedfastexception-s”}}]

``

It doesn’t really tells much.

It looks like Druid connects back to Tranquility client in some way, I’ve just verified that I have to enable all the ports in EC2 security group in order to make this work from another machine. Is there a way to know what ports/protocols must be opened? When looking at the “netstat” output, I’m not seeing anything is listening in the Tranquility process, neither I’ve found something suspicious when Looking at the source code of Tranquility.

Do you have any ideas?