FailureAccrualFactory marking connection to "disco!druid:overlord" as dead

I am unable to send any data to druid using docker-druid image.

Getting following exception:

Dec 04, 2018 12:24:17 PM com.twitter.finagle.service.FailureAccrualFactory$$anon$2$$anon$1 didMarkDead

INFO: FailureAccrualFactory marking connection to “disco!druid:overlord” as dead. Remote Address: Inet(/172.17.0.2:8081,Map())

Failed to send message: %sjava.lang.IllegalStateException: Failed to create merged beam: druid:overlord/pageviews

``

why is tranquility trying to send to 172.17.0.2? shouldn’t this be the address of the ‘default’ docker machine?

Hi Prathamesh,

This error sounds like some sort of issue with docker networking. You might be able to fix it by taking a look at that.

Also, you could consider using the Druid Kafka indexing service – a direct connection from Druid to Kafka that works more seamlessly than using tranquillity. We have a quickstart here: https://docs.imply.io/on-prem/tutorial/kafka-indexing-service and wrote up a post here about the design details: https://imply.io/post/exactly-once-streaming-ingestion

Gian

Hi Gian

Been stuck on using dockerized druid for quite some time now.

It could be a docker networking issue as you said but i was thinking this might have something to do with how tranquility locates druid services. I didn’t face any issues when i setup druid on a ubuntu VM , although i had to make a small change without which i was getting a similar issue. i.e “Failed to create merged beam”.

I had to set druid.host in common.runtime.properties to machines ip address. Wondering if some config changes are required for the druid docker image to work.

Meanwhile i also tried the Imply docker image. The dockerized imply distribution as well doesn’t seem to accept data over tranquility.

Thanks,

Prathamesh

Tranquility finds Druid services using ZK and should respect the ‘druid.host’ you set on the overlord. If it’s detecting 172.17.0.2 then it is probably seeing an announcement from the overlord with that ip set.

Hi Gian,

The instructions to start the docker-druid container mentioned in the wiki are :

docker run --rm -i -p 3000:8082 -p 3001:8081 druidio/example-cluster

``

The zookeeper ports doesn’t seem to be exposed in this case. Since you mentioned that tranquility would locate druid services using zk, i believe the zookeeper ports (2181,2888, 3888) also would need to be exposed then. Is that correct?

Aslo i am not setting druid.host as of now. I did that while running druid locally (not using the docker image). Not sure how to set that when using the docker-druid image.

Thanks,

Prathamesh

Hi Gian,

I think we can set druid.host in the config file *supervisord.conf. * As of now druid.host is set to %(ENV_HOSTIP)s. This assigns 172.17.0.2 . Not sure if this should have been the IP of the ‘default’ docker machine (running on windows)

I’ll give it a try.

Thanks,

Prathamesh

If you’re using Tranquility Server then it is the one that needs to contact ZK, but it’s already inside the container, so it’s good with the default ports list.

Gian

Hi Gian,

Could you validate following points regarding tranquility?

following are the different setup i tried and the part of tranquility configuration that is relevant:

  1. Druid installed on a ubuntu VM and sending data to it using tranquility as a library. - This works fine.

zookeeper.connect:VM’s IP address

``

druid.host set to VM’s IP address (without this data cannot be send to druid from the application using tranquility library)

Note: console is accessible at the same address

  1. docker-druid image running on my machine (windows) and sending data to it using tranquility as library (same application as 1). - This doesn’t work

zookeeper.connect:IP address of the default docker machine machine

``

Note: console is accessible at the same address.

If i access the docker-druid container, the container’s ip address is 172.17.0.2. The coordinator console however is accessible at ‘default’ docker machine address (192.168.99.100) in my case.

What i think is happening here is that tranquility library is not able to locate druid services since druid host is not set to the reachable ip address (192.167.99.100). Is this a correct assumption?

Also along with the above, shouldn’t zookeeper port 2181 be exposed in order for tranquility to find find? How would tranquility locate druid services otherwise?

Thanks,

Prathamesh

Hey Gian,

Can you help me with the command to see what address is stored under the node *druid/discovery/druid:overlord in zookeeper. *

Thanks,

Prathamesh

Pathamesh,

You can use the zkCli for this - either run zkCli on one of the zookeeper machines or use connect host:port from a remote machine. Once you’re in, you can use ls /druid/discovery/druid:overlord to list the znodes, and then get /druid/discovery/druid:overlord/<znode_name> on the znodes to get the address and port.

Thanks Julian.

Gian, I checked the address by logging to the docker container. I can see following address for the overlord zk node

{“name”:“druid:overlord”,“id”:“91bb6dbd-9b8c-4863-9530-e70c9f7f6587”,“address”:"

172.17.0.2",“port”:8081,“sslPort”:-1,“payload”:null,“registrationTimeUTC”:154416

6430226,“serviceType”:“DYNAMIC”,“uriSpec”:null}

``

When tranquility find the overlord, the address would be 172.17.0.2. I believe this is coming from -Ddruid.host=%(ENV_HOSTIP)s property in supervisor.conf.

Is this supposed to be something else? since this is the IP of the container how would an application be able to reach the overlord using this IP address?

Tranquility, after finding the overlord would then find existing realtime task(or create one) using the address for overlord it receives from zookeeper right?

Thanks,

Prathamesh

Hi Gian,

I was able to partially solve this problem. I could get the data flowing from my application using tranquility library to a docker-druid container. Although i have some queries about some limitations.

Solution:

  • I cloned the docker-druid repo and set -Ddruid.host to ip of my ‘default’ machine. In my case 192.168.99.102 (using docker on windows)

- apart from the ports specified in the github wiki (8081,8082) i expose other ports specified in the Dockerfile like 2181.

  • This time when i executed the application, tranquility was able to find the overlord (since overlord’s address in zknode is set to reachable ip address and not localhost). So i didn’t see the error mentioned

in this thread. i.e.

FailureAccrualFactory marking connection to “disco!druid:overlord” as dead. Remote Address: Inet(/172.17.0.2:8081,Map())

``

-This enable tranquility to submit realtime tasks which i was able to see on the coordinator console under running indexing tasks.

  • However this time there was a different error as shown below:

FailureAccrualFactory marking connection to “disco!firehose:druid:overlord:pageviews_1-090-0000-0000” as dead. Remote Address: Inet(/192.168.99.102:8100,Map())

``

  • I realized after looking at logs that druid announces the segments also under zookeeper nodes.

  • This means port 8100 would also need to be opened. I restarted the container exposing 8100 and other ports specified in the Dockerfile of Imply docker distruction like 8101,8102 etc.

  • After this i was able to send data from tranquility to druid running on the druid container. :slight_smile:

However the problem still remains.

How would this work when docker-druid is deployed to kubernetes? For cloud providers like IBM one can get only specific ports.

From IBM cloud kubernetes service:

When you expose your app by creating a Kubernetes service of type NodePort, a NodePort in the range of 30000 - 32767 and an internal cluster IP address is assigned to the service.

on my local setup i could just open port 8100. I could still do that on the kubernetes cluster but it may have to be mapped to a different port. How would this image work then?

Tranquility would retrieve the data about segments from zookeeper but the ports that one can map to exposed ports may not be the same.

Thanks,

Prathamesh

Hey Gian,

Sorry for such a long thread.

Could you answer this one last question? How would one expose the druid services when running it on kubernetes or any other container service?

As in, Druid would register overlord/discovery services with zookeeper when it starts up and later announce segments when real-time task are created.

eg.

Overlord service:

{“name”:“druid:overlord”,“id”:“91bb6dbd-9b8c-4863-9530-e70c9f7f6587”,“address”:"

172.17.0.2",“port”:8081,“sslPort”:-1,“payload”:null,“registrationTimeUTC”:154416

6430226,“serviceType”:“DYNAMIC”,“uriSpec”:null}

``

``

segment announcement:

2018-12-07T10:23:58,835 INFO [task-runner-0-priority-0] io.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName=‘firehose:druid:overlord:pageviews_1-075-0000-0000’, host=‘192.168.99.102’, port=-1, plaintextPort=8100, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}]

``

The address and port that druid stored in zk may not be configurable. A client (tranquility) would always find the address:port which is stored in zookeeper.

If running on a container how would these ports be exposed ( for eg. 8100). specifically if port 8100 has to be mapped to some other port (8100:32000) while exposing, how could that be done?

Thanks,

Prathamesh