Tranquility - UnknownHostException when ingesting data from spark into druid

I am trying to ingest data from spark into druid through tranquility. I am following the exact sample code in the Tranquility repo.

While ingesting, I am seeing the following when tranquility is trying to establish a ZK connection:

16/03/10 03:08:14 ERROR imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up  <HOST>.com
	at Method)
	at org.apache.zookeeper.client.StaticHostProvider.<init>(
	at org.apache.zookeeper.ZooKeeper.<init>(
	at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(
	at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(
	at org.apache.curator.HandleHolder$1.getZooKeeper(
	at org.apache.curator.HandleHolder.getZooKeeper(
	at org.apache.curator.ConnectionState.reset(
	at org.apache.curator.ConnectionState.start(
	at org.apache.curator.CuratorZookeeperClient.start(
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory$.BeamInstance$lzycompute(DruidEventRDDBeamFactory.scala:29)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory$.BeamInstance(DruidEventRDDBeamFactory.scala:25)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory.makeBeam(DruidEventRDDBeamFactory.scala:17)
	at com.metamx.tranquility.spark.BeamFactory$class.tranquilizer(BeamFactory.scala:39)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory.tranquilizer$lzycompute(DruidEventRDDBeamFactory.scala:12)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory.tranquilizer(DruidEventRDDBeamFactory.scala:12)
	at com.metamx.tranquility.spark.BeamRDD$$anonfun$propagate$1.apply(BeamRDD.scala:36)
	at com.metamx.tranquility.spark.BeamRDD$$anonfun$propagate$1.apply(BeamRDD.scala:35)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:902)

From the host that runs the spark task, I am able to ping and netcat the ZK url:port, so there is no firewall block here.

I tried to fiddle around a bit, I get a similar error when I directly give the IP of my ZK cluster to the curator config: Temporary failure in name resolution
	at Method)

Please note that the ZK host or the IP is not present in /etc/hosts file. Do we need it to be present in the hosts file for tranquility/curator towork? Can someone please give me some pointers here?

Hey Ram,

It looks like there might be an extra space at the start of your zk connect string, since there two spaces after “” but there should generally only be one. Could you double-check how you’re providing that connect string?

Thanks Gian for the help. Extra space in the comma-separated connect string was the issue :slight_smile:

@Gian - Looking at this in zookeeper:

Looks like the issue was fixed 3 years back. Think we might have to bump the curator version in tranquility (it is on 2.6.0 now). I have submitted an issue to Tranquility.