druid tranquility

I’m struggling with this error for some days. The problem seems i can’t connect to zookeeper through curator but i don’t understand why.I´m able to connect it using zkclient.

I was able to start all the services: coordinator,historical,broker,overlord and middle Manager

I’m using scala api to inset data into druid.

Thanks!

[info] Running druid_tranquil

ERROR [org.apache.zookeeper.ClientCnxn] Event thread exiting due to interruption

java.lang.InterruptedException

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

[success] Total time: 1 s, completed May 22, 2015 5:05:18 PM

ERROR [org.apache.zookeeper.ClientCnxnSocketNIO] Unable to open socket to 127.0.0.1/127.0.0.1:2181

WARN [org.apache.zookeeper.ClientCnxn] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.nio.channels.ClosedByInterruptException

at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)

at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)

at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)

at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)

at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

FIles-config/

_common/common.runtime.properties:

druid.extensions.coordinates=[“io.druid.extensions:druid-examples”,“io.druid.extensions:druid-kafka-eight”,“io.druid.extensions:mysql-metadata-storage”]

druid.zk.service.host=127.0.0.1

druid.zk.service.port=2181

druid.zk.paths.base=/druid/prod

druid.discovery.curator.path=/prod/discovery

druid.request.logging.type=emitter

druid.request.logging.feed=druid_requests

druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://localhost:3306/druid

druid.metadata.storage.connector.user=root

druid.metadata.storage.connector.password=druid

druid.storage.type=local

druid.storage.storage.storageDirectory=/tmp/druid/localStorage

druid.cache.type=local

druid.cache.sizeInBytes=10000000

druid.selectors.indexing.serviceName=druid:prod:overlord

druid.emitter=noop

overload/runtime.properties:

druid.host=localhost:8085

druid.port=8085

druid.service=druid/prod/overlord

druid.indexer.storage.type=metadatadruid.extensions.coordinates=[“io.druid.extensions:druid-s3-extensions”, “io.druid.extensions:druid-histogram”, “io.druid.extensions:mysql-metadata-storage”]

druid.storage.type=local

druid.storage.storage.storageDirectory=/tmp/druid/localStorage

druid.indexer.storage.type=metadata

middlemanager/runtime.properties:

druid.host=localhost:8084

druid.port=8084

druid.service=druid/prod/middlemanager

druid.storage.type=local

druid.storage.storage.storageDirectory=/tmp/druid/localStorage

druid.indexer.runner.javaOpts=-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

druid.indexer.task.baseTaskDir=/mnt/persistent/task/

druid.indexer.fork.property.druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912

druid.indexer.fork.property.druid.processing.numThreads=2

druid.indexer.fork.property.druid.segmentCache.locations=[{“path”: “/mnt/persistent/zk_druid”, “maxSize”: 0}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.indexer.fork.property.druid.storage.type=local

druid.indexer.fork.property.druid.storageDirectory=/tmp/druid/localStorage

druid.worker.capacity=9

druid.worker.ip=localhost

.scala

import com.metamx.common.Granularity

import com.metamx.common.scala.net.curator

import com.metamx.tranquility.beam.ClusteredBeamTuning

import com.metamx.tranquility.druid._

import com.twitter.util.Await

import io.druid.granularity.QueryGranularity

import io.druid.query.aggregation.{CountAggregatorFactory, LongSumAggregatorFactory}

import org.apache.curator.framework.{CuratorFramework, CuratorFrameworkFactory}

import org.apache.curator.retry.{RetryOneTime, ExponentialBackoffRetry, BoundedExponentialBackoffRetry}

import org.joda.time.{Period, DateTime}

object druid_tranquil {

def main(args: Array[String]): Unit = {

try {

val dataSource = “foo”

val discoveryPath = “/prod/discovery” // Your overlord’s druid.discovery.curator.path.

val indexService = “druid:prod:overlord” // Your overlord’s druid.service, with slashes replaced by colons.

val firehosePattern = “druid:firehose:%s” // Make up a service pattern, include %s somewhere in it.

val dimensions = IndexedSeq(“bar”)

val aggregators = Seq(new LongSumAggregatorFactory(“baz”, “baz”))

val curator = CuratorFrameworkFactory.builder()

.connectString(“127.0.0.1:2181”)

.retryPolicy(new ExponentialBackoffRetry(500, 15, 10000))

.build()

curator.start();

val timestamper = (eventMap: Map[String, Any]) => new DateTime(eventMap(“timestamp”))

val service = DruidBeams

.builder(timestamper)

.curator(curator)

.discoveryPath(discoveryPath)

.location(DruidLocation(indexService, firehosePattern, dataSource))

.rollup(DruidRollup(SpecificDruidDimensions(dimensions), aggregators, QueryGranularity.MINUTE))

.tuning(

ClusteredBeamTuning(

segmentGranularity = Granularity.HOUR,

windowPeriod = new Period(“PT10M”),

partitions = 1,

replicants = 1

)

)

.buildService()

val future = service(Seq(Map(“timestamp” -> “2010-01-02T03:04:05.678Z”, “bar” -> “hey”, “baz” -> 3)))

println(“result = %s” format Await.result(future))

}catch {

case ex: Exception => {

println(ex)

} } }}

Hi Rafael, this is indeed strange. Do the Druid nodes complain at all about connecting to ZK? Have you tried connecting locally via zkCli?

I start all druid nodes and i get no problem, all of them connect to zookeeper.

Yes, i am able to connect to zkCli.

On the same host, the port is open and listening. I still dont get the problem

Any other suggestion?

Thanks !

Hi Rafael, do you have the full logs of when you start tranq? I’m not sure what is happening based on the description so far but perhaps having all the logs will yield more information.

Looking closer, i found this Exception. Has they were INFO i didn´t saw it before

broker

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.13.0-36-generic

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=atnogcloud

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/atnogcloud

2015-05-28T10:20:52,226 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/atnogcloud/hadoop/druid-0.7.1.1

2015-05-28T10:20:52,227 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@61184753

2015-05-28T10:20:52,252 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.client.ServerInventoryView.start() throws java.lang.Exception] on object[io.druid.client.BatchServerInventoryView@43405236].

2015-05-28T10:20:52,264 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-05-28T10:20:52,273 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session

2015-05-28T10:20:52,274 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.coordination.broker.DruidBroker.start()] on object[io.druid.server.coordination.broker.DruidBroker@6f89de90].

2015-05-28T10:20:52,277 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112

2015-05-28T10:20:52,280 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14d7c3d7a3c000e, negotiated timeout = 30000

2015-05-28T10:20:52,289 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

2015-05-28T10:20:52,495 INFO [ServerInventoryView-0] io.druid.curator.inventory.CuratorInventoryManager - Created new InventoryCacheListener for /druid/prod/segments/localhost:8083

2015-05-28T10:20:52,495 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - New Server[DruidServerMetadata{name=‘localhost:8083’, host=‘localhost:8083’, maxSize=10000000000, tier=’_default_tier’, type=‘historical’, priority=‘0’}]

2015-05-28T10:20:52,516 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - Inventory Initialized

2015-05-28T10:20:52,539 INFO [ServerInventoryView-0] io.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName=‘broker’, host=‘localhost’, port=8082}]

2015-05-28T10:20:53,314 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@23a7bed0{/,null,AVAILABLE}

2015-05-28T10:20:53,322 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@67fcee4a{HTTP/1.1}{0.0.0.0:8082}

2015-05-28T10:20:53,323 INFO [main] org.eclipse.jetty.server.Server - Started @44945ms

and coordinator

2015-05-28T10:20:53,849 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2015-05-28T10:20:53,849 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

2015-05-28T10:20:53,849 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=

2015-05-28T10:20:53,850 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

2015-05-28T10:20:53,850 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64

2015-05-28T10:20:53,850 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.13.0-36-generic

2015-05-28T10:20:53,850 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=atnogcloud

2015-05-28T10:20:53,850 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/atnogcloud

2015-05-28T10:20:53,850 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/atnogcloud/hadoop/druid-0.7.1.1

2015-05-28T10:20:53,851 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@4df3f011

2015-05-28T10:20:53,889 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.client.ServerInventoryView.start() throws java.lang.Exception] on object[io.druid.client.BatchServerInventoryView@3777cf76].

2015-05-28T10:20:53,901 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-05-28T10:20:53,906 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session

2015-05-28T10:20:53,914 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14d7c3d7a3c000f, negotiated timeout = 30000

2015-05-28T10:20:53,920 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

2015-05-28T10:20:53,926 INFO [main] io.druid.metadata.SQLMetadataConnector - Table[druid_rules] already exists

2015-05-28T10:20:53,928 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.metadata.SQLMetadataRuleManager.start()] on object[io.druid.metadata.SQLMetadataRuleManager@710157b0].

2015-05-28T10:20:53,957 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.NettyHttpClient.start()] on object[com.metamx.http.client.NettyHttpClient@2172c025].

2015-05-28T10:20:53,958 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@1e69e4b7].

2015-05-28T10:20:53,988 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.coordinator.DruidCoordinator.start()] on object[io.druid.server.coordinator.DruidCoordinator@41bde91b].

2015-05-28T10:20:54,016 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112

2015-05-28T10:20:54,103 INFO [CoordinatorLeader-0] io.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow!

2015-05-28T10:20:54,116 INFO [CoordinatorLeader-0] io.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName=‘coordinator’, host=‘localhost’, port=8081}]

2015-05-28T10:20:54,298 INFO [ServerInventoryView-0] io.druid.curator.inventory.CuratorInventoryManager - Created new InventoryCacheListener for /druid/prod/segments/localhost:8083

2015-05-28T10:20:54,299 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - New Server[DruidServerMetadata{name=‘localhost:8083’, host=‘localhost:8083’, maxSize=10000000000, tier=’_default_tier’, type=‘historical’, priority=‘0’}]

2015-05-28T10:20:54,341 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - Inventory Initialized

2015-05-28T10:20:54,355 INFO [DatabaseRuleManager-Exec–0] io.druid.metadata.SQLMetadataRuleManager - Polled and found rules for 1 datasource(s)

2015-05-28T10:20:54,439 INFO [CoordinatorLeader-0] io.druid.common.config.ConfigManager - Creating watch for key[coordinator.config]

2015-05-28T10:20:54,460 INFO [CoordinatorLeader-0] io.druid.common.config.ConfigManager - Creating watch for key[coordinator.whitelist]

2015-05-28T10:20:55,294 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@98694f1{/,jar:file:/home/atnogcloud/hadoop/druid-0.7.1.1/lib/druid-server-0.7.1.1.jar!/static,AVAILABLE}

2015-05-28T10:20:55,300 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@4aee6d59{HTTP/1.1}{0.0.0.0:8081}

and overlord

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.13.0-36-generic

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=atnogcloud

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/atnogcloud

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/atnogcloud/hadoop/druid-0.7.1.1

2015-05-28T10:20:37,918 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@5daabc59

2015-05-28T10:20:38,184 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.WorkerCuratorCoordinator.start() throws java.lang.Exception] on object[io.druid.indexing.worker.WorkerCuratorCoordinator@2be2e55b].

2015-05-28T10:20:38,184 INFO [main] io.druid.indexing.worker.WorkerCuratorCoordinator - WorkerCuratorCoordinator good to go sir. Server[localhost:8084]

2015-05-28T10:20:38,198 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-05-28T10:20:38,222 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session

2015-05-28T10:20:38,278 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14d7c3d7a3c000c, negotiated timeout = 30000

2015-05-28T10:20:38,293 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

2015-05-28T10:20:38,669 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.WorkerTaskMonitor.start()] on object[io.druid.indexing.worker.WorkerTaskMonitor@343159b].

2015-05-28T10:20:38,742 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112

2015-05-28T10:20:42,638 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7a86f82e{/,null,AVAILABLE}

2015-05-28T10:20:42,646 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@62283451{HTTP/1.1}{0.0.0.0:8084}

2015-05-28T10:20:42,646 INFO [main] org.eclipse.jetty.server.Server - Started @34328ms

and middlemanager

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.13.0-36-generic

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=atnogcloud

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/atnogcloud

2015-05-28T10:20:37,917 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/atnogcloud/hadoop/druid-0.7.1.1

2015-05-28T10:20:37,918 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@5daabc59

2015-05-28T10:20:38,184 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.WorkerCuratorCoordinator.start() throws java.lang.Exception] on object[io.druid.indexing.worker.WorkerCuratorCoordinator@2be2e55b].

2015-05-28T10:20:38,184 INFO [main] io.druid.indexing.worker.WorkerCuratorCoordinator - WorkerCuratorCoordinator good to go sir. Server[localhost:8084]

2015-05-28T10:20:38,198 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-05-28T10:20:38,222 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session

2015-05-28T10:20:38,278 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14d7c3d7a3c000c, negotiated timeout = 30000

2015-05-28T10:20:38,293 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

2015-05-28T10:20:38,669 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.WorkerTaskMonitor.start()] on object[io.druid.indexing.worker.WorkerTaskMonitor@343159b].

2015-05-28T10:20:38,742 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112

2015-05-28T10:20:42,638 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7a86f82e{/,null,AVAILABLE}

2015-05-28T10:20:42,646 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@62283451{HTTP/1.1}{0.0.0.0:8084}

2015-05-28T10:20:42,646 INFO [main] org.eclipse.jetty.server.Server - Started @34328ms

Overlord got none

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.13.0-36-generic

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=atnogcloud

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/atnogcloud

2015-05-28T10:20:53,869 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/atnogcloud/hadoop/druid-0.7.1.1

2015-05-28T10:20:53,870 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@a5ae373

2015-05-28T10:20:53,908 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.overlord.TaskMaster.start()] on object[io.druid.indexing.overlord.TaskMaster@41671bc1].

2015-05-28T10:20:53,935 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112

2015-05-28T10:20:53,918 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-05-28T10:20:53,949 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session

2015-05-28T10:20:53,980 INFO [main-SendThread(127.0.0.1:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14d7c3d7a3c0010, negotiated timeout = 30000

2015-05-28T10:20:53,992 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

2015-05-28T10:20:54,088 INFO [LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power!

2015-05-28T10:20:54,096 INFO [LeaderSelector-0] io.druid.indexing.overlord.TaskLockbox - Synced 0 locks for 0 tasks from storage (0 locks ignored).

2015-05-28T10:20:54,113 INFO [LeaderSelector-0] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.overlord.TaskQueue.start()] on object[io.druid.indexing.overlord.TaskQueue@55dacb95].

2015-05-28T10:20:54,116 INFO [LeaderSelector-0] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

2015-05-28T10:20:54,121 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskQueue - Beginning management in PT60S.

2015-05-28T10:20:54,144 INFO [LeaderSelector-0] io.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName=‘druid/prod/overlord’, host=‘localhost’, port=8085}]

2015-05-28T10:20:55,072 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@5599ea12{/,[jar:file:/home/atnogcloud/hadoop/druid-0.7.1.1/lib/druid-server-0.7.1.1.jar!/static, jar:file:/home/atnogcloud/hadoop/druid-0.7.1.1/lib/druid-indexing-service-0.7.1.1.jar!/indexer_static],AVAILABLE}

2015-05-28T10:20:55,078 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@1671998a{HTTP/1.1}{0.0.0.0:8085}

2015-05-28T10:20:55,078 INFO [main] org.eclipse.jetty.server.Server - Started @46777ms

2015-05-28T10:21:54,123 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

2015-05-28T10:22:54,123 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

2015-05-28T10:23:54,124 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

Thank You!

Rafael, I don’t see any exceptions in these logs, everything appears normal. Do you have the tranquility logs when you start up the service that is running tranq?

There are all the logs from Coordinator,historical,mmiddlemanager,overlord and from tranquility when i run my jar who has my main class.
Thanks again!

coordinator.txt (315 KB)

druidtranquility.txt (1.3 KB)

historical.txt (62.5 KB)

middlemanager.txt (35.6 KB)

overlord.txt (125 KB)

Hi Rafael, is it possible for you just to write a very simple main() to test connection to ZK with curator?

i.e.

val curator = CuratorFrameworkFactory.builder()

.connectString(“127.0.0.1:2181”)

.retryPolicy(new ExponentialBackoffRetry(500, 15, 10000))

.build()

curator.start();

Does that work?

Yes, it gives the following exception

[info] Running druid_tranquil
ERROR [org.apache.zookeeper.ClientCnxn] Event thread exiting due to interruption
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
ERROR [org.apache.zookeeper.ClientCnxnSocketNIO] Unable to open socket to 127.0.0.1/127.0.0.1:2181
 WARN [org.apache.zookeeper.ClientCnxn] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.nio.channels.ClosedByInterruptException
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)
	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Thank You!

Okay, so I think this problem has nothing to do with Druid/Tranquility. It appears Curator is just unable to connect to ZK. How are you running this block of code? Are you running it in your IDE?

I am running it using sbt clean, package assembly, run.