Druid Kafka cluster: tranquility not listening

Its hard to explain out problem because we do not get any log outputs that would help here.

Main we have the problem, that our kafka tranquility is not listening to the kafka inserts, we have the following cluster:

3x kafka instances (with tranquility, zookeeper)

2x middlemanager instances (with zookeeper)

2x broker instances (with zookeeper)

2x overlord,coordinator instances (with zookeeper)

Our problems:

  1. When we shoot a message to the kafka, we can see in the logs, that it is correctly inserted.

The tranquility runs without any error and listens to the right topic but does not recognize any kafka message?

  1. In the druid console (overlord-ip:8081) we do not see any workers, but in the coordinator console (overlord-ip:8090) we see the 2 workers?

Our tranquility kafka.json:

{

“dataSources” : {

“test” : {

“spec” : {

“dataSchema” : {

“dataSource” : “test”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“timestampSpec” : {

“column” : “last_change”,

“format” : “auto”

},

“dimensionsSpec” : {

“dimensions” : [

“test”

],

“dimensionExclusions” : [

“last_change”

]

},

“format” : “json”

}

},

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “hour”,

“queryGranularity” : “none”

},

“metricsSpec” : [

{

“name” : “num”,

“type” : “longSum”,

“fieldName” : “num”

}

]

},

“ioConfig” : {

“type” : “realtime”

},

“tuningConfig” : {

“type” : “realtime”,

“maxRowsInMemory” : “100000”,

“intermediatePersistPeriod” : “PT10M”,

“windowPeriod” : “PT10M”

}

},

“properties” : {

“task.partitions” : “2”,

“task.replicants” : “2”,

“topicPattern” : “test.*”,

“topicPattern.priority” : “1”

}

}

},

“properties” : {

“zookeeper.connect” : “xxxxxxx:2181”,

“zookeeper.timeout” : “PT20S”,

“druid.selectors.indexing.serviceName” : “druid/overlord”,

“druid.discovery.curator.path” : “/druid/discovery”,

“kafka.zookeeper.connect” : “xxxxxxxx:2181”,

“kafka.group.id” : “tranquility-kafka”,

“consumer.numThreads” : “2”,

“commit.periodMillis” : “15000”,

“reportDropsAsExceptions” : “true”

}

}

Our Zookeeper config on Kafka instances (on overlord cfg, IPs from both overlords are set):

myid=1

tickTime=2000

initLimit=10

syncLimit=10

dataDir=/var/lib/zookeeper

clientPort=2181

maxClientCnxns=60

autopurge.snapRetainCount=3

autopurge.purgeInterval=0

leaderServes=yes

preAllocSize=65536

server.1=kafka-1:2888:3888

server.2=kafka-2:2888:3888

server.3=kafka-3:2888:3888

Any help would be greatly appreciated! :slight_smile: