tranquility kafka can't talk to Kafka

Hi,

I’m trying to ingest data from Kafka to druid (latest stable version 0.9.0) using tranquitily kafka. But looks like it doesn’t pickup kafka topic. The below is my tranquility config:

{

“dataSources” : [

{

“spec” : {

“dataSchema” : {

“granularitySpec” : {

“queryGranularity” : “none”,

“type” : “uniform”,

“segmentGranularity” : “FIVE_MINUTE”

},

“dataSource” : “wikipedia”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“timestampSpec” : {

“format” : “auto”,

“column” : “timestamp”

},

“format” : “json”,

“dimensionsSpec” : {

“dimensions” : [

“page”,

“language”,

“user”,

“unpatrolled”,

“newPage”,

“robot”,

“anonymous”,

“namespace”,

“continent”,

“country”,

“region”,

“city”

]

}

}

},

“metricsSpec” : [

{

“type” : “count”,

“name” : “count”

},

{

“type” : “doubleSum”,

“name” : “added”,

“fieldName” : “added”

},

{

“name” : “deleted”,

“type” : “doubleSum”,

“fieldName” : “deleted”

},

{

“name” : “delta”,

“type” : “doubleSum”,

“fieldName” : “delta”

}

]

},

“ioConfig” : {

“type” : “realtime”

},

“tuningConfig” : {

“type” : “realtime”,

“intermediatePersistPeriod” : “PT2M”,

“windowPeriod” : “PT2M”,

“maxRowsInMemory” : 75000

}

},

“properties” : {

“task.partitions” : “1”,

“task.replicants” : “1”,

“topicPattern” : “wikipedia”,

“topicPattern.priority” : “1”

}

}

],

“properties” : {

“zookeeper.connect” : “localhost:2181”,

“zookeeper.timeout” : “PT20S”,

“druid.selectors.indexing.serviceName” : “druid/overlord”,

“druid.discovery.curator.path” : “/druid/discovery”,

“kafka.zookeeper.connect” : “localhost:2181”,

“kafka.group.id” : “druid-example”,

“consumer.numThreads” : “2”,

“commit.periodMillis” : “15000”,

“reportDropsAsExceptions” : “false”

}

}

Looks like it’s working, I did provided wrong parameter when I started tranquility

HI,
Were you trying to do the Wikipedia edits with real time data?

yes, I have kafka-druid

I am trying to do the same can you explain me how to get the realtime data. I read the documentation and all the data that is being used is from the json file which we input. can u tell me the process

Thank You

Hi Ujwal,

You can follow this instruction: http://druid.io/docs/0.9.0/tutorials/quickstart.html

And when you start tranquility, use this command:

bin/tranquility kafka -configFile <path_to_druid_distro>/conf-quickstart/tranquility/kafka.json

Hoa.

I am facing the exact same issue. The issue is something to do with Tranquility not finding the topic. I can see the Zookeeper show all listed topics and consumer groups in its shell, kafka broker accepting the topics and also capturing data and offsets and kafka console consumer when connected to the topic can consume the events. Tranquility-kafka.log shows connection to Zookeeper however, the problem starts somewhere around here:

2016-08-09 16:44:44,271 [main] INFO k.c.ZookeeperConsumerConnector - [bb17-tranquility_skoppar-mac15.ecdc.edgecast.com-1470761083011-d74c5b83], begin rebalancing consumer bb17-tranquility_skoppar-mac15.ecdc.edgecast.com-1470761083011-d74c5b83 try #0

2016-08-09 16:44:44,409 [main] WARN k.c.ZookeeperConsumerConnector - [bb17-tranquility_skoppar-mac15.ecdc.edgecast.com-1470761083011-d74c5b83], no brokers found when trying to rebalance.

2016-08-09 16:44:44,411 [main] INFO k.c.ZookeeperConsumerConnector - [bb17-tranquility_skoppar-mac15.ecdc.edgecast.com-1470761083011-d74c5b83], end rebalancing consumer bb17-tranquility_skoppar-mac15.ecdc.edgecast.com-1470761083011-d74c5b83 try #0

2016-08-09 16:44:44,413 [main] INFO k.c.ZookeeperConsumerConnector - [bb17-tranquility_skoppar-mac15.ecdc.edgecast.com-1470761083011-d74c5b83], Creating topic event watcher for topics ((?!))

2016-08-09 16:44:44,421 [main] INFO k.c.ZookeeperConsumerConnector - [bb17-tranquility_skoppar-mac15.ecdc.edgecast.com-1470761083011-d74c5b83], Topics to consume = List()

2016-08-09 16:44:59,177 [KafkaConsumer-CommitThread] INFO c.m.tranquility.kafka.KafkaConsumer - Flushed 0 pending messages in 0ms and committed offsets in 2ms.

2016-08-09 16:45:14,180 [KafkaConsumer-CommitThread] INFO c.m.tranquility.kafka.KafkaConsumer - Flushed 0 pending messages in 0ms and committed offsets in 0ms.

Why is the topics list empty?

Attaching the kafka.json (renamed it here publicAvroKafka) I am using. Also, I am using supervise from imply to trigger the services and it uses exact same command as mentioned below.

Appreciate any suggestion on this.

regards

Sunita

publicAvroKafka.txt (7.28 KB)

In the attached spec file, the datasource level kafka specific properties are specified at wrong level and therefore not being picked up.
you need to move them outside the “spec” block.

Refer https://github.com/druid-io/tranquility/blob/master/distribution/src/universal/conf/kafka.json.example for example.