Trouble pushing data from Kafka to Druid

Hi,

We are evaluating Kafka with Druid via Tranquility, but can’t see our message columns showing in Druid.

Our conf/kakfa.json snippet

{

“dataSources” : [

{

“spec” : {

“dataSchema” : {

“dataSource” : “requests”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“timestampSpec” : {

“format” : “auto”,

“column” : “timestamp”

},

“format” : “json”,

“dimensionsSpec” : {

“dimensions” : [

“pipeline”,

“consumerToken”,

“ip”

]

}

}

},

“metricsSpec” : [

{

“type” : “count”,

“name” : “count”

}

]

}

},

}

]

}

We are sending events like: {“pipeline”:“radio”,“consumerToken”:“bfc5c8e4-4559-47a7-a6fd-c964eecbdd40”,“amount”:1,“ip”:“68.61.82.62”,“timestamp”:1472736695000}

But we are seeing only timestamp and count, but not the other dimension tables.

No errors in the logs…

What could be the reason?

Thanks in advance,

Yardena

Hi Yardena,

I have not used Tranquility much, but I have had great success with the Kafka Indexing Service extension for ingesting kafka messages into druid. You may want to give that a shot.

~ Ankush

did you put the datasource server.json as well?

Thanks for help guys, turned out we just didn’t wait enough to see the data in Druid via REST API.