Tranquility ingestion spec. Read from many kafka topics

Apache Druid Q&A Google Groups
#1

Hi guys,
I’m trying to configure tranquility to poll from all topics that start with “ds-impression-”. For that i have next snippet of config:
“properties”: {
“task.partitions”: “1”,
“task.replicants”: “1”,
“topicPattern”: “ds-impression-”
}
It should work according to https://github.com/druid-io/tranquility/blob/master/docs/kafka.md
However in tranquility logs i see:
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - de-opportunity-clearchannel filtered
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - ds-impression-181fm filtered
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - ds-impression-clearchannel filtered
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - de-impression-181fm filtered
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - de-opportunity-181fm filtered
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - ds-opportunity-clearchannel filtered
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - ds-opportunity-181fm filtered
2/6/2017 5:33:00 PM15:33:00.944 [main] DEBUG kafka.consumer.Whitelist - de-impression-clearchannel filtered

What is wrong in my configuration?

Thanks, Vadim.

#2

The topicPattern is a regex, so try “ds-impression-.*”

#3

Yeap, i’ve tried it meanwhile and it worked but the problem is that it writes to different datasources, even if useTopicAsDataSource is false by default and i did not change it.

“dataSources”: {

“impression”: {

“spec”: {

“dataSchema”: {

“dataSource”: “impression”,

…

}

},

“properties”: {

“task.partitions”: “1”,

“task.replicants”: “1”,

“topicPattern”: “ds-impression-.*”

}

}

#4

Any ideas?