Problem with Kafka Lookups

Hello all,

I’m about to test Kafka lookups, but I have a problem. I have started Kafka 2.11-1.1.0 Quickstart and Druid 0.12.3. I have prepared a lookup configuration file, which I have registered via POST:

{

“__default”: {

“kafka_test”: {

“version”: “v0”,

“lookupExtractorFactory”: {

“type”:“kafka”,

“kafkaTopic”:“testTopic”,

“kafkaProperties”:{“zookeeper.connect”:“localhost/kafka”}

}

}

}

}

``

After registration I can see the status of my lookup using GET:

"__default"``:{``"kafka_test"``:{``"loaded"``:``true``}}

``

Unfortunately after a minute the status changes to

"__default"``:{``"kafka_test"``:{``"loaded"``:``**false**``}}

``

Even within the active minute the lookup does’t work, if I use it in a query:

...

"dimensions"``: [

``{

``"type"``: ``"extraction"``,

``"dimension"``: ``"uid"``,

``"outputName"``: ``"uid"``,

``"outputType"``: ``"STRING"``,

``"extractionFn"``: {

``"type"``: ``"registeredLookup"``,

``"lookup"``: ``"kafka_test"``,

``"retainMissingValue"``: ``true

``}

``},

``...

]

...

``

The output after sending the query is:

{

``"error" : ``"Unknown exception"``,

``"errorMessage" : ``null``,

``"errorClass" : ``"java.lang.IllegalStateException"``,

``"host" : ``null

}

``

In Kafka i have used the producer from the Druid documentation:

./bin/kafka-console-producer.sh --property parse.key=true --property key.separator="->" --broker-list localhost:9092 --topic testTopic

``

Have anybody of you tested Kafka lookups successfuly? Do you see a problem in my code?

Best regards

Martin

I have definitely seen it work before. I wonder if there’s some more detail available on the IllegalStateException – perhaps in the logs of a broker or historical node?

Thank you Gian. Meanwhile I could solve the problem. It did not work for two reasons:

  • The order of loading extensions is important. druid-kafka-extraction-namespace must be loaded before druid-lookups-cached-global
  • I needed to remove “/kafka” from “kafkaProperties”:{“zookeeper.connect”:“somehost:2181/kafka”} from the config file
    After fixing the two issues everything works fine.

Best regards

Martin