Help Using Lookups from Kafka

Hi,

I have been trying to use lookups and had success using the lookup type “map”, with some key value pairs. I was able to use lookups in dimensions and filters for the map lookup type.

However, i have not had much success with Kafka Lookups (http://druid.io/docs/latest/development/extensions-core/kafka-extraction-namespace.html)

When I try to use the lookup in the dimension spec in my select query, I get an unknown exception.

Following are the steps I took.

1) Added, druid-kafka-extraction-namespace and druid-lookups-cached-global in the extensions list. The load list now looks like, [“druid-histogram”, “druid-datasketches”, “druid-kafka-indexing-service”, “druid-kafka-extraction-namespace”, “druid-lookups-cached-global”]

2) Adding the Lookup,

POST to http://localhost:8081/druid/coordinator/v1/lookups

{

“__default”: {

“mylookupname”: {

“type”: “kafka”,

“kafkaTopic”: “testlookup”,

“kafkaProperties”: {

“zookeeper.connect”: “localhost:2181/kafka”

}

}

}

}

3) GET http://localhost:8081/druid/coordinator/v1/lookups/__default,

returns

[

“mylookupname”

]

4) Query that throws the unknown exception,

POST localhost:8082/druid/v2/

{

“queryType”: “select”,

“dataSource”: “mydatasource”,

“descending”: “false”,

“dimensions”: [{

“type”: “lookup”,

“dimension”: “DimensionName”,

“outputName”: “DimensionName”,

“name”: “mylookupname”

}],

“metrics”: ,

“granularity”: “all”,

“intervals”: [“2016-09-01/2016-09-15”],

“pagingSpec”: {

“pagingIdentifiers”: {

},

“threshold”: 500

}

}

5) Since this did not work, I posted the lookup directly to the broker and historical nodes just to force it to use my lookup. Not not make any difference.

POST http://localhost:8082/druid/listen/v1/lookups/mylookupname

POST http://localhost:8083/druid/listen/v1/lookups/mylookupname

{

“type”: “kafka”,

“kafkaTopic”: “testlookup”,

“kafkaProperties”: {

“zookeeper.connect”: “localhost:2181/kafka”

}

}

I followed similar steps for a lookup type “map” and I was able to use the lookup in my query, successfully.

Not sure what mistake I am making that is causing this to not work.

Hope someone can help with this!

I should also mention that POST to http://localhost:8082/druid/listen/v1/lookups/mylookupname is successful and the following response is returned.

{

“status”: “accepted”,

“failedUpdates”: {}

}

But, If I do a GET on http://localhost:8082/druid/listen/v1/lookups , I do not see the lookup that I just posted.

Are the kafka lookups treated differently than the regular map lookups?

Thanks,

Ankush

Hi,

I have been experiencing the same problem. Have you possibly found the solution?

Thanks,

Jan

Hi Ankush,

Can you please share the logs of the broker/historical that show the exception ?

Also when you post the lookup to the coordinator is there any error on the logs ? can you check if you can issue a get to historical to check if then know about the lookups ?

Hi,

I have been digging more and this looks related to https://github.com/druid-io/druid/issues/2329 which is merged to 0.9.2 branch. I have backported the fix and lookup works.

Regards,

Jan