Issue creating Globally Cached Lookups in druid

TROUBLESHOOTING Ingestion
#1

Hi,

I am facing issue with creation of Globally Cached Lookups.

I am trying to create cachedNamespace in druid and my json file is sitting S3 bucket. The json file in S3 is of the following form:

{“foo”: “bar”}
{“baz”: “bat”}
{“buck”: “truck”}

Here is the config i am passing to druid as a POST request(druid/coordinator/v1/lookups/config/__default/dim_product_manf_desc_v2)

{
“version”: “v0”,
“lookupExtractorFactory”: {
“type”: “cachedNamespace”,
“extractionNamespace”: {
“type”: “uri”,
“uri”: “s3://data-dev/test/dim_product_manf_desc_v1.json”,
“namespaceParseSpec”: {
“format”: “simpleJson”
},
“pollPeriod”: “0”
},
“firstCacheTimeout”: 120000,
“injective”: true
}
}

With this the cachedNamespace is created and i can see that from the lookup table of console.

image
image1001×260 14.2 KB

But i am not able to see the lookup schema and when query druid says object doesn’t exist. The error is as shown below:

select * from LOOKUP.test2

Error: Plan validation failed
org.apache.calcite.runtime.CalciteContextException: From line 1, column 15 to line 1, column 26: Object 'LOOKUP' not found; did you mean 'lookup'?
org.apache.calcite.tools.ValidationException
#3

have you loaded the extension druid-lookups-cached-global?

1 Like
#4

also the query uses lookup not LOOKUP

image
image1222×1010 48.4 KB

#5

just verified that … extension is not loaded by default

#6

Thank you very much!
Loaded the extension (druid-lookups-cached-global) and i am able to create cachedNamespace from S3 bucket via druid console and able to query.

Also, could you please answer the below questions:

  1. Does cachedNamespace use off-heap or on-heap memory?
  2. Is cachedNamespace based lookup better than SQL join query between two tables? fact-dim model joins
  3. What exactly is the difference between map and cachedNamespace?