Nested Druid Queries Not Selecting Transformed Dimensions from Lookups in Inner Query

Hey I have been using Druid for a while now, and it’s been generally great!

I have recently run into an issue with Nested Queries.

If I define a lookup dimension in the inner query, and try to propagate it to the outer query by selecting the inner dimension’s outputName as a default dimension in the outer query I get the original non-transformed dimension instead of the lookup dimension value.

At first, I thought the issue was that maybe because the lookups weren’t JDBC because that seemed to be the trend among the dimensions which were not working for me, however upon starting to upgrade the lookups to JDBC I am still having the issue. It is very strange that it is inconsistently working.

For example:

Inner query

{

“dimension”: “country_id”,

“outputName”: “country”,

“name”: “country_name_by_id_lookup_v2”,

“type”: “lookup”,

“replaceMissingValueWith”: null,

“retainMissingValue”: false,

“injective”: false,

“optimize”: true

}

Outer query

{

“type”: “default”,

“dimension”: “country”,

“outputName”: “country_name”

}

Could you attach the full query you’re using? What version of Druid is this? Are you using groupBy v1 or v2?

I’m currently using GroupBy v1 still, the lookup values are propagating for domain and publisher, but not country. I tried querying the inner query by itself, and I confirmed that it is indeed getting transformed properly.

{

“queryType”: “groupBy”,

“dataSource”: {

“type”: “query”,

“query”: {

“queryType”: “groupBy”,

“dataSource”: “dataSource”,

“granularity”: “all”,

“dimensions”: [

{

“dimension”: “country_id”,

“outputName”: “country”,

“name”: “country_name_by_id_lookup_v2”,

“type”: “lookup”,

“replaceMissingValueWith”: null,

“retainMissingValue”: false,

“injective”: false,

“optimize”: true

},

{

“dimension”: “domain_id”,

“outputName”: “domain_name”,

“name”: “domain_lookup_v2”,

“type”: “lookup”,

“replaceMissingValueWith”: null,

“retainMissingValue”: false,

“injective”: false,

“optimize”: true

},

{

“dimension”: “publisher_id”,

“outputName”: “publisher_name”,

“name”: “publisher_lookup_v2”,

“type”: “lookup”,

“replaceMissingValueWith”: null,

“retainMissingValue”: false,

“injective”: false,

“optimize”: true

}

],

“filter”: {

“type”: “and”,

“fields”: [

{

“type”: “in”,

“dimension”: “country_id”,

“values”: [

“6252001”

],

“extractionFn”: null

}

]

},

“aggregations”: [

{

“type”: “longSum”,

“name”: “ad_requests”,

“fieldName”: “impressions”

}

],

“postAggregations”: ,

“intervals”: [

“2017-04-01T04:00:00+00:00/2017-04-26T03:59:59+00:00”

]

}

},

“granularity”: “all”,

“dimensions”: [

{

“type”: “default”,

“dimension”: “country”,

“outputName”: “country_name”

},

{

“type”: “default”,

“dimension”: “domain_name”,

“outputName”: “domain_name”

},

{

“type”: “default”,

“dimension”: “publisher_name”,

“outputName”: “publisher_name”

}

],

“aggregations”: [

{

“type”: “longSum”,

“name”: “ad_requests”,

“fieldName”: “ad_requests”

}

],

“intervals”: [

“2017-04-01T04:00:00+00:00/2017-04-26T03:59:59+00:00”

]

}

Also currently using version Druid version 0.9.2

Does the query work as expected with groupBy v2?

We still have not reconfigured Druid for GroupBy V2, so I haven’t tried yet. Although we are planning on upgrading at some point soon.

We still have not reconfigured Druid for GroupBy V2, so I haven’t tried yet. Although we are planning on upgrading at some point soon.

Hmm, I tried writing a unit test with an inner query that has lookup dimension specs, and it passed OK. Are you able to get the problem you’re seeing reduced to a test case? It could be a unit test in GroupByQueryRunnerTest, or it could be a dataset + a query that others could use to reproduce the problem. This is the one I wrote that did pass: https://gist.github.com/gianm/97c08f7910cae1a08bec4e1cbae509a3

Let’s move this to a github issue too, please file one here and link to this discussion thread: https://github.com/druid-io/druid/issues

I’m going to play around with it a bit more, maybe I messed something up. If I cannot resolve I’ll file a github issue.

Thanks!