Select query: problem with following pagingIdentifiers across segments [8.2]

Hi,

I’m running into a problem when trying to “walk” a Select query page by page using the paging identifiers returned by previous Select queries. Am I doing something wrong, or is this a bug? I’m on a fairly old version (8.2) so I’m hoping it’s an already-fixed bug, but I couldn’t find it in the github issue list. (We do plan to upgrade to the next release once it’s ready.)

(I’ve only seen this in a case where I have a couple of filters set, but I don’t know that they have any real bearing on things. I wonder if it might be just that they restrict the result set enough so that it falls below a certain threshold to trigger the behavior or something along those lines. Still, I’ll include them in my example queries below.)

The interval I’m using spans two segments in the data source. I start with a timeseries query that tells me there are 51 total results to be found:

{
“queryType”: “timeseries”,
“dataSource”: “myDataSource”,
“granularity”: “all”,
“aggregations”: [{“name”: “count”, “type”: “count”}],
“intervals”: [“2016-04-16T00:00:00.000Z/2016-04-18T00:00:00.000Z”],
“filter”: {
“type”: “and”,
“fields”: [
{“dimension”: “dim1”, “value”: “value”, “type”: “selector”},
{“dimension”: “dim2”, “value”: “value”, “type”: “selector”}
]
}
}

``

My first Select query, with a threshold of 40:

{
“queryType”: “select”,
“dimensions”:,
“metrics”:,
“dataSource”: “myDataSource”,
“granularity”: “all”,
“intervals”: [“2016-04-16T00:00:00.000Z/2016-04-18T00:00:00.000Z”],
“pagingSpec”: {
“pagingIdentifiers”: {},
“threshold”: 40
},
“filter”: {
“type”: “and”,
“fields”: [
{“dimension”: “dim1”, “value”: “value”, “type”: “selector”},
{“dimension”: “dim2”, “value”: “value”, “type”: “selector”}
]
}
}

``

It returns 37 rows from the first segment, 3 rows from the second segment, and these paging identifiers:

“pagingIdentifiers”: {
“myDataSource_2016-04-16T00:00:00.000Z_2016-04-17T00:00:00.000Z_2016-04-18T22:27:37.240Z”: 36,
“myDataSource_2016-04-17T00:00:00.000Z_2016-04-18T00:00:00.000Z_2016-05-16T21:14:24.599Z”: 2
}

``

Following the documentation, I increment the values in the paging identifier and include it in my next Select query (all other values in the query are the same as above):

{
“queryType”: “select”,
// …
“pagingSpec”: {
“pagingIdentifiers”: {
“myDataSource_2016-04-16T00:00:00.000Z_2016-04-17T00:00:00.000Z_2016-04-18T22:27:37.240Z”: 37,
“myDataSource_2016-04-17T00:00:00.000Z_2016-04-18T00:00:00.000Z_2016-05-16T21:14:24.599Z”: 3
},
“threshold”: 40
}
}

``

This returns 11 rows, bringing us to 51 total, so I should be at the end of the line. However, my RESTful service is not tracking that state and doesn’t know any better, so I build another Select query based on the new result’s paging identifiers:

“pagingIdentifiers”: {
“myDataSource_2016-04-17T00:00:00.000Z_2016-04-18T00:00:00.000Z_2016-05-16T21:14:24.599Z”: 13
}

``

Incrementing and querying again:

{
“queryType”: “select”,
// …
“pagingSpec”: {
“pagingIdentifiers”: {
“myDataSource_2016-04-17T00:00:00.000Z_2016-04-18T00:00:00.000Z_2016-05-16T21:14:24.599Z”: 14
},
“threshold”: 40
}
}

``

Instead of the expected empty result, instead I get back all 37 rows from the first segment, plus another paging identifier to blindly follow:

“pagingIdentifiers”: {
“myDataSource_2016-04-16T00:00:00.000Z_2016-04-17T00:00:00.000Z_2016-04-18T22:27:37.240Z”: 36
}

``

And building a query from that one returns all 14 rows from the second segment, plus the same pagingIdentifiers shown above. From this point following my procedure will just loop between the last two queries.

Is this a bug or am I doing something wrong?

Thanks,

Ryan

Hi Ryan,

I’m not super familiar with the internals of the Select query, but it sounds like you may be encountering a problem similar to this one, this was merged in 0.9.1:

“Select query cannot span to next segment with paging”

https://github.com/druid-io/druid/pull/2480

Thanks,

Jon

Jon -

Yes, it looks like that is the problem I am encountering. Thank you!

-Ryan

As noted in the pull request https://github.com/druid-io/druid/pull/2480 it does not appear that this bug is actually fixed in 0.9.1.1. I posted comments there about a workaround we are attempting.