thanks for your reply Eric. Sorry for the late response.
To make it easier to read my replies without too much confusion, I have tried to state more explicitly whether I’m talking about the query cache or the lookup cache and I furthermore capitalize query/lookup so that its easier to follow along.
So, by the way, I was mostly referring to the query caching in my post, not the lookup caches.
I tried to verify whether the optimization in the QueryToolChest for injective lookups works, but I do not observe it so far. I attached a query that I used for testing. I submit it and it has a dimension split on an injective lookup. I declared it to be injective but see that after each poll period, Druid recomputes the query.
First what version of druid are we talking about ?
We are running Druid 0.9.1.1 with a patch that uses guice 4.1 because the older guice version would mask stack traces
Currently, both the legacy lookup extension and the lookups-cached-global extensions are loaded and have lookups defined as a leftover from the transition phase but only the lookups-cached-global lookups are in use.
Second What kind of lookup are you using ? can you share the spec of the lookup ?
We use lookups-cached-global and have two kinds of lookups:
One kind is based on mapping files residing in S3
All low-cardinality lookups have been migrated to map-based lookups that look like this:
The S3 based lookups have poll periods of 3 hours and we observe that Druid doesn’t find queries that make use of these lookups within the QUERY caches between poll periods.
For the map-based lookups we observe that Druid can make use of the cached entries in the QUERY caches for longer than the poll period (of course). So that’s nice, but I still need to test whether the cache keys stay the same even after cluster restarts.
FYI having one to one mapping doesn’t imply that after every lookup poll the cache key is the same.
If you are sure that the lookup doesn’t change over the life cycle of the druid process then set the poll period to ZERO that will ensure that you will have the same version always.
I’m trying to make Druid use this optimization that you brought to my attention, namely, that the historical nodes don’t even get to know about the lookup because the broker already rewrites the query such that the historicals query and merge based on a dimension and the broker then translates the results via a lookup mapping. Prerequisites for this seem to be a one to one mapping definition and numeric sort order.
I made a query that features both but I still observe that Druid recomputes the query instead of fetching the intermediate results from its QUERY caches after about 3 hours which is what we have set as a poll period. So it seems that the cache keys in the QUERY caches are sill containing lookup-related tokens which differ across poll periods.
Setting the poll period to zero doesn’t unfortunately work for our usecases because our lookup definitions ocasionally change.
The poll period to my understanding tells Druid how often to check for updates to a mapping file. Each time Druid checks, I figure that there might be two outcomes: either the content of the lookup mapping file has changed since the last poll or it hasn’t.
By “robust behaviour” I meant that Druid ideally would only produce different cache keys for the QUERY cache if it detects that the contents of a mapping file actually changed. Polling doesn’t mean that they did, only that Druid should check.
I’d like it better if instead a strong consistent hash value would be computed and used in the cache key.
not sure i am getting this ? which cache key ? as part of the API every lookup need to return a cache key, i am i missing something ?
Sorry. my bad. I’m mostly concerned with whats going on in the QUERY cache. The cache keys that get created for the QUERY cache and that refer to a lookup contain some notion of a “revision”. Now, depending on which KIND of lookup that cache key is based on, it makes sense to use different schemes for what this revision is.
The usecase for file-based lookups (S3) is that they are high-cardinality and might change occasionally. Ideally, Druid would change the revision ID in its QUERY cache keys for that lookup if and only if the contents of the mapping file actually changed.
The more interesting case is map-based lookups, so these inline maps one can define. It would be sad if the QUERY cache keys for those lookups would change every time somebody submits some change to a lookup spec or the cluster restarts because as long as the contents of a mapping doesn’t change, the cache keys should also stay the same. This seems to be already the case as the whole content of the mappin becomes the cache key. An alternative way of giving the operator control would be to let him just specify a revision ID along with the mapping definition. In the case of inline-mappings, the usecase is probably that these lookups only change if a human manually modifies them.
Then he could also bump up the revision id. The downside of this is clear, the upside would be to not have long cache keys anymore.
To get rid of the downside of needing to remember to update the revision id manually, one could instead use a strong hash algorithm, such as a maximum entropy hash computed over the whole key-value content.
Even better would be if one could manually declare a version for each lookup and this version alone would make it into the cache key.
how is that different from hashing the entries ?
Good question. I assumed that some developers would regard the use of a hash as not safe enough in that they might assume that it could happen that the content changes while the hash key stays the same. Alternatively, it might be the case that computing the hash might be considered too expensive. My intuition to both things is otherwise, but for those who have these concerns, it might be nice to offer a revision Id that can be manually controlled by operators. In that case, I would be in control over when cache keys would change and I would change them only when I know that my lookup content has changed. Caching in Druid is highly complex and that brings with it corner cases in which Druid would behave differently from what operators expect. Simple is beautiful. If I could know that Druid would just take my lookup revision as that part of the QUERY cache-key which needs to change with each update to the lookup content, then I can be more confident about whats going on inside Druid while I’m in investigation mode.
Sure, if one forgets to update that version one would get corrupt results back, but it would give users more control over caching.
If I know that the contents of a lookup don’t change, I don’t increase the version and vice versa.
if the lookup doesn’t change set the poll to ZERO.
The lookup period only tells Druid how often to check for updates. So, for a time-range spanning 10 lookups, the mapping data might change only once. So I would want Druid to detect this one update which would not happen with poll ZERO and at the same time I would expect Druid in the ideal case to also detect for the other 8 times that the content did not change and not update its QUERY cache keys.
With regards to the lookup cache, perhaps one could have an additional check during each poll for whether the content really changed.
That will be very expensive in some cases. the idea behind the poll is to be agnostic to what was there in the cache before.
I think I’ve meanwhile seen that the last-modification timestamp is being checked. If so, maybe I could try to optimize things within our application that provisions the files in S3. Computing a strong hash based on the content is not expensive though, right? Druid reads the whole content anyways and puts it either into heap or offheap mem.
BTW for JDBC cached global you can define a ‘tsColumn’ which is the column in table which contains when the key was updated, BUT there is a nasty bug that need to be fixed before you can move to JDBC.
I’m curious to learn about it and about the new cached-single extension, but I don’t understand how individual revisions per lookup key-value mapping fits into the big picture of Druids lookup mechanism.
Our main and actually only objective is that a query that Druid has computed stays in a persistent remote QUERY cache for all times and never becomes stale. So far we observe that lookups interfere too strongly with this vision.
On this note, it is great that Druid can reuse some query cache entries. If I query the last 7 days and then extend the query to 30 days then the second query is different form the first and yet Druid can facilitate the cache entries that the first query produced because the cache entries are per segment.
What’s not so great is that if I next query for a subset of metrics, it would again be possible to serve the query up from QUERY cache but this Druis is not capable of doing.
It would be very desirable to improve this in the future either by caching per metric or by merging cache entries that match in filter and aggregation conditions and only differ on metrics within the cache.
Maybe then, it would also make sense to have an external application that submits occational queries to warm up the cache.