Safe reloading of lookups?


we run Druid in production and use the lookups-cached-global extension to rewrite ids to labels. Some of our lookup maps are map-based but others lookup maps reside as CSV files in Amazon S3 and get reloaded/refreshed once per hour.

We seem to see that the NamespaceExtractionCacheManager first unloads all namespaces and then loads them again.
If queries come in in between, then they receive a “Namespace vanished” exception.

To my understanding, this is a current shortcoming in how Druid manages lookups or is there a way to get Druid to refresh lookups in a non-disruptive way?

Is there anything in Druid 0.9.2 or 0.10.0 or on the roadmap that improves this situation?
Is the lookups-cached-single extension also prone to these disruptions?


Hi Sascha,

one of the reason we added the single cached lookup is to reduce the lookups lifecycle inter dependency.

So technically polling strategy of single cached extension should meet your expectations, but there is only support for jdbc lookups.

I can not speak for the global cached, but i know that there was multiple fixes to enhance locking management.