namespace lookup not working - NamespacedExtractionModule not loading (0.8.2)

Trying to get namespace lookups working, and I’m able to get them to work as expected on my workstation. However, when running it in a different environment (a QA environment), even though I’m using essentially the same configuration, the lookup query fails. Looking at the historical node log, I see that on my workstation the NamespacedExtractionModule is loaded and does stuff. On the QA machine though, it never shows up in the logs at all. No errors, no clues.

I’m really stumped. I expect this may be difficult to diagnose by forum post, but any advice about where to look would be greatly appreciated! One thing is that the QA environment may not be able to connect to a remote repository via http. But I have remote repositories set to “”. (If that is the issue I would be fine with manually loading them up and/or copying them from some other place to the QA host, but I have no idea what/where to look for that.)

I am running a “baby” instance where I’m starting all the nodes on a single host, as support for development and testing… version is 0.8.2. Debug logging is on.

Here is the log output on the QA host where it fails, as filtered via “grep -i namespace logs/historical.log”:

2016-03-23T21:57:53,066 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-examples, io.druid.extensions:druid-kafka-eight, io.druid.extensions:druid-namespace-lookup, io.druid.extensions:druid-s3-extensions], defaultVersion=‘0.8.2’, localRepository=‘extensions-repo’, remoteRepositories=}]
2016-03-23T21:57:57,484 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-namespace-lookup] for class[io.druid.cli.CliCommandCreator]
Resolving artifact io.druid.extensions:druid-namespace-lookup:pom:0.8.2
Resolved artifact io.druid.extensions:druid-namespace-lookup:pom:0.8.2 from /site/zookeeper/zookeeper-3.4.6/extensions-repo (simple)
Resolving artifact io.druid.extensions:druid-namespace-lookup:jar:0.8.2
Resolved artifact io.druid.extensions:druid-namespace-lookup:jar:0.8.2 from /site/zookeeper/zookeeper-3.4.6/extensions-repo (simple)
2016-03-23T21:57:58,813 INFO [main] io.druid.initialization.Initialization - Added URL[file:/site/zookeeper/zookeeper-3.4.6/extensions-repo/io/druid/extensions/druid-namespace-lookup/0.8.2/druid-namespace-lookup-0.8.2.jar]
2016-03-23T21:58:00,244 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-namespace-lookup] for class[io.druid.initialization.DruidModule]

You can see that it is loading druid-namespace-lookup. But then, nothing more.

Here is the log output on my workstation where it succeeds, with the same “grep” (a few repeated lines omitted):

2016-03-23T21:33:02,740 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-examples, io.druid.extensions:druid-kafka-eight, io.druid.extensions:druid-namespace-lookup], defaultVersion=‘0.8.2’, localRepository=‘extensions-repo’, remoteRepositories=}]
2016-03-23T21:33:06,732 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-namespace-lookup] for class[io.druid.cli.CliCommandCreator]
Resolving artifact io.druid.extensions:druid-namespace-lookup:pom:0.8.2
Resolved artifact io.druid.extensions:druid-namespace-lookup:pom:0.8.2 from /site/zookeeper/zookeeper-3.4.6/extensions-repo (simple)
Resolving artifact io.druid.extensions:druid-namespace-lookup:jar:0.8.2
Resolved artifact io.druid.extensions:druid-namespace-lookup:jar:0.8.2 from /site/zookeeper/zookeeper-3.4.6/extensions-repo (simple)
2016-03-23T21:33:08,397 INFO [main] io.druid.initialization.Initialization - Added URL[file:/site/zookeeper/zookeeper-3.4.6/extensions-repo/io/druid/extensions/druid-namespace-lookup/0.8.2/druid-namespace-lookup-0.8.2.jar]
2016-03-23T21:33:09,157 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-namespace-lookup] for class[io.druid.initialization.DruidModule]
2016-03-23T21:33:09,176 INFO [main] io.druid.initialization.Initialization - Adding remote extension module[io.druid.server.namespace.NamespacedExtractionModule] for class[io.druid.initialization.DruidModule]
2016-03-23T21:33:12,367 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.NamespaceLookupStaticConfig] from props[druid.query.extraction.namespace.] as [io.druid.server.initialization.NamespaceLookupStaticConfig@30364216]
2016-03-23T21:33:13,137 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.namespace.NamespacedExtractionModule$NamespaceStaticConfiguration.start()] on object[io.druid.server.namespace.NamespacedExtractionModule$NamespaceStaticConfiguration@42a0501e].
2016-03-23T21:33:13,137 INFO [main] io.druid.server.namespace.NamespacedExtractionModule - Loading configuration as static configuration
2016-03-23T21:33:13,140 DEBUG [main] io.druid.server.namespace.cache.NamespaceExtractionCacheManager - Deleting 0 namespaces:
2016-03-23T21:33:13,141 DEBUG [main] io.druid.server.namespace.cache.NamespaceExtractionCacheManager - Trying to update namespace [locationName_lookup]

2016-03-23T21:33:13,165 INFO [main] io.druid.server.namespace.NamespacedExtractionModule - Loaded 6 namespace-lookup configuration
2016-03-23T21:33:13,247 INFO [NamespaceExtractionCacheManager-0] io.druid.server.namespace.URIExtractionNamespaceFunctionFactory - Finished loading 514 lines for namespace [locationName_lookup]

And here is my common.runtime.properties (slightly edited to protect the innocent):

Well, it does appear that it was an issue with that particular host not being able to pull needed dependencies. We weren’t able to determine exactly what it was missing. But my colleague used “tools pull-deps” to grab all needed dependencies, then we just included the preloaded dependencies as part of our deployment package, and the problem was fixed.

I gather from the release notes that 0.9.0 will always load dependencies from local file system, so it’s kind of a moot point, but maybe this information will be helpful if anyone else is having similar troubles with an older version.