Druid on GCS Missing Config Key

Hi Folks,

I’m trying to use the GCS connector to link the Druid indexer with GS.

I have specified the fs.gs.project.id in the core-site.xml file, but I’m still receiving the below error. Does anyone have some insight?

Thanks so much!

2017-09-01T16:53:23,908 INFO [main] io.druid.storage.google.GoogleStorageDruidModule - Configuring GoogleStorageDruidModule…

Sep 01, 2017 4:53:23 PM com.google.inject.internal.MessageProcessor visit

INFO: An exception was caught and reported. Message: java.io.IOException: Must supply a value for configuration setting: fs.gs.project.id

java.lang.RuntimeException: java.io.IOException: Must supply a value for configuration setting: fs.gs.project.id

at com.google.common.base.Throwables.propagate(Throwables.java:160)

at io.druid.indexer.HadoopDruidIndexerConfig.fromDistributedFileSystem(HadoopDruidIndexerConfig.java:205)

at io.druid.cli.CliInternalHadoopIndexer.getHadoopDruidIndexerConfig(CliInternalHadoopIndexer.java:164)

at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:90)

at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

at com.google.inject.spi.Elements.getElements(Elements.java:110)

at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)

at com.google.inject.AbstractModule.configure(AbstractModule.java:62)

at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

at com.google.inject.spi.Elements.getElements(Elements.java:110)

at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:177)

at com.google.inject.AbstractModule.configure(AbstractModule.java:62)

at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)

at com.google.inject.spi.Elements.getElements(Elements.java:110)

at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)

at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)

at com.google.inject.Guice.createInjector(Guice.java:99)

at com.google.inject.Guice.createInjector(Guice.java:73)

at com.google.inject.Guice.createInjector(Guice.java:62)

at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:370)

at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:63)

at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:109)

at io.druid.cli.Main.main(Main.java:108)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:117)

at io.druid.cli.Main.main(Main.java:108)

Caused by: java.io.IOException: Must supply a value for configuration setting: fs.gs.project.id

at com.google.cloud.hadoop.util.ConfigurationUtil.getMandatoryConfig(ConfigurationUtil.java:39)

at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.createOptionsBuilderFromConfig(GoogleHadoopFileSystemBase.java:2175)

at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.configure(GoogleHadoopFileSystemBase.java:1822)

at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:1003)

at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:966)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2316)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)

at io.druid.indexer.HadoopDruidIndexerConfig.fromDistributedFileSystem(HadoopDruidIndexerConfig.java:193)

… 27 more

Hmm, I’m not sure why it wouldn’t be picked up from there, but maybe try adding it to the jobProperties too?

We use the following core-site.xml which works fine for us: https://gist.github.com/erikdubbelboer/ef724d68b77c234a913058d903d2bcf9