Upgrade from 0.12 to 0.14 batch indexation on a hadoop cluster are failing because of the AWS region

Hello,

I use our local hadoop (Cloudera) cluster to do batch ingestion of daily segments into a Druid datasource. Our deep storage is S3.

I recently upgraded our Druid cluster from 0.12 to 0.14 in our staging environment and noticed the changes in the AWS Library. I followed all the instructions here: http://druid.io/docs/latest/development/extensions-core/s3.html

All our Druid components have the aws.region setting in the JVM file as well as the druid.indexer.runner.javaOpts in the middle manager. In the logs of the indexation task I even see it propagated as such:

2019-05-25T05:12:11,468 INFO [main] org.apache.druid.cli.CliPeon - Starting up with processors[8], memory[251,133,952], maxMemory[1,908,932,608], directMemory[1,908,932,608].
__2019-05-25T05:12:11,470 INFO [main] org.apache.druid.cli.CliPeon - * aws.region: ca-central-1__
2019-05-25T05:12:11,470 INFO [main] org.apache.druid.cli.CliPeon - * awt.toolkit: sun.awt.X11.XToolkit
2019-05-25T05:12:11,471 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter: composing
2019-05-25T05:12:11,471 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.composing.emitters: ["logging", "http", "statsd", "kafka"]
2019-05-25T05:12:11,471 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.http.recipientBaseUrl: XXXXX
2019-05-25T05:12:11,471 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.kafka.alert.topic: druid_alerts
2019-05-25T05:12:11,471 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.kafka.bootstrap.servers: XXX YYY ZZZ
2019-05-25T05:12:11,472 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.kafka.clusterName: main_druid
2019-05-25T05:12:11,472 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.kafka.metric.topic: druid_metrics
2019-05-25T05:12:11,472 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.logging.logLevel: debug
2019-05-25T05:12:11,472 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.hostname: localhost
2019-05-25T05:12:11,472 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.statsd.port: 8125
2019-05-25T05:12:11,473 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.directory: extensions
2019-05-25T05:12:11,473 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.hadoopDependenciesDir: hadoop-dependencies
2019-05-25T05:12:11,473 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.loadList: ["mysql-metadata-storage", "druid-s3-extensions", "druid-lookups-cached-global", "statsd-emitter", "druid-avro-extensions", "druid-kafka-indexing-service", "druid-parquet-extensions", "kafka-emitter"]
2019-05-25T05:12:11,473 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.remoteRepositories: []
2019-05-25T05:12:11,473 INFO [main] org.apache.druid.cli.CliPeon - * druid.host: XXXX
2019-05-25T05:12:11,473 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.kill.durationToRetain: 604800000
2019-05-25T05:12:11,474 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.kill.enabled: true
2019-05-25T05:12:11,474 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.s3Bucket: XXXX
2019-05-25T05:12:11,474 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.s3Prefix: XXXX
2019-05-25T05:12:11,474 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.type: s3
__2019-05-25T05:12:11,474 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.runner.javaOpts: -server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Daws.region=ca-central-1__

The indexation job proceeds and then fails at the end with this exception:

Error: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error in custom provider, com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
  at org.apache.druid.storage.s3.S3StorageDruidModule.getAmazonS3Client(S3StorageDruidModule.java:132) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.s3.S3StorageDruidModule)
  at org.apache.druid.storage.s3.S3StorageDruidModule.getAmazonS3Client(S3StorageDruidModule.java:132) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.s3.S3StorageDruidModule)
  while locating org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3
    for the 1st parameter of org.apache.druid.storage.s3.S3DataSegmentPusher.<init>(S3DataSegmentPusher.java:57)
  while locating org.apache.druid.storage.s3.S3DataSegmentPusher
  at org.apache.druid.storage.s3.S3StorageDruidModule.configure(S3StorageDruidModule.java:109) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.s3.S3StorageDruidModule)
  while locating org.apache.druid.segment.loading.DataSegmentPusher annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=89, type=MAPBINDER, keyType=java.lang.String)
  at org.apache.druid.guice.PolyBind.createChoice(PolyBind.java:71) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.LocalDataStorageDruidModule)
  while locating org.apache.druid.segment.loading.DataSegmentPusher

1 error
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028)
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
        at org.apache.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:126)
        at org.apache.druid.indexer.HadoopDruidIndexerMapper.setup(HadoopDruidIndexerMapper.java:53)
        at org.apache.druid.indexer.IndexGeneratorJob$IndexGeneratorMapper.setup(IndexGeneratorJob.java:322)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:371)
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:337)
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
        at org.apache.druid.storage.s3.S3StorageDruidModule.getAmazonS3Client(S3StorageDruidModule.java:148)
        at org.apache.druid.storage.s3.S3StorageDruidModule$$FastClassByGuice$$3f7cfccc.invoke(<generated>)
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
        at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
        at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)
        at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)
        at com.google.inject.multibindings.MapBinder$RealMapBinder$ValueProvider.get(MapBinder.java:821)
        at org.apache.druid.guice.PolyBind$ConfiggedProvider.get(PolyBind.java:202)
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)

I even tried to set it in the jobProperties of the indexation json (which is not a decent permanent solution as the indexation requester is not concerned with where the deep storage of Druid is) like so (exerpt):

"jobProperties": {
          "aws.region": "ca-central-1",
          "mapreduce.job.jvm.numtasks": "20",
          "mapreduce.map.memory.mb": "2048",
          "mapreduce.map.java.opts": "-server -Xmx2048m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps",
          "mapreduce.reduce.memory.mb": "2048",
          "mapreduce.reduce.cpu.vcores": "1",
          "mapreduce.reduce.java.opts": "-server -Xmx13312m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps",
....

But that did not seem to change anything. What am I missing?

Thanks!

C

Hey Cherif,

In between those versions (I think it was in 0.13) we switched from jets3t to the aws sdk. You now need to configure a region, either through JVM system property or environment variable. See here for details: http://druid.io/docs/latest/development/extensions-core/s3.html

Hey Gian,

Thanks for your reply. As I mentioned above I already did that. If you look at some of the logs I pasted, you can even see the property propagated.

Looking fwd to hear your thoughts.

C