Unable to load AWS credentials from any provider in the chain

Hello there!

I am running a large Hadoop cluster on version 2.6.0 so none of my Hadoop indexing jobs work. I run into the jackson dependency issue. The issue has been documented several times:

https://groups.google.com/forum/#!msg/druid-user/UM-Cgj750sY/xWF94hTjt-AJ

https://groups.google.com/forum/#!msg/druid-development/jNxhMZpp-rc/XwAFP2xYe60J

In order to fix the jackson dependency issue, I am forced to build a fat Druid jar excluding the jackson dependency with sbt. I followed the solution documented here:

https://github.com/druid-io/druid/blob/master/docs/content/operations/other-hadoop.md

However, with this sbt built fat jar, I run into a really weird a random issue:

com.google.inject.ProvisionException: Guice provision errors:

  1. Error in custom provider, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain

at io.druid.storage.s3.S3StorageDruidModule.getRestS3Service(S3StorageDruidModule.java:107)

at io.druid.storage.s3.S3StorageDruidModule.getRestS3Service(S3StorageDruidModule.java:107)

while locating org.jets3t.service.impl.rest.httpclient.RestS3Service

for parameter 0 at io.druid.storage.s3.S3DataSegmentKiller.(S3DataSegmentKiller.java:43)

while locating io.druid.storage.s3.S3DataSegmentKiller

at io.druid.storage.s3.S3StorageDruidModule.configure(S3StorageDruidModule.java:84)

while locating io.druid.segment.loading.DataSegmentKiller annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=124, type=MAPBINDER)

at io.druid.guice.Binders.dataSegmentKillerBinder(Binders.java:24)

while locating java.util.Map<java.lang.String, io.druid.segment.loading.DataSegmentKiller>

for parameter 0 at io.druid.segment.loading.OmniDataSegmentKiller.(OmniDataSegmentKiller.java:36)

while locating io.druid.segment.loading.OmniDataSegmentKiller

at io.druid.cli.CliPeon$1.configure(CliPeon.java:142)

while locating io.druid.segment.loading.DataSegmentKiller

for parameter 4 at io.druid.indexing.common.TaskToolboxFactory.(TaskToolboxFactory.java:76)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:131)

while locating io.druid.indexing.common.TaskToolboxFactory

for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.(ThreadPoolTaskRunner.java:71)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.(QueryResource.java:90)

while locating io.druid.server.QueryResource

Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain

at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)

at io.druid.storage.s3.S3StorageDruidModule.getRestS3Service(S3StorageDruidModule.java:107)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:105)

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55)

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)

at com.google.inject.Scopes$1$1.get(Scopes.java:65)

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)

at com.google.inject.Scopes$1$1.get(Scopes.java:65)

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)

at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:90)

at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:90)

at com.google.inject.multibindings.MapBinder$RealMapBinder$2.get(MapBinder.java:389)

at com.google.inject.multibindings.MapBinder$RealMapBinder$2.get(MapBinder.java:385)

at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55)

at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)

at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)

at com.google.inject.Scopes$1$1.get(Scopes.java:65)

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)

at com.google.inject.Scopes$1$1.get(Scopes.java:65)

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)

at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)

at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)

at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:49)

at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)

at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)

at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)

at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)

at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)

at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)

at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)

at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)

at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)

at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)

at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036)

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:134)

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71)

at io.druid.cli.CliPeon.run(CliPeon.java:211)

at io.druid.cli.Main.main(Main.java:91)

``

I don’t use AWS at all so I am not sure why I am getting this issue. Must be an issue with the build. Here is my spec:

{

“type” : “index_hadoop”,

“spec” : {

“dataSchema” : {

“dataSource” : “api_created_trips_02”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column” : “SecondsSinceEpoch”,

“format” : “posix”

},

“dimensionsSpec”: {

“dimensions”: [

“uuid”,

“country_id”,

“city_id”,

“request_at”,

“status”

],

“dimensionExclusions”: ,

“spatialDimensions”:

}

}

},

“metricsSpec”: [

{

“type”: “doubleSum”,

“name”: “duration”,

“fieldName”: “duration”

}

],

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “HOUR”,

“intervals” : [“2015-12-05/2015-12-06”]

}

},

“ioConfig” : {

“type” : “hadoop”,

“inputSpec” : {

“type” : “static”,

“paths” : “hdfs://nameservice1/user/bwhite/api_created_trips_1day_json”

}

}

}

}

``

Thanks!

Hey bwhite,

If you aren’t using aws at all, try removing the “io.druid.extensions” % “druid-s3-extensions” and “com.amazonaws” % “aws-java-sdk” lines from that build.sbt linked in the docs (along with any other extensions you aren’t using).

Also, are you using a special build of Hadoop, or the stock Apache one? If it’s stock Apache could you give some more details about the Jackson related exception you saw? We are using Jackson 2.4.6 because we have seen it work well with most versions of Apache Hadoop, so it would be good to know if there’s somewhere that’s not working there.