S3 Deep Storage with fileSessionCredntials - AccessDenied

Hi Everyone,

I'm trying to integrate S3 for Deep Storage. We have IAM - AssumeRoles with temporary STS tokens fetched every hour.
I have the plumbing done to write these tokens into ~/.aws/credentials file in the mesos workers. The S3 buckets are created and the policy entails to rw into these buckets for the role (assumed)

The common.runtime.properties file is configured as below.

For S3:

druid.storage.type=s3
druid.storage.bucket=
druid.storage.baseKey=segments/sample-data
druid.s3.fileSessionCredentials=/.aws/credentials

Any help here guys?

Hey Druiders,

I presume you guys should have ran into this. Can you point me to a direction here?

Best Regards

Varaga

It’s worth to inform you guys that I am using a 0.13.0-SNAPSHOT distribution built locally (that includes my PR fix for zookeeper)…

Figured out that this latest contains aws-java-sdk implementation.
The problem seems to be the fact that the region comes back as unidentified.

It is a bit of pain of to go through the code to find out what property needs to be set or how the code behaves…
It’d be great if the documentation is updated for new features added?

Some debug logs worth your look.

2018-04-27T09:18:46,231 INFO [appenderator_merge_0] io.druid.storage.s3.S3DataSegmentPusher - Pushing [/tmp/druid400080604628751070index.zip] to bucket[] and key[segments/s
ample-data/caliper-sample-s3-1field/2017-09-30T00:00:00.000Z_2017-10-01T00:00:00.000Z/2018-04-27T09:10:50.350Z/0/index.zip].
2018-04-27T09:18:46,232 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.internal.Mimetypes - Recognised extension ‘zip’, mimetype is: ‘application/zip’
2018-04-27T09:18:46,232 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.AmazonS3Client - Bucket region cache doesn’t have an entry for <<bucket>>. Trying to get bucket region
from Amazon S3.




2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0]
com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec - Connection can be kept alive for 60000 MILLISECONDS
2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 1][route: {s}->https://s3.amazonaws.com:443] can be kept alive for 60.0 seconds
2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 1][route: {s}->https://s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0] com.amazonaws.request - Received successful response: 200, AWS Request ID: null
2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0] com.amazonaws.requestId - x-amzn-RequestId: not available
2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0] com.amazonaws.requestId - AWS Request ID: not available
2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.AmazonS3Client - Not able to derive region of the hmheng-data-services/druid-segments/dev from the HEAD Bucket requests.
2018-04-27T09:18:46,241 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.AmazonS3Client - Region for <bucket> is null

``

Hi Varaga,

druid.s3.fileSessionCredentials=/.aws/credentials is not a valid configuration. If you have the ‘credentials’ file under /your/home/.aws/, Druid should be aware of it automatically.

Would you check that the region is set properly?

Jihoon

2018년 4월 27일 (금) 오전 3:34, Varaga chakravarthyvp@gmail.com님이 작성:

Thanks Jihoon

The S3 credentials file are picked up with that configuration and you can see that in the logs attached. The region is also set in config file under. /…/.aws/ directory. However this config file may not have been picked up.

Also it was the InstanceProfileProvider that was handling the credentials.

The process runs as root in an Ubuntu base (in house) image that sets the root user to a location in mesos mount.

Are you suggesting to run the process in into its own user and then sts tokens fetched as this user without specifying the fileSessionCredentials property

Hi Jihoon,

The **druid.s3.**fileSessionCredentials=~/.aws/credentials

was what was set.

I tried to set the region through the environment (exported variable) as well but it didn’t work. With the aws-cli assume role command the config file is fetched and written as ~/.aws/config

This should be autodetected by the aws client. However I see it is not!

How else do you recommend to set the region? Also, I’m not sure if this is the issue as well !

2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.headers - http-outgoing-4 << Transfer-Encoding: chunked
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.headers - http-outgoing-4 << Server: AmazonS3
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec - Connection can be kept alive for 60000 MILLISECONDS
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser - Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$AccessControlListHandler
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.wire - http-outgoing-4 << “23c[\r][\n]”
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.wire - http-outgoing-4 << “<?xml version="1.0" encoding="UTF-8"?>[\n]”
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.wire - http-outgoing-4 << “55a2a99348b42f91bd02796243cb3c4be7fc15c422defec115804ab3c93608e8bedrock-nonprod-aws55a2a99348b42f91bd02796243cb3c4be7fc15c422defec115804ab3c93608e8bedrock-nonprod-awsFULL_CONTROL[\r][\n]”
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.wire - http-outgoing-4 << “0[\r][\n]”
2018-04-30T10:10:19,008 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.wire - http-outgoing-4 << “[\r][\n]”
2018-04-30T10:10:19,009 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 4][route: {s}->https://s3.amazonaws.com:443] can be kept alive for 60.0 seconds
2018-04-30T10:10:19,009 DEBUG [appenderator_merge_0] com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 4][route: {s}->https://s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
2018-04-30T10:10:19,009 DEBUG [appenderator_merge_0] com.amazonaws.request - Received successful response: 200, AWS Request ID: F0245F222F551DF8
2018-04-30T10:10:19,009 DEBUG [appenderator_merge_0] com.amazonaws.requestId - x-amzn-RequestId: not available
2018-04-30T10:10:19,009 DEBUG [appenderator_merge_0] com.amazonaws.requestId - AWS Request ID: F0245F222F551DF8
2018-04-30T10:10:19,009 INFO [appenderator_merge_0] io.druid.storage.s3.S3DataSegmentPusher - Pushing [/tmp/druid3309108450302028265index.zip] to bucket[] and key[segments/sample-data/caliper-sample-s3-1field/2017-09-30T00:00:00.000Z_2017-10-01T00:00:00.000Z/2018-04-30T10:01:57.504Z/0/index.zip].
2018-04-30T10:10:19,009 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.internal.Mimetypes - Recognised extension ‘zip’, mimetype is: ‘application/zip’
2018-04-30T10:10:19,009 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.AmazonS3Client - Bucket region cache doesn’t have an entry for . Trying to get bucket region from Amazon S3.

2018-04-30T10:10:19,020 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.AmazonS3Client - Not able to derive region of the from the HEAD Bucket requests.

2018-04-30T10:10:19,020 DEBUG [appenderator_merge_0] com.amazonaws.services.s3.AmazonS3Client - Region for is null
2018-04-30T10:10:19,020 DEBUG [appenderator_merge_0] com.amazonaws.request - Sending Request: PUT https://s3.amazonaws.com /segments/sample-data/caliper-sample-s3-1field/2017-09-30T00%3A00%3A00.000Z_2017-10-01T00%3A00%3A00.000Z/2018-04-30T10%3A01%3A57.504Z/0/index.zip Headers: (x-amz-grant-full-control: id="", id="", User-Agent: aws-sdk-java/1.11.199 Linux/4.9.76-3.78.amzn1.x86_64 Java_HotSpot™_64-Bit_Server_VM/25.161-b12 java/1.8.0_161, amz-sdk-invocation-id: 2048b4ca-c713-1757-1fd2-49ef46e3a3a5, Content-Length: 37654, Content-MD5: 28eKWEVL6KllzpPX/Tkg5A==, Content-Type: application/zip, )
2018-04-30T10:10:19,020 DEBUG [appenderator_merge_0] com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '“PUT
segments/sample-data/caliper-sample-s3-1field/2017-09-30T00%3A00%3A00.000Z_2017-10-01T00%3A00%3A00.000Z/2018-04-30T10%3A01%3A57.504Z/0/index.zip
amz-sdk-invocation-id:2048b4ca-c713-1757-1fd2-49ef46e3a3a5
amz-sdk-retry:0/0/500
content-length:37654
content-md5:28eKWEVL6KllzpPX/Tkg5A==
content-type:application/zip
host:s3.amazonaws.com
user-agent:aws-sdk-java/1.11.199 Linux/4.9.76-3.78.amzn1.x86_64 Java_HotSpot™_64-Bit_Server_VM/25.161-b12 java/1.8.0_161
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20180430T101019Z
x-amz-grant-full-control:id="
”, id=""
x-amz-security-token:FQoDYXdzEOv/…

amz-sdk-invocation-id;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-grant-full-control;x-amz-security-token
UNSIGNED-PAYLOAD"
2018-04-30T10:10:19,020 DEBUG [appenderator_merge_0] com.amazonaws.auth.AWS4Signer - AWS4 String to Sign: '"AWS4-HMAC-SHA256
20180430T101019Z
20180430/us-east-1/s3/aws4_request


my common.runtime.properties file

druid.storage.type=s3
druid.storage.bucket=
druid.storage.baseKey=segments/sample-data
druid.s3.fileSessionCredentials=~/.aws/credentials
druid.s3.endpoint.url=s3.amazonaws.com
druid.s3.endpoint.serviceName=s3
druid.s3.endpoint.signingRegion=us-east-1

Hi Varaga,

which version of Druid are you using?

I also set my client region in /my/home/.aws/config file. Or, you should be able to set the region by 'export AWS_REGION=‘us-east-1’.

Jihoon

2018년 4월 30일 (월) 오전 6:08, Chakravarthy varaga chakravarthyvp@gmail.com님이 작성:

Hi Jihoon

I have sorted this. I had to set the AWS_CREDENTIALS_PROPERTIES_FILE env to they creds., File.

The problem was the ~(home) was the mesos slave directory where the token was fetched. The credentials provider in the have process was fetching the ‘user.home’ for the tokens and this was the process user ( root inside the container).

Thanks for your prompt responses. I guess some documentation around this would have been great. I’m using 13.0.snapshot version.

Best Regards

Varaga

Glad to hear that you solved the issue!

Jihoon

2018년 5월 1일 (화) 오전 1:05, Chakravarthy varaga chakravarthyvp@gmail.com님이 작성: