Need help with IAM role configuration for s3 extension

Hi,

I know that [Glenn Nethercutt] has, way back, contributed a change which allowed usage of an IAM role instead of an access key/secret key combo in the properties file. I was able to use s3 using access/secret key but now trying to switch over to IAM role… I need some help. I see the stack trace below -

java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8028E25792B43D64; S3 Extended Request ID: Ul1PU1aqZuF0Q1w9H6WdXzMeXb1Sh6KCFVMSyxym7m4m04o7BVfEE1HL4ZUiCTDFDmb/yejNErQ=), S3 Extended Request ID: Ul1PU1aqZuF0Q1w9H6WdXzMeXb1Sh6KCFVMSyxym7m4m04o7BVfEE1HL4ZUiCTDFDmb/yejNErQ=

at org.apache.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:122) ~[?:?]

at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:740) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:736) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:623) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]

at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]

at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8028E25792B43D64; S3 Extended Request ID: Ul1PU1aqZuF0Q1w9H6WdXzMeXb1Sh6KCFVMSyxym7m4m04o7BVfEE1HL4ZUiCTDFDmb/yejNErQ=)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.199.jar:?]

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229) ~[aws-java-sdk-s3-1.11.199.jar:?]

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176) ~[aws-java-sdk-s3-1.11.199.jar:?]

at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3381) ~[aws-java-sdk-s3-1.11.199.jar:?]

at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1160) ~[aws-java-sdk-s3-1.11.199.jar:?]

at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1150) ~[aws-java-sdk-s3-1.11.199.jar:?]

at org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3.getBucketAcl(ServerSideEncryptingAmazonS3.java:70) ~[?:?]

at org.apache.druid.storage.s3.S3Utils.grantFullControlToBucketOwner(S3Utils.java:199) ~[?:?]

at org.apache.druid.storage.s3.S3DataSegmentPusher.uploadFileIfPossible(S3DataSegmentPusher.java:165) ~[?:?]

at org.apache.druid.storage.s3.S3DataSegmentPusher.lambda$push$0(S3DataSegmentPusher.java:110) ~[?:?]

at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]

at org.apache.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:82) ~[?:?]

at org.apache.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:108) ~[?:?]

… 11 more

``

I have done the following -

  1. Set up an s3 bucket with AES-256 encryption.

  2. Created an IAM role which has Get/Put/Delete permissions on this bucket.

  3. Created an ec2 instance (where I am running my druid stack, my working stack is a three node cluster but now to test, its on a single node) and attacked the IAM role to this instance.

  4. Added the below in my properties file

druid.storage.type=s3

druid.storage.bucket=

druid.storage.baseKey=druid/segments

druid.storage.sse.type=s3

druid.s3.fileSessionCredentials=

druid.indexer.logs.type=s3

druid.indexer.logs.s3Bucket=

druid.indexer.logs.s3Prefix=druid/indexing-logs

``

All the indexing logs data is making its way no problem, just the segment data is not coming through. What am I missing here?

Btw, picked up the fileSessionCredentials bit (setting it to a role name) from this thread - https://groups.google.com/forum/#!topic/druid-user/Lu_3XDi2l4w

I feel I am missing something basic, would really appreciate nudge in the right direction.

Fixed it…

Basically, this is what I did

  1. I removed all IAM User/ role based references from the common.runtime.properties file.
  2. Created jets3t.properties in _common folder with below contents
    s3service.s3-endpoint=s3.us-east-1.amazonaws.com

s3service.https-only=true

s3service.s3-endpoint-https-port=443

s3service.server-side-encryption=AES256

``

  1. Created a role with full access to s3 bucket

  2. Attached the IAM role to the EC2 instance hosting druid.

Can someone tell me what exact permissions are needed for using s3 as deep storage? I had these but apparently its not enough (as full access works fine)

“s3:ListBucket”,

“s3:GetBucketTagging”,

“s3:HeadBucket”,

“s3:GetObject”,

“s3:GetObjectTagging”,

“s3:ListObjectsV2”,

“s3:ListObjects”,

“s3:ListBucketByTags”,

“s3:ListBucketMultipartUploads”,

“s3:ListMultipartUploadParts”,

“s3:PutBucketTagging”,

“s3:PutObject”,

“s3:PutObjectTagging”,

“s3:DeleteObject”,

“s3:DeleteObjectTagging”

``

1 Like