How to use IAM roles to configure AWS S3 as Deep Storage?

Hi Everyone,

I have been trying to setup S3 as deep storage using IAM roles. Haven’t had much success.

Have followed the following thread as well: https://groups.google.com/forum/#!topic/druid-user/Lu_3XDi2l4w

It also doesn’t seem to work for me.

As I have been looking for the correct configuration to setup the same & haven’t found anywhere documented.

It’d be really great if anyone can help me with the same.

Note: Have also attached log file which contains exception that I’m getting on Middle Manager nodes while it is trying to handoff segments to Deep Storage.

Thanks in advance!

prod_S3_error.txt (13.1 KB)

I have gone thru this pain too. Please check out this post (the last comment from Karthik).

https://groups.google.com/forum/#!searchin/druid-user/s3%7Csort:date/druid-user/kHYBSuAJ_gI/2nw_IjiYCgAJ

You basically can omit the access key and secret key from the config file.

Hope this helps.

Thanks for your reply, cklh28.

Actually, I connected with Karthik to debug the issue. The issue was that I had given selected number of privileges to IAM_user_role for the S3 bucket. Indeed, it requires all(as there is no exhaustive list of privileges documented to use) the privileges. That solved our issue.

I’ll be posting another question in forum to get to know what all specific privileges need to be assigned to the IAM_user_role. As providing all the action privilege is not recommended in production.