IAM policy to connect to druid

Hi All,

I am a newbie to Apache Druid and aws. We have druid installed on ec2 instances of AWS. We have a shell script that ingests the data to druid. To connect to druid, we made use of access key and secret key. THe data was loaded successfully. We are now looking to ingest the data using the IAM roles. We created a IAM policy something like this:

Action:ec2:*
resources : *

When we give full permissions to ec2 , we are able to run our job successfully. However we want to know specific permissions that we should give ec2 to access the Druid. If anyone has come across this scenario kindly help us on this. If you can provide the list of Actions for ec2 that would be really helpful.

Welcome @Robin1! The AWS Authentication doc may be useful.

The policy attached to your IAM role has to contain the necessary permissions, and those permissions depend on the value of useListShards.

If useListShards is set to true:

  • ListStreams: required to list your data streams
  • Get*: required for GetShardIterator
  • GetRecords: required to get data records from a data stream’s shard
  • ListShards : required to get the shards for a stream of interest

Here’s a corresponding example policy:

[
  {
    "Effect": "Allow",
    "Action": ["kinesis:List*"],
    "Resource": ["*"]
  },
  {
    "Effect": "Allow",
    "Action": ["kinesis:Get*"],
    "Resource": [<ARN for shards to be ingested>]
  }
]

If useListShards is set to false:

  • ListStreams: required to list your data streams
  • Get*: required for GetShardIterator
  • GetRecords: required to get data records from a data stream’s shard
  • DescribeStream: required to describe the specified data stream

Here’s a corresponding example policy:

[
  {
    "Effect": "Allow",
    "Action": ["kinesis:ListStreams"],
    "Resource": ["*"]
  },
  {
    "Effect": "Allow",
    "Action": ["kinesis:DescribeStreams"],
    "Resource": ["*"]
  },
  {
    "Effect": "Allow",
    "Action": ["kinesis:Get*"],
    "Resource": [<ARN for shards to be ingested>]
  }
]

Let us know how it goes.

Thank you Mark for your response. We are using native batch ingestion(index_parallel) and in the ingestion spec we had given the access key and secret key. It works fine this way. We also have the access key and secret key stored in common_runtime.properties but the ask is now to remove the access key and secret key from these places and instead use IAM role. In the documentation, it is mentioned that the access key and secret key is to be placed in the common_runtime.properties. Is it possible that we can use IAM roles instead and if yes what permissions should we use for this ? Thank you again for looking into this.