Getting AccessDeniedException while reading data from kinesis stream[where/how to keep aws credentials for middle manager tasks]

Hi,

I am trying ingesting data from kinesis stream.

Initial load data is happening properly, i,e, druid is able to parse data from DescribeStream API from Druid Load data.

But, during task, I am getting com.amazonaws.services.kinesis.model.AmazonKinesisException: User: arn:aws:iam::xxxxxxxxxxxxx:user/xxxxxx is not authorized to perform: kinesis:GetShardIterator on resource: arn:aws:kinesis:us-east-1:xxxxxxxxxxxxx:stream/xxxxxxxx (Service: AmazonKinesis; Status Code: 400; Error Code: AccessDeniedException;

kinesis credentials are added in _common/common_runtime_properties and middleManager/runtime.properties

as

druid.kinesis.accessKey=xxx

druid.kinesis.secretKey=xxxx

Followed this - https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#operations

My doubt, why describeStream is working in Load data and why tasks is not able to access stream due to authentication. Is there another way to specify aws credential. Please help.

Regards,

Paras

You should attach the right IAM profile to the EC2 instance running your services. IAM profiles is definitely the better approach to IAM users with access/secret keys.