Getting AccessDeniedException while reading data from kinesis stream[where/how to keep aws credentials for middle manager tasks]


I am trying ingesting data from kinesis stream.

Initial load data is happening properly, i,e, druid is able to parse data from DescribeStream API from Druid Load data.

But, during task, I am getting User: arn:aws:iam::xxxxxxxxxxxxx:user/xxxxxx is not authorized to perform: kinesis:GetShardIterator on resource: arn:aws:kinesis:us-east-1:xxxxxxxxxxxxx:stream/xxxxxxxx (Service: AmazonKinesis; Status Code: 400; Error Code: AccessDeniedException;

kinesis credentials are added in _common/common_runtime_properties and middleManager/




Followed this -

My doubt, why describeStream is working in Load data and why tasks is not able to access stream due to authentication. Is there another way to specify aws credential. Please help.



You should attach the right IAM profile to the EC2 instance running your services. IAM profiles is definitely the better approach to IAM users with access/secret keys.