Cannot ingest data from s3 in a different region than the Druid installation

I have been trying to ingest data from a different aws region than the druid installation and it is failing with the following:

Error: Failed to sample data: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: 1ZDYZGH4JZV5V4KS; S3 Extended Request ID: op5gLNzAkRscuSHJHcy+4vJk3wHClhnTNbTyQS8HxOJDS77N8JTp5Xszx38pZh13iAeWoWk5IcI=; Proxy: null), S3 Extended Request ID: op5gLNzAkRscuSHJHcy+4vJk3wHClhnTNbTyQS8HxOJDS77N8JTp5Xszx38pZh13iAeWoWk5IcI=

I need to support ingestion from multiple regions. I looked at the documentation and other threads and could not find a solution. The solutions that they propose include adding jvm parameters, but from my understanding that will be for the entire druid cluster and not for the specific ingestion spec. Any ideas?

Hi Alon,

You could try using

druid.s3.forceGlobalBucketAccessEnabled=true

in the common runtime file. This will obviously apply at the cluster level but will allow you to use buckets not in the default region.

Now I am getting the following error:

Error: Failed to sample data: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: X9XE35NHN29VY1NB; S3 Extended Request ID: aNppwZRPN1LCbbKFat60ZXnfJGEFj0uUhI0vpeYFJnizsxfQr89MQaJCvNJmC0x3mNGBBCU9hGI=; Proxy: null), S3 Extended Request ID: aNppwZRPN1LCbbKFat60ZXnfJGEFj0uUhI0vpeYFJnizsxfQr89MQaJCvNJmC0x3mNGBBCU9hGI=

I validated that I have permission to the bucket. And the key exists

hi Alon,

Have you checked the S3 druid authentication page to make sure you are using the most appropriate method for your environment?

My mistake… I was adding the prefix to the uri’s section not to prefixes. Works great! Thanks for helping a druid newbie.

You bet! Glad I was able to help.