How to use non-Amazon S3 for deep storage?

I am attempting to configure non-Amazon S3 deep storage for my druid cluster. I’ve configured my common-properties file with the following properties for S3 storage and commented out the local storage properties:

S3 access key.
Must be set.
S3 secret key.
Must be set.
Bucket to store in.
Must be set.
Base key prefix to use, i.e. what directory.
Must be set.

There is no mention of a property to specify for the S3 host, so based on searching past posts in the group I also added this property:


and created a file conf/druid/_common/ that contains the following:

Uncomment to use s3 compatibility mode on GCE




Despite all of this, after restarting all of the druid processes none of my data is being stored in S3 when I run my batch index job. Druid still appears to be using local storage since I can successfully query the data after loading. I don’t receive any errors during the load.

Any suggestions on how to load data to non-Amazon S3 deep storage would be much appreciated. I am running Druid version 0.10.0.

Have the same issue - any feedback would be great.

Lots of experimentation on difference configurations… no dice so far. Am able to write logs to S3, but can’t write index


Latest results are that I can get my log files to write to S3 but not the segments, which seems to be an issue others have faced. Just to be clear, I’m trying to connect to s3 on the endpoint “” and load segments into the bucket “mybucket”.

I’ve tried a few options:

  • Add an entry to the file for When I do this alone, then my ingestion task uses an incorrect segmentOutputPath and the job fails:

“segmentOutputPath” : “s3n://mybucket/druid/segments”

java.lang.Exception: java.lang.IllegalArgumentException: Invalid hostname in URI s3n://mybucket/druid/segments

  • Don’t use and just specify my S3 endpoint address as a prefix to my bucket in the file like If I do just this then it attempts to connect to amazon rather than my local s3 and the connection fails:
2017-08-01T20:58:32,391 DEBUG [pool-20-thread-1] org.jets3t.service.utils.RestUtils$ThreadSafeConnManager - Get connection: {s}->****:443, timeout = 0

2017-08-01T20:58:32,393 ERROR [pool-20-thread-1] io.druid.indexer.JobHelper - Exception in retry loop Connection refused (Connection refused)
  • Specify the endpoint in both locations listed above. If I do this, the job seems to make the most progress but eventually it still dies because it ends up duplicating the hostname like this:

ERROR [pool-20-thread-1] io.druid.indexer.JobHelper - Exception in retry loop Name or service not known

At this point, I’m running out of ideas. Is anyone out there willing to share a sample configuration where they are successfully storing segments on a non-Amazon instance of S3?

I’m about to the point of abandoning s3 as a backend data storage and writing to hadoop but I am hopeful that someone out there has a working config they are willing to share.

Hi Brandon,
did you also set in your

Also, you should add “druid-s3-extensions” to druid.extensions.loadList in your



2017년 8월 2일 (수) 오전 6:07, Brandon Dean engrdean@gmail.com님이 작성:

Yes, I have set all of the below properties in the file:









As I mentioned, the logs are being successfully stored in s3, it’s just the segments that I can’t get working.



I got this to work using pithos + cassandra. Here were my configs:






I haven’t yet configured HTTPS (this is an internal system, but I will still do that next). If your host supports https (via s3cmd’s ~/.s3cfg file, for instance, which is the default), then that should work.

My common.runtime.properities is not much different than yours:






I have noticed that you have the s3n above. I do have that when I submit my batch jobs, but not here in the common config file.

So all that used to work as-is until yesterday, where I switch to use minio, and now druid won’t store anything in S3. Need to get to the bottom of this one, but I thought I would let you guys know I had a successful setup for the last 2 months.

Joel was recently reported against master,
but maybe there was the same issue in old versions.