How is druid.indexer.logs.s3Prefix used by druid

Apache Druid Q&A Google Groups
#1

Hi,

I am using a S3 API compatible service from IBM (object storage). I am able to set my deep storage to S3 and provide the required details and see data being pushed to my bucket on IBM cloud. This is done while indexing logs set to use local storage.

However, when i try to use S3 for both indexer logs and deep storage i see errors indicating that bucket doesn’t exists. What exactly is s3Prefix supposed to be? and why is this only required for indexer logs and not deep storage?

Druid is able to push data into the bucket i use for deepstorage so it is able to connect to the service. What is s3Prefix when using S3 for indexer logs?

Thanks,

Prathamesh

#2

druid.storage.bucket=bucket_1 (do not include s3:// etc)

#3

Hi Venkat,

Yes. I had set that property. I was confused about the purpose of druid.indexer.logs.s3Prefix. I thought this is something that S3 compatible API service would have to provide.

I later learned that it can be any value and would act just as a prefix when logs are pushed to deep storage. I am able to use a S3 compatible API as my deep storage now.

Thanks,

Prathamesh