Using Druid with Google Cloud Storage

Hello,

Is it possible to configure Druid to work with GCS (Google Cloud Storage) as a deep storage backend? If so, how?

Thank you,

David

Hey David,

Not out of the box, although for our S3 deep storage we use jets3t which also works with GCS. If you’re willing to get your hands dirty, it should be possible to adapt that extension to work with both S3 and GCS.

Hello,
it is not supported yet.

You would have to implement a module that looks like this https://github.com/druid-io/druid/tree/master/extensions/hdfs-storage/src/main/java/io/druid/storage/hdfs

Thank you for your replies!

Hi David,

Were you able to get a GCS extension to work? If so, any chance you’ll be able to open source it (put it in extensions-contrib I guess)?

If not, we’ll try to work on this.

Thanks,

Ravi

Hi Ravi,
I think you are looking for this - https://github.com/druid-io/druid/pull/2458

also - https://github.com/druid-io/druid/pull/2645

Oh, this is awesome - exactly our use case! Thanks for pointing this out, Nishant.