Can't get Druid to ingest directly from Google Cloud Storage

I have made the necessary changes in the “environment” file to enable GCS on Druid’s console. But when I want to ingest I get this error:

Error: Cannot construct instance of org.apache.druid.data.input.google.GoogleCloudStorageInputSource, problem: Unable to provision, see the following errors: 1) Error in custom provider, java.io.IOException: Error reading credential file from environment variable GOOGLE_APPLICATION_CREDENTIALS, value ‘/home/user/druid/my_service_account_key.json’: File does not exist. at org.apache.druid.common.gcp.GcpModule.getHttpRequestInitializer(GcpModule.java:60)

I don’t know what information is necessary to be shared to diagnose the problem but I have set the path like this:

GOOGLE_APPLICATION_CREDENTIALS=/home/user/druid/my_service_account_key.json

Is there a way to see if it actually can’t find the file or it’s a permission issue?

1 Like

Hands up I’ve not done GCS ingestion … but I note in the docs that there’s no “json” extension in the environment variable — should it have a json extension?

https://druid.apache.org/docs/latest/development/extensions-core/google.html#required-configuration

The google credentials file is in JSON format (at least the one I got following the GCS instructions). The Druid documentation is referring to a file which can’t be located. I didn’t try to remove the .json from the file but I got it working by mapping the VM folders to the docker folders (Volumes) in the compose YML file. So the file name might not be the issue. I guess my problem is the mapping volume issues.

So is this variable supposed to be a shell environment variable? or should I define this in my “environment” file? Can someone elaborate?

Hm let me see if I can find someone who knows to chip in…