Using Self-Signed SSL Certs in Druid S3 Extension

  • Druid Version: 0.22.1
  • Deep Storage: In-house/internally developed s3 API compatible Service

Hi, I have a question regarding using a S3 API compatible service instead of traditional AWS S3. We are currently using an in-house/internal (built by our internal teams) and we have a different problem then the usual AWS S3 users.

Our internal deep storage with druid s3 extension works fine when using HTTP protocol but when using HTTPs we are unable to verify the certificate authority which is expected since the internal deep storage service uses self-signed certs (not my choice but I’m guessing the other teams wanted to save costs). This problem would also occur if you had a MiM set up that was validating/intercepting/tracing your traffic like some orgs might have.

My question is how do we do one of the following by specifying additional configuration/flags?

  1. Use HTTPs but set a property for DISABLE_SSL_VERIFICATION/DISABLE_CERT_CHECKING or similar?
  2. How do we load the certs into a java keystore so that we can validate against our certs that we load for deep storage?

I believe for option #1: you can reference this: SDKGlobalConfiguration (AWS SDK for Java - 1.12.253) and set the DISABLE_CERT_CHECKING_SYSTEM_PROPERTY via jvm argument:

-Dcom.amazonaws.sdk.disableCertChecking=true

But I know that option #1 is not a recommended practice since that opens you up for MiM attack in a production environment that may have access to internet/or malicious internal processes. But it is still useful for internal testing and quick development. I am not as familiar with Java/JVM, I usually work with GoLang, Python, JS, etc. but I guess I will have to look into how to create/load into a keystore. But was wondering if anyone has already done this ask to save me some work/time.

I took a quick look into option 2 and appears you will have to do the following (general steps):

  1. Get SSL cert chain into a format that is compatible, example:
echo quit | openssl s_client -showcerts -servername server -connect server:443 > cacert.pem
  1. Load certificate into Java keystore (where/how depends on how it was installed/OS), reference: https://www.tutorialworks.com/java-trust-ssl/
keytool -import -alias CHOOSE-AN-ALIAS \
    -file certificate.pem \
    -keystore /path/to/your/truststore
  1. Restart process and see if change has taken affect

My question/ask here is still semi-open since I am wondering if there is any easy way to do option #2 with flags/configuration options especially for those who have things set up in containers/helm. I am unsure if you could do those steps for #2 without modifying and customizing charts/dockerfiles

I hope you are using a self-signed certificate or self-signed ca certificate for connecting to ur inhouse s3 deployment. In that case, you can add this certificate to your jdk/jre/security/cacert Keystore. This is the Keystore java used internally and holds all the trusted public certs including ca certs. Hope this helps

Hi @TijoThomas

Yes, I did cover that in my previous reply and gave an example on how to get the cert and load into a keystore.

But now I am more wondering what is the expectation for users that are using helm/docker (we have hybrid setups where our stateless druid apps like overlord, coordinator, etc. run in a Kubernetes cluster for scalability and VM compute capacity issues). They would have to modify the start commands/exec of the helm chart and/or create a new Dockerfile to actually load into the keystore I believe.

Perhaps my suggestion here is that we should have the capability already included in the helm chart and the process in general should also be documented in Druid official docs for users.

@Sergio_Ferragut did you run into this with your HELM stuff at all?
Maybe even @Saydul_Bashar ?