[druid-user] I can't ingest data from Amazon MSK with SSL, please help

I’m having trouble ingesting data on Amazon MSK with SSL.

First, I did a test locally with a Kafka client that could retrieve data from Kafka over SSL.
I created a client.properties file as such:

security.protocol=SSL
ssl.truststore.location=$JDK_HOME/lib/security/cacerts
ssl.truststore.password=changeit

Then I tested the connectivity and received data:

kafka-console-consumer --bootstrap-server --consumer.config client.propertie --topic my-kafka-topic --from-beginning

However, when I pass along these properties to my

“ioConfig”: {
“consumerProperties”: {
“bootstrap.servers”: “bootstrap servers with port”,“security.protocol”: “SSL”
“ssl.truststore.location”: “$JDK_HOME/lib/security/cacerts”
“ssl.truststore.password”: “changeit”
}
}

in my druid ingestion spec, well, it fails to ingest data.

Please note that $JDK_HOME is an actual path to my JDK

Also, I use the root signed JDK cacerts file because it contains the AWS certificate.

Also, I tested my configuration with Kafka running in Docker with no SSL and it all worked fine. However, my deployment environment requires SSL.

Any help would be deeply appreciated.

Are there any hints in the supervisor or task logs out of interest?

It finally started working after I used the following consumer property: isolation.level = read_uncommitted.
I’m not sure why that fixed this.

Thank you.

Oh interesting… that’s on the Kafka Consumer Properties? Hmm I can’t even pretend to be a Kafka expert – I did a cursory Google… Would that override the topic offset that Druid is storing internally? Or is it nothing to do with that… :smiley: :smiley: