[druid-user] I can't ingest data from Amazon MSK with SSL, please help

Apache Druid Q&A Google Groups
#1

I’m having trouble ingesting data on Amazon MSK with SSL.

First, I did a test locally with a Kafka client that could retrieve data from Kafka over SSL.
I created a client.properties file as such:

security.protocol=SSL
ssl.truststore.location=$JDK_HOME/lib/security/cacerts
ssl.truststore.password=changeit

Then I tested the connectivity and received data:

kafka-console-consumer --bootstrap-server --consumer.config client.propertie --topic my-kafka-topic --from-beginning

However, when I pass along these properties to my

“ioConfig”: {
“consumerProperties”: {
“bootstrap.servers”: “bootstrap servers with port”,“security.protocol”: “SSL”
“ssl.truststore.location”: “$JDK_HOME/lib/security/cacerts”
“ssl.truststore.password”: “changeit”
}
}

in my druid ingestion spec, well, it fails to ingest data.

Please note that $JDK_HOME is an actual path to my JDK

Also, I use the root signed JDK cacerts file because it contains the AWS certificate.

Also, I tested my configuration with Kafka running in Docker with no SSL and it all worked fine. However, my deployment environment requires SSL.

Any help would be deeply appreciated.

#2

Are there any hints in the supervisor or task logs out of interest?

#3

It finally started working after I used the following consumer property: isolation.level = read_uncommitted.
I’m not sure why that fixed this.

Thank you.

#4

Oh interesting… that’s on the Kafka Consumer Properties? Hmm I can’t even pretend to be a Kafka expert – I did a cursory Google… Would that override the topic offset that Druid is storing internally? Or is it nothing to do with that… :smiley: :smiley: