Is anyone encrypting Kafka consumption using Kafka indexing service?

I really need help and advice from someone that is submitting supervisor specs for the Kafka indexing service using SSL. I need help trying to figure out what to put in my spec to ensure that data from Kafka to Druid is being transmitted with encryption. I'm having trouble with the key store truststore and key password parameters that you can find in the druid documentation.

Hey Chris,

I would expect the Kafka consumer will handle its own TLS based on the consumerProperties you set. I don’t think the Druid settings would have any effect on it (the Druid settings are really for the Druid servers themselves, but for talking to Kafka, that’s handled through the Kafka consumer library).

The minimal config should involve setting security.protocol, ssl.truststore.location, and ssl.truststore.password. You might need to set more than that based on what kind of setup you are going for.

Gian

Thank you Gian. The Kafka guys were adamant that the settings had nothing to do with Kafka but rather to do with druid. I am passing this information on so that we can get it figured it out.

Hi Chris,

I faced a similar problem recently. Below is a snippet of my config that works (some names are changed). These settings mirror Kafka’s consumer API

“ioConfig”: {
“topic”: “stream”,
“consumerProperties”: {
“security.protocol”: “SSL”,
“group.id”: “group_id”,
“bootstrap.servers”: “broker-0,broker-1,broker-2”,
“ssl.truststore.location”: “truststore.jks”,
“ssl.keystore.location”: “keystore.jks”,
“ssl.truststore.password”: {
“type”: “environment”,
“variable”: “PKI_TRUSTPASS”
},
“ssl.keystore.password”: {
“type”: “environment”,
“variable”: “PKI_STOREPASS”
},
“ssl.key.password”: {
“type”: “environment”,
“variable”: “PKI_KEYPASS”
}
},
“taskCount”: 1,
“replicas”: 1,
“taskDuration” : “PT2M”
}

``

Look here: https://kafka.apache.org/documentation/#consumerapi and search for ‘ssl’ to find the options available.

Thank you very much Kiefer!! This is very helpful.