Using Password Provider in Kafka Ingestion Spec


I would like to pull SSL configs for a Kafka spec from an environment variable. The documentation mentions that this is possible using a password provider in and the Kafka indexing service docs mention that a password provider may be used in consumerProperties. Is this possible? If so, is there an example somewhere?

“ioConfig”: {
“topic”: “<stream_name>”,
“consumerProperties”: {
“bootstrap.servers”: [broker_list],
“security.protocol”: “ssl”,
“ssl.truststore.location”: “<location_1>”,
“ssl.truststore.password”: “{“type”: “environment”, “variable”: “KAFKA_SSL_TRUSTSTORE_PASSWORD”}”,
“ssl.keystore.location”: “<location_2>”,
“ssl.keystore.password”: “{“type”: “environment”, “variable”: “KAFKA_SSL_KEYSTORE_PASSWORD”}”,
“ssl.key.password”: “{“type”: “environment”, “variable”: “KAFKA_SSL_KEY_PASSWORD”}”


Thank you,


Hey Kiefer,

I don’t think that technique would work in the consumerProperties, since those are passed as-is to the Kafka consumer library. If the Kafka consumer supports some way of pulling keys out of the environment, then using that mechanism (if it exists) should work.

Then the documentation is wrong? consumerProperties under KafkaSupervisorIOConfig here:

Oh! Maybe that’s a feature I didn’t know about :slight_smile:

In that case, I bet what you need to do is not escape it. So, rather than embedding it into a string, do this:

“ssl.keystore.password”: {“type”: “environment”, “variable”: “KAFKA_SSL_KEYSTORE_PASSWORD”}


The more you learn, eh?

It seems to have worked better as JSON as opposed to a string.

Hi Kiefer,

Here is my working fragment of supervisor job:

"consumerProperties": {
  "bootstrap.servers": "k1:9093,k2:9093,k3:9093",
  "": "druid",
  "security.protocol": "SSL",
  "ssl.keystore.location": "/var/druid/keystore/kafka.druid.keystore.jks",
  "ssl.truststore.location": "/var/druid/keystore/kafka.druid.truststore.jks",
  "ssl.keystore.password": { "type": "environment", "variable": "KAFKA_KEYSTORE_PWD" },
  "ssl.key.password": { "type": "environment", "variable": "KAFKA_KEY_PWD" },
  "ssl.truststore.password": { "type": "environment", "variable": "KAFKA_TRUSTSTORE_PWD" }


Environment variables must be accessible from Overlord service.




Best regards,