Using Password Provider in Kafka Ingestion Spec

Hello,

I would like to pull SSL configs for a Kafka spec from an environment variable. The documentation mentions that this is possible using a password provider in runtime.properties and the Kafka indexing service docs mention that a password provider may be used in consumerProperties. Is this possible? If so, is there an example somewhere?

“ioConfig”: {
“topic”: “<stream_name>”,
“consumerProperties”: {
“bootstrap.servers”: [broker_list],
“security.protocol”: “ssl”,
“ssl.truststore.location”: “<location_1>”,
“ssl.truststore.password”: “{“type”: “environment”, “variable”: “KAFKA_SSL_TRUSTSTORE_PASSWORD”}”,
“ssl.keystore.location”: “<location_2>”,
“ssl.keystore.password”: “{“type”: “environment”, “variable”: “KAFKA_SSL_KEYSTORE_PASSWORD”}”,
“ssl.key.password”: “{“type”: “environment”, “variable”: “KAFKA_SSL_KEY_PASSWORD”}”
}

``

Thank you,

Kiefer

Hey Kiefer,

I don’t think that technique would work in the consumerProperties, since those are passed as-is to the Kafka consumer library. If the Kafka consumer supports some way of pulling keys out of the environment, then using that mechanism (if it exists) should work.

Then the documentation is wrong? consumerProperties under KafkaSupervisorIOConfig here: http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html

Oh! Maybe that’s a feature I didn’t know about :slight_smile:

In that case, I bet what you need to do is not escape it. So, rather than embedding it into a string, do this:

“ssl.keystore.password”: {“type”: “environment”, “variable”: “KAFKA_SSL_KEYSTORE_PASSWORD”}

Gian

The more you learn, eh?

It seems to have worked better as JSON as opposed to a string.

Hi Kiefer,

Here is my working fragment of supervisor job:

"consumerProperties": {
  "bootstrap.servers": "k1:9093,k2:9093,k3:9093",
  "group.id": "druid",
  "security.protocol": "SSL",
  "ssl.keystore.location": "/var/druid/keystore/kafka.druid.keystore.jks",
  "ssl.truststore.location": "/var/druid/keystore/kafka.druid.truststore.jks",
  "ssl.keystore.password": { "type": "environment", "variable": "KAFKA_KEYSTORE_PWD" },
  "ssl.key.password": { "type": "environment", "variable": "KAFKA_KEY_PWD" },
  "ssl.truststore.password": { "type": "environment", "variable": "KAFKA_TRUSTSTORE_PWD" }
},

``

Environment variables must be accessible from Overlord service.

KAFKA_KEYSTORE_PWD

KAFKA_KEY_PWD

KAFKA_TRUSTSTORE_PWD

Best regards,
Yevgen