Cannot get Kafka indexing service running with Secured Kafka with Kerberos (SASL_PLAINTEXT)

Hi all,

I have been trying to get Kafka indexing service in imply.io running with with Secured Kafka with Kerberos (SASL_PLAINTEXT) for some time, but with no success.

I have running Kerberos secured Kafka broker, producer and consumer, they are all tested .

After setting javaopts parameters relavant to security given in red below in conf-quickstart/druid/_common/common.runtime.properties

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Djava.security.auth.login.config=/etc/kafka/kafka_client_kerberos_jaas.conf -Djava.security.krb5.conf=/etc/krb.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.kdc=server.myhostname.com -Djava.security.krb5.realm=MYREALM.COM

, I follow the steps given in tutorial kafka indexing service in imply.io and add the consumer properties (“security.protocol”: “SASL_PLAINTEXT”, “sasl.kerberos.service.name”: “kafka” ) related to security in json file, but I am just getting stuck in the step "curl -XPOST -H’Content-Type: application/json’ -d

@/home/kafka/kafka-task.json http://localhost:8090/druid/indexer/v1/supervisor", and receiving the following error message

Problem accessing /druid/indexer/v1/supervisor. Reason:

    org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

Powered by Jetty://

In overlord.log , that error turns out to be caused by: “org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: You must pass java.security.auth.login.config in secure mode”, although I pass this argument in the way given above.

Could you please help me in getting Kafka indexing service running with with Secured Kafka ?

overlord.log and kafka_task.json attached.

yours,

Hamit

overlord.log (160 KB)

kafka_task.json (2.17 KB)

Hey Hamit,

The supervisor, which lives on the overlord, also has a Kafka consumer that it uses to determine the task configuration and therefore the overlord also needs to have the Kerberos configuration. Try setting

-Djava.security.auth.login.config=/etc/kafka/kafka_client_kerberos_jaas.conf

in your overlord/jvm.config file as well.

Hi David,
It worked.

Thanks for the help.

yours,

Hamit

Hi David,

I have another question for you regarding this issue.

I said before it worked for the case all the users are super users.

In my case, I have two users with the names “kafka” and “kafka-client”.

User “kafka” is running zookeeper and kafka server in secured mode with kerberos and user “kafka-client” is running only druid.

In kafka server properties file, if they are set as super users, the druid works as I said before.

But when kafka-client is removed from the super.users with and “allow.everyone.if.no.acl.found” is set to false, and then trying to give the permission to it with the following kafka console command

“bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=kafka1.myhostname.com:2181 --add --allow-principal User:kafka-client --operation Read --topic test-topic --group test-group”

after I add “group.id”: “test-group” line to json file, I can not get it running.

Error mesage I got in the overlord.log is as follows

ERROR [KafkaSupervisor-igwcdrv1-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[igwcdrv1] failed to handle notice:
{class=io.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class org.apache.kafka.common.errors.GroupAuthorizationException, exceptionMessage=Not aut
horized to access group: kafka-supervisor-ebjnpajj, noticeClass=RunNotice}

org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: kafka-supervisor-ebjnpajj

2016-12-08T10:04:45,033 INFO [KafkaSupervisor-igwcdrv1-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2016-12-08T10:04:45.032Z”,"
service":“druid/overlord”,“host”:“hamits:8090”,“severity”:“component-failure”,“description”:“KafkaSupervisor[igwcdrv1] failed to handle notice”,“data”:{“class”:“io
.druid.indexing.kafka.supervisor.KafkaSupervisor”,“exceptionType”:“org.apache.kafka.common.errors.GroupAuthorizationException”,“exceptionMessage”:“Not authorized to access group: kafka-supervisor-ebjnpajj”,“exceptionStackTrace”:“org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: kafka-supervisor-ebjnpajj\n”,“noticeClass”:“RunNotice”}}]

It turns out that group name I enter in json is not passed to the system. It is “kafka-supervisor-ebjnpajj” that druid itself puts

I would like to stress out that I don’t have any problem when running kafka console consumer instead of druid after typing the command given above for read permission

Do you have an idea about that?

Thanks in advance.

yours,

Hamit.

Hey Hamit,

Unfortunately it looks like the supervisor overrides the group ID with its own as you can see here: https://github.com/druid-io/druid/blob/master/extensions-core/kafka-indexing-service/src/main/java/io/druid/indexing/kafka/supervisor/KafkaSupervisor.java#L601

You could either build the extension from source and move that line before ‘props.putAll(ioConfig.getConsumerProperties());’, or the alternative option would be to use a wildcard (i.e. --group *) when setting the Kafka ACLs.

I’ll check in a patch for this for next release.

See: https://github.com/druid-io/druid/pull/3765

Hi David,

I went with the building of extension after making changes you proposed and it worked.

Thank again.

yours,

Hamit.