Cannot get Kafka indexing service running with Secured Kafka with PLAINTEXTSASL in Hortonworks

I have been trying to get Kafka indexing service running with with Secured Kafka with Kerberos (PLAINTEXTSASL) for some time.

I have running Kerberos secured Kafka broker, producer and consumer, they are all tested .

consumer properties (“security.protocol”: “PLAINTEXTSASL”, “sasl.kerberos.service.name”: “kafka” ) related to security in json file, but still getting below error with curl command

[root@devupgnew4 ~]# curl -ikv --negotiate -X ‘POST’ -H ‘Content-Type:application/json’ -d @/tmp/examples/wikipedia-kafka-supervisor.json http://devupgnew4.azure.ril.com:8090/druid/indexer/v1/supervisor

POST /druid/indexer/v1/supervisor HTTP/1.1

User-Agent: curl/7.29.0

Host: devupgnew4.azure.ril.com:8090

Accept: /

Content-Type:application/json

Content-Length: 1457

Expect: 100-continue

< HTTP/1.1 100 Continue

HTTP/1.1 100 Continue

< HTTP/1.1 500 Server Error

HTTP/1.1 500 Server Error

< Date: Thu, 31 Jan 2019 15:02:24 GMT

< Cache-Control: must-revalidate,no-cache,no-store

Cache-Control: must-revalidate,no-cache,no-store

< Content-Type: text/html;charset=iso-8859-1

Content-Type: text/html;charset=iso-8859-1

< Content-Length: 409

Content-Length: 409

< Connection: close

Connection: close

< Server: Jetty(9.3.19.v20170502)

Server: Jetty(9.3.19.v20170502)

<

Error 500

HTTP ERROR: 500

Problem accessing /druid/indexer/v1/supervisor. Reason:

    org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

Powered by Jetty:// 9.3.19.v20170502
  • Closing connection 0

My json schema for Kafka indexing

[root@devupgnew4 ~]# cat /tmp/examples/wikipedia-kafka-supervisor.json

{

“type”: “kafka”,

“dataSchema”: {

“dataSource”: “kafkawikipedia”,

“parser”: {

“type”: “string”,

“parseSpec”: {

“format”: “json”,

“timestampSpec”: {

“column”: “time”,

“format”: “auto”

},

“dimensionsSpec”: {

“dimensions”: [

“channel”,

“cityName”,

“comment”,

“countryIsoCode”,

“countryName”,

“isAnonymous”,

“isMinor”,

“isNew”,

“isRobot”,

“isUnpatrolled”,

“metroCode”,

“namespace”,

“page”,

“regionIsoCode”,

“regionName”,

“user”,

{ “name”: “added”, “type”: “long” },

{ “name”: “deleted”, “type”: “long” },

{ “name”: “delta”, “type”: “long” }

]

}

}

},

“metricsSpec” : ,

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “DAY”,

“queryGranularity”: “NONE”,

“rollup”: false

}

},

“tuningConfig”: {

“type”: “kafka”,

“reportParseExceptions”: false

},

“ioConfig”: {

“topic”: “kafkawiki”,

“replicas”: 1,

“taskDuration”: “PT10M”,

“completionTimeout”: “PT20M”,

“consumerProperties”: {

“bootstrap.servers”: “devupgnew2.azure.ril.com:6667,devupgnew3.azure.ril.com:6667,devupgnew4.azure.ril.com:6667”,

“security.protocol”: “PLAINTEXTSASL”,

“sasl.kerberos.service.name”: “kafka”

}

}

}

Could you please help me in getting Kafka indexing service running with with Secured Kafka ?

overlord.log attached.

overload.log (120 KB)

Hi Team,

Can any one help me on this. We have HDP kerberos cluster.

Thanks

Mohammad Layeeq

Team can any one help me on this.