Supervisor Status Response Fields Meaning for Kafka Indexing Service

Hi,
Could anyone please explain the meaning of the fields in the supervisor status response, as I am struggling with setting up Kafka Indexing Service with Protobuf.

Its driving me crazy. I want to understand the meaning of each field of Supervisor Status, however there is nothing which I can find out on the Druid DOC page.

Following is my Supervisor Spec Submitted to Druid:

{

“type”: “kafka”,

“dataSchema”: {

“dataSource”: “kafkaIndexing-protobuf-test”,

“parser”: {

“type”: “protobuf”,

“descriptor”: “DruidMessage.desc”,

“protoMessageType”: “DruidMessage”,

“parseSpec”: {

“format”: “json”,

“timestampSpec”: {

“column”: “timestamp”,

“format”: “millis”

},

“dimensionsSpec”: {

“dimensions”: [

“eventType”,

“failureType”,

…{Few More Dimensions}

],

“dimensionExclusions”: [

“timestamp”,

“value”

]

}

}

},

“metricsSpec”: [

{

“type”: “longSum”,

“name”: “eventCount”,

“fieldName”: “eventCount”

}

],

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “HOUR”,

“queryGranularity”: “minute”

}

},

“ioConfig”: {

“topic”: “protobufKafkaIndexing”,

“consumerProperties”: {

“bootstrap.servers”: “Kafka Broker”

},

“taskCount”: 4,

“replicas”: 1,

“taskDuration”: “PT1H”

}

}

Folllowing is my Supervisor Status Response:

{

“id”: “kafkaIndexing-protobuf-test”,

“generationTime”: “2017-07-13T12:05:41.541Z”,

“payload”: {

“dataSource”: “kafkaIndexing-protobuf-test”,

“topic”: “protobufKafkaIndexing”,

“partitions”: 4,

“replicas”: 1,

“durationSeconds”: 3600,

“activeTasks”: [

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_e8667072be1fef7_ahiiikfn”,

“startingOffsets”: {

“0”: 6652

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {},

“lag”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_45f200cfc0fd17b_nnfnfaja”,

“startingOffsets”: {

“1”: 6653

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {},

“lag”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_bd9095606261eb2_npojcand”,

“startingOffsets”: {

“2”: 6653

},

“startTime”: “2017-07-13T11:59:52.387Z”,

“remainingSeconds”: 3250,

“type”: “ACTIVE”,

“currentOffsets”: {

“2”: 15064

},

“lag”: {

“2”: 0

}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_56db020af47f096_lepnnilk”,

“startingOffsets”: {

“3”: 6653

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {},

“lag”: {}

}

],

“publishingTasks”: ,

“latestOffsets”: {

“0”: 15063,

“1”: 15065,

“2”: 15064,

“3”: 15065

},

“minimumLag”: {

“2”: 0

},

“aggregateLag”: 0,

“offsetsLastUpdated”: “2017-07-13T12:05:11.476Z”

}

}

The “latestOffsets” value keeps on increasing while as I keep on pushing Events to the Kafka ,but there is no data in the druid . No special logs also in the console.

Please help. Let me know if is there anything else to provide.

I am attaching the indexing task log(Only one is shown as running and other three as Pending , dont know why )

Thanks,

Pravesh Gupta

index_kafka_kafkaIndexing-protobuf-test_bd9095606261eb2_npojcand.txt (435 KB)

Now When I see the Supervisor Status, I see following ::

{

“id”: “kafkaIndexing-protobuf-test”,

“generationTime”: “2017-07-16T08:07:42.086Z”,

“payload”: {

“dataSource”: “kafkaIndexing-protobuf-test”,

“topic”: “protobufKafkaIndexing”,

“partitions”: 4,

“replicas”: 1,

“durationSeconds”: 3600,

“activeTasks”: [

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_e8667072be1fef7_oloiekgi”,

“startingOffsets”: {

“0”: 6652

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {},

“lag”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_45f200cfc0fd17b_cpohmjba”,

“startingOffsets”: {

“1”: 6653

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {},

“lag”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_cd46cc3aa772f37_cdagflnm”,

“startingOffsets”: {

“2”: 15064

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {},

“lag”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_56db020af47f096_lepnnilk”,

“startingOffsets”: {

“3”: 6653

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {},

“lag”: {}

}

],

“publishingTasks”: ,

“latestOffsets”: {

“0”: 15063,

“1”: 15065,

“2”: 15064,

“3”: 15065

},

“minimumLag”: {},

“aggregateLag”: 0,

“offsetsLastUpdated”: “2017-07-16T08:07:41.441Z”

}

}

Not able to figure out whats happening here …

I can see some segments created in druid . But from above status looks like the all the data has not been read from Kafka.

Please help.

Thanks,

Pravesh Gupta

Any help on this ??

Please let me know the meaning of fields returned in Supervisor Status. I do not seem to find out anything either from Online Docs or from the Codebase.

Thanks,

Pravesh Gupta