Kafka Indexing Service - With Protobuf - Incorrect Supervisor Status

Hi,

I am running Kafka Indexing Service (with taskDuration aas 1 hour).

When I try to get the status of the Supervisor Task, I see following

“id”: “kafkaIndexing-protobuf-test”,

“generationTime”: “2017-07-10T13:29:56.172Z”,

“payload”: {

“dataSource”: “kafkaIndexing-protobuf-test”,

“topic”: “protobufKafkaIndexing”,

“partitions”: 4,

“replicas”: 2,

“durationSeconds”: 3600,

“activeTasks”: [

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_148a0c3d26b7d7e_jifiaagc”,

“startingOffsets”: {

“0”: 3802

},

"startTime": null,

"remainingSeconds": null,

“type”: “ACTIVE”,

“currentOffsets”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_148a0c3d26b7d7e_bcjoladp”,

“startingOffsets”: {

“0”: 3802

},

"startTime": null,

"remainingSeconds": null,

“type”: “ACTIVE”,

“currentOffsets”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_9151246efd50065_dpdckiki”,

“startingOffsets”: {

“1”: 3802

},

"startTime": null,

"remainingSeconds": null,

“type”: “ACTIVE”,

“currentOffsets”: {}

},

Why the startTime and remainingSeconds are “null” here. Is this expected or some issue ??

Thanks,

Pravesh Gupta

That’s not normal, but neither is currentOffsets being blank. It’s possible that the tasks haven’t fully started up yet. Maybe check their logs in the overlord console for clues.

Thanks Gian.

What about the startingOffset then ??

I though currentOffsets are blank because as there is no remaining data in the Kafka Topic. Isnt that make sense ? What does all these parameters mean ? Is there any doc which explains it, Could not find anything at http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html .

Also I am attaching the Overlord Console Snapshot and log of two task. As per my supervisor spec , I had configured for 4 tasks with 2 replicas , but only two ran.

Also in the above Supervisor Status Response, the task index_kafka_kafkaIndexing-protobuf-test_0498ade5a0bc196_jcagkldk shows startingOffset as 3802 but when I see the status of same task on Overlord console, I see it is still in pending tasks list . How is that possible.

OverlordKafkaIndexingTask1Log.txt (165 KB)

Also I keep seeing these messages in overlord log when I am ending events to Kafka now :

2017-07-11T07:08:29,069 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-3] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_148a0c3d26b7d7e_jifiaagc], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:29,070 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-6] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_148a0c3d26b7d7e_bcjoladp], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:29,071 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-5] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_9151246efd50065_dpdckiki], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:29,077 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-0] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_0498ade5a0bc196_jcagkldk], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:29,078 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-1] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_0498ade5a0bc196_beeadnng], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:29,079 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-2] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_c4927e74a46b245_oodgekgo], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:29,082 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-7] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_c4927e74a46b245_mlnghmaa], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:29,083 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-4] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_9151246efd50065_opeaadaf], this task may not have been assigned to a worker yet or may have already completed

2017-07-11T07:08:32,661 INFO [KafkaIndexTaskClient-kafkaIndexing-protobuf-test-3] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafkaIndexing-protobuf-test_148a0c3d26b7d7e_jifiaagc], this task may not have been assigned to a worker yet or may have already completed

I restarted the Druid Nodes and also restarted the Kafka and Produced few messages again in the Kafka.

Now when i see the Supervisor Status, I see all the fields like “startingOffsets” , “remainingSeconds” , “currentOffsets” are filled UNLIKE The initial Problem I mentioned.

{

“id”: “kafkaIndexing-protobuf-test”,

“generationTime”: “2017-07-12T06:43:31.402Z”,

“payload”: {

“dataSource”: “kafkaIndexing-protobuf-test”,

“topic”: “protobufKafkaIndexing”,

“partitions”: 4,

“replicas”: 1,

“durationSeconds”: 3600,

“activeTasks”: [

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_7860c0189a17b4d_njlpcbic”,

“startingOffsets”: {

“0”: 6652

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_60cdd1a6204d338_ghgkddkm”,

“startingOffsets”: {

“1”: 6653

},

“startTime”: null,

“remainingSeconds”: null,

“type”: “ACTIVE”,

“currentOffsets”: {}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_06b2d68a4c7bd7b_hajbdlbm”,

“startingOffsets”: {

“2”: 6653

},

“startTime”: “2017-07-12T06:30:32.667Z”,

“remainingSeconds”: 2821,

“type”: “ACTIVE”,

“currentOffsets”: {

“2”: 6653

}

},

{

“id”: “index_kafka_kafkaIndexing-protobuf-test_f1baeec4b3dd5c2_mmilpmob”,

“startingOffsets”: {

“3”: 6653

},

“startTime”: “2017-07-12T06:31:07.736Z”,

“remainingSeconds”: 2856,

“type”: “ACTIVE”,

“currentOffsets”: {

“3”: 6653

}

}

],

“publishingTasks”:

}

}

But Still NO Segments got created . I have waited for more than the task Duration now.

For the record , I am using the imply distribution of druid and followed Imply Quickstart DOC for Kafka Indexing Service.

Is there any issue with the Protobuf Deserializer , although I cant see any visible logs in logger files.

Any help would be appreciated.

Thanks,

Pravesh Gupta