About "enable.auto.commit" property in Kafka Index Service

Hi all,
We’re using Kafka Index Service in our company. We want to monitor the lag of consuming Kafka.
As I know, KIS maintains the offset information of Kafka in the metadata storage. So it doesn’t have
to commit offset to Kafka. But as I understand, KIS also doesn’t have any reason to forbid auto
committing offset to Kafka (it maintains the offset by itself, and don’t care about the offset in Kafka).
But when I read the KIS source code, I found this line:

props.setProperty("enable.auto.commit", "false");

``

So, my question is: why KIS explicitly setting “enable.auto.commit” to “false”? Did I misunderstand
something?

Thanks

Hi Kevin,

It’s because we have no need for auto-committing since Druid manages its own Kafka offsets.

For monitoring the lag, would it work to use the supervisor status endpoint and check the “aggregateLag” field? That’s the total lag across all Kafka partitions, computed for you.

Hi Gian,
Thanks for your reply. I’ll try to use the supervisor status endpoint to get the lag.

Kevin

在 2017年11月2日星期四 UTC+8上午11:39:31,Gian Merlino写道:

Hi Gian,
Sorry to bother again. But when I used “GET” method to call the supervisor status endpoint, I did get response. But I didn’t see the “aggregateLag” field as you mentioned.

The cmd I used was:
curl -X GET http://:/druid/indexer/v1/supervisor//status

``

The response was like this:
{
“id”: “”,
“generationTime”: “2017-11-02T14:26:23.177+08:00”,
“payload”: {
“dataSource”: ",
“topic”: ",
“partitions”: 15,
“replicas”: 2,
“durationSeconds”: 3600,
“activeTasks”: [
{
“id”: “”,
“startingOffsets”: {
“0”: 3140315022,
“3”: 3127526185,
“6”: 3125620301,
“9”: 3140048744,
“12”: 3125618173
},
“startTime”: “2017-11-02T14:20:29.455+08:00”,
“remainingSeconds”: 3246,
“type”: “ACTIVE”,
“currentOffsets”: {
“0”: 3140923617,
“3”: 3128134780,
“6”: 3126228896,
“9”: 3140657340,
“12”: 3126226768
}
},

],
“publishingTasks”:
}
}

``

Could you please kindly point out what’s wrong?

Thanks,
Kevin

在 2017年11月2日星期四 UTC+8上午11:39:31,Gian Merlino写道:

You might be using an older version. The feature was added in Druid 0.10.1.