Kafka ingest service is automatically resets

Hi,

I have 2 questions regarding this post: (Durid 11)

  1. Kafka group.id property is not being effective in kafka topic. I don’t see this topic has this particular consumer group. I can see in overlord logs that it registers this property, but doesn’t show in kafka consumer group.

  2. Kafka ingest service suddenly jumps to an offset which is past. Current use case is like my kafka ingest agg offset lag 5,000,000. After a while the offset lag jumps to 30,000,000. I checked there were no failed tasks and there is no new data in kafka.

Did anyone encounter any similar behavior? Is there a way to avoid it?

Below is spec:

{
“type”: “kafka”,
“dataSchema”: {
“dataSource”: “foo_bar”,
“parser”: {
“type”: “string”,
“parseSpec”: {
“format”: “json”,
“timestampSpec”: {
“column”: “timestamp”,
“format”: “auto”
},
“dimensionsSpec”: {
“dimensions”: ,
“dimensionExclusions”:
}
}
},
“metricsSpec”: [
{
“name”: “count”,
“type”: “count”
}
],
“granularitySpec”: {
“type”: “uniform”,
“segmentGranularity”: “HOUR”,
“queryGranularity”: “NONE”
}
},
“tuningConfig”: {
“type”: “kafka”,
“maxRowsPerSegment”: 5000000,
“resetOffsetAutomatically”: true
},
“ioConfig”: {
“topic”: “kafka_topic”,
“useEarliestOffset”: true,
“lateMessageRejectionPeriod”: “PT2H”,
“consumerProperties”: {
“bootstrap.servers”: “node:9092”,
“group.id”: “group_id”
},
“taskCount”: 2,
“replicas”: 2,
“taskDuration”: “PT30M”
}
}

``

Did anyone encounter this behavior?