[druid-user] Kafka Log Clean Up Druid Issue

Hello,
I have retention policy as 1 hour and log.cleanup as delete. After 1 or 2 cleanups Druid stops consuming the messages and I have to reset the supervisor to start it again. Can someone please help me with this issue?

Thank you

Druid will note the latest topic offset at ingestion time so that it knows where to start running from again when the supervisors start again.

What length of time is your task duration? Maybe it is too long given your Kafka retention period?

Check

useEarliestOffset and resetOffsetAutomatically are likely of interest - and also maybe check those message rejection period settings…

Hello,
Thank you for your reply. I have tried task duration to 1h,5h, and 24h. My supervisour config is as per follow:


{
"type": "kafka",
"spec": {
"ioConfig": {
"type": "kafka",
"topic": "table1",
"consumerProperties": {
"bootstrap.servers": "IP:9092"
},
"inputFormat": {
"type": "json"
},
"useEarliestOffset": true,
"appendToExisting": true,
"taskDuration": "PT1H",
"taskCount": 2
},
"tuningConfig": {
"type": "kafka",
"resetOffsetAutomatically": true
},
"dataSchema": {
"dataSource": "table1",
"granularitySpec": {
"type": "uniform",
"queryGranularity": "NONE",
"segmentGranularity": "HOUR",
"rollup": false
},
"timestampSpec": {
"column": "Time",
"format": "millis"
},
"dimensionsSpec": {
"dimensions": [
"Field1",
"Field2",
"Field3",
{
"type": "long",
"name": "Field4"
},
{
"type": "long",
"name": "Field5"
},
"Field6",
"Field7",
"Field8",
"Field9",
]

}
}
}
}