Error while creating supervisor task for kafka ingestion service in druid

Hello,
I’m using kafka indexing service using supervisor spec in druid. I tried to submit the supervisor spec to druid overlord process, but i’m encountering the following exception

ERROR [KafkaSupervisor-metrics-kafka-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[metrics-kafka] failed to handle notice: {class=io.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class com.metamx.common.ISE, exceptionMessage=Offset in metadata storage [29] > latest Kafka offset [6] for partition [0]. If your Kafka offsets have been reset, you will need to remove the entry for [metrics-kafka] from the dataSource table., noticeClass=RunNotice}

Why is this exception coming? And how can I resolve it?

Thanks
Rajnandini

Deleting metrics-kafka entry from the datasources will result in loss of the data.

It means that the last message that Druid read (offset 29) no longer exists in Kafka (the latest message in Kafka is offset 6). This typically happens if your Kafka log files get deleted or you delete and recreate the Kafka topic.

To recover, you need to delete the entry from the druid_dataSource metadata table. This is not the same as deleting the datasource itself. You won’t lose any data that has already been ingested into Druid.

Thank you. It worked :slight_smile:

-Rajnandini

Hello,
Even after deletion of druid datasource, the supervisor task is not getting created. It still throwing the same error.

ERROR [KafkaSupervisor-metrics-
kafka-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[metrics-kafka] failed to handle notice: {class=io.druid.indexing.kafka.supervisor. KafkaSupervisor,
exceptionType=class com.metamx.common.ISE, exceptionMessage=Offset in metadata storage [29] > latest Kafka offset [6] for partition [0]. If
your Kafka offsets have been reset, you will need to remove the entry for [metrics-kafka] from the dataSource table., noticeClass=RunNotice}

Thanks,
Rajnandini

You need to remove the entry for metrics-kafka in the druid_dataSource table in SQL, not delete the datasource from Druid.

Is there any documentation about how to delete druid datasource from druid_datasource table in SQL?

Thanks,
Rajnandini

No there isn’t. You just need to issue a SQL command similar to: DELETE FROM druid.druid_dataSource WHERE dataSource=‘metrics-kafka’

Starting in 0.9.2 there will be a reset endpoint to help remove the entry.

Hi,
I wanted to delete unwanted segments form deep storage. I disabled those datasources from cordinator node. And also submitted Kill task to overlord. This task is successful but still there are those segments still available on deep storage.
I’m really running out of memory.

Thanks
Rajnandini

kill task should work

are you sure you have the right intervals ?