Kafka indexing task failing

Hey ,

We are using kafka indexing service to ingest that data. Previously it as working fine. We have submitted the new kafka supervisor config with topic change. After that all the task are failing for that datasource.

**Exception in the task log : **

data-10_2016-09-19T00:00:00.000Z_2016-09-20T00:00:00.000Z_2016-09-23T11:42:42.084Z_65].

2016-09-24T17:10:36,116 INFO [task-runner-0-priority-0] io.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - Unregistering chat handler[index_kafka_prism-data-10_ea19f44b877d897_ccnmoioe]

2016-09-24T17:10:36,118 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KafkaIndexTask{id=index_kafka_prism-data-10_ea19f44b877d897_ccnmoioe, type=index_kafka, dataSource=prism-data-10}]

com.metamx.common.ISE: Transaction failure publishing segments, aborting

at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:505) ~[?:?]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-rc1.jar:0.9.2-rc1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-rc1.jar:0.9.2-rc1]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

2016-09-24T17:10:36,126 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_prism-data-10_ea19f44b877d897_ccnmoioe] status changed to [FAILED].

2016-09-24T17:10:36,129 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

“id” : “index_kafka_prism-data-10_ea19f44b877d897_ccnmoioe”,

“status” : “FAILED”,

“duration” : 5743847

}

Same time overlord logs shows :

2016-09-24T23:29:42,754 INFO [qtp2040548959-29] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - ** Not updating metadata, existing state is not the expected start state.**

2016-09-24T23:29:42,757 INFO [qtp2040548959-29] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-24T23:29:42.756Z”,“service”:“druid/overlord”,“host”:“druid-overlord-coordinate-001.c.inshorts-1374.internal:8090”,“metric”:“segment/txn/failure”,“value”:1,“dataSource”:“prism-data-10”,“taskType”:“index_kafka”}]

How can we fix this if we want to change the topic for the same datasource ? Does kafka supervisor config doesn’t allow topic changes ?

Thanks,

Saurabh

Hey Saurabh,

Changing the topic messes with the ingestion metadata used to guarantee exactly-once ingestion – it’s all about offsets and partitions for a particular topic. But you can delete the ingestion metadata row from the druid_dataSource metadata table to “reset” this and allow a topic change. In 0.9.2 we’ll be adding an API to do this without having to muck with the metadata store directly.

Hi Gian,

I am using 0.10.0-SNAPSHOT (taken latest on March 29) and i can see this exception for few of my tasks and logs in overlord saying 'Not updating metadata, existing state is not the expected start state.’.

Please note that i have not changed topic - the topic i am using is same from quite some time now.

Regards,

arpan