FAILED Task : KAFKA Indexing Service

HI Al,

I am having 2 issues while running real time ingestion through kafka indexing service.

  1. Some of the tasks are getting failed as soon as they start . I do not see anything in middle Manager logs - I checked overload logs and found some exception. Complete logs of overload for one task is attached. filename - Failed_Task_Logs_Druid.txt.

If someone have faced similar issue please let me know what exactly the issue is and how to improve.

  1. I can task getting failed after running for more than 80-90 minutes with below logs in middleManager

2016-11-11T02:36:56,312 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[telemetry_2016-11-10T23:45:00.000Z_2
016-11-11T00:00:00.000Z_2016-11-10T23:45:02.911Z_12] at path[/druid/segments/172.25.182.157:8102/172.25.182.157:8102_indexer-executor__default_tier_2016-11-11T01:08:17.
472Z_e237c477dc8f4ca790aaaa1408b716740]
2016-11-11T02:36:56,312 INFO [appenderator_persist_0] io.druid.curator.announcement.Announcer - unannouncing [/druid/segments/172.25.182.157:8102/172.25.182.157:8102_in
dexer-executor__default_tier_2016-11-11T01:08:17.472Z_e237c477dc8f4ca790aaaa1408b716740]
2016-11-11T02:36:56,325 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[telemetry_2016-11-10T23:45:00.
000Z_2016-11-11T00:00:00.000Z_2016-11-10T23:45:02.911Z_12].
2016-11-11T02:36:56,327 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KafkaIndexTask{id=index_kafka_te
lemetry_1de5b639b1dd5b9_jhnmjplh, type=index_kafka, dataSource=telemetry}]
com.metamx.common.ISE: Transaction failure publishing segments, aborting
at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:506) ~[?:?]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
2016-11-11T02:36:56,332 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_telemetry_1de5b639b1dd5b9_jhnmjplh] status change
d to [FAILED].
2016-11-11T02:36:56,335 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
“id” : “index_kafka_telemetry_1de5b639b1dd5b9_jhnmjplh”,
“status” : “FAILED”,
“duration” : 5320173
}

Failed_Task_Logs_Druid.txt (24.5 KB)

Hey Arpan,

Issue 1 is a bug that was in 0.9.1.1 that was fixed in 0.9.2. If you’d like, you can download the 0.9.2-rc2 here and see if that fixes it: http://druid.io/downloads.html

For Issue 2, there should be a corresponding exception in the overlord logs that details why the transaction failed. One thing that’ll likely fix it is removing the entry for that datasource from the druid_dataSource metadata table.

Hi David,

Okay - yes I recently got to know about this new release of Druid but wants to know if it’s available with Imply ? Or I can directly copy folders from new release to current imply tar im using ?

Regards,

Arpan Khagram

+91 8308993200

Hi Arpan,

Yes, you should be able to replace the contents of /dist/druid with the 0.9.2-rc2 version. An official Imply release will come out sometime after 0.9.2 is declared stable.

Thanks David. Done and looks to be working fine.