FAILED Task : KAFKA Indexing Service

HI Al,

I am having 2 issues while running real time ingestion through kafka indexing service.

  1. Some of the tasks are getting failed as soon as they start . I do not see anything in middle Manager logs - I checked overload logs and found some exception. Complete logs of overload for one task is attached. filename - Failed_Task_Logs_Druid.txt.

If someone have faced similar issue please let me know what exactly the issue is and how to improve.

  1. I can task getting failed after running for more than 80-90 minutes with below logs in middleManager

2016-11-11T02:36:56,312 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[telemetry_2016-11-10T23:45:00.000Z_2
016-11-11T00:00:00.000Z_2016-11-10T23:45:02.911Z_12] at path[/druid/segments/
2016-11-11T02:36:56,312 INFO [appenderator_persist_0] io.druid.curator.announcement.Announcer - unannouncing [/druid/segments/
2016-11-11T02:36:56,325 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[telemetry_2016-11-10T23:45:00.
2016-11-11T02:36:56,327 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KafkaIndexTask{id=index_kafka_te
lemetry_1de5b639b1dd5b9_jhnmjplh, type=index_kafka, dataSource=telemetry}]
com.metamx.common.ISE: Transaction failure publishing segments, aborting
at ~[?:?]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ [druid-indexing-service-]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ [druid-indexing-service-]
at [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker( [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$ [?:1.8.0_91]
at [?:1.8.0_91]
2016-11-11T02:36:56,332 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_telemetry_1de5b639b1dd5b9_jhnmjplh] status change
d to [FAILED].
2016-11-11T02:36:56,335 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
“id” : “index_kafka_telemetry_1de5b639b1dd5b9_jhnmjplh”,
“status” : “FAILED”,
“duration” : 5320173

Failed_Task_Logs_Druid.txt (24.5 KB)

Hey Arpan,

Issue 1 is a bug that was in that was fixed in 0.9.2. If you’d like, you can download the 0.9.2-rc2 here and see if that fixes it:

For Issue 2, there should be a corresponding exception in the overlord logs that details why the transaction failed. One thing that’ll likely fix it is removing the entry for that datasource from the druid_dataSource metadata table.

Hi David,

Okay - yes I recently got to know about this new release of Druid but wants to know if it’s available with Imply ? Or I can directly copy folders from new release to current imply tar im using ?


Arpan Khagram

+91 8308993200

Hi Arpan,

Yes, you should be able to replace the contents of /dist/druid with the 0.9.2-rc2 version. An official Imply release will come out sometime after 0.9.2 is declared stable.

Thanks David. Done and looks to be working fine.