realtime index job never finishes

Hi all

I have a realtime index job running since 2016-7-25 but now it’s still running.

By clicking the payload link from the coordinator console I can see it’s supposed to be shutdown by 2016-07-25-19:00:00.

I’m wondering why it’s blocking. Any ideas?

fyi my druid version is 0.8.0 and here are some of the tail logs

2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/gc/count","value":0,"gcName":"G1 Young Generation"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/gc/time","value":0,"gcName":"G1 Old Generation"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/gc/count","value":0,"gcName":"G1 Old Generation"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/bufferpool/capacity","value":1548152977,"bufferpoolName":"direct"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/bufferpool/used","value":1548152977,"bufferpoolName":"direct"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/bufferpool/count","value":12041,"bufferpoolName":"direct"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/bufferpool/capacity","value":2079899244,"bufferpoolName":"mapped"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/bufferpool/used","value":2079899244,"bufferpoolName":"mapped"}]
2016-08-19T01:15:20,870 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:20.870Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"jvm/bufferpool/count","value":158,"bufferpoolName":"mapped"}]
2016-08-19T01:15:23,813 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:23.813Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"ingest/events/thrownAway","value":0,"dataSource":"MiddletiersBillingProd"}]
2016-08-19T01:15:23,813 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:23.813Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"ingest/events/unparseable","value":0,"dataSource":"MiddletiersBillingProd"}]
2016-08-19T01:15:23,813 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-08-19T01:15:23.813Z","service":"druid/miuiad/production/prod/middlemanager","host":"c3-miui-ad-sup41.bj:8102","metric":"ingest/events/processed","value":0,"dataSource":"MiddletiersBillingProd"}]

Can you share your entire task json?

The most common reason for this is problems with handoff. See here: https://github.com/druid-io/tranquility/blob/master/docs/trouble.md#my-tasks-are-never-exiting. Also you could consider upgrading Druid, if possible. There have been a lot of improvements and fixes since 0.8.0.