Kafka-indexing-service task fail without any reason

hi, all:

We use druid 0.11.0 on the production env for 2years , but recently, we have a issue that it seems like the indexing task fail without any reason.

The end log show nothing.

2019-10-10T17:08:55,077 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:08:55.077Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8101”,“version”:“0.11.0”,“metric”:“ingest/handoff/failed”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh”]}]

2019-10-10T17:08:55,077 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:08:55.077Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8101”,“version”:“0.11.0”,“metric”:“ingest/merge/time”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh”]}]

2019-10-10T17:08:55,077 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:08:55.077Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8101”,“version”:“0.11.0”,“metric”:“ingest/merge/cpu”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh”]}]

2019-10-10T17:08:55,077 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:08:55.077Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8101”,“version”:“0.11.0”,“metric”:“ingest/handoff/count”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh”]}]

2019-10-10T17:08:55,077 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:08:55.077Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8101”,“version”:“0.11.0”,“metric”:“ingest/sink/count”,“value”:37,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh”]}]

2019-10-10T17:08:55,077 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:08:55.077Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8101”,“version”:“0.11.0”,“metric”:“ingest/events/messageGap”,“value”:251702,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh”]}]

2019-10-10T17:08:59,939 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorDriver - New segment[transaction_crm_merchant_org_ds_2019-10-10T08:00:00.000Z_2019-10-10T09:00:00.000Z_2019-10-10T08:00:09.687Z_23] for sequenceName[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_5].

2019-10-10T17:08:59,951 INFO [task-runner-0-priority-0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[transaction_crm_merchant_org_ds_2019-10-10T08:00:00.000Z_2019-10-10T09:00:00.000Z_2019-10-10T08:00:09.687Z_23] at existing path[/druid_olap/segments/druid-others-001:8101/druid-others-001:8101_indexer-executor__default_tier_2019-10-10T16:06:57.231Z_334172ae273d46fd87b126b8d8dc734c0]

2019-10-10T17:08:59,951 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_org_ds’, timestamp=2019-10-09T22:32:43.335Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3’, previousSegmentId=‘transaction_crm_merchant_org_ds_2019-10-10T17:00:00.000Z_2019-10-10T18:00:00.000Z_2019-10-10T17:00:12.808Z_6’}

2019-10-10T17:08:59,952 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] to overlord: [SegmentAllocateAction{dataSource=‘transaction_crm_merchant_org_ds’, timestamp=2019-10-09T22:32:43.335Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3’, previousSegmentId=‘transaction_crm_merchant_org_ds_2019-10-10T17:00:00.000Z_2019-10-10T18:00:00.000Z_2019-10-10T17:00:12.808Z_6’}].

2019-10-10T17:09:07,057 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorDriver - New segment[transaction_crm_merchant_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.282Z_27] for sequenceName[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3].

2019-10-10T17:09:07,069 INFO [task-runner-0-priority-0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[transaction_crm_merchant_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.282Z_27] at existing path[/druid_olap/segments/druid-others-001:8101/druid-others-001:8101_indexer-executor__default_tier_2019-10-10T16:06:57.231Z_334172ae273d46fd87b126b8d8dc734c0]

2019-10-10T17:09:07,070 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_org_ds’, timestamp=2019-10-10T00:09:14.901Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3’, previousSegmentId=‘transaction_crm_merchant_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.282Z_27’}

2019-10-10T17:09:07,070 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] to overlord: [SegmentAllocateAction{dataSource=‘transaction_crm_merchant_org_ds’, timestamp=2019-10-10T00:09:14.901Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3’, previousSegmentId=‘transaction_crm_merchant_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.282Z_27’}].

Is anyone meet this situation?

index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh (2.73 MB)

From another task consume same topic, it seems like the segments push to HDFS hang! How can I improve segments pushment throughout?

leManager",“host”:“druid-others-001:8100”,“version”:“0.11.0”,“metric”:“ingest/persists/failed”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff”]}]

2019-10-10T18:15:21,803 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T18:15:21.803Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8100”,“version”:“0.11.0”,“metric”:“ingest/handoff/failed”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff”]}]

2019-10-10T18:15:21,803 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T18:15:21.803Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8100”,“version”:“0.11.0”,“metric”:“ingest/merge/time”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff”]}]

2019-10-10T18:15:21,803 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T18:15:21.803Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8100”,“version”:“0.11.0”,“metric”:“ingest/merge/cpu”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff”]}]

2019-10-10T18:15:21,803 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T18:15:21.803Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8100”,“version”:“0.11.0”,“metric”:“ingest/handoff/count”,“value”:0,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff”]}]

2019-10-10T18:15:21,803 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T18:15:21.803Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8100”,“version”:“0.11.0”,“metric”:“ingest/sink/count”,“value”:122,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff”]}]

2019-10-10T18:15:21,803 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T18:15:21.803Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8100”,“version”:“0.11.0”,“metric”:“ingest/events/messageGap”,“value”:93878,“dataSource”:“transaction_crm_merchant_org_ds”,“taskId”:[“index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff”]}]

2019-10-10T18:15:21,908 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,923 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,927 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,933 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,937 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,942 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,945 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,949 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,953 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,956 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,960 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,965 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,969 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,973 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,978 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,982 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,987 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,991 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,994 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:21,997 INFO [coordinator_handoff_scheduled_0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8081

2019-10-10T18:15:22,211 INFO [coordinator_handoff_scheduled_0] io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for Segments : [[SegmentDescriptor{interval=2019-10-10T08:00:00.000Z/2019-10-10T09:00:00.000Z, version=‘2019-10-10T08:00:09.687Z’, partitionNumber=25}, SegmentDescriptor{interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, version=‘2019-10-10T16:00:02.742Z’, partitionNumber=20}, SegmentDescriptor{interval=2019-10-10T10:00:00.000Z/2019-10-10T11:00:00.000Z, version=‘2019-10-10T10:00:08.459Z’, partitionNumber=34}, SegmentDescriptor{interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, version=‘2019-10-10T16:00:02.742Z’, partitionNumber=21}, SegmentDescriptor{interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, version=‘2019-10-10T16:00:02.742Z’, partitionNumber=22}, SegmentDescriptor{interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, version=‘2019-10-10T16:00:02.742Z’, partitionNumber=23},

在 2019年10月11日星期五 UTC+8下午5:57:07,朱健写道:

Please check Middle manager and overlord log for index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh if you don’t see any error in the task log and task has failed.

Thanks,

Vaibhav

thanks for your advice.

I’ve got some information in middle-manager.log:

2019-10-11 01:08:44,089 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:08:44.089Z”,“service”:“druid/middleManager”,“host”:“druid-others-001:8091”,“version”:“0.11.0”,“metric”:“jvm/gc/mem/init”,“value”:44761088,“gcGen”:[“old”],“gcGenSpaceName”:“sun.gc.generation.1.space.0.name: old\u0000 string [internal]”,“gcName”:[“cms”]}]

2019-10-11 01:09:11,129 INFO [qtp1651251663-57] io.druid.indexing.overlord.ForkingTaskRunner - Killing process for task: index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 01:09:11,130 INFO [forking-task-runner-7] io.druid.storage.hdfs.tasklog.HdfsTaskLogs - Writing task log to: /druid_olap/indexing-logs/index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 01:09:11,180 INFO [forking-task-runner-7] io.druid.storage.hdfs.tasklog.HdfsTaskLogs - Wrote task log to: /druid_olap/indexing-logs/index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 01:09:11,180 INFO [forking-task-runner-7] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution

java.io.IOException: Stream closed

at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[?:1.8.0_162]

at java.io.BufferedInputStream.read1(BufferedInputStream.java:291) ~[?:1.8.0_162]

at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_162]

at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_162]

at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:?]

at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:452) [druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:224) [druid-indexing-service-0.11.0.jar:0.11.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]

2019-10-11 01:09:11,180 INFO [forking-task-runner-7] io.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 01:09:11,189 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Job’s finished. Completed [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] with status [FAILED]

2019-10-11 01:09:16,916 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Submitting runnable for task[index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff]

2019-10-11 01:09:16,919 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Affirmative. Running task [index_kafka_transaction_crm_merchant_org_ds_664898b21b0d98a_mlfcbmff]

在 2019年10月11日星期五 UTC+8下午6:19:08,Vaibhav Vaibhav写道:

log in overload:

superuser@druid-others-002:/app/druid/log/druid$ grep ‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh’ overlord-2019-10-10.1.log -C10

2019-10-11 00:06:46,641 INFO [KafkaIndexTaskClient-crm_bd_capacity_ds-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-002:8102

2019-10-11 00:06:47,183 INFO [KafkaSupervisor-transaction_crm_store_org_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘transaction_crm_store_org_ds’, generationTime=2019-10-10T16:06:47.183Z, payload={dataSource=‘transaction_crm_store_org_ds’, topic=‘transaction_crm_store_org’, partitions=10, replicas=1, durationSeconds=3600, active=[{id=‘index_kafka_transaction_crm_store_org_ds_1e10c6a7f1baa91_fadhbjpk’, startTime=2019-10-10T15:43:53.528Z, remainingSeconds=2226}, {id=‘index_kafka_transaction_crm_store_org_ds_4f082286f80947e_bpnnckap’, startTime=2019-10-10T15:51:17.642Z, remainingSeconds=2670}], publishing=}}

2019-10-11 00:06:47,636 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T16:06:47.636Z”,“service”:“druid/overlord”,“host”:“druid-others-002:8090”,“version”:“0.11.0”,“metric”:“jetty/numOpenConnections”,“value”:20}]

2019-10-11 00:06:49,629 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Task group [1] has run for [PT3600S]

2019-10-11 00:06:49,633 INFO [KafkaIndexTaskClient-transaction_crm_merchant_org_ds-1] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-002:8101

2019-10-11 00:06:49,697 INFO [KafkaIndexTaskClient-transaction_crm_merchant_org_ds-1] io.druid.indexing.kafka.KafkaIndexTaskClient - Task [index_kafka_transaction_crm_merchant_org_ds_8787a46c1fd81ab_ohgljhhg] paused successfully

2019-10-11 00:06:49,697 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Setting endOffsets for tasks in taskGroup [1] to {1=872409919, 3=872390684, 5=872375262, 7=872400859, 9=872374534} and resuming

2019-10-11 00:06:49,700 INFO [KafkaIndexTaskClient-transaction_crm_merchant_org_ds-0] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-002:8101

2019-10-11 00:06:50,078 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Creating new task group [1] for partitions [1, 3, 5, 7, 9]

2019-10-11 00:06:50,078 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Number of tasks [0] does not match configured numReplicas [1] in task group [1], creating more tasks

2019-10-11 00:06:50,078 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds] io.druid.indexing.overlord.MetadataTaskStorage - Inserting task index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh with status: TaskStatus{id=index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh, status=RUNNING, duration=-1}

2019-10-11 00:06:50,084 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds] io.druid.indexing.overlord.TaskLockbox - Adding task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] to activeTasks

2019-10-11 00:06:50,084 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 00:06:50,084 INFO [TaskQueue-Manager] io.druid.indexing.overlord.RemoteTaskRunner - Added pending task index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 00:06:50,084 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘transaction_crm_merchant_org_ds’, generationTime=2019-10-10T16:06:50.084Z, payload={dataSource=‘transaction_crm_merchant_org_ds’, topic=‘transaction_crm_merchant_org’, partitions=10, replicas=1, durationSeconds=3600, active=[{id=‘index_kafka_transaction_crm_merchant_org_ds_994131cb0527b58_pidolkne’, startTime=2019-10-10T15:14:47.060Z, remainingSeconds=476}], publishing=[{id=‘index_kafka_transaction_crm_merchant_org_ds_8787a46c1fd81ab_ohgljhhg’, startTime=2019-10-10T15:06:47.184Z, remainingSeconds=1799}]}}

2019-10-11 00:06:50,084 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - No worker selection strategy set. Using default of [EqualDistributionWorkerSelectStrategy]

2019-10-11 00:06:50,084 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - Coordinator asking Worker[druid-others-001:8091] to add task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]

2019-10-11 00:06:50,090 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.RemoteTaskRunner - Task index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh switched from pending to running (on [druid-others-001:8091])

2019-10-11 00:06:50,090 INFO [rtr-pending-tasks-runner-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] status changed to [RUNNING].

2019-10-11 00:06:50,103 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Worker[druid-others-001:8091] wrote RUNNING status for task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] on [TaskLocation{host=‘null’, port=-1, tlsPort=-1}]

2019-10-11 00:06:50,107 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Worker[druid-others-001:8091] wrote RUNNING status for task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] on [TaskLocation{host=‘druid-others-001’, port=8101, tlsPort=-1}]

2019-10-11 00:06:50,107 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] location changed to [TaskLocation{host=‘druid-others-001’, port=8101, tlsPort=-1}].

2019-10-11 00:06:55,205 INFO [qtp1108651663-102] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_org_ds_8787a46c1fd81ab_ohgljhhg]: SegmentInsertAction{segments=[DataSegment{size=300660, shardSpec=NumberedShardSpec{partitionNum=8, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/8_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=10066, shardSpec=NumberedShardSpec{partitionNum=21, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, merchant_id, pay_way], version=‘2019-10-10T14:00:08.677Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T140000.000Z_20191010T150000.000Z/2019-10-10T14_00_08.677Z/21_index.zip}, interval=2019-10-10T14:00:00.000Z/2019-10-10T15:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=297707, shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/0_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2124860, shardSpec=NumberedShardSpec{partitionNum=14, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/14_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=300290, shardSpec=NumberedShardSpec{partitionNum=5, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/5_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2139588, shardSpec=NumberedShardSpec{partitionNum=12, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/12_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=280609, shardSpec=NumberedShardSpec{partitionNum=2, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/2_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2119068, shardSpec=NumberedShardSpec{partitionNum=13, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/13_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=10066, shardSpec=NumberedShardSpec{partitionNum=20, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, merchant_id, pay_way], version=‘2019-10-10T14:00:08.677Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T140000.000Z_20191010T150000.000Z/2019-10-10T14_00_08.677Z/20_index.zip}, interval=2019-10-10T14:00:00.000Z/2019-10-10T15:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2122732, shardSpec=NumberedShardSpec{partitionNum=10, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/10_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=297469, shardSpec=NumberedShardSpec{partitionNum=1, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/1_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2146424, shardSpec=NumberedShardSpec{partitionNum=11, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, level8, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/11_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}], startMetadata=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic=‘transaction_crm_merchant_org’, partitionOffsetMap={1=872369451, 3=872350723, 5=872334957, 7=872360299, 9=872334498}}}, endMetadata=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic=‘transaction_crm_merchant_org’, partitionOffsetMap={1=872409919, 3=872390684, 5=872375262, 7=872400859, 9=872374534}}}}

2019-10-11 00:06:55,206 INFO [qtp1108651663-102] io.druid.indexing.overlord.MetadataTaskStorage - Logging action for task[index_kafka_transaction_crm_merchant_org_ds_8787a46c1fd81ab_ohgljhhg]: SegmentInsertAction{segments=[DataSegment{size=300660, shardSpec=NumberedShardSpec{partitionNum=8, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/8_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=10066, shardSpec=NumberedShardSpec{partitionNum=21, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, merchant_id, pay_way], version=‘2019-10-10T14:00:08.677Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T140000.000Z_20191010T150000.000Z/2019-10-10T14_00_08.677Z/21_index.zip}, interval=2019-10-10T14:00:00.000Z/2019-10-10T15:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=297707, shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/0_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2124860, shardSpec=NumberedShardSpec{partitionNum=14, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/14_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=300290, shardSpec=NumberedShardSpec{partitionNum=5, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/5_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2139588, shardSpec=NumberedShardSpec{partitionNum=12, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/12_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=280609, shardSpec=NumberedShardSpec{partitionNum=2, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/2_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2119068, shardSpec=NumberedShardSpec{partitionNum=13, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/13_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=10066, shardSpec=NumberedShardSpec{partitionNum=20, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, merchant_id, pay_way], version=‘2019-10-10T14:00:08.677Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T140000.000Z_20191010T150000.000Z/2019-10-10T14_00_08.677Z/20_index.zip}, interval=2019-10-10T14:00:00.000Z/2019-10-10T15:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2122732, shardSpec=NumberedShardSpec{partitionNum=10, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/10_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=297469, shardSpec=NumberedShardSpec{partitionNum=1, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, merchant_id, pay_way], version=‘2019-10-10T16:00:02.742Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T160000.000Z_20191010T170000.000Z/2019-10-10T16_00_02.742Z/1_index.zip}, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}, DataSegment{size=2146424, shardSpec=NumberedShardSpec{partitionNum=11, partitions=0}, metrics=[cnt, amount, ef_amount, ef_cnt, success_cnt, success_amount, refund_cancel_cnt, refund_cancel_amount], dimensions=[level1, level2, level3, level4, level5, level6, level7, level8, merchant_id, pay_way], version=‘2019-10-10T15:00:07.736Z’, loadSpec={type=hdfs, path=hdfs://s1-kafka-001:9000/druid_olap/segments/transaction_crm_merchant_org_ds/20191010T150000.000Z_20191010T160000.000Z/2019-10-10T15_00_07.736Z/11_index.zip}, interval=2019-10-10T15:00:00.000Z/2019-10-10T16:00:00.000Z, dataSource=‘transaction_crm_merchant_org_ds’, binaryVersion=‘9’}], startMetadata=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic=‘transaction_crm_merchant_org’, partitionOffsetMap={1=872369451, 3=872350723, 5=872334957, 7=872360299, 9=872334498}}}, endMetadata=KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic=‘transaction_crm_merchant_org’, partitionOffsetMap={1=872409919, 3=872390684, 5=872375262, 7=872400859, 9=872374534}}}}

2019-10-11 00:06:55,221 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Updated metadata from[KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic=‘transaction_crm_merchant_org’, partitionOffsetMap={0=872310801, 1=872369451, 2=872368651, 3=872350723, 4=872342067, 5=872334957, 6=872347539, 7=872360299, 8=872330651, 9=872334498}}}] to[KafkaDataSourceMetadata{kafkaPartitions=KafkaPartitions{topic=‘transaction_crm_merchant_org’, partitionOffsetMap={0=872310801, 1=872409919, 2=872368651, 3=872390684, 4=872342067, 5=872375262, 6=872347539, 7=872400859, 8=872330651, 9=872374534}}}].

2019-10-11 00:06:55,223 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Published segment [transaction_crm_merchant_org_ds_2019-10-10T16:00:00.000Z_2019-10-10T17:00:00.000Z_2019-10-10T16:00:02.742Z_8] to DB with used flag [true]

2019-10-11 00:06:55,225 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Published segment [transaction_crm_merchant_org_ds_2019-10-10T14:00:00.000Z_2019-10-10T15:00:00.000Z_2019-10-10T14:00:08.677Z_21] to DB with used flag [true]

2019-10-11 00:06:55,228 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Published segment [transaction_crm_merchant_org_ds_2019-10-10T16:00:00.000Z_2019-10-10T17:00:00.000Z_2019-10-10T16:00:02.742Z] to DB with used flag [true]

2019-10-11 00:06:55,230 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Published segment [transaction_crm_merchant_org_ds_2019-10-10T15:00:00.000Z_2019-10-10T16:00:00.000Z_2019-10-10T15:00:07.736Z_14] to DB with used flag [true]

2019-10-11 00:06:55,232 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Published segment [transaction_crm_merchant_org_ds_2019-10-10T16:00:00.000Z_2019-10-10T17:00:00.000Z_2019-10-10T16:00:02.742Z_5] to DB with used flag [true]

2019-10-11 00:06:55,234 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Published segment [transaction_crm_merchant_org_ds_2019-10-10T15:00:00.000Z_2019-10-10T16:00:00.000Z_2019-10-10T15:00:07.736Z_12] to DB with used flag [true]

2019-10-11 00:06:55,236 INFO [qtp1108651663-102] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Published segment [transaction_crm_merchant_org_ds_2019-10-10T16:00:00.000Z_2019-10-10T17:00:00.000Z_2019-10-10T16:00:02.742Z_2] to DB with used flag [true]

from overlord.log , There isn’t any obvious cause , The task status just assigned to FAIL and removed.

2019-10-11 01:09:11,123 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 12 tasks from storage (0 tasks added, 0 tasks removed).

2019-10-11 01:09:11,127 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-001:8091

2019-10-11 01:09:11,130 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: druid-others-001:8091, status 200 OK, response: {“task”:“index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh”}

2019-10-11 01:09:11,130 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from activeTasks

2019-10-11 01:09:11,130 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,130 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-09T17:00:00.000Z/2019-10-09T18:00:00.000Z, version=2019-10-10T17:05:09.865Z}

2019-10-11 01:09:11,132 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442516]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-09T17:00:00.000Z/2019-10-09T18:00:00.000Z, version=2019-10-10T17:05:09.865Z}

2019-10-11 01:09:11,135 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,137 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442502]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-09T18:00:00.000Z/2019-10-09T19:00:00.000Z, version=2019-10-10T17:04:41.584Z}

2019-10-11 01:09:11,139 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,141 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442510]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-09T22:00:00.000Z/2019-10-09T23:00:00.000Z, version=2019-10-10T17:04:52.667Z}

2019-10-11 01:09:11,144 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,146 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442522]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-09T23:00:00.000Z/2019-10-10T00:00:00.000Z, version=2019-10-10T17:05:17.081Z}

2019-10-11 01:09:11,148 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,150 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442528]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T00:00:00.000Z/2019-10-10T01:00:00.000Z, version=2019-10-10T17:05:24.543Z}

2019-10-11 01:09:11,153 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,155 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442537]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T01:00:00.000Z/2019-10-10T02:00:00.000Z, version=2019-10-10T17:05:47.784Z}

2019-10-11 01:09:11,158 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,160 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442544]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T02:00:00.000Z/2019-10-10T03:00:00.000Z, version=2019-10-10T17:06:04.134Z}

2019-10-11 01:09:11,162 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,164 INFO [qtp1108651663-124] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_mpgbpacc]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_keeper_ds’, timestamp=2019-10-09T17:42:45.292Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_2’, previousSegmentId=‘transaction_crm_merchant_keeper_ds_2019-10-10T15:00:00.000Z_2019-10-10T16:00:00.000Z_2019-10-10T15:00:07.879Z_20’}

2019-10-11 01:09:11,164 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442552]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T03:00:00.000Z/2019-10-10T04:00:00.000Z, version=2019-10-10T17:05:58.853Z}

2019-10-11 01:09:11,190 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Worker[druid-others-001:8091] wrote FAILED status for task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] on [TaskLocation{host=‘druid-others-001’, port=8101, tlsPort=-1}]

2019-10-11 01:09:11,190 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Worker[druid-others-001:8091] completed task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] with status[FAILED]

2019-10-11 01:09:11,190 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 01:09:11,197 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,198 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442559]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T04:00:00.000Z/2019-10-10T05:00:00.000Z, version=2019-10-10T17:06:12.783Z}

2019-10-11 01:09:11,201 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,203 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442569]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T05:00:00.000Z/2019-10-10T06:00:00.000Z, version=2019-10-10T17:06:34.669Z}

2019-10-11 01:09:11,206 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,207 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442577]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T06:00:00.000Z/2019-10-10T07:00:00.000Z, version=2019-10-10T17:06:49.812Z}

2019-10-11 01:09:11,210 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,212 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442585]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T07:00:00.000Z/2019-10-10T08:00:00.000Z, version=2019-10-10T17:07:05.908Z}

2019-10-11 01:09:11,214 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,216 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442616]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T08:00:00.000Z/2019-10-10T09:00:00.000Z, version=2019-10-10T17:07:45.757Z}

2019-10-11 01:09:11,219 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,220 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442593]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T09:00:00.000Z/2019-10-10T10:00:00.000Z, version=2019-10-10T17:06:59.667Z}

2019-10-11 01:09:11,223 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,225 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442602]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T10:00:00.000Z/2019-10-10T11:00:00.000Z, version=2019-10-10T17:07:12.223Z}

2019-10-11 01:09:11,227 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,229 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442609]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T13:00:00.000Z/2019-10-10T14:00:00.000Z, version=2019-10-10T17:07:29.858Z}

2019-10-11 01:09:11,232 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,233 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442475]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T16:00:00.000Z/2019-10-10T17:00:00.000Z, version=2019-10-10T16:00:02.742Z}

2019-10-11 01:09:11,236 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] from TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:11,238 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[442488]: TaskLock{groupId=index_kafka_transaction_crm_merchant_org_ds, dataSource=transaction_crm_merchant_org_ds, interval=2019-10-10T17:00:00.000Z/2019-10-10T18:00:00.000Z, version=2019-10-10T17:00:12.808Z}

2019-10-11 01:09:11,240 INFO [qtp1108651663-124] io.druid.indexing.overlord.TaskLockbox - Added task[index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_mpgbpacc] to TaskLock[index_kafka_transaction_crm_merchant_keeper_ds]

2019-10-11 01:09:11,240 INFO [qtp1108651663-124] io.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2019-10-09T17:00:00.000Z/2019-10-09T18:00:00.000Z] version[2019-10-10T17:05:11.115Z] for task: index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_mpgbpacc

2019-10-11 01:09:11,243 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.MetadataTaskStorage - Updating task index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh to status: TaskStatus{id=index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh, status=FAILED, duration=-1}

2019-10-11 01:09:11,307 INFO [qtp1108651663-113] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Allocated pending segment [transaction_crm_store_keeper_ds_2019-10-10T02:00:00.000Z_2019-10-10T03:00:00.000Z_2019-10-10T02:00:02.941Z_24] for sequence[index_kafka_transaction_crm_store_keeper_ds_ad6c39adbefb48d_9] (previous = [transaction_crm_store_keeper_ds_2019-10-10T01:00:00.000Z_2019-10-10T02:00:00.000Z_2019-10-10T01:00:08.652Z_23]) in DB

2019-10-11 01:09:11,312 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.overlord.TaskQueue - Task done: KafkaIndexTask{id=index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh, type=index_kafka, dataSource=transaction_crm_merchant_org_ds}

2019-10-11 01:09:11,312 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Worker-1] io.druid.indexing.kafka.supervisor.KafkaSupervisor - All tasks in taskGroup [1] have failed, tasks will be re-created

2019-10-11 01:09:11,312 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] on worker[druid-others-001:8091]

2019-10-11 01:09:11,312 WARN [KafkaSupervisor-transaction_crm_merchant_org_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - All tasks in group [1] failed to transition to publishing state, killing tasks []

2019-10-11 01:09:11,315 WARN [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskQueue - Unknown task completed: index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

2019-10-11 01:09:11,315 INFO [Curator-PathChildrenCache-1] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:09:11.315Z”,“service”:“druid/overlord”,“host”:“druid-others-002:8090”,“version”:“0.11.0”,“metric”:“task/run/time”,“value”:3741084,“dataSource”:“transaction_crm_merchant_org_ds”,“taskStatus”:“FAILED”,“taskType”:“index_kafka”}]

2019-10-11 01:09:11,315 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskQueue - Task FAILED: KafkaIndexTask{id=index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh, type=index_kafka, dataSource=transaction_crm_merchant_org_ds} (3741084 run duration)

2019-10-11 01:09:11,315 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] status changed to [FAILED].

2019-10-11 01:09:11,316 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] went bye bye.

2019-10-11 01:09:11,856 INFO [qtp1108651663-124] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Allocated pending segment [transaction_crm_merchant_keeper_ds_2019-10-09T17:00:00.000Z_2019-10-09T18:00:00.000Z_2019-10-09T17:00:08.228Z_21] for sequence[index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_2] (previous = [transaction_crm_merchant_keeper_ds_2019-10-10T15:00:00.000Z_2019-10-10T16:00:00.000Z_2019-10-10T15:00:07.879Z_20]) in DB

2019-10-11 01:09:12,023 INFO [KafkaIndexTaskClient-transaction_crm_merchant_org_ds-1] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-002:8101

2019-10-11 01:09:12,106 INFO [qtp1108651663-107] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_jkcknifn]: SegmentAllocateAction{dataSource=‘transaction_crm_store_org_ds’, timestamp=2019-10-09T22:24:27.933Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_9’, previousSegmentId=‘transaction_crm_store_org_ds_2019-10-10T17:00:00.000Z_2019-10-10T18:00:00.000Z_2019-10-10T17:00:24.701Z_8’}

2019-10-11 01:09:12,134 INFO [qtp1108651663-107] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_jkcknifn] already present in TaskLock[index_kafka_transaction_crm_store_org_ds]

2019-10-11 01:09:12,728 INFO [qtp1108651663-97] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_org_ds’, timestamp=2019-10-10T00:09:14.901Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3’, previousSegmentId=‘transaction_crm_merchant_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.282Z_27’}

2019-10-11 01:09:12,743 INFO [qtp1108651663-107] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Allocated pending segment [transaction_crm_store_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.311Z_24] for sequence[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_9] (previous = [transaction_crm_store_org_ds_2019-10-10T17:00:00.000Z_2019-10-10T18:00:00.000Z_2019-10-10T17:00:24.701Z_8]) in DB

2019-10-11 01:09:12,773 ERROR [qtp1108651663-97] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container

io.druid.java.util.common.ISE: Unable to grant lock to inactive Task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]

at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:267) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:244) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:177) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:54) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:64) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:339) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:328) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:635) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:325) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) ~[?:?]

After task status update to FAILED and removed, when re-create the task , the tasklock conflict, any suggestion about this ?

在 2019年10月11日星期五 UTC+8下午6:19:08,Vaibhav Vaibhav写道:

MM log: It clearly says Killing the task : The task has got KILL signal from the overlord hence the `Exception caught during execution

java.io.IOException: Stream closed`

2019-10-11 01:09:11,129 INFO [qtp1651251663-57] io.druid.indexing.overlord.ForkingTaskRunner - Killing process for task: index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh

Ovelord log: The log seems incomplegte, Please see the overlord around time 2019-10-11 01:09:11

The last line of the overlord log you have patsed is for**2019-10-11 00:09:24 **

2019-10-11 00:09:24,178 INFO [KafkaSupervisor-transaction_crm_merchant_keeper_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘transaction_crm_merchant_keeper_ds’, generationTime=2019-10-10T16:09:24.178Z

Thanks ,

Vaibhav

Since time 2019-10-11 01:09:11, there is always a tasklock problem.

2019-10-11 01:09:11,315 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] status changed to [FAILED].

2019-10-11 01:09:11,316 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh] went bye bye.

2019-10-11 01:09:11,856 INFO [qtp1108651663-124] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Allocated pending segment [transaction_crm_merchant_keeper_ds_2019-10-09T17:00:00.000Z_2019-10-09T18:00:00.000Z_2019-10-09T17:00:08.228Z_21] for sequence[index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_2] (previous = [transaction_crm_merchant_keeper_ds_2019-10-10T15:00:00.000Z_2019-10-10T16:00:00.000Z_2019-10-10T15:00:07.879Z_20]) in DB

2019-10-11 01:09:12,023 INFO [KafkaIndexTaskClient-transaction_crm_merchant_org_ds-1] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-others-002:8101

2019-10-11 01:09:12,106 INFO [qtp1108651663-107] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_jkcknifn]: SegmentAllocateAction{dataSource=‘transaction_crm_store_org_ds’, timestamp=2019-10-09T22:24:27.933Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_9’, previousSegmentId=‘transaction_crm_store_org_ds_2019-10-10T17:00:00.000Z_2019-10-10T18:00:00.000Z_2019-10-10T17:00:24.701Z_8’}

2019-10-11 01:09:12,134 INFO [qtp1108651663-107] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_jkcknifn] already present in TaskLock[index_kafka_transaction_crm_store_org_ds]

2019-10-11 01:09:12,728 INFO [qtp1108651663-97] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_org_ds’, timestamp=2019-10-10T00:09:14.901Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3’, previousSegmentId=‘transaction_crm_merchant_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.282Z_27’}

2019-10-11 01:09:12,743 INFO [qtp1108651663-107] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Allocated pending segment [transaction_crm_store_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.311Z_24] for sequence[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_9] (previous = [transaction_crm_store_org_ds_2019-10-10T17:00:00.000Z_2019-10-10T18:00:00.000Z_2019-10-10T17:00:24.701Z_8]) in DB

2019-10-11 01:09:12,773 ERROR [qtp1108651663-97] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container

io.druid.java.util.common.ISE: Unable to grant lock to inactive Task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]

at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:267) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:244) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:177) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:54) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:64) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:339) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:328) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:635) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:325) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) ~[?:?]

From 2019-10-11 01:09:11 to 2019-10-11 01:11:58,521 , there is always a tasklock problem, deadlock?

2019-10-11 01:09:21,442 INFO [qtp1108651663-96] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_store_org_ds_98e3e460d729cb0_amdmlefn]: SegmentAllocateAction{dataSource=‘transaction_crm_store_org_ds’, timestamp=2019-10-10T13:57:51.469Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_store_org_ds_98e3e460d729cb0_4’, previousSegmentId=‘transaction_crm_store_org_ds_2019-10-10T06:00:00.000Z_2019-10-10T07:00:00.000Z_2019-10-10T06:00:08.043Z_23’}

2019-10-11 01:09:22,021 WARN [KafkaSupervisor-transaction_crm_merchant_org_ds-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Lag metric: Kafka partitions [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] do not match task partitions [0, 2, 4, 6, 8]

2019-10-11 01:09:22,021 INFO [KafkaSupervisor-transaction_crm_merchant_org_ds-Reporting-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-10-10T17:09:22.021Z”,“service”:“druid/overlord”,“host”:“druid-others-002:8090”,“version”:“0.11.0”,“metric”:“ingest/kafka/lag”,“value”:10413,“dataSource”:“transaction_crm_merchant_org_ds”}]

2019-10-11 01:09:22,211 INFO [qtp1108651663-98] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_keeper_ds_78faa073e753151_labnmmhk]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_keeper_ds’, timestamp=2019-10-10T12:45:18.776Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_keeper_ds_78faa073e753151_5’, previousSegmentId=‘transaction_crm_merchant_keeper_ds_2019-10-10T08:00:00.000Z_2019-10-10T09:00:00.000Z_2019-10-10T08:00:08.485Z_22’}

2019-10-11 01:09:22,943 INFO [qtp1108651663-104] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_store_keeper_ds_ad6c39adbefb48d_dabdmeke]: SegmentAllocateAction{dataSource=‘transaction_crm_store_keeper_ds’, timestamp=2019-10-10T04:20:45.259Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_store_keeper_ds_ad6c39adbefb48d_9’, previousSegmentId=‘transaction_crm_store_keeper_ds_2019-10-10T03:00:00.000Z_2019-10-10T04:00:00.000Z_2019-10-10T03:00:09.502Z_24’}

2019-10-11 01:09:23,920 INFO [qtp1108651663-122] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_mpgbpacc]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_keeper_ds’, timestamp=2019-10-10T02:11:20.284Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_2’, previousSegmentId=‘transaction_crm_merchant_keeper_ds_2019-10-09T23:00:00.000Z_2019-10-10T00:00:00.000Z_2019-10-09T23:00:07.098Z_23’}

2019-10-11 01:09:24,336 INFO [qtp1108651663-110] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_transaction_crm_merchant_org_ds_5b0d7c46d692076_ichaoinh] already present in TaskLock[index_kafka_transaction_crm_merchant_org_ds]

2019-10-11 01:09:24,357 INFO [qtp1108651663-105] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_jkcknifn]: SegmentAllocateAction{dataSource=‘transaction_crm_store_org_ds’, timestamp=2019-10-10T00:43:00.562Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_9’, previousSegmentId=‘transaction_crm_store_org_ds_2019-10-09T23:00:00.000Z_2019-10-10T00:00:00.000Z_2019-10-09T23:00:07.795Z_24’}

2019-10-11 01:09:24,364 INFO [qtp1108651663-96] io.druid.indexing.overlord.TaskLockbox - Added task[index_kafka_transaction_crm_store_org_ds_98e3e460d729cb0_amdmlefn] to TaskLock[index_kafka_transaction_crm_store_org_ds]

2019-10-11 01:09:24,364 INFO [qtp1108651663-96] io.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2019-10-10T13:00:00.000Z/2019-10-10T14:00:00.000Z] version[2019-10-10T17:08:54.114Z] for task: index_kafka_transaction_crm_store_org_ds_98e3e460d729cb0_amdmlefn

2019-10-11 01:09:24,886 INFO [qtp1108651663-106] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]: SegmentAllocateAction{dataSource=‘transaction_crm_merchant_org_ds’, timestamp=2019-10-10T00:09:14.901Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_3’, previousSegmentId=‘transaction_crm_merchant_org_ds_2019-10-09T22:00:00.000Z_2019-10-09T23:00:00.000Z_2019-10-09T22:00:02.282Z_27’}

2019-10-11 01:09:24,974 INFO [qtp1108651663-126] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_transaction_crm_store_keeper_ds_db3adac35e9a62c_nceaeahf]: SegmentAllocateAction{dataSource=‘transaction_crm_store_keeper_ds’, timestamp=2019-10-10T00:12:00.936Z, queryGranularity=DurationGranularity{duration=5000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_transaction_crm_store_keeper_ds_db3adac35e9a62c_6’, previousSegmentId=‘transaction_crm_store_keeper_ds_2019-10-09T23:00:00.000Z_2019-10-10T00:00:00.000Z_2019-10-09T23:00:07.202Z_25’}

2019-10-11 01:09:25,049 INFO [qtp1108651663-110] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Allocated pending segment [transaction_crm_merchant_org_ds_2019-10-10T05:00:00.000Z_2019-10-10T06:00:00.000Z_2019-10-10T05:00:11.335Z_28] for sequence[index_kafka_transaction_crm_merchant_org_ds_5b0d7c46d692076_6] (previous = [transaction_crm_merchant_org_ds_2019-10-10T13:00:00.000Z_2019-10-10T14:00:00.000Z_2019-10-10T13:00:04.463Z_23]) in DB

2019-10-11 01:09:25,083 INFO [qtp1108651663-98] io.druid.indexing.overlord.TaskLockbox - Added task[index_kafka_transaction_crm_merchant_keeper_ds_78faa073e753151_labnmmhk] to TaskLock[index_kafka_transaction_crm_merchant_keeper_ds]

2019-10-11 01:09:25,083 INFO [qtp1108651663-98] io.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2019-10-10T12:00:00.000Z/2019-10-10T13:00:00.000Z] version[2019-10-10T17:07:22.844Z] for task: index_kafka_transaction_crm_merchant_keeper_ds_78faa073e753151_labnmmhk

2019-10-11 01:09:25,123 INFO [qtp1108651663-113] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_crm_bd_capacity_ds_5623f306ec8bf88_gdfgjbhd]: SegmentAllocateAction{dataSource=‘crm_bd_capacity_ds’, timestamp=2019-10-10T17:09:25.000Z, queryGranularity=DurationGranularity{duration=1000, origin=0}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName=‘index_kafka_crm_bd_capacity_ds_5623f306ec8bf88_2’, previousSegmentId=‘crm_bd_capacity_ds_2019-10-10T16:00:00.000Z_2019-10-10T17:00:00.000Z_2019-10-10T16:15:56.532Z_4’}

2019-10-11 01:09:25,700 INFO [qtp1108651663-96] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Allocated pending segment [transaction_crm_store_org_ds_2019-10-10T13:00:00.000Z_2019-10-10T14:00:00.000Z_2019-10-10T13:00:05.660Z_21] for sequence[index_kafka_transaction_crm_store_org_ds_98e3e460d729cb0_4] (previous = [transaction_crm_store_org_ds_2019-10-10T06:00:00.000Z_2019-10-10T07:00:00.000Z_2019-10-10T06:00:08.043Z_23]) in DB

2019-10-11 01:09:25,774 INFO [KafkaSupervisor-crm_bd_capacity_ds] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘crm_bd_capacity_ds’, generationTime=2019-10-10T17:09:25.774Z, payload={dataSource=‘crm_bd_capacity_ds’, topic=‘crm.capacity.event’, partitions=3, replicas=1, durationSeconds=3600, active=[{id=‘index_kafka_crm_bd_capacity_ds_5623f306ec8bf88_gdfgjbhd’, startTime=2019-10-10T16:17:29.939Z, remainingSeconds=484}], publishing=}}

2019-10-11 01:09:25,777 INFO [qtp1108651663-104] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_transaction_crm_store_keeper_ds_ad6c39adbefb48d_dabdmeke] already present in TaskLock[index_kafka_transaction_crm_store_keeper_ds]

2019-10-11 01:09:25,777 INFO [qtp1108651663-122] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_transaction_crm_merchant_keeper_ds_22117d9986c3f22_mpgbpacc] already present in TaskLock[index_kafka_transaction_crm_merchant_keeper_ds]

2019-10-11 01:09:25,777 INFO [qtp1108651663-105] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_transaction_crm_store_org_ds_0789cdd72eb9175_jkcknifn] already present in TaskLock[index_kafka_transaction_crm_store_org_ds]

2019-10-11 01:09:25,778 INFO [qtp1108651663-126] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_transaction_crm_store_keeper_ds_db3adac35e9a62c_nceaeahf] already present in TaskLock[index_kafka_transaction_crm_store_keeper_ds]

2019-10-11 01:09:25,806 INFO [qtp1108651663-113] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_crm_bd_capacity_ds_5623f306ec8bf88_gdfgjbhd] already present in TaskLock[index_kafka_crm_bd_capacity_ds]

2019-10-11 01:09:25,778 ERROR [qtp1108651663-106] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container

io.druid.java.util.common.ISE: Unable to grant lock to inactive Task [index_kafka_transaction_crm_merchant_org_ds_6566349e42a5e35_jjlhblgh]

at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:267) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:244) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:177) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.SegmentAllocateAction.perform(SegmentAllocateAction.java:54) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:64) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:339) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:328) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:635) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at io.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:325) ~[druid-indexing-service-0.11.0.jar:0.11.0]

at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) ~[?:?]

I’ve found the reason of this problem. At 01:10 there is a hive job running and The HDFS balancer also running. Some datanode’s iostat metric util% is 100. So the stream closed.

@Vaibhav Vaibhav Thanks a lot for your advice.

在 2019年10月11日星期五 UTC+8下午7:15:37,Vaibhav Vaibhav写道:

:+1: