Datasource not created after Kafka indexing

I am using Druid 0.9.1.1 and am testing Kafka indexing service. I have a topic in Kafka with around 98M records. I have created Kafka indexing supervisor and defined a new, non-existing before, datasource name. During the ingestion process I was able to query this new datasource with Pivot. I assume this was done by middle manager itself.

Now. Indexing task has finished, publishing task has finished, I see in logs:

2016-07-22T16:57:25,460 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo",
  "status" : "SUCCESS",
  "duration" : 7139994
}

and on my middle manager

2016-07-22T16:57:26,656 INFO [forking-task-runner-1] io.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[0] for task: index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo

2016-07-22T16:57:26,656 INFO [forking-task-runner-1] io.druid.storage.s3.S3TaskLogs - Pushing task log var/druid/task/index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo/log to: druid/indexing-logs/index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo/log

2016-07-22T16:57:27,881 INFO [forking-task-runner-1] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] status changed to [SUCCESS].

2016-07-22T16:57:27,888 INFO [forking-task-runner-1] io.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo

2016-07-22T16:57:30,337 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Job's finished. Completed [index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] with status [SUCCESS]

and on my coordinator:

2016-07-22T16:57:36,592 INFO [ServerInventoryView-0] io.druid.client.BatchServerInventoryView - Server Disappeared[DruidServerMetadata{name=‘ip-10-0-0-94.eu-west-1.compute.internal:8102’, host=‘ip-10-0-0-94.eu-west-1.compute.internal:8102’, maxSize=0, tier=’_default_tier’, type=‘indexer-executor’, priority=‘0’}]

But, there is no such datasource in Druid console and in output of “/druid/coordinator/v1/metadata/datasources” and Pivot now says “Error: No such datasource”.

What happened?

Thank you in advance,

Nikita

Hi Nikita,

Can you post the logs from the overlord console for that task?

Do you mean this attached log?

Just in case, here is the relevant portion from overlord log file:

2016-07-22T16:57:32,522 INFO [KafkaSupervisor-load-test-2-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘load-test-2’, generationTime=2016-07-22T16:57:32.522Z, payload={dataSource=‘load-test-2’, topic=‘load-test’, partitions=5, replicas=1, durationSeconds=3600, active=[{id=‘index_kafka_load-test-2_69312fa8fedba0e_dfjnbeel’, startTime=2016-07-22T16:28:57.638Z, remainingSeconds=1885}], publishing=[{id=‘index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo’, startTime=null, remainingSeconds=90}]}}

2016-07-22T16:57:41,462 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Worker[ip-10-0-0-94.eu-west-1.compute.internal:8091] wrote SUCCESS status for task [index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] on [TaskLocation{host=‘ip-10-0-0-94.eu-west-1.compute.internal’, port=8102}]

2016-07-22T16:57:41,462 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Worker[ip-10-0-0-94.eu-west-1.compute.internal:8091] completed task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] with status[SUCCESS]

2016-07-22T16:57:41,462 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskQueue - Received SUCCESS status for task: index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo

2016-07-22T16:57:41,462 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] on worker[ip-10-0-0-94.eu-west-1.compute.internal:8091]

2016-07-22T16:57:41,464 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from activeTasks

2016-07-22T16:57:41,464 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,464 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-05-27T00:00:00.000Z/2016-05-28T00:00:00.000Z, version=2016-07-22T15:50:57.652Z}

2016-07-22T16:57:41,472 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4534]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-05-27T00:00:00.000Z/2016-05-28T00:00:00.000Z, version=2016-07-22T15:50:57.652Z}

2016-07-22T16:57:41,476 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,476 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-05-28T00:00:00.000Z/2016-05-29T00:00:00.000Z, version=2016-07-22T15:51:38.243Z}

2016-07-22T16:57:41,482 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4535]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-05-28T00:00:00.000Z/2016-05-29T00:00:00.000Z, version=2016-07-22T15:51:38.243Z}

2016-07-22T16:57:41,484 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,484 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-24T00:00:00.000Z/2016-06-25T00:00:00.000Z, version=2016-07-22T14:58:37.497Z}

2016-07-22T16:57:41,490 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4502]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-24T00:00:00.000Z/2016-06-25T00:00:00.000Z, version=2016-07-22T14:58:37.497Z}

2016-07-22T16:57:41,491 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,491 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-25T00:00:00.000Z/2016-06-26T00:00:00.000Z, version=2016-07-22T14:59:33.496Z}

2016-07-22T16:57:41,496 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4503]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-25T00:00:00.000Z/2016-06-26T00:00:00.000Z, version=2016-07-22T14:59:33.496Z}

2016-07-22T16:57:41,497 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,497 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-26T00:00:00.000Z/2016-06-27T00:00:00.000Z, version=2016-07-22T15:01:24.792Z}

2016-07-22T16:57:41,502 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4506]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-26T00:00:00.000Z/2016-06-27T00:00:00.000Z, version=2016-07-22T15:01:24.792Z}

2016-07-22T16:57:41,503 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,503 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-27T00:00:00.000Z/2016-06-28T00:00:00.000Z, version=2016-07-22T15:03:04.834Z}

2016-07-22T16:57:41,508 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4507]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-27T00:00:00.000Z/2016-06-28T00:00:00.000Z, version=2016-07-22T15:03:04.834Z}

2016-07-22T16:57:41,509 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,509 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-28T00:00:00.000Z/2016-06-29T00:00:00.000Z, version=2016-07-22T15:04:45.922Z}

2016-07-22T16:57:41,516 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4508]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-28T00:00:00.000Z/2016-06-29T00:00:00.000Z, version=2016-07-22T15:04:45.922Z}

2016-07-22T16:57:41,521 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,521 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-29T00:00:00.000Z/2016-06-30T00:00:00.000Z, version=2016-07-22T15:06:26.500Z}

2016-07-22T16:57:41,525 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4509]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-29T00:00:00.000Z/2016-06-30T00:00:00.000Z, version=2016-07-22T15:06:26.500Z}

2016-07-22T16:57:41,526 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,526 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-30T00:00:00.000Z/2016-07-01T00:00:00.000Z, version=2016-07-22T15:08:06.913Z}

2016-07-22T16:57:41,532 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4510]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-06-30T00:00:00.000Z/2016-07-01T00:00:00.000Z, version=2016-07-22T15:08:06.913Z}

2016-07-22T16:57:41,533 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,533 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-01T00:00:00.000Z/2016-07-02T00:00:00.000Z, version=2016-07-22T15:09:48.474Z}

2016-07-22T16:57:41,539 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4511]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-01T00:00:00.000Z/2016-07-02T00:00:00.000Z, version=2016-07-22T15:09:48.474Z}

2016-07-22T16:57:41,540 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,540 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-02T00:00:00.000Z/2016-07-03T00:00:00.000Z, version=2016-07-22T15:11:30.749Z}

2016-07-22T16:57:41,546 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4512]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-02T00:00:00.000Z/2016-07-03T00:00:00.000Z, version=2016-07-22T15:11:30.749Z}

2016-07-22T16:57:41,547 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,547 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-03T00:00:00.000Z/2016-07-04T00:00:00.000Z, version=2016-07-22T15:13:13.744Z}

2016-07-22T16:57:41,553 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4513]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-03T00:00:00.000Z/2016-07-04T00:00:00.000Z, version=2016-07-22T15:13:13.744Z}

2016-07-22T16:57:41,554 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,554 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-04T00:00:00.000Z/2016-07-05T00:00:00.000Z, version=2016-07-22T15:14:55.604Z}

2016-07-22T16:57:41,559 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4514]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-04T00:00:00.000Z/2016-07-05T00:00:00.000Z, version=2016-07-22T15:14:55.604Z}

2016-07-22T16:57:41,560 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,560 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-05T00:00:00.000Z/2016-07-06T00:00:00.000Z, version=2016-07-22T15:16:47.560Z}

2016-07-22T16:57:41,565 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4515]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-05T00:00:00.000Z/2016-07-06T00:00:00.000Z, version=2016-07-22T15:16:47.560Z}

2016-07-22T16:57:41,565 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,565 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-06T00:00:00.000Z/2016-07-07T00:00:00.000Z, version=2016-07-22T15:18:36.286Z}

2016-07-22T16:57:41,570 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4517]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-06T00:00:00.000Z/2016-07-07T00:00:00.000Z, version=2016-07-22T15:18:36.286Z}

2016-07-22T16:57:41,571 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,571 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-07T00:00:00.000Z/2016-07-08T00:00:00.000Z, version=2016-07-22T15:20:24.317Z}

2016-07-22T16:57:41,575 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4518]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-07T00:00:00.000Z/2016-07-08T00:00:00.000Z, version=2016-07-22T15:20:24.317Z}

2016-07-22T16:57:41,582 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,582 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-08T00:00:00.000Z/2016-07-09T00:00:00.000Z, version=2016-07-22T15:22:16.754Z}

2016-07-22T16:57:41,586 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4519]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-08T00:00:00.000Z/2016-07-09T00:00:00.000Z, version=2016-07-22T15:22:16.754Z}

2016-07-22T16:57:41,589 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,589 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-09T00:00:00.000Z/2016-07-10T00:00:00.000Z, version=2016-07-22T15:24:14.524Z}

2016-07-22T16:57:41,593 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4520]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-09T00:00:00.000Z/2016-07-10T00:00:00.000Z, version=2016-07-22T15:24:14.524Z}

2016-07-22T16:57:41,594 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,594 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-10T00:00:00.000Z/2016-07-11T00:00:00.000Z, version=2016-07-22T15:26:06.744Z}

2016-07-22T16:57:41,597 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4521]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-10T00:00:00.000Z/2016-07-11T00:00:00.000Z, version=2016-07-22T15:26:06.744Z}

2016-07-22T16:57:41,598 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,598 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-11T00:00:00.000Z/2016-07-12T00:00:00.000Z, version=2016-07-22T15:27:54.214Z}

2016-07-22T16:57:41,602 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4522]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-11T00:00:00.000Z/2016-07-12T00:00:00.000Z, version=2016-07-22T15:27:54.214Z}

2016-07-22T16:57:41,603 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,603 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-12T00:00:00.000Z/2016-07-13T00:00:00.000Z, version=2016-07-22T15:29:43.641Z}

2016-07-22T16:57:41,606 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4523]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-12T00:00:00.000Z/2016-07-13T00:00:00.000Z, version=2016-07-22T15:29:43.641Z}

2016-07-22T16:57:41,607 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,607 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-13T00:00:00.000Z/2016-07-14T00:00:00.000Z, version=2016-07-22T15:31:21.695Z}

2016-07-22T16:57:41,609 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4524]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-13T00:00:00.000Z/2016-07-14T00:00:00.000Z, version=2016-07-22T15:31:21.695Z}

2016-07-22T16:57:41,610 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,610 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-14T00:00:00.000Z/2016-07-15T00:00:00.000Z, version=2016-07-22T15:32:58.002Z}

2016-07-22T16:57:41,613 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4525]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-14T00:00:00.000Z/2016-07-15T00:00:00.000Z, version=2016-07-22T15:32:58.002Z}

2016-07-22T16:57:41,614 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,614 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-15T00:00:00.000Z/2016-07-16T00:00:00.000Z, version=2016-07-22T15:34:30.666Z}

2016-07-22T16:57:41,616 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4526]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-15T00:00:00.000Z/2016-07-16T00:00:00.000Z, version=2016-07-22T15:34:30.666Z}

2016-07-22T16:57:41,617 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,617 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-16T00:00:00.000Z/2016-07-17T00:00:00.000Z, version=2016-07-22T15:36:04.955Z}

2016-07-22T16:57:41,619 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4527]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-16T00:00:00.000Z/2016-07-17T00:00:00.000Z, version=2016-07-22T15:36:04.955Z}

2016-07-22T16:57:41,620 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,620 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-17T00:00:00.000Z/2016-07-18T00:00:00.000Z, version=2016-07-22T15:37:41.327Z}

2016-07-22T16:57:41,622 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4528]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-17T00:00:00.000Z/2016-07-18T00:00:00.000Z, version=2016-07-22T15:37:41.327Z}

2016-07-22T16:57:41,623 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,623 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-18T00:00:00.000Z/2016-07-19T00:00:00.000Z, version=2016-07-22T15:39:39.472Z}

2016-07-22T16:57:41,625 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4529]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-18T00:00:00.000Z/2016-07-19T00:00:00.000Z, version=2016-07-22T15:39:39.472Z}

2016-07-22T16:57:41,626 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,626 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-19T00:00:00.000Z/2016-07-20T00:00:00.000Z, version=2016-07-22T15:41:38.781Z}

2016-07-22T16:57:41,627 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4530]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-19T00:00:00.000Z/2016-07-20T00:00:00.000Z, version=2016-07-22T15:41:38.781Z}

2016-07-22T16:57:41,628 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,628 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-20T00:00:00.000Z/2016-07-21T00:00:00.000Z, version=2016-07-22T15:43:34.942Z}

2016-07-22T16:57:41,630 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4531]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-20T00:00:00.000Z/2016-07-21T00:00:00.000Z, version=2016-07-22T15:43:34.942Z}

2016-07-22T16:57:41,630 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,630 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-21T00:00:00.000Z/2016-07-22T00:00:00.000Z, version=2016-07-22T15:45:30.092Z}

2016-07-22T16:57:41,631 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4532]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-21T00:00:00.000Z/2016-07-22T00:00:00.000Z, version=2016-07-22T15:45:30.092Z}

2016-07-22T16:57:41,632 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] from TaskLock[index_kafka_load-test-2]

2016-07-22T16:57:41,632 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-22T00:00:00.000Z/2016-07-23T00:00:00.000Z, version=2016-07-22T15:47:20.748Z}

2016-07-22T16:57:41,633 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with id[4533]: TaskLock{groupId=index_kafka_load-test-2, dataSource=load-test-2, interval=2016-07-22T00:00:00.000Z/2016-07-23T00:00:00.000Z, version=2016-07-22T15:47:20.748Z}

2016-07-22T16:57:41,635 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.MetadataTaskStorage - Updating task index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo to status: TaskStatus{id=index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo, status=SUCCESS, duration=7148668}

2016-07-22T16:57:41,636 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskQueue - Task done: KafkaIndexTask{id=index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo, type=index_kafka, dataSource=load-test-2}

2016-07-22T16:57:41,636 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskQueue - Task SUCCESS: KafkaIndexTask{id=index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo, type=index_kafka, dataSource=load-test-2} (7148668 run duration)

2016-07-22T16:57:41,636 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] status changed to [SUCCESS].

2016-07-22T16:57:41,636 INFO [Curator-PathChildrenCache-1] io.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] went bye bye.

2016-07-22T16:57:41,642 INFO [KafkaSupervisor-load-test-2-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Task [index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo] completed successfully, stopping tasks [index_kafka_load-test-2_2a9f4e0023eca2b_gijiclbo]

index-task.log.gz (1.6 MB)

Hey Nikita,

What happened was the indexing task timed out before it could finish publishing. It looks like you already set a completionTimeout of 1H, but you could try extending this even longer to give the task time to publish. A better solution would be to run a shorter taskDuration or split the workload across multiple tasks so that any given task is doing less work and thus has less segments to publish. A publishing time that takes longer than an hour isn’t really ideal.

Thank you, I will try that.

Is there any connection between taskDuration and segmentGranularity? If I have large segmentGranularity, say 1 day, is it Ok to finish tasks much earlier? How the final segment will be composed then?

Nikita

Hey Nikita,

Yes, it is okay to have a segmentGranularity larger than the taskDuration. What will happen is that subsequent tasks will create additional shards for the same time interval that will be loaded together with the previous shards.