Overlord issue: No TaskLocation available for task

Hello,

I’m running a Kafka Indexing Service task (spec attacked), but it has been pending since the beginning. The workers are all available and I have no other runnig task.

and in the overlord logs is showing this:

2016-12-16T17:09:59,922 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-12-16T17:09:59.922-03:00”,“service”:“druid/overlord”,“host”:“monsterprobe:8090”,“metric”:“jvm/bufferpool/count”,“value”:0,“bufferpoolName”:“mapped”}]

2016-12-16T17:10:18,050 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 1 tasks from storage (0 tasks added, 0 tasks removed).

2016-12-16T17:10:21,941 INFO [KafkaIndexTaskClient-sep-druid-0] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_sep-druid_b5d076c4daa7781_aadbnkhb], this task may not have been assigned to a worker yet or may have already completed

2016-12-16T17:10:21,946 INFO [KafkaSupervisor-sep-druid] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id=‘sep-druid’, generationTime=2016-12-16T17:10:21.946-03:00, payload={dataSource=‘sep-druid’, topic=‘sep’, partitions=1, replicas=1, durationSeconds=3600, active=[{id=‘index_kafka_sep-druid_b5d076c4daa7781_aadbnkhb’, startTime=null, remainingSeconds=null}], publishing=}}

I don’t know if this is the cause of the issue. But is the only strange log that I found.

Regards,

Joaquín Silva

kafka-spec.json (6.62 KB)

Hi,

I have the same issue.
Have you resolved it yet?

Regards,

Chanh

Hey guys,

If you’re running Druid 0.9.2 and Java 7, you’re likely running into this bug: https://github.com/druid-io/druid/issues/3795.

If this is your case, your options would be to:

  1. update to Java 8
  2. downgrade to Druid 0.9.1.1 until Druid 0.10 is released
  3. build from master which includes the fix here: https://github.com/druid-io/druid/pull/3796

I’d recommend option 1 if this is a possibility for you, as we are looking to phase out support for Java 7 in the not-too-distant future.

Thank you David,

I am using Java 8.

To clear my issue:

I am upgrading druid from 0.9.1.1 to 0.9.2 and I already roll out historical and brokers node -> these work fine.

Then I am rolling overlord but It seems like new overlord and old middle manager was conflict -> TaskLocation available for task [index_kafka_sep-druid_b5d076c4daa7781_aadbnkhb], this task may not have been assigned to a worker yet or may have already completed

After I shutdown all my old version middle manger and start new one -> working

Even I tried to start both version of middle manger at the same time -> TaskLocation available for task [index_kafka_sep-druid_b5d076c4daa7781_aadbnkhb], this task may not have been assigned to a worker yet or may have already completed

This process make my system don’t match the zero time.

Batch task works fine when I try to roll new overlord on old middle manager but Kafka index task don’t.

Regards,

Chanh

Hmm okay, can you try setting workerThreads in the supervisor tuning config to 10 or 20 and see if that helps at all?