java.util.concurrent.RejectedExecutionException: Got Interrupted while while adding to the Queue

Hello,

we are using druid 0.12.3 with Kafka supervisors.

I see below error for tasks,

019-01-22T21:33:42,693 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[AbstractTask{id='index_kafka_test-requests_f0d3ac38f6e0667_iolafpnp', groupId='index_kafka_test-requests', taskResource=TaskResource{availabilityGroup='index_kafka_test-requests_f0d3ac38f6e0667', requiredCapacity=1}, dataSource='test-requests', context={checkpoints={"0":{"0":25144554956}}, IS_INCREMENTAL_HANDOFF_SUPPORTED=true}}]
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Got Interrupted while adding to the Queue
	at io.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:223) ~[druid-server-0.12.3.jar:0.12.3]
	at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:729) ~[?:?]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.3.jar:0.12.3]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.3.jar:0.12.3]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: java.util.concurrent.RejectedExecutionException: Got Interrupted while adding to the Queue
	at io.druid.java.util.common.concurrent.Execs$1.rejectedExecution(Execs.java:144) ~[java-util-0.12.3.jar:0.12.3]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_144]
	at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:484) ~[guava-16.0.1.jar:?]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_144]
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58) ~[guava-16.0.1.jar:?]
	at io.druid.segment.realtime.appenderator.AppenderatorImpl.persistAll(AppenderatorImpl.java:460) ~[druid-server-0.12.3.jar:0.12.3]
	at io.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:215) ~[druid-server-0.12.3.jar:0.12.3]
	... 7 more
Caused by: java.lang.InterruptedException
	at java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:879) ~[?:1.8.0_144]
	at io.druid.java.util.common.concurrent.Execs$1.rejectedExecution(Execs.java:141) ~[java-util-0.12.3.jar:0.12.3]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_144]
	at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:484) ~[guava-16.0.1.jar:?]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_144]
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58) ~[guava-16.0.1.jar:?]
	at io.druid.segment.realtime.appenderator.AppenderatorImpl.persistAll(AppenderatorImpl.java:460) ~[druid-server-0.12.3.jar:0.12.3]
	at io.druid.segment.realtime.appenderator.StreamAppenderatorDriver.persist(StreamAppenderatorDriver.java:215) ~[druid-server-0.12.3.jar:0.12.3]
	... 7 more
2019-01-22T21:33:42,699 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_test-requests_f0d3ac38f6e0667_iolafpnp] status changed to [FAILED].

How would i go to debug this issue as i havent found any errors in coordinator logs or overlord logs.

Thanks !

looks like configuration related problem. Does this tutorial ingestion work well on your cluster?

http://druid.io/docs/latest/tutorials/tutorial-kafka.html

Ming

We have multiple data-sources through Kafka ingestion and only one data-source is giving us this error. Supervisors are successful for some of the datasources. So my first assumption was its was not a config problem.