Index tasks are failed with ERROR io.druid.curator.discovery.ServerDiscoverySelector - No server instance found for [druid/overlord]

Hi,

Index task are failed with below reason.

All overlord services are running but it is failing with No server instance found for [druid/overlord]

Please suggest how to address this issue.

Below is the full stack trace.

2019-Aug-23 11:31:26 AM [main] INFO io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_test_summary_121_2019-08-23T11:28:59.112Z] to overlord: [LockListAction{}].
2019-Aug-23 11:31:26 AM [main] ERROR io.druid.curator.discovery.ServerDiscoverySelector - No server instance found for [druid/overlord]
2019-Aug-23 11:31:26 AM [main] WARN io.druid.indexing.common.actions.RemoteTaskActionClient - Exception submitting action for task[index_test_summary_121_2019-08-23T11:28:59.112Z]
io.druid.java.util.common.IOE: No known server
at io.druid.discovery.DruidLeaderClient.getCurrentKnownLeader(DruidLeaderClient.java:276) ~[druid-server-0.12.3.jar:0.12.3]
at io.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:128) ~[druid-server-0.12.3.jar:0.12.3]
at io.druid.indexing.common.actions.RemoteTaskActionClient.submit(RemoteTaskActionClient.java:81) [druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.common.task.AbstractTask.getTaskLocks(AbstractTask.java:232) [druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.common.task.IndexTask.isReady(IndexTask.java:200) [druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.common.task.IndexTask.isReady(IndexTask.java:192) [druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:170) [druid-indexing-service-0.12.3.jar:0.12.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:413) [java-util-0.12.3.jar:0.12.3]
at io.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:311) [java-util-0.12.3.jar:0.12.3]
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:134) [druid-api-0.12.3.jar:0.12.3]
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:101) [druid-services-0.12.3.jar:0.12.3]
at io.druid.cli.CliPeon.run(CliPeon.java:301) [druid-services-0.12.3.jar:0.12.3]
at io.druid.cli.Main.main(Main.java:116) [druid-services-0.12.3.jar:0.12.3]
2019-Aug-23 11:31:26 AM [main] ERROR io.druid.cli.CliPeon - Error when starting up. Failing.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:413) ~[java-util-0.12.3.jar:0.12.3]
at io.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:311) ~[java-util-0.12.3.jar:0.12.3]
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:134) ~[druid-api-0.12.3.jar:0.12.3]
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:101) [druid-services-0.12.3.jar:0.12.3]
at io.druid.cli.CliPeon.run(CliPeon.java:301) [druid-services-0.12.3.jar:0.12.3]
at io.druid.cli.Main.main(Main.java:116) [druid-services-0.12.3.jar:0.12.3]
Caused by: io.druid.java.util.common.ISE: Failed to run task[index_test_summary_121_2019-08-23T11:28:59.112Z] isReady
at io.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:175) ~[druid-indexing-service-0.12.3.jar:0.12.3]
… 10 more
Caused by: io.druid.java.util.common.IOE: No known server
at io.druid.discovery.DruidLeaderClient.getCurrentKnownLeader(DruidLeaderClient.java:276) ~[druid-server-0.12.3.jar:0.12.3]
at io.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:128) ~[druid-server-0.12.3.jar:0.12.3]
at io.druid.indexing.common.actions.RemoteTaskActionClient.submit(RemoteTaskActionClient.java:81) ~[druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.common.task.AbstractTask.getTaskLocks(AbstractTask.java:232) ~[druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.common.task.IndexTask.isReady(IndexTask.java:200) ~[druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.common.task.IndexTask.isReady(IndexTask.java:192) ~[druid-indexing-service-0.12.3.jar:0.12.3]
at io.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:170) ~[druid-indexing-service-0.12.3.jar:0.12.3]

Thanks in advance.

I submitted three index tasks with same request in that two index tasks are failed ,one index task success,

Iam very much confused with this behaviour like same request some are failing some are success,

The error indicated the master server “overlord” was not stable. Can you check if the zookeeper the cluster used was in good condition?

Thanks