[druid-user] Batch Ingestion Tasks getting failed because of InvocationTargetException

Hi,

We are using apache-druid-0.22.1 version. Many of the Ingestion tasks are getting failed. I have verified Middle Manager and Overlord and Default druid logs and found that below is the root cause for this. It’s just showing (null).

Please check the below exception and guide me what should be changed and what is the root cause for this?

Exception :

java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:143) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:115) [druid-services-0.22.1.jar:0.22.1]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:304) [druid-services-0.22.1.jar:0.22.1]
at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.22.1.jar:0.22.1]
Caused by: org.apache.druid.java.util.common.ISE: Failed to run task[----TaskID-----] isReady
at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:177) ~[druid-indexing-service-0.22.1.jar:0.22.1]
… 10 more
Caused by: org.apache.druid.java.util.common.ISE: Task[----TaskID-----] is not ready to run yet!
at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:173) ~[druid-indexing-service-0.22.1.jar:0.22.1]
… 10 more

Thanks & Regards
Punnaiah Gupta Kota.

Hi Punnaiah Gupta Kota,

I’ve been researching and the closest thing I can find is this:

This PR makes shutdown hooks work again by automatically setting this to false if it is not explicitly set in system properties (similar to the shutdown hook properties themselves). I don’t think this should have any impact on Druid since we are not technically a web app, and this setting only appears to be used by a few jmx related settings.

Is it possible that a change in your system properties might resolve the issue?

Best,

Mark

Please share me what system property i should and where should add to resolve the issue.

Thanks & Regards
Punnaiah Guptha Kota

Hi Mark

The above is different completely which is not relevant to the above posted exception.
This is purely related to Ingestion task id. I have verified the link which you shared which is related to log4j.

Could you please help me with any configuration exactly to resolve this issue.

Thanks & Regards
Punnaiah Gupta Kota

Hi Punnaiah,
Based on line “at org.apache.druid.cli.CliPeon.run(CliPeon.java:304) [druid-services-0.22.1.jar:0.22.1]”, I think this is a Peon startup issue.
How many workers are you running in the middle manager? What are your jvmOptsArray settings?
One thought is that it is unable to start because of resource starvation.
How much memory and CPU do you have on the Middle Manager node?

Sergio