[druid-user] Unable to run ingestion with specific java version

Hi,

I hit today an issue when AWS did some security updates on my Druid instances and set the java package to version 8u282.

This version prevented the coordinator to run with the following error

org.apache.curator.framework.listen.ListenerContainer - Listener (org.apache.druid.curator.discovery.CuratorDruidLeaderSelector$1@4cc12db2) threw an exception
java.lang.ClassFormatError: Illegal field name “org.apache.druid.server.coordinator.DruidCoordinator$this” in class org/apache/druid/server/coordinator/DruidCoordinator$CoordinatorHistoricalManagerRunnable
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_282]
at java.lang.ClassLoader.defineClass(ClassLoader.java:756) ~[?:1.8.0_282]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_282]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_282]
at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_282]
at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_282]
at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_282]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_282]
at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_282]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_282]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_282]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_282]
at org.apache.druid.server.coordinator.DruidCoordinator.becomeLeader(DruidCoordinator.java:570) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.apache.druid.server.coordinator.DruidCoordinator.access$000(DruidCoordinator.java:99) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.apache.druid.server.coordinator.DruidCoordinator$1.becomeLeader(DruidCoordinator.java:524) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.apache.druid.curator.discovery.CuratorDruidLeaderSelector$1.isLeader(CuratorDruidLeaderSelector.java:97) ~[druid-server-0.15.1-incubating.jar:0.15.1-incubating]
at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:665) ~[curator-recipes-4.1.0.jar:4.1.0]
at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:661) ~[curator-recipes-4.1.0.jar:4.1.0]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.1.0.jar:4.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]

To workaround this issue (as yum failed to downgrade java), I installed java (from AdoptOpenJDK · GitHub) to /opt/java
and added in my systemd unit files the following line
Environment=JAVA_HOME=/opt/java

This made the coordinator working again. Great.
However, I’m having another issue now : no ingestion success.

All tasks fails with
2021-03-24T20:28:35,472 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution
java.io.IOException: Cannot run program “java”: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_272]
at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:411) [druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating]
at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:224) [druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]

I can’t figure out how to make the java command available to this fork.

I tried to make it available to all users via /etc/profile, but that didn’t work.

Any idea ?

Hi Guillaume,

Could you try creating a symbolic link for the java executable at /usr/bin/java and try.

It seems like the processbuilder is looking for Java command in /usr/bin/java on unix/linix machine and doesn’t use the locations in environment variables like PATH and that could reason for the above failure.

Thanks and Regards,
Vaibhav

Druid should work with java 8u282. Was there something else done to the java configuration during the update?

By the way, I have a quickstart on my laptop. The laptop has java 12 as it’s main installation. Before starting up services, I just run

export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home"

and things run fine, including ingestion.

Maybe in your case, setting DRUID_JAVA_HOME would work, if your druid is recent enough (see this pull).

Sorry for the spam… DRUID_JAVA_HOME is mentioned here - https://druid.apache.org/docs/latest/tutorials/index.html#requirements