Druid historical query problem: Exception with one of the sequences! Caused by: ClassCastException

Hi!
In one of the historicals I’m getting a lot of these errors - so the node is practically unavailable:

2018-11-22_10:34:54.45405 2018-11-22T10:34:54,451 ERROR [processing-1] io.druid.query.ChainedExecutionQueryRunner - Exception with one of the sequences!

2018-11-22_10:34:54.45406 java.lang.IllegalArgumentException: Self-suppression not permitted

2018-11-22_10:34:54.45406 at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[?:1.8.0_131]

2018-11-22_10:34:54.45406 at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:60) ~[java-util-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45407 at io.druid.query.spec.SpecificSegmentQueryRunner$2.accumulate(SpecificSegmentQueryRunner.java:86) ~[druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45407 at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45407 at io.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:172) ~[druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45408 at io.druid.query.spec.SpecificSegmentQueryRunner.access$200(SpecificSegmentQueryRunner.java:45) ~[druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45408 at io.druid.query.spec.SpecificSegmentQueryRunner$3.wrap(SpecificSegmentQueryRunner.java:152) ~[druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45410 at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45410 at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45411 at io.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45411 at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45411 at io.druid.java.util.common.guava.Sequences.toList(Sequences.java:150) ~[java-util-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45412 at io.druid.query.ChainedExecutionQueryRunner$1$1$1.call(ChainedExecutionQueryRunner.java:130) [druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45412 at io.druid.query.ChainedExecutionQueryRunner$1$1$1.call(ChainedExecutionQueryRunner.java:120) [druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45413 at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]

2018-11-22_10:34:54.45413 at io.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:249) [druid-processing-0.12.2.jar:0.12.2]

2018-11-22_10:34:54.45415 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]

2018-11-22_10:34:54.45415 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]

2018-11-22_10:34:54.45415 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

2018-11-22_10:34:54.45416 Caused by: java.lang.ClassCastException

``

I have stopped the problematic historical node and since I have 2 replicas, the query started working - ending in the second historical - which has no problem.

Some older exceptions I have found in the log:

2018-11-22_10:21:46.74773 2018-11-22T10:21:46,744 ERROR [qtp1141783782-91] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container

2018-11-22_10:21:46.74774 java.lang.ClassCastException

2018-11-22_10:21:46.74775 2018-11-22T10:21:46,744 WARN [qtp1141783782-91] org.eclipse.jetty.servlet.ServletHandler - /druid/v2/

2018-11-22_10:21:46.74775 java.lang.ClassCastException

``

Do you have any idea why is this happening and how to fix it?

I’m running Druid 0.12.2

Hey Davor,

The exception has probably gotten thrown so many times that the JVM has started skipping the stack trace. You can ‘reset’ this behavior by restarting the historical and getting it to throw the exception again. You should see a real stack trace and be able to raise that as an issue on GitHub: https://github.com/apache/druid/issues