Observed qtp Threads Hanging on Broker with Druid 0.22.1

Since upgrading from Druid 0.19.0 to 0.22.1 a couple months ago we have been struggling with Broker performance issues. Before the upgrade we never needed to restart the Brokers, and they ran with low CPU and GC values. The Brokers were running with 8 CPU, 32GB RAM, using 24G Heap, and mostly default settings from the Basic Cluster Tuning guide (7 Proc Threads, 3 Merge Buffers, 500MB Buffer Size).

After the upgrade the Brokers starting running out of memory and crashing within 12 hours. We figured that the new version just needed more resources, and so we began the exercise of scaling up the Brokers. We are now running them with 16 CPU, 128GB RAM, using 80G Heap, 40G MaxDirect Mem, and are using 15 Proc Threads, 16 Merge Buffers, and 500MB Buffer Size (we tried going to 1GB Buffers with poor results). We have even gone to settings higher than this, but consistently have the Brokers exhaust their resources.

Here is our current Broker runtime.properties and JVM settings.

Broker Config

runtime.properties

druid.service=druid/broker
druid.plaintextPort=8082
druid.tlsPort=8282

# HTTP server settings
druid.server.http.numThreads=40
druid.server.http.gracefulShutdownTimeout=PT1M
druid.server.http.defaultQueryTimeout=60000

# HTTP client settings
druid.broker.http.numConnections=30
druid.broker.http.maxQueuedBytes=10000000

# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=16
druid.processing.numThreads=15

# Query protection settings
druid.query.scheduler.prioritization.strategy=threshold
druid.query.scheduler.prioritization.durationThreshold=PT24H
druid.query.scheduler.prioritization.adjustment=-1
druid.sql.planner.requireTimeCondition=false
druid.sql.planner.metadataSegmentCacheEnable=true
druid.sql.planner.metadataSegmentPollPeriod=60000

# Query cache disabled -- push down caching and merging to historicals and middlemanagers
druid.broker.cache.useCache=false
druid.broker.cache.populateCache=false

# Monitors that emit self-monitoring metrics
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.SysMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.client.cache.CacheMonitor","org.apache.druid.server.metrics.QueryCountStatsMonitor"]

JVM:

java -Xms12G -Xmx12G -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger -Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown -Dlog4j.shutdownHookEnabled=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/var/log/druid.gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=50 -XX:GCLogFileSize=10m -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:ParallelGCThreads=13 -XX:ConcGCThreads=8 -XX:+UseG1GC -XX:-UseBiasedLocking -Xms80G -Xmx80G -XX:MaxDirectMemorySize=40G -Dcom.sun.management.jmxremote -Dcom.sun.magement.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/opt/druid/conf/druid/_common/jmxremote.password -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Djava.rmi.server.hostname=10.10.203.27 -cp /opt/druid/conf/druid/_common:/opt/druid/conf/druid/broker:/opt/druid/lib/* org.apache.druid.cli.Main server broker

We have observed that the CPU and GC values both increase at the same. We keep seeing sudden increases until the Broker API becomes unresponsive, the Kubernetes Liveness Probe fails, and the pod is restarted. We also start experiencing “org.apache.druid.query.QueryCapacityExceededException” errors because the Brokers cannot acquire a merge buffer within 1-3 hours before they crash. The Brokers come back online with low CPU and GC, and work fine for 1-4 days before having the issue again.

This graph over the past few days shows the most recent example of this behavior.

This is happening on all three of our Druid clusters, with two being smaller clusters (1k queries per min with 3 Brokers each) and one being a med-large cluster (4k queries per min with 8 Brokers).

To get a better understanding of what is going on we got JMX configured on the Druid JVM, and used the VisualVM tool to analyze the Heap and Threads.

What we found is that observing the thread allocations allowed us to view which qtp threads SQL queries were being run on. As an SQL query is handled the thread name changes from ‘qtp1266602046-###’ to ‘sql[SQL-UUID]’. We can look at those query IDs in our logs and see the corresponding SQL statement. Once the SQL query is completed the thread name goes back to the qtp ID. The following GIF shows this typical behavior.
Screen Recording 2022-03-18 at 11.51.20 AM

As we observed these threads during a CPU/GC increase, and saw that one of the qtp threads had changed its name to an SQL ID, and then never changed back. The thread seemed to get stuck pulling in data from somewhere. The allocated space for the thread just constantly increases. As this happened we saw Heap increasing faster, leading to more GC, which required more CPU. This GIF shows this behavior, where the four qtp threads at the top are the ones which are stuck.
Screen Recording 2022-03-18 at 12.01.11 PM

We determined what one of the stuck qtp thread IDs was, and saw that it stopped producing any logs the moment it hung. The SQL ID in the thread name could not be found in the logs, and so we couldn’t see what the query was. The thread remains in this hung state producing no logs until the Broker is restarted.

We did a thread dump, and were able to find the hung threads, and what functions they were running at the time. We were hoping to find a common issue, but instead found that the four queries in this example were at three different functions. I will put the thread dump in a comment since it is too long to include here.

We are wondering if this is evidence of a Druid bug, a misconfiguration of our Brokers, or an example of very bad queries hanging up Druid? Since this wasn’t happening in Druid 0.19 we are currently labeling this as a bug.

Thank you for any help.

Here is thread dump 1 of 3.

Thread Dump for abaf08ef-ea40-484d-a070-ed620c16c37b
"sql[abaf08ef-ea40-484d-a070-ed620c16c37b]" - Thread t@160
   java.lang.Thread.State: RUNNABLE
        at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39)
        at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41)
        at org.apache.druid.java.util.common.guava.ConcatSequence$$Lambda$1800/1857032586.accumulate(Unknown Source)
        at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
        at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
        at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
        at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41)
        at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:245)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:232)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.druid.java.util.common.concurrent.DirectExecutorService.execute(DirectExecutorService.java:81)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
        at org.apache.druid.query.DirectQueryProcessingPool.submitRunnerTask(DirectQueryProcessingPool.java:42)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:230)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:222)
        at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
        at com.google.common.collect.Iterators.addAll(Iterators.java:357)
        at com.google.common.collect.Lists.newArrayList(Lists.java:147)
        at com.google.common.collect.Lists.newArrayList(Lists.java:129)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:218)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:164)
        at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:66)
        at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32)
        at org.apache.druid.sql.http.SqlResource.doPost(SqlResource.java:118)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:208)
        at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:212)
        at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel$$Lambda$391/230030817.dispatch(Unknown Source)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - locked <31f1fa2b> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

Here is thread dump 2 of 3.

Thread Dump for 6f44b64e-59c8-4cbd-8bfd-140b7df484ab
"sql[6f44b64e-59c8-4cbd-8bfd-140b7df484ab]" - Thread t@185
   java.lang.Thread.State: RUNNABLE
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:521)
        at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2.isAllSingleValueDims(GroupByQueryEngineV2.java:331)
        at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$1.make(GroupByQueryEngineV2.java:245)
        at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$1.make(GroupByQueryEngineV2.java:203)
        at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39)
        at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41)
        at org.apache.druid.java.util.common.guava.ConcatSequence$$Lambda$1800/1857032586.accumulate(Unknown Source)
        at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
        at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
        at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
        at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41)
        at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:245)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:232)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.druid.java.util.common.concurrent.DirectExecutorService.execute(DirectExecutorService.java:81)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
        at org.apache.druid.query.DirectQueryProcessingPool.submitRunnerTask(DirectQueryProcessingPool.java:42)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:230)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:222)
        at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
        at com.google.common.collect.Iterators.addAll(Iterators.java:357)
        at com.google.common.collect.Lists.newArrayList(Lists.java:147)
        at com.google.common.collect.Lists.newArrayList(Lists.java:129)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:218)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:164)
        at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:66)
        at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32)
        at org.apache.druid.sql.http.SqlResource.doPost(SqlResource.java:118)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:208)
        at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:212)
        at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel$$Lambda$391/230030817.dispatch(Unknown Source)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - locked <34b5f995> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

Here is thread dump 3 of 3.

Thread Dump for 6f44b64e-59c8-4cbd-8bfd-140b7df484ab
"sql[b4b15bca-feb2-4f0d-855e-1f4d6b8df8b1]" - Thread t@197
   java.lang.Thread.State: RUNNABLE
        at org.apache.druid.segment.VirtualColumns.exists(VirtualColumns.java:159)
        at org.apache.druid.segment.virtual.VirtualizedColumnInspector.getColumnCapabilities(VirtualizedColumnInspector.java:54)
        at org.apache.druid.segment.DimensionHandlerUtils.createColumnSelectorPluses(DimensionHandlerUtils.java:209)
        at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$1.make(GroupByQueryEngineV2.java:209)
        at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$1.make(GroupByQueryEngineV2.java:203)
        at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39)
        at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41)
        at org.apache.druid.java.util.common.guava.ConcatSequence$$Lambda$1800/1857032586.accumulate(Unknown Source)
        at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
        at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
        at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
        at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41)
        at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:245)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:232)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.druid.java.util.common.concurrent.DirectExecutorService.execute(DirectExecutorService.java:81)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
        at org.apache.druid.query.DirectQueryProcessingPool.submitRunnerTask(DirectQueryProcessingPool.java:42)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:230)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:222)
        at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
        at com.google.common.collect.Iterators.addAll(Iterators.java:357)
        at com.google.common.collect.Lists.newArrayList(Lists.java:147)
        at com.google.common.collect.Lists.newArrayList(Lists.java:129)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:218)
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:164)
        at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:66)
        at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84)
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
        at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32)
        at org.apache.druid.sql.http.SqlResource.doPost(SqlResource.java:118)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:208)
        at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:212)
        at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel$$Lambda$391/230030817.dispatch(Unknown Source)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - locked <1622afea> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

We were finally able to determine the type of SQL queries causing this problem, and can now reproduce the issue any time. We have documented the problem in this Github issue.

2 Likes

Thank you for taking the time to document all of this and create a GitHub issue.