Druid UI (unified console) sometimes is painfully slow

I have very recently migrated to Druid 0.18.1. I’m not sure if it is anything to do with the update or not but I have been very frequently observing painfully slow response time in the unified console. Most of the times the UI comes up, and each pane describing the meta info of segments, data sources, etc are just stuck at “Loading…” Same thing is being observed in the segments page, where the Druid logo keeps on loading. I tried hard refreshing multiple times and it loads just fine, only to go back to this slow state in sometime. I can see that it is making a call to /druid/v2/sql with a bunch of SQL queries. Is this call trying to read from Druid’s internal store or the external metadata (MySQL). I have provided some of them here, and also the screenshot for reference. Any help would be much appreciated.

Sample queries in /druid/v2/sql:

  • SELECT server_type AS “service_type”, COUNT(*) as “count” FROM sys.servers GROUP BY 1
  • SELECT COUNT() FILTER (WHERE “suspended” = 0) AS “runningSupervisorCount”, COUNT() FILTER (WHERE “suspended” = 1) AS “suspendedSupervisorCount” FROM sys.supervisors

Browser used: Google Chrome 83.0.4103.97.

Screenshot 2020-06-09 at 8.47.43 PM.png

Screenshot 2020-06-09 at 8.47.53 PM.png

What type of metadata service are you using ? It looks like performance problems from metadata storage layer. We are using postgres instead of the default local derby database and there is no performance issues for us, even for our dev cluster, which is relatively small.

Screenshot 2020-06-09 at 8.47.43 PM.png

Screenshot 2020-06-09 at 8.47.53 PM.png

We have a production grade MySQL server as the metadata store but I doubt whether it is trying to query these info from the metadata store or Druid’s internal store. And I’m guessing this purely because of the namespaces the query uses such as sys.servers and sys.supervisors.

Hi,

The sys schema tables like sys.segments, sys.servers etc. are not physical tables in your metadata store. These are in-memory representations of Druid internals which are stored in broker and broker syncs segment metadata from the coordinator.

Are you observing the slowness in UI after upgrading to Druid 0.18.1 or perhaps your segment count has increased a lot recently, but that should only affect sys.segments view, and not the sys.servers and others to be slow. From the screenshot it seems all the panes are taking time to load. You can also try these queries directly from the broker to rule out any UI issues.

Thanks,

Surekha

Hi Surekha,

Thanks for the response. It is kind of both. As in, we upgraded to 0.18.1 a couple of weeks back and our segments have also started growing recently since we actually started using Druid heavily only after the update. BTW, like you said, I tried hitting the Broker directly with these queries (<BROKER_HOST>:<BROKER_PORT>/druid/v2/sql). We have three brokers. The response from two brokers are quick but the third one times out every single time. And since we also have router processes sitting along in these broker nodes and use that as the UI, it kinda makes sense that maybe the router is actually hitting this unresponsive broker for the information and times out, causing the painful UX experience on the console.

Having said that, I’m not seeing anything fishy in my log file generated by this particular broker. How should I proceed ahead with debugging this further?

Thanks,

Sivaprasanna

Siva, do you have druid.sql.planner.metadataSegmentCacheEnable set to true on your broker?

Hi Sivaprasanna,

If 2 brokers are fine and only one is slow, I’d definitely check that instance, if there is low disk space, any OOM errors. Are your broker instances behind a load balancer, are the health checks fine.

The broker setting which Samarth pointed out could help with sys.segments query performance.

HTH

Surekha

Thanks Surekha & Samarth. I think druid.sql.planner.metadataSegmentCacheEnable can be of help but I believe I kinda found out the problem. We are running a benchmark with various kind of queries (affecting realtime and historical in a varying degree) to understand the response time & capture latency info, and I observed that this particular broker goes to a stale state exactly after the benchmark runs.

Our cluster setup is this:

  • 5 Historicals
  • 5 MiddleManagers
  • 3 Brokers & 3 Routers (co-located)

According to connection pool sizing recommendation[1], the druid.server.http.threads on Historicals & MiddleManagers should be set to a higher value than the sum of druid.broker.http.numConnections across brokers. Unfortunately, that wasn’t the case before since so far, on our Historicals & MiddleManagers, druid.server.http.threads was configured as 50 and druid.broker.http.numConnections on the broker was set to 20. Now I changed them to the following as per the recommendation.

  • Historicals & MiddleManagers’ druid.server.http.threads is 70 (sum of druid.broker.http.numConnections + 10)
  • Broker’s druid.broker.http.numConnections is 20.
  • Broker’s druid.server.http.numThreads is 30
    But even after increasing the druid.server.http.threads on Historicals & MiddleManagers, I am observing that right after I run the benchmark, this broker immediately goes to the stale state. What makes my hunch more solid is when this particular broker goes unresponsive, and when I stop the broker process, I see this Jetty server related message (ERROR [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Jetty lifecycle event failed [class org.eclipse.jetty.server.Server]) with a bunch of Jetty QueuedThreadPool WARN message and the broker process isn’t gracefully stopped. Please find the full error message below.

Diggin deeper, I think what I’m seeing is probably what was being discussed here: https://github.com/apache/druid/issues/325 Can you please help us in resolving this issue?

2020-06-11T08:26:27,889 INFO [Thread-43] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/BROKER/prod-euclid-druid-queryserver1-1735672:8082]

2020-06-11T08:26:27,899 INFO [NodeRoleWatcher[BROKER]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Node[http://prod-euclid-druid-queryserver1-1735672:8082] of role[broker] went offline.

2020-06-11T08:26:27,900 INFO [Thread-43] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{“druidNode”:{“service”:“druid/broker”,“host”:“prod-euclid-druid-queryserver1-1735672”,“bindOnHost”:false,“plaintextPort”:8082,“port”:-1,“tlsPort”:-1,“enablePlaintextPort”:true,“enableTlsPort”:false},“nodeType”:“broker”,“services”:{“lookupNodeService”:{“type”:“lookupNodeService”,“lookupTier”:"__default"}}}].

2020-06-11T08:26:27,901 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]

2020-06-11T08:26:57,908 INFO [Thread-43] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@3909a854{HTTP/1.1,[http/1.1]}{0.0.0.0:8082}

2020-06-11T08:26:57,909 INFO [Thread-43] org.eclipse.jetty.server.session - node0 Stopped scavenging

2020-06-11T08:26:57,913 INFO [Thread-43] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@186dcb05{/,null,UNAVAILABLE}

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-162[timeseries_[euclid_cdo_live_hourly]_18f82248-c0a4-4a1b-b163-7a9626c9d91b],5,main]

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-165[topN_[euclid_cdo_live_hourly]_549ddbd3-f42e-4f69-b75f-ee053d5bb0d5],5,main]

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-177[topN_[euclid_cdo_live_hourly]_c99ddc98-0077-4d09-a53d-efd9f21eb5d9],5,main]

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-159[topN_[euclid_cdo_live_hourly]_2cbbaac8-171b-4039-b8ec-78439c0ca563],5,main]

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-175[topN_[euclid_cdo_live_hourly]_748433e0-294f-4df3-837b-3836b20d5ef5],5,main]

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-163[topN_[euclid_cdo_live_hourly]_e41aeaac-5830-4e3e-9f78-1c5fbfca4656],5,main]

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-182[topN_[euclid_cdo_live_hourly]_0381d5c9-d858-4c9a-aa57-6353e0334453],5,main]

2020-06-11T08:27:27,917 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-150[topN_[euclid_cdo_live_hourly]_51dd4cda-9fcf-4533-af6f-71d11568e4ae],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-169[topN_[euclid_cdo_live_hourly]_9a40b9f3-e77c-4393-af88-a949df285d61],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-181[topN_[euclid_cdo_live_hourly]_f3a2af6d-78cd-4ff5-8df6-909edb8040bb],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-184[topN_[euclid_cdo_live_hourly]_4ef89502-6e54-436a-b54e-47776a11ee31],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-155[topN_[euclid_cdo_live_hourly]_a4e2f32b-62a7-40b1-9160-54b36205908d],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-173[topN_[euclid_cdo_live_hourly]_c38ae97c-fb65-4b12-91c2-3fb8511d3939],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-180[topN_[euclid_cdo_live_hourly]_2049db4c-6796-4a69-acbc-f38a4fa1a666],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-149[topN_[euclid_cdo_live_hourly]_85f8f14e-7eda-4176-9408-b723728f2a8e],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-179[topN_[euclid_cdo_live_hourly]_6d83af84-35eb-4ecd-b40a-4e615061e0a0],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-174[topN_[euclid_cdo_live_hourly]_1e84dba4-80d5-4bbf-bc52-7373a3adc2fe],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-186[topN_[euclid_cdo_live_hourly]_2c476c10-7e41-45f7-8437-02bd6907d96d],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-161[topN_[euclid_cdo_live_hourly]_312791d2-704d-4b76-8104-8a87c2baa42a],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-166[topN_[euclid_cdo_live_hourly]_a2a2ddc7-cd92-4f99-85af-77cb4b7d050c],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-168[topN_[euclid_cdo_live_hourly]_b352dcb8-6dbc-4b1c-83bf-54f37427a122],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-171[topN_[euclid_cdo_live_hourly]_65d58a84-b67d-444a-b261-5947dec1def9],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-178[topN_[euclid_cdo_live_hourly]_37b5c479-cb6d-4e9e-9917-4b99d510653a],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-157[topN_[euclid_cdo_live_hourly]_14a9ad98-f3a5-407e-b183-f953f9f93df1],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-154[topN_[euclid_cdo_live_hourly]_aa2697cd-5e1c-419b-891b-1c1563dc07b2],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-167[topN_[euclid_cdo_live_hourly]_848e3e21-5da2-4449-a96f-9a79e8581cfa],5,main]

2020-06-11T08:27:27,918 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-183[topN_[euclid_cdo_live_hourly]_2b2727fa-bc35-455e-b947-bb90463965bb],5,main]

2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-160[topN_[euclid_cdo_live_hourly]_b3f4badf-f5da-4103-820e-37d97afba7d5],5,main]

2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-170[topN_[euclid_cdo_live_hourly]_714624b8-fd38-46a3-8044-17cb91f54acd],5,main]

2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-164[topN_[euclid_cdo_live_hourly]_1fd07432-53bd-433b-9cda-714660401bcc],5,main]

2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-156[topN_[euclid_cdo_live_hourly]_cf25e506-acd1-4979-bfe4-07c0b67b5547],5,main]

2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-158[topN_[euclid_cdo_live_hourly]_e3c5d7c0-0dc1-4098-a8cb-2d0ee3c04a4f],5,main]

2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-176[timeseries_[euclid_cdo_live_hourly]_7755967f-6778-4064-8a23-0b3bdb8742ac],5,main]

2020-06-11T08:27:27,919 WARN [Thread-43] org.eclipse.jetty.util.thread.QueuedThreadPool - QueuedThreadPool[qtp510793091]@1e721583{STOPPING,38<=38<=38,i=0,q=37}[org.eclipse.jetty.util.thread.TryExecutor$$Lambda$38/2089816579@5d457b16] Couldn’t stop Thread[qtp510793091-172[topN_[euclid_cdo_live_hourly]_c20848de-6dde-43ae-86e2-4cfe54f8f126],5,main]

2020-06-11T08:27:27,920 ERROR [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Jetty lifecycle event failed [class org.eclipse.jetty.server.Server]

java.util.concurrent.TimeoutException: null

at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) [jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1]

at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1]

at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]

2020-06-11T08:27:27,920 WARN [Thread-43] org.apache.druid.server.initialization.jetty.JettyServerModule - Unable to stop Jetty server.

java.util.concurrent.TimeoutException: null

at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:129) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:181) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.server.Server.doStop(Server.java:439) ~[jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]

at org.apache.druid.server.initialization.jetty.JettyServerModule$2.stop(JettyServerModule.java:450) [druid-server-0.18.1.jar:0.18.1]

at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1]

at org.apache.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:401) [druid-core-0.18.1.jar:0.18.1]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]

2020-06-11T08:27:27,920 INFO [Thread-43] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]

2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView is stopping.

2020-06-11T08:27:27,924 INFO [Thread-43] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView Stopped.

2020-06-11T08:27:27,925 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.

2020-06-11T08:27:27,937 INFO [Thread-43] org.apache.druid.guice.LifecycleForkJoinPoolProvider - Shutting down ForkJoinPool [org.apache.druid.guice.LifecycleForkJoinPoolProvider@35bd2eec]

``

[1] https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#connection-pool-guidelines

Thanks. : )

Hi Siva,
Thanks for digging into this issue. Could you comment on the github issue that you are running into this on version 0.18.1 along with the broker stack trace, that might bump this issue and get this prioritized?

-Surekha

Hi Surekha,

I have commented on the GitHub issue along with the stack trace[1]. Do you know of any way to mitigate this or if there is a work around (other than having to restart the broker process every now and then), that would be really helpful.

[1] https://github.com/apache/druid/issues/325#issuecomment-643065616