We observe some slow answers by Druid when we have concurrents queries.
For instance, I could have a query A which lasts about 1s but when I do the query A 10x it lasts about 30s!
Which parameters impact the speed answer in this case?
We use Apache Druid 0.21.1
This post is a good place to start, but it is long and fairly comprehensive:
Data-driven applications and dashboards can place performance and scalability demands on the underlying datastore. In a world where each app or dashboard may fire off dozens of queries per second, a single poorly designed query can make or break the user experience.
This tutorial explains the interplay between data modeling and query performance in Apache Druid. We dig into how a query is submitted to Druid and how Druid responds, providing several techniques for optimizing query …
The docs also have something to say here. If you have already applied these, please respond with more details about your cluster set up:
On Broker RunTime Properties –
Set druid.broker.http.numConnections to Number of Available Cores, as a starting point
Set druid.processing.buffer.sizeBytes to 500MB to 1GB. If you set it to higher value, please make sure you have enough available memory on the instances.
On Historical and Task RunTime Properties –
Set druid.server.http.numThreads to Number of Query Nodes x druid.broker.http.numConnections
Set druid.processing.buffer.sizeBytes= 500MB to 1GB. Try and keep this in sync with the values on Broker Service.
If you start seeing TOO MANY OPEN FILES. Please increase ulimit size, at the time of Service start.