thanks again Gian, much appreciated.
I forgot to attach the query in the previous post.
I used Druid 0.9.0 for the tests. I did some early tests with Druid 0.9.1rc1 and noticed a roughly 20% performance boost, but I was just playing around, not doing serious testing. I guess the boost is due to the better distribution of segments across historicals.
We need to have 30 days of data in a hot tier. Currently, for one datasource that has 5 mil records per segment/shard, we get very good query latencies for short-term query ranges but if someone queries 30 days and doesn’t filter on anything, then the query would be rather slow 45 seconds or so. We now tested with 10 mil records per segment/shard which only yields 4-5 segments per hour. So far we only have a couple of hours of data with this setup and see that the short-term query times are twice as long as before. My hope is that this ratio will eventually flip and mean lower query latencies for the segments with 10mil records, but we cannot ingest data for a whole month just to find out whether this hypothesis will pan out to be true. So I find it kind of difficult to arrive at a good setup. I’ve spent months trying to tune Druid, but so far, every variation I tried only made Druid slower, not faster. Still waiting for the knot to untie itself and give way to a speed improvement. How likely is it that our very first setup happened to be the optimal one?
topnquery-six-metrics.txt (4.36 KB)