Druid - Historical - better few servers with lots of memory or lots of server with less memory?

Hi,

I finally set up the first version of my Druid cluster (0.14) and performances are amazing (and we are just in dev environment…)

Now, I’m going to dive in performance tuning to get the best of my cluster.

For now, I have 2 historicals nodes (r5a.2xlarge) with 64 Gb of memory each

I’m wondering what is the best configuration : 2 historicals with 64 Gb of memory or 4 historicals with 32 Gb of memory ?

If I have 4 historicals, will the requests be more parallelized and will I get better performances ?

Hint : lots of servers with lots of memory is not an accepted answer :smiley:

Hey Guillaume,

Glad to hear you are getting good performance so far!!

Assuming the total number of CPUs on your historicals is the same in both scenarios (i.e. the 64GB mem historicals also have 2x the CPUs) then generally it’s best to do larger servers. This is because it offloads more merging work from the broker.

By the way, we’ve been working on a cluster tuning guide, which you can get a sneak peek at here: https://github.com/apache/incubator-druid/blob/master/docs/content/operations/basic-cluster-tuning.md. It will be available at http://druid.io/docs/latest/operations/basic-cluster-tuning.html once the next release is out.

Gian

Awesome !

I’ll take a first read this afternoon and maybe come back here with more questions (if that can help improve this guide)

Thanks