Memcached recommendations

Hi all,

Wondering if anyone could share tips on memcached cluster building for use with druid. I’m using the production configs from druid.io for now and running on a single AWS box, wondering about sizing/replication of servers and stuff like that.

Thanks,

Michael

Hi Michael,

First of all, memcached is not required in order to run Druid in production. Even for non-trivial deployments, enabling local caching on the historical nodes or the brokers should be sufficient. Memcached is useful if you need the cache to persist during rolling upgrades or if you somehow need the cache to be shared across nodes, but it adds a point of failure and requires tuning based on your workload.

I would experiment with various sizes and check cache hit-rate metrics emitted by Druid to determine what’s best for your work-load. Also look at how fast your cache is filling up to get a sense of eviction rates. In practice you’ll have to adjust sizes and determine what memory / hit-rate trade-offs you are willing to make.

Either lots of historical nodes with a smaller local cache each, or a single memcached instance can handle sizeables clusters.

Druid doesn’t replicate the cache, so I would only worry about multiple memcached instances if you need more ram.