Historical Node Memcached Properties Recommendations for druid.cache.numConnections

I am really new in supporting our Druid Cluster.

I have the following properties in our historical nodes.

druid.cache.type=memcached

druid.historical.cache.useCache=true

druid.historical.cache.populateCache=true

druid.historical.cache.unCacheable=

druid.cache.numConnections=12

At this point, we have about 110 historical nodes and 7 broker nodes connecting to 3 nodes of Memcached.

Initially, I ran into a problem earlier while trying to add more historical nodes back when we still had around 85 historical nodes. This is the error in memcached.

2019-02-07T00:37:46.580375027Z accept4(): Too many open files

2019-02-07T00:37:46.580980546Z Too many open connections

The historical nodes are showing errors below:

2019-01-28T19:26:12.460319291Z 2019-01-28 19:26:12.459 INFO net.spy.memcached.MemcachedConnection: Reconnecting due to exception on {QA sa=/0.0.0.0:11211, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 52430408 Key: druid:fde2d720a3e69a8ec0353cec2bf9bc6aae7d4deb:b8609b3082eb0688a4e98483976d79ceadb2db3e, topWop=null, toWrite=0, interested=1}

And it seems like the historical nodes are not able to get connections to the memcached nodes, because the max allowed connections in memcached was set to 1024 by default. And I was able to resolved this by updating the max allowed connections in memcached.

using this -c parameter. I simply doubled the number

memcached -m 50000m -c 2048

I am assuming that the limit in the allowed connections in memcached is just based on the resource limits that was initially when the node is configured.

cat /etc/security/limits.conf

$ cat /proc/sys/fs/file-max

8937707

$ ulimit -Hn

100000

$ ulimit -Sn

65535

My questions is really about how should I configure the value for druid.cache.numConnections? What should be the basis of setting this value? I can set this value high enough but I will just running into the problem of the limits in my memcached. In the Druid documentation, the default value is druid.cache.numConnections=1. When do we considering increasing this value?

Any recommendations or insight to this parameter is greatly appreciated.