Druid HIstorical Node Disk 90% full, what next?

Hello everybody,

I searched for a similar topic but didnt find one, so here is my question.

We are running a Druid Cluster (obviously) version 0.13-incubating. Cluster consists of 1x Broker, 1x Master and 3x Data nodes. Each Data node has 2x 1tb disks for data. Every data node is now at 90% on both disks.

If I am not mistaken, those are cached segments based on per data source retention policies (Corrections are welcome). My question is, Should we add another 1tb disk on every data node or should we ad another data node?

Is there anything else we can do configuration wise?

Thank you very much

George

Hi George,
Somethings like available RAM to segment cache ratios play a role here.

In general scaling out ( adding a new node ) plays out well than scaling up ( adding a new disk to existing node ) as you

are also adding additional RAM, cores etc in addition to storage when you add a new node and segments start to get balanced between all the nodes(including new node).

Coming to cost, it might be cheaper to add disk rather than entirely new node but if cost is not a severe constraint, I think you will be better off with adding a new node(both from ingestions and query perspectives)

Thanks,

–siva

Hello Siva,

First of all thanks for responding, your input is very valuable. 99% I will add a new node. I just have another question regarding a specific optional config value:

Bellow you can read that it references an optional “freeSpacePercent” . Do you know what that does?

druid.segmentCache.locations

Segments assigned to a Historical process are first stored on the local file system (in a disk cache) and then served by the Historical process.

These locations define where that local cache resides. This value cannot be NULL or EMPTY.

Here is an example druid.segmentCache.locations=[{"path": "/mnt/druidSegments", "maxSize": 10000, "freeSpacePercent": 1.0}].

“freeSpacePercent” is optional, if provided then enforces that much of free disk partition space while storing segments.

But, it depends on File.getTotalSpace() and File.getFreeSpace() methods, so enable if only if they work for your File System.

Thanks again

Best regards,

George

Hi George,
You can leave that empty as it is optional.

Basically whenever you create a file, your OS follows one of the methods of allocation as described in

https://www.geeksforgeeks.org/file-allocation-methods/ for efficient storage and / or faster access.

in that context, totalSpace and freeSpace comes into picture.

But in your case, you will leave it and OS will take care of it.

Thanks,

–siva