Multiple segment cache locations

Hi Team,

In our historical nodes, we have multiple mounts can we have multiple segment cache locations and add the overall maxSize as shown below

druid.service=druid/historical

druid.port=8080

Processing threads and buffers

druid.processing.buffer.sizeBytes=1073741824

druid.processing.numThreads=31

HTTP server threads

druid.server.http.numThreads=50

druid.server.maxSize=7200000000000

Segment storage

druid.segmentCache.locations=[{“path”: “/mnt01/data/indexCache”, “maxSize”: 1800000000000},{“path”: “/mnt02/data/indexCache”, “maxSize”: 1800000000000},{“path”: “/mnt03/data/indexCache”, “maxSize”: 1800000000000},{“path”: “/mnt04/data/indexCache”, “maxSize”: 1800000000000}]

druid.monitoring.monitors=[“io.druid.server.metrics.HistoricalMetricsMonitor”, “com.metamx.metrics.JvmMonitor”]

Thanks,

Sathish

Hi Sathish, what is your question?

A related question for me is - how are the multiple segment cache locations used by druid - is the second location used after the first one fills up? Or is there a way to map segments of a datasource to a particular cache location? The documentation is a bit thin on this.

Hi Satish/Neelesh,

Yes, It is possible to specify multiple segment cache locations in Druid as specified in configs above.

When a new segment is to loaded in segment cache, The historical node tries to select the locations in the order of remaining capacity i.e the one having the most available space is selected for loading the segment.

No, currently it is not possible specify an affinity based on datasource for segment cache to be used.

Hi Nishant,

Thanks for the clarification, it is working as expected

-Sathish

Thanks for the details, Nishant! We were exploring if it was possible to have encrypted segment cache for specific datasources, hence the question.