Historical segment data not being swapped/refreshed from deep storage

Hi,
We have total memory of 2TB ( 8 historical nodes) and have 4TB of segments in deep storage. We have enabled the caching ( not a Memcache) on historical nodes and after 2TB of segments loaded to historical nodes it’s not loading any further segments even if we fire the query spanning all 4Tb segments. I am expecting if the queried data is not in the cache, the druid should load it from deep storage by swapping some of the segments from historical hash/cache.

I want to understand how historical nodes load/swaps the segments and cache them ?. It is not practical to have enough memory to match your deep storage segment size. I am thinking I may be missing a setting or configuration which does swap/refresh the segments/cache on demand in both historical/broker nodes. Should I not enable caching ?, or I’m I missing a trick here ?

Following are my historical and Broker settings.

Broker Runtime:

druid.service=druid/broker

druid.port=8082

HTTP server threads

druid.broker.http.numConnections=5

druid.server.http.numThreads=25

Processing threads and buffers

#druid.processing.buffer.sizeBytes=536870912

druid.processing.buffer.sizeBytes=2073741824

druid.processing.numThreads=15

Query cache

druid.broker.cache.useCache=true

druid.broker.cache.populateCache=true

druid.cache.type=local

druid.cache.sizeInBytes=2000000000

Hi Malik,

Even I’m facing similar issue. Please let me know what configuration need to be added, if you have solved this issue

Thanks in advance!!

  • Ramya

Druid cannot fetch the segments from deep storage on demand, that’s by design.
You can set “druid.segmentCache.locations” and “druid.server.maxSize” as maximum as possible to fit your segment data into historical nodes.