I am still at the early stages to druid
I noticed that when I’m ingesting new data using batch ingestion, two new folders are created inside deep storage
I am wondering what is segment-cache?
Is there any different between a segment-cache and a normal segment folder?
Any help would be appreciated.
Segment-cache is the segments that are loaded and readily available on historical node local storage. This data which is memory-mapped is served quickly to broker nodes to satisfy data queries.
The actual segments data is essentially the same…but includes all of the data ever sent to deep storage. Coordinator nodes determine when and if (and to which historical node) data might need to be extracted back from deep storage to serve queries.
Thank you for the reply,
I see that the contents of both folders are the same
So would it be possible that the data queries would be affected if the segment-cache folder is deleted/missing?
The brokers that respond to queries have a broker cache as well…but in general Druid would be in deep trouble if the segment cache folder is deleted/missing.