Cold-storage - how is actually used/accessed/recovered

I all, being new…still learning a lot in regards to druid.

My question(s) is…

How exactly is cold-storage used?

First of my current use-case/scenario: (running druid 14/imply 2.9.3)

  1. Have built out: 1 Master Node, 1 Query Node, 10 Data Nodes (Using Cold-Storage S3)

  2. Have ingested 2 Billion rows, approx 900GB after injest. (via Kafka-Indexing-service)

I have been led to believe:

  1. When data is committed to the Historical Processes, it will also ‘commit’ to cold Storage

  2. If I lose a datanode…(with zero replication…)…then the data is still in cold-storage…How is that data accessed or recovered?

  • In fact I accidentally rebuilt 2 of the my data nodes, losing the data on the local disk (ebs volumes), now my ‘data’ count is off…which makes me think the data from cold-storage was not recovered…so…How do I get back?
  1. Cold-Storage is never queried
  • If true, then it really is offline backup, so there must be a way to recover it?

  • if not true, then when would it be accessed?

Can we in fact have say 2TB storage on the Historical-nodes themselves, and have ‘infinite’ cold-storage, then have the system poll data from cold-storage when queries intersect with the data in cold-storage but not the actual historicals? (essentially using Historical has a caching layer for recent data)

I’m guessing this where hot-historicals / cold-historicals come into play?

also of note/interest:

When I loaded my 2 Billion records, I see we are consuming about 900gb on our data nodes. In S3, it shows we are consuming about <200GB.

The data size difference I assume is due to??

  1. Index ?

  2. compression?

  3. segment compaction?

oh yeah…loving having avatica ‘native’ sql interface :slight_smile:

  1. When data is committed to the Historical Processes, it will also ‘commit’ to cold Storage

The commit to deep storage happens before a segment is downloaded by historical processes, this document and section should be helpful: http://druid.io/docs/latest/ingestion/index.html#indexing-and-handoff

  1. If I lose a datanode…(with zero replication…)…then the data is still in cold-storage…How is that data accessed or recovered?

The coordinator will periodically check to see what segments should be pushed to historicals, and will eventually try to reallocate the data to other historicals.

  1. Cold-Storage is never queried

Correct, the deep storage is never in the query path.

Can we in fact have say 2TB storage on the Historical-nodes themselves, and have ‘infinite’ cold-storage, then have the system poll data from cold-storage when queries intersect with the data in cold-storage but not the actual historicals? (essentially using Historical has a caching layer for recent data)

Data will never be pulled “on-demand” from deep storage, the pool of historicals collectively need to have enough storage (including replication) to hold all of your data.

I’m guessing this where hot-historicals / cold-historicals come into play?

The tiering is more for sizing of hardware/configuration for different query patterns.

When I loaded my 2 Billion records, I see we are consuming about 900gb on our data nodes. In S3, it shows we are consuming about <200GB.

Is this with replication?

When I loaded my 2 Billion records, I
see we are consuming about 900gb on our data nodes. In S3, it shows we
are consuming about <200GB.

Is this with replication?

Indeed we do have replication at 2, which doesn’t account for that much of a difference, I’m guessing…when the data is pulled into the historicals it builds indexes…and that accounts for larger space requirements on the actual druid nodes.

Hi Daniel,

Druid segments are compressed when storing in deep storage, but decompressed once historicals load them into their local storage. The difference in size comes from this.

The index is created together when a segment is generated and they are stored in deep storage together.

Jihoon