When it comes to recovery in druid, There are two important components we need to backup regularly -
1. Metadata store and
2. Deep storage
As long as they are available we should be able to bring up a brand new Druid cluster with all the data intact.
There isn’t a strong need to back up deep storage at a particular point in time. The important part is you backup metadata storage at a particular point in time. It contains pointers to deep storage, and as long as you have a consistent snapshot of metadata storage, you will have a consistent view of deep storage.
Here is what I would recommend.
Take regular snapshots of the metadata storage. Maybe one snapshot per day, and store them for however long you like. It’s good to store them for at least a couple of weeks, in case you need to roll back for some reason.
Back up new segments in deep storage when they appear. It isn’t important to take consistent snapshots, just to get every file eventually.
People often take this approach with S3 (just trusting it). It’s up to the user. In that case, (1) is still important, but you can skip (2).