How to take metadata backup?

I have 6 Datanodes and 1 NameNode, I have taken a snapshot from the name node but I also want to take a backup of druid metadata, I am using Postgres for metadata storage.

I don’t believe druid has any mechanism for taking backups of metadata since it’s Bring Your Own style of metadata storage.

For postgres, you have a few options (I’m not a postgres expert but I’ve used some of these in the past) and I’m probably missing a lot of them:

  1. pg_dump/pg_dumpall - PostgreSQL: Documentation: 14: pg_dump
  • I have used this one personally to back up single DB or entire DB. Easy to use and test on test DBs
  1. File system backup - PostgreSQL: Documentation: 14: 26.2. File System Level Backup
  • never actually used this before, I would rather have pg_dump do it personally since it can rebuild the entire DB from commands (meaning the process is visible to the user since you can read the commands that will occur to build/rebuild the DB), allows single table/DB restore vs whole DB, and also doesn’t have to have everything shutdown.
  1. PITR (Point in Time Recovery) - PostgreSQL: Documentation: 14: 26.3. Continuous Archiving and Point-in-Time Recovery (PITR)
  • Uses WAL (write ahead logs) to restore a DB and requires a base filesystem to be archived. This is closer to a traditional snapshot (base files need to be there and snapshot restores new/changes to filesystem) you may be looking for.
2 Likes