Zum Inhalt

Cryptographic Deduplicated Backups: BorgBackup, Borgmatic, Append-Only SSH & 3-2-1-1-0 Recovery

Executive Summary: Relying on RAID arrays or unencrypted rsync scripts leaves infrastructure vulnerable to ransomware encryption, silent filesystem bit-rot, and accidental rm -rf propagation. This architectural guide details the deployment of BorgBackup orchestrated via Borgmatic, implementing content-defined Buzhash chunk deduplication, authenticated AEAD AES-256-OCB / BLAKE2b client-side encryption, atomic pre-backup PostgreSQL/MariaDB database dumps, and Append-Only (--append-only) SSH authorized_keys isolation to guarantee immutable ransomware-proof off-site recovery.

  • Buzhash Variable-Size Deduplication

    Split files into content-defined rolling-hash chunks so moving, renaming, or appending to multi-gigabyte VM images only transmits changed byte blocks.

  • repokey-blake2 AEAD Encryption

    Encrypt all data chunks, filenames, and metadata client-side before a single byte leaves the host, leaving the remote storage server zero-knowledge.

  • Immutable Append-Only SSH Target

    Restrict the backup client's SSH key via command="borg serve --append-only" so even a root-compromised server cannot delete or overwrite historical archives.

  • Declarative Borgmatic Pipelines

    Automate streaming pg_dump/mariadb-dump hooks, GFS (keep_daily/keep_monthly) retention pruning, and cryptographic borg check verification in one YAML file.

1. Architecture: Buzhash Chunking, Client-Side AEAD & Append-Only Storage

When BorgBackup processes a directory tree, it never trusts timestamps alone for new files: it streams data through a Buzhash rolling window, splitting streams into variable-length chunks (between $64\text{ KiB}$ and $4\text{ MiB}$). Each chunk is hashed using a keyed HMAC/BLAKE2b MAC, compressed (zstd), and encrypted client-side before being transmitted over SSH to the remote repository:

%%{init: {'flowchart': {'wrappingWidth': 600, 'subGraphTitleMargin': {'top': 15, 'bottom': 15}}}}%%
flowchart TB
    subgraph SourceHost["Production Host (Borgmatic + Borg Client)"]
        DBHook["Borgmatic Pre-Hook<br/>Streaming pg_dump / MariaDB"]
        Buzhash["Buzhash Rolling Chunker<br/>Deduplication Index"]
        Crypto["zstd-6 Compression +<br/>repokey-blake2 Encryption"]
    end

    subgraph UntrustedSSH["SSH Transport (Port 22)"]
        SSHStream["Encrypted SSH Pipe<br/>Zero Server-Side Plaintext"]
    end

    subgraph BackupVault["Off-Site Storage Vault (Borg Server)"]
        AuthKeys["SSH authorized_keys<br/>borg serve --append-only"]
        Repo[("Immutable Borg Repository<br/>Transaction Log Segments")]
    end

    DBHook -->|"Stdout Pipe"| Buzhash
    Buzhash -->|"Unique Chunks Only"| Crypto
    Crypto --> SSHStream
    SSHStream --> AuthKeys
    AuthKeys -->|"Write New Segment Only"| Repo

Why Append-Only Mode Defeats Ransomware

If an attacker gains root access on your production server and inspects /root/.ssh and /etc/borgmatic/config.yaml, they obtain the SSH private key and repository passphrase. On a standard SSH backup server, the attacker can immediately run borg delete ssh://backup-vault/./repo or rm -rf, destroying all backups before deploying ransomware.

When the remote backup server enforces command="borg serve --append-only --restrict-to-path /srv/borg/prod-node" in ~/.ssh/authorized_keys, the remote borg serve process ignores or converts every delete and prune RPC into a non-destructive transaction log entry. All historical segments remain intact on disk.

2. Hardening the Remote Backup Server (--append-only)

On your off-site backup server (or Hetzner Storage Box / ZFS NAS), create a dedicated borg user and restrict the production server's Ed25519 SSH public key inside /srv/borg/.ssh/authorized_keys:

/srv/borg/.ssh/authorized_keys (On the Remote Backup Vault)
command="borg serve --append-only --restrict-to-path /srv/borg/repos/prod-node-01",restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... root@prod-node-01

Initialize the repository from the production host using repokey-blake2 (optimal on 64-bit CPUs without AES-NI bottlenecks, or repokey for AES-256-CTR + HMAC-SHA256) and immediately export an offline paper/password-manager copy of the key:

Initializing the Borg Repository & Exporting the Emergency Key Backup
export BORG_PASSPHRASE="Your-High-Entropy-64-Char-Passphrase"

# 1. Initialize encrypted repository over SSH
borg init --encryption=repokey-blake2 ssh://borg@vault.nejat-hakan.de:22/srv/borg/repos/prod-node-01

# 2. MANDATORY: Export the encrypted key material and store it inside Vaultwarden / printed paper!
borg key export ssh://borg@vault.nejat-hakan.de:22/srv/borg/repos/prod-node-01 ~/borg-prod-node-01-key-backup.txt
borg key export --paper ssh://borg@vault.nejat-hakan.de:22/srv/borg/repos/prod-node-01 ~/borg-prod-node-01-paper.txt

Without BOTH the Key File and the Passphrase, Recovery is Impossible

In repokey and repokey-blake2 modes, the encrypted chunk-encryption key is stored inside the repository config file. However, if the repository header is corrupted, or if you use keyfile-blake2, losing the exported key means permanent cryptographic loss of all backups—even if you know the passphrase. Always store borg key export output in a separate physical location (3-2-1-1-0 rule).

3. Declarative Production Pipeline with Borgmatic (/etc/borgmatic/config.yaml)

Install borgbackup and borgmatic (sudo apt install borgbackup borgmatic) and define your complete backup, database-dump, retention, consistency check, and Uptime Kuma monitoring pipeline in /etc/borgmatic/config.yaml (chmod 0600):

/etc/borgmatic/config.yaml
source_directories:
  - /etc
  - /opt
  - /var/vmail
  - /home

repositories:
  - path: ssh://borg@vault.nejat-hakan.de:22/srv/borg/repos/prod-node-01
    label: offsite_vault
  - path: /mnt/local_zfs_backup/borg/prod-node-01
    label: local_zfs

exclude_patterns:
  - "*.pyc"
  - "*/.cache/*"
  - "/var/lib/docker/overlay2/*"
  - "/opt/*/data/logs/*"
exclude_caches: true
exclude_if_present:
  - .nobackup

# --- Compression & Encryption ---
compression: auto,zstd,6
encryption_passcommand: "cat /etc/borgmatic/.borg_passphrase"
archive_name_format: "{hostname}-{now:%Y-%m-%dT%H:%M:%S}"
ssh_command: "ssh -i /root/.ssh/id_ed25519_borg -o StrictHostKeyChecking=accept-new"

# --- Grandfather-Father-Son (GFS) Retention Policy ---
keep_hourly: 24
keep_daily: 14
keep_weekly: 8
keep_monthly: 12
keep_yearly: 3

# --- Cryptographic Repository & Archive Verification ---
checks:
  - name: repository
    frequency: 1 week
  - name: archives
    frequency: 2 weeks
  - name: data
    frequency: 1 month
    max_duration: 3600

# --- Atomic Streaming Database Dumps (Zero Temporary Disk Overhead) ---
postgresql_databases:
  - name: all
    hostname: 127.0.0.1
    port: 5432
    username: postgres
    format: custom

mariadb_databases:
  - name: all
    hostname: 127.0.0.1
    username: root
    password: "${MARIADB_ROOT_PASSWORD}"
    options: "--single-transaction --quick"

# --- Dead-Man's-Switch Uptime Kuma Heartbeat Integration ---
uptime_kuma:
  push_url: "https://status.nejat-hakan.de/api/push/x9Km2pL8qR"
  states:
    - start
    - finish
    - fail

4. Verification, Live FUSE Mounting & Bare-Metal Restore Drills

A backup is only valid once a restore drill has been verified. Borg allows mounting any historical archive as a read-only userspace FUSE filesystem (borg mount) to inspect or copy individual files, or extracting specific database dumps:

Executing Borgmatic & Performing a Live FUSE Restore Drill
# 1. Validate Borgmatic configuration syntax
sudo borgmatic config validate

# 2. Execute full backup + database dump + prune + check cycle with verbose telemetry
sudo borgmatic --verbosity 1 --stats

# 3. List all historical archives and inspect deduplicated size
sudo borgmatic list
sudo borgmatic info --archive latest

# 4. Mount the latest archive as a read-only FUSE filesystem for granular file recovery
sudo mkdir -p /mnt/borg-restore
sudo borgmatic mount --archive latest --mount-point /mnt/borg-restore
ls -la /mnt/borg-restore/etc
sudo borgmatic umount --mount-point /mnt/borg-restore