Author	Nejat Hakan
eMail	nejat.hakan@outlook.de
PayPal Me	https://paypal.me/nejathakan

Backup Server - BorgBackup + Borgmatic

Introduction to Backup Strategies and Borg

In the digital age, data is undeniably one of the most valuable assets for individuals and organizations alike. Whether it's personal photos, critical research data, business documents, or application configurations, the loss of this data can range from inconvenient to catastrophic. This underscores the critical importance of robust backup strategies. A well-thought-out backup plan is not just a good idea; it's an essential component of responsible data management and system administration.

At its core, a backup is a copy of data taken and stored elsewhere so that it may be used to restore the original after a data loss event. These events can stem from various causes: hardware failure (e.g., disk crashes), software corruption, human error (accidental deletion), malicious attacks (ransomware), or natural disasters.

Key Backup Concepts

Before diving into specific tools, it's crucial to understand some fundamental concepts:

Recovery Point Objective (RPO): This refers to the maximum acceptable amount of data loss measured in time. For example, an RPO of 1 hour means the organization can tolerate losing up to 1 hour of data. This dictates how frequently backups should occur. If you back up daily, your RPO is 24 hours.
Recovery Time Objective (RTO): This is the maximum acceptable downtime for a system or application after a disaster or failure. It defines how quickly you need to restore the data and get the system operational again. RTO influences the choice of backup media, restoration procedures, and potentially the need for failover systems.
The 3-2-1 Rule: This is a widely recommended best practice for backups:
- Keep at least three copies of your data. This includes the original production data and at least two backups.
- Store these copies on at least two different types of media. This protects against failures specific to one type of storage (e.g., all hard drives failing, or a particular brand of SSD having a fault). Examples include internal hard drives, external hard drives, USB drives, magnetic tapes, network-attached storage (NAS), and cloud storage.
- Keep at least one copy offsite. This protects against localized disasters like fire, flood, or theft affecting your primary location. An offsite copy could be on a remote server, in cloud storage, or a physical disk stored at a different geographical location.

Types of Backups

Understanding the different types of backups helps in designing an efficient strategy:

Full Backup: A full backup copies all selected data. While it's the most straightforward and provides the simplest restoration process (only one backup set is needed), it's also the most time-consuming and requires the most storage space. Full backups are typically performed periodically (e.g., weekly or monthly).
Incremental Backup: An incremental backup only copies the data that has changed since the last backup of any type (full or incremental). Restoring from incremental backups requires the last full backup and all subsequent incremental backups in sequence. They are fast to create and use minimal storage but can make restoration more complex and time-consuming.
Differential Backup: A differential backup copies all data that has changed since the last full backup. To restore, you only need the last full backup and the latest differential backup. They use more space than incrementals (as they grow over time until the next full backup) but offer a faster restore process than a series of incrementals.

Why BorgBackup?

There are numerous backup tools available, but BorgBackup (often just called "Borg") stands out for several compelling reasons, particularly in the self-hosting and Linux communities:

Deduplication: This is Borg's flagship feature. Borg doesn't just back up files; it breaks data into smaller, variable-sized chunks. It then stores each unique chunk only once in the repository. If multiple files share the same content, or if a file has only minor changes between backups, Borg only stores the new or changed chunks. This results in significant storage space savings, especially for repetitive data or frequent backups of large files with small modifications.
Compression: Borg supports various compression algorithms (e.g., lz4, zstd, zlib, lzma) to further reduce the storage footprint of your backups. You can choose the algorithm that best balances compression ratio and CPU usage for your needs.
Encryption: Security is paramount for backups. Borg provides robust, authenticated client-side encryption. This means your data is encrypted before it leaves your machine and is stored encrypted in the repository. Only someone with the correct key or passphrase can decrypt the data. Borg supports different encryption modes, allowing for flexibility and strong security.
Efficiency: Borg is designed to be fast and efficient, both in terms of backup speed and resource usage, especially when combined with its deduplication capabilities.
Mountable Backups: Borg allows you to mount your backup archives as a filesystem using FUSE (Filesystem in Userspace). This makes it incredibly easy to browse your backups, compare versions, and restore individual files without needing to extract the entire archive.
Cross-Platform: While primarily developed and used on Linux, Borg can also run on macOS, BSD, and Windows (via Windows Subsystem for Linux - WSL, or Cygwin).
Open Source: BorgBackup is free and open-source software, giving you full control and transparency.

Introducing Borgmatic

While BorgBackup itself is a powerful command-line tool, managing backup configurations, scheduling, pruning old backups, and performing consistency checks can become tedious. This is where Borgmatic comes in.

Borgmatic is a wrapper script and configuration tool for BorgBackup that simplifies and automates common backup tasks. Key features of Borgmatic include:

Configuration-driven: It uses simple YAML configuration files to define backup sources, repositories, encryption settings, retention policies, and hooks.
Automation: Easily integrates with system schedulers like cron or systemd timers to automate your backup processes.
Pruning Policies: Borgmatic allows you to define sophisticated policies for automatically deleting old backups (e.g., keep the last 7 daily, 4 weekly, and 6 monthly backups), helping to manage storage space effectively.
Consistency Checks: It can automate running borg check to ensure the integrity of your repositories and archives.
Hooks: You can define scripts to run before or after backups, or on error. This is useful for tasks like dumping databases before a backup, sending notifications, or custom cleanup actions.
Multiple Configurations: Manage backups for different sets of data or to different repositories using a single Borgmatic setup.

Together, BorgBackup provides the core backup engine with its advanced features like deduplication and encryption, while Borgmatic provides the user-friendly configuration layer and automation capabilities, making them an excellent duo for a robust self-hosted backup server.

Workshop Getting Started with Backup Concepts

This workshop aims to solidify your understanding of basic backup concepts through a practical scenario.

Scenario: You are a university student working on your final year thesis. This document is critical, and its loss would be devastating. You need to devise a basic backup strategy.

Project Goals:

Define a Recovery Point Objective (RPO) and Recovery Time Objective (RTO) for your thesis.
Identify at least three different storage media options for your backups.
Manually simulate a simple backup and versioning process to understand the effort involved without automation, highlighting why tools like Borg are beneficial.

Steps:

Define RPO and RTO:
- RPO: How much work (measured in time) are you willing to lose on your thesis? If you work on it for 3 hours straight, and your computer crashes, would losing those 3 hours of work be acceptable? Or would you prefer to only lose, say, the last 30 minutes of work?
  - Think about it: For a critical thesis, a shorter RPO is generally better. Perhaps an RPO of 1 hour, or even 30 minutes if you're making rapid progress, might be appropriate. This means you should save your work frequently and aim to back it up in line with this RPO.
- RTO: If your primary computer fails, how quickly do you need to be able to access your thesis and resume working? Can you wait a day to get a new laptop and restore, or do you need access within an hour from another device?
  - Think about it: For a thesis with a deadline, a shorter RTO is desirable. Perhaps an RTO of a few hours would be a good target. This influences where and how you store backups for quick accessibility.
Identify Backup Media Options (Applying the 3-2-1 Rule Conceptually):
- Your first copy is the live document on your laptop's hard drive.
- Think of at least two other different types of media for your backups:
  - Example 1 (Different Media Type, Onsite): A USB flash drive or an external SSD/HDD.
  - Example 2 (Different Media Type, Offsite): Cloud storage (like Google Drive, Dropbox, or a university-provided OneDrive), or another computer you own at a different location.
- Ensure at least one copy is offsite: Cloud storage inherently fulfills this. If using two physical drives, one should ideally be stored away from your primary work location (e.g., at a friend's house, a safety deposit box, or taken home if you work at the university).
Manual Backup and Versioning Simulation:
- Create a new folder on your computer named MyThesis_Project.
- Inside this folder, create a simple text file named thesis_v1.txt. Add some initial content, for example:
```
Thesis Draft - Chapter 1 Introduction
Version 1 - Initial thoughts.
Date: [Today's Date]
```
- Now, simulate making a backup to an "external drive." Create another folder on your computer (e.g., MyThesis_Backup_USB).
- Copy thesis_v1.txt from MyThesis_Project to MyThesis_Backup_USB.
- Go back to MyThesis_Project/thesis_v1.txt. Make some changes and add more content.
- Before saving, you decide you want to keep the previous version. So, in MyThesis_Project, rename thesis_v1.txt to thesis_v1_old_datetime.txt (e.g., thesis_v1_old_20231027_1000.txt) and then save your current work as a new file, say thesis_v2.txt.
- Now, copy thesis_v2.txt to your MyThesis_Backup_USB folder. You might also want to copy the versioned thesis_v1_old_datetime.txt there for completeness.
- Repeat this process a few times, making minor changes, saving new versions (e.g., thesis_v3.txt), and manually copying them to your backup folder.
Reflection:
- Consider the manual effort involved in:
  - Remembering to make copies.
  - Naming versions consistently.
  - Copying to the backup location.
  - What if you forget? What if you overwrite the wrong file?
  - How much storage space are all these full copies taking, even if changes are small? (This is where deduplication will shine).
- This manual process, while simple for one file, becomes cumbersome and error-prone for many files or entire projects. It also doesn't efficiently use storage space if only small parts of your files change.

This workshop should give you a tangible appreciation for the challenges of manual backups and set the stage for understanding how tools like BorgBackup and Borgmatic can automate and optimize this crucial process, addressing issues like versioning, storage efficiency, and reliability.

1. Setting Up Your First BorgBackup Repository

Before you can back up any data with BorgBackup, you need a place to store those backups. This storage location, managed by Borg, is called a repository. Setting up a repository is the foundational first step. This section will guide you through installing BorgBackup, choosing a repository location, and initializing your first repository with appropriate encryption.

Prerequisites

BorgBackup has a few dependencies, primarily Python and some related libraries.

Python: BorgBackup is written in Python. You'll typically need Python 3.7 or newer. Most modern Linux distributions come with a suitable Python version.
BorgBackup Software: You need to install the BorgBackup package itself.
OpenSSH (for remote repositories): If you plan to use remote repositories over SSH (which is very common and highly recommended for offsite backups), you'll need OpenSSH client on the machine performing the backup (the client) and OpenSSH server on the machine hosting the repository (the server).

Installation

The installation method varies slightly depending on your operating system.

On Linux (Debian/Ubuntu based):

sudo apt update
sudo apt install borgbackup

On Linux (Fedora/RHEL based):
```
sudo dnf install borgbackup
```
On macOS (using Homebrew):
```
brew install borgbackup
```
Using pip (Python's package installer - can be used on most OSes if a system package isn't available or you need a specific version): It's generally recommended to install Borg within a Python virtual environment to avoid conflicts with system Python packages.
```
python3 -m venv borg-env
source borg-env/bin/activate
pip install borgbackup
# To deactivate the virtual environment later: deactivate
```
Standalone Binaries: Borg also offers standalone binaries for Linux and macOS on their official website or GitHub releases page. These are convenient as they bundle all dependencies.

To verify the installation, open a terminal and type:

borg --version

This should output the installed BorgBackup version.

Choosing a Repository Location

A Borg repository is essentially a directory that Borg manages. You can create a repository in several places:

Local Directory: On the same machine where you're running Borg. This could be an internal hard drive, an external USB drive, or a separate partition.
- Pros: Simple to set up, fast access.
- Cons: Doesn't protect against local disasters (fire, theft, machine failure) unless the local directory is on a physically separate, detachable medium that is stored offsite.
- Example path: /mnt/backup_drive/borg_repo or ~/borg_backups/my_project_repo
Remote Server via SSH: On a remote machine accessible via SSH. This is the most common setup for robust backups, especially for achieving offsite storage.
- Pros: Excellent for offsite backups (3-2-1 rule), data is protected from local disasters.
- Cons: Requires a remote server with SSH access and Borg installed on it. Network speed can be a bottleneck.
- Example path (SSH URL format): ssh://user@your-remote-server.com/path/to/borg_repo or user@your-remote-server.com:/path/to/borg_repo (the ssh:// prefix is optional for recent Borg versions if the path contains @ and :).

For this initial setup, we'll focus on a local repository for simplicity. Remote repositories will be covered in an intermediate section.

Initializing a Borg Repository (borg init)

Once you've decided on a location, you use the borg init command to prepare the directory as a Borg repository. This command sets up the necessary internal structure and, crucially, configures encryption.

Encryption Options

Borg mandates encryption for repositories. You cannot create an unencrypted Borg repository. This is a strong security feature. When you initialize a repository, you must choose an encryption mode:

repokey mode (Recommended for most users):
- The encryption key is stored within the repository directory itself (config file), but it is encrypted with a passphrase you provide during initialization.
- To access the repository (backup, restore, prune, etc.), you'll need to provide this passphrase.
- Pros: The key material travels with the repository. If you copy the repository directory to another location, the key (albeit passphrase-protected) is part of it. Simpler key management than keyfile mode if you are consistent with your passphrase.
- Cons: If someone gains access to your repository directory and your passphrase, they can decrypt your data. The security heavily relies on the strength of your passphrase.
keyfile mode:
- The encryption key is stored in a separate file (the "keyfile") located outside the repository directory, typically in your home directory (e.g., ~/.config/borg/keys/my_repo_key).
- To access the repository, Borg needs access to this keyfile. The keyfile itself can optionally be passphrase-protected.
- Pros: Strong separation of key material from the repository data. If someone copies only the repository directory, they don't have the key. Potentially more secure if the keyfile is well-protected and on a different storage medium.
- Cons: You are responsible for backing up and securing this keyfile. If you lose the keyfile, you lose access to all data in your repository, permanently. This is a critical point.
none (Unencrypted - This is NOT for the repository itself, but for the key when using repokey/keyfile): This option means that the key (either the repokey within the repository or the external keyfile) is not itself encrypted with a passphrase.
- For repokey mode: borg init --encryption=repokey-blake2-chacha20-poly1305 (example using a modern cipher, the passphrase prompt will still appear, but you could use BORG_PASSPHRASE environment variable or --passphrase for automation, potentially from a script or password manager, but be careful with plaintext passphrases). Borg documentation often refers to providing the passphrase via BORG_PASSPHRASE or BORG_PASSCOMMAND when saying "no interactive passphrase".
- For keyfile mode: borg init --encryption=keyfile-blake2-chacha20-poly1305 (here, if you don't set a passphrase for the keyfile, the keyfile is stored unencrypted on disk).
- Pros: Useful for automated backups where interactively entering a passphrase is not feasible.
- Cons: Reduced security. If someone gains access to the unencrypted key (either the config file in repokey mode if the passphrase was empty/easily guessed, or the unencrypted keyfile), they can decrypt all your data. This mode should only be used if the key itself or the passphrase source is very well protected.

Choosing an Encryption Cipher (Advanced detail, but good to know):

Borg supports different underlying encryption algorithms.
Examples:

repokey-aes-ocb (older, might be removed in future Borg versions)
repokey-blake2-chacha20-poly1305 (modern, strong, and fast authenticated encryption)
keyfile-aes-ocb
keyfile-blake2-chacha20-poly1305

If you don't specify, Borg usually picks a secure default (often repokey-blake2). For new repositories, using one of the blake2 variants is generally recommended (e.g., repokey-blake2 or keyfile-blake2). The blake2 refers to the MAC (Message Authentication Code) algorithm used for integrity and authenticity, while chacha20-poly1305 or aes-256-gcm (another modern option if aes-ocb is phased out) would be the encryption ciphers. Borg's defaults are strong. authenticated_blake2_chacha20_poly1305 is a common strong default.

Best Practices for Passphrases/Keyfiles:

Strong Passphrases: If using passphrase-based encryption (common with repokey mode), choose a long, complex, and unique passphrase. Consider using a password manager to generate and store it. A weak passphrase makes your encrypted backup vulnerable.
Secure Keyfile Storage: If using keyfile mode, treat the keyfile like the crown jewels. Store it securely, back it up to multiple safe locations (separate from the repository itself), and restrict access to it.
Environment Variables for Automation: For automated backups (e.g., scripts, cron jobs), you can provide the passphrase via the BORG_PASSPHRASE environment variable or a command specified by BORG_PASSCOMMAND. Be extremely cautious about how and where you store this passphrase or script.

The borg init Command Syntax:

borg init --encryption=<mode> /path/to/your/repository

Example using repokey mode (Borg will prompt for a passphrase):

borg init --encryption=repokey /mnt/external_drive/my_borg_backups

If you want to specify a particular cipher combination like the modern BLAKE2b MAC with ChaCha20/Poly1305 encryption, you might use:

borg init --encryption=repokey-blake2-chacha20-poly1305 /mnt/external_drive/my_borg_backups

Borg also has shorter aliases like repokey-blake2. Check borg init --help for the most up-to-date list of encryption modes and their full names. For simplicity, repokey often defaults to a strong modern cipher.

Verifying Repository Initialization

After running borg init, the specified directory will be populated with a config file, a data directory, README, and an index.N file. You shouldn't manually modify these files.

You can verify basic repository accessibility by listing its (currently empty) contents:

borg list /path/to/your/repository

This command will prompt for the passphrase if you used passphrase-protected repokey or keyfile mode. If successful, it will show nothing (as no archives exist yet) or just confirm connection.

Workshop Initializing a Local Borg Repository

This workshop will guide you through installing BorgBackup, creating a local directory for your backup repository, and initializing it using repokey encryption with a passphrase.

Project Goals:

Install BorgBackup on your system.
Create a dedicated directory for a local Borg repository.
Initialize this repository using borg init with repokey encryption mode.
Verify the initialization.
Understand the importance of the chosen passphrase.

Prerequisites:

A Linux, macOS, or WSL environment.
Sudo/administrator privileges if needed for installation.

Steps:

Install BorgBackup:
- Open your terminal.
- Follow the installation instructions for your OS provided earlier in this section. For example, on Debian/Ubuntu:
```
sudo apt update
sudo apt install borgbackup -y
```
- Verify the installation:
```
borg --version
```
  You should see the BorgBackup version number printed (e.g., borg 1.2.6).
Create a Local Repository Directory:
- Choose a location for your repository. For this workshop, let's create a directory in your home folder named borg_repo_local.
```
mkdir ~/borg_repo_local
```
- You can verify its creation:
```
ls -d ~/borg_repo_local/
```
Initialize the Borg Repository:
- Now, use the borg init command. We'll use the repokey encryption mode, which is common and relatively straightforward for starting. Borg will choose a strong default cipher suite for repokey.
```
borg init --encryption=repokey ~/borg_repo_local
```
- Crucial Step - Passphrase: Borg will now prompt you to enter a new passphrase for the repository:
```
Enter new passphrase for key /home/youruser/borg_repo_local/config:
Enter same passphrase again:
```
  Choose a strong, memorable passphrase. Write this passphrase down and store it in a secure location (e.g., a password manager). If you forget this passphrase, you will lose all access to the backups stored in this repository. For this workshop, you can use something simple like "borgtesting123", but understand this is NOT for production.
- After successful initialization, Borg might output a message like:
```
IMPORTANT: You need to commit /home/youruser/borg_repo_local/config to சகses.
The repository key is stored in the file /home/youruser/borg_repo_local/config.
The integrity of the repository metadata is protected by a MAC.
```
  (The "commit ... to சகses" message is a bit cryptic, it's essentially saying the config file now contains the encrypted key.) Some versions might just return to the prompt silently on success.
Inspect the Repository Directory (Optional):
- Take a look inside the ~/borg_repo_local directory:
```
ls -la ~/borg_repo_local/
```
- You should see files like config, README, and a directory named data.
  - The config file contains repository settings and the (encrypted) repository key.
  - The data directory is where Borg will store the actual backup data chunks.
  - Do not manually delete or modify files within this directory unless you know exactly what you are doing (e.g., following official Borg documentation for rare recovery scenarios).
Verify Repository Accessibility:
- Use the borg list command to check if you can access the repository. Since it's new, it won't contain any archives.
```
borg list ~/borg_repo_local
```
- Borg will prompt you for the passphrase you just set:
```
Enter passphrase for /home/youruser/borg_repo_local/config:
```
- Enter the passphrase. If it's correct, the command will execute without error but will likely show no output, or just the repository path, because there are no archives yet. If you enter an incorrect passphrase, you'll get an error.
Discuss Common Errors and Troubleshooting:
- "Repository does not exist" or "Is not a Borg repository": This usually means you've pointed borg init or borg list to the wrong path, or you forgot to run borg init on that path. Double-check your directory paths.
- Passphrase incorrect: If borg list (or any other command accessing the repo) fails with a decryption error or "Invalid passphrase," you're entering the wrong passphrase. This highlights the importance of securely storing it.
- Permission errors: If the directory ~/borg_repo_local is not writable by your user, borg init will fail. Ensure you have write permissions.

You have now successfully initialized your first local BorgBackup repository! It's encrypted and ready to store backups. The next step will be to create your first backup. Remember the passphrase – it's your key to your backed-up data.

2. Performing Your First Backup with Borg

With your Borg repository initialized, you're ready to create your first backup. The borg create command is central to this process. It takes the data you specify, processes it (chunking, deduplicating, compressing, encrypting), and stores it as a new "archive" within your repository. An archive is like a snapshot of your data at a specific point in time.

Basic borg create Command Syntax

The fundamental structure of the borg create command is:

borg create [options] REPOSITORY::ARCHIVE_NAME PATH_TO_BACKUP_1 [PATH_TO_BACKUP_2 ...]

Let's break this down:

borg create: The Borg command to create a new backup archive.
[options]: Various flags you can use to modify Borg's behavior (e.g., compression, exclusion patterns, statistics). We'll explore some key options.
REPOSITORY: The path to your Borg repository (e.g., ~/borg_repo_local or ssh://user@server/path/to/repo).
::ARCHIVE_NAME: This is crucial. The :: (double colon) separates the repository path from the name you give to this specific backup archive. Choose descriptive archive names.
PATH_TO_BACKUP_1 [PATH_TO_BACKUP_2 ...]: One or more paths to the files or directories you want to include in this backup.

Understanding Archive Naming Conventions

The ARCHIVE_NAME part is user-defined. It's good practice to use a consistent and informative naming scheme. A very common convention is to include the hostname and the date/time of the backup:

myhostname-%Y-%m-%d_%H-%M-%S (e.g., desktop-2023-10-28_14-30-00)
{hostname}-{now:%Y-%m-%d_%H-%M} (using Borg's placeholder substitution)

Borg itself provides placeholders that it can automatically fill in. To see available placeholders, you can check borg help placeholders. Some useful ones include:

{hostname}: The hostname of the machine creating the backup.
{now}: The current timestamp. You can format it using standard strftime directives (e.g., {now:%Y-%m-%d}).
{user}: The current username.

Example using placeholders:

borg create ~/borg_repo_local::'{hostname}-{now:%Y-%m-%d-%H%M%S}' ~/Documents ~/Pictures

This command would create an archive in ~/borg_repo_local named something like yourpc-20231028-150000 (if hostname is yourpc and the time is 3 PM on Oct 28, 2023), and it would back up the ~/Documents and ~/Pictures directories. Note the single quotes around the archive name string when using placeholders, to prevent the shell from interpreting characters like { and }.

Selecting Files and Directories to Back Up

You simply list the paths to the files or directories you want to include after the archive name.

~/important_document.txt: Backs up a single file.
~/Documents: Backs up the entire Documents directory and its contents recursively.
~/Documents /var/log: Backs up both the Documents directory and the /var/log directory.

Borg stores the full path of the files as they are on the source system, relative to the root of the backup operation if multiple sources are given, or directly if only one source is specified. When restoring, you'll see these paths.

Excluding Files and Directories (--exclude, --exclude-from)

Often, you'll want to exclude certain files or directories from your backups, such as:

Temporary files (*.tmp, *.bak, *~)
Cache directories (.cache)
Trash directories
Large, easily reproducible files (e.g., virtual machine images if you only want their configs, downloaded ISOs)

Borg provides several ways to exclude items:

--exclude PATTERN or -e PATTERN: Exclude files or directories matching the given pattern. Patterns can be shell-like glob patterns or, with a prefix, regular expressions or path prefixes.
- --exclude '*.tmp': Excludes all files ending with .tmp.
- --exclude 'sh:**/node_modules/*': Excludes all node_modules directories and their contents. The sh: prefix indicates a shell-style pattern. ** matches directories recursively.
- --exclude 're:.*\.log$': Excludes files ending with .log using a regular expression (re: prefix).
- --exclude 'pp:/var/cache': Excludes everything under /var/cache (path prefix pp:).
--exclude-from EXCLUDEFILE: Read exclusion patterns from a specified file, one pattern per line. This is very useful for managing longer lists of exclusions. Example my_exclusions.txt:
```
# Comments are allowed
*.log
sh:**/.cache/*
/home/user/Downloads/*
```
Then use: borg create --exclude-from=my_exclusions.txt ...
--exclude-caches: A convenient option to exclude directories containing a CACHEDIR.TAG file. Many applications create such tags in their cache directories, adhering to the Cache Directory Tagging Standard.
.borgignore or BORG_EXCLUDE_FROM: If a file named .borgignore exists in a directory Borg is processing, or if the environment variable BORG_EXCLUDE_FROM points to a file, patterns from these files will be used for exclusion. This is similar to .gitignore.

Compression Options (--compression ALGO[,LEVEL])

Borg can compress your backup data to save space. You can specify the compression algorithm and, for some, a compression level.

Syntax: --compression ALGO[,LEVEL]
Common Algorithms (ALGO):
- none: No compression (fastest, but uses most space).
- lz4 (Default): Very fast compression and decompression, good for general use where CPU speed is more critical than maximum compression.
- zstd[,LEVEL]: Zstandard. Offers a great balance between compression ratio and speed. Often recommended. Levels range from 1 (fastest) to 22 (highest compression). Default level is 3. E.g., --compression zstd,6.
- zlib[,LEVEL]: Good compression, but slower than lz4 or zstd. Levels 0-9. Default level is 6.
- lzma[,LEVEL]: High compression ratio, but very slow (especially compression). Levels 0-9. Default level is 6.
Choosing:
- For most users, lz4 (the default) or zstd (e.g., zstd,3 or zstd,6) are excellent choices.
- If storage space is extremely limited and backup time is not a major concern, lzma could be considered, but test performance.
- If your data is already compressed (e.g., JPEG images, MP3s, video files), further compression might yield little benefit and just consume CPU. Deduplication will still be effective for identical files.

Example:

borg create --compression zstd,5 ~/borg_repo_local::myarchive-zstd ~/my_data

Dry Runs (--dry-run or -n)

Before performing an actual backup, especially when experimenting with new exclusion patterns or options, it's wise to do a dry run.

borg create --dry-run --list --stats ~/borg_repo_local::test_run ~/Documents

A dry run will simulate the backup process, showing you what files would be backed up, what would be excluded, and estimated statistics, but it won't actually write any data to the repository.

Checking Backup Statistics and Contents (--stats, --list)

These options provide valuable feedback during and after the backup:

--stats: Displays detailed statistics at the end of the backup operation, including:
- Original size (total size of source files).
- Compressed size (size of data after compression).
- Deduplicated size (size of new data actually added to the repository, considering deduplication with existing data).
- Number of files.
- Chunk statistics. This is extremely useful for understanding how effective deduplication and compression are.
--list: Outputs the list of files and directories being backed up (or that would be backed up in a --dry-run). Useful for verifying your include/exclude patterns.
--progress: Shows a progress bar during the backup, indicating files being processed.

A common combination for a verbose backup command:

borg create --verbose --stats --progress --list \
    ~/borg_repo_local::backup-{now:%Y-%m-%d} \
    ~/Documents \
    --exclude '*.tmp' \
    --exclude 'sh:**/node_modules/*'

(The \ at the end of lines is a line continuation character for long commands in the shell.)

You'll be prompted for your repository passphrase when you run borg create.

Workshop Backing Up Your Documents Folder

In this workshop, you'll create your first actual backup using Borg. You will back up a sample "Documents" folder to the local repository you initialized in the previous workshop. You'll experiment with archive naming, view statistics, and practice excluding files.

Project Goals:

Create a sample "Documents" folder with diverse content.
Use borg create to back up this folder to your local Borg repository.
Experiment with different archive naming schemes, including Borg's placeholders.
Use --stats and --list to understand the backup process and results.
Practice excluding a specific subfolder and files of a certain type.

Prerequisites:

BorgBackup installed.
A local Borg repository initialized (e.g., ~/borg_repo_local from the previous workshop).
The passphrase for your repository.

Steps:

Prepare a Sample "Documents" Folder:

If you don't have a suitable Documents folder for testing, create one.

mkdir -p ~/my_sample_docs/work_files
mkdir -p ~/my_sample_docs/temporary_stuff

Populate it with some sample files of different types:

Create a few text files:

echo "This is important document 1." > ~/my_sample_docs/report.txt
echo "Project notes and ideas." > ~/my_sample_docs/work_files/project_alpha.txt
cp ~/my_sample_docs/report.txt ~/my_sample_docs/report_copy.txt # For deduplication demo

Create some temporary files:

echo "This is a temporary file." > ~/my_sample_docs/temporary_stuff/temp_file.tmp
echo "Another temp file." > ~/my_sample_docs/temporary_stuff/scratch.bak

(Optional) Add a few small image files or other binary files if you have them handy. The more varied the content, the better for observing Borg's behavior.

Perform Your First Backup:
- Construct your borg create command. Let's use a simple archive name first, and include --stats, --list, and --progress.
```
borg create --verbose --stats --progress --list \
    ~/borg_repo_local::mydocs-backup-01 \
    ~/my_sample_docs
```
- Borg will prompt for the passphrase for ~/borg_repo_local. Enter it.
- Observe the output:
  - --list will show the files being added.
  - --progress will show activity.
  - --stats at the end will show "Original size," "Compressed size," and "Deduplicated size." For the very first backup, "Deduplicated size" will be close to "Compressed size" as there's no prior data in the repo to deduplicate against.
  - Note the "This archive" vs "All archives" statistics.
Experiment with Archive Naming (Placeholders):
- Let's create another backup, this time using Borg's placeholders for a more dynamic archive name.
```
borg create --verbose --stats \
    ~/borg_repo_local::'{hostname}-{now:%Y-%m-%d_%H%M}' \
    ~/my_sample_docs
```
- Enter the passphrase.
- Observe the statistics. You should see that the "Deduplicated size" for "This archive" is very small (or even zero if no files changed), because most/all data was already in the repository from the first backup. This demonstrates Borg's powerful deduplication.

List Archives in the Repository:

Now, let's see the archives you've created:
```
borg list ~/borg_repo_local
```

Enter the passphrase. You should see your two archives listed, e.g.:

mydocs-backup-01                       Mon, 2023-10-28 10:00:00 [someid]
yourhostname-2023-10-28_1005           Mon, 2023-10-28 10:05:00 [anotherid]

Practice Exclusions:
- Let's say you want to back up ~/my_sample_docs but exclude the temporary_stuff subfolder and all *.bak files.
- Create a new backup with these exclusions:
```
borg create --verbose --stats --list \
    ~/borg_repo_local::mydocs-exclusions-{now:%Y-%m-%d} \
    ~/my_sample_docs \
    --exclude 'sh:**/temporary_stuff/*' \
    --exclude '*.bak'
```
  - sh:**/temporary_stuff/* : The sh: specifies a shell pattern. ** matches any preceding directories (making the pattern work even if temporary_stuff is nested deeper), and /* matches all files and folders directly inside temporary_stuff.
  - '*.bak': Excludes any file ending with .bak anywhere in the backup source.
- Enter the passphrase.
- Examine the output from --list. You should not see temp_file.tmp or scratch.bak listed as being added. The temporary_stuff directory itself might be created as an empty directory in the archive if it's not explicitly excluded with a trailing slash for the directory itself (e.g., sh:**/temporary_stuff/). To exclude the directory and its content, sh:**/temporary_stuff is often enough if it's a directory name. Borg's exclude patterns can be nuanced. For robustness, sh:**/temporary_stuff/* for contents and sh:**/temporary_stuff for the directory itself, or just sh:**/temporary_stuff often works. A common pattern is some/path/* to exclude contents and some/path to exclude the directory itself if it was empty.
- Check the statistics.

Exclusion using --exclude-from (Optional Advanced Step):

Create a file named my_borg_exclusions.txt in your home directory:
```
nano ~/my_borg_exclusions.txt
```

Add the following lines to it:

# Exclude all .tmp files
*.tmp

# Exclude the entire Downloads directory if it were part of the backup
# sh:**/Downloads/* (example, not backing up Downloads in this workshop)

# Exclude the temporary_stuff directory content
sh:**/temporary_stuff/*

Save and close the file.

Now, run a backup using this exclusion file:

borg create --verbose --stats --list \
    ~/borg_repo_local::mydocs-exclude-file-{now:%Y-%m-%d} \
    ~/my_sample_docs \
    --exclude-from=~/my_borg_exclusions.txt

Verify from the --list output that files matching these patterns (like temp_file.tmp) are excluded.

You've now successfully created several backups, seen deduplication in action, and learned how to control what gets backed up using exclusions. This forms the core of using borg create. The next step is to learn how to restore your data.

3. Restoring Files with Borg

Backups are only useful if you can reliably restore your data from them. BorgBackup provides flexible and powerful ways to retrieve your backed-up files, whether you need an entire snapshot, a specific folder, or just a single file. You can even mount your backups as a regular filesystem for easy browsing.

Listing Archives and Their Contents

Before you can restore, you usually need to know which archive contains the data you're interested in and what exactly is inside that archive.

Listing Archives in a Repository (borg list REPOSITORY):
As seen previously, this command shows all archives stored in the repository, along with their creation timestamps and unique IDs.

borg list ~/borg_repo_local

Example output:

Enter passphrase for /home/user/borg_repo_local/config:
mydocs-backup-01                       Sat, 2023-10-28 15:30:00 [abcdef12...]
desktop-2023-10-28_1535                Sat, 2023-10-28 15:35:00 [fedcba98...]
mydocs-exclusions-2023-10-28           Sat, 2023-10-28 15:40:00 [12345678...]

Listing Contents of a Specific Archive (borg list REPOSITORY::ARCHIVE_NAME):
To see the files and directories stored within a particular archive:

borg list ~/borg_repo_local::mydocs-backup-01

This will output a tree-like structure of the archive's contents, similar to the ls -R or tree command. You'll see permissions, owner, group, size, date, and path for each item.

Enter passphrase for /home/user/borg_repo_local/config:
drwxr-xr-x user group       0 Sat, 2023-10-28 15:25:00 home/user/my_sample_docs
-rw-r--r-- user group      30 Sat, 2023-10-28 15:20:00 home/user/my_sample_docs/report.txt
-rw-r--r-- user group      30 Sat, 2023-10-28 15:20:00 home/user/my_sample_docs/report_copy.txt
drwxr-xr-x user group       0 Sat, 2023-10-28 15:25:00 home/user/my_sample_docs/work_files
-rw-r--r-- user group      25 Sat, 2023-10-28 15:21:00 home/user/my_sample_docs/work_files/project_alpha.txt
...

You can also list specific subdirectories within an archive:

borg list ~/borg_repo_local::mydocs-backup-01 home/user/my_sample_docs/work_files

Extracting Data (borg extract)

The borg extract command is used to restore files and directories from an archive to your filesystem.

Extracting an Entire Archive:
To restore everything from an archive to the current directory:
```
# First, navigate to where you want to restore the data, or specify an output path
mkdir ~/restore_location
cd ~/restore_location

borg extract ~/borg_repo_local::mydocs-backup-01
```
This will recreate the directory structure and files from the mydocs-backup-01 archive inside ~/restore_location. Borg by default strips leading components of paths until it finds a common directory or just uses the basename.
- Path Specificity: Borg attempts to be smart about paths. If an archive contains home/user/file.txt, and you extract it, it will typically create that structure.
- --strip-components NUMBER: Similar to tar, you can strip a specified number of leading path components.
- --stdout: Extract file contents to standard output (useful for piping or single files).

Extracting Specific Files or Directories:
You can specify paths within the archive to restore only those items.

# Restore only the report.txt file to the current directory
borg extract ~/borg_repo_local::mydocs-backup-01 home/user/my_sample_docs/report.txt

# Restore the entire work_files directory to the current directory
borg extract ~/borg_repo_local::mydocs-backup-01 home/user/my_sample_docs/work_files

The paths you provide must match the paths as they are stored in the archive (which you can see with borg list REPOSITORY::ARCHIVE_NAME).

Restoring to a Different Location:
By default, borg extract restores to the current working directory. While you can cd to a target directory, it's often cleaner to use shell redirection if restoring single files or specify paths carefully. For multiple files/directories from an archive, Borg recreates the archived path structure. If you want to restore home/user/my_sample_docs/report.txt from an archive into /tmp/recovery/report.txt, you would typically:
```
mkdir -p /tmp/recovery
cd /tmp/recovery
borg extract ~/borg_repo_local::mydocs-backup-01 home/user/my_sample_docs/report.txt
# This would create /tmp/recovery/home/user/my_sample_docs/report.txt
```
To have more control, you might extract and then move, or use borg mount.

If you are restoring specific files and want them in the current directory without their full archived path structure, you might need to extract to a temporary location and then move them, or use --stdout for single files.
```
borg extract --stdout ~/borg_repo_local::mydocs-backup-01 home/user/my_sample_docs/report.txt > ./restored_report.txt
```
Handling Existing Files:
By default, borg extract will not overwrite existing files. It will skip them and print a warning.
- Use the --force option to overwrite existing files. Use with caution!
- Consider restoring to a new, empty directory to avoid accidental overwrites and then manually compare or merge.

Mounting Archives with FUSE (borg mount)

One of Borg's most convenient features for browsing and restoring is its ability to mount an archive (or an entire repository) as a filesystem using FUSE (Filesystem in Userspace). This allows you to navigate your backups with standard file manager tools or command-line utilities as if they were regular directories.

Prerequisites: You need FUSE installed on your system.
- Debian/Ubuntu: sudo apt install fuse3 libfuse3-dev (or fuse for older systems)
- Fedora: sudo dnf install fuse fuse-devel
- macOS: FUSE for macOS (osxfuse/macFUSE) needs to be installed.
Mounting a Specific Archive:
```
# Create a mount point (an empty directory)
mkdir ~/borg_mount_point

# Mount the archive
borg mount ~/borg_repo_local::mydocs-backup-01 ~/borg_mount_point
```
Enter the repository passphrase when prompted. Now, you can cd ~/borg_mount_point or open it in your file manager and browse the contents of mydocs-backup-01 as if it were a live filesystem. You can copy files from here to any other location using standard cp commands or drag-and-drop. The filesystem is read-only by default, which is safe.
Mounting an Entire Repository: You can also mount the entire repository. This will create a directory structure at the mount point where each top-level directory corresponds to an archive name.
```
borg mount ~/borg_repo_local ~/borg_mount_point
```
Then, ~/borg_mount_point would contain directories like mydocs-backup-01, desktop-2023-10-28_1535, etc.
Unmounting: When you're done, you must unmount the FUSE filesystem:
```
borg umount ~/borg_mount_point
# Or, if that fails or Borg is not active:
# fusermount -u ~/borg_mount_point (Linux)
# umount ~/borg_mount_point (macOS, sometimes Linux)
```
It's important to unmount cleanly. If the borg mount process is killed abruptly, the mount point might become unresponsive, sometimes requiring a fusermount -u -z mountpoint (lazy unmount) or even a reboot in worst-case FUSE glitches.

Verifying Restored Data

After restoring files, it's crucial to verify their integrity and correctness.

File sizes and timestamps: Compare them with what you expect.
File content: Open important files (documents, configuration files) to ensure they are readable and contain the correct data.
Checksums: For critical data, if you have checksums (e.g., md5sum, sha256sum) of the original files, you can re-calculate them on the restored files and compare. Borg does internal integrity checks using checksums and MACs for all data it stores, so data corruption within the repository is detected. This verification step is more about ensuring the restore process itself worked as expected and the files are usable in their restored context.

Borg itself has a borg check command (which we'll cover more later) to verify repository integrity, but verifying the usability of restored files is a good practice for your peace of mind.

Workshop Restoring a Deleted File and a Previous Version

This workshop will give you hands-on practice with borg extract and borg mount. You will simulate accidental data loss and version recovery scenarios.

Project Goals:

"Accidentally" delete a file from your ~/my_sample_docs folder.
Use borg extract to restore the deleted file from a previous backup.
Modify a file, create a new backup, and then restore the previous version of that file to a different location.
Use borg mount to browse an archive and copy a file.

Prerequisites:

Your local Borg repository (~/borg_repo_local) with at least two archives of ~/my_sample_docs (as created in the previous workshop). One archive should represent an older state, and another a newer state after some modifications.
The FUSE library installed for borg mount.
Passphrase for your repository.

Steps:

Identify Archives:
- List the archives in your repository to choose which ones you'll use.
```
borg list ~/borg_repo_local
```
- Note down two archive names: an "older" one and a "newer" one. Let's assume you have mydocs-backup-01 (older) and yourhostname-2023-10-28_1005 (newer), or similar names from your previous work.
Scenario 1: Restore a "Deleted" File:
- Let's simulate deleting report.txt from your live ~/my_sample_docs folder.
```
ls ~/my_sample_docs/report.txt # Verify it exists
rm ~/my_sample_docs/report.txt
ls ~/my_sample_docs/report.txt # Verify it's gone (will error)
```
- Now, restore report.txt from your older backup (e.g., mydocs-backup-01) into a temporary recovery directory.
```
mkdir ~/restore_temp
cd ~/restore_temp
borg extract ~/borg_repo_local::mydocs-backup-01 home/user/my_sample_docs/report.txt
```
  (Replace home/user/my_sample_docs/report.txt with the actual path shown by borg list ~/borg_repo_local::mydocs-backup-01 if it's different on your system, for example, it might be just my_sample_docs/report.txt if ~/ was resolved differently during backup.) To find the exact path in the archive:
```
borg list ~/borg_repo_local::mydocs-backup-01 | grep report.txt
```
- Verify the restored file:
```
ls -l home/user/my_sample_docs/report.txt
cat home/user/my_sample_docs/report.txt
```
- You can now copy this file back to its original location if desired:
```
cp home/user/my_sample_docs/report.txt ~/my_sample_docs/
```
- Clean up the temporary restore directory:
```
cd ~ # Go back to home directory
# rm -r ~/restore_temp # Optional, or keep for next steps
```
Scenario 2: Restore a Previous Version of a File:
- First, ensure you have at least two versions of a file across two different backups.
  - Open ~/my_sample_docs/work_files/project_alpha.txt in your live folder. Add a new line like "Version 2 changes made here." Save it.
  - Create a new backup of ~/my_sample_docs:
```
borg create --stats ~/borg_repo_local::mydocs-version2-{now:%Y%m%d} ~/my_sample_docs
```
    You now have an archive (let's call it ARCHIVE_V2 e.g. mydocs-version2-20231028) containing the modified project_alpha.txt, and an older archive (e.g., yourhostname-2023-10-28_1005, let's call it ARCHIVE_V1) containing the original version.
- Let's say you want to retrieve the content of project_alpha.txt before you made the "Version 2" changes.
- Restore the older version from ARCHIVE_V1 to a new file name or new location to avoid overwriting your current live file.
```
# Ensure you are in a directory where you want to restore, e.g., ~/restore_temp
# If ~/restore_temp doesn't exist or was deleted, recreate it:
mkdir -p ~/restore_temp
cd ~/restore_temp

# Extract the older version of the file from ARCHIVE_V1
# Adjust ARCHIVE_V1 name and path as per your setup
borg extract ~/borg_repo_local::yourhostname-2023-10-28_1005 home/user/my_sample_docs/work_files/project_alpha.txt
```
  This will create ~/restore_temp/home/user/my_sample_docs/work_files/project_alpha.txt (with the older content).
- To make it clearer, let's rename it:
```
mv home/user/my_sample_docs/work_files/project_alpha.txt ./project_alpha_v1_restored.txt
rm -rf home # clean up the created directory structure in restore_temp
```
- Verify:
```
cat ./project_alpha_v1_restored.txt # Should show the content before "Version 2 changes"
cat ~/my_sample_docs/work_files/project_alpha.txt # Should show the content with "Version 2 changes"
```

Scenario 3: Browse and Restore using borg mount:

Ensure FUSE is installed (e.g., sudo apt install fuse3 on Debian/Ubuntu).
Create a mount point:
```
mkdir ~/borg_browse_mount
```

Mount one of your archives (e.g., ARCHIVE_V2 which is mydocs-version2-...):

borg mount ~/borg_repo_local::mydocs-version2-$(date +%Y%m%d) ~/borg_browse_mount
# Adjust the archive name to match one you created.
# If you don't remember the exact name use: borg list ~/borg_repo_local
# Example: borg mount ~/borg_repo_local::mydocs-version2-20231028 ~/borg_browse_mount

Enter your repository passphrase.

Now, browse the mounted archive:

ls -l ~/borg_browse_mount/
# You should see the contents of your backup, likely starting with 'home/' or 'my_sample_docs/'
# Navigate into the structure, e.g.:
ls -l ~/borg_browse_mount/my_sample_docs/work_files/ # Adjust path based on what `ls ~/borg_browse_mount/` shows

Copy a file (e.g., project_alpha.txt) from the mount point to your ~/restore_temp directory using standard system commands:

# Adjust the source path according to how it appears in your mount
cp ~/borg_browse_mount/my_sample_docs/work_files/project_alpha.txt ~/restore_temp/project_alpha_from_mount.txt

Verify the copied file:

cat ~/restore_temp/project_alpha_from_mount.txt

Crucial: Unmount the Borg FUSE filesystem:

borg umount ~/borg_browse_mount
# If that gives an error like "borg_browse_mount is not a borg mount point",
# it might be because the borg mount command terminated. Try:
# sudo umount ~/borg_browse_mount  OR  fusermount -u ~/borg_browse_mount

Clean up:

# rm -r ~/restore_temp # If you're done with it
# rmdir ~/borg_browse_mount

You have now practiced restoring individual files, recovering previous versions, and browsing backups using FUSE. These are essential skills for effectively utilizing your Borg backups. Remember that regular testing of your restore procedures is a critical part of any reliable backup strategy.

4. Introduction to Borgmatic for Automation

While BorgBackup is a potent tool for creating and managing backups, manually running borg create commands, remembering complex options, and handling retention (pruning old backups) can become tedious and error-prone, especially for regular backups. This is where Borgmatic shines. Borgmatic is a wrapper script for BorgBackup that uses simple YAML configuration files to automate and simplify your backup operations.

Why Automate Backups?

Consistency:
Automated backups run on a regular schedule without manual intervention, ensuring data is backed up consistently.
Reliability:
Reduces the chance of human error (e.g., forgetting to run a backup, typing incorrect commands).
Efficiency:
Saves you time and effort compared to manual execution.
Retention Management:
Automating the pruning of old backups is crucial for managing storage space.

Installing Borgmatic

Borgmatic is typically installed using Python's package installer, pip. It's highly recommended to install it within a Python virtual environment to avoid conflicts with system packages and manage dependencies cleanly.

Ensure you have pip and venv (Python virtual environment tool): Most Python 3 installations include these. If not:

# For Debian/Ubuntu
sudo apt install python3-pip python3-venv
# For Fedora
sudo dnf install python3-pip python3-virtualenv

Create and activate a virtual environment (optional but recommended):
```
python3 -m venv ~/borgmatic-env
source ~/borgmatic-env/bin/activate
```
Your shell prompt will likely change to indicate you're in the virtual environment (e.g., (borgmatic-env) user@host:$).
Install Borgmatic using pip:
```
pip install borgmatic
```
This will also pull in BorgBackup as a dependency if it's not already available in a way Borgmatic can use, or if a specific compatible version is needed.
Verify installation:
```
borgmatic --version
```
This should output the installed Borgmatic version.

To deactivate the virtual environment later (when you're done using Borgmatic directly from that shell session):
```
deactivate
```
When you schedule Borgmatic (e.g., with cron or systemd), you'll need to make sure to call the borgmatic executable from within its virtual environment, or ensure Borgmatic and its dependencies are on the system PATH.

Basic Borgmatic Configuration File Structure (config.yaml)

Borgmatic uses YAML (YAML Ain't Markup Language) for its configuration files. YAML is human-readable and uses indentation (spaces, not tabs) to define structure.

The default configuration file location is ~/.config/borgmatic/config.yaml or /etc/borgmatic/config.yaml for system-wide configuration. You can also specify a config file using the -c or --config flag.

Let's look at a minimal config.yaml:

location:
    # List of source directories to backup.
    source_directories:
        - /home/youruser/Documents
        - /home/youruser/Pictures

    # List of repository paths.
    repositories:
        - /mnt/backup_drive/my_borg_repo # Path to your Borg repository

storage:
    # Borg repository encryption passphrase.
    # For security, this is often better handled by BORG_PASSPHRASE environment variable
    # or BORG_PASSCOMMAND, especially in production.
    # For this intro, we might set it here or use an environment variable.
    # encryption_passphrase: "your-borg-repo-passphrase" # Less secure if file is readable

    # Alternatively, use BORG_PASSPHRASE environment variable or BORG_PASSCOMMAND for better security
    # For example, to use an environment variable, you'd omit encryption_passphrase here
    # and ensure BORG_PASSPHRASE is set when borgmatic runs.
    # Or use:
    # encryption_passcommand: "cat /path/to/secure/passphrase-file" # More secure

    # Archive name format. {hostname} and {now} are placeholders.
    archive_name_format: '{hostname}-{now:%Y-%m-%d_%H:%M:%S}'

retention:
    # Retention policy for pruning archives.
    # Borgmatic will keep this many daily archives.
    keep_daily: 7
    # Keep this many weekly archives.
    keep_weekly: 4
    # Keep this many monthly archives.
    keep_monthly: 6

    # Pruning prefix. Only consider archives matching this prefix for pruning.
    # Useful if you have different types of backups in the same repository.
    # prefix: '{hostname}-' # By default, matches the hostname part of archive_name_format

# Optional: Hooks for actions before/after backups
# hooks:
#    before_backup:
#        - echo "Starting backup..."
#    after_backup:
#        - echo "Backup finished."
#    on_error:
#        - echo "Backup failed!" >&2

# Optional: Consistency checks
# consistency:
#    checks:
#        - repository
#        - archives
#    check_last: 3 # Check the last 3 archives

Key Sections Explained:

location:
- source_directories: A list of paths on your local system that you want to back up.
- repositories: A list of Borg repository paths where the backups will be stored. You can specify multiple repositories (e.g., one local, one remote).
storage:
- encryption_passphrase: Be very careful with storing plaintext passphrases in config files. While convenient for a quick start, this is a security risk if the config file is compromised. Better alternatives for production:
  - Set the BORG_PASSPHRASE environment variable before running Borgmatic.
  - Use encryption_passcommand: "command_to_output_passphrase" (e.g., pass show borg/myrepo if using the pass password manager, or cat /path/to/protected_file).
  - Let Borg/Borgmatic prompt you if run interactively (not suitable for automation).
- archive_name_format: Defines the naming scheme for archives created by Borgmatic. It uses the same placeholders as borg create.
- Other options like compression, exclude_patterns, exclude_from can also be set here, mirroring borg create flags.
retention:
- Defines how many old backups to keep. This is crucial for managing storage space.
- keep_daily, keep_weekly, keep_monthly, keep_yearly: Specify the number of backups to retain for each period. Borgmatic uses the archive timestamp to determine this.
- prefix: (Optional but often good practice) Borgmatic will only consider archives whose names start with this prefix for pruning. This helps if you have multiple machines or backup sets going to the same repository, ensuring one machine's prune job doesn't delete another's unrelated backups. It often defaults to {hostname}- or can be explicitly set.
hooks (Optional):
- Allows you to run custom shell commands before_backup, after_backup, or on_error. Useful for tasks like dumping a database before backup, sending notifications, etc.
consistency (Optional):
- Configures automated checks (borg check) on your repository and archives to ensure data integrity.

Running Your First Borgmatic Backup

Once you have a config.yaml file:

Validate Configuration:

borgmatic config validate
# Or, if your config is not at the default location:
# borgmatic --config /path/to/your/config.yaml config validate

This checks for YAML syntax errors and basic configuration issues.

Simulate a Run (Dry Run):
It's always a good idea to do a dry run first to see what Borgmatic would do without actually changing anything.
```
borgmatic --verbosity 1 --dry-run
# or shorter: borgmatic -v 1 -n
```
- --verbosity 1 (or -v 1) shows more detailed output about what Borgmatic is doing. -v 0 is less verbose, -v 2 is for debugging.
- --dry-run (or -n) simulates actions like creating archives and pruning, but doesn't execute them.
Perform an Actual Backup:
If the dry run looks good, run Borgmatic to create a backup:
```
borgmatic --verbosity 1
```
This will:
- Read the configuration.
- Execute any before_backup hooks.
- Run borg create for each source directory to each repository.
- Run borg prune according to your retention policies (for each repository).
- Execute any after_backup or on_error hooks.
- Perform consistency checks if configured.
Other useful Borgmatic commands:
- borgmatic init -e repokey: Can help initialize a repository specified in the config if it doesn't exist.
- borgmatic list: Lists archives (uses settings from config).
- borgmatic check: Performs consistency checks.
- borgmatic prune: Only performs pruning.
- borgmatic extract: Helps extract files (though often borg extract direct is also used).

If you're using a virtual environment for Borgmatic, remember to activate it (source ~/borgmatic-env/bin/activate) before running these commands, or call the executable directly: ~/borgmatic-env/bin/borgmatic ....

Workshop Automating Your Local Backup with Borgmatic

In this workshop, you'll set up Borgmatic to automate backups of your ~/my_sample_docs folder to the local Borg repository (~/borg_repo_local) you've been using.

Project Goals:

Install Borgmatic (if not already done).
Create a basic Borgmatic config.yaml file.
Configure source_directories, repositories, and archive_name_format.
Securely handle the Borg repository passphrase (using BORG_PASSPHRASE environment variable for this workshop).
Run Borgmatic to create a backup.
Add a simple retention policy and observe pruning (conceptually or by running borgmatic prune).

Prerequisites:

BorgBackup installed and a local repository initialized (e.g., ~/borg_repo_local) with a known passphrase.
~/my_sample_docs folder with some content.
Python 3, pip, and venv.

Steps:

Install Borgmatic (if you haven't already):

Create and activate a virtual environment:

python3 -m venv ~/borgmatic-env
source ~/borgmatic-env/bin/activate

Install Borgmatic:
```
pip install borgmatic
```
Verify:
```
borgmatic --version
```

Create Borgmatic Configuration Directory and File:

mkdir -p ~/.config/borgmatic
nano ~/.config/borgmatic/config.yaml

Populate config.yaml:

Paste the following basic configuration into config.yaml. Adjust youruser and the repository path if they are different on your system.

location:
    source_directories:
        - /home/youruser/my_sample_docs # Adjust 'youruser'

    repositories:
        - /home/youruser/borg_repo_local # Adjust 'youruser' if needed

storage:
    # We will use the BORG_PASSPHRASE environment variable for the passphrase.
    # So, do NOT uncomment or add 'encryption_passphrase' here for this workshop.
    # If you had to use a command, it would be like:
    # encryption_passcommand: "cat /secure/path/to/passfile"

    archive_name_format: '{hostname}-docs-{now:%Y-%m-%d-%H%M%S}'

retention:
    # Keep the last 7 daily backups
    keep_daily: 7
    # Always keep at least 2 backups, regardless of other policies
    keep_minimum: 2
    # Pruning prefix - important if multiple configs use the same repo
    prefix: '{hostname}-docs-'

# For now, we'll keep hooks and consistency commented out or minimal
# hooks:
#    before_backup:
#        - echo "Borgmatic: Starting backup for my_sample_docs..."
#    after_backup:
#        - echo "Borgmatic: Backup for my_sample_docs finished."
#    on_error:
#        - echo "Borgmatic: Backup for my_sample_docs FAILED!" >&2

Important Note on prefix: The prefix in retention should generally match the static part of your archive_name_format. Here, archive_name_format is '{hostname}-docs-{now...}', so prefix: '{hostname}-docs-' is appropriate. This ensures Borgmatic only prunes archives created by this specific configuration.
Save and close the file (Ctrl+O, Enter, Ctrl+X in nano).

Handle the Repository Passphrase Securely:
- For this workshop, we'll set the BORG_PASSPHRASE environment variable in the terminal session just before running Borgmatic. Remember, this is only for the current terminal session. For persistent automation (cron/systemd), you'd need a more robust way (covered later).
- In your terminal (where borgmatic-env is active):
```
export BORG_PASSPHRASE='your-borg-repo-passphrase'
# Replace 'your-borg-repo-passphrase' with the actual passphrase
# for ~/borg_repo_local
```
  Security Warning: Typing your passphrase directly into export can store it in your shell history. For slightly better interactive security, you could use read -s BORG_PASSPHRASE which doesn't echo:
```
read -s -p "Enter Borg repository passphrase: " BORG_PASSPHRASE
export BORG_PASSPHRASE
echo # To get a new line after the prompt
```
Validate and Dry Run Borgmatic:
- Validate the configuration:
```
borgmatic config validate
# If you used a custom path: borgmatic -c /path/to/config.yaml config validate
```
  It should report success. If not, check your YAML for syntax errors (indentation is key!).
- Perform a dry run:
```
borgmatic --verbosity 1 --dry-run
```
  Observe the output. It should indicate it would create an archive with the name format you specified and then list files to be backed up. It might also simulate pruning if you already have many archives.
Run Borgmatic to Create a Backup:
- If the dry run looks good:
```
borgmatic --verbosity 1
```
- Borgmatic should now create a new archive in ~/borg_repo_local. It will also attempt to prune, but since keep_daily: 7 and keep_minimum: 2, it likely won't delete much unless you have many old archives already matching the prefix.
- You can verify the new archive with borg list ~/borg_repo_local (you'll be prompted for passphrase again if BORG_PASSPHRASE isn't set for the borg list command itself, or if it's in a different shell).
Observe Pruning (Conceptual or by Creating More Archives):
- The retention policy (keep_daily: 7, keep_minimum: 2, prefix: '{hostname}-docs-') means Borgmatic will try to:
  - Keep the 7 most recent daily backups that match the prefix.
  - Ensure at least 2 backups matching the prefix are kept, even if they are older than 7 days.
- To really see pruning in action, you'd need to:
  - Create more than 7 backups with Borgmatic over several (simulated) days (you can just run borgmatic -v 1 multiple times; each run creates a new archive).
  - Or, manually create some older archives with names matching the prefix, e.g., yourhostname-docs-2023-10-15-120000, yourhostname-docs-2023-10-16-120000, etc., then run borgmatic prune -v 1.
- Let's just run the prune command explicitly to see what it would do (it runs as part of the main borgmatic command anyway):
```
# Ensure BORG_PASSPHRASE is still set
borgmatic prune --verbosity 1
```
  If you have enough archives matching the prefix and some are older than 7 distinct days, you might see it prune some. Otherwise, it will state it's keeping them according to the policy.

Deactivate Virtual Environment (if done for now):

deactivate
unset BORG_PASSPHRASE # Good practice to clear the passphrase from env

You have now successfully configured Borgmatic to manage backups for your my_sample_docs folder, including basic retention. This setup forms the foundation for more advanced configurations, such as backing up to remote servers and using hooks, which will be covered in the intermediate sections.

5. Setting Up a Remote Borg Repository via SSH

Storing backups on a separate physical machine, especially one located offsite, is a cornerstone of the 3-2-1 backup rule and significantly enhances data resilience against local disasters (fire, theft, hardware failure of the primary machine). BorgBackup excels at working with remote repositories accessible via SSH (Secure Shell).

This section covers setting up Borg to back up to a remote server, focusing on SSH key-based authentication and security best practices for restricting Borg's access on the server.

Prerequisites for the Remote Server

The machine that will host your Borg repository (the "backup server") needs:

SSH Server:
An OpenSSH server installed and running. Most Linux server distributions include this by default or make it easy to install (e.g., sudo apt install openssh-server on Debian/Ubuntu).
BorgBackup Installed:
BorgBackup must be installed on the remote server as well. The version should ideally be compatible with the Borg version on your client machine (the machine performing the backup). It's generally best if they are the same major.minor version.
User Account:
A dedicated user account on the remote server to own the Borg repository and handle SSH connections is recommended for better security and isolation. Avoid using root.
Storage Space:
Sufficient disk space on the remote server to hold the anticipated backup data.

SSH Key-Based Authentication (Passwordless Login)

Using SSH keys for authentication is more secure and convenient than passwords, especially for automated processes. It allows your client machine to log into the remote server without interactively typing a passphrase.

How it Works (Briefly):
You generate a pair of cryptographic keys: a private key (kept secret on your client machine) and a public key (copied to the remote server). When the client tries to connect, the server uses the public key to issue a challenge that only the corresponding private key can solve.

Steps to Set Up SSH Key-Based Authentication:

On your Client Machine (the one that will run borg create or borgmatic):
- Check if you already have an SSH key pair:
```
ls -al ~/.ssh/id_rsa ~/.ssh/id_ed25519
```
  (Common key types are RSA: id_rsa, id_rsa.pub; Ed25519: id_ed25519, id_ed25519.pub - Ed25519 is generally preferred if supported).
- If you don't have a key pair or want a new one specifically for Borg:
```
# Using Ed25519 (recommended)
ssh-keygen -t ed25519 -C "borg_backup_key_for_myclient_$(date +%Y-%m-%d)"

# Or using RSA (older, but widely compatible)
# ssh-keygen -t rsa -b 4096 -C "borg_backup_key_for_myclient_$(date +%Y-%m-%d)"
```
  - It will ask where to save the key. You can press Enter for the default (e.g., ~/.ssh/id_ed25519). If you choose a custom name (e.g., ~/.ssh/borg_specific_key), you'll need to tell SSH to use it (via ~/.ssh/config or ssh -i).
  - It will ask for a passphrase for the key itself. This is highly recommended. This passphrase protects your private key if it's ever compromised. You'll need to enter this passphrase when the key is first used by SSH (an SSH agent can cache it for your session). For fully automated backups without user interaction, you might use an unencrypted key if the private key file itself is extremely well protected, or use an SSH agent with a pre-loaded key.
- This creates two files: ~/.ssh/id_ed25519 (private key - KEEP SECRET!) and ~/.ssh/id_ed25519.pub (public key - safe to share).
On your Remote Server (the backup host):
- Log in to the remote server as the user who will own the Borg repository (e.g., borguser).
```
ssh borguser@your-remote-server.com
```
- Ensure the ~/.ssh directory exists and has correct permissions:
```
mkdir -p ~/.ssh
chmod 700 ~/.ssh
```
- Create or edit the ~/.ssh/authorized_keys file. This file lists all public keys that are authorized to log in as this user.
```
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```
- Now, copy the content of your client machine's public key (~/.ssh/id_ed25519.pub) into the remote server's ~/.ssh/authorized_keys file.
  - Method A (from client, if ssh-copy-id is available):
```
# On your client machine
ssh-copy-id -i ~/.ssh/id_ed25519.pub borguser@your-remote-server.com
```
    (If you used a custom key name, specify it with -i path/to/public_key.pub).
  - Method B (Manual copy):
    1. On your client machine, display the public key: cat ~/.ssh/id_ed25519.pub
    2. Copy the entire output line (starts with ssh-ed25519 ...).
    3. On the remote server, open ~/.ssh/authorized_keys with an editor (e.g., nano ~/.ssh/authorized_keys) and paste the copied public key as a new line. Save and exit.
Test the Connection (from Client):
- Try logging into the remote server from your client machine:
```
ssh borguser@your-remote-server.com
```
- If you set a passphrase for your SSH key, you'll be prompted for it now.
- If everything is set up correctly, you should log in without being asked for the borguser's password on the server.
- If it still asks for a password, troubleshoot SSH configuration (check permissions on ~/.ssh and authorized_keys on the server, ensure PubkeyAuthentication yes is in /etc/ssh/sshd_config on the server, and check SSH logs usually in /var/log/auth.log or journalctl -u sshd on the server).

Restricting SSH Key Access for Borg (command= option)

Allowing full shell access with an SSH key used for backups is a security risk. If the client machine (or that specific SSH key) is compromised, an attacker could log into your backup server and potentially delete or tamper with your backups.

To mitigate this, SSH's authorized_keys file allows you to restrict what command(s) can be run when a specific key is used for authentication. This is done by prefixing the public key entry with command="your_command_here".

For Borg, the remote server essentially needs to run the borg serve command when the client connects.

Example of a restricted authorized_keys entry on the server:

command="borg serve --restrict-to-path /path/to/borg_repo --append-only",restrict ssh-ed25519 AAAA...your...public...key...here... borg_backup_key

Let's break down the options for the command= string:

borg serve: This is the Borg command that runs on the server side to handle requests from a Borg client.
--restrict-to-path /path/to/borg_repo: Crucial security measure. This confines Borg operations initiated by this key to only the specified repository path. The client cannot use this key to access or create repositories elsewhere on the server. Replace /path/to/borg_repo with the actual absolute path to the directory where this user's Borg repository will be stored on the server (e.g., /home/borguser/backups/client_A_repo).
--append-only (Optional but recommended for added safety): This puts the repository into append-only mode for connections using this key. It means the client can create new backups and list archives, but cannot delete archives or compact the repository. Pruning would need to be done either by a different key without this restriction or locally on the server by a trusted process. This protects against a compromised client maliciously deleting backups.
restrict (after the command quotation, before the key type): This is an additional SSH option that applies a set of default restrictions: disables TTY allocation, X11 forwarding, agent forwarding, port forwarding, etc., further hardening the connection.

To implement this:

On the remote server, edit ~/.ssh/authorized_keys for borguser.
Find the line corresponding to the client's public key.
Prepend it with command="..." and the restrict option as shown above. Ensure there's a space between restrict and the key type (ssh-ed25519 or ssh-rsa).
- Example before: ssh-ed25519 AAAA... key_comment
- Example after: command="borg serve --restrict-to-path /srv/backups/myclient_repo",restrict ssh-ed25519 AAAA... key_comment
Save the file.
Now, if you try to SSH normally with this key (ssh borguser@your-remote-server.com), it should fail or immediately disconnect because you're no longer granted an interactive shell. This is expected and desired. Borg commands from the client should still work.

Initializing a Remote Borg Repository

Once SSH key auth (ideally restricted) is set up:

On the Remote Server: Create the directory that will house the repository (e.g., /srv/backups/myclient_repo). Ensure borguser owns it and has write permissions.

# As root or with sudo on the server
sudo mkdir -p /srv/backups/myclient_repo
sudo chown borguser:borguser /srv/backups/myclient_repo
sudo chmod 700 /srv/backups/myclient_repo # Restrict access to owner

On the Client Machine: Use borg init with the SSH URL.
```
borg init --encryption=repokey ssh://borguser@your-remote-server.com/srv/backups/myclient_repo
```
- Replace borguser@your-remote-server.com with your server details.
- Replace /srv/backups/myclient_repo with the exact path you used in --restrict-to-path (if you used that restriction) and created on the server.
- Borg will prompt for an encryption passphrase for the new repository. This passphrase is for the Borg repository's encryption, separate from your SSH key's passphrase.

Borgmatic Configuration for Remote Repositories

In your Borgmatic config.yaml on the client, you simply use the SSH URL for the repository path:

location:
    source_directories:
        - /home/your_local_user/Documents
    repositories:
        - ssh://borguser@your-remote-server.com/srv/backups/myclient_repo # SSH URL
# ... rest of your Borgmatic configuration ...

If your SSH key is not the default (~/.ssh/id_rsa or ~/.ssh/id_ed25519) or requires specific options, you might need to configure an entry in your client's ~/.ssh/config file:

Host mybackupserver
    HostName your-remote-server.com
    User borguser
    IdentityFile ~/.ssh/borg_specific_key
    # Add other options like Port if not standard 22

Then, in your Borgmatic config (and borg commands), you can use the alias: ssh://mybackupserver/srv/backups/myclient_repo

Workshop Configuring a Remote Backup Target

Scenario:
You want to back up your client machine (your current machine or a primary VM) to a "remote" server. For this workshop, the "remote server" can be:

A Virtual Machine (VM) running Linux on your host computer.
Another physical computer on your local network.
If you have access to a cloud VPS, you can use that too. The key is to simulate the client-server SSH interaction.

Project Goals:

Set up a "remote server" environment (VM or another machine).
Install BorgBackup and OpenSSH server on the "remote server".
Configure SSH key-based authentication from your client to the server.
Initialize a Borg repository on the "remote server" from your client machine.
Modify your Borgmatic config.yaml to use this remote repository.
Run Borgmatic to create a backup on the remote server.
(Bonus) Implement the command= restriction in authorized_keys on the server.

Prerequisites:

Your client machine with Borg and Borgmatic installed.
Access to set up a "remote server" (Linux VM recommended, e.g., using VirtualBox/VMware with a Debian or Ubuntu server image).
Network connectivity between client and "server".

Steps:

Part 1: Setting up the "Remote Server"

(Perform these steps on your chosen "remote server" machine/VM)

Install OS (if a new VM): Install a minimal Linux server OS (e.g., Ubuntu Server).

Update System and Install Prerequisites:

sudo apt update && sudo apt upgrade -y
sudo apt install openssh-server borgbackup -y

Create a Backup User:

sudo adduser borguser # Follow prompts to set password, etc.

Create Repository Directory (as root/sudo, then chown):
This is where the Borg repository will live on the server.

sudo mkdir -p /var/borg-repos/client1_repo
sudo chown borguser:borguser /var/borg-repos/client1_repo
sudo chmod 700 /var/borg-repos/client1_repo # Only borguser can access

Note: Using /var/borg-repos/ or /srv/borg-repos/ is a common convention.

Ensure SSH Server is Running:

sudo systemctl status ssh
# Should be active (running). If not, sudo systemctl enable --now ssh

Get Server's IP Address:
```
ip addr show
```
Note down the IP address of this server on your local network (e.g., 192.168.1.101). You'll need this for the client.

Part 2: Setting up SSH Key Authentication (Client -> Server)

On Your Client Machine:

Generate an SSH key pair if you don't have one you want to use:

ssh-keygen -t ed25519 -C "borg_client_$(whoami)@$(hostname)"
# Press Enter for default file location (~/.ssh/id_ed25519)
# Enter a strong passphrase for the key (recommended!)

Copy the public key to the borguser on the remote server (replace REMOTE_SERVER_IP):
```
ssh-copy-id -i ~/.ssh/id_ed25519.pub borguser@REMOTE_SERVER_IP
```
You'll be asked for borguser's password on the remote server for this one-time copy.

Test SSH Login (from Client):
```
ssh borguser@REMOTE_SERVER_IP
```
You should be prompted for your SSH key's passphrase (if you set one), and then log in without needing borguser's password. Type exit to log out.

Part 3: Initialize Remote Borg Repository (from Client)

On Your Client Machine:
Initialize the Borg repository on the remote server.
```
# Replace REMOTE_SERVER_IP and path if different
borg init --encryption=repokey ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/client1_repo
```
- You'll be prompted for your SSH key passphrase (if any).
- Then, you'll be prompted to set a new encryption passphrase for the Borg repository itself. Choose a strong one and save it securely. This is the passphrase Borgmatic will need.

Part 4: Configure Borgmatic for Remote Backup (on Client)

On Your Client Machine:

Activate your Borgmatic virtual environment (if using one):
```
source ~/borgmatic-env/bin/activate
```

Edit your Borgmatic config.yaml (e.g., ~/.config/borgmatic/config.yaml):

location:
    source_directories:
        - /home/yourlocaluser/my_sample_docs # Adjust yourlocaluser

    repositories:
        # Comment out or remove your old local repository path
        # - /home/yourlocaluser/borg_repo_local
        # Add the new remote repository path
        - ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/client1_repo # Adjust IP

storage:
    # Ensure BORG_PASSPHRASE will be set for the *Borg repository encryption*
    # encryption_passcommand: "cat /path/to/borg_repo_passphrase_file" # Recommended
    archive_name_format: '{hostname}-docs-{now:%Y-%m-%d-%H%M%S}'

retention:
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 3
    keep_minimum: 2 # Make sure this makes sense with other policies
    prefix: '{hostname}-docs-' # Ensure this matches your archive_name_format

# ... other settings like hooks, consistency can remain or be added ...

Crucially, ensure Borgmatic can get the Borg repository passphrase. For this workshop, set BORG_PASSPHRASE in your terminal to the Borg repository's encryption passphrase you just created in step 3.1.
```
# In your client terminal, where borgmatic-env is active:
export BORG_PASSPHRASE='your-NEW-borg-repo-passphrase' # The one for the remote repo
# Or use read -s BORG_PASSPHRASE ...
```

Test Borgmatic with the Remote Repository:
- Validate: borgmatic config validate
- Dry run: borgmatic --verbosity 1 --dry-run (You'll be prompted for SSH key passphrase if it's not cached by an agent).
- Actual backup: borgmatic --verbosity 1
If successful, a new archive will be created in /var/borg-repos/client1_repo on your remote server. You can verify by SSHing into the server and listing contents with sudo -u borguser borg list /var/borg-repos/client1_repo (will ask for Borg repo passphrase there).

Part 5: (Bonus) Implement SSH Command Restriction (on Server)

On the Remote Server:
- Edit borguser's authorized_keys file:
```
sudo -u borguser nano /home/borguser/.ssh/authorized_keys
```
- Find the line for your client's public key. It will look like ssh-ed25519 AAAA... comment.
- Prepend the command= restriction. Ensure the path in --restrict-to-path matches EXACTLY the repository path you are using.
```
command="borg serve --restrict-to-path /var/borg-repos/client1_repo",restrict ssh-ed25519 AAAA...your...public...key...here... borg_client_...
```
  (If you also want append-only, add --append-only within the quotes: command="borg serve --restrict-to-path /var/borg-repos/client1_repo --append-only",restrict ...)
- Save the file.
Test (from Client):
- Try to SSH normally: ssh borguser@REMOTE_SERVER_IP. It should fail to give you a shell.
- Run borgmatic --verbosity 1 again. It should still work for creating backups because borg serve is the allowed command.
- If you added --append-only on the server, try running borgmatic prune --verbosity 1 from the client. It should fail with an error indicating the repository is append-only, proving the restriction works. (You'd then need another way to prune, like a local cron job on the server, or a separate SSH key for administrative tasks without the append-only restriction).

You have now successfully configured Borgmatic to back up to a remote server using secure SSH key authentication, and optionally hardened it with command restrictions. This is a significant step towards a robust backup strategy. Remember to unset BORG_PASSPHRASE and deactivate your virtual environment on the client when done.

6. Advanced Borgmatic Configuration

Borgmatic's power extends far beyond simple backup and prune operations. Its YAML configuration allows for sophisticated setups involving multiple backup sets, custom actions via hooks, automated health checks, and fine-grained retention policies. Mastering these can tailor your backup strategy precisely to your needs.

Multiple Backup Configurations in a Single Borgmatic File

While you can have multiple separate Borgmatic configuration files (e.g., /etc/borgmatic/config.yaml, /etc/borgmatic/config_databases.yaml), Borgmatic also supports defining multiple, distinct backup configurations within a single YAML file. This is achieved by having a top-level list of configurations.

Each item in this list is a complete Borgmatic configuration block, similar to what you'd put in a standalone config.yaml.

Example config.yaml with multiple configurations:

# This is a list of configurations
-
    # Configuration 1: Backing up documents to a local and a remote repository
    location:
        source_directories:
            - /home/user/Documents
            - /home/user/Projects
        repositories:
            - /mnt/local_backup_drive/borg_main
            - ssh://borguser@remote_server/backup/borg_main
        exclude_patterns:
            - '*.tmp'
            - '**/node_modules/'
    storage:
        archive_name_format: '{hostname}-main-{now:%Y-%m-%d}'
        # Passphrase handled by BORG_PASSPHRASE or encryption_passcommand
        compression: zstd,3
    retention:
        prefix: '{hostname}-main-'
        keep_daily: 7
        keep_weekly: 4
        keep_monthly: 6
    hooks:
        before_backup:
            - echo "Starting main backup..."
        after_backup:
            - echo "Main backup complete."
    consistency:
        checks:
            - repository
            - archives
        check_last: 3

-
    # Configuration 2: Backing up system configuration files to the remote repository only
    location:
        source_directories:
            - /etc
            - /opt/mycustomapp/config
        repositories:
            # Only remote for this one
            - ssh://borguser@remote_server/backup/borg_system_configs
        exclude_from: /home/user/.config/borgmatic/system_excludes.txt
    storage:
        archive_name_format: '{hostname}-system-{now:%Y-%m-%d}'
        # Assumes same passphrase mechanism as above or a different one via passcommand for this repo
        compression: zstd,1
    retention:
        prefix: '{hostname}-system-'
        keep_daily: 14
        keep_weekly: 8
        keep_monthly: 12
    hooks:
        before_backup:
            - echo "Starting system config backup..."
        after_backup:
            - echo "System config backup complete."

When you run borgmatic (e.g., borgmatic create, borgmatic prune), it will process each configuration block in the list sequentially. This means before_backup hooks for the first block run, then its borg create, then its borg prune, then its after_backup, then its consistency checks. Then, the same sequence happens for the second configuration block, and so on.

Hooks: before_backup, after_backup, on_error

Hooks are shell commands or scripts that Borgmatic executes at specific points in its workflow. This is incredibly powerful for integrating backups with other system tasks.

before_backup: Executed before borg create runs for that configuration block.
- Use cases:
  - Dumping databases (e.g., mysqldump, pg_dump) to a file that is then included in source_directories.
  - Stopping services that need to be quiescent for a consistent backup.
  - Creating LVM snapshots.
  - Sending a "backup starting" notification.
after_backup: Executed after borg create and borg prune (and consistency checks if enabled for that block and run after create/prune) complete successfully for that configuration block.
- Use cases:
  - Removing temporary database dumps.
  - Restarting services that were stopped.
  - Deleting LVM snapshots.
  - Sending a "backup successful" notification.
on_error: Executed if any command within the Borgmatic workflow for that configuration block (including borg create, borg prune, borg check, or even other hooks) fails (returns a non-zero exit code).
- Use cases:
  - Sending a "backup failed" notification with error details.
  - Attempting cleanup actions.
  - Logging detailed error information to a custom file.

Hook Syntax:
Hooks are defined as a list of strings, where each string is a command to be executed.

hooks:
    before_backup:
        - /usr/local/bin/dump_database.sh
        - echo "Database dump complete. Starting Borg backup."
    after_backup:
        - rm /tmp/database_dump.sql
        - /usr/local/bin/send_success_email.sh
    on_error:
        - /usr/local/bin/send_failure_email.sh "Borgmatic backup failed for $(hostname)"

Important Considerations for Hooks:

Permissions: Ensure the user running Borgmatic has permission to execute the hook scripts/commands.
Environment: Hooks run in a shell environment. PATH and other environment variables might be minimal, especially when run via cron or systemd. Use absolute paths for executables in scripts, or set the PATH within your script.
Error Handling: A hook script should exit with a non-zero status if it fails. This will trigger Borgmatic's on_error hooks (if defined) and cause Borgmatic itself to report an error.
Idempotency: If possible, make your hook scripts idempotent (safe to run multiple times with the same outcome if a backup is retried).

Health Checks: borgmatic check and the consistency section

Ensuring your backups are valid and restorable is as important as creating them. Borg provides the borg check command for this, and Borgmatic can automate its execution.

borg check command:
- borg check REPOSITORY: Verifies the integrity of the repository metadata and the data chunks themselves. It checks for corruption, missing data, etc.
- borg check REPOSITORY::ARCHIVE_NAME: Checks a specific archive.
- --verify-data: This is a more intensive check that actually re-reads, decrypts, and decompresses all data chunks associated with the checked archives/repository. It's slower but provides a higher level of assurance. Highly recommended to run periodically.

Borgmatic consistency section: This section in config.yaml automates running borg check.

consistency:
    # List of checks to perform. Can be "repository", "archives", or both.
    checks:
        - repository  # Verifies repository structure and metadata.
        - archives    # Verifies integrity of archive metadata.

    # Which archives to check if "archives" is in checks.
    # "all" or "last" (mutually exclusive).
    # check_last: 3       # Check the N most recent archives.
    check_all: false    # Set to true to check all archives (can be slow).

    # If you only want to check archives matching a certain prefix (within the overall repo).
    # prefix: '{hostname}-main-'

    # Add the --verify-data flag to borg check. Recommended but slower.
    verify_data: true

Running Checks with Borgmatic:
- Checks defined in the consistency section are typically run after borg create and borg prune for each configuration block if consistency is defined within that block.
- You can also run checks manually for all configured repositories:
```
borgmatic check --verbosity 1
# To force --verify-data if not in config (or override config):
# borgmatic check --verbosity 1 --override consistency.verify_data=true
```

borgmatic rcreate --health-check (deprecated in favor of borgmatic init -- επίσης)

Older versions of Borgmatic had borgmatic rcreate --health-check for initializing a repository and immediately checking it. Newer Borgmatic versions integrate this into borgmatic init. When you run borgmatic init (e.g., borgmatic init --encryption repokey), it creates the repository and performs basic checks. The consistency section is for ongoing checks of existing repositories.

Advanced Retention Policies

The basic keep_daily, keep_weekly, etc., are powerful, but Borg (and thus Borgmatic via borg prune) offers more. The key command is borg prune, and Borgmatic's retention section maps directly to its options.

retention:
    # Standard time-based retention
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6
    keep_yearly: 1

    # Keep within a certain timeframe (e.g., all backups from last 30 days)
    # keep_within: 30d # 'H' for hours, 'd' for days, 'w' for weeks, 'm' for months, 'y' for years

    # Always keep at least N archives, regardless of other policies
    keep_minimum: 2

    # Pruning prefix (essential for shared repositories)
    # Borgmatic defaults this to the static part of archive_name_format if not set,
    # or just {hostname}- if archive_name_format is very dynamic.
    # Explicitly setting it is good practice.
    prefix: '{hostname}-main-'

    # You can also specify --glob-archives PATTERN for pruning,
    # although prefix is usually sufficient for Borgmatic's per-config pruning.
    # glob_archives: '*-database-*' # Example: prune only archives matching this glob

Borgmatic intelligently applies these: it first determines which archives to keep based on keep_daily/weekly/monthly/yearly/within, then ensures keep_minimum is met. Pruning only affects archives matching the prefix (or glob_archives if used).

Overriding Storage Quota Options for Pruning

Borg's prune command has a --storage-quota SIZE option (e.g., 100G, 1T). If set, Borg will prune archives (oldest first, respecting other keep options) until the repository size is below this quota. Borgmatic doesn't have a direct YAML key for this in retention, but you can pass arbitrary arguments to borg prune using the borg section if needed (less common for this specific flag, as retention policies are usually preferred).

A more direct approach for storage quotas is often managed at the filesystem level or by monitoring repository size and adjusting retention policies.

Logging and Monitoring Options in Borgmatic

Verbosity: Control with -v 0/1/2 or --verbosity 0/1/2.

Syslog: Borgmatic can log to syslog.

# In your config.yaml
monitoring:
    syslog_verbosity: 1 # 0 for less, 1 for normal, 2 for debug

Healthchecks.io / Cronitor.io / similar "dead man's switch" services: You can use hooks to ping these services. If the service doesn't receive a ping after a successful backup, it can alert you.

hooks:
    after_backup:
        - curl -fsS --retry 3 https://hc-ping.com/your-uuid-here
    on_error:
        - curl -fsS --retry 3 https://hc-ping.com/your-uuid-here/fail

Custom Logging in Hooks: Your hook scripts can write to custom log files.
Systemd Journal: When running Borgmatic via systemd (covered later), output is automatically captured by the systemd journal.

Workshop Implementing Pre-Backup Hooks and Health Checks

Scenario:
You want to enhance your Borgmatic setup for ~/my_sample_docs. You'll simulate dumping a "database" (a simple text file for this workshop) before the backup, and then configure Borgmatic to perform consistency checks.

Project Goals:

Create a simple shell script to act as a before_backup hook (simulating a database dump).
Configure this script in Borgmatic's hooks section.
Run Borgmatic and verify the hook executes and the "dump" is backed up.
Add an after_backup hook to clean up the "dump" file.
Configure Borgmatic's consistency section to check the repository and the last archive with data verification.
Implement a more complex retention policy.

Prerequisites:

Borgmatic installed and configured for ~/my_sample_docs (local or remote repo).
BORG_PASSPHRASE handling established for your Borgmatic runs.
Your borgmatic-env activated if you use one.

Steps:

Create a Simulated Database Dump Script (before_backup hook):

In your home directory (or a scripts directory), create simulate_db_dump.sh:
```
nano ~/simulate_db_dump.sh
```

Add the following content:

#!/bin/bash
# This script simulates a database dump.
DUMP_DIR="$HOME/my_sample_docs/db_dumps" # Dump inside a source_directory
DUMP_FILE="$DUMP_DIR/simulated_db_$(date +%Y%m%d_%H%M%S).sql"

echo "INFO: Starting simulated database dump to $DUMP_FILE"
mkdir -p "$DUMP_DIR"
echo "-- Simulated SQL Dump --" > "$DUMP_FILE"
echo "CREATE TABLE my_table (id INT, data VARCHAR(255));" >> "$DUMP_FILE"
echo "INSERT INTO my_table VALUES (1, 'Sample data at $(date)');" >> "$DUMP_FILE"
echo "INFO: Simulated database dump complete."

# Important: Exit with 0 on success
exit 0

Make it executable:
```
chmod +x ~/simulate_db_dump.sh
```

Create a Cleanup Script (after_backup hook):

Create cleanup_db_dump.sh:
```
nano ~/cleanup_db_dump.sh
```

Add the following content:

#!/bin/bash
# This script cleans up old simulated database dumps.
# It keeps the latest one for inspection, removes others.
DUMP_DIR="$HOME/my_sample_docs/db_dumps"

echo "INFO: Cleaning up old simulated database dumps in $DUMP_DIR..."
# Example: Remove all but the newest .sql file in the DUMP_DIR
# This is a simple example; real cleanup might be more complex.
# Ensure DUMP_DIR exists and is not empty before proceeding with complex find/rm
if [ -d "$DUMP_DIR" ]; then
    # List all .sql files, sort them, keep the last (newest), delete the rest.
    # The "head -n -1" part lists all but the last N files.
    # If only one file, ls ... | head -n -1 will be empty.
    find "$DUMP_DIR" -maxdepth 1 -type f -name "*.sql" -print0 | xargs -0 ls -t | tail -n +2 | xargs -I {} rm -- {}
    echo "INFO: Cleanup complete. Kept the latest dump (if any)."
else
    echo "INFO: Dump directory $DUMP_DIR not found, nothing to clean."
fi
exit 0

Make it executable:
```
chmod +x ~/cleanup_db_dump.sh
```
Note on Cleanup Script Logic:
The example cleanup script is basic. It attempts to delete all but the newest .sql file in the dump directory. For production, you'd want more robust logic, perhaps deleting files older than X days or ensuring a certain number are kept. Test such scripts carefully! For the workshop, we want the current dump to be backed up, then cleaned. The script above keeps the latest dump. A better after_backup hook might be to simply rm $DUMP_DIR/*.sql if the dumps are only needed for that specific backup instance. Let's simplify for the workshop: Modify cleanup_db_dump.sh to just remove all .sql files:
```
#!/bin/bash
DUMP_DIR="$HOME/my_sample_docs/db_dumps"
echo "INFO: Removing simulated database dumps from $DUMP_DIR"
rm -f "$DUMP_DIR"/*.sql
echo "INFO: Removal complete."
exit 0
```
Re-save and ensure it's executable. This revised script removes the dump after it has been backed up.

Update Borgmatic config.yaml for Hooks:

Edit your ~/.config/borgmatic/config.yaml. Add or modify the hooks section.
Also, ensure /home/youruser/my_sample_docs (which will contain db_dumps) is in source_directories.

# ... location and storage sections ...
location:
    source_directories:
        - /home/youruser/my_sample_docs # Ensure this path is correct for your setup
    # ... repositories ...

# ... storage ...

retention:
    # ... (we'll modify this in step 5) ...
    prefix: '{hostname}-docs-' # Or your existing prefix
    keep_daily: 3 # For easier observation of pruning later
    keep_minimum: 1

hooks:
    before_backup:
        - /home/youruser/simulate_db_dump.sh # Adjust path if you put it elsewhere
    after_backup:
        - /home/youruser/cleanup_db_dump.sh # Adjust path
    on_error:
        - echo "BORGMATIC ERROR: Backup or hook failed for my_sample_docs!" >&2

consistency:
    # Run checks after create/prune for this config block
    checks:
        - repository
        - archives
    check_last: 1    # Check the integrity of the most recent archive
    verify_data: true # Perform thorough data verification (slower but safer)

Replace youruser with your actual username.

Run Borgmatic and Verify Hooks and Backup:

Ensure BORG_PASSPHRASE is set for your repository.

Run Borgmatic:

# If using a virtual env, ensure it's active: source ~/borgmatic-env/bin/activate
borgmatic --verbosity 1

Observe the output:
- You should see the echo messages from your simulate_db_dump.sh script.
- Borgmatic should then create the backup.
- After the backup, you should see messages from cleanup_db_dump.sh.
- Then, consistency checks should run.

Verification:

Check your live ~/my_sample_docs/db_dumps/ directory. It should be empty (or not contain the .sql file from this run) because the after_backup hook cleaned it.

List the contents of the newly created archive to ensure the dump file was backed up:

# Get the latest archive name (adjust repo path and use your actual BORG_PASSPHRASE method)
LATEST_ARCHIVE=$(BORG_PASSPHRASE='your-repo-pass' borg list --short --sort-by timestamp ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/client1_repo | tail -n 1)
# Or for local repo:
# LATEST_ARCHIVE=$(BORG_PASSPHRASE='your-repo-pass' borg list --short --sort-by timestamp ~/borg_repo_local | tail -n 1)

echo "Latest archive: $LATEST_ARCHIVE"
BORG_PASSPHRASE='your-repo-pass' borg list ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/client1_repo::$LATEST_ARCHIVE | grep .sql
# Or for local:
# BORG_PASSPHRASE='your-repo-pass' borg list ~/borg_repo_local::$LATEST_ARCHIVE | grep .sql

You should see your simulated_db_....sql file listed in the archive.

Implement a More Complex Retention Policy:
- Modify the retention section in your config.yaml to something like this:
```
retention:
    prefix: '{hostname}-docs-' # Or your existing prefix
    keep_daily: 7           # Keep the last 7 daily backups
    keep_weekly: 4          # Keep 4 weekly backups (one per week for the last 4 weeks)
    keep_monthly: 3         # Keep 3 monthly backups
    keep_minimum: 2         # Always keep at least 2, regardless of other rules
```
- To observe this, you'd need to generate backups over a simulated period. For the workshop, simply run borgmatic --verbosity 1 several times (e.g., 5-10 times).
- After creating several backups, run:
```
borgmatic prune --verbosity 1
# Or just run 'borgmatic --verbosity 1' again, as pruning is part of the default flow.
```
  Observe the output. Borgmatic (via borg prune) will tell you which archives it's keeping and deleting based on your new policy. With the keep_daily: 7, keep_weekly: 4 etc., it might not prune much initially until you have archives spanning those timeframes. If you made only, say, 5 backups today, it will likely keep all of them due to keep_daily: 7 and keep_minimum: 2. The more complex policies show their effect over longer periods of backup history.

You've now successfully integrated pre-backup and post-backup actions using hooks, ensuring temporary data is included and then cleaned up. You've also configured automated health checks for your repository and archives, adding a layer of confidence in your backup integrity, and explored more nuanced retention policies. These advanced configurations make Borgmatic a very flexible and robust backup automation tool.

7. Security Considerations for Your Backup Server

Backups contain copies of your valuable, often sensitive, data. Therefore, the security of your backup server and the backup data itself is paramount. A compromised backup system can lead to data breaches, data loss (if backups are deleted or corrupted by an attacker), or ransomware holding your backups hostage. This section delves into critical security aspects.

Repository Encryption Deep Dive

BorgBackup's mandatory client-side encryption is a foundational security feature. This means data is encrypted before it leaves the client machine and is stored in an encrypted state in the repository.

Key Modes Revisited:
- repokey mode: The master encryption key is stored within the repository's config file, but this key itself is encrypted with your chosen passphrase.
  - Pros: Simpler key management if you only need to remember/manage one passphrase per repository. The key travels with the repository data (convenient for moving/copying the repo).
  - Cons: Security hinges on the strength of your passphrase. If an attacker gets the repository files and the passphrase, they can decrypt everything.
- keyfile mode: The master encryption key is stored in a separate file outside the repository (e.g., ~/.config/borg/keys/my_repo_key). This keyfile can optionally be passphrase-protected.
  - Pros: Stronger separation of key and data. An attacker needs both the repository files and the keyfile. If the keyfile is stored very securely (e.g., on a separate encrypted USB, or managed by a hardware security module if Borg ever supports it directly), this can be more secure.
  - Cons: You are solely responsible for the security and backup of this keyfile. If you lose the keyfile (and it wasn't passphrase protected, or you forget that passphrase too), all data in the repository is irrevocably lost. This cannot be overstated.
Passphrase Strength:
- For repokey mode or passphrase-protected keyfile mode, use a long, strong, unique passphrase.
- Consider using a reputable password manager to generate and store these passphrases.
- Avoid common words, dictionary attacks, or easily guessable patterns. Think 15+ characters, mixed case, numbers, symbols.
Securely Managing Encryption Keys/Passphrases:
- Password Managers: Tools like KeePassXC, Bitwarden, 1Password are excellent for storing complex passphrases.
- Physical Storage (for ultimate keyfile security): For extremely sensitive keyfiles, consider storing them on an encrypted USB drive, kept offline in a secure physical location (like a safe). Back up this USB drive!
- BORG_PASSPHRASE / BORG_PASSCOMMAND: When automating backups:
  - export BORG_PASSPHRASE='yourpass' in a script: Convenient but exposes the passphrase in the script file and potentially process list. The script file must be heavily permission-restricted (e.g., chmod 600).
  - BORG_PASSCOMMAND='cat /path/to/secure/passphrasefile': The passphrasefile should be readable only by the user running Borg (e.g., chmod 400). This is generally better than embedding in the script.
  - BORG_PASSCOMMAND='pass borg/myrepopass' (using pass password manager): A good option if you use pass.
  - Systemd credentials (covered later): Another mechanism for providing secrets to services.
Key Export/Backup (for repokey mode): Even with repokey mode, the key is inside the repository's config file (encrypted by your passphrase). If the config file itself gets corrupted, you might have issues. Borg allows you to export the key material:
```
# Assuming BORG_PASSPHRASE is set or you'll be prompted
borg key export /path/to/repo /path/to/exported_key_backup
```
Store this exported_key_backup file very securely and separately from the repository. You would use borg key import if needed. This is an extra layer of safety for the key material itself.

Securing the Backup Server Itself (OS Hardening, Firewall, SSH Hardening)

If your backup repository is on a remote server, that server becomes a critical piece of infrastructure.

Minimal Necessary Services:
- The backup server should run only the essential services required for its function (SSH, Borg, potentially monitoring agents).
- Disable or uninstall any unused services to reduce the attack surface.
Regular Updates: Keep the server's operating system and all software (especially SSH and Borg) patched and up-to-date to protect against known vulnerabilities.
Firewall:
- Implement a host-based firewall (e.g., ufw on Ubuntu, firewalld on Fedora/CentOS).
- Allow incoming connections only on the necessary ports (e.g., SSH port, typically 22, or a custom port if you've changed it). Deny all other incoming traffic by default.
- Example with ufw:
```
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh # Or 'sudo ufw allow 2222/tcp' if using custom port 2222
sudo ufw enable
```
SSH Hardening:
- Use key-based authentication only: Disable password authentication in /etc/ssh/sshd_config:
```
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
```
- Change default SSH port (optional, security through obscurity): While not a primary defense, changing from port 22 can reduce automated bot scans. If you do, remember to update firewall rules and client configurations. Port 2222 (in sshd_config)
- Limit users who can SSH: Use AllowUsers or AllowGroups in sshd_config. AllowUsers borguser adminuser
- Disable root login via SSH: PermitRootLogin no (or prohibit-password if root needs key access for specific tasks, but generally avoid).
- Use strong ciphers and MACs: Modern SSH versions have good defaults. You can review and harden these in sshd_config if needed (advanced topic).
- Regularly review SSH logs: (/var/log/auth.log or journalctl -u sshd) for suspicious activity.
- Install Fail2Ban: This tool monitors log files (like SSH logs) for malicious patterns (e.g., repeated failed login attempts) and temporarily or permanently bans the offending IP addresses by updating firewall rules.
```
sudo apt install fail2ban
# Basic configuration is often sufficient for SSH protection out of the box.
# Custom jails can be configured in /etc/fail2ban/jail.local.
```

Limiting Borg's Access on the Remote Server (Revisiting command= in authorized_keys)

This was covered in Section 5 but is critical for security: When using SSH key authentication for Borg, always restrict the key in the server's ~/.ssh/authorized_keys file:

command="borg serve --restrict-to-path /path/to/actual_repo --append-only",restrict ssh-ed25519 AAAA...key...comment

--restrict-to-path /absolute/path/to/repo: Prevents the client from accessing any other part of the server's filesystem via Borg.
--append-only: (Highly recommended for client keys) Prevents the client from deleting archives (borg delete, borg prune) or compacting the repository (borg compact). This protects against a compromised client destroying backups. Pruning would then need to be done by a separate, more privileged process on the server itself or via a different SSH key without this restriction, used only for administrative tasks.
restrict: SSH option that disables TTY, port forwarding, X11 forwarding, etc., further limiting the capabilities of the SSH session.

Append-Only Mode for Repositories

Borg repositories can be set globally (in their config file on the server) to be append-only. This is a stronger guarantee than just the SSH key restriction.

Enabling on the server:

# On the server, as the user owning the repository
# BORG_PASSPHRASE must be set for the repo
borg config /path/to/repo append_only true

Pros: Strong protection against accidental or malicious deletion from any client, regardless of their SSH key options (unless they can also run borg config on the server to disable it).
Cons: Pruning (borg prune) and compacting (borg compact) cannot be done remotely if this is set. These operations must be performed directly on the server by a process that can temporarily disable append-only mode (e.g., a local cron job running borg config ... append_only false; borg prune ...; borg config ... append_only true). This adds complexity to maintenance.

A common strategy is to use --append-only on the client's SSH key and run prune/compact jobs locally on the backup server via cron, using a script that has full access to the repository.

Regularly Testing Restore Procedures

This is arguably the most important security (and reliability) practice.
Backups are useless if they can't be restored.

Periodically (e.g., monthly or quarterly, or after major system changes) perform test restores of:
- Critical individual files.
- Entire directories.
- A full system restore (if applicable, to a test environment).
Verify the integrity and usability of the restored data.
This practice not only ensures your backups are working but also keeps you familiar with the restore process, which is crucial during a real emergency.

Physical Security of Backup Media

Onsite Backups: If you use external drives for local backups, store them securely when not in use. Disconnect them to protect against ransomware that might encrypt connected drives.
Offsite Backups:
- If using physical media for offsite storage (e.g., USB drives taken to another location), ensure that location is physically secure.
- Consider the security of the transport process.
- For remote backup servers (self-hosted or provider), ensure the data center or location has adequate physical security measures.

Air-Gapped Backups

For ultimate protection against online threats, consider an air-gapped backup. This means the backup media is physically disconnected from any network or computer most of the time.

Example: Backing up to an external hard drive that is then unplugged and stored securely.
This is highly effective against ransomware but makes automation more challenging (requires manual intervention to connect/disconnect).

Workshop Enhancing Backup Security

Scenario: You have a remote Borg repository set up. Now, you want to tighten its security by refining SSH key restrictions and discussing passphrase/key management strategies.

Project Goals:

Review and discuss the transition from passphrase-based repokey to keyfile mode for a test repository (conceptual understanding or actual execution on a test repo).
Practice exporting a repokey and discuss its secure storage.
On your "remote server" (VM or other machine), review its SSH configuration for best practices.
Implement a stricter command= line in authorized_keys on the server, including --append-only.
Discuss setting up a basic firewall (ufw) on the server.

Prerequisites:

A client machine and a "remote server" (VM) with Borg installed.
SSH key-based login from client to server is functional.
A Borg repository existing on the remote server, accessible by the client.
sudo access on the remote server.

Steps:

Passphrase vs. Keyfile Mode - Discussion and Optional Test:
- Discussion:
  - Currently, your remote repository likely uses repokey mode with a passphrase. Discuss the pros and cons vs. keyfile mode as outlined in the theory section.
  - When would keyfile mode be significantly more advantageous? (e.g., if the repository storage itself is on less trusted media, but the keyfile can be kept on highly secure, separate media).
  - What are the risks of keyfile mode? (Losing the keyfile = losing the data).
- (Optional) Experiment with keyfile on a new, temporary repository:
  - On client: mkdir ~/borg_keyfile_test_repo_source
  - On client, create a key: borg key export --paper ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/test_keyfile_repo ~/borg_test_keyfile (This is for repokey export actually. For keyfile init, Borg creates it.)
  - Let's initialize a new repo with keyfile mode:
```
# On client:
# First, create the directory on the server:
ssh borguser@REMOTE_SERVER_IP "mkdir -p /var/borg-repos/test_keyfile_repo"

# Now initialize with keyfile. Borg will create the key in ~/.config/borg/keys/
borg init --encryption=keyfile-blake2 ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/test_keyfile_repo
```
    Borg will tell you where it saved the keyfile (e.g., ~/.config/borg/keys/REMOTE_SERVER_IP_var_borg-repos_test_keyfile_repo).
    - Secure this keyfile!
    - Back it up separately.
    - If you move to another client machine, you'll need this keyfile to access the repo.
  - Try creating a backup: borg create ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/test_keyfile_repo::test_archive ~/borg_keyfile_test_repo_source (It should not ask for a passphrase for the repo itself, but might for your SSH key if it's protected).
  - Clean up (delete the test repo on server and local keyfile when done).
Exporting repokey for Backup:
- Assume your main remote repository (ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/client1_repo) uses repokey mode.
- On your client machine, export its key:
```
# Ensure BORG_PASSPHRASE is set to your main repo's passphrase, or you'll be prompted.
# export BORG_PASSPHRASE='your-main-repo-passphrase'
mkdir -p ~/borg_keys_backup
borg key export ssh://borguser@REMOTE_SERVER_IP/var/borg-repos/client1_repo ~/borg_keys_backup/client1_repo.keybackup
```
- client1_repo.keybackup now contains the repository's master key, still encrypted by your original passphrase.
- Discussion: How would you securely store client1_repo.keybackup? (e.g., on an encrypted USB drive, stored in a different physical location, in a password manager's secure notes if small enough). Why is this backup useful? (Protects against corruption of the config file within the repo itself).
Review SSH Server Configuration (on Remote Server):
- Log in to your remote server.
- Open /etc/ssh/sshd_config with an editor (e.g., sudo nano /etc/ssh/sshd_config).
- Check for these recommended settings (uncomment or change if necessary):
```
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no # Usually paired with PasswordAuthentication no
PermitEmptyPasswords no

# Optional:
# Port 2222 (if you changed from default 22)
# AllowUsers borguser youradminuser
# LogLevel VERBOSE (for more detailed logs during troubleshooting)
```
- If you made changes, save the file and restart the SSH service:
```
sudo systemctl restart sshd
```
  Be careful: A mistake in sshd_config could lock you out. Always test login from another terminal before disconnecting your current session if making significant changes.
Implement Stricter command= in authorized_keys (on Remote Server):
- Edit borguser's authorized_keys file:
```
sudo -u borguser nano /home/borguser/.ssh/authorized_keys
```
- Modify the line for your client's key to include --append-only and ensure --restrict-to-path points to the correct absolute path of the repository. Original might be: command="borg serve --restrict-to-path /var/borg-repos/client1_repo",restrict ssh-ed25519 AAAA... Change to:
```
command="borg serve --restrict-to-path /var/borg-repos/client1_repo --append-only",restrict ssh-ed25519 AAAA...your...public...key...here... client_key_comment
```
- Save the file.
- Test from Client:
  - Try to run a backup with Borgmatic: borgmatic -v 1 (should work).
  - Try to prune from the client: borgmatic prune -v 1. This should now fail with an error message indicating the repository is append-only or the operation is not permitted. This confirms the restriction is working.
  - Discussion: How would you now prune this repository? (Answer: A cron job or systemd timer on the server itself, running borg prune directly on the local repository path, or using a different SSH key for admin tasks that doesn't have the --append-only restriction).
Discuss and (Optional) Implement Basic Firewall (ufw on Remote Server):
- Discussion: Why is a firewall important? (Defense in depth, blocks unwanted network access).
- (Optional) Implement ufw on the remote server (if it's a Linux VM/server):
  - Install ufw: sudo apt update && sudo apt install ufw -y
  - Set defaults:
```
sudo ufw default deny incoming
sudo ufw default allow outgoing
```
  - Allow SSH (use your actual SSH port if not 22):
```
sudo ufw allow ssh  # Or 'sudo ufw allow 2222/tcp' if port is 2222
```
  - Enable ufw:
```
sudo ufw enable
# It will warn that this may disrupt existing ssh connections. Type 'y'.
```
  - Check status: sudo ufw status verbose
  - Test: Ensure you can still SSH into the server from your client. If not, you might have blocked SSH. sudo ufw disable temporarily if you get locked out, fix the rule, then re-enable.
- Discussion: What if Borgmatic needed to connect to other services from the backup server (e.g., for healthcheck pings via curl in a hook)? (The default allow outgoing usually covers this. If outgoing was also denied, you'd need specific allow out rules).

This workshop has emphasized several layers of security: strong encryption key management, hardening the SSH server, restricting SSH key capabilities significantly, and network firewalling. Regularly reviewing and testing these security measures is crucial for maintaining a trustworthy backup system.

8. Advanced Borg Features and Techniques

Beyond the daily tasks of creating, restoring, and pruning backups, BorgBackup offers a suite of advanced commands and features for repository maintenance, in-depth analysis, and handling specific data types. Understanding these can help you optimize storage, troubleshoot issues, and manage your backups more effectively.

borg compact Reclaiming Space in the Repository

When you delete archives using borg prune or borg delete, Borg marks the data chunks exclusively used by those deleted archives as "unreferenced." However, it doesn't immediately remove them from the repository or reduce the repository's disk footprint. The borg compact command is used to free up this space.

How it works: borg compact rewrites segments of the repository, removing unreferenced chunks and compacting the remaining data.

Syntax:

borg compact REPOSITORY_OR_ARCHIVE

Or, more commonly for the whole repository:

borg compact /path/to/your/repo
# Or for remote:
# borg compact ssh://user@server/path/to/repo

When to use it:
- After significant pruning or deletion of archives, especially if they contained a lot of unique data.
- Periodically, as part of repository maintenance (e.g., monthly or quarterly).
- It can be I/O intensive and may take time on large repositories.
Options:
- --progress: Shows progress.
- --verbose: More detailed output.
- --cleanup-commits: (Advanced) Can remove old transaction commit tags from the repository index, potentially freeing a small amount of additional space. This is usually safe but makes it harder to roll back to a very old repository state if there was index corruption. Borg now does some of this automatically.

borg check --repair Attempting to Repair a Corrupted Repository

This is a last-resort command. If borg check reports errors (e.g., "segment inconsistency," "missing chunks"), your repository might be corrupted.

Syntax:

borg check --repair REPOSITORY_OR_ARCHIVE

What it does: It attempts to fix inconsistencies. This might involve:
- Rebuilding the repository index from segment data.
- Removing references to missing or corrupted chunks, which means data loss. Archives that relied on those chunks might become incomplete or unrestorable.
Caveats - VERY IMPORTANT:
- ALWAYS back up your repository directory before running --repair if possible. If the repair makes things worse or causes unacceptable data loss, you might want to revert.
- --repair is not magic. It cannot recover lost data. Its goal is to bring the repository back to a consistent state, even if that means acknowledging data loss.
- Understand what caused the corruption (hardware issues, unsafe shutdowns, software bugs) to prevent recurrence.
When to use: Only after borg check (without --repair) indicates problems and you've exhausted other options or are prepared for potential data loss from affected archives.

borg diff ::archive1 ::archive2 Comparing Archive Contents

The borg diff command allows you to see what changed between two archives in a repository. It shows files that were added, deleted, or modified.

Syntax:
```
borg diff REPOSITORY::ARCHIVE1 REPOSITORY::ARCHIVE2 [PATH]
```
Or if the repository is implied by context (e.g., after cding to a repo dir or setting BORG_REPO):
```
borg diff ::ARCHIVE1 ::ARCHIVE2
```
Output:
- Lines starting with + indicate added files/directories.
- Lines starting with - indicate removed files/directories.
- Lines starting with m indicate modified files (metadata like permissions or timestamps changed, or content changed).
- It shows changes in file content (by comparing chunk IDs), permissions, ownership, and timestamps.
Use cases:
- Understanding why a backup is larger or smaller than expected.
- Tracking changes to specific files or directories over time.
- Verifying if a particular change was indeed backed up.

borg delete Deleting Specific Archives or Entire Repositories

While borg prune is used for automated, policy-based deletion of old archives, borg delete gives you manual control.

Deleting Specific Archives:
```
borg delete REPOSITORY::ARCHIVE_NAME_TO_DELETE [REPOSITORY::ANOTHER_ARCHIVE ...]
```
- --stats: Shows statistics about reclaimed space (potential, before borg compact).
- --dry-run or -n: See what would be deleted without actually doing it.
- Use with caution! Deleted archives are generally not recoverable unless you have other backups of the repository itself.
Deleting an Entire Repository: This is more than just deleting archives; it removes the entire repository structure.
```
borg delete REPOSITORY
```
- This command is highly destructive. It will ask for confirmation.
- It's equivalent to rm -rf /path/to/repo_directory but with Borg's safety checks (like passphrase confirmation).
- Useful if you want to completely decommission a repository.

Understanding Deduplication in Depth

Borg's deduplication is a core strength. It works by breaking files into variable-sized chunks using a rolling hash algorithm (typically Buzhash).

Chunking:
- As Borg reads a file, it calculates a hash of a sliding window of data. When the hash meets certain criteria (e.g., some bits are zero), it marks the end of a chunk.
- This results in variable-sized chunks. The advantage is that if you insert or delete data in the middle of a large file, only the chunks around the modification (and the new data itself) are likely to change. Chunks after the modification often remain identical, leading to excellent deduplication even for files that shift content.
- Fixed-size chunking (used by some other tools) would cause all subsequent chunks to change after an insertion/deletion, leading to poorer deduplication for such files.
Chunk ID and Storage:
- For each chunk, Borg calculates a strong cryptographic hash (e.g., SHA256, BLAKE2b) which serves as its unique ID.
- The repository stores a mapping of these chunk IDs to the actual (compressed, encrypted) chunk data.
- When backing up, if Borg encounters a chunk whose ID is already in the repository, it simply stores a reference to the existing chunk instead of storing the data again.
The chunks Cache (~/.cache/borg/ or BORG_CACHE_DIR):
- To speed up deduplication, Borg maintains a local cache on the client machine. This cache stores the IDs and some metadata of chunks already known to be in the repository.
- When creating a backup, Borg first checks this local cache. If a chunk ID is found, Borg assumes the chunk is in the repository (unless --force-check-known-chunks is used) and doesn't need to send its data. This significantly speeds up backups to remote repositories by reducing data transfer.
- If the cache gets corrupted or is missing, backups will still work correctly but might be slower as Borg has to query the repository more often to see if chunks exist. You can safely delete the cache directory; Borg will recreate it on the next run.
- The cache can grow quite large for repositories with many unique chunks. You can control its location with BORG_CACHE_DIR.
Performance Implications:
- Deduplication is CPU-intensive (hashing, chunking).
- A fast CPU helps. Compression also adds CPU load.
- I/O speed (reading source files, writing to repository or local cache) is also a factor.
- Network speed is critical for remote repositories.

borg benchmark cpu CHUNKERALGO [ENCRYPTIONALGO [COMPRESSIONALGO]] Borg 1.2.x introduced this command to benchmark the performance of different chunking, encryption, and compression algorithm combinations on your specific hardware. This can help you choose the optimal settings for your performance needs.

Example:

borg benchmark cpu buzhash blake2,chacha20,poly1305 zstd,3

This will test the Buzhash chunker, BLAKE2b MAC with ChaCha20/Poly1305 encryption, and Zstd level 3 compression. It reports throughput (MiB/s).

Using Borg with Different Filesystems and Their Quirks

Borg aims to preserve as much metadata as possible, but filesystems have differences.

Permissions, Ownership, Timestamps: Borg generally handles standard Unix permissions (read/write/execute for user/group/other), user/group IDs, and modification/access times well.
ACLs (Access Control Lists):
- If your source filesystem uses POSIX ACLs (common on Linux for fine-grained permissions), you need to tell Borg to back them up using the --acls flag during borg create.
```
borg create --acls ...
```
- Restoring ACLs also requires appropriate support on the target filesystem and often root privileges.
- Not all systems/filesystems support ACLs or support them identically.
Extended Attributes (xattrs):
- xattrs store additional metadata (e.g., SELinux labels, user comments, user.mime_type). To back these up, use --xattrs.
```
borg create --xattrs ...
```
- Similar to ACLs, restoring xattrs depends on target filesystem support.
Special Files: Borg handles symlinks, hardlinks, and device files.
Filesystem-Specific Issues:
- Case sensitivity: Backing up from a case-sensitive filesystem (Linux ext4) to a repository that might be restored to a case-insensitive filesystem (Windows NTFS, macOS HFS+ by default) can cause issues if filenames differ only by case (e.g., File.txt and file.txt). Borg stores paths as they are.
- Path length limits: Very long paths might hit limits on certain filesystems.
- Character sets: Non-UTF-8 filenames could be problematic, though Borg generally tries to handle them. UTF-8 is standard on modern Linux.

Always test restores to your intended target systems/filesystems if you rely heavily on specific metadata like ACLs or xattrs.

Workshop Optimizing and Managing Repository Space

Scenario: You have a Borg repository that has seen several backups, and some archives have been pruned. You'll now explore compacting the repository and using borg diff.

Project Goals:

Create several similar backups to generate deduplicatable data and simulate some history.
Examine repository information and size (borg info, du -sh).
Delete a few older archives manually using borg delete.
Run borg compact and observe its effect (if any significant change in size).
Use borg diff to compare two slightly different archives.
(Conceptual) Discuss when borg check --repair would be considered.

Prerequisites:

An existing Borg repository (local or remote) with some archives. Let's use the local ~/borg_repo_local for simplicity.
BORG_PASSPHRASE handling established.
my_sample_docs folder for creating new backups.

Steps:

Generate More Backup History and Deduplicatable Data:
- Ensure your borgmatic-env is active if you use one and BORG_PASSPHRASE is set.
- Modify a file in ~/my_sample_docs, e.g., add a line to ~/my_sample_docs/report.txt.
- Create a new backup using borg create (or borgmatic create via its config):
```
# Using direct borg create for quick iteration
borg create --stats ~/borg_repo_local::data-v1-{now:%Y%m%d%H%M%S} ~/my_sample_docs
```
- Repeat this process 2-3 more times, making small changes to files in ~/my_sample_docs each time before creating a new archive. This will ensure there's data to deduplicate and some history.
Examine Repository Information:
- Get overall repository info:
```
borg info ~/borg_repo_local
```
  Note the "Deduplicated size" (total unique data) and "Compressed size." Also, see "Total size of all archives."
- Check the actual disk usage of the repository directory:
```
du -sh ~/borg_repo_local
```
  This disk usage should be closer to the "Deduplicated size (compressed)" reported by borg info than the "Total size of all archives (compressed)" because of deduplication.
Delete Some Archives Manually:
- List your archives:
```
borg list ~/borg_repo_local
```
- Identify 2-3 older archives you want to delete. Choose archives that are not your very latest ones. Note their exact names.
- Delete them using borg delete:
```
# Replace with your actual archive names
borg delete --stats ~/borg_repo_local::archive-name-1 ::archive-name-2
```
  The --stats option will show "Deleted data size" (this is an estimate of unique data freed by these deletions).
- Run borg info ~/borg_repo_local again. You might see "Space reclaims suggested" or notice that "Deduplicated size" hasn't changed much yet, but the number of archives has.
- Check du -sh ~/borg_repo_local again. The size might not have decreased significantly yet, as the space is marked for reclamation but not yet freed.
Run borg compact:
- Now, compact the repository:
```
borg compact --progress --verbose ~/borg_repo_local
```
- This might take some time depending on the repository size and the amount of space to reclaim.
- After it finishes, check du -sh ~/borg_repo_local again. You should now see a reduction in disk usage if borg delete freed up unique chunks.
- Run borg info ~/borg_repo_local one more time. The "Deduplicated size" should now more accurately reflect the currently referenced data.
Use borg diff to Compare Archives:
- List your archives again with borg list ~/borg_repo_local and pick two archives that you know have some differences (e.g., one from before you modified report.txt and one after).
- Let's say you have data-v1- iniziale and data-v1-modified.
- Run borg diff:
```
borg diff ~/borg_repo_local::data-v1-iniziale ::data-v1-modified
# Replace with your actual archive names
```
- Examine the output. You should see the file(s) you changed listed with an m (modified) or perhaps + or - if you added/deleted files between those backups.
  - Example output for a modified file:
```
m my_sample_docs/report.txt
```
- Try diffing two archives that should be identical (if you ran a backup twice without changing source files). The diff should be empty.
Conceptual Discussion: borg check --repair:
- When would you consider using borg check --repair?
  - Only if borg check (without --repair) reports errors.
  - If you suspect repository corruption due to:
    - Unreliable storage media.
    - Sudden power loss or system crash during a write operation to the repo.
    - Filesystem corruption on the drive hosting the repo.
- What are the risks?
  - Potential data loss from affected archives. --repair tries to make the repo consistent, not necessarily recover all data.
- What precaution should you take before running it?
  - Back up the entire corrupted repository directory to another location if at all possible. This gives you a fallback if --repair makes things worse or if you want to try other recovery methods on the original copy.

You've now practiced key repository maintenance tasks: reclaiming space with borg compact, identifying changes with borg diff, and manually deleting archives. These tools give you finer control over your Borg repositories beyond automated pruning.

9. Integrating BorgBackup with System Services (systemd)

For reliable, automated backups on Linux systems, integrating Borgmatic (which wraps BorgBackup) with systemd is a robust approach. systemd is the init system and service manager used by most modern Linux distributions. It provides fine-grained control over starting, stopping, and scheduling services, along with excellent logging capabilities.

We'll create two types of systemd units:

Service Unit (.service): Defines how to run Borgmatic as a service. This includes specifying the command, user, environment variables (like BORG_PASSPHRASE_COMMAND), and dependencies.
Timer Unit (.timer): Defines when and how often the corresponding service unit should be started. This replaces traditional cron jobs for scheduling.

Creating systemd Service Units for Borgmatic (borgmatic.service)

A systemd service unit for Borgmatic typically looks like this. You would save it in /etc/systemd/system/borgmatic.service.

[Unit]
Description=Borgmatic backup service
Documentation=man:borgmatic(1) https://torsion.org/borgmatic/
# Ensures network is up if backing up to remote target or using network for passcommand
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot

# User to run Borgmatic as. Choose a non-root user if possible,
# but ensure this user can read all source_directories and access BORG_PASSPHRASE.
User=your_backup_user # Or root if backing up system-wide files not readable by a normal user

# Group for the process
# Group=your_backup_group

# --- IMPORTANT: Passphrase Handling ---
# Option 1: Using BORG_PASSPHRASE_COMMAND (Recommended)
# Create a script that outputs the passphrase, e.g., /usr/local/bin/borg-passphrase.sh
# Ensure this script is executable and VERY securely permissioned (e.g., chmod 700, owned by User).
# Environment=BORG_PASSPHRASE_COMMAND=/usr/local/bin/borg-passphrase.sh

# Option 2: Using systemd's LoadCredential / SetCredential (More complex but very secure)
# This involves systemd managing the secret. See systemd.exec(5) man page.
# Example (simplified concept - actual implementation needs more setup):
# LoadCredential=borg_repo_passphrase:/path/to/secure/passphrase_file
# ExecStart=/usr/local/bin/borgmatic --verbosity 1 --syslog-verbosity 1 --override storage.encryption_passcommand="cat ${CREDENTIALS_DIRECTORY}/borg_repo_passphrase"

# Option 3: Using an EnvironmentFile (Less secure than BORG_PASSPHRASE_COMMAND if file is just plaintext)
# Create a file, e.g., /etc/borgmatic/borg_env with BORG_PASSPHRASE="yourpass"
# Ensure this file is chmod 600 and owned by User.
# EnvironmentFile=/etc/borgmatic/borg_env

# For this example, let's assume BORG_PASSPHRASE_COMMAND using a script:
Environment=BORG_PASSPHRASE_COMMAND=/etc/borgmatic/scripts/get_borg_pass.sh
# And ensure borgmatic's config uses it or expects BORG_PASSPHRASE to be set by such a command.
# Or, if borgmatic config.yaml has `encryption_passcommand` set, you might not need it here.
# Check your Borgmatic config; this service unit should COMPLEMENT it.

# Path to borgmatic executable (adjust if installed in a venv or different location)
# If installed in a venv for 'your_backup_user' at ~your_backup_user/borgmatic-env:
# ExecStart=/home/your_backup_user/borgmatic-env/bin/borgmatic --verbosity 1 --syslog-verbosity 1
# If installed system-wide (e.g., via pip install --user or system package manager):
ExecStart=/usr/local/bin/borgmatic --verbosity 1 --syslog-verbosity 1
# Add --config /path/to/config.yaml if not using default locations.

# Optional: Resource control
# CPUQuota=50%
# IOSchedulingClass=best-effort
# IOSchedulingPriority=7

[Install]
WantedBy=multi-user.target

Explanation of Key Parts:

[Unit] Section:
- Description: Human-readable description.
- Documentation: Links to relevant documentation.
- After=network-online.target, Wants=network-online.target: Ensures the network is fully up before this service runs, crucial if your repository is remote or your BORG_PASSPHRASE_COMMAND uses the network (e.g., fetching from Vault).
[Service] Section:
- Type=oneshot: Borgmatic runs, does its job, and then exits. This type is suitable for tasks that perform a single action.
- User: Specifies the system user under which Borgmatic will run. This user needs:
  - Read access to all source_directories defined in Borgmatic's config.
  - Write access to the Borgmatic cache directory (~/.cache/borgmatic/ or ~user/.cache/borg/ for Borg's cache).
  - Access to the Borg repository (e.g., SSH keys configured if remote).
  - Ability to execute BORG_PASSPHRASE_COMMAND or read the EnvironmentFile/LoadCredential if used.
- Environment=BORG_PASSPHRASE_COMMAND=...: This is a common and secure way to provide the passphrase.
  - Create a script (e.g., /etc/borgmatic/scripts/get_borg_pass.sh) that simply echoes the passphrase.
  - This script must be owned by the User (or root) and have very strict permissions (e.g., chmod 500 or chmod 700 allowing only owner execute/read).
  - Example get_borg_pass.sh:
```
#!/bin/sh
echo "your-actual-borg-repo-passphrase"
```
- EnvironmentFile=...: An alternative where a file contains KEY=VALUE pairs. The file itself must be secured.
- ExecStart: The actual command to run.
  - Use the full path to the borgmatic executable. If installed in a Python virtual environment, point to the borgmatic inside that venv's bin/ directory.
  - --verbosity 1 and --syslog-verbosity 1 are good defaults for logging to the systemd journal.
  - If your Borgmatic config file is not at a default location (/etc/borgmatic/config.yaml or ~User/.config/borgmatic/config.yaml), specify it with --config /path/to/your/config.yaml.
[Install] Section:
- WantedBy=multi-user.target: Makes the service part of the standard multi-user system startup, though it's the timer that will actually trigger it regularly.

Creating systemd Timer Units for Scheduled Backups (borgmatic.timer)

A timer unit defines the schedule for running the associated service unit. Save this as /etc/systemd/system/borgmatic.timer. The timer unit must have the same base name as the service unit it controls (e.g., borgmatic.service is controlled by borgmatic.timer).

[Unit]
Description=Daily Borgmatic backup timer
# If the service unit has After/Wants network-online.target, this timer implicitly depends on it too.

[Timer]
# Define when to run the backup.
# OnCalendar=daily                                  # Runs once a day at midnight
# OnCalendar=*-*-* 02:00:00                         # Runs daily at 2 AM
OnCalendar=*-*-* 02,14:00:00                      # Runs daily at 2 AM and 2 PM
# OnCalendar=Sun *-*-* 03:00:00                     # Runs weekly on Sunday at 3 AM

# Optional: Add some randomized delay to avoid thundering herd problem if many timers fire at once.
RandomizedDelaySec=15min                          # Waits a random time between 0 and 15 minutes

# Optional: Wake the system from suspend if it's asleep at the scheduled time.
# WakeSystem=false                                # Set to true if needed, requires ACPI wake support

# Optional: Run job if missed due to system being off.
Persistent=true                                   # If true, the service unit will be run once
                                                  # as soon as possible after the timer missed its last start time.

Unit=borgmatic.service                            # Specifies the service unit to activate

[Install]
WantedBy=timers.target                            # Enables the timer to be started at boot

Explanation of Key Parts:

[Unit] Section:
- Description: Human-readable description.
[Timer] Section:
- OnCalendar: This is the most flexible way to specify schedules. It uses a format similar to cron but with more expressive power. See man systemd.time for details.
  - daily is equivalent to *-*-* 00:00:00.
  - weekly is Mon *-*-* 00:00:00.
  - *-*-* HH:MM:SS means daily at that specific time.
  - You can list multiple times: *-*-* 02,14:00:00 runs at 2 AM and 2 PM.
- RandomizedDelaySec: Spreads out the execution of timers that have the same OnCalendar value. Good practice for system stability.
- Persistent=true: If the machine was off when the timer was due, the job will run as soon as possible after the next boot (or when the timer is next checked).
- Unit=borgmatic.service: Links this timer to the service unit we created earlier.
[Install] Section:
- WantedBy=timers.target: Ensures this timer is started when timers.target is activated (usually at boot).

Managing Borgmatic Services and Timers

After creating these unit files:

Reload systemd daemon: Tell systemd to pick up the new/changed unit files.
```
sudo systemctl daemon-reload
```
Enable the timer: This makes the timer start automatically at boot. It does not start it immediately.
```
sudo systemctl enable borgmatic.timer
```
Start the timer: This activates the timer, and it will begin its schedule.
```
sudo systemctl start borgmatic.timer
```
Check the status of the timer and service:
- Timer status:
```
sudo systemctl status borgmatic.timer
```
  This shows when the timer last ran (Last Triggered) and when it's next due (Next Elapses).
- Service status (to see if it ran recently):
```
sudo systemctl status borgmatic.service
```
Manually trigger the service (for testing): If you want to run the backup job immediately without waiting for the timer:
```
sudo systemctl start borgmatic.service
```
Then check its status and logs.
View logs with journalctl: Systemd captures stdout and stderr from services into the system journal.
- View logs for the borgmatic.service:
```
sudo journalctl -u borgmatic.service
```
- Follow logs in real-time:
```
sudo journalctl -f -u borgmatic.service
```
- Show logs since a certain time:
```
sudo journalctl -u borgmatic.service --since "1 hour ago"
```

Handling Passphrase Prompts (Security Considerations)

As seen in the borgmatic.service example, Environment=BORG_PASSPHRASE_COMMAND=/path/to/script is a good method.

The Script (/etc/borgmatic/scripts/get_borg_pass.sh):
- Must be executable: sudo chmod +x /etc/borgmatic/scripts/get_borg_pass.sh
- Must be owned by root or the user running the service: sudo chown root:root /etc/borgmatic/scripts/get_borg_pass.sh (if service User=root) or sudo chown your_backup_user:your_backup_user ... (if service User=your_backup_user).
- Permissions must be very strict: sudo chmod 500 /etc/borgmatic/scripts/get_borg_pass.sh (read and execute only for owner) or even sudo chmod 700 (read, write, execute for owner only). No group or other access.
- The script simply contains:
```
#!/bin/sh
echo "your-ACTUAL-borg-repository-passphrase"
```
Security Implications: The passphrase is still in plaintext in this script file. The security relies entirely on the script's file permissions restricting access. This is generally considered acceptable if permissions are correct, as only root (or the designated user) can read/execute it.
Alternative: systemd credentials: A more advanced method where systemd itself can manage encrypted credentials. This involves systemd-creds encrypt /path/to/secret /path/to/encrypted-secret and then using LoadCredential= in the service unit. The kernel keyring is often used. This avoids having plaintext passphrases on disk accessible even to root after initial setup. This is more complex to set up.
Never embed passphrases directly in ExecStart= or Environment= within the systemd unit file itself if it's not BORG_PASSPHRASE_COMMAND.

Workshop Scheduling Borgmatic with systemd

Scenario: You will take your existing Borgmatic setup (local or remote repository) and schedule it to run automatically using systemd service and timer units.

Project Goals:

Create a borgmatic.service unit file.
Create a secure passphrase script for BORG_PASSPHRASE_COMMAND.
Create a borgmatic.timer unit file for a daily backup.
Enable and start the timer.
Manually trigger the service to test the setup.
Check logs using journalctl.

Prerequisites:

A working Borgmatic setup (config file exists, repository initialized, passphrase known). Let's assume Borgmatic is installed system-wide or in a known venv path, and its config is at /etc/borgmatic/config.yaml or ~youruser/.config/borgmatic/config.yaml.
sudo privileges on the machine where Borgmatic runs.
This workshop assumes Borgmatic will run as your current user, or a dedicated backupuser. If running as current user, ensure paths in service file reflect this (e.g., path to borgmatic venv, path to config if in home dir).

Steps:

Prepare Passphrase Script:
- Decide which user will run the Borgmatic service (e.g., youruser or a dedicated backupuser). For simplicity, let's assume youruser. If you used root for backups previously, you might continue with that, but generally, a dedicated user is better if possible.
- Create a directory for Borgmatic systemd helper scripts (if it doesn't exist):
```
sudo mkdir -p /etc/borgmatic/scripts
```
- Create the passphrase script (e.g., /etc/borgmatic/scripts/get_borg_pass.sh):
```
sudo nano /etc/borgmatic/scripts/get_borg_pass.sh
```
- Add your Borg repository passphrase to it:
```
#!/bin/sh
# IMPORTANT: Replace with your actual Borg repository passphrase!
echo "your-borg-repository-passphrase"
```
- Save and close.
- Set strict permissions and ownership. If borgmatic.service will run as youruser:
```
sudo chown youruser:youruser /etc/borgmatic/scripts/get_borg_pass.sh
sudo chmod 500 /etc/borgmatic/scripts/get_borg_pass.sh # Read/Execute for owner only
```
  If running as root:
```
sudo chown root:root /etc/borgmatic/scripts/get_borg_pass.sh
sudo chmod 500 /etc/borgmatic/scripts/get_borg_pass.sh
```

Create borgmatic.service File:

Create the service file:

sudo nano /etc/systemd/system/borgmatic.service

Paste the following, adjusting User, ExecStart path, and --config path as needed for your setup:

[Unit]
Description=Borgmatic backup service (scheduled)
Documentation=man:borgmatic(1) https://torsion.org/borgmatic/
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot

# --- ADJUST THESE ---
User=youruser # Replace 'youruser' with the user running the backup
              # Or 'root' if necessary.
# If borgmatic installed in a venv for 'youruser':
# ExecStart=/home/youruser/borgmatic-env/bin/borgmatic --verbosity 1 --syslog-verbosity 1 --config /home/youruser/.config/borgmatic/config.yaml
# If borgmatic installed system-wide and config is at /etc/borgmatic/config.yaml:
ExecStart=/usr/local/bin/borgmatic --verbosity 1 --syslog-verbosity 1 --config /etc/borgmatic/config.yaml
# --- END ADJUST ---

Environment=BORG_PASSPHRASE_COMMAND=/etc/borgmatic/scripts/get_borg_pass.sh
# Consider adding BORG_CACHE_DIR and BORG_CONFIG_DIR if not default for the user
# Environment="BORG_CACHE_DIR=/home/youruser/.cache/borg"
# Environment="BORG_CONFIG_DIR=/home/youruser/.config/borg"

[Install]
WantedBy=multi-user.target

Key Adjustments:

User: The user who owns the source files (or can read them) and whose SSH keys are set up for remote repos.
ExecStart:
- The path to borgmatic executable. If you installed it in a virtual environment like ~/borgmatic-env/bin/borgmatic, use that full path. If installed system-wide (e.g., /usr/local/bin/borgmatic), use that.
- The --config flag: If your borgmatic config.yaml is not in /etc/borgmatic/config.yaml (for root user) or ~YOUR_SERVICE_USER/.config/borgmatic/config.yaml, you must specify its path with --config. For example, if User=youruser and config is ~youruser/.config/borgmatic/config.yaml, then the path is /home/youruser/.config/borgmatic/config.yaml.
- Save and close.

Create borgmatic.timer File:

Create the timer file:

sudo nano /etc/systemd/system/borgmatic.timer

Paste the following (adjust OnCalendar for your desired schedule):

[Unit]
Description=Daily Borgmatic backup timer (02:30 AM)

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=10min
Persistent=true
Unit=borgmatic.service

[Install]
WantedBy=timers.target

Save and close.

Reload systemd, Enable and Start Timer:

sudo systemctl daemon-reload
sudo systemctl enable borgmatic.timer
sudo systemctl start borgmatic.timer

Verify Timer Status:
```
sudo systemctl status borgmatic.timer
```
Look for Active: active (waiting) and Next Elapses: to see when it's scheduled.
Manually Test the Service:
- Trigger the service to run immediately:
```
sudo systemctl start borgmatic.service
```
- Check its status (it will run and then go inactive, which is normal for Type=oneshot):
```
sudo systemctl status borgmatic.service
```
  Look for Active: inactive (dead) and a successful main process exit code (e.g., Main PID: ... (code=exited, status=0/SUCCESS)).
- Check the logs in detail:
```
sudo journalctl -u borgmatic.service -e
# The -e flag jumps to the end of the log.
```
  You should see output from Borgmatic, including archive creation, pruning, consistency checks, and any hook outputs. If there are errors (e.g., passphrase script not found, permission issues, Borg errors), they will appear here.
Troubleshooting Common Issues:
- Passphrase script errors: journalctl will show if get_borg_pass.sh failed (e.g., "permission denied," "file not found"). Double-check path, ownership, and permissions of the script and its directory. Ensure it's executable by the User defined in borgmatic.service.
- Borgmatic config not found: Ensure the --config /path/to/config.yaml in ExecStart is correct for the User running the service. If User=root, Borgmatic looks in /etc/borgmatic/ and ~root/.config/borgmatic/. If User=youruser, it looks in /etc/borgmatic/ (less common for user-specific configs) and ~youruser/.config/borgmatic/.
- Permission denied reading source files: The User in borgmatic.service must be able to read all source_directories.
- SSH key issues for remote repos: If User in borgmatic.service is root, root's SSH keys (/root/.ssh/) must be set up. If User=youruser, then ~youruser/.ssh/ keys are used. Ensure the correct SSH key has its passphrase managed (e.g., by an ssh-agent if the key is encrypted, or use an unencrypted key only for this purpose and very securely stored if no agent is used by systemd). The BORG_PASSPHRASE_COMMAND handles the Borg repository passphrase, not SSH key passphrases. For SSH key passphrases with systemd, you might need ssh-agent integration or unencrypted keys.

You have now successfully scheduled your Borgmatic backups using systemd, with a secure way to handle the repository passphrase. Your backups will run automatically according to the timer's schedule, and you can monitor them via journalctl.

10. Advanced Backup Strategies and Scenarios

With a solid foundation in Borg, Borgmatic, and systemd automation, you can now tackle more complex backup scenarios. This section explores strategies for backing up dynamic application data like databases, container volumes, and implementing multi-repository or air-gapped setups for enhanced data protection.

Backing Up Databases (MySQL/MariaDB, PostgreSQL) using Pre-Backup Hooks

Databases store their data in live files that are constantly changing. Simply copying these files often results in a corrupted, inconsistent backup. The proper way to back up databases is to use their native dump utilities.

General Strategy:
1. Use a before_backup hook in Borgmatic to execute the database's dump command (e.g., mysqldump, pg_dump).
2. Direct the dump output to a file within one of Borgmatic's source_directories.
3. Borgmatic then backs up this SQL dump file.
4. (Optional) Use an after_backup hook to delete the temporary SQL dump file.

MySQL/MariaDB Example:

before_backup script (e.g., /usr/local/bin/backup_mysql.sh):

#!/bin/bash
DB_USER="backupuser"
DB_PASS="backuppassword" # Securely manage this! Use .my.cnf or other methods.
DB_NAME="mydatabase"
BACKUP_DIR="/var/backup_dumps/mysql" # Ensure this is in Borgmatic's source_directories
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
OUTPUT_FILE="${BACKUP_DIR}/${DB_NAME}-${TIMESTAMP}.sql.gz" # Compress the dump

mkdir -p "$BACKUP_DIR"

echo "Dumping MySQL database: $DB_NAME to $OUTPUT_FILE"
mysqldump -u "$DB_USER" -p"$DB_PASS" --single-transaction --quick --lock-tables=false "$DB_NAME" | gzip > "$OUTPUT_FILE"

if [ $? -eq 0 ]; then
    echo "MySQL dump successful."
    # Optional: Prune old dumps in BACKUP_DIR, keeping a few recent ones locally
    find "$BACKUP_DIR" -name "*.sql.gz" -type f -mtime +7 -delete
    exit 0
else
    echo "ERROR: MySQL dump failed!" >&2
    rm -f "$OUTPUT_FILE" # Clean up failed dump
    exit 1
fi

Security Note for DB_PASS: Storing passwords in scripts is bad. For MySQL, create a ~/.my.cnf file for backupuser with credentials:
```
[mysqldump]
user=backupuser
password=backuppassword
```
Make ~/.my.cnf readable only by backupuser (chmod 600). Then mysqldump can be run without -p"$DB_PASS".
--single-transaction: For InnoDB tables, this starts a transaction and dumps data from that consistent snapshot, minimizing locking.
--quick: Dumps row by row, useful for large tables.
--lock-tables=false: Avoids global read lock if using --single-transaction.

Borgmatic config.yaml relevant part:

location:
    source_directories:
        - /var/backup_dumps/mysql # Where the .sql.gz files are saved
        # ... other source directories ...
hooks:
    before_backup:
        - /usr/local/bin/backup_mysql.sh
    # Optional after_backup to remove dump if you only want it in Borg:
    # after_backup:
    #    - find /var/backup_dumps/mysql -name "*.sql.gz" -type f -delete

PostgreSQL Example:

before_backup script (e.g., /usr/local/bin/backup_postgres.sh):

#!/bin/bash
DB_USER="backupuser_pg"
DB_NAME="mypgdatabase"
BACKUP_DIR="/var/backup_dumps/postgres"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
OUTPUT_FILE="${BACKUP_DIR}/${DB_NAME}-${TIMESTAMP}.sql.gz" # Or .custom.gz for custom format

mkdir -p "$BACKUP_DIR"

echo "Dumping PostgreSQL database: $DB_NAME to $OUTPUT_FILE"
# For plain SQL dump:
# pg_dump -U "$DB_USER" -d "$DB_NAME" | gzip > "$OUTPUT_FILE"
# For custom format (recommended for pg_restore flexibility):
pg_dump -U "$DB_USER" -d "$DB_NAME" -Fc | gzip > "${BACKUP_DIR}/${DB_NAME}-${TIMESTAMP}.custom.gz"
OUTPUT_FILE="${BACKUP_DIR}/${DB_NAME}-${TIMESTAMP}.custom.gz" # update for message

if [ $? -eq 0 ]; then
    echo "PostgreSQL dump successful."
    find "$BACKUP_DIR" -name "*.custom.gz" -type f -mtime +7 -delete # Prune old local dumps
    exit 0
else
    echo "ERROR: PostgreSQL dump failed!" >&2
    rm -f "$OUTPUT_FILE"
    exit 1
fi

Authentication: PostgreSQL uses pg_hba.conf for client authentication. You can also use a ~/.pgpass file for backupuser_pg (chmod 600): hostname:port:database:username:password.
-Fc: Custom format, compressed by default (though gzipping again is fine), allows for more flexible restores with pg_restore (e.g., selecting specific tables, parallel restore).

Borgmatic config.yaml: Similar to MySQL, add script to before_backup and BACKUP_DIR to source_directories.

Consistency Considerations:
- LVM Snapshots: For very large or high-transaction databases, dumping might take too long or impact performance. An alternative is to:
  1. (Hook) Briefly lock/quiesce the database.
  2. (Hook) Create an LVM (Logical Volume Manager) snapshot of the filesystem holding the database files.
  3. (Hook) Unlock the database.
  4. Mount the LVM snapshot read-only.
  5. Add the path to the database files on the snapshot to Borgmatic's source_directories (or rsync them to a backup area and back that up). This backs up the raw database files from a consistent point in time.
  6. (Hook) Unmount and remove the LVM snapshot after backup. This is more complex but provides a near-instantaneous consistent copy. Restoring involves replacing the live database files with the backed-up ones (after stopping the DB server).

Backing Up Container Volumes (e.g., Docker)

Containers often store persistent data in Docker volumes.

Strategy 1: Back up from within the container (if backup tools are present).
- Use docker exec CONTAINER_NAME /path/to/backup_script.sh in a before_backup hook.
- The script inside the container dumps data to a directory within the mounted volume.
- Borgmatic on the host then backs up that volume's path on the host.
Strategy 2: Back up the volume path from the host.
- Find the Docker volume's mount point on the host:
```
docker volume inspect my_volume_name
# Look for "Mountpoint": "/var/lib/docker/volumes/my_volume_name/_data"
```
- Add this host path to Borgmatic's source_directories.
- Consistency: For live application data (like databases) inside volumes, this direct file backup might not be consistent. You'd ideally stop the container or use an in-container dump method first.
  - before_backup: docker stop my_app_container
  - (Borgmatic backs up /var/lib/docker/volumes/my_volume_name/_data)
  - after_backup: docker start my_app_container
Strategy 3: Docker-specific backup tools.
- Tools like offen/docker-volume-backup can create tarballs of volumes, potentially executing pre/post commands within containers. You could integrate these with Borgmatic hooks.

Multi-Repository Strategies (Onsite and Offsite)

The 3-2-1 backup rule emphasizes multiple copies on different media, with one offsite. Borgmatic makes this easy:

location:
    source_directories:
        - /home/myuser/important_data
    repositories:
        - /mnt/usb_backup_drive/borg_repo_local  # Onsite copy
        - ssh://backupuser@my-remote-nas.example.com/borg_repo_offsite # Offsite copy
# ... rest of config ...

When Borgmatic runs create, it will create the same archive in both repositories. Pruning and consistency checks will also run against both.

Considerations:
- Bandwidth: Backing up to a remote offsite repository will be limited by your upload speed. Initial backups can be large.
- Seeding: For very large initial remote backups, you can "seed" the repository:
  1. Create the initial backup to an external drive locally.
  2. Physically transport this drive to the remote location.
  3. Copy the repository from the drive to the remote server's storage.
  4. Future Borgmatic backups will then only send incremental changes over the network.
- Different Retention Policies: If you want different retention for local vs. remote (e.g., keep more local, less remote due to cost/space), you'd need two separate Borgmatic configuration blocks (as shown in Section 6), each pointing to a different repository and having its own retention settings.

Air-Gapped Backups

Air-gapped backups provide maximum protection against online threats like ransomware. The backup media is physically disconnected from any network or computer most of the time.

How to implement with Borg/Borgmatic:
1. Primary backups (daily/hourly) go to an always-on local or remote Borg repository.
2. Periodically (e.g., weekly/monthly), perform an additional backup to a dedicated external USB drive.
  - Connect the USB drive.
  - Run a specific Borgmatic configuration (or a manual borg create command) targeting a repository on this USB drive.
  - Once complete, eject and physically disconnect the USB drive. Store it securely and offline.
3. This requires manual intervention for the air-gapped step.
4. The repository on the air-gapped drive can be a full standalone repo or you could potentially use borg rcreate to synchronize from your primary repo to the air-gapped one when it's connected (though this links them; a fully separate borg create is more "air-gapped" in spirit for recovery from primary repo compromise).

Client-Side vs. Server-Side Deduplication Considerations

Borg performs client-side deduplication.

The client reads data, chunks it, checks its local cache (~/.cache/borg/) for known chunks, and if a chunk is unknown or cache is missing, it queries the server repository to see if the chunk ID exists.
Only new, unique chunks (after compression and encryption) are sent to the server.
Pros: Saves network bandwidth significantly, especially for remote backups. Reduces storage on the server.
Cons: Client needs CPU resources for chunking, hashing, compression, encryption. Client needs disk space for the chunks cache.

Some other backup systems might use server-side deduplication, where raw data is sent to the server, and the server then deduplicates it. This shifts CPU/storage load to the server but uses more bandwidth. Borg's client-side approach is generally preferred for self-hosted scenarios where bandwidth might be a constraint.

Cross-Platform Backups (Linux, macOS, Windows via WSL)

Linux & macOS: Borg works natively. Ensure paths and permissions are handled as expected.
Windows:
- WSL (Windows Subsystem for Linux): You can install Borg within a WSL distribution and back up Windows filesystem paths mounted under /mnt/c, /mnt/d, etc.
  - Caveats: Windows ACLs and some metadata might not translate perfectly to what Borg (a POSIX-focused tool) understands and stores. NTFS alternate data streams are generally not backed up. Test restores thoroughly.
- Cygwin: Another option, but WSL is generally more integrated.
- Native Windows Borg builds (less common): Some community efforts might exist, but official Borg is Linux-first.
- For critical Windows system state, dedicated Windows backup tools (like Windows Server Backup or Veeam Agent for Windows) are often used in conjunction with Borg for file-level data.

Disaster Recovery (DR) Planning and Testing

Having backups is only half the battle; knowing how to restore them in a disaster is crucial.

DR Plan: Document your recovery procedures:
- How to access backups (passphrases, key locations).
- Steps to restore different types of data (files, databases, full system).
- Hardware/software requirements for restoration.
- Contact information for key personnel.
DR Testing: Regularly simulate disaster scenarios:
- Restore a random selection of files.
- Restore a database to a test server and verify its integrity.
- If feasible, perform a full system restore to a spare machine or VM.
- Time your restore process to understand your RTO (Recovery Time Objective).
- Identify any gaps or issues in your plan or backup integrity.

Workshop Backing Up a Live Application (e.g., a Simple Web App with a Database)

Scenario: You have a simple application, perhaps a locally hosted WordPress site (using SQLite for simplicity in this workshop, or a small MariaDB/MySQL instance if you're comfortable). You want to back it up using Borgmatic, including its files and database.

For this workshop, we'll simulate a "flat-file CMS" or a simple app that uses an SQLite database, as it's easier to set up for everyone.

Application Setup (Example: Hugo static site with a "content" SQLite DB):

Install Hugo (Optional - or use any directory of files and a dummy SQLite DB): If you want to simulate a static site generator:

# On Debian/Ubuntu
sudo apt install hugo
hugo new site mySimpleWebApp
cd mySimpleWebApp
git init # Often needed for themes
git submodule add https://github.com/theNewDynamic/gohugo-theme-ananke.git themes/ananke
echo "theme = 'ananke'" >> hugo.toml # Or config.toml / config.yaml
hugo new content posts/my-first-post.md
echo "Some content for the post" >> content/posts/my-first-post.md
# hugo server -D # To test locally

The mySimpleWebApp directory contains the site files.

Create a Dummy SQLite Database for the App:

# Ensure sqlite3 is installed: sudo apt install sqlite3
cd mySimpleWebApp # Or your app directory
mkdir -p app_database
sqlite3 app_database/application_data.db <<EOF
CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT);
INSERT INTO settings (key, value) VALUES ('site_name', 'My Awesome App');
INSERT INTO settings (key, value) VALUES ('last_updated', '$(date)');
CREATE TABLE user_content (id INTEGER PRIMARY KEY AUTOINCREMENT, content TEXT);
INSERT INTO user_content (content) VALUES ('Initial piece of user data.');
.quit
EOF

Now, mySimpleWebApp/app_database/application_data.db is our database.

Project Goals:

Write a before_backup hook script to dump the SQLite database.
Configure Borgmatic to back up the application files (e.g., mySimpleWebApp directory) and the database dump.
(Optional) Write an after_backup hook to clean up the dump file.
Simulate a "disaster" (delete the application files and DB).
Perform a restore of the application files and discuss database restoration.

Prerequisites:

A working Borgmatic setup.
The simple application (e.g., mySimpleWebApp with application_data.db) created.
sqlite3 command-line tool installed.

Steps:

Create Database Dump Script (before_backup hook):

Create /usr/local/bin/backup_sqlite_app.sh (or in ~/scripts):
```
sudo nano /usr/local/bin/backup_sqlite_app.sh
```

Add the following content (adjust APP_DIR and DUMP_SUBDIR if needed):

#!/bin/bash
APP_DIR="$HOME/mySimpleWebApp" # Path to your application's root
DB_FILE="$APP_DIR/app_database/application_data.db"
DUMP_SUBDIR="db_dumps_sqlite" # Subdirectory within APP_DIR for dumps
BACKUP_DUMP_DIR="$APP_DIR/$DUMP_SUBDIR"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
OUTPUT_FILE="${BACKUP_DUMP_DIR}/app_db-${TIMESTAMP}.sql"

echo "INFO: Starting SQLite dump for $DB_FILE to $OUTPUT_FILE"
mkdir -p "$BACKUP_DUMP_DIR"

# Dump the SQLite database
sqlite3 "$DB_FILE" ".dump" > "$OUTPUT_FILE"

if [ $? -eq 0 ]; then
    echo "INFO: SQLite dump successful."
    # Optional: Prune old .sql files in BACKUP_DUMP_DIR locally
    find "$BACKUP_DUMP_DIR" -name "*.sql" -type f -mtime +3 -delete
    exit 0
else
    echo "ERROR: SQLite dump failed for $DB_FILE!" >&2
    rm -f "$OUTPUT_FILE" # Clean up failed dump
    exit 1
fi

Make it executable: sudo chmod +x /usr/local/bin/backup_sqlite_app.sh
Chown it to the user that will run Borgmatic if not root: sudo chown youruser:youruser /usr/local/bin/backup_sqlite_app.sh

Create Cleanup Script (Optional after_backup hook):

If you only want the dump within the Borg archive and not lingering in the source directory:
```
sudo nano /usr/local/bin/cleanup_sqlite_dump.sh
```

Content:

#!/bin/bash
APP_DIR="$HOME/mySimpleWebApp" # Path to your application's root
DUMP_SUBDIR="db_dumps_sqlite"
BACKUP_DUMP_DIR="$APP_DIR/$DUMP_SUBDIR"

echo "INFO: Cleaning up SQLite dump file(s) from $BACKUP_DUMP_DIR"
rm -f "$BACKUP_DUMP_DIR"/*.sql
echo "INFO: SQLite dump cleanup complete."
exit 0

Make executable and chown as above.

Configure Borgmatic:

Edit your Borgmatic config.yaml. You might create a new configuration block if you have others, or modify an existing one.

# Example for a new configuration block
# -
# location: ... (if part of a list of configs)

location:
    source_directories:
        - /home/youruser/mySimpleWebApp # Backs up app files AND the db_dumps_sqlite subdir

    # ... repositories ...
    repositories:
        - /path/to/your/borg/repo # Or remote ssh://...

storage:
    # ... encryption, archive_name_format ...
    archive_name_format: '{hostname}-mySimpleWebApp-{now:%Y-%m-%d}'

retention:
    # ... your retention policy ...
    prefix: '{hostname}-mySimpleWebApp-'
    keep_daily: 5

hooks:
    before_backup:
        - /usr/local/bin/backup_sqlite_app.sh
    # Uncomment if using the cleanup script
    # after_backup:
    #    - /usr/local/bin/cleanup_sqlite_dump.sh
    on_error:
        - echo "BORGMATIC ERROR: Backup for mySimpleWebApp FAILED!" >&2

consistency: # Optional, but good practice
    checks:
        - repository
        - archives
    check_last: 1
    verify_data: true

Replace youruser and paths as appropriate.
Ensure the source_directories includes the parent directory (mySimpleWebApp) where the db_dumps_sqlite subdirectory will be created by the hook.

Run Borgmatic Backup:
- Ensure BORG_PASSPHRASE is handled (e.g., set in env, or service file if using systemd later).
- Run Borgmatic (e.g., borgmatic -v 1 or sudo systemctl start borgmatic.service if configured).
- Verify:
  - The hook script runs.
  - The SQL dump is created in mySimpleWebApp/db_dumps_sqlite/.
  - Borgmatic backs it up.
  - The cleanup script (if used) removes the SQL dump from the source after backup.
  - Check the Borg archive content (borg list REPO::ARCHIVE) to see the .sql file and application files.
Simulate Disaster:
- CRITICAL: Ensure you have a successful backup in Borg BEFORE doing this!
- cd ~ (move out of the app directory).
- rm -rf mySimpleWebApp
- Verify it's gone: ls mySimpleWebApp (should show "No such file or directory").

Perform Restore:

Identify the latest (or correct) archive:

# Adjust repo path and passphrase handling
LATEST_ARCHIVE=$(BORG_PASSPHRASE='yourpass' borg list --short --sort-by timestamp /path/to/your/borg/repo | tail -n 1)
echo "Will restore from: $LATEST_ARCHIVE"

Restore application files (including the SQL dump):
```
mkdir ~/restore_location
cd ~/restore_location
# Adjust repo path and passphrase handling
BORG_PASSPHRASE='yourpass' borg extract /path/to/your/borg/repo::$LATEST_ARCHIVE
```
This will restore the entire content of the archive, likely into a subdirectory matching the original path (e.g., home/youruser/mySimpleWebApp).

Move the restored application to its original location:

# Example: if restored to ~/restore_location/home/youruser/mySimpleWebApp
mv home/youruser/mySimpleWebApp ~/
cd ~/mySimpleWebApp # Go into the restored app directory

Restore the Database: The SQL dump (e.g., db_dumps_sqlite/app_db-....sql) is now restored along with other files.

# List the dump files to find the correct one
ls db_dumps_sqlite/

# Assuming the dump is db_dumps_sqlite/app_db-YYYYMMDD_HHMMSS.sql
# And the live database file should be app_database/application_data.db

# Remove any potentially empty/corrupt placeholder DB file from file restore
rm -f app_database/application_data.db
mkdir -p app_database # Ensure directory exists

sqlite3 app_database/application_data.db < db_dumps_sqlite/app_db-YYYYMMDD_HHMMSS.sql # Replace with actual dump file name

Verify Application:
- Check contents of app_database/application_data.db:
```
sqlite3 app_database/application_data.db "SELECT * FROM settings;"
```
- If it was a Hugo site, try hugo server -D within ~/mySimpleWebApp to see if it runs.

This workshop demonstrates a common pattern for backing up applications with databases: dump the database to a file, back up the application code and the dump file, and then have a clear process for restoring both. The specifics vary by database type and application complexity, but the principle of using hooks for pre/post processing is key.

11. Troubleshooting Common Issues

Even with well-configured systems, issues can arise with backup processes. Being able to diagnose and resolve these problems is crucial for maintaining a reliable backup strategy. This section covers common problems encountered with BorgBackup and Borgmatic, along with troubleshooting steps. Remember that verbose output (-v or --verbose for Borg, --verbosity 1 or 2 for Borgmatic) and system logs (journalctl, /var/log/) are your best friends in troubleshooting.

Permission Errors

Permission errors are among the most frequent issues, occurring either on the client (reading source files, writing to cache) or on the server (writing to the repository).

Client-Side Error Reading Source Files
- Symptom: borg create or Borgmatic fails with messages like "Permission denied: '/path/to/source/file'" or "OSError: [Errno 13] Permission denied".
- Cause: The user account under which borg or borgmatic is running (this could be your interactive user, or a specific user defined in a cron job or systemd service file, like User=your_backup_user) lacks the necessary read permissions for one or more of the specified source_directories or the files/subdirectories within them. For directories, the user also needs execute permission to traverse into them.
- Troubleshooting:
  1. Identify the User: Determine exactly which user is running the backup command. For interactive sessions, it's your current user (whoami). For automated jobs (systemd/cron), check the User= directive in the service file or the crontab entry.
  2. Pinpoint the Problematic Path: The error message from Borg usually indicates the exact file or directory causing the permission issue.
  3. Check Permissions: Use ls -ld /path/to/problematic/file_or_dir to view its ownership and permissions. For example, drwxr-x--- 1 root data_group 4096 Oct 30 10:00 /srv/important_data. This shows root as owner, data_group as group, and only owner can write, owner and group can read/execute. Others have no permissions.
  4. Verify Access: If your backup user is, say, bkpuser, and bkpuser is not root and not in data_group, they won't be able to access /srv/important_data.
  5. Solutions:
    - Adjust Permissions (use with caution): Modify permissions on the source files/directories to grant the backup user read access. For example, sudo chmod o+r /path/to/file or sudo chmod -R o+rx /path/to/directory. Be mindful of the security implications of opening up permissions too broadly.
    - Use Groups: Add the backup user to a group that already has appropriate access to the source data (e.g., sudo usermod -a -G data_group bkpuser). You might need to log out and back in for group changes to take effect.
    - Run as Privileged User (last resort): If backing up system-critical files that only root can read, you might need to run Borg/Borgmatic as root. If doing so, ensure all configurations (passphrase scripts, SSH keys) are owned and secured by root. This increases the potential impact if the backup process is compromised.
    - ACLs (Access Control Lists): For more granular control, you can use setfacl to grant specific permissions to the backup user without changing base ownership or group permissions (e.g., sudo setfacl -R -m u:bkpuser:rX /path/to/source_directory).
Client-Side Error Writing to Cache/Keys
- Symptom: borg create fails with "Permission denied" related to ~/.cache/borg/, ~/.config/borg/security/, or ~/.config/borg/keys/.
- Cause: The user running Borg does not have write permission to their own home directory's cache or configuration subdirectories used by Borg. This is uncommon unless home directory permissions are severely misconfigured, or BORG_CACHE_DIR, BORG_SECURITY_DIR, or BORG_KEYS_DIR environment variables are set to point to a non-writable location.
- Troubleshooting:
  1. Verify the user running Borg (as above).
  2. Check the ownership and permissions of the problematic directory (e.g., ls -ld ~/.cache/borg). The user running Borg should own this directory and have write permissions.
  3. If environment variables are used, ensure the specified paths are correct and writable by the user.
  4. Correct permissions if needed (e.g., sudo chown -R bkpuser:bkpuser /home/bkpuser/.cache/borg; sudo chmod -R u+rwX /home/bkpuser/.cache/borg).
Server-Side Error Writing to Repository (Remote Backups)
- Symptom: borg create (when targeting a remote SSH repository) fails with "Permission denied," "Repository write error," or an SSH-related error that implies a permissions problem on the server after successfully connecting.
- Cause:
  1. The SSH user on the remote server (e.g., borguser in ssh://borguser@server/...) does not have write (and execute for directories) permissions to the specified repository path on the server (e.g., /srv/borg_repositories/client_A_repo).
  2. The command="borg serve --restrict-to-path /some/path ..." line in the server's ~/.ssh/authorized_keys file for that SSH key might:
    - Specify an incorrect path in --restrict-to-path.
    - Specify a path that the SSH user cannot actually write to, even if the directory itself has wider permissions (the restriction takes precedence for Borg operations).
- Troubleshooting (on the Remote Server):
  1. Verify Repository Path Permissions: Log into the remote server. Check ownership and permissions of the repository directory: ls -ld /path/to/actual/repository_on_server. Ensure the SSH user (e.g., borguser) owns this directory and has rwx (read, write, execute) permissions. For example: sudo chown -R borguser:borguser /srv/borg_repositories/client_A_repo; sudo chmod -R u+rwx /srv/borg_repositories/client_A_repo.
  2. Inspect authorized_keys: Carefully examine the ~borguser/.ssh/authorized_keys file on the server. Find the line corresponding to the client's SSH key.
    - Ensure the path provided to --restrict-to-path is the exact, absolute path to the repository directory intended for this client. Typos are common.
    - Ensure this path is indeed writable by borguser.
  3. Check for Mount Options: If the repository is on a separately mounted filesystem on the server, ensure it's not mounted read-only (check mount output or /etc/fstab).
  4. SELinux/AppArmor: If enabled on the server, security policies might be preventing borg serve (run via SSH) from writing to the target directory. Check audit logs (/var/log/audit/audit.log or journalctl -t audit) for denials. You may need to adjust SELinux contexts (chcon) or AppArmor profiles.
  5. Server-Side SSH Logs: Examine the SSH daemon logs on the server (often /var/log/auth.log, /var/log/secure, or via journalctl -u sshd) for more detailed error messages related to the SSH session or borg serve execution.

SSH Connection Problems

When backing up to a remote repository via SSH, various connection issues can occur.

Symptom: borg commands targeting a remote repository fail with messages like "Connection refused," "Connection timed out," "No route to host," "Host key verification failed," or "Permission denied (publickey,gssapi-keyex,password)."
Causes & Troubleshooting:
1. Connection Refused / Connection Timed Out / No Route to Host:
  - Server Offline/Unreachable: The remote server might be powered off or experiencing network issues. Try ping your_remote_server_ip from the client.
  - SSH Daemon Not Running on Server: The SSH service (sshd) might not be active on the remote server.
    - On the server: sudo systemctl status sshd (or sudo systemctl status ssh on older systems). If not active, sudo systemctl start sshd and sudo systemctl enable sshd.
  - Firewall Blocking SSH Port: A firewall on the server (e.g., ufw, firewalld) or an intermediate network firewall could be blocking incoming connections on the SSH port (default is 22).
    - On the server: Check firewall status (e.g., sudo ufw status, sudo firewall-cmd --list-all). Ensure the SSH port is allowed.
  - Incorrect Server IP/Hostname or Port: Double-check the repository URL in your Borg/Borgmatic configuration: ssh://user@correct_hostname_or_ip:port/path/to/repo. If using a non-standard port, ensure it's specified.
  - Network Configuration Issues: DNS problems (if using hostname), incorrect routing, or general network outages.
2. Host Key Verification Failed:
  - Symptom: WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ... Host key verification failed.
  - Cause: The SSH host key of the remote server has changed since the last time the client connected. This is expected if the server's OS was reinstalled or its SSH keys were regenerated. It could also indicate a man-in-the-middle (MITM) attack, though this is less common on trusted networks.
  - Solution:
    1. Verify (if possible): If you can, verify the new host key fingerprint with an out-of-band method (e.g., ask the server admin or check the server console).
    2. Remove Old Key: On the client machine, remove the old host key entry for the server from ~/.ssh/known_hosts.
      ssh-keygen -R your_remote_server_hostname_or_ip
    3. Reconnect: The next time you attempt an SSH connection (or run Borg), you'll be prompted to accept the new host key. Type yes if you trust the connection.
3. Permission Denied (publickey,gssapi-keyex,password) or similar authentication failures:
  - Cause: SSH server on the remote machine rejected the authentication attempt. This usually means issues with SSH key-based authentication.
  - Troubleshooting:
    - Client's Public Key Missing on Server: Ensure the client's public SSH key (~/.ssh/id_rsa.pub, ~/.ssh/id_ed25519.pub, etc.) is correctly added to the ~REMOTE_USER/.ssh/authorized_keys file on the remote server.
    - Incorrect Permissions on Server: Permissions on the server for ~REMOTE_USER/.ssh directory must be 700 (drwx------). Permissions for ~REMOTE_USER/.ssh/authorized_keys file must be 600 (-rw-------). The home directory itself (~REMOTE_USER) should not be group-writable.
      - On the server: chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys
    - Client Presenting Wrong Key: If the client has multiple SSH private keys, it might be trying to use the wrong one.
      - Specify the correct key with ssh -i /path/to/private_key ... for manual testing.
      - Configure ~/.ssh/config on the client to specify the IdentityFile for that host. Example:
        Host myremoteserver HostName actual_ip_or_domain User borguser IdentityFile ~/.ssh/borg_specific_key
    - SSH Key Passphrase: If your private key is encrypted with a passphrase, you'll need to enter it, or use ssh-agent to manage it for you. If running automated backups via systemd/cron, ssh-agent forwarding or an unencrypted key (with very strict file permissions) might be necessary.
    - Server's sshd_config: Ensure PubkeyAuthentication yes is set in /etc/ssh/sshd_config on the server. If you only want key-based auth, also set PasswordAuthentication no and ChallengeResponseAuthentication no. Restart sshd after changes.
4. command= Restriction Issues in authorized_keys:
  - Symptom: SSH connection might seem to establish then immediately close, or Borg commands fail with cryptic errors.
  - Cause: Syntax errors in the command="borg serve ..." line, incorrect path to borg serve on the server (if not in SSH user's PATH), or wrong path in --restrict-to-path.
  - Troubleshooting:
    - Double-check the syntax of the entire command="..." string in ~REMOTE_USER/.ssh/authorized_keys on the server.
    - Ensure borg serve is executable and in the PATH of the remote SSH user, or provide an absolute path to borg serve within the command string (e.g., command="/usr/bin/borg serve ...").
    - Verify the --restrict-to-path argument points to the correct, existing, and accessible (by the remote SSH user) repository path.
5. Verbose SSH Client Output:
  - For deep SSH debugging from the client, use ssh -vvv user@host. This will show detailed information about the connection establishment, key exchange, and authentication attempts, often revealing the point of failure.
  - You can also set BORG_RSH='ssh -vvv' before running a Borg command to get this verbose SSH output within Borg's execution.

Passphrase Issues (Borg Repository Passphrase)

These errors relate to the encryption passphrase of the Borg repository itself, not SSH key passphrases.

Symptom:
- borg init (on an existing repo) or other commands fail with "Failed to create/acquire lock", followed by passphrase prompts or errors.
- "Decryption failed" messages.
- Persistent incorrect passphrase prompts when you believe you're providing the correct one.
Causes & Troubleshooting:
1. Incorrect Passphrase Entered: This is the most common cause.
  - Double-check for typos, case sensitivity (passphrases ARE case-sensitive), or an accidentally enabled Caps Lock key.
  - Ensure no leading/trailing whitespace if copying/pasting.
2. BORG_PASSPHRASE Environment Variable:
  - If you rely on BORG_PASSPHRASE for automation, ensure it's set correctly in the environment where Borg/Borgmatic runs.
  - echo "$BORG_PASSPHRASE" to verify its value just before running the command (but be careful, this might print it to logs if not careful).
  - Ensure it's exported if set in a script: export BORG_PASSPHRASE="yourpass"
3. BORG_PASSPHRASE_COMMAND Script Errors:
  - If using BORG_PASSPHRASE_COMMAND (e.g., in a systemd service file), the script specified might be failing.
  - Script Not Executable or Wrong Path: Ensure the script has execute permissions for the user running Borg and the path is correct.
  - Script Output: The script must output the passphrase only to standard output, followed by a newline. Any other output (e.g., echo "My passphrase is:" followed by the pass) or errors sent to stdout will result in an incorrect passphrase.
  - Script Permissions: The script itself should have very tight permissions (e.g., chmod 500 or 700, owned by the user running Borg or root).
  - Test the script manually: sudo -u <backup_user> /path/to/pass_script and see what it outputs.
4. Borgmatic Configuration:
  - If encryption_passphrase is set directly in Borgmatic's config.yaml (generally discouraged for security), verify it's correct.
  - If encryption_passcommand is set in config.yaml, the same script checks as for BORG_PASSPHRASE_COMMAND apply.
5. Keyfile Mode Issues: If using keyfile encryption mode:
  - Ensure the keyfile (e.g., in ~/.config/borg/keys/) is present, readable by the user, and not corrupted.
  - If the keyfile itself was passphrase-protected during borg init, that passphrase is required.
6. Forgetting the Passphrase:
  - This is a critical situation. If you used repokey mode and have truly forgotten the passphrase, and you do not have an exported key backup (borg key export), your data is irrecoverably lost. Borg's encryption is strong.
  - If you used keyfile mode and lost the keyfile, the data is also lost, unless the keyfile was passphrase-protected and you remember that passphrase (in which case you'd need to restore the keyfile from a backup).
  - This underscores the importance of securely storing your passphrase or keyfiles.

Full Repository / Out of Space

Symptom: borg create fails with errors like "No space left on device," "Create append only failed: ... ENOSPC", "Repository is full," or I/O errors during write operations.
Causes & Troubleshooting:
1. Storage Device Genuinely Full: The underlying storage (local disk, server's disk, NAS share) where the Borg repository resides is full.
  - Check disk space: df -h /path/to/repository_mount_point (on the machine hosting the repository).
2. Ineffective Pruning Policies: Your Borgmatic retention settings (or manual borg prune parameters) are not deleting enough old archives to keep space usage in check.
  - Review keep_daily, keep_weekly, keep_monthly, prefix, etc., settings. Are they appropriate for your data change rate and available storage?
3. borg compact Not Run: After pruning or deleting archives, borg compact is needed to actually reclaim disk space. If this isn't run regularly, the repository size on disk won't decrease even if archives are deleted.
4. Unexpectedly Large Source Data: The data you're backing up has grown significantly, or new large sources were added, exceeding storage capacity faster than anticipated.
5. Filesystem Quotas Reached: If the user owning the repository on the server is subject to filesystem quotas, they might have hit their limit even if the disk itself has free space.
6. Solutions:
  - Adjust Retention Policies: Make your pruning more aggressive (e.g., keep fewer daily/weekly archives).
  - Run borg compact: Schedule borg compact /path/to/repo to run periodically (e.g., after pruning) if not already part of your Borgmatic flow or automated separately.
  - Add More Storage: Increase the capacity of the storage device.
  - Exclude Unnecessary Data: Review your backup sources and exclusion patterns (--exclude, exclude_from in Borgmatic) to ensure you're not backing up large, unnecessary files (caches, temporary files, downloads, media you can re-download).
  - Monitor Disk Usage: Implement monitoring for the repository storage to get alerted before it becomes critically full.

Slow Backups

Symptom: Backups take an unusually long time to complete compared to previous runs or expectations.
Causes & Troubleshooting:
1. Network Bottleneck (Remote Repositories):
  - Low bandwidth between client and server. Test with tools like iperf3.
  - High latency. ping your_remote_server_ip.
  - Network congestion or faulty network hardware.
  - MTU issues: Incorrect Maximum Transmission Unit settings can cause packet fragmentation and retransmissions.
2. Slow Storage I/O:
  - Client Source Disk: Slow reading from the client's source disks. Check disk health (SMART status) and performance (e.g., using hdparm -tT /dev/sdX or fio).
  - Server Repository Disk (or Client's local repo disk): Slow writing to the repository disk. Check disk health and performance on the server. USB drives, especially older ones or those on slow USB ports, can be a bottleneck.
3. CPU Limitation (Client or Server):
  - Borg's operations (chunking with Buzhash, hashing with SHA256/BLAKE2, compression, encryption) are CPU-intensive.
  - Monitor CPU usage on both client and server during a backup using top or htop. If CPU is consistently at 100% on one or more cores, this could be the bottleneck.
  - Less powerful CPUs (e.g., on Raspberry Pi or older NAS devices) will naturally lead to slower backups.
4. Borg Cache Issues:
  - Missing or Small Cache: If the client's Borg cache (~/.cache/borg/ or BORG_CACHE_DIR) is missing, corrupted, or too small to hold information about existing chunks, Borg has to query the repository more frequently, slowing down deduplication, especially for remote repositories. The first backup after deleting the cache will be slower.
  - Ensure the cache directory has sufficient disk space and correct permissions.
5. Compression Algorithm Choice:
  - High-compression algorithms like lzma or high levels of zstd/zlib are very CPU-intensive and slow, although they save more space. lz4 (Borg's default) or zstd with a low-to-mid level (e.g., zstd,3) offer better speed.
  - Consider using borg benchmark cpu to test different algorithm combinations on your hardware.
6. Very Large Number of Small Files:
  - Backing up millions of tiny files involves significant filesystem metadata overhead (opening, reading, closing each file), which can be slower than backing up a few large files of the same total size.
7. Inefficient Exclusion Patterns:
  - Complex or poorly written exclusion patterns might cause Borg to spend excessive time evaluating files that should be skipped.
8. borg check --verify-data: If consistency checks with --verify-data are run as part of the same Borgmatic job after every backup, this can significantly extend the total job time as it re-reads, decrypts, and decompresses data. Schedule such intensive checks less frequently if speed is an issue.
9. Initial Backup: The very first backup of a dataset will always be the slowest as all data needs to be chunked, processed, and sent to the repository. Subsequent backups are much faster due to deduplication.

Borg Cache Issues

The client-side cache (~/.cache/borg/ or as specified by BORG_CACHE_DIR) is crucial for Borg's performance, especially with remote repositories. It stores metadata about chunks already present in the repository.

Symptom:
- Backups are noticeably slower than usual, particularly for remote repositories.
- You might see warnings related to cache operations if verbosity is high.
Causes & Troubleshooting:
1. Cache Deleted or Corrupted: If the cache directory is accidentally deleted or its files become corrupted (e.g., due to an unsafe system shutdown), Borg will effectively operate as if it's a new repository for deduplication purposes (it will query the server for every chunk).
  - Solution: Borg will automatically rebuild the cache during the next backup(s). The backup where the cache is being rebuilt will be slower. There's usually no need for manual intervention unless the cache directory itself has permission/space issues.
2. Insufficient Disk Space for Cache: The cache can grow quite large, proportional to the number of unique chunks in your repositories. If the filesystem holding the cache runs out of space, operations will fail.
  - Solution: Ensure adequate free space on the filesystem where the cache resides. Consider moving the cache to a larger partition using BORG_CACHE_DIR.
3. Permissions Issues on Cache Directory: The user running Borg must have read, write, and execute permissions for the cache directory and its contents.
  - Solution: Correct permissions (e.g., chown -R user:group ~/.cache/borg; chmod -R u+rwx ~/.cache/borg).
4. Multiple Repositories, Single Cache: If you back up to many different repositories, the single cache directory will store info for all of them. This is normal, but it means the cache size reflects the total chunk diversity across all repositories it knows about.
5. Cache Sync Issues (rare): In very rare cases or with older Borg versions, the cache might get out of sync with the repository state. Running borg check on the repository can sometimes help, but usually, if the cache is suspect, letting Borg rebuild it is the simplest approach.

Borgmatic Configuration Errors

Borgmatic relies on YAML configuration files. Errors in these files are a common source of problems.

Symptom: Borgmatic fails to start or execute, often printing an error message pointing to a problem in its config.yaml file (or the file specified with --config).
Causes & Troubleshooting:
1. YAML Syntax Errors: YAML is very sensitive to indentation (spaces, not tabs!). Incorrect indentation is the most common syntax error.
  - Solution: Carefully review your config.yaml file. Use a text editor that can show whitespace characters or has YAML syntax highlighting. Online YAML validators or linters can also help identify syntax issues.
2. Incorrect Paths: Paths specified for source_directories, repositories, hook scripts, exclude_from files, or encryption_passcommand might be incorrect or typos.
  - Solution: Verify that all paths are absolute (unless relative paths are explicitly supported and intended) and point to existing, accessible locations.
3. Misspelled Configuration Keys: Using an incorrect name for a configuration option (e.g., source_directory instead of source_directories).
  - Solution: Refer to the official Borgmatic documentation (e.g., man borgmatic or the online documentation) for the correct key names and structure.
4. borgmatic config validate: This is your primary tool for checking Borgmatic configurations.
```
borgmatic config validate
# Or if using a non-default config file:
# borgmatic --config /path/to/your_config.yaml config validate
```
  This command will parse the configuration and report syntax errors or structural issues.
5. Borgmatic Logs: Increase Borgmatic's verbosity (--verbosity 1 or --verbosity 2) to get more detailed output about what it's trying to do and where it's failing. If run via systemd, check journalctl -u your_borgmatic_service_name.

Interrupted Backups

Backups can be interrupted due to power loss, system crashes, network disconnects, or manual termination.

Symptom: The backup process (borg create or borgmatic) terminates prematurely.
State of Repository & Incomplete Archives:
- Borg is designed to be robust against interruptions. It uses a transactional commit mechanism. If a backup is interrupted during the creation of an archive, that archive is typically not fully committed and will not appear in borg list as a valid, complete archive.
- You might find a checkpoint archive if the interruption was somewhat graceful or if checkpoints were enabled. These are named like myarchive.checkpoint. They represent an intermediate, resumable state.
- The repository metadata itself should remain consistent.
Troubleshooting:
1. Run borg check: After an interruption, it's good practice to run borg check REPOSITORY_PATH to verify the overall integrity of the repository. It should report no errors if Borg handled the interruption correctly.
2. Handle Checkpoint Archives:
  - List archives: borg list REPOSITORY_PATH. If you see a .checkpoint archive for the interrupted backup, you can usually just delete it:
```
borg delete REPOSITORY_PATH::archive_name.checkpoint
```
  - Alternatively, if you want to try and resume from it (less common for automated setups), you could rename it (though this is advanced and typically not needed as re-running the backup is easier).
3. Re-run the Backup: The simplest solution is to re-run the borg create or borgmatic command. Borg's deduplication will ensure that only data not already processed and committed from the previous attempt (or already existing from prior successful backups) will be processed and sent. The new attempt will create a fresh, complete archive.

Data Corruption

This is the most serious issue, potentially indicating problems with your storage media or other systemic issues.

Symptom:
- borg check (run without --repair) reports errors like "segment inconsistency," "obj N: Error: Did not find mandatory part P of (CHUNK_ID)", "archive consistency check failed," or similar integrity validation failures.
- borg extract or borg mount fails with errors when trying to access specific data.
Causes:
- Hardware Issues: Failing hard drives or SSDs (bad sectors), faulty RAM (on client or server), unstable storage controllers, bad cables. This is the most common cause of persistent corruption.
- Filesystem Corruption: Underlying filesystem on which the Borg repository resides may be corrupted.
- Unsafe Shutdowns/Crashes: System crashing or losing power while Borg is actively writing to the repository segments. While Borg is designed to be robust, extreme scenarios can sometimes lead to issues.
- Software Bugs: Extremely rare in Borg itself for causing data corruption, but bugs in underlying OS, filesystem drivers, or network protocols could theoretically contribute.
- Bit Rot (Silent Data Corruption): Data on storage media degrading over time without overt errors. Borg's internal checksums and MACs are designed to detect this.
Troubleshooting - Proceed with Extreme Caution:
1. DO NOT PANIC. DO NOT IMMEDIATELY RUN borg check --repair.
2. ISOLATE THE REPOSITORY: If possible, stop any further writes (backups) to the affected repository until the issue is understood.
3. BACK UP THE CORRUPTED REPOSITORY: Before attempting any repair, if at all feasible, make a raw, block-level copy (e.g., using dd) or a file-level copy (rsync -a) of the entire Borg repository directory to a different, reliable storage medium. This is your critical safety net. If --repair makes things worse or causes unacceptable data loss, you can try to revert or use other tools on this copy.
4. GATHER INFORMATION:
  - Run borg check REPOSITORY_PATH --verbose to get as much detail as possible about the errors. Note which archives or data segments are affected.
  - Check system logs (dmesg, journalctl, /var/log/syslog) on both the client and server (if remote) for any hardware errors, I/O errors, or filesystem warnings around the time the corruption might have occurred.
  - Check S.M.A.R.T. status of the storage drives hosting the repository (e.g., sudo smartctl -a /dev/sdX).
  - Run a filesystem check (e.g., fsck) on the filesystem hosting the repository (unmount it first if possible, or run from a live CD).
5. ASSESS THE DAMAGE AND CAUSE: Try to understand the extent of the corruption and, crucially, identify and fix the underlying cause (e.g., replace a failing disk). If you don't fix the cause, corruption will likely recur.
6. CONSIDER borg check --repair (as a last resort for this copy):
  - If you have a backup of the corrupted repo (Step 3), you can try running borg check --repair REPOSITORY_PATH on the original (or another copy).
  - Understand the implications: --repair attempts to make the repository internally consistent. This often means discarding corrupted data chunks or metadata. Consequently, archives that relied on this data may become incomplete or entirely unrestorable. --repair is a data recovery tool in the sense of salvaging what's left, not necessarily restoring lost data.
  - Read the output of --repair carefully to see what actions it took.
7. TEST RESTORES: After any repair attempt, thoroughly test restoring critical data from various archives to see what is recoverable.
8. IF REPAIR FAILS OR DATA LOSS IS UNACCEPTABLE:
  - Revert to the backup of the repository you made in Step 3 and try other advanced data recovery techniques if you have the expertise, or consult a professional.
  - If you have older, known-good backups of the repository itself (e.g., from a separate backup system that backed up your Borg server), you might consider restoring one of those.
  - Worst case: You may need to abandon the corrupted repository and start a new one, losing the history in the corrupted one. Restore what you can from any still-accessible archives.

Workshop Troubleshooting a Failed Backup Scenario

This workshop will guide you through simulating a common backup failure, using logs to diagnose it, and then fixing it.

Scenario: We'll simulate a Borgmatic backup failing because a BORG_PASSPHRASE_COMMAND script used by a systemd service is misconfigured (e.g., wrong permissions or incorrect output).

Project Goals:

Set up (or use an existing) borgmatic.service with BORG_PASSPHRASE_COMMAND.
Intentionally misconfigure the passphrase script to cause a failure.
Attempt to run the Borgmatic service.
Use journalctl to diagnose the failure.
Correct the misconfiguration and verify the backup now succeeds.

Prerequisites:

A working Borgmatic setup, preferably scheduled with systemd (as per Chapter 9 workshop).
A borgmatic.service file that uses Environment=BORG_PASSPHRASE_COMMAND=/path/to/get_borg_pass.sh.
The actual /path/to/get_borg_pass.sh script.
sudo privileges.

Steps:

Verify Current Working Setup (Optional):

If you have a working systemd setup for Borgmatic, first ensure it runs correctly:

sudo systemctl start borgmatic.service
sleep 10 # Give it a moment to run
sudo systemctl status borgmatic.service
sudo journalctl -u borgmatic.service -n 20 --no-pager # View last 20 log lines

It should show a successful run (status=0/SUCCESS).

Introduce a Misconfiguration in the Passphrase Script:
- Let's modify the passphrase script to make it non-executable by the user running the service or make it output an incorrect passphrase.
- Option A: Permission Issue Assume your borgmatic.service runs as User=youruser.
```
# Find your script, e.g., /etc/borgmatic/scripts/get_borg_pass.sh
# Temporarily remove execute permission for the owner
sudo chmod u-x /etc/borgmatic/scripts/get_borg_pass.sh
```
- Option B: Incorrect Output (More subtle) Edit the script (e.g., sudo nano /etc/borgmatic/scripts/get_borg_pass.sh): Change echo "your-actual-borg-repo-passphrase" To echo "WRONGPASS_your-actual-borg-repo-passphrase" or even just echo "debug: starting script..." ; echo "your-actual-borg-repo-passphrase" (extra output before passphrase) Save the script. (Remember to revert permissions if you did Option A and are now trying B).
Attempt to Run the Borgmatic Service:
```
sudo systemctl start borgmatic.service
```

Diagnose the Failure using systemctl status and journalctl:

Check the service status immediately:
```
sudo systemctl status borgmatic.service
```
You'll likely see Active: failed (Result: exit-code) and possibly a hint in the last few log lines shown by status.

Dive into the detailed logs:

sudo journalctl -u borgmatic.service -e --no-pager
# -e jumps to the end, --no-pager shows all relevant logs at once

Interpreting Logs for Option A (Permission Issue on script): You might see errors from the shell indicating it couldn't execute the script:

systemd[...]: Starting Borgmatic backup service...
sh[...]: /etc/borgmatic/scripts/get_borg_pass.sh: Permission denied
borgmatic[...]: Critical: Failed to acquire lock ... The read passphrase command ... exited with status 126.
borgmatic[...]: HINT: Poses as a GPG-compatible passphrase агенты. Check your configuration.
systemd[...]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
systemd[...]: borgmatic.service: Failed with result 'exit-code'.

The key is "Permission denied" for the script and "exited with status 126" (command invoked cannot execute).

Interpreting Logs for Option B (Incorrect Output from script): Borgmatic/Borg itself will complain about an incorrect passphrase or decryption failure.

systemd[...]: Starting Borgmatic backup service...
borgmatic[...]: INFO: Doing relevant actions for repository /path/to/your/repo...
borgmatic[...]: Critical: Failed to create/acquire the lock /path/to/your/repo/lock.exclusive (timeout).
borgmatic[...]: HINT: Maybe a stale lock from a crashed backup? Either wait or manually remove the lock.
borgmatic[...]: Borgmatic backup FAILED!
systemd[...]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
systemd[...]: borgmatic.service: Failed with result 'exit-code'.

Or more directly:

borgmatic[...]: Enter passphrase for repository /path/to/your/repo: Incorrect passphrase
borgmatic[...]: Critical: Decryption failed.

The "Failed to create/acquire lock" can sometimes be a misleading symptom if the underlying cause is a passphrase failure preventing Borg from even interacting correctly with the repository.

Correct the Misconfiguration:
- For Option A (Permission Issue): Restore execute permission:
```
sudo chmod u+x /etc/borgmatic/scripts/get_borg_pass.sh
```
- For Option B (Incorrect Output): Edit the script again (sudo nano /etc/borgmatic/scripts/get_borg_pass.sh) and ensure it only echoes the correct passphrase, with no other text and no errors.
Verify the Fix:
- Re-run the service:
```
sudo systemctl start borgmatic.service
```
- Check status and logs again:
```
sudo systemctl status borgmatic.service
sudo journalctl -u borgmatic.service -e --no-pager
```
- The backup should now complete successfully (status=0/SUCCESS), and the logs should reflect normal operation.

This workshop demonstrates a typical troubleshooting workflow: observe failure, consult detailed logs to pinpoint the cause (often an error message will point directly or indirectly to it), formulate a hypothesis, apply a fix, and verify. Always remember the importance of verbose logging when diagnosing issues.