Author	Nejat Hakan
eMail	nejat.hakan@outlook.de
PayPal Me	https://paypal.me/nejathakan

Backup Server - Duplicati

Introduction to Duplicati and Backup Strategies

Welcome to this comprehensive guide on setting up and managing your own backup server using Duplicati. In an era where data is invaluable, ensuring its safety and recoverability is paramount. This guide will walk you through the essentials of Duplicati, from basic setup to advanced configurations, empowering you to take control of your data protection strategy. We will delve into not just the "how" but also the "why," providing you with a deep understanding of backup principles and Duplicati's capabilities. This material is designed for university students and self-hosting enthusiasts who are eager to learn and implement robust backup solutions.

What is Duplicati?

Duplicati is a free, open-source backup client that securely stores encrypted, incremental, compressed backups on cloud storage services and remote file servers. It works with a wide range of backends, including local storage, SSH/SFTP, WebDAV, and popular cloud services like Amazon S3, Backblaze B2, Google Drive, Microsoft OneDrive, and many others. Its web-based user interface makes it accessible from any operating system with a modern browser, and it also offers a powerful command-line interface for automation and advanced users.

Key Features

Duplicati stands out due to a rich set of features designed for flexibility, security, and efficiency:

• Strong Encryption: Duplicati uses robust AES-256 encryption by default to protect your backups before they leave your computer. This means your data is secure even if stored on a third-party cloud service. The encryption key remains with you, ensuring only you can access your backed-up data.
• Incremental Backups and Deduplication: After an initial full backup, Duplicati only uploads the changed parts of files (incrementally). More importantly, it employs block-based deduplication. This means if you have multiple copies of the same data (even across different files or different machines backing up to the same location with the same key), Duplicati stores that data only once. This significantly saves storage space and bandwidth.
• Compression: Duplicati compresses backup data to further reduce storage space requirements. It uses standard Gzip/Zip compression.
• Web-Based User Interface: Configuration and monitoring of backups are primarily done through a user-friendly web interface, which can be accessed locally or remotely (if configured).
• Scheduling: Backups can be scheduled to run automatically at specific times or intervals, ensuring your data is consistently protected without manual intervention.
• Versatile Backend Support: Duplicati can back up to a vast array of destinations, including local disks, USB drives, network-attached storage (NAS), FTP, SFTP, WebDAV, and numerous commercial cloud storage providers.
• Cross-Platform: Duplicati runs on Windows, macOS, Linux, and other systems where Mono or .NET can run.
• Open Source: Being open-source (LGPL license), Duplicati benefits from community contributions, transparency, and the freedom to use and modify it.
• Backup Verification: Duplicati can periodically download a random set of backup files, decrypt them, and verify their integrity to ensure your backups are restorable.
• Retention Policies: You can configure how many old backups to keep, allowing for fine-grained control over storage usage versus backup history depth.

How Duplicati Works (Brief Overview)

Understanding the core mechanism of Duplicati helps in appreciating its efficiency and security:

1. File Scanning: Duplicati scans the source folders to identify files that need to be backed up or have changed since the last backup.
2. Deduplication: Files are broken down into smaller, fixed-size (or variable-size, depending on configuration) blocks. Duplicati calculates a hash for each block. If a block with the same hash has already been backed up (even from a different file or an earlier version of the same file), it's not uploaded again; only a reference to the existing block is stored.
3. Compression: The new, unique blocks of data are compressed to reduce their size.
4. Encryption: The compressed blocks are then encrypted using your chosen passphrase and encryption algorithm (AES-256 by default).
5. Packaging: Encrypted blocks are bundled into larger volumes (called dblock files). Index files (dindex) are created to map which blocks are in which volumes, and list files (dlist) are created to represent the state of your files and folders at the time of the backup.
6. Upload: These dblock, dindex, and dlist files are uploaded to the configured backup destination.

This process ensures that backups are space-efficient, secure, and that only necessary data is transferred.

Why Self-Host Your Backup Server?

While commercial backup services offer convenience, self-hosting your backup solution with a tool like Duplicati provides several compelling advantages, especially for those who value control and understanding of their systems.

Control and Privacy

• Data Sovereignty: You decide where your data is stored. This could be on your own hardware in your home, on a server you control in a data center, or even on a trusted friend's machine. You are not bound by the terms of service or privacy policies of a third-party provider that might change.
• Encryption Key Management: With Duplicati, you hold the encryption keys. The service provider (if you use one for storage) only sees encrypted blobs of data and cannot access your files. This is a significant privacy advantage.
• No Vendor Lock-in: Because Duplicati is open-source and supports standard protocols and various backends, you can easily switch storage providers or change your backup strategy without losing access to your backups or being locked into a proprietary format.

Cost-Effectiveness

• Leverage Existing Infrastructure: You might already have a NAS, a home server, or spare hard drives that can be repurposed for backups, significantly reducing costs compared to paying monthly subscription fees for cloud backup services, especially for large data volumes.
• Choice of Storage: You can choose the most cost-effective storage solution that meets your needs, from cheap deep archive cloud storage to local disks.
• No Per-Device Fees: Many commercial backup services charge per device. With Duplicati, you can back up as many devices as you want to your self-hosted solution without incurring extra licensing costs from Duplicati itself.

Customization

• Tailored Configuration: Self-hosting allows you to fine-tune every aspect of your backup process: scheduling, retention policies, encryption algorithms, network usage, and more, to perfectly match your specific requirements.
• Integration: You can integrate Duplicati with other self-hosted services, scripts, and monitoring tools for a more cohesive and automated environment.
• Learning Opportunity: Setting up and managing your own backup server is an excellent learning experience, deepening your understanding of data management, network protocols, and system administration.

Fundamental Backup Concepts

Before diving into Duplicati's specifics, it's crucial to understand some fundamental backup concepts. These principles will guide your decisions in configuring an effective backup strategy.

The 3-2-1 Backup Rule

This is a widely recognized and highly recommended best practice for data protection:

• 3 Copies of Your Data: Keep at least three copies of any important data. This includes the original "production" data and at least two backups.
• 2 Different Storage Media: Store these copies on at least two different types of storage media. For example, an internal hard drive and an external hard drive, or an internal hard drive and cloud storage. This protects against media failure.
• 1 Copy Off-Site: Keep at least one of these copies in a geographically separate location. This protects against local disasters like fire, flood, or theft. Cloud storage inherently satisfies this, or you could use a NAS at a friend's or family member's house.

Full, Incremental, and Differential Backups

Duplicati primarily uses a sophisticated form of incremental backup with deduplication. However, it's useful to understand the traditional definitions:

• Full Backup: A full backup copies all selected data. It's the most straightforward but also the most time-consuming and storage-intensive. Restoring is simple as you only need the latest full backup.
• Incremental Backup: An incremental backup only copies data that has changed since the last backup of any type (full or incremental). Restores require the last full backup and all subsequent incremental backups in order. They are fast to create and use less storage than differential backups. Duplicati's block-based approach is a more advanced version of this.
• Differential Backup: A differential backup copies data that has changed since the last full backup. Restores require the last full backup and the latest differential backup. They take longer to create and use more storage than incrementals over time but are generally faster to restore.

Duplicati's model is "synthetically full." Each backup operation behaves like an incremental, but because of how it stores data (blocks and metadata files), each backup version can be thought of as a "full" snapshot in time, without the storage penalty of traditional full backups.

Encryption and Data Security

• Encryption: As mentioned, encrypting your backups is crucial, especially if they are stored off-site or on third-party services. AES-256 is a strong, industry-standard encryption algorithm.
• Passphrase: The passphrase you choose for encryption is critical. It should be long, complex, unique, and stored securely (e.g., in a password manager). Losing your passphrase means losing access to your backups.
• Data in Transit: Ensure data is encrypted during transfer to the backup destination (e.g., using HTTPS, SFTP, or a VPN). Duplicati encrypts data before transmission.
• Data at Rest: Data stored at the destination should be encrypted. Duplicati handles this by encrypting the data itself. Some storage providers offer additional server-side encryption, which can be an extra layer of security.

Retention Policies

A retention policy defines how long you keep your backups and how many versions you retain.

• Purpose: Balances the need to recover older versions of files against the cost of storage space.
• Considerations:
  • Recovery Point Objective (RPO): How much data can you afford to lose? This influences backup frequency. If you back up daily, your RPO is up to 24 hours.
  • Recovery Time Objective (RTO): How quickly do you need to recover data? This influences your choice of backup media and restore procedures.
  • Storage Capacity: Available storage space will limit how many versions you can keep.
  • Compliance/Legal Requirements: Some data may have legal retention requirements.
• Common Strategies:
  • Keep daily backups for a week.
  • Keep weekly backups for a month.
  • Keep monthly backups for a year or more. Duplicati offers flexible options to implement such policies.

Workshop Understanding Backup Needs

This workshop will guide you through the process of assessing your data and defining your backup requirements. This foundational step is crucial before configuring any backup software.

Project Assess Your Personal Data

The goal is to create a simple inventory of your important data and determine your basic backup requirements.

Step 1 Identify Critical Data

Think about the data on your computer(s) that would be irreplaceable or very difficult/costly to recreate if lost.

• Action: Make a list of these data categories.
• Examples:
  • Personal documents (resumes, letters, financial records, academic papers)
  • Photos and videos (family memories, creative projects)
  • Work or school projects
  • Application settings or configuration files for critical software
  • Code repositories
  • Email archives
  • Databases (e.g., from a password manager, personal finance software)
• Guidance: For each category, note down the typical location(s) on your computer (e.g., C:\Users\YourName\Documents, /home/yourname/Pictures, specific project folders). This will help when configuring Duplicati's source selection.
• Consider: What data, if lost, would cause significant distress or disruption? Prioritize these.

Step 2 Estimate Data Volume

Estimate the total size of the critical data you identified.

• Action: For each category or primary folder, check its size using your operating system's file explorer properties. Sum these up to get a rough total.
• Example:
  • Documents: 5 GB
  • Photos: 150 GB
  • Projects: 20 GB
  • Total: ~175 GB
• Guidance: This estimate will help you determine the required storage capacity for your backups. Remember that Duplicati uses compression and deduplication, so the actual storage used might be less, but it's good to have an estimate of the raw source data size. Also, consider future growth.

Step 3 Define Recovery Point Objective (RPO) and Recovery Time Objective (RTO)

These are crucial metrics for any backup strategy.

• Recovery Point Objective (RPO): This refers to the maximum acceptable amount of data loss measured in time. For example, if your RPO is 24 hours, it means you can tolerate losing up to 24 hours' worth of data changes. This directly influences how frequently you need to back up.
  • Action: For your most critical data, ask yourself: "If my hard drive died right now, how much of the work I did in the last X hours/days would I be willing to lose?"
  • Examples:
    • For active research papers: RPO of 1 hour might be desired (though frequent backups can be intensive).
    • For general documents: RPO of 1 day might be acceptable.
    • For a photo archive that changes infrequently: RPO of 1 week might be acceptable.
• Recovery Time Objective (RTO): This refers to the maximum acceptable amount of time to restore data after a disaster or data loss event. How long can you afford to be without access to your critical data?
  • Action: Ask yourself: "If I lost all my critical files, how quickly would I need them back to resume normal operations or avoid significant inconvenience?"
  • Examples:
    • For critical work files: RTO of a few hours.
    • For less critical archives: RTO of a day or two.
• Guidance: Be realistic. A very short RPO (e.g., minutes) requires very frequent backups, which can impact system performance and storage. A very short RTO might require faster, more expensive backup media or more complex recovery procedures. Your RPO will directly influence your backup schedule, and your RTO will influence your choice of backup destination and restore process planning.

Step 4 Choose a Backup Destination (Conceptual)

Based on your understanding of the 3-2-1 rule, data volume, and RTO/RPO, start thinking conceptually about where your backups will live. We will cover specific destination configurations later.

• Action: Consider the following options and which might fit your needs and resources:
  • Local Storage:
    • External USB Hard Drive: Good for quick backups and restores. Portable.
    • Second Internal Hard Drive: Fast, but not protected from system-wide failures (e.g., power surge, malware affecting all connected drives) or local disasters (fire, theft).
    • Network Attached Storage (NAS): Centralized storage accessible by multiple devices on your local network. Good for a home or small office.
  • Off-Site Storage:
    • Cloud Storage (e.g., Backblaze B2, Wasabi, Google Drive, OneDrive): Provides geographical separation, often with good durability. Costs can vary based on storage and bandwidth.
    • Remote Server (e.g., a friend's computer, a VPS, a dedicated server): Requires setting up SSH/SFTP or another protocol. You manage the remote end.
• Guidance: Aim for at least one local and one off-site backup destination for critical data to align with the 3-2-1 rule. For example:
  • Primary backup to a local NAS (quick RTO for common restores).
  • Secondary backup (or a copy of the primary backup) to cloud storage (disaster recovery, off-site). Duplicati can be configured with multiple backup jobs, potentially targeting different destinations for different data or different retention policies.

By completing this workshop, you've laid the groundwork for creating a meaningful and effective backup strategy. You now have a clearer picture of what you need to protect and what your recovery expectations are, which will inform all your Duplicati configurations moving forward.

Basic Level Duplicati Setup and Operation

This section will guide you through the initial steps of getting Duplicati up and running, configuring your first backup job, and performing a restore. We will focus on fundamental operations to build a solid understanding of Duplicati's core functionality.

1. Installing Duplicati

Duplicati is designed to run on a variety of operating systems. The installation process is generally straightforward.

System Requirements

Duplicati's resource requirements are relatively modest for basic operation, but they can increase with the size of the backup set and the frequency of backups.

• Operating System:
  • Windows: Windows 7 or later, Windows Server 2008 R2 or later. Requires .NET Framework 4.6.2 or newer (Duplicati installer usually bundles a compatible .NET version if needed, or will prompt).
  • Linux: Most modern distributions. Requires Mono (version 5.0 or newer recommended). Common dependencies include libsqlite3-0, libssl-dev (or equivalent OpenSSL libraries), and ca-certificates.
  • macOS: macOS 10.10 Yosemite or later. It's distributed as a .dmg package.
• Processor: A modern CPU (e.g., 1 GHz or faster) is sufficient. Faster CPUs will improve compression and encryption speeds.
• Memory (RAM): At least 512 MB of RAM is recommended, but 1 GB or more is better, especially for larger backups. Duplicati can consume more memory when processing large file lists or during intense operations like database recreation.
• Disk Space:
  • For Duplicati installation: Around 100-200 MB.
  • For local Duplicati database: This can grow depending on the number of files backed up and the number of versions kept. It can range from a few megabytes to several gigabytes for very large backup sets (millions of files). Ensure the drive where Duplicati stores its configuration and local databases has ample free space. This is typically in your user profile directory.

Installation on Windows

1. Download: Go to the official Duplicati website (duplicati.com) and download the latest stable or beta version .msi installer for Windows. Beta versions often have more features and bug fixes but might be less stable than official releases. For learning, the latest beta is usually fine.
2. Run Installer: Double-click the downloaded .msi file.
3. Follow Prompts: The installation wizard will guide you. You'll typically accept the license agreement and choose an installation location (the default is usually fine).
4. Launch Duplicati: After installation, Duplicati may offer to launch. If not, you can find it in your Start Menu. By default, Duplicati runs as a tray icon. When it starts, it also starts a web server on localhost (your own computer) on port 8200. Your default web browser should open automatically to http://localhost:8200. If not, you can manually open this address.

Installation on Linux (e.g., Debian/Ubuntu, Fedora)

The installation process varies slightly depending on your Linux distribution.

• For Debian/Ubuntu and derivatives (e.g., Mint, Pop!_OS):

  1. Download: Go to duplicati.com and download the .deb package.
  2. Install Dependencies (if not already present):

     sudo apt update
     sudo apt install mono-complete libsqlite3-0 ca-certificates # Potentially libssl-dev for some setups
     

     mono-complete installs the full Mono runtime.
  3. Install Duplicati: Navigate to the directory where you downloaded the .deb file and run:

     sudo dpkg -i duplicati_VERSION_all.deb 
     

     Replace duplicati_VERSION_all.deb with the actual filename. If you encounter dependency issues, run:

     sudo apt-get install -f
     

     This command attempts to fix broken dependencies.
  4. Start and Enable Service (Recommended for server use):

     sudo systemctl start duplicati
     sudo systemctl enable duplicati
     

     This starts the Duplicati service and ensures it starts automatically on boot. The service usually runs the web UI on http://localhost:8200. If you are on a headless server, you'll need to access this from another machine on your network, potentially via an SSH tunnel or by configuring Duplicati to listen on all interfaces (see advanced section).

• For Fedora/RHEL/CentOS and derivatives:

  1. Download: Go to duplicati.com and download the .rpm package.
  2. Install Dependencies:

     sudo dnf install mono-core sqlite # Or yum for older RHEL/CentOS
     

  3. Install Duplicati:

     sudo rpm -ivh duplicati-VERSION.noarch.rpm
     

     Or, using dnf:

     sudo dnf install duplicati-VERSION.noarch.rpm
     

     Replace duplicati-VERSION.noarch.rpm with the actual filename.
  4. Start and Enable Service:

     sudo systemctl start duplicati
     sudo systemctl enable duplicati
     

Installation on macOS

1. Download: Go to duplicati.com and download the .dmg disk image file for macOS.
2. Mount Image: Double-click the .dmg file to mount it. A Finder window will open showing the Duplicati application icon.
3. Install: Drag the Duplicati application icon to your Applications folder.
4. First Run: Open Duplicati from your Applications folder. macOS might warn you that it's an application downloaded from the internet. You may need to right-click (or Control-click) the application icon and choose "Open," then confirm.
5. Tray Icon: Duplicati will run as an icon in the macOS menu bar. Clicking this icon will give you options, including opening the web interface, which runs on http://localhost:8200.

Initial Configuration and Accessing the Web UI

Once Duplicati is installed and running (either as a tray application or a service), it hosts a web server.

• Access: Open your web browser and navigate to http://localhost:8200.
• Interface: You should see the Duplicati Web UI. If this is the first time running Duplicati on the machine, it might ask if this machine should host Duplicati server instances or if it's just for accessing another server. For a standard setup where Duplicati manages its own backups, choose the default/first option.
• Password Protection (Highly Recommended):
  1. In the Web UI, go to Settings (usually in the left-hand menu or under a main menu icon).
  2. Under the "Server" section (or similar, depending on UI version), you'll find an option to set a password for accessing the interface.
  3. Enter a strong password and save the settings. You will be logged out and will need to re-enter the password to access the UI. This is crucial if other users have access to your computer or if you plan to expose the Duplicati web interface to your network.

Workshop Installing Duplicati on a Virtual Machine

This workshop will guide you through installing Duplicati on a Linux Virtual Machine (VM). Using a VM is a great way to experiment without affecting your main operating system. We'll use Ubuntu Server as an example.

Project Set up Duplicati on a Linux VM

The goal is to have a functional Duplicati instance running on an Ubuntu Server VM, accessible via its web UI.

Step 1 Prepare Your Virtual Machine

• Action:
  1. Using your preferred virtualization software (e.g., VirtualBox, VMware Workstation/Player, Hyper-V, KVM/QEMU), create a new virtual machine.
  2. Install Ubuntu Server (e.g., latest LTS version). Allocate at least 1 CPU, 1-2 GB RAM, and 20 GB disk space.
  3. Ensure the VM has network access. A "Bridged Adapter" or "NAT Network" (if you want to access it from other machines on your physical network) is often preferred. For initial setup, NAT is fine if you can access ports from the host.
  4. After installation, log into your Ubuntu Server VM.
• Guidance: During Ubuntu Server installation, you can choose to install an OpenSSH server, which is highly recommended for headless management.

Step 2 Download Duplicati

• Action:
  1. On your Ubuntu Server VM, open a terminal.
  2. Update your package list:

     sudo apt update
     

  3. Install wget if it's not already installed (it usually is):

     sudo apt install wget -y
     

  4. Go to the Duplicati downloads page (https://www.duplicati.com/download) using a browser on your host machine to find the URL of the latest .deb package.
  5. Use wget to download the .deb package into your VM. For example:

     wget https://updates.duplicati.com/beta/duplicati_2.0.X.Y_beta_YYYYMMDD-1_all.deb 
     

     (Replace https://updates.duplicati.com/beta/duplicati_2.0.X.Y_beta_YYYYMMDD-1_all.deb with the actual current link).
• Guidance: Always download software from official sources. The "beta" channel for Duplicati is often more up-to-date and generally stable enough for personal use and learning.

Step 3 Install Duplicati

• Action:
  1. First, install the necessary dependencies:

     sudo apt install mono-complete libsqlite3-0 ca-certificates -y
     

  2. Navigate to the directory where you downloaded the Duplicati .deb file (usually your home directory).
  3. Install the package using dpkg:

     sudo dpkg -i duplicati_*.deb 
     

     (The * will match the downloaded filename).
  4. If dpkg reports any missing dependencies, fix them with:

     sudo apt-get install -f -y
     

• Guidance: mono-complete ensures all necessary parts of the Mono framework are available. libsqlite3-0 is for Duplicati's local database, and ca-certificates is for SSL/TLS connections.

Step 4 Start and Enable Duplicati Service

• Action:
  1. Start the Duplicati service:

     sudo systemctl start duplicati
     

  2. Enable the service to start automatically on boot:

     sudo systemctl enable duplicati
     

  3. Check the status of the service to ensure it's running:

     sudo systemctl status duplicati
     

     You should see output indicating it's "active (running)". Press q to exit the status view.
• Guidance: Running Duplicati as a service is the standard way for server installations. It ensures Duplicati is always running in the background to perform scheduled backups.

Step 5 Access the Duplicati Web UI

• Action:
  1. The Duplicati service by default listens on localhost:8200 within the VM. To access this from your host machine's browser, you have a few options:
     • SSH Tunneling (Recommended for security if VM is not on a trusted network): On your host machine (not the VM), open a terminal or PowerShell and run:

       ssh -L 8200:localhost:8200 your_vm_user@your_vm_ip_address
       

       Replace your_vm_user with your username on the VM and your_vm_ip_address with the VM's IP address (you can find this with ip a on the VM). Now, on your host machine, open a browser to http://localhost:8200.
     • Configure Duplicati to Listen on All Interfaces (Less Secure, use with caution on untrusted networks): Edit the Duplicati service configuration:

       sudo nano /etc/default/duplicati 
       

       Find the line DAEMON_OPTS="" and change it to:

       DAEMON_OPTS="--webservice-interface=any --webservice-port=8200"
       

       Save the file (Ctrl+O, Enter, then Ctrl+X in nano). Then restart the Duplicati service:

       sudo systemctl restart duplicati
       

       Now you can access the UI from your host browser using http://your_vm_ip_address:8200. Make sure your VM's firewall (if enabled, e.g., ufw) allows incoming connections on port 8200.

       sudo ufw allow 8200/tcp 
       

  2. Once you can access the UI, you'll see the initial Duplicati setup screen. It might ask "Is this your first time running Duplicati?". Choose "Yes, this is the first time."
• Guidance: The SSH tunnel is a more secure method if your VM is exposed to a wider network. If it's just a local VM for testing, modifying DAEMON_OPTS is simpler.

Step 6 Secure the Web UI (Basic Password)

• Action:
  1. In the Duplicati Web UI (accessed via http://localhost:8200 if using SSH tunnel, or http://your_vm_ip_address:8200 otherwise), click on "Settings" in the left sidebar.
  2. Scroll down to the "Password" field (under "Server state and settings" or similar section).
  3. Enter a strong, unique password. Confirm it.
  4. Click "OK" or "Save" at the bottom of the page. Duplicati will apply the settings and likely log you out.
  5. You will now be prompted to enter this password every time you access the Duplicati Web UI.
• Guidance: This is the minimum security step. For production environments exposed to a network, further hardening (like a reverse proxy with HTTPS) is recommended, which we'll cover in advanced sections.

You have now successfully installed Duplicati on a Linux VM and performed basic security hardening. This VM can serve as your sandbox for the rest of this guide.

2. Your First Backup Job

With Duplicati installed and accessible, it's time to create your first backup job. This will involve defining what data to back up, where to store it, how to encrypt it, and when to run the backup.

Understanding the Duplicati Interface

When you first access the Duplicati Web UI, it's relatively clean:

• Home: Shows the status of existing backup jobs.
• Add backup: The primary button or link to create a new backup job.
• Restore: To restore files from existing backups.
• Settings: Global Duplicati settings (like UI password, default options, server port).
• About: Information about the Duplicati version and system.

When you "Add backup," you'll be guided through a multi-page wizard:

1. General: Name, description, encryption type, and passphrase.
2. Destination: Where the backup data will be stored.
3. Source Data: What files and folders to back up.
4. Schedule: When the backup job should run automatically.
5. Options: Advanced settings like retention policy, upload volume size, throttling, etc.

Defining a Backup Source

This is where you specify the local files and folders you want to protect.

• Computer: The local machine where Duplicati is running.
• Source data: You can browse your file system and check boxes next to the folders or individual files you wish to include.
• Common choices for a first backup:
  • Documents folder (C:\Users\YourName\Documents or /home/yourname/Documents)
  • Pictures folder
  • A specific project folder
• Exclusions: Later, you'll learn about filters to exclude subfolders or specific file types (e.g., temporary files, cache directories). For a first backup, keeping it simple is fine.

Choosing a Backup Destination (Local Folder, USB Drive)

For your first backup, using a local destination is straightforward and quick. This could be another folder on your internal hard drive (less safe, as it's on the same physical device), an external USB drive, or a network share (though we'll cover network shares in more detail later).

• Storage Type: Select "Local folder or drive."
• Path: Specify the full path to the destination folder.
  • Example (Windows): D:\DuplicatiBackups\MyFirstBackup (assuming D: is an external drive)
  • Example (Linux): /mnt/usb_backup_drive/MyFirstBackup or /srv/backups/MyFirstBackup
• Username/Password: For local folders, these are usually not required unless the folder itself has specific permissions that the user Duplicati is running as cannot access.

Configuring Basic Scheduling

Duplicati allows you to automate your backups.

• Automatically run backups: Check this box to enable scheduling.
• Time: Specify the time of day the backup should run.
• Days: Choose which days of the week the backup should run (e.g., daily, specific weekdays).
• Repeat: You can also set it to repeat (e.g., every X hours), but for a first backup, a daily schedule is common.
• Considerations: Schedule backups during off-peak hours if they are large or might impact system performance.

Setting Basic Encryption (AES-256)

Encryption is a cornerstone of Duplicati's security.

• Encryption: Duplicati defaults to "AES-256 encryption, built-in." This is a strong and recommended option.
• Passphrase: You must enter a strong, unique passphrase. This passphrase is critical. If you lose it, you cannot restore your data. Store it securely in a password manager.
• Generate Passphrase: Duplicati often has a button to generate a strong random passphrase. This is a good option. Make sure to copy and save it immediately.
• Confirm Passphrase: Re-enter the passphrase to ensure you typed it correctly.

Running and Monitoring Your First Backup

After configuring all five steps of the wizard, you'll save the backup job.

• Saving the Job: Duplicati will save the configuration.
• Running Manually: You can usually click "Run now" on the newly created backup job from the home screen to start it immediately, even if a schedule is set.
• Monitoring Progress:
  • The Duplicati UI will show the status of the backup job (e.g., "Verifying backend data...", "Counting files...", "Backing up...", "Compacting...").
  • It will display progress bars, files processed, data uploaded, estimated time remaining, etc.
  • You can click on the job to see more details, including logs.
• First Backup Duration: The first backup is always a full backup of the selected source data. It will take the longest as all data needs to be processed, compressed, encrypted, and transferred to the destination. Subsequent backups will be much faster as they are incremental.

Workshop Backing Up Your Documents Folder Locally

This workshop will walk you through creating your very first backup job in Duplicati, targeting your user's "Documents" folder and storing it in a separate local folder.

Project Create a Local Backup of "My Documents"

The goal is to successfully back up your Documents folder to another location on your local machine (or an attached USB drive) and understand the process.

Step 1 Launch Duplicati and Add a New Backup

• Action:
  1. Open the Duplicati Web UI (e.g., http://localhost:8200).
  2. If you set a password, log in.
  3. On the home page, click "Add backup."
  4. Choose "Configure a new backup" and click "Next."
• Guidance: This starts the 5-step backup configuration wizard.

Step 2 Configure General Settings (Name, Encryption)

This is the first page of the wizard.

• Action:
  1. Name: Give your backup job a descriptive name. Example: My Documents Backup (Local)
  2. Description (Optional): Add a brief description. Example: Daily backup of my main documents folder to an external USB drive.
  3. Encryption: Select "AES-256 encryption, built-in" (this is usually the default and is highly recommended).
  4. Passphrase:
     • Enter a strong, unique passphrase. Write this down and store it in a secure location (like a password manager) immediately. Do not lose it!
     • Alternatively, click the "Generate" button next to the passphrase field to create a strong random one. Copy this generated passphrase and store it securely.
  5. Confirm passphrase: Re-enter the passphrase.
  6. Click "Next."
• Guidance: The passphrase is your only key to decrypting your backups. If you use the "Generate" button, ensure you copy it before moving to the next step or saving.

Step 3 Select Source Data

This is the third page of the wizard ("Source Data").

• Action:
  1. Under "What do you want to backup?", you will see a tree view of your computer's file system.
  2. Navigate to your user's "Documents" folder.
     • On Windows, this is typically C:\Users\<YourUserName>\Documents.
     • On Linux, this is typically /home/<yourusername>/Documents.
     • On macOS, this is typically /Users/<yourusername>/Documents.
  3. Check the box next to the "Documents" folder.
  4. (Optional for this first backup) You can expand the folder and uncheck specific sub-folders if you wish to exclude them, but for now, let's back up the whole thing.
  5. Click "Next."
• Guidance: Be mindful of the size of the folder you select. For a first test, a smaller folder might be quicker. However, backing up "Documents" is a realistic scenario.

Step 4 Choose a Local Destination

This is the second page of the wizard ("Destination").

• Action:
  1. Storage Type: Select "Local folder or drive" from the dropdown menu.
  2. Folder path:
     • Click the "..." button or manually type the path to a folder where you want to store the backups.
     • Important: Choose a location different from your source data. Ideally, on a separate physical drive (e.g., an external USB drive or a different internal partition).
     • Example (Windows, external drive E:): E:\DuplicatiBackups\DocsBackup
     • Example (Linux, assuming a mounted USB drive at /mnt/backup_usb): /mnt/backup_usb/DuplicatiBackups/DocsBackup
     • If the folder doesn't exist, Duplicati will usually create it.
  3. Username and Password: Leave these blank for a standard local folder or USB drive unless specific permissions require them (uncommon for this scenario).
  4. Click the "Test connection" button. Duplicati will attempt to create a small file in the destination folder to verify it has write access. You should see a "Connection worked!" message. If not, check the path and permissions.
  5. Click "Next."
• Guidance: Storing backups on the same physical drive as the original data offers protection against file deletion or corruption but not against drive failure. Using an external drive is a better first step towards the 3-2-1 rule.

Step 5 Set a Simple Schedule

This is the fourth page of the wizard ("Schedule").

• Action:
  1. Check the box "Automatically run backups."
  2. Time: Choose a time for the backup to run (e.g., 10:00 PM or 22:00).
  3. Days: Select "Daily" or check specific days (e.g., Monday through Friday). For this workshop, "Daily" is fine.
  4. Leave "Allowed days" and "Repeat" at their defaults for now.
  5. Click "Next."
• Guidance: Choose a time when your computer is likely to be on but not under heavy use.

Step 6 Configure Remote Volume Size and Retention (Basic)

This is the fifth and final page of the wizard ("Options").

• Action:
  1. Remote volume size: This setting determines the size of the dblock files Duplicati uploads. A good default is often 50MB or 100MB. For local backups, larger sizes (e.g., 200MB or 500MB) can sometimes be more efficient if you have very large files, but the default is fine to start. Let's leave it at the default (often 50MB).
  2. Backup retention: This defines how many old backup versions Duplicati keeps. Select "Keep a specific number of backups." For this initial workshop, set it to something simple like "5" (meaning it will keep the 5 most recent backup versions). We will explore more advanced retention later.
  3. Scroll to the bottom and click "Save."
• Guidance: The remote volume size impacts how many files are created at the destination and can affect performance with some cloud storage. For local storage, it's less critical. The retention policy is important for managing storage space.

Step 7 Run the Backup and Verify

• Action:
  1. After saving, you'll be taken back to the Duplicati home page. Your new backup job My Documents Backup (Local) will be listed.
  2. Click "Run now" next to your backup job.
  3. Observe the progress bar and status messages at the top of the page. It will show phases like "Counting files...", "Starting backup...", "Uploading files...".
  4. The first backup might take some time depending on the size of your Documents folder.
  5. Once completed, it should say "Last successful run: [Date and Time]" and show the source size and backup size.
  6. Verification (Optional but Recommended):
     • Navigate to the destination folder you specified (e.g., E:\DuplicatiBackups\DocsBackup). You should see several files with names like duplicati-xxxxxxxx.dlist.zip.aes, duplicati-xxxxxxxx.dblock.zip.aes, and duplicati-xxxxxxxx.dindex.zip.aes. These are your encrypted backup files. Do not manually delete or alter these files.
• Guidance: If the backup fails, click on the job name, then go to the "Log" tab (or "Show log" link) to see error messages. Common initial issues might be incorrect destination paths or permissions problems.

Congratulations! You've successfully configured and run your first Duplicati backup job. This is a fundamental skill upon which all further Duplicati usage will build.

3. Restoring Files

Backups are useless if you can't restore them. Duplicati provides a straightforward way to restore individual files, folders, or entire backup sets. Understanding the restore process is as crucial as setting up the backup itself.

Navigating Backup Versions

Each time a backup job runs successfully, Duplicati creates a new "version" or "snapshot" of your data at that point in time. When you initiate a restore, Duplicati will typically show you a list of available backup versions, timestamped, allowing you to choose from which point in time you want to restore.

• Accessing Restore:
  1. In the Duplicati Web UI, click "Restore" from the main menu or next to a specific backup job on the home page.
  2. If you clicked the general "Restore" button, you'll first need to select which backup job you want to restore from.
• Choosing a Version:
  • Duplicati will list available backup dates/times in a dropdown menu or list. The most recent backup is usually selected by default.
  • You can select an older version if you need to recover a file as it existed previously, or if the most recent backup doesn't contain the file (e.g., it was deleted before the last backup ran).

Restoring Specific Files or Folders

You don't always need to restore everything. Duplicati allows for granular recovery.

• File Browser: Once you've selected a backup version, Duplicati presents a file browser interface that mirrors the folder structure of your backed-up data as it existed at the time of that specific backup version.
• Selection: Navigate through the folders and check the boxes next to the specific files or entire folders you wish to restore.
• Search: There's usually a search bar to help you quickly find files or folders by name within the selected backup version.

Restoring to Original Location vs. New Location

Duplicati gives you flexibility in where the restored files are placed.

• Restore to original location:
  • If you select this option, Duplicati will attempt to put the restored files back where they originally came from.
  • Overwrite options: You'll typically have choices for how to handle existing files at the original location:
    • Save different versions (rename): If a file with the same name exists, the restored file will be renamed (e.g., myfile (restored).txt). This is often the safest default.
    • Overwrite: If a file with the same name exists, it will be replaced by the restored version. Use with caution.
• Restore to different location (alternative location):
  • You can specify a new, empty folder where Duplicati will place the restored files. This is very useful for:
    • Comparing a restored version with the current version without overwriting anything.
    • Restoring files to a temporary location for inspection.
    • Recovering data if the original path no longer exists or is on a different machine.
  • You'll need to provide the path to this new location.

Understanding "Direct Restore from Backup Files"

This is a powerful feature for disaster recovery scenarios.

• Purpose: Allows you to restore files directly from the backup destination (e.g., your external drive or cloud storage) even if:
  • The Duplicati local database for the backup job is lost or corrupted.
  • You are on a completely new machine where Duplicati has just been installed and has no prior knowledge of your backup jobs.
• Process:
  1. Click "Restore" in the Duplicati UI.
  2. Choose the option "Direct restore from backup files."
  3. You will then need to configure the connection to your backup destination (e.g., select "Local folder or drive," provide the path, or configure cloud storage details).
  4. Crucially, you will need to provide the encryption passphrase that was used when creating the backup. Without it, the data cannot be decrypted.
  5. Duplicati will then scan the destination, identify backup sets, and allow you to proceed with selecting versions and files for restore, similar to a regular restore.
• Significance: This feature means your backups are not entirely dependent on the local Duplicati installation that created them, as long as you have the destination files and the passphrase.

Workshop Restoring a Deleted File

This workshop will simulate a common scenario: accidentally deleting an important file and needing to recover it from your Duplicati backup. We will use the backup job created in the previous workshop.

Project Simulate Data Loss and Recover

The goal is to delete a file from your "Documents" folder, then use Duplicati to restore it, verifying the process.

Step 1 Identify a Test File in Your Backup Source

• Action:
  1. Open your "Documents" folder (the one you configured as the source for My Documents Backup (Local)).
  2. Create a new, simple text file. Name it something like TestRestoreFile.txt.
  3. Open the file and type some unique text into it, for example: "This is a test file for Duplicati restore. Original version." Save and close the file.
  4. (Optional but good practice) Run your backup job My Documents Backup (Local) manually by clicking "Run now" in the Duplicati UI. This ensures TestRestoreFile.txt is included in the latest backup. Wait for it to complete.
• Guidance: Using a dedicated test file makes it easy to verify the restore without risking important data. Running the backup ensures the file exists in at least one backup version.

Step 2 Delete the Test File

• Action:
  1. Go back to your "Documents" folder in your file explorer.
  2. Delete TestRestoreFile.txt. For a true test, also empty it from your Recycle Bin / Trash to simulate permanent deletion.
  3. Verify the file is gone from your Documents folder.
• Guidance: This simulates accidental data loss.

Step 3 Open Duplicati and Select "Restore"

• Action:
  1. Go to the Duplicati Web UI.
  2. On the home page, find your backup job My Documents Backup (Local).
  3. Click the "Restore" button/link associated with this specific backup job. Alternatively, you can click "Restore" from the main left-hand menu and then select "My Documents Backup (Local)" from the list of backup jobs.
• Guidance: Restoring directly from the job is usually quicker if you know which job contains the file.

Step 4 Choose the Backup Job and Version

• Action:
  1. If you didn't click "Restore" next to the job, you'll be prompted to select the backup job. Choose My Documents Backup (Local).
  2. Restore from: Duplicati will show a dropdown list of available backup versions (dates and times).
     • Select the latest version (usually pre-selected), or if you ran the backup after creating the file but before deleting it, that version should contain the file. If you are unsure, you might need to check a couple of recent versions.
  3. The file browser below will populate with the contents of your "Documents" folder as it existed in that selected backup version.
• Guidance: If you made changes to the file and ran multiple backups, you could choose an older version to get a previous state of the file.

Step 5 Find and Select the Deleted File

• Action:
  1. In the file browser area (labeled "Search for files" or similar), you can either:
     • Navigate through the folder structure to where TestRestoreFile.txt was located.
     • Or, type TestRestoreFile.txt into the search bar and press Enter or click "Search".
  2. Once you find TestRestoreFile.txt, check the box next to it.
  3. Click the "Continue" button.
• Guidance: The search function is very helpful for large backups.

Step 6 Choose Restore Location and Options

• Action:
  1. You'll be presented with "Restore Options."
  2. Original location: This is usually the default. It will try to restore TestRestoreFile.txt back into your "Documents" folder.
  3. Overwrite: Choose an option. For this test, "Save different versions (rename)" is safe. If you are certain the file is deleted and there's no conflict, "Overwrite" would also work. Let's select "Save different versions (rename)" to see how it behaves.
  4. (Optional) You could choose "Restore to a specific folder" and provide a path to a temporary folder if you wanted to inspect it first. For this exercise, let's stick with the original location.
  5. Click the "Restore" button.
• Guidance: Understanding the overwrite options is important to prevent accidental data loss of current files if they share names with restored files.

Step 7 Verify Successful Restoration

• Action:
  1. Duplicati will show a progress bar for the restore operation. It should be very quick for a single small file.
  2. Once it says "Successfully restored N files," go to your "Documents" folder using your file explorer.
  3. Look for TestRestoreFile.txt.
     • If you chose "Save different versions (rename)" and for some reason a file with that name already existed (unlikely in this specific test if you emptied trash), it might be named TestRestoreFile (restored_by_duplicati_YYYYMMDDTHHMMSS).txt or similar.
     • If you chose "Overwrite" (and it was truly deleted), it should be TestRestoreFile.txt.
  4. Open the file and verify its contents are "This is a test file for Duplicati restore. Original version."
• Guidance: Always verify restored files to ensure data integrity and that the correct version was recovered.

You have now successfully restored a "deleted" file using Duplicati. This exercise demonstrates the core restore functionality and its importance in a backup strategy. Regularly testing your restore process is a critical best practice.

Intermediate Level Duplicati Configuration and Management

Having mastered the basics of Duplicati installation, first backups, and restores, we now move to intermediate-level topics. This section will cover more diverse backup destinations, advanced scheduling, fine-tuning backups with filters, and using Duplicati's command-line interface. These skills will enable you to create more robust, efficient, and customized backup solutions.

4. Exploring Backup Destinations

Duplicati's versatility shines in its support for a wide array of backup destinations. Moving beyond local folders and USB drives allows for off-site backups and integration with various storage services, aligning with the 3-2-1 backup rule.

Network Shares (SMB/CIFS, NFS)

Network Attached Storage (NAS) devices or shares on other computers within your local network are common backup targets.

• SMB/CIFS (Server Message Block / Common Internet File System):
  • Commonly used by Windows systems for file sharing, but also supported by Linux (Samba) and macOS.
  • Configuration in Duplicati:
    • Storage Type: "SMB / CIFS"
    • Server and Port: The hostname or IP address of the SMB server (e.g., mynas.local, 192.168.1.100). The default SMB port is 445.
    • Path on server: The name of the shared folder and any subpath (e.g., backups/duplicati_pc1).
    • Username and Password: Credentials required to access the SMB share.
  • Considerations: Ensure the user Duplicati runs as (or the specified user) has write permissions to the share. Network stability is important.
• NFS (Network File System):
  • Predominantly used in Linux/Unix environments.
  • Configuration in Duplicati: Duplicati doesn't have a specific "NFS" storage type. Instead, you would typically mount the NFS share to a local directory on the machine running Duplicati, and then configure Duplicati to use that local mount point as a "Local folder or drive" destination.
  • Mounting NFS (Linux example):
    1. Install NFS client tools: sudo apt install nfs-common (Debian/Ubuntu) or sudo dnf install nfs-utils (Fedora).
    2. Create a local mount point: sudo mkdir /mnt/nfs_backup_share
    3. Mount the NFS share: sudo mount -t nfs nfs_server_ip_or_hostname:/path/to/exported_share /mnt/nfs_backup_share
    4. Configure Duplicati to use /mnt/nfs_backup_share as the destination.
    5. For permanent mounting, add an entry to /etc/fstab.
  • Considerations: Proper NFS export permissions on the server and mount options on the client are crucial.

Cloud Storage Providers

Cloud storage is an excellent option for off-site backups, providing data redundancy and accessibility. Duplicati supports many popular services.

• Pros of Cloud Storage:
  • Off-site by nature (good for disaster recovery).
  • Often high durability (data replicated by provider).
  • Scalable storage capacity.
  • Accessible from anywhere with an internet connection.

• Cons of Cloud Storage:

  • Can have ongoing costs (storage fees, download/egress fees).
  • Reliant on internet connectivity for backups and restores.
  • Upload/download speeds can be a bottleneck.
  • Privacy concerns (mitigated by Duplicati's client-side encryption).

• Examples of Supported Providers and Authentication:

  • Backblaze B2 Cloud Storage:
    • Known for its cost-effective storage.
    • Authentication: Uses an "Application Key" (Key ID and Application Key). You create these in your Backblaze B2 account.
    • Storage Type in Duplicati: "B2 Cloud Storage."
  • Wasabi Hot Cloud Storage:
    • S3-compatible, often with no egress fees (check current terms).
    • Authentication: S3-compatible Access Key ID and Secret Access Key.
    • Storage Type in Duplicati: "S3 compatible."
  • Amazon S3 (Simple Storage Service):
    • Highly scalable and feature-rich, but can be complex to price.
    • Authentication: AWS Access Key ID and Secret Access Key. Can also use IAM roles for more secure access if Duplicati runs on an EC2 instance.
    • Storage Type in Duplicati: "Amazon S3."
  • Google Drive:
    • Popular consumer cloud storage, can be used with Duplicati.
    • Authentication: OAuth. Duplicati will provide an "AuthID" link. You click it, log into your Google account, authorize Duplicati, and Google provides a code to paste back into Duplicati.
    • Storage Type in Duplicati: "Google Drive."
    • Considerations: API rate limits and storage quotas on free/consumer accounts.
  • Microsoft OneDrive:
    • Similar to Google Drive, integrated with Microsoft accounts.
    • Authentication: OAuth, similar to Google Drive.
    • Storage Type in Duplicati: "Microsoft OneDrive v2."
    • Considerations: API rate limits and storage quotas.
  • S3-Compatible Generic:
    • Many other cloud providers (e.g., DigitalOcean Spaces, Linode Object Storage, MinIO self-hosted S3) offer S3-compatible APIs.
    • Authentication: Access Key ID, Secret Access Key, and the S3 service URL (endpoint).
    • Storage Type in Duplicati: "S3 compatible."

SSH/SFTP Backups

If you have access to another server (e.g., a VPS, a dedicated server, another computer on your network, or even a Raspberry Pi) with an SSH server running, SFTP (SSH File Transfer Protocol) is a secure way to transfer backups.

• Storage Type in Duplicati: "SSH with SFTP."
• Configuration:
  • Server and Port: Hostname or IP address of the SSH server, and port (default 22).
  • Path on server: The remote directory where backups will be stored (e.g., /home/user/duplicati_backups).
  • Username: The username to log in to the SSH server.
  • Password or SSH Key:
    • Password: You can use a password, but it's generally less secure and not recommended for automation.
    • SSH Key: Using an SSH key pair (public/private) is much more secure and better for automated tasks. You generate a key pair on the machine running Duplicati, copy the public key to the SSH server's ~/.ssh/authorized_keys file for the specified user, and then provide the path to the private key file in Duplicati's configuration. You can also specify a passphrase for the private key if it's encrypted.
• Considerations: Ensure the SSH server is properly secured. The user must have write permissions to the specified path on the server.

WebDAV Backups

WebDAV (Web Distributed Authoring and Versioning) is an extension of HTTP that allows clients to perform remote web content authoring operations. Some NAS devices and cloud services (like Nextcloud/OwnCloud) support WebDAV.

• Storage Type in Duplicati: "WebDAV."
• Configuration:
  • Server URL: The full URL to the WebDAV endpoint (e.g., https://yourdavserver.com/remote.php/dav/files/username/backups).
  • Username and Password: Credentials for WebDAV access.
• Considerations: Ensure the WebDAV server supports large file uploads and is reliably available. Performance can vary. Use HTTPS for secure transmission.

Workshop Backing Up to a Raspberry Pi via SSH (SFTP)

This workshop will guide you through setting up a backup job that sends data to a Raspberry Pi (or any Linux machine acting as an SSH server) using SFTP. We'll focus on using SSH key-based authentication for better security.

Project Securely Backup Data to a Remote Raspberry Pi

The goal is to configure Duplicati to back up a folder to a Raspberry Pi over SFTP using SSH keys.

Step 1 Prepare Your Raspberry Pi (or other Linux SSH Server)

• Action:
  1. Ensure SSH is Enabled and Running: On your Raspberry Pi (or Linux server), SSH is usually enabled by default on Raspberry Pi OS and many server distributions. If not, install and enable it:

     sudo apt update
     sudo apt install openssh-server -y
     sudo systemctl enable ssh
     sudo systemctl start ssh
     

  2. Create a Dedicated Backup User (Recommended):

     sudo adduser duplicatiuser 
     

     Follow the prompts to set a password (you might not use it if relying solely on SSH keys, but it's good practice).
  3. Create a Backup Storage Directory: As duplicatiuser, or as root and then chown:

     sudo mkdir -p /mnt/backup_storage/my_pc_backups
     sudo chown duplicatiuser:duplicatiuser /mnt/backup_storage/my_pc_backups
     sudo chmod 700 /mnt/backup_storage/my_pc_backups # Restrict access
     

     (Adjust /mnt/backup_storage if you have an external drive mounted there on the Pi).
  4. Find Pi's IP Address:

     ip a
     

     Note the IP address of your Raspberry Pi on your local network.
• Guidance: Using a dedicated user and directory for backups enhances security and organization. chmod 700 ensures only duplicatiuser can access this directory.

Step 2 Generate SSH Key Pair on the Duplicati Machine

• Action: On the machine where Duplicati is running (your client machine, e.g., your main PC or the VM from the previous workshop):
  1. Open a terminal (Linux/macOS) or PowerShell/Git Bash (Windows).
  2. Generate an SSH key pair:

     ssh-keygen -t ed25519 -C "duplicati_backup_key_for_pi"
     

     • -t ed25519: Specifies the modern and secure Ed25519 algorithm. RSA is also an option (ssh-keygen -t rsa -b 4096).
     • -C "comment": Adds a comment to the key for identification.
  3. When prompted for a file to save the key: Press Enter to accept the default (e.g., ~/.ssh/id_ed25519 on Linux/macOS, C:\Users\YourName\.ssh\id_ed25519 on Windows). Note this path.
  4. When prompted for a passphrase: You can enter a passphrase to encrypt your private key. This adds an extra layer of security. If you do, Duplicati will need this passphrase when configuring the backup job. For fully automated backups, you might leave it empty, but understand the security implication: if someone gets your private key file, they can access the server. For this workshop, let's try it with a passphrase. Remember it!
• Guidance: Your private key (e.g., id_ed25519) must be kept secret. The public key (e.g., id_ed25519.pub) is what you'll copy to the server.

Step 3 Copy Public Key to Raspberry Pi

• Action:
  1. Use ssh-copy-id (if available on your Duplicati machine) for the easiest method:

     ssh-copy-id duplicatiuser@RASPBERRY_PI_IP_ADDRESS
     

     Replace RASPBERRY_PI_IP_ADDRESS with the actual IP. It will prompt for duplicatiuser's password on the Pi once to authorize the key copying.
  2. Manual Method (if ssh-copy-id is not available):
     • First, display your public key on the Duplicati machine:

       cat ~/.ssh/id_ed25519.pub 
       

       (Adjust path if you chose a non-default one or are on Windows).
     • Copy the entire output (starts with ssh-ed25519 ...).
     • SSH into your Raspberry Pi as duplicatiuser:

       ssh duplicatiuser@RASPBERRY_PI_IP_ADDRESS
       

     • On the Pi, create the .ssh directory and authorized_keys file if they don't exist, and set correct permissions:

       mkdir -p ~/.ssh
       chmod 700 ~/.ssh
       touch ~/.ssh/authorized_keys
       chmod 600 ~/.ssh/authorized_keys
       

     • Open ~/.ssh/authorized_keys with an editor (e.g., nano ~/.ssh/authorized_keys) and paste the public key content on a new line. Save and exit.
  3. Test SSH Key Authentication: From your Duplicati machine, try to SSH into the Pi again as duplicatiuser. It should now log you in without asking for the user's password (it might ask for your SSH key's passphrase if you set one).

     ssh duplicatiuser@RASPBERRY_PI_IP_ADDRESS
     

     If it works, type exit to close the SSH session.
• Guidance: ssh-copy-id is the preferred method as it handles permissions correctly. Ensure the .ssh directory and authorized_keys file on the Pi have strict permissions.

Step 4 Configure Duplicati for SFTP Destination

• Action:
  1. Open Duplicati Web UI and start creating a new backup job or edit an existing one.
  2. Page 1: General:
     • Name: e.g., My Data to Pi (SFTP)
     • Encryption: Set a strong passphrase (as usual).
  3. Page 2: Destination:
     • Storage Type: Select "SSH with SFTP".
     • Server and port: Enter the Raspberry Pi's IP address (e.g., 192.168.1.XX) and port 22 (or your custom SSH port if changed).
     • Path on server: Enter the path you created on the Pi: /mnt/backup_storage/my_pc_backups (or your chosen path).
     • Username: duplicatiuser
     • Password: Leave this blank (we are using SSH keys).
     • SSH key file: Provide the full path to your private SSH key file on the Duplicati machine (e.g., /home/your_user/.ssh/id_ed25519 or C:\Users\YourUser\.ssh\id_ed25519).
     • SSH key file passphrase: If you set a passphrase for your private key in Step 2, enter it here.
  4. Click "Test connection." Duplicati will attempt to connect to the Pi using SFTP and your SSH key. You should see "Connection worked!"
     • If it fails: Double-check IP, path, username, private key path, and key passphrase. Check SSH server logs on the Pi (sudo journalctl -u ssh or /var/log/auth.log) for clues.
  5. Page 3: Source Data: Select a small folder for this test (e.g., create a new folder with a few test files).
  6. Page 4: Schedule: Configure as desired (e.g., daily).
  7. Page 5: Options:
     • Remote volume size: Default (e.g., 50MB) is fine.
     • Backup retention: Choose a simple policy (e.g., keep 5).
  8. Save the backup job.
• Guidance: The path to the private key must be accessible by the user Duplicati is running as. On Linux, if Duplicati runs as a system service (e.g., duplicati user), ensure that user can read the key file. Often, it's easier if Duplicati runs as your user, or you place the key in a location accessible by the duplicati service user with appropriate permissions. For this workshop, assuming Duplicati runs as your desktop user is fine.

Step 5 Run the Backup and Verify on Raspberry Pi

• Action:
  1. In Duplicati, click "Run now" for the new backup job.
  2. Monitor the progress.
  3. Once complete, SSH into your Raspberry Pi:

     ssh duplicatiuser@RASPBERRY_PI_IP_ADDRESS
     

  4. Navigate to the backup directory:

     cd /mnt/backup_storage/my_pc_backups
     ls -l
     

  5. You should see the Duplicati dblock, dindex, and dlist files, encrypted and named with *.aes.
• Guidance: This confirms that files are being successfully transferred and stored on the Pi.

You've now configured Duplicati to back up data to a remote Linux server (your Raspberry Pi) using the secure SFTP protocol with SSH key-based authentication. This is a robust setup for off-site or network backups.

5. Advanced Scheduling and Retention Policies

Duplicati's default scheduling options (daily, weekly, etc.) are convenient, but for more precise control, you can use its advanced scheduling syntax. Similarly, managing storage space effectively requires understanding and implementing smart retention policies beyond just keeping a specific number of backups.

Cron-like Scheduling Syntax

Duplicati allows for a more granular scheduling mechanism similar to cron expressions found on Unix-like systems. This is typically accessed by choosing "Custom" or "Advanced" in the schedule settings.

• Format: The syntax usually consists of specifying minutes, hours, days of the month, months, and days of the week.
  • M H DoM M DOW (Minute, Hour, Day of Month, Month, Day of Week)
  • * (asterisk) means "any value" or "every".
  • , (comma) separates multiple values (e.g., 0,15,30,45 for minutes).
  • - (hyphen) defines a range (e.g., 1-5 for Monday to Friday in DOW).
  • / (slash) specifies step values (e.g., */15 for every 15 minutes).
• Examples:
  • 0 2 * * *: Run at 2:00 AM every day. (Minute 0, Hour 2, any Day of Month, any Month, any Day of Week)
  • 0 0 * * 1,3,5: Run at midnight (00:00) every Monday, Wednesday, and Friday. (DOW: 1=Monday, ..., 7=Sunday or 0=Sunday, check Duplicati's specific convention, often 0-6 for Sun-Sat or 1-7 for Mon-Sun)
  • 30 22 * * 1-5: Run at 10:30 PM (22:30) on weekdays (Monday to Friday).
  • 0 */6 * * *: Run every 6 hours, on the hour (00:00, 06:00, 12:00, 18:00).
• Duplicati's Implementation:
  • Access this in the "Schedule" page of your backup job configuration. You might need to select an "Advanced" or "Custom" option for the repeat type.
  • Duplicati's UI might provide fields for each component (Minutes, Hours, etc.) or a single text box for the cron-like string. Refer to the UI tooltips or documentation for the exact format it expects.
  • Commonly, you might specify a start time and then a repeat interval (e.g., "Run every 4 hours"). The advanced syntax gives more control like "run at 2 AM and 2 PM only".

Smart Retention Policy

Duplicati offers flexible ways to manage how many old backup versions are kept, balancing storage usage with the ability to restore older data. This is configured on the "Options" (page 5) of a backup job.

• Default Option ("Keep a specific number of backups"): Simple, e.g., "keep the last 5 backups."
• Advanced Option ("Custom backup retention"): This is where more sophisticated policies are set. Duplicati's approach can be "Keep all backups" (uses most space), "Delete backups that are older than X" (time-based), or a "Smart retention" like scheme. The most common way to configure this is using a rule string, e.g.: 7D:1D,4W:1W,12M:1M
  • 7D:1D: For backups within the last 7 Days, keep one backup per Day.
  • 4W:1W: For backups older than 7 days but within the last 4 Weeks, keep one backup per Week.
  • 12M:1M: For backups older than 4 weeks but within the last 12 Months, keep one backup per Month.
  • U:U (Unlimited time, Unlimited backups): Can be used at the end to keep all older backups, or combined with a time limit for all older backups.
  • Duplicati typically keeps the first backup of each interval. For example, with 4W:1W, if you have daily backups, for a given week, it might keep the backup made on Monday of that week.
• The "Delete backups that are older than" setting: A simpler time-based rule. For instance, set to 3 months, and any backup version older than 3 months from the current date will be deleted.
• The "Keep a specific number of full backups" (or similar wording referring to versions): Directly limits the total number of versions.

Understanding keep-time, keep-versions

These are command-line options that map to the UI settings for retention:

• --keep-time=<time>: Specifies how long to keep backups. Examples: 3M (3 months), 1Y (1 year), 7D (7 days). Backups older than this will be deleted.
• --keep-versions=<count>: Specifies the maximum number of backup versions to keep. If set to 5, Duplicati will keep the 5 newest versions.
• Combining keep-time and keep-versions: If both are specified, Duplicati keeps backups that satisfy either condition. For example, if keep-time=30D and keep-versions=5, Duplicati will keep the last 5 versions and any version within the last 30 days, even if it's more than 5 versions. This usually means it keeps more backups than you might expect. It's often better to use one or the other, or the "Smart retention" policy.
• Duplicati's Smart Retention / Custom Policy: This is the most flexible. It doesn't directly map to a single keep-time or keep-versions but rather a series of rules. The UI option "Custom backup retention" (or similar) is the place to configure this. The syntax 7D:1D,4W:1W,12M:1M is a good example of a Grandfather-Father-Son (GFS)-like scheme.

Balancing Storage Space with Backup History

• Why it matters: Storing too many backup versions can consume a lot of storage, especially with large datasets or frequent changes. Storing too few might mean you can't recover a file from far enough back in time.
• Strategy:
  1. Assess your RPO: How far back do you realistically need to go to recover files?
  2. Consider data change frequency: Data that changes rapidly might need more frequent backups and a shorter, but denser, retention period for recent history. Archival data might need fewer versions but kept for longer.
  3. Monitor storage usage: Regularly check how much space your backups are consuming at the destination.
  4. Start with a reasonable policy and adjust: A policy like 7D:1D,4W:1W,6M:1M (daily for a week, weekly for a month, monthly for 6 months) is a common starting point. Adjust based on your needs and available storage.
• Deduplication's role: Duplicati's deduplication helps significantly, but new data or heavily changed existing data will still consume new space. Retention policies manage the metadata (the dlist files representing versions) and trigger the removal of dblock files that are no longer referenced by any retained version.

Workshop Implementing a Granular Retention Policy

This workshop will guide you through setting up a "smart" or GFS-like retention policy for one of your backup jobs.

Project Optimize Backup Storage with a Smart Policy

The goal is to configure a Duplicati backup job to keep daily backups for the last week, weekly backups for the last month, and monthly backups for the last six months.

Step 1 Analyze Your Backup Needs (Daily, Weekly, Monthly)

• Action:
  1. Revisit your RPO (Recovery Point Objective) from the initial "Understanding Backup Needs" workshop.
  2. Decide on the following:
     • Short-term recovery: For how many recent days do you want daily restore points? (e.g., 7 days)
     • Mid-term recovery: For how many recent weeks (beyond the daily ones) do you want weekly restore points? (e.g., 4 weeks)
     • Long-term recovery: For how many recent months (beyond the weekly ones) do you want monthly restore points? (e.g., 6 months)
     • Oldest backup: Do you want to delete all backups older than a certain period (e.g., 1 year)?
  3. For this workshop, we'll aim for:
     • Daily backups for the past 7 days.
     • Weekly backups for the past 4 weeks (i.e., 1 month).
     • Monthly backups for the past 6 months.
     • Delete anything older than approximately 6 months implicitly through this policy.
• Guidance: This defines our target retention: 7D:1D,4W:1W,6M:1M. This means:
  • In the last 7 days, keep 1 backup per day.
  • For time between 1 week ago and 4 weeks ago, keep 1 backup per week.
  • For time between 1 month ago and 6 months ago, keep 1 backup per month.
  • Anything older than what these rules cover will be implicitly deleted.

Step 2 Configure Duplicati's Advanced Retention Options

• Action:
  1. Open the Duplicati Web UI.
  2. Choose an existing backup job (e.g., My Documents Backup (Local) or your SFTP backup job). Click "Configure."
  3. Navigate to page 5, "Options."
  4. Find the "Backup retention" setting.
  5. Select the option that allows for custom or smart retention. This might be labeled "Custom backup retention," "Keep backups using a smart retention policy," or you might need to select "Delete backups that are older than" and then also "Keep a specific number of backups" if your UI version uses a combination. The most powerful option often involves inputting a rule string.
     • If there's a direct input for a rule string (like 7D:1D,...), that's preferred.
     • Some UIs might have separate fields like "Keep daily backups for X days", "Keep weekly for Y weeks", etc.
  6. If using the rule string format: Enter: 7D:1D,4W:1W,6M:1M
     • 7D:1D: For backups within the last 7 Days from now, keep one backup every 1 Day.
     • 4W:1W: For backups older than 7 Days but within the last 4 Weeks from now, keep one backup every 1 Week.
     • 6M:1M: For backups older than 4 Weeks but within the last 6 Months from now, keep one backup every 1 Month.
     • (Implicitly, anything older than 6 months will be removed as it doesn't match any rule to be kept).
  7. If your UI uses separate fields, try to match the logic:
     • Keep 1 backup per day for 7 days.
     • Then, keep 1 backup per week for the next 3 weeks (to make up 4 weeks total, with the first week covered by dailies).
     • Then, keep 1 backup per month for the next 5 months (to make up 6 months total). The string format 7D:1D,4W:1W,6M:1M is generally more reliable and directly expresses the GFS intent.
  8. Save the backup job configuration.
• Guidance: Duplicati processes these rules from left to right. A backup is kept if it matches any rule. Duplicati typically keeps the earliest backup within an interval that satisfies the rule. For example, for 4W:1W, it would keep the first backup made in each of those weeks.

Step 3 Example Policy Clarification 7D:1D,4W:1W,6M:1M

Let's assume you run daily backups.

• Day 1 to Day 7: You will have 7 backups (one for each day).
• Day 8: The backup from Day 1 is now 7 days old. It's kept by 7D:1D. The new backup for Day 8 is also kept.
• Day 10: The backup from Day 1 is now 9 days old. It no longer matches 7D:1D. It now falls into the 4W:1W category. If it's the first backup of its week, it will be kept. The daily backups from Day 4 to Day 10 will be kept by 7D:1D.
• After 5 weeks: You'll have approximately:
  • 7 daily backups (for the most recent week).
  • 3-4 weekly backups (for the 3-4 weeks prior to the most recent week).
• After 7 months: You'll have approximately:
  • 7 daily backups.
  • 3-4 weekly backups.
  • 5 monthly backups (for months 2 through 6).
  • Backups from month 1 (the oldest month) will be deleted as they are now older than 6 months and don't fit any rule.
• Key Point: The policy defines what to keep. If a backup version doesn't match any keep rule, it's eligible for deletion. Duplicati will then automatically run compact operations as needed to reclaim space from deleted versions.

Step 4 Monitor Storage Usage Over Time

• Action:
  1. Ensure your backup job is scheduled to run regularly (e.g., daily).
  2. Over the next few weeks and months, occasionally check:
     • The list of restore points for this backup job in Duplicati (click Restore > select job > see version dropdown). Verify that the number and age of backups align with your policy.
     • The actual storage space used at your backup destination.
• Guidance: It takes time for a retention policy to fully take effect and show its impact on storage, as old backups need to age out of the retention window. If storage consumption is higher than expected, you might need to make your retention policy more aggressive (e.g., 3M:1M instead of 6M:1M). If you have ample space, you might make it more generous.

By implementing a smart retention policy, you are taking a significant step towards professional backup management, ensuring you have a useful history of backups without letting storage costs run wild.

6. Filters and Exclusions

Often, you don't need or want to back up every single file within a selected source folder. Temporary files, cache directories, large unimportant media files, or system-specific files can bloat your backups, consume unnecessary storage, and slow down the backup process. Duplicati's filter system allows you to precisely include or exclude files and folders.

Why Use Filters?

• Save Storage Space: Exclude large, unnecessary files (e.g., virtual machine disk images if backed up elsewhere, downloaded ISOs, personal video libraries if not critical).
• Speed Up Backups: Fewer files to scan, process, and upload means faster backup times.
• Reduce Clutter: Keep your backup sets clean and focused on truly important data.
• Avoid Issues: Exclude actively changing log files or database files that might be locked or inconsistent if backed up live without proper quiescing (though for critical databases, dedicated backup tools are often better).
• Security/Privacy: Exclude folders containing sensitive information that you explicitly do not want in this particular backup set.

Including/Excluding Specific Files and Folders

Duplicati filters are rule-based. You can define multiple rules, and they are processed in order.

• Filter Types:
  • Exclude folder: Excludes a specific folder and all its contents.
  • Exclude file: Excludes a specific file.
  • Exclude files whose names contain: Excludes files if their name includes a certain string.
  • Exclude file extension: Excludes all files with a specific extension (e.g., .tmp, .log).
  • Include folder: Explicitly includes a folder.
  • Include file: Explicitly includes a file.
  • Exclude expression (Regular Expression): Allows complex pattern matching using regular expressions.
  • Include expression (Regular Expression):
• How Filters are Applied:
  1. Duplicati first determines the list of all files and folders under the selected "Source Data."
  2. Then, it applies the filter rules.
  3. An "Exclude" filter removes items from the list.
  4. An "Include" filter can add items back if they were part of an initially excluded broader scope but are now explicitly included. The interaction can be complex. A common pattern is to exclude a parent folder and then include specific subfolders.
  5. The order of filters can matter, especially with overlapping rules.

Using Wildcards and Regular Expressions

• Wildcards (Simple Pattern Matching):
  • Often supported in simpler filter types (like "Exclude files whose names contain" or when specifying paths).
  • *: Matches any sequence of characters (except path separators). Example: *.tmp matches all files ending in .tmp. MyFolder/* might match all files directly within MyFolder.
  • ?: Matches any single character. Example: report_202?.txt matches report_2020.txt, report_2021.txt, etc.
  • Duplicati's wildcard support might vary by filter type; regular expressions are more powerful and consistent.
• Regular Expressions (Regex):
  • Provide a very powerful way to define complex patterns for matching file/folder names or paths.
  • Used with "Exclude expression" or "Include expression" filter types.
  • Syntax: Duplicati uses standard .NET regular expressions. Learning regex is a skill in itself.
  • Examples:
    • Exclude all files ending with .log: .*\.log$ (or \.log$ if applied to filenames only)
    • Exclude a folder named cache anywhere in the path: .*/cache/.* (to match the folder itself) or .*/cache/.* (if paths are full paths) or simply [/\\]cache[/\\] to match folder names.
    • Exclude hidden files/folders (starting with a dot) on Linux/macOS: ^.*/\..* (for full paths) or ^\..* (for filenames/foldernames within a directory).
  • Testing: Regex can be tricky. It's highly recommended to test your regular expressions using an online regex tester (e.g., regex101.com) with sample paths from your system.
• Filter Groups: Duplicati allows creating groups of filters, which can be enabled or disabled together. This is useful for managing complex sets of rules.

System Files and Temporary Files Exclusion Best Practices

It's generally a good idea to exclude common system and temporary files/folders.

• Windows:
  • Thumbs.db (image thumbnails)
  • desktop.ini (folder customization)
  • $RECYCLE.BIN or RECYCLER (Recycle Bin)
  • System Volume Information (System Restore points, indexing - usually not accessible anyway)
  • pagefile.sys, hiberfil.sys, swapfile.sys (system files)
  • C:\Windows\Temp\*
  • %TEMP%\* (user's temporary folder, e.g., C:\Users\<User>\AppData\Local\Temp\*)
• Linux:
  • lost+found (filesystem recovery directory)
  • /tmp/*
  • /var/tmp/*
  • User-specific cache folders like ~/.cache/
  • Files like .DS_Store (macOS specific, but can appear on shared drives)
• macOS:
  • .DS_Store (folder view settings)
  • ~/.Trash/ (User's Trash)
  • /.Spotlight-V100/ (Spotlight index)
  • /.fseventsd/ (file system events log)
  • ~Library/Caches/ (User's cache folder)
• Duplicati's Built-in Filters: Duplicati often comes with some default exclusion filters for common system files. Check the "Default filters" section in the global "Settings" page or when setting up a job. You can often choose to "Exclude system files" or "Exclude temporary files" via checkboxes, which apply a predefined set of rules.

Workshop Refining Backups with Filters

This workshop will guide you through adding filters to an existing backup job to exclude common temporary files and a specific large media folder you don't want to back up.

Project Exclude Temporary Files and Large Media from a Backup

The goal is to modify a backup job (e.g., My Documents Backup (Local)) to exclude all files with .tmp and .log extensions, and also exclude a specific subfolder named LargeVideos from being backed up.

Step 1 Identify Files/Folders to Exclude

• Action:
  1. Within your "Documents" folder (or whatever source you are using for your backup job), create some dummy files and folders to test the exclusions:
     • Create a file: MyData.tmp
     • Create a file: ImportantReport.docx
     • Create a file: session.log
     • Create a subfolder: MyProject
       • Inside MyProject, create a file: ProjectNotes.txt
       • Inside MyProject, create a file: debug.log
     • Create a subfolder: LargeVideos
       • Inside LargeVideos, create a file: VacationFootage.mp4 (this can be an empty file for testing)
       • Inside LargeVideos, create another file: AnotherMovie.mkv
• Guidance: This setup gives us .tmp files, .log files at different levels, and a specific folder (LargeVideos) to target for exclusion.

Step 2 Add Exclusion Filters in Duplicati

• Action:
  1. Open the Duplicati Web UI.
  2. Select your backup job (e.g., My Documents Backup (Local)) and click "Configure."
  3. Navigate to page 3, "Source Data." Below the file browser where you select source folders, you'll find the "Filters" section.
  4. We will add three filters:
     • Exclude all .tmp files:
       • Click "Add filter."
       • From the dropdown, select "Exclude file extension."
       • In the text box, type tmp (without the dot).
       • Alternatively, you could use "Exclude expression" and type .*\.tmp$ or *\.tmp (depending on whether it expects full regex or wildcard path). For "Exclude file extension", just the extension is enough.
     • Exclude all .log files:
       • Click "Add filter" again.
       • Select "Exclude file extension."
       • Type log.
     • Exclude the LargeVideos folder:
       • Click "Add filter" again.
       • Select "Exclude folder."
       • Type the name of the folder: LargeVideos
         • Important Note on Path Specificity: If you just type LargeVideos, Duplicati might exclude any folder named LargeVideos found within your source data. If you want to be more specific (e.g., exclude LargeVideos only if it's directly under Documents), you might need to use a path-based filter or a regular expression. For this workshop, LargeVideos as a folder name exclusion is fine.
         • A more robust way, especially if LargeVideos could appear elsewhere and you only want to exclude a specific one (e.g., C:\Users\You\Documents\LargeVideos), is to use "Exclude expression" with a regex matching the path. Example: [/\\]LargeVideos[/\\]$ (matches a folder named LargeVideos at the end of a path segment).
  5. Your filters list should now look something like this:
     • Exclude extension: tmp
     • Exclude extension: log
     • Exclude folder: LargeVideos
  6. Proceed to page 5 ("Options") and click "Save."
• Guidance: Duplicati applies filters relative to the selected source folders. "Exclude file extension" is quite broad. If you need finer control (e.g., exclude .log only from a specific subfolder), regular expressions become necessary.

Step 3 Test the Backup to Ensure Exclusions Work

• Action:
  1. On the Duplicati home page, click "Run now" for the modified backup job.
  2. Wait for the backup to complete.
  3. Verify by attempting a restore (the best way to check):
     • Click "Restore" for this backup job.
     • Select the latest backup version.
     • Browse the file tree in the restore interface:
       • Verify that MyData.tmp is NOT present.
       • Verify that session.log is NOT present.
       • Verify that the LargeVideos folder is NOT present (or is empty if the filter was for its contents only, but "Exclude folder" should remove the folder itself from the backup).
       • Navigate into MyProject. Verify that debug.log is NOT present.
       • Verify that ImportantReport.docx and MyProject/ProjectNotes.txt ARE present (as they were not excluded).
  4. (Alternative) Check the logs:
     • After the backup, click on the backup job, then "Show log".
     • Go to the "Remote" tab (if available and shows uploaded files) or look for file lists in the general log. This is more advanced, but sometimes you can see what was processed. The restore interface is a more direct check.
• Guidance: The restore interface is the most definitive way to confirm what Duplicati actually backed up after applying filters. If files you expected to be excluded are still there, review your filter rules for typos or logic errors. If files you expected to be included are missing, your filter might be too broad.

Step 4 (Optional) Use Regular Expressions for Complex Patterns

Let's say you want to exclude any folder named cache or Cache anywhere within your source data.

• Action (Illustrative):
  1. Go back to configuring the filters for your backup job (Page 3, "Source Data").
  2. Click "Add filter."
  3. Select "Exclude expression."
  4. In the text box, enter a regular expression. To match a folder named cache (case-insensitive) anywhere:
     • A common regex for this is (?i)[/\\]cache[/\\]
       • (?i): Makes the match case-insensitive.
       • [/\\]: Matches either a forward slash / or a backslash \ (path separators).
       • cache: The folder name.
       • The surrounding [/\\] ensures it's matching a whole directory name.
  5. Save and test as before.
• Guidance: Regular expressions offer immense power but require careful construction. Tools like regex101.com are invaluable for testing your expressions against sample file paths before implementing them in Duplicati. Remember that Duplicati's regex engine is .NET's.

By using filters effectively, you can significantly optimize your backups, making them more efficient and manageable. This workshop demonstrates the basic principles, which you can expand upon with more complex rules and regular expressions as your needs evolve.

7. Command-Line Interface (CLI) Usage

While Duplicati's Web UI is convenient for configuration and monitoring, the command-line interface (CLI) offers powerful capabilities for automation, scripting, advanced operations, and disaster recovery. The CLI allows you to perform most actions available in the Web UI and more.

Introduction to Duplicati.CommandLine.exe (or duplicati-cli)

• Location: The CLI executable is part of the standard Duplicati installation.
  • Windows: Typically found in C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe.
  • Linux: Often installed as duplicati-cli and available in your system's PATH. If not, it might be in /opt/duplicati/duplicati-cli or /usr/bin/duplicati-cli or similar. You can also invoke it directly via mono: mono /opt/duplicati/Duplicati.CommandLine.exe.
  • macOS: Located within the Duplicati.app bundle, e.g., /Applications/Duplicati.app/Contents/MacOS/duplicati-cli.
• Basic Syntax:

  Duplicati.CommandLine.exe <command> <destination> <sources> [options]
  

  Or on Linux/macOS:

  duplicati-cli <command> <destination> <sources> [options]
  

• Key Commands:
  • backup: Performs a backup.
  • restore: Restores files.
  • find: Searches for files in a backup.
  • list: Lists the contents of a backup.
  • delete: Deletes a backup set or specific versions.
  • compact: Compacts the backend data.
  • repair: Attempts to repair a broken backend or local database.
  • test: Tests the integrity of backup data.
  • list-versions: Shows available backup versions.
• Help: To get help on commands and options:

  Duplicati.CommandLine.exe help
  Duplicati.CommandLine.exe help <command> 
  

  (e.g., duplicati-cli help backup)

Running Backups and Restores from CLI

• Backup Example:

  duplicati-cli backup "sftp://user@yourserver:22//backup/destination" "/path/to/source/data" --backup-name="My CLI Backup" --encryption-module=aes -- passphrase="your_secret_passphrase" --dblock-size=100MB --retention-policy="7D:1D,4W:1W"
  

  • sftp://user@yourserver:22//backup/destination: The destination URL. Note the double slash // before the path for some protocols like SFTP.
  • /path/to/source/data: The source folder(s). Multiple sources can be listed.
  • --backup-name: Name of the backup set (important for Duplicati to manage its state).
  • --encryption-module=aes: Specifies AES encryption.
  • --passphrase="your_secret_passphrase": Your encryption passphrase. Be careful with exposing passphrases in scripts; consider environment variables or secure input methods.
  • --dblock-size=100MB: Sets the remote volume size.
  • --retention-policy="7D:1D,4W:1W": Sets the retention policy.
  • Many other options are available (filters, throttling, etc.). Use duplicati-cli help backup to see them all.
• Restore Example:

  duplicati-cli restore "sftp://user@yourserver:22//backup/destination" "/path/to/restore/location/" --passphrase="your_secret_passphrase" --version=0 --overwrite=true
  

  • /path/to/restore/location/: Where to restore the files.
  • --version=0: Restore the latest version. Use a specific number (from list-versions) or a date/time string to restore older versions.
  • --overwrite=true: Allows overwriting existing files.
  • You can also specify specific files to restore by adding them after the restore location path.

Automating Tasks with Scripts

The CLI is ideal for automation using shell scripts (Bash on Linux/macOS, PowerShell or Batch on Windows).

• Benefits:
  • Run backups outside of Duplicati's built-in scheduler (e.g., triggered by system events).
  • Integrate Duplicati operations into larger automation workflows.
  • Perform pre-backup or post-backup actions (e.g., stopping/starting services, sending custom notifications).
• Example Bash Script Snippet:

  #!/bin/bash
  BACKUP_NAME="MyImportantData"
  DESTINATION_URL="b2://your-b2-bucket-name/path/to/backup?auth-username=YOUR_B2_KEY_ID&auth-password=YOUR_B2_APP_KEY"
  SOURCE_DIR="/home/user/important_docs"
  PASSPHRASE_FILE="/home/user/.duplicati_passphrase" # Store passphrase in a protected file
  
  if [ ! -f "$PASSPHRASE_FILE" ]; then
      echo "Error: Passphrase file not found at $PASSPHRASE_FILE"
      exit 1
  fi
  
  # Read passphrase from file
  ENCRYPTION_PASSPHRASE=$(cat "$PASSPHRASE_FILE")
  
  echo "Starting backup for $BACKUP_NAME..."
  duplicati-cli backup "$DESTINATION_URL" "$SOURCE_DIR" \
      --backup-name="$BACKUP_NAME" \
      --encryption-module=aes \
      --passphrase="$ENCRYPTION_PASSPHRASE" \
      --dblock-size=50MB \
      --retention-policy="30D:1D,12M:1W" \
      --log-file="/var/log/duplicati/$BACKUP_NAME.log" \
      --log-level=Information \
      --send-mail-url="smtp://mail.example.com:587/?starttls=when-available" \
      --send-mail-subject="Duplicati Backup Report: $BACKUP_NAME %PARSEDRESULT%" \
      --send-mail-to="your-email@example.com" \
      --send-mail-from="duplicati@example.com" \
      --send-mail-username="smtp_user" \
      --send-mail-password="smtp_password" # Consider secure credential handling
  
  RESULT=$?
  if [ $RESULT -eq 0 ]; then
      echo "Backup for $BACKUP_NAME completed successfully."
  else
      echo "Backup for $BACKUP_NAME failed with exit code $RESULT."
  fi
  
  exit $RESULT
  

  Important Security Note: Storing passphrases directly in scripts is risky. Consider using environment variables, dedicated credential management tools (like HashiCorp Vault, or OS-specific credential managers), or reading from a file with restricted permissions (as shown above, ensure chmod 600 $PASSPHRASE_FILE).

Exporting Backup Configuration as CLI Command

Duplicati's Web UI provides a very handy feature to export an existing backup job's configuration directly as a CLI command.

1. In the Duplicati Web UI, select your backup job.
2. Click "Configure."
3. On the final page (page 5, "Options"), scroll down to the bottom.
4. Click the "Export..." button.
5. Choose "As Command-line."
6. A dialog will pop up showing the full Duplicati.CommandLine.exe (or duplicati-cli) command with all the configured options, source paths, and destination URL.
7. You can copy this command and use it directly in your terminal or as a basis for your scripts. Remember to handle the passphrase securely if it's included.

Workshop Automating a Backup Job with a Script

This workshop will guide you through creating a simple shell script (Bash for Linux/macOS) to run a Duplicati backup job using the CLI. We'll export an existing job configuration and then build a script around it.

Project Create a Shell Script to Run a Duplicati Backup

The goal is to create a reusable Bash script that executes a specific Duplicati backup job, with basic logging and error indication.

Step 1 Export an Existing Backup Job to CLI Command

• Action:
  1. Ensure you have at least one backup job configured in the Duplicati Web UI (e.g., My Documents Backup (Local) or the SFTP backup to Raspberry Pi).
  2. In the Duplicati Web UI, select the backup job.
  3. Click the "Configure" button/link for that job.
  4. Navigate through the wizard to the last page (page 5, "Options").
  5. Scroll to the bottom and click the "Export..." button.
  6. Select "As Command-line" from the dropdown.
  7. A text box will appear containing the full command. Copy this entire command string to your clipboard.
     • Example of exported command (will vary based on your setup):

       "/usr/bin/duplicati-cli" backup "file:///mnt/usb_backup_drive/MyFirstBackup" "/home/student/Documents/" --backup-name="My Documents Backup (Local)" --dbpath="/home/student/.config/Duplicati/ABCDE12345.sqlite" --encryption-module="aes" --compression-module="zip" --dblock-size="50mb" --passphrase="YOUR_BACKUP_JOB_PASSPHRASE" --retention-policy="5" --disable-module="console-password-input"
       

       (Note: The --passphrase will be your actual job's passphrase. Be very careful with this!)
• Guidance: This exported command is the core of your script. It contains all the specific settings for that job. The --dbpath is important as it points to the local database for this backup job.

Step 2 Create a Script (Bash)

• Action:
  1. Open a text editor (e.g., nano, vim, gedit, VS Code) on your Linux/macOS machine.
  2. Create a new file, for example, run_my_docs_backup.sh.
  3. Start with the shebang and some basic structure:

     #!/bin/bash
     
     echo "Starting Duplicati backup script..."
     
     # Paste the exported command here, but modify it for security and clarity
     # Example (modify paths and passphrase handling according to your needs)
     DUPLICATI_CLI_PATH="/usr/bin/duplicati-cli" # Adjust if different
     DESTINATION_URL="file:///mnt/usb_backup_drive/MyFirstBackup" # From your export
     SOURCE_PATH="/home/student/Documents/" # From your export
     BACKUP_NAME="My Documents Backup (Local)" # From your export
     DB_PATH="/home/student/.config/Duplicati/ABCDE12345.sqlite" # From your export
     # --- SECURITY: DO NOT PUT PLAINTEXT PASSPHRASE HERE ---
     # We will handle passphrase via an environment variable or a protected file
     # For this workshop, let's use an environment variable.
     # Ensure DUPLICATI_PASSPHRASE is set in your environment before running the script.
     # Example: export DUPLICATI_PASSPHRASE="your_actual_passphrase"
     
     if [ -z "$DUPLICATI_PASSPHRASE" ]; then
         echo "Error: DUPLICATI_PASSPHRASE environment variable is not set."
         exit 1
     fi
     
     COMMAND_OPTIONS=(
         "--backup-name=$BACKUP_NAME"
         "--dbpath=$DB_PATH"
         "--encryption-module=aes"
         "--compression-module=zip"
         "--dblock-size=50mb"
         "--passphrase=$DUPLICATI_PASSPHRASE" # Use the env var
         "--retention-policy=5" # Or your configured retention
         "--disable-module=console-password-input" # Good for scripts
         # Add any other options from your export
     )
     
     echo "Running backup job: $BACKUP_NAME"
     "$DUPLICATI_CLI_PATH" backup "$DESTINATION_URL" "$SOURCE_PATH" "${COMMAND_OPTIONS[@]}"
     
     # Capture exit code in the next step
     

  4. Modify the script:
     • Replace placeholder paths and values with those from your exported command.
     • Crucially, remove the --passphrase="YOUR_ACTUAL_PASSPHRASE" part from the direct command. We are using an environment variable DUPLICATI_PASSPHRASE for better security.
     • You can break down the command into variables for readability as shown. The COMMAND_OPTIONS array is a good way to manage many options.
  5. Save the file.
  6. Make the script executable:

     chmod +x run_my_docs_backup.sh
     

• Guidance on Passphrase Handling:
  • Environment Variable (as in script): Before running the script, set the variable: export DUPLICATI_PASSPHRASE="your_secret_passphrase". This is better than plaintext in the script but can still be visible in process lists or shell history.
  • Protected File: Store the passphrase in a file with chmod 600 permissions and read it in the script: PASSPHRASE=$(cat /path/to/secure/passphrase_file). This is generally more secure for automated scripts.
  • Avoid --passphrase argument directly in script if possible. Duplicati might also support reading it from stdin or a specific environment variable name it recognizes (e.g., DUPLICATI2_PASSPHRASE, check duplicati-cli help). For simplicity in this workshop, we use a custom env var.

Step 3 Add Error Handling and Logging (Basic)

• Action:
  1. Modify your script to include basic error checking and logging:

     #!/bin/bash
     
     echo "Starting Duplicati backup script at $(date)"
     
     DUPLICATI_CLI_PATH="/usr/bin/duplicati-cli" # Adjust
     DESTINATION_URL="file:///mnt/usb_backup_drive/MyFirstBackup" # Your export
     SOURCE_PATH="/home/student/Documents/" # Your export
     BACKUP_NAME="My Documents Backup (Local)" # Your export
     DB_PATH="/home/student/.config/Duplicati/ABCDE12345.sqlite" # Your export
     LOG_FILE="/var/log/duplicati_script_$(date +%Y%m%d).log" # Simple daily log
     
     # Check for passphrase environment variable
     if [ -z "$DUPLICATI_PASSPHRASE" ]; then
         echo "Error: DUPLICATI_PASSPHRASE environment variable is not set." | tee -a "$LOG_FILE"
         exit 1
     fi
     
     # Ensure log directory exists (example)
     # sudo mkdir -p /var/log/duplicati
     # sudo chown youruser:yourgroup /var/log/duplicati 
     # Or log to a user-writable location: LOG_FILE="/home/student/duplicati_script.log"
     
     echo "Running backup job: $BACKUP_NAME" | tee -a "$LOG_FILE"
     
     # Array for command options
     COMMAND_OPTIONS=(
         "--backup-name=$BACKUP_NAME"
         "--dbpath=$DB_PATH"
         "--encryption-module=aes"
         "--compression-module=zip"
         "--dblock-size=50mb"
         "--passphrase=$DUPLICATI_PASSPHRASE"
         "--retention-policy=5"
         "--disable-module=console-password-input"
         "--log-file=$LOG_FILE"  # Duplicati's own logging
         "--log-level=Information" # Or Warning, Error
         # Consider adding --auto-compact=true if not default
     )
     
     # Execute Duplicati command
     "$DUPLICATI_CLI_PATH" backup "$DESTINATION_URL" "$SOURCE_PATH" "${COMMAND_OPTIONS[@]}"
     EXIT_CODE=$?
     
     # Check exit code
     if [ $EXIT_CODE -eq 0 ]; then
         echo "Duplicati backup for $BACKUP_NAME completed successfully at $(date)." | tee -a "$LOG_FILE"
     elif [ $EXIT_CODE -eq 1 ]; then
         echo "Duplicati backup for $BACKUP_NAME completed with warnings at $(date)." | tee -a "$LOG_FILE"
     else
         echo "Error: Duplicati backup for $BACKUP_NAME failed with exit code $EXIT_CODE at $(date)." | tee -a "$LOG_FILE"
     fi
     
     echo "Duplicati backup script finished at $(date)." | tee -a "$LOG_FILE"
     exit $EXIT_CODE
     

  2. Explanation of Additions:
     • LOG_FILE: Defines a path for the script's own logging (and Duplicati's, using --log-file). Ensure the directory exists and is writable by the user running the script.
     • tee -a "$LOG_FILE": Prints messages to both the console and appends them to the log file.
     • EXIT_CODE=$?: Captures the exit code of the duplicati-cli command.
     • Conditional checks on EXIT_CODE: Duplicati uses specific exit codes (0 for success, 1 for success with warnings, others for errors).
     • --log-file and --log-level for Duplicati's internal logging.
• Guidance: For logging to /var/log/, the script might need to be run with sudo, or permissions on the log directory/file adjusted. Logging to a user's home directory is simpler for testing.

Step 4 Schedule the Script with Cron (Linux) or Task Scheduler (Windows)

• Action (Linux - Cron):
  1. Set the DUPLICATI_PASSPHRASE environment variable in a way that cron can access it. One common method is to define it at the top of your crontab file or within the cron job line itself (less secure for crontab file view). A better way for cron: In your script, source a file containing the variable:

     # At the top of your run_my_docs_backup.sh script:
     # Load environment variables if they exist
     # if [ -f /home/student/.duplicati_env ]; then
     # . /home/student/.duplicati_env 
     # fi
     

     Then create /home/student/.duplicati_env with export DUPLICATI_PASSPHRASE="your_secret_passphrase" and chmod 600 /home/student/.duplicati_env.
  2. Open your user's crontab for editing: crontab -e
  3. Add a line to schedule your script. For example, to run daily at 2:30 AM:

     # Example: Ensure DUPLICATI_PASSPHRASE is set if not sourcing from script
     # DUPLICATI_PASSPHRASE="your_secret_passphrase" 
     30 2 * * * /path/to/your/run_my_docs_backup.sh
     

     Or if your script sources the env file:

     30 2 * * * /path/to/your/run_my_docs_backup.sh > /home/student/cron_backup.log 2>&1
     

     (The > ... 2>&1 redirects stdout and stderr of the cron job itself to a log, separate from Duplicati's internal logging).
  4. Save and exit the crontab editor. Cron will automatically pick up the new schedule.
• Action (Windows - Task Scheduler):
  1. Open Task Scheduler.
  2. Click "Create Basic Task..." or "Create Task...".
  3. Give it a name (e.g., "Duplicati CLI Backup").
  4. Set a trigger (e.g., Daily, at a specific time).
  5. For "Action," select "Start a program."
  6. Program/script: C:\Windows\System32\cmd.exe (or powershell.exe)
  7. Add arguments: /c C:\path\to\your\script.bat (or for PowerShell: -File C:\path\to\your\script.ps1)
  8. Environment Variable for Passphrase: Set DUPLICATI_PASSPHRASE as a system or user environment variable, or use PowerShell's secure credential management features to store and retrieve it within the .ps1 script.
  9. Configure other settings as needed (e.g., run whether user is logged on or not).
• Guidance: Test your script thoroughly by running it manually from the terminal first:

  export DUPLICATI_PASSPHRASE="your_secret_passphrase" # Set it for the current session
  ./run_my_docs_backup.sh
  

  Check the output and the log files. Once it works manually, then set up the cron job or scheduled task.

You have now created a basic script to automate a Duplicati backup using its CLI, including considerations for passphrase handling and logging. This is a powerful way to integrate Duplicati into more complex environments or achieve automation beyond the Web UI's scheduler.

This concludes the intermediate level. You've learned to use diverse backup destinations, implement sophisticated scheduling and retention, refine backups with filters, and leverage the command-line interface for automation. These skills form a strong foundation for tackling advanced Duplicati topics.