| Author | Nejat Hakan |
| nejat.hakan@outlook.de | |
| PayPal Me | https://paypal.me/nejathakan |
Team Chat Matrix Synapse
Introduction
Welcome to this comprehensive guide on self-hosting your own team chat server using Matrix Synapse. In an era where digital communication is paramount, controlling your own communication platform offers significant advantages in terms of privacy, security, data ownership, and customization. Matrix is an open standard for interoperable, decentralized, real-time communication, and Synapse is the most mature reference "homeserver" implementation developed by the core Matrix team.
Think of Matrix like email – it's a protocol that allows different servers run by different people or organizations to communicate seamlessly (federation). Synapse is like the mail server software (e.g., Postfix or Exchange) that manages user accounts, message history, and communication routing for users on that specific server. Users connect to their homeserver using client applications (like Element, Fluffychat, or Nheko), similar to how you use Thunderbird or Outlook for email.
Self-hosting Synapse gives you complete control. You decide who can register, how data is stored, which features are enabled, and how your server interacts (or doesn't interact) with the wider public Matrix network (the "Fediverse"). This is particularly valuable for businesses, organizations, communities, or even families who require enhanced privacy or specific configurations not available on public servers.
This guide is structured progressively:
- Basic: Covers fundamental concepts, prerequisites, and getting your first simple Synapse instance running using Docker.
- Intermediate: Focuses on essential production configurations like setting up a reverse proxy for secure HTTPS connections, managing users, and migrating to a more robust database (PostgreSQL).
- Advanced: Explores deeper topics like federation intricacies, performance tuning with workers and monitoring, bridging to other chat networks (like IRC or Telegram), and advanced security hardening techniques.
Each section builds upon the previous one and includes practical, hands-on "Workshop" subsections. These workshops provide step-by-step instructions for applying the concepts you've learned, allowing you to build and refine your own Synapse deployment. We assume a technical aptitude suitable for university students, diving deep into the "why" behind configurations, not just the "how". Let's begin your journey into the world of decentralized, self-hosted communication!
1. Understanding Matrix and Synapse Concepts
Before diving into installation, it's crucial to grasp the fundamental concepts underpinning the Matrix ecosystem. Understanding these principles will help you make informed decisions during setup and troubleshooting later on.
The Matrix Protocol
Matrix itself isn't software; it's an open standard, a protocol specification published by the non-profit Matrix.org Foundation. It defines how real-time communication (instant messaging, VoIP/video calls, file transfers, etc.) should work in a decentralized and interoperable manner. Key characteristics include:
- Decentralization: There is no single central server or company controlling the network. Anyone can run their own server (called a "homeserver").
- Federation: Homeservers can communicate with each other, allowing users on different servers to join the same rooms and exchange messages seamlessly. If Alice is on
wonderland.comand Bob is onlooking-glass.org, they can still chat in the same Matrix room. This is analogous to how email servers interoperate. - End-to-End Encryption (E2EE): Matrix provides robust E2EE capabilities, primarily using the Olm and Megolm cryptographic ratchets (an implementation based on the Signal protocol's Double Ratchet). This ensures that only the intended participants in a conversation can decrypt messages, not even the administrators of the homeservers involved (once E2EE is enabled and active in a room).
- Real-time Communication: Designed for low-latency exchange of messages and data.
- Extensibility: The protocol is designed to be extended with new features and message types.
- Open Standard and Open Source: The specification is openly published, and the reference implementations (like Synapse) are open source, fostering transparency and community contribution.
Homeservers The Role of Synapse
A homeserver is the server software that implements the Matrix protocol. It's where user accounts reside, where room history is stored (for rooms hosted primarily on that server or where its users are participating), and it handles communication both with clients connected to it and with other homeservers via federation.
Synapse is the reference homeserver implementation developed and maintained by the core Matrix team. It is:
- Written primarily in Python.
- The most feature-complete and widely used homeserver implementation currently.
- Relatively resource-intensive compared to some newer, alternative homeservers (like Dendrite or Conduit), but offers stability and extensive features.
- The focus of this guide due to its maturity and prevalence.
When you self-host Synapse, you are running your own instance of this server software, defining your own corner of the Matrix network (e.g., yourdomain.com).
Clients Connecting to Your Homeserver
Users interact with Matrix via client applications. These clients connect to a specific homeserver to send/receive messages, manage rooms, and handle encryption keys. Popular clients include:
- Element: The flagship client developed by Element (formerly New Vector), the company that employs many core Matrix developers. Available for Web, Desktop (Windows, macOS, Linux), Android, and iOS. Known for its feature richness.
- Fluffychat: A user-friendly client with a focus on ease of use, available for Web, Android, iOS, and Desktop (Linux).
- Nheko Reborn: A native desktop client (Windows, macOS, Linux) focusing on performance and a clean interface.
- SchildiChat: A fork of Element with some UI/UX tweaks and additional features.
- ...and many others, including terminal-based clients and specialized bots.
Clients implement the Client-Server API specified by the Matrix protocol to communicate with the user's homeserver.
Identity Servers (Optional)
Identity Servers (IS) provide a way to map third-party identifiers (3PIDs) like email addresses or phone numbers to Matrix IDs (MXIDs, e.g., @alice:wonderland.com). This allows users to discover each other using familiar contact information.
- Using an IS is optional. You can run a Matrix homeserver without one.
- If used, they introduce a potential privacy consideration, as the IS learns the mapping between MXIDs and 3PIDs.
- Historically,
matrix.orgran the primary public IS, but others exist (e.g.,vector.im). You can also self-host an IS (likesydent, the reference implementation), but this adds complexity. - For a private team chat server where users know each other's Matrix IDs, an IS is often unnecessary.
Integration Managers (Optional)
Integration Managers provide user-friendly interfaces for managing bots, bridges, and widgets within Matrix rooms. Element integrates with integration managers (like the default Dimension manager hosted by Element) to offer features like adding sticker packs, configuring bridges, or embedding tools via widgets.
- Like Identity Servers, these are optional.
- You can self-host your own integration manager if desired, but it's another component to manage.
- For basic chat functionality, an integration manager isn't required.
Workshop Exploring the Matrix Ecosystem
This workshop aims to familiarize you with the practical side of the Matrix ecosystem before you build your own server.
Objective: Experience Matrix as a user on the public network and identify key components.
Steps:
- Explore the Matrix.org Website:
- Navigate to https://matrix.org/ in your web browser.
- Spend some time exploring the "Try Matrix Now", "Clients", "Servers", and "Bridges" sections.
- Task: Identify at least three different Matrix clients (besides Element), one alternative homeserver (besides Synapse), and two types of bridges (e.g., IRC, Slack). Note down their names.
- Create a Test Account:
- Go to the Element Web client at https://app.element.io/.
- Click on "Create Account".
- Choose the default
matrix.orghomeserver option. - Follow the registration process to create a free account. Remember your username and password, but treat this as a test account – don't use a critical password. Secure your account recovery key/passphrase when prompted.
- Join Public Rooms:
- Once logged in, you'll see the Element interface.
- Use the "Explore public rooms" button (often a compass icon or "+" next to Rooms) or the search bar.
- Search for and join a few well-known public rooms to see how conversations look. Good examples include:
#matrix:matrix.org(Matrix HQ)#synapse:matrix.org(Synapse Admins room)#element-web:matrix.org(Element Web/Desktop room)
- Observe the format of user IDs (
@username:homeserver.domain) and room aliases (#roomname:homeserver.domain).
- Send a Message:
- In one of the rooms you joined, try sending a simple "Hello!" message.
- Notice how messages appear in real-time.
Outcome: You should now have a better feel for the Matrix user experience, understand the components listed on the official site, and recognize the structure of Matrix IDs and room aliases. This context will be invaluable as you set up your own homeserver.
2. Prerequisites and Planning
Setting up a self-hosted Synapse instance requires careful planning and ensuring your environment meets the necessary requirements. Rushing this stage often leads to problems later on.
Hardware Requirements
Synapse's resource usage depends heavily on the number of users, their activity level, the number of rooms they join (especially large, federated rooms), and whether features like end-to-end encryption are heavily used.
- CPU:
- Minimum (Testing/Few Users): 1 modern CPU core.
- Recommended (Small Team < 50 active users): 2-4 modern CPU cores.
- Larger Deployments: Scales significantly with concurrent users and federation activity. Start with 4+ cores and monitor.
- RAM: Synapse can be memory-intensive, especially with many active connections or large rooms.
- Minimum (Testing/Few Users): 1-2 GB RAM dedicated to Synapse.
- Recommended (Small Team < 50 active users): 4 GB RAM dedicated to Synapse. Add more if using a resource-heavy database like PostgreSQL on the same machine. 8GB total system RAM is a good starting point.
- Larger Deployments: 8GB+ RAM for Synapse is common. Monitor usage closely.
- Disk: Disk space is needed for the operating system, Synapse itself, the database, and stored media (uploaded files, images, videos). Media storage can grow significantly over time.
- Minimum (Testing): 20-30 GB (will fill up quickly with media).
- Recommended (Small Team): 50-100 GB SSD. Using an SSD is highly recommended for database performance.
- Larger Deployments: Plan for hundreds of GBs or even TBs, potentially using separate storage for media. Database I/O performance is critical.
Note: These are estimates. Always monitor your resource usage after setup.
Software Requirements
- Operating System: A modern Linux distribution is strongly recommended. Debian (Stable) or Ubuntu LTS (Long Term Support) are excellent choices due to their stability, large communities, and extensive documentation. This guide will primarily assume a Debian/Ubuntu-based environment.
- Docker and Docker Compose: While manual installation is possible, using Docker greatly simplifies dependency management, updates, and reproducibility. We will focus on the Docker-based approach. You'll need
dockeranddocker-composeinstalled. - Reverse Proxy (Recommended): Software like Nginx, Caddy, or Apache is needed to handle HTTPS/TLS termination and route traffic correctly to Synapse. Covered in the Intermediate section.
- Firewall: A properly configured firewall (like
ufwon Ubuntu/Debian orfirewalldon CentOS/Fedora) is essential for security.
Domain Name and DNS Configuration
You need a registered domain name (or a subdomain) that you control. This domain name forms the basis of your users' Matrix IDs (e.g., @user:yourdomain.com) and your server's identity on the Matrix network.
You will need access to your domain's DNS settings (usually via your domain registrar or DNS hosting provider like Cloudflare, Namecheap, GoDaddy, etc.). You must configure the following DNS records:
- A Record (IPv4): Points your chosen hostname (e.g.,
matrix.yourdomain.comor justyourdomain.com) to the public IPv4 address of your server. - AAAA Record (IPv6 - Optional but Recommended): Points your hostname to the public IPv6 address of your server, if available.
-
SRV Record (for Federation - Recommended): This record tells other Matrix servers how to find your Synapse server for server-to-server communication, which typically runs on port 8448. This avoids exposing the Synapse port directly via the main domain A/AAAA record.
- Service:
_matrix-fed._tcp(or_matrix._tcpfor legacy/client fallback, see Intermediate section) - Proto:
_tcp - Name: Your server name (e.g.,
yourdomain.com) - Priority: Lower number means higher preference (e.g.,
10). - Weight: Relative weight for records with the same priority (e.g.,
0). - Port:
8448(the default federation port for Synapse). - Target: The hostname that has the A/AAAA record pointing to your server (e.g.,
matrix.yourdomain.com).
- Service:
Important: DNS changes can take time to propagate (minutes to hours, sometimes up to 48 hours). You can use tools like dig or online DNS checkers to verify propagation.
Firewall Configuration
Your server's firewall (and any network firewalls between your server and the internet) must allow incoming traffic on specific ports:
- TCP 80 (HTTP): Required for Let's Encrypt certificate validation (HTTP-01 challenge).
- TCP 443 (HTTPS): The standard port for secure client-to-server communication via your reverse proxy. Also used for server-to-server federation if using
.well-knowndelegation instead of SRV records. - TCP 8448 (Matrix Federation): The default port for server-to-server federation traffic if using SRV records. This should ideally not be handled by the reverse proxy initially, but directly by Synapse (though TLS is still required).
You will configure your firewall to allow traffic on these ports, typically only from the internet (Any or 0.0.0.0/0).
TLS/SSL Certificates
Secure communication via HTTPS is mandatory for Matrix. Clients and other servers will refuse to connect to your homeserver over plain HTTP. You need a valid TLS/SSL certificate for your domain.
- Let's Encrypt: The highly recommended solution. Provides free, automated certificates. Tools like
Certbotintegrate well with common web servers/reverse proxies. - Commercial Certificates: You can purchase certificates from traditional Certificate Authorities (CAs).
- Self-Signed Certificates: Not suitable for production or federation. Only for very limited internal testing, as clients and other servers will reject them.
We will cover setting up Let's Encrypt with a reverse proxy in the Intermediate section. For the initial basic setup, we might run Synapse directly, but it won't be securely accessible from the wider internet or usable for federation until TLS is properly configured.
Workshop Setting Up Your Environment
This workshop guides you through the essential preparatory steps involving DNS and basic server/firewall considerations.
Objective: Configure DNS records for your Synapse server and understand the necessary network port allowances.
Prerequisites:
- A registered domain name (e.g.,
yourdomain.com). - Access to your domain's DNS management panel.
- A server (Virtual Private Server (VPS), cloud instance, or even a local machine with a public IP address if you intend to federate). Note down its public IPv4 address (and IPv6 if available).
- Access to configure the firewall on the server and/or your network router if the server is behind NAT.
Steps:
- Choose Your Hostname: Decide on the fully qualified domain name (FQDN) clients will use to connect and that will host Synapse. Common choices:
- A subdomain:
matrix.yourdomain.com(Recommended, cleaner separation) - The apex domain:
yourdomain.com(Possible, but can complicate hosting other services like a website on the same domain) - For this workshop, let's assume you chose
matrix.yourdomain.comas the FQDN andyourdomain.comas the logical "server name" for Matrix IDs (@user:yourdomain.com).
- A subdomain:
- Configure DNS A/AAAA Records:
- Log in to your DNS provider's control panel.
- Navigate to the DNS management section for
yourdomain.com. - Create an A record:
- Host/Name:
matrix(or@if using the apex domain) - Value/Points to: Your server's public IPv4 address.
- TTL (Time To Live): Default is usually fine (e.g., 1 hour or Auto).
- Host/Name:
- (Optional) Create an AAAA record if you have an IPv6 address:
- Host/Name:
matrix(or@) - Value/Points to: Your server's public IPv6 address.
- TTL: Default.
- Host/Name:
- Configure DNS SRV Record for Federation:
- Still in your DNS management panel, look for the option to add an SRV record. The input fields might vary slightly.
- Create an SRV record for federation:
- Service:
_matrix-fed(Some providers might require_matrix-fed._tcp) - Protocol:
_tcp - Name:
@oryourdomain.com(This specifies the domain part of Matrix IDs) - Priority:
10 - Weight:
0 - Port:
8448 - Target/Host:
matrix.yourdomain.com(The FQDN with the A/AAAA records) - TTL: Default.
- Service:
- Configure DNS SRV Record for Client Connections (Optional but Recommended):
- Create another SRV record for client connections:
- Service:
_matrix(Some providers might require_matrix._tcp) - Protocol:
_tcp - Name:
@oryourdomain.com - Priority:
10 - Weight:
0 - Port:
443(Clients should connect over standard HTTPS) - Target/Host:
matrix.yourdomain.com - TTL: Default.
- Service:
- Create another SRV record for client connections:
- Verify DNS Propagation (Wait ~15-60 minutes):
- Open a terminal or command prompt on your local machine.
- Use the
digcommand (on Linux/macOS) ornslookup(on Windows/Linux/macOS). Replacematrix.yourdomain.comandyourdomain.comwith your actual domain names.# Check A record dig A matrix.yourdomain.com +short # Check AAAA record (if configured) dig AAAA matrix.yourdomain.com +short # Check Federation SRV record dig SRV _matrix-fed._tcp.yourdomain.com +short # Check Client SRV record dig SRV _matrix._tcp.yourdomain.com +short # --- OR using nslookup --- # Check A/AAAA records nslookup matrix.yourdomain.com # Check SRV record (set query type first) nslookup -query=SRV _matrix-fed._tcp.yourdomain.com nslookup -query=SRV _matrix._tcp.yourdomain.com - Verify that the commands return the IP address(es) and SRV record details you configured. If not, wait longer or double-check your DNS settings. You can also use online tools like Google's Dig tool (https://toolbox.googleapps.com/apps/dig/).
- Plan Firewall Rules (Conceptual):
- Identify the firewall software on your server (e.g.,
ufw,firewalld,iptables) or your network router's firewall settings if applicable. - Mentally (or actually, if comfortable) prepare the rules to allow incoming TCP traffic on ports 80, 443, and 8448. For example, using
ufw: - Do not blindly run these commands yet if you are unsure. We will implement firewall rules more concretely when needed, but understand which ports need to be open.
- Identify the firewall software on your server (e.g.,
Outcome: You should have correctly configured DNS records pointing to your server, verified their propagation, and understand which network ports need to be accessible for Synapse to function correctly with clients and federation. This lays the groundwork for the actual installation.
3. Basic Synapse Installation with Docker
Now that the prerequisites are handled, we can proceed with installing Synapse. Using Docker and Docker Compose is the recommended method as it encapsulates Synapse and its Python dependencies, simplifying setup and upgrades.
Why Docker?
- Isolation: Synapse runs in a container, isolated from your host operating system and other applications. This prevents dependency conflicts.
- Dependency Management: The Docker image includes the correct versions of Python, libraries, and system dependencies needed by Synapse. You don't need to manage these manually on your host.
- Reproducibility: Docker Compose uses a configuration file (
docker-compose.yml) to define the entire service stack (Synapse, potentially a database, reverse proxy, etc.). This makes your setup easy to replicate or move to another server. - Simplified Updates: Updating Synapse often involves just pulling a newer Docker image and restarting the container.
- Consistency: Ensures Synapse runs in a predictable environment, regardless of the host OS specifics (as long as Docker is supported).
Choosing a Docker Image
The official Synapse Docker image is maintained by the Matrix core team and is the recommended choice.
- Image Name:
matrixdotorg/synapse - Tags: You can specify versions (e.g.,
v1.70.0) or uselatest. It's generally recommended to pin to a specific version tag for stability in production, updating deliberately rather than automatically gettinglatest. Find available tags on Docker Hub: https://hub.docker.com/r/matrixdotorg/synapse/tags
Generating the Initial Configuration
Synapse is configured primarily through a YAML file, typically named homeserver.yaml. Before you can run Synapse, you need to generate a base configuration file tailored to your server name.
The Synapse Docker image provides a command to do this:
# Create a directory to hold Synapse data and configuration
mkdir -p /path/to/your/synapse/data
cd /path/to/your/synapse/data
# Run the image with the 'generate' command
# Replace yourdomain.com with your actual server name (the domain part of user IDs)
# Replace matrix.yourdomain.com with the FQDN pointing to your server if different
# Use --report-stats=yes or --report-stats=no
docker run -it --rm \
-v $(pwd):/data \
-e SYNAPSE_SERVER_NAME=yourdomain.com \
-e SYNAPSE_REPORT_STATS=yes \
matrixdotorg/synapse:latest generate
Explanation:
docker run -it --rm: Runs a command in a new container.-itallows interaction (though not strictly needed forgenerate),--rmremoves the container after it exits.-v $(pwd):/data: Mounts the current directory (which should be/path/to/your/synapse/data) inside the container at/data. This is where the generatedhomeserver.yamlwill be placed.-e SYNAPSE_SERVER_NAME=yourdomain.com: Sets an environment variable inside the container. Thegeneratecommand uses this to set theserver_namein the configuration file. This MUST match the domain part you want for your user IDs (e.g.,@user:yourdomain.com) and should be the domain covered by your SRV records or.well-knownfile later.-e SYNAPSE_REPORT_STATS=yes: Sets another environment variable.yesenables sharing anonymous statistics with the Matrix developers to help improve Synapse.nodisables it. Choose according to your preference.matrixdotorg/synapse:latest generate: Specifies the image to use and the command to run inside it (generate).
After running this, you will find a homeserver.yaml file (and potentially log config and signing key files) inside your /path/to/your/synapse/data directory.
Basic Configuration Parameters
Open the generated homeserver.yaml file in a text editor. It's heavily commented, which is helpful. For a basic setup, ensure these are correct:
server_name: Should match what you provided during generation (e.g.,yourdomain.com). Do not change this lightly after setup.public_baseurl: (May not be strictly needed if using a reverse proxy later, but good to review). This should ideally be the HTTPS URL clients will connect to (e.g.,https://matrix.yourdomain.com). For now, it might be unset or point to HTTP.listeners: This section defines where Synapse listens for connections.- The default configuration usually includes listeners on port 8008 (HTTP client-server API) and 8448 (HTTPS federation API).
- For our initial Docker setup without a reverse proxy yet, we might expose port 8008 directly. This is insecure for public access.
database: By default, it will be configured to use SQLite. The path/data/homeserver.dbis relative to the container. Since we mounted our host directory/path/to/your/synapse/datato/datain the container, the SQLite file will appear on our host at/path/to/your/synapse/data/homeserver.db.log_config: Points to the logging configuration file.media_store_path: Path inside the container where uploaded media will be stored (e.g.,/data/media_store). This will map to/path/to/your/synapse/data/media_storeon the host.signing_key_path: Path to the server's signing key, used for federation.registration_shared_secret: (May be commented out) Used for registration via tools likeregister_new_matrix_user. Generate a strong secret if you plan to use this.enable_registration: By default, this might befalse. If you want to allow registration (even if only via the command-line tool initially), you might need to set it totrueor configure specific registration methods later. For initial setup, leaving itfalseand using the command-line tool with a shared secret (if configured) or admin commands is safer.
Important Security Note: The default homeserver.yaml exposes many ports and options. We will refine this significantly when adding the reverse proxy. For now, we focus on getting it running in a basic, locally accessible way.
Running Synapse with Docker Compose
Using docker-compose is much more convenient than raw docker run commands for managing services with persistent data and specific configurations.
Create a file named docker-compose.yml in a convenient location (e.g., one level above your Synapse data directory):
version: '3.8'
services:
synapse:
image: matrixdotorg/synapse:latest # Or pin to a specific version like v1.70.0
container_name: matrix_synapse
restart: unless-stopped
volumes:
# Mount the directory containing homeserver.yaml and other data
# Replace /path/to/your/synapse/data with the actual path on your host
- /path/to/your/synapse/data:/data
ports:
# Expose the HTTP listener port (8008) to the host
# WARNING: Exposing 8008 directly to the internet is insecure.
# This is for initial testing ONLY. We will remove this mapping
# once a reverse proxy is configured.
- "127.0.0.1:8008:8008" # Bind to localhost only for safety initially
# If you NEED federation immediately (without reverse proxy, less secure):
# - "0.0.0.0:8448:8448" # Expose federation port (requires TLS configuration within Synapse)
environment:
# Synapse needs to know its own external address for federation etc.
# Even if using localhost binding above, these help it construct URLs.
# Set these if they differ from the defaults Synapse might guess.
# - SYNAPSE_PUBLIC_HOSTNAME=matrix.yourdomain.com
# - SYNAPSE_LISTEN_ADDRESS=0.0.0.0 # Address Synapse listens on *inside* the container
- TZ=Etc/UTC # Set your server's timezone
# Optional: If you need to run Synapse as a specific user/group on the host
# user: "1000:1000" # Replace with your user/group ID if needed
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8008/health"]
interval: 30s
timeout: 10s
retries: 3
volumes:
# Define the volume if you prefer Docker-managed volumes (alternative to bind mount)
# synapse_data:
# networks:
# matrix_network:
# driver: bridge
Explanation:
version: '3.8': Specifies the Docker Compose file format version.services: synapse:: Defines the Synapse service.image: matrixdotorg/synapse:latest: Specifies the Docker image to use.container_name: matrix_synapse: Assigns a specific name to the container.restart: unless-stopped: Ensures the container restarts automatically unless manually stopped.volumes:: Defines data persistence.- /path/to/your/synapse/data:/data: This is a bind mount. It maps the directory/path/to/your/synapse/dataon your host machine to the/datadirectory inside the container. This is where Synapse expects its configuration and data based on thehomeserver.yamldefaults. Make sure the host path exists and has appropriate permissions.
ports:: Maps ports between the host and the container (HOST:CONTAINER)."127.0.0.1:8008:8008": This maps port 8008 inside the container to port 8008 on the host machine, but only listens on the host's loopback interface (127.0.0.1). This means Synapse will only be accessible from the server itself, not from the outside network. This is a safer default before TLS is set up. If you were to use"8008:8008"or"0.0.0.0:8008:8008", it would be exposed externally (insecurely!).- The
8448port mapping is commented out. We typically won't expose this directly in the final setup; the reverse proxy or direct federation (with TLS handled by Synapse) will manage access.
environment:: Sets environment variables within the container.TZis good practice. Others can override specific Synapse settings if needed, though configuration viahomeserver.yamlis generally preferred.healthcheck:: Defines a command Docker can run periodically to check if the Synapse service is healthy.
To run Synapse:
- Navigate to the directory containing your
docker-compose.ymlfile. - Start the service in detached mode (runs in the background):
- Check the container status:
You should see the
matrix_synapsecontainer running. - View the logs to check for errors:
Look for messages indicating Synapse has started successfully and is listening on the configured ports (inside the container). Press
Ctrl+Cto stop following the logs.
Verifying the Installation
Since we bound port 8008 to 127.0.0.1, you can only test from the server itself.
Use curl from the server's terminal:
curl http://localhost:8008/_matrix/static/
# Or to check the client API version endpoint:
curl http://localhost:8008/_matrix/client/versions
You should get an HTML response (likely a "Not Found" page for /_matrix/static/`, which is fine) or a JSON response listing API versions. This confirms Synapse is running and responding on port 8008 locally.
Important: At this stage, your Synapse instance is:
- Running.
- Using an SQLite database.
- Accessible only via HTTP on
localhost:8008. - Not accessible to the public internet or other Matrix servers (no federation).
- Likely has user registration disabled by default.
The next steps involve securing it with a reverse proxy and TLS, configuring user management, and potentially switching the database.
Workshop Your First Synapse Instance
This workshop guides you through installing Docker, generating the configuration, setting up Docker Compose, and running your basic Synapse instance.
Objective: Get a minimal Synapse server running locally using Docker Compose.
Prerequisites:
- A Linux server (Debian/Ubuntu recommended) meeting the minimum hardware specs.
- SSH access to the server.
- The domain name preparations from the previous workshop should be conceptually complete (though not strictly needed for this local test).
Steps:
- Install Docker and Docker Compose:
- Log in to your server via SSH.
- Follow the official Docker installation instructions for your Linux distribution. For Debian/Ubuntu:
# Update package lists sudo apt update # Install prerequisites sudo apt install -y apt-transport-https ca-certificates curl gnupg lsb-release software-properties-common # Add Docker's official GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg # Set up the stable repository echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null # Install Docker Engine sudo apt update sudo apt install -y docker-ce docker-ce-cli containerd.io # Verify Docker installation sudo docker run hello-world - Install Docker Compose (v2 is integrated as a plugin):
- Add your user to the
dockergroup to run Docker commands withoutsudo(requires logout/login ornewgrp docker):
- Create Synapse Data Directory:
- Choose a location for Synapse's configuration and data.
/opt/synapseor/srv/synapseare common choices. - Create the directory:
- Choose a location for Synapse's configuration and data.
- Generate
homeserver.yaml:- Navigate into the parent directory:
cd /srv/synapse - Run the Docker command to generate the config, replacing
yourdomain.comwith your actual server name (e.g.,example.org) and choosingyesornofor stats reporting. - Verify
homeserver.yamland other files are created in/srv/synapse/data. You might needsudo ls -l /srv/synapse/datato see them. - (Optional but Recommended) Generate a registration secret (if you didn't when prompted by
generateor want to change it):- Edit
/srv/synapse/data/homeserver.yaml(e.g.,sudo nano /srv/synapse/data/homeserver.yaml). - Find the
registration_shared_secretline (it might be commented out). - Uncomment it and set it to a strong, randomly generated password. You can generate one with:
openssl rand -hex 32 - Save the file. Also ensure
enable_registration: trueis set temporarily if you want to use the command-line tool immediately (we'll disable public registration later).
- Edit
- Navigate into the parent directory:
-
Create
docker-compose.yml:- Stay in the
/srv/synapsedirectory (or wherever you want to keep the compose file). -
Create the
docker-compose.ymlfile (e.g.,sudo nano docker-compose.yml) with the following content, making sure to replace/srv/synapse/datawith your actual data path:version: '3.8' services: synapse: image: matrixdotorg/synapse:latest # Consider pinning e.g. matrixdotorg/synapse:v1.90.0 container_name: matrix_synapse restart: unless-stopped volumes: # Ensure this path matches where homeserver.yaml is located - /srv/synapse/data:/data ports: # !! BINDING TO LOCALHOST ONLY FOR INITIAL TEST !! - "127.0.0.1:8008:8008" environment: - TZ=Etc/UTC # Adjust to your timezone e.g. Europe/Berlin healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8008/health"] interval: 30s timeout: 10s retries: 3 -
Save the file.
- Stay in the
-
Start Synapse:
- Run Docker Compose:
- Check Logs and Status:
- Verify the container is running:
- Check the logs for successful startup messages:
- Register Your First User:
- If you enabled registration and set a
registration_shared_secretinhomeserver.yaml, you can use theregister_new_matrix_usercommand-line tool. You need to run this inside the running Synapse container or use a separate instance of the image connected to the same data. - Execute the command via
docker exec:Note: If you didn't set# Replace <USERNAME> and <PASSWORD> with your desired admin user credentials # Use -a for admin rights, -p for password, -u for username, -c for config file path inside container # The URL http://localhost:8008 tells the tool where to reach Synapse (inside the container network) sudo docker exec matrix_synapse register_new_matrix_user \ -c /data/homeserver.yaml \ http://localhost:8008 \ -u <USERNAME> \ -p <PASSWORD> \ -a # Make this user an adminregistration_shared_secretorenable_registration: true, this command might fail. You can alternatively create users via the Synapse Admin API later. - If successful, it will print the full Matrix ID (e.g.,
@youruser:yourdomain.com). Save this ID and password.
- If you enabled registration and set a
- Connect with Element Web (via SSH Tunnel - Advanced):
- Since Synapse is only listening on
localhost:8008on the server, you can't connect directly from your browser yet. To test connectivity, you can use an SSH tunnel (this is optional and advanced). - On your local machine (not the server), run: This command forwards connections to port 8008 on your local machine to port 8008 on the server's localhost interface, through the SSH connection. Keep this SSH session running.
- Now, open your web browser on your local machine and navigate to
http://localhost:8008. - You should see a Synapse confirmation page ("It works! Synapse is running").
- Open Element Web (https://app.element.io/).
- Click "Sign In".
- Click "Edit" next to the Homeserver setting.
- Enter
http://localhost:8008as your homeserver URL (only works because of the SSH tunnel). - Enter the username and password you registered in step 7.
- Click "Sign In".
- If successful, you are connected to your own Synapse server! You won't see other users or public rooms yet.
- Close the SSH tunnel when finished testing.
- Since Synapse is only listening on
Outcome: You have successfully deployed a basic Synapse instance using Docker Compose, verified it's running, registered an admin user, and (optionally) connected to it using Element Web via an SSH tunnel. This instance is not yet ready for production use but serves as a vital foundation. Remember to disable enable_registration: true in homeserver.yaml again if you set it temporarily.
4. Configuring a Reverse Proxy and TLS
Your basic Synapse instance is running, but it's only accessible locally via HTTP. This is neither secure nor practical for real-world use. A reverse proxy is essential to:
- Handle TLS/SSL Termination: Provide secure HTTPS encryption for all client and server communication.
- Expose Synapse Securely: Act as the public gateway to your Synapse instance, hiding the internal
8008port. - Simplify Port Management: Clients and servers connect to standard ports (443), while the proxy routes traffic internally.
- Enable Features: Required for certain configurations like
.well-knowndelegation and can facilitate load balancing later.
We will focus on Nginx as the reverse proxy due to its popularity, performance, and excellent documentation. We'll also use Let's Encrypt via the Certbot tool to obtain free, trusted TLS certificates automatically.
Why a Reverse Proxy?
Imagine your Synapse server running on port 8008 inside its container. Without a reverse proxy, you'd have to expose port 8008 directly to the internet. This is bad because:
- HTTP is Insecure: Passwords and message data would be sent unencrypted.
- Non-Standard Port: Clients and servers expect to connect via standard HTTPS port 443.
- Direct Exposure: Exposes Synapse's internal workings more directly to potential attackers.
A reverse proxy sits in front of Synapse. All external traffic comes to the proxy (typically on port 443). The proxy handles the TLS encryption/decryption and then forwards the requests (usually as plain HTTP) to Synapse listening on its internal port (e.g., localhost:8008).
Internet ---> Firewall (Allow 443) ---> Reverse Proxy (Nginx on Port 443) ---+
^ | (HTTPS)
| |
+-------> Nginx handles TLS Termination <----------------------------------+
Forwards request (HTTP) to Synapse on localhost:8008
Synapse (Docker Container) <------------------+
(Listens on 127.0.0.1:8008)
Choosing Nginx
Nginx is a high-performance, open-source web server, reverse proxy, load balancer, and cache. It's known for its stability, efficiency (low memory footprint), and powerful configuration options. Certbot has excellent integration with Nginx, making TLS setup straightforward.
Configuring Nginx for Synapse
We need to configure Nginx to listen for requests for your Matrix domain (e.g., matrix.yourdomain.com) and proxy them correctly to the Synapse container. Matrix requires specific paths to be routed:
/_matrix/...: Client-Server and Server-Server APIs./_synapse/client/...: Older client API path (still needed for some clients/bots)./_media/...: Media repository requests.
Here's a sample Nginx configuration snippet, typically placed in /etc/nginx/sites-available/matrix.yourdomain.com.conf (replace matrix.yourdomain.com and yourdomain.com with your actual domains):
# /etc/nginx/sites-available/matrix.yourdomain.com.conf
server {
# Listen on port 80 for ACME challenges (Let's Encrypt)
listen 80;
listen [::]:80;
server_name matrix.yourdomain.com; # Your FQDN
# Redirect all HTTP traffic to HTTPS
location / {
return 301 https://$host$request_uri;
}
# Location for Let's Encrypt validation files
location /.well-known/acme-challenge/ {
root /var/www/html; # Or another directory accessible by Nginx
allow all;
}
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name matrix.yourdomain.com; # Your FQDN
# TLS configuration (Certbot will manage these paths)
ssl_certificate /etc/letsencrypt/live/matrix.yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/matrix.yourdomain.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf; # Recommended SSL parameters
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # Diffie-Hellman parameters
# Optional: Add security headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options SAMEORIGIN always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "no-referrer" always; # Adjust if widgets need referrer
# Increase max upload size (adjust as needed)
client_max_body_size 128M; # Example: 128 Megabytes
location ~ ^(/_matrix|/_synapse/client|/_media) {
# Proxy requests to Synapse listening on localhost:8008
# Assumes Synapse container port 8008 is mapped to host's 127.0.0.1:8008
proxy_pass http://127.0.0.1:8008;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
# Allow large uploads here too
client_max_body_size 128M;
# Required for Synapse websockets (for client push/notifications)
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400s; # Long timeout for active connections
}
# Optional: Serve a landing page at the root (/)
# location / {
# root /var/www/matrix-landing; # Example path
# index index.html;
# }
}
# Federation: .well-known delegation (Alternative/Supplement to SRV records)
# This separate server block handles requests specifically for your server_name
# (e.g., yourdomain.com) if it differs from your matrix FQDN.
# It tells clients and servers where the *actual* Synapse API is located.
# You only need this if your server_name (yourdomain.com) is different from
# the hostname Synapse is served on (matrix.yourdomain.com).
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name yourdomain.com; # Your Matrix server_name (domain part of MXIDs)
# TLS Configuration (Needs its OWN certificate for yourdomain.com)
ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
# Optional Security Headers (repeat as above)
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
# ... other headers
# Redirect HTTP to HTTPS for the base domain
if ($scheme = http) {
return 301 https://$host$request_uri;
}
# Serve the .well-known files for Matrix delegation
location /.well-known/matrix/server {
return 200 '{"m.server": "matrix.yourdomain.com:443"}';
add_header Content-Type application/json;
# Optional: Allow CORS for web clients to discover the homeserver.
add_header 'Access-Control-Allow-Origin' '*' always;
add_header "Access-Control-Allow-Methods" "GET, POST, OPTIONS" always;
add_header "Access-Control-Allow-Headers" "Origin, X-Requested-With, Content-Type, Accept, Authorization" always;
}
location /.well-known/matrix/client {
return 200 '{ "m.homeserver": { "base_url": "https://matrix.yourdomain.com" }, "m.identity_server": { "base_url": "https://vector.im" } }'; # Optional IS example
add_header Content-Type application/json;
add_header 'Access-Control-Allow-Origin' '*' always;
add_header "Access-Control-Allow-Methods" "GET, POST, OPTIONS" always;
add_header "Access-Control-Allow-Headers" "Origin, X-Requested-With, Content-Type, Accept, Authorization" always;
}
# Optional: Redirect root of base domain to Element or a landing page
# location / {
# return 301 https://app.element.io/; # Redirect to Element Web
# }
}
Key points:
- Two
serverblocks for HTTP->HTTPS: The firstserverblock listens on port 80 simply to handle Let's Encrypt challenges and redirect users to HTTPS. - Main HTTPS
serverblock: Listens on 443, defines the TLS certificate paths (Certbot will create these), includes recommended SSL settings, sets security headers, and defines the proxy logic. location ~ ^(/_matrix|/_synapse/client|/_media): This regular expression matches the required Matrix API paths.proxy_pass http://127.0.0.1:8008;: Forwards matching requests to Synapse (assuming the Docker port mapping is127.0.0.1:8008:8008).proxy_set_header: Passes necessary information like the original host and protocol to Synapse. Essential for Synapse to generate correct URLs.client_max_body_size: Increases the maximum allowed size for file uploads. Adjust as needed.- Websocket Headers:
UpgradeandConnectionheaders are crucial for real-time communication. - Federation
.well-knownblock (Optional but Recommended): This separateserverblock handles requests foryourdomain.com(yourserver_name). It serves JSON files at specific.well-knownpaths./.well-known/matrix/server: Tells other servers that the actual API foryourdomain.comcan be found atmatrix.yourdomain.com:443. This is an alternative or supplement to using SRV DNS records for federation. Using.well-knownoften simplifies firewall rules (only need 443 open)./.well-known/matrix/client: Tells clients that when they typeyourdomain.comas their homeserver, the actual API is athttps://matrix.yourdomain.com. It can also suggest an identity server.- This block needs its own TLS certificate for
yourdomain.com. - CORS headers (
Access-Control-Allow-Originetc.) are needed for web clients to read these files directly.
Setting Up Let's Encrypt with Certbot
Certbot is a tool that automates the process of obtaining and renewing Let's Encrypt TLS certificates. It integrates well with Nginx.
- Install Certbot and the Nginx Plugin:
- Ensure Firewall Allows Port 80: Let's Encrypt's ACME protocol often uses the HTTP-01 challenge, which requires your server to be reachable on port 80 from the internet.
- Create Nginx Configuration File:
- Create the file
/etc/nginx/sites-available/matrix.yourdomain.com.confwith the Nginx configuration provided above (adjust domain names!). - If you are using the
.well-knownapproach, also create/etc/nginx/sites-available/yourdomain.com.conf(or combine them if appropriate, ensuringserver_namedirectives are correct).
- Create the file
- Enable the Nginx Site(s):
sudo ln -s /etc/nginx/sites-available/matrix.yourdomain.com.conf /etc/nginx/sites-enabled/ # If using a separate config for the base domain: # sudo ln -s /etc/nginx/sites-available/yourdomain.com.conf /etc/nginx/sites-enabled/ # Test Nginx configuration syntax sudo nginx -t # If syntax is ok, reload Nginx (or restart if not running) sudo systemctl reload nginx - Run Certbot:
- Use the
certbotcommand with the--nginxplugin. It will analyze your Nginx configuration, identify theserver_namedirectives, obtain certificates, and automatically modify your Nginx config to include the necessaryssl_certificate,ssl_certificate_key, and other TLS-related directives. - Run Certbot for each domain you need a certificate for (e.g.,
matrix.yourdomain.comandyourdomain.comif using.well-knowndelegation). - Certbot will ask for an email address (for renewal reminders) and agreement to the Terms of Service.
- It should automatically detect your Nginx blocks and ask if you want it to configure HTTPS. Choose the option to redirect HTTP to HTTPS if offered.
- Use the
- Verify Auto-Renewal: Certbot typically sets up a systemd timer or cron job to automatically renew certificates before they expire. You can test the renewal process:
Federation Considerations Port 8448 vs. .well-known
How do other Matrix servers find your server to exchange federated messages?
- SRV Record (Port 8448):
- How it works: You configure a
_matrix-fed._tcpSRV record pointing to your FQDN (e.g.,matrix.yourdomain.com) and port8448. - Synapse Config: Synapse needs a listener configured for federation, usually on port 8448, secured with TLS. The default
homeserver.yamloften includes this, but you need to ensure Synapse has access to valid TLS certificates for the FQDN specified in the SRV target (e.g.,matrix.yourdomain.com). This can be tricky with Docker if Nginx handles TLS externally on 443. One approach is to share the Let's Encrypt certs with the Synapse container or run a separate proxy just for 8448. - Firewall: You must open TCP port 8448.
- How it works: You configure a
.well-known/matrix/serverDelegation (Port 443):- How it works: Other servers first check
https://yourdomain.com/.well-known/matrix/server. Your Nginx configuration serves a JSON response telling them the actual server is atmatrix.yourdomain.com:443. - Nginx Config: Requires the second
serverblock in the Nginx example, serving the correct JSON response on your base domain (yourdomain.com), secured with a valid TLS certificate for that base domain. - Synapse Config: No special Synapse listener needed beyond the standard client listener proxied by Nginx on port 443. Synapse must be configured to expect federation traffic via the reverse proxy on port 443 (this is often handled automatically if
public_baseurlis set correctly or headers likeX-Forwarded-Protoare passed). - Firewall: Only need TCP port 443 open. Generally simpler.
- How it works: Other servers first check
Recommendation: Using the .well-known delegation method via Nginx on port 443 is often easier to manage, especially with Docker and Certbot, as Nginx handles all TLS on the standard port. You still need the SRV record for clients (_matrix._tcp), but the federation SRV (_matrix-fed._tcp) becomes less critical (though still good practice as a fallback).
Update Docker Compose
Now that Nginx handles external connections on port 443 and proxies to 127.0.0.1:8008, we no longer need to expose port 8008 directly from the Synapse container in docker-compose.yml.
Modify your docker-compose.yml:
version: '3.8'
services:
synapse:
image: matrixdotorg/synapse:latest # Or pinned version
container_name: matrix_synapse
restart: unless-stopped
volumes:
- /srv/synapse/data:/data
# Optional: Mount Let's Encrypt certificates if Synapse needs direct access
# (e.g., for federation on 8448 without separate proxy, or for TURN server)
# - /etc/letsencrypt:/etc/letsencrypt:ro
# NO LONGER EXPOSE PORTS EXTERNALLY - Nginx handles this
# ports:
# - "127.0.0.1:8008:8008" # Keep this if Nginx is on the SAME host
# Remove if Nginx is on a different host/container
# Expose port 8008 *only to the Docker network* if Nginx runs in a separate container
# expose:
# - 8008
networks:
# Connect to a common network if Nginx runs in another Docker Compose service
- matrix_network # Example network name
environment:
- TZ=Etc/UTC
# Ensure Synapse knows it's behind a proxy (often automatic via X-Forwarded headers)
# You might need these if Synapse can't detect the public URL correctly
# - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml # Default
# - SYNAPSE_HTTP_PROXY_MODE=1 # Set if Synapse is behind a reverse proxy
# - SYNAPSE_PUBLIC_HOSTNAME=matrix.yourdomain.com
healthcheck:
# Healthcheck still uses internal port
test: ["CMD", "curl", "-f", "http://localhost:8008/health"]
interval: 30s
timeout: 10s
retries: 3
# Define the network if Nginx is in the same docker-compose file or needs to connect
networks:
matrix_network:
driver: bridge
Key Changes:
- Removed the
portssection (or commented it out). Nginx now handles external access. - If Nginx runs on the same host machine as the Docker daemon (not in another container), Synapse still needs to listen on
127.0.0.1:8008for Nginx to connect to it. So theproxy_pass http://127.0.0.1:8008;in Nginx is correct, and the127.0.0.1:8008:8008port mapping indocker-compose.ymlmight still be needed if Nginx is outside Docker. - If Nginx runs as another Docker container managed by Docker Compose (a common pattern), you would remove the
portsmapping entirely, add both Nginx and Synapse to the same Docker network (e.g.,matrix_network), and change theproxy_passin Nginx tohttp://synapse:8008;(using the service namesynapse). You'd also addexpose: - 8008to the Synapse service definition. - Added
networksdefinition (adapt if needed).
After modifying docker-compose.yml, apply the changes:
Workshop Securing Synapse with Nginx and Let's Encrypt
This workshop guides you through installing Nginx, configuring it as a reverse proxy for Synapse, obtaining Let's Encrypt certificates, and setting up .well-known delegation.
Objective: Make your Synapse instance accessible via HTTPS on the standard port 443 using Nginx and a valid TLS certificate. Enable federation via .well-known delegation.
Prerequisites:
- A running Synapse instance from the previous workshop (accessible on
127.0.0.1:8008on the server). - A Linux server with Docker/Docker Compose.
- Correctly configured DNS A/AAAA records for
matrix.yourdomain.compointing to your server's IP. - Correctly configured DNS A/AAAA records for
yourdomain.compointing to your server's IP (required for.well-knowndelegation). - Firewall allowing incoming TCP traffic on ports 80 and 443.
Steps:
- Install Nginx:
- Configure Firewall: Ensure ports 80 and 443 are open. Port 8448 is not needed if using only
.well-knowndelegation. - Create Nginx Configuration for
matrix.yourdomain.com:- Create the config file:
- Paste the first two
serverblocks (the port 80 redirect block and the main port 443 proxy block) from the Nginx example section above. - Carefully replace
matrix.yourdomain.comwith your actual FQDN everywhere it appears. - Set
client_max_body_sizeappropriately (e.g.,128M). - Ensure
proxy_passpoints tohttp://127.0.0.1:8008;(assuming Synapse port 8008 is mapped to the host's localhost). - For the port 80 block's
location /.well-known/acme-challenge/, ensure therootdirectory exists and is writable by Nginx (or use a standard path like/var/www/htmlwhich usually exists): - Save and close the file.
- Create Nginx Configuration for
yourdomain.com(.well-known):- Create the config file:
- Paste the third
serverblock (the.well-knowndelegation block) from the Nginx example section above. - Carefully replace
yourdomain.comwith your base domain (server name) andmatrix.yourdomain.comwith your FQDN in thereturnstatements. - Save and close the file.
- Enable Nginx Sites:
- Test Nginx Configuration:
- If it reports syntax errors, carefully review your configuration files. Common issues are missing semicolons
;, incorrect domain names, or brace mismatches{}. - If syntax is OK, reload Nginx:
- If it reports syntax errors, carefully review your configuration files. Common issues are missing semicolons
- Install Certbot:
- Obtain Let's Encrypt Certificates:
- Run Certbot for both domain names:
- Certbot should automatically find the
server_namedirectives, obtain the certificates, and modify your Nginx configuration files (/etc/nginx/sites-available/...) to include thessl_certificate,ssl_certificate_key, and other related lines. It will likely ask if you want to redirect HTTP to HTTPS - choose Redirect. - If Certbot fails, common reasons include:
- Firewall blocking port 80.
- DNS records not fully propagated yet.
- Incorrect
server_namein Nginx config. - Nginx not running or unable to bind to port 80.
- Check the Certbot logs (
/var/log/letsencrypt/letsencrypt.log) for details if errors occur.
- Verify Nginx Reloaded: Certbot should have reloaded Nginx automatically. You can check its status:
- Update Docker Compose (If Necessary):
- Edit your
docker-compose.yml. - Ensure the
portssection for Synapse only includes127.0.0.1:8008:8008(or is removed entirely if Nginx is also containerized and connected via a Docker network). Do not expose 8008 publicly. - Apply changes:
- Edit your
- Test HTTPS Access:
- Open your web browser and navigate to
https://matrix.yourdomain.com. You should see the Synapse "It Works!" page, served over HTTPS with a valid certificate (check the lock icon in your browser). - Navigate to
https://yourdomain.com/.well-known/matrix/server. You should see the JSON output{"m.server": "matrix.yourdomain.com:443"}. - Navigate to
https://yourdomain.com/.well-known/matrix/client. You should see the JSON output{"m.homeserver": {"base_url": "https://matrix.yourdomain.com"}, ... }.
- Open your web browser and navigate to
- Test Federation:
- Use the Matrix Federation Tester tool: https://federationtester.matrix.org/
- Enter your base domain (
yourdomain.com) and click "Test". - It should discover your server via
.well-known(or SRV if configured) and report success connecting over HTTPS on port 443. Look for green checkmarks!
- Test Client Connection:
- Go back to Element Web (https://app.element.io/) or your preferred client.
- Sign out if you were previously logged in via the SSH tunnel.
- Sign in again. This time, when editing the Homeserver:
- Enter your actual HTTPS FQDN:
https://matrix.yourdomain.com
- Enter your actual HTTPS FQDN:
- Use the username and password created earlier.
- You should be able to log in successfully over a secure connection.
Outcome: Your Synapse instance is now securely accessible via HTTPS using Nginx as a reverse proxy. Let's Encrypt certificates provide valid TLS encryption and will auto-renew. Federation should be working correctly via the .well-known delegation method. You can now connect securely using standard Matrix clients.
5. User Management and Authentication
With your Synapse server running securely behind a reverse proxy, the next crucial step is managing user access. By default, Synapse might allow open registration, which is undesirable for a private team chat server. This section covers disabling public registration, creating users manually or via tokens, and introduces concepts for more advanced authentication methods.
Disabling Public Registration
Allowing anyone on the internet to create an account on your private homeserver is generally a bad idea. It consumes resources, opens potential abuse vectors, and dilutes the purpose of a private server.
To disable open registration:
- Edit
homeserver.yaml:- Open your Synapse configuration file (e.g.,
sudo nano /srv/synapse/data/homeserver.yaml).
- Open your Synapse configuration file (e.g.,
- Find Registration Settings: Locate the
enable_registrationsetting. It might be under the main level or within alistenerssection depending on your Synapse version. - Set to
false: - Check Other Registration Options: Review related options like
enable_registration_without_verification,registrations_require_3pid, etc., and ensure they are either disabled or commented out if you want absolutely no open registration path. - Restart Synapse: Apply the changes by restarting the Synapse container:
Now, if someone tries to register an account through a client like Element pointing to your server, they should receive an error message indicating registration is disabled.
Manual User Creation (Command-Line)
Since public registration is disabled, you need a way to create accounts for your team members or authorized users. The primary method is using the register_new_matrix_user command-line tool, executed within the Synapse container's context.
Steps:
- SSH into your server.
-
Use
docker exec: Run the tool inside the runningmatrix_synapsecontainer:# Basic user creation: sudo docker exec matrix_synapse register_new_matrix_user \ -c /data/homeserver.yaml \ http://localhost:8008 \ -u <USERNAME> \ -p <PASSWORD> # Create an admin user: sudo docker exec matrix_synapse register_new_matrix_user \ -c /data/homeserver.yaml \ http://localhost:8008 \ -u <ADMIN_USERNAME> \ -p <ADMIN_PASSWORD> \ -a # The -a flag designates the user as an admin # Create a user without admin rights: sudo docker exec matrix_synapse register_new_matrix_user \ -c /data/homeserver.yaml \ http://localhost:8008 \ -u <REGULAR_USERNAME> \ -p <REGULAR_PASSWORD> \ # No -a flag- Replace
<USERNAME>,<PASSWORD>, etc., with the desired values. -c /data/homeserver.yaml: Specifies the path to the config file inside the container.http://localhost:8008: Tells the tool how to contact the Synapse process (using its internal listener).-u: Specifies the username (localpart). The full Matrix ID will be@<USERNAME>:yourdomain.com.-p: Sets the user's password. Choose strong passwords.-a: (Optional) Grants server administrator privileges to the user. Use this sparingly.
- Replace
-
Provide Credentials: Securely communicate the full Matrix ID (e.g.,
@alice:yourdomain.com) and the password to the intended user.
This method gives you direct control over who gets an account.
Exploring Registration Tokens
Manually creating every user via command line can be tedious, especially if users prefer to choose their own passwords securely. Synapse supports registration tokens. You, as an admin, generate one-time-use tokens. Users can then visit a specific registration page (usually enabled via configuration), enter the token along with their desired username and password, and create their own account.
Configuration (homeserver.yaml):
To enable registration only via tokens, you typically need:
# Keep general registration disabled
enable_registration: false
# Enable registration with tokens
registration_requires_token: true
# You might also need to re-enable registration on the specific listener
# if it was completely disabled there. Check your 'listeners' section.
# Sometimes, just having 'registration_requires_token: true' is sufficient
# alongside 'enable_registration: false'. Test your specific version.
# Optional: Define allowed validity period for tokens
# registration_token_validity_period: 1d # e.g., 1 day
Generating Tokens (Admin API):
Tokens are usually managed via the Synapse Admin API. You need an admin user account (created via register_new_matrix_user -a) and tools like curl to interact with the API.
- Get an Access Token for your Admin User:
- Log in to Element (or another client) as your admin user (
@adminuser:yourdomain.com). - Go to Settings -> Help & About -> Advanced -> Click to reveal Access Token. Copy this long token string. Treat this token like a password!
- Log in to Element (or another client) as your admin user (
-
Use
curlto Create a Token:# Replace <YOUR_ADMIN_ACCESS_TOKEN> and your server FQDN curl -X POST \ -H "Authorization: Bearer <YOUR_ADMIN_ACCESS_TOKEN>" \ -H "Content-Type: application/json" \ -d '{ "uses_allowed": 1, "expiry_time": null, "token_length": 16 }' \ "https://matrix.yourdomain.com/_synapse/admin/v1/registration_tokens/new"uses_allowed: 1: Makes it a one-time use token.expiry_time: null: Token doesn't expire automatically (you can set a timestamp).token_length: 16: Length of the generated token string.- The command will return JSON containing the
tokenstring.
-
Provide Token to User: Securely give the generated token string to the user who needs an account.
- User Registration: The user goes to your homeserver's registration page in their client (e.g., Element, pointing to
https://matrix.yourdomain.com). The interface should now prompt for a token alongside username/password.
This method balances security (no open registration) with user convenience (self-chosen password).
Introduction to Single Sign-On (SSO) Concepts
For organizations already using centralized identity management systems, integrating Synapse with Single Sign-On (SSO) offers a seamless user experience. Users log in using their existing company credentials instead of managing a separate Matrix password.
Synapse supports several SSO protocols:
- OpenID Connect (OIDC): A modern standard built on OAuth 2.0. Widely supported by identity providers like Keycloak, Okta, Google Workspace, Azure AD, etc. Generally the recommended SSO method.
- SAML (Security Assertion Markup Language): An older but still widely used XML-based standard for enterprise SSO.
- CAS (Central Authentication Service): Another SSO protocol, often found in academic institutions.
- LDAP (Password Auth): Synapse can be configured to authenticate users against an LDAP directory (like OpenLDAP or Active Directory) using their LDAP password. This isn't strictly SSO but achieves centralized password management.
Configuration: Setting up SSO involves configuring specific sections in homeserver.yaml detailing the identity provider's endpoints, client IDs/secrets, attribute mappings (how to map email/username from the SSO provider to a Matrix ID), and potentially TLS settings for communication with the provider. The setup is highly dependent on the chosen protocol and the specific identity provider being used. It typically requires familiarity with both Synapse configuration and the identity provider's administration.
Due to the complexity and variance, detailed SSO setup is beyond the scope of this intermediate section but is a common next step for organizational deployments. Refer to the official Synapse documentation on SSO for specific guides.
Configuring Basic Password Policies
Synapse itself has limited built-in support for enforcing complex password policies (like minimum length, complexity requirements, history). However, you can influence password handling indirectly:
- SSO/LDAP: If you use SSO or LDAP password authentication, the password policies are enforced by your central identity provider or LDAP directory, not Synapse.
- Client-Side: Some clients might offer suggestions or checks, but these aren't server-enforced.
- External Scripts/Admin Tools: You could potentially use the Admin API to periodically check for weak passwords or enforce resets, but this requires custom tooling.
For most self-hosted scenarios without SSO/LDAP, relying on user education and potentially using registration tokens (so users set their own hopefully strong passwords) is the common approach.
Workshop Controlling User Access
This workshop guides you through disabling public registration and practicing user creation using both the command-line tool and registration tokens.
Objective: Secure your Synapse server by disabling open registration and learn how to add authorized users.
Prerequisites:
- Your Synapse server running securely behind Nginx with HTTPS (from the previous workshop).
- An admin user account created on your Synapse server (e.g., using
register_new_matrix_user -a). - Access to your server's command line (SSH).
- A Matrix client like Element Web.
Steps:
- Disable Public Registration:
- SSH into your server.
- Edit the Synapse configuration file:
- Find the line
enable_registration:and set it tofalse. - Ensure any other registration options like
enable_registration_without_verificationare also commented out or explicitlyfalse. - Save the file.
- Restart Synapse:
- Verify: Try opening Element Web in an incognito/private browser window, point it to your homeserver (
https://matrix.yourdomain.com), and click "Create Account". You should receive an error stating that registration is disabled on this server.
- Create a User via Command Line:
- Use
docker execto run the registration tool: - Note the full Matrix ID (
@testuser:yourdomain.com) and the password. - Verify: Log in to Element Web using this new username and password. It should work. Log out again.
- Use
- Enable Registration via Tokens:
- Edit
homeserver.yamlagain: - Add or uncomment the following line:
- Ensure
enable_registration: falseremainsfalse. - Save the file.
- Restart Synapse:
- Edit
- Generate a Registration Token:
- Log in to Element Web using your admin account.
- Go to Settings -> Help & About -> Advanced (you might need to enable developer mode first in Settings -> Labs) -> Scroll down to "Access Token" and click "Click to reveal".
- Copy the long access token string.
- Go back to your server's SSH terminal.
- Use
curlto request a token (replace<YOUR_ADMIN_ACCESS_TOKEN>andmatrix.yourdomain.com):ADMIN_TOKEN="<YOUR_ADMIN_ACCESS_TOKEN>" YOUR_FQDN="matrix.yourdomain.com" curl -X POST \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "uses_allowed": 1, "expiry_time": null, "token_length": 16 }' \ "https://$YOUR_FQDN/_synapse/admin/v1/registration_tokens/new" - The output will be JSON. Copy the value of the
"token"field (e.g.,"AbCdEfGhIjKlMnOp"). This is your registration token.
- Register a User with the Token:
- Open Element Web in a fresh incognito/private browser window.
- Point it to your homeserver (
https://matrix.yourdomain.com). - Click "Create Account".
- The registration form should now have an additional field asking for a "Registration Token" (or similar wording).
- Enter the token you copied.
- Choose a new username (e.g.,
tokenuser) and a new password. - Complete the registration.
- Verify: You should be successfully registered and logged in as
@tokenuser:yourdomain.com.
- Deactivate a User (Optional):
- You might want to remove the
testusercreated earlier. You can do this via the Admin API. - In the SSH terminal (you still need the admin token):
- A successful deactivation returns an empty JSON object
{}. - Verify: Try logging in as
testuseragain. It should fail.
- You might want to remove the
Outcome: You have successfully configured your Synapse server to prevent unwanted public sign-ups. You can now confidently create users either directly via the command line or by providing them with secure, single-use registration tokens. You have also seen how to deactivate users using the admin API.
6. Database Choices and Configuration
By default, Synapse generated via the Docker command uses SQLite as its database backend. While SQLite is incredibly simple to set up (just a single file on disk), it has significant performance limitations and is not recommended for anything beyond small, low-usage instances or initial testing. For any serious deployment, migrating to PostgreSQL is highly advised.
Default Database SQLite Limitations
SQLite stores the entire database in a single file (e.g., homeserver.db). Its limitations in the context of Synapse include:
- Concurrency: SQLite has limited write concurrency. When Synapse needs to write data (new messages, user updates, federation events), it locks parts of the database file. With many concurrent users or high federation activity, these locks become a major bottleneck, leading to slow response times and high I/O wait.
- Scalability: Performance degrades significantly as the database size and number of concurrent operations grow.
- Resource Usage: While seemingly lightweight, frequent locking under load can sometimes lead to high CPU usage as processes wait for locks.
- Replication/High Availability: No built-in support for replication or clustering for high availability, unlike mature databases like PostgreSQL.
- Tooling: While tools exist, the ecosystem for monitoring, tuning, and backing up PostgreSQL is generally more mature and feature-rich for server applications.
Essentially, for a chat server designed for potentially many users and real-time interactions, SQLite quickly becomes the bottleneck.
Using PostgreSQL for Performance and Scalability
PostgreSQL (or Postgres) is a powerful, open-source object-relational database system known for its:
- Robustness and Reliability: ACID compliant, battle-tested over decades.
- Concurrency Control: Uses Multi-Version Concurrency Control (MVCC), allowing multiple readers and writers to operate much more efficiently without blocking each other as aggressively as SQLite. This is critical for Synapse's performance under load.
- Scalability: Handles large databases and high transaction volumes much more gracefully than SQLite.
- Extensibility: Supports advanced data types, indexing methods, and procedural languages.
- Advanced Features: Offers replication, connection pooling, sophisticated monitoring, and tuning options.
Migrating Synapse to use PostgreSQL is one of the single most impactful changes you can make to improve its performance and reliability.
Reasons to Switch to PostgreSQL
- Performance: Significantly faster message sending/receiving, room joining, and general responsiveness, especially with more than a handful of active users or federation enabled.
- Scalability: Allows your Synapse instance to grow to support many more users and handle higher loads.
- Stability: Reduced chance of database corruption compared to SQLite under heavy load or unexpected shutdowns (though proper backups are always essential).
- Foundation for Scaling: Necessary if you plan to implement Synapse workers (covered in Advanced section) for further scaling.
Basic PostgreSQL Setup
You need a running PostgreSQL server. It can be installed directly on the host machine or, more commonly in a Docker environment, run as a separate Docker container. Running it as a container keeps your host system cleaner and simplifies management alongside Synapse.
Option 1: PostgreSQL in Docker (Recommended)
Add a PostgreSQL service to your docker-compose.yml:
version: '3.8'
services:
synapse:
image: matrixdotorg/synapse:latest
container_name: matrix_synapse
restart: unless-stopped
volumes:
- /srv/synapse/data:/data
# No ports exposed directly for Synapse
networks:
- matrix_network
depends_on: # Ensure database starts before Synapse attempts to connect
- postgres
environment:
- TZ=Etc/UTC
# We will configure the database in homeserver.yaml
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8008/health"]
interval: 30s
timeout: 10s
retries: 3
postgres:
image: postgres:15-alpine # Use a specific major version (e.g., 15)
container_name: matrix_postgres
restart: unless-stopped
networks:
- matrix_network
environment:
# Set PostgreSQL credentials
# CHANGE THESE to strong, unique passwords!
- POSTGRES_USER=synapse_user
- POSTGRES_PASSWORD=synapse_very_strong_password
- POSTGRES_DB=synapse_db
# Optional: Set locale for database collation (affects sorting)
# Important: Choose this carefully ONCE during setup. UTF8 is recommended.
- POSTGRES_INITDB_ARGS=--encoding=UTF8 --lc-collate=C --lc-ctype=C
# Or use your system locale if preferred, e.g.:
# - POSTGRES_INITDB_ARGS=--encoding=UTF8 --locale=en_US.UTF-8
volumes:
# Persist PostgreSQL data
# Use a Docker named volume or a host bind mount
- postgres_data:/var/lib/postgresql/data
# Example bind mount:
# - /srv/synapse/postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U synapse_user -d synapse_db"]
interval: 10s
timeout: 5s
retries: 5
volumes:
# Define the named volume for Postgres data (if not using bind mount)
postgres_data:
networks:
matrix_network:
driver: bridge
Key PostgreSQL Service Details:
image: postgres:15-alpine: Uses the official Postgres image (version 15, Alpine variant for smaller size). Pinning to a major version is recommended.environment: Sets up the initial database (POSTGRES_DB), user (POSTGRES_USER), and password (POSTGRES_PASSWORD) when the container starts for the first time and the data volume is empty. Choose strong passwords.POSTGRES_INITDB_ARGS: Crucial for setting the database encoding and collation at creation time.UTF8encoding is essential. Collation (lc-collate,lc-ctype) affects string sorting and comparison;C(orC.UTF-8) provides byte-order sorting, which is often recommended for performance and consistency with Synapse. Alternatively, use a specific locale likeen_US.UTF-8. This cannot be easily changed later.volumes: Persists the actual database data outside the container. Using a named volume (postgres_data) managed by Docker is often simplest.networks: Connects the Postgres container to the same Docker network as Synapse, allowing them to communicate using service names (e.g.,postgres).healthcheck: Checks if the Postgres server is ready to accept connections.depends_on: - postgresin the Synapse service definition tells Docker Compose to start thepostgresservice before starting thesynapseservice.
Option 2: PostgreSQL on Host
If you prefer to install PostgreSQL directly on the host OS:
# On Debian/Ubuntu
sudo apt update
sudo apt install postgresql postgresql-contrib -y
# Switch to the postgres user to create the database and user
sudo -u postgres psql <<EOF
-- Create the database user (use a strong password!)
CREATE USER synapse_user WITH PASSWORD 'synapse_very_strong_password';
-- Create the database with recommended encoding and locale
-- Use 'C' or 'C.UTF-8' for collation unless you have specific locale needs
-- Ensure template0 is used to allow specifying encoding/locale
CREATE DATABASE synapse_db
OWNER synapse_user
LC_COLLATE='C'
LC_CTYPE='C'
ENCODING 'UTF8'
TEMPLATE template0;
EOF
# Edit pg_hba.conf to allow connections from Synapse (e.g., from Docker)
# sudo nano /etc/postgresql/<VERSION>/main/pg_hba.conf
# Add a line like (adjust IP range if Docker network is different):
# host synapse_db synapse_user 172.17.0.0/16 md5
# Or for localhost connections if Synapse runs directly on host:
# host synapse_db synapse_user 127.0.0.1/32 md5
# Edit postgresql.conf to listen on appropriate interfaces if needed
# sudo nano /etc/postgresql/<VERSION>/main/postgresql.conf
# Modify 'listen_addresses' (e.g., 'localhost, <docker_bridge_ip>')
# Restart PostgreSQL to apply changes
sudo systemctl restart postgresql
Configuring Synapse to Use PostgreSQL
Once the PostgreSQL server is running and the database/user are created, you need to tell Synapse how to connect to it.
- Edit
homeserver.yaml: -
Modify the
databaseSection: Comment out or remove thesqlite3configuration and add the PostgreSQL settings:database: name: psycopg2 # The Python library Synapse uses for Postgres args: user: synapse_user password: synapse_very_strong_password database: synapse_db host: postgres # Use the service name from docker-compose.yml port: 5432 # Default Postgres port cp_min: 5 # Minimum number of connections in pool cp_max: 10 # Maximum number of connections in pool # Optional: Add SSL settings if connecting to Postgres over TLS # sslmode: require # E.g., 'require', 'verify-full' # sslcert: /path/to/client.crt # Inside container # sslkey: /path/to/client.key # Inside container # sslrootcert: /path/to/ca.crt # Inside containername: psycopg2: Specifies the PostgreSQL driver.user,password,database: Must match the credentials you set up in PostgreSQL.host: Crucially, use the service name defined indocker-compose.yml(e.g.,postgres) if both are running in Docker containers on the same network. If Postgres is on the host, you might need the host's IP address accessible from the Docker bridge (e.g.,172.17.0.1often works, but checkip addr show docker0).port: The standard PostgreSQL port.cp_min,cp_max: Configure the connection pool size. Defaults are usually okay to start, but can be tuned later.
-
Restart Synapse:
Synapse should now start, connect to the PostgreSQL database, and automatically create the necessary tables and schema within the# If you modified docker-compose.yml (e.g., added the postgres service) sudo docker compose up -d --force-recreate # Force recreate to apply changes # If you only modified homeserver.yaml # sudo docker compose restart synapsesynapse_dbdatabase. Check the logs (sudo docker compose logs -f synapse) for any connection errors.
Migrating from SQLite to PostgreSQL
If you already have existing users and data in an SQLite database, you need to migrate that data to the new PostgreSQL database. This is a non-trivial process that requires downtime and careful execution.
The core Matrix team provides a script called synapse_port_db designed for this purpose.
High-Level Steps (Use with Caution):
- STOP Synapse completely. This is essential to prevent data changes during migration.
- Set up the new PostgreSQL database (as described above: create the DB, user, password). Ensure it's empty.
- Modify
homeserver.yamlto point to the new PostgreSQL database. -
Run the
synapse_port_dbscript. This script reads the old SQLite config, connects to both SQLite and the new Postgres DB (using the updatedhomeserver.yaml), and copies data across. It's usually run using the Synapse Docker image:sudo docker compose run --rm -v /srv/synapse/data:/data synapse \ python /synapse/scripts/synapse_port_db \ --sqlite-database /data/homeserver.db \ --postgres-config /data/homeserver.yaml- This command runs the
synapse_port_dbscript inside a temporary Synapse container. - It mounts your data volume so the script can access both the SQLite file (
/data/homeserver.db) and thehomeserver.yaml(which now contains the Postgres connection details). - The script will output progress as it copies tables. This can take a significant amount of time for large databases.
- Review Output: Check carefully for any errors reported by the script.
- Backup SQLite DB: Once confident the migration succeeded, back up the old
homeserver.dbfile somewhere safe. - Restart Synapse: Start Synapse using the configuration pointing to PostgreSQL.
- Verify: Log in, check rooms, history, users to ensure data integrity.
- This command runs the
Important Considerations for Migration:
- Downtime: The migration requires Synapse to be offline. Plan accordingly.
- Disk Space: You need enough space for both the SQLite file and the growing PostgreSQL database during the migration.
- Complexity: Errors can occur. Ensure you have backups before starting.
- Test First: If possible, test the migration process on a non-production copy of your database first.
For new setups, starting directly with PostgreSQL avoids this migration complexity entirely.
Workshop Migrating to PostgreSQL
This workshop guides you through setting up a PostgreSQL container and configuring a new Synapse instance to use it from the start. We will not perform the complex SQLite-to-Postgres data migration here, focusing instead on the cleaner initial setup with Postgres.
Objective: Configure and run Synapse using PostgreSQL as the database backend within a Docker Compose setup.
Prerequisites:
- Docker and Docker Compose installed.
- Nginx reverse proxy configured (optional for this specific workshop, but assumed for a complete setup).
- Directory structure ready (e.g.,
/srv/synapse).
Steps:
-
Stop Existing Synapse (If Running):
- If you have a Synapse container running from previous workshops, stop and remove it to avoid conflicts.
- Navigate to the directory with your old
docker-compose.yml. - ```bash
- sudo docker compose down
-
```
-
(Optional but Recommended) Backup or move your old
/srv/synapse/datadirectory if you want to keep the SQLite data safe, e.g.,sudo mv /srv/synapse/data /srv/synapse/data_sqlite_backup. Then recreate an empty data directory:sudo mkdir /srv/synapse/data. - Create New
docker-compose.ymlwith PostgreSQL: - Navigate to your main Synapse configuration directory (e.g.,
/srv/synapse). - Create a new
docker-compose.ymlfile (or replace the old one). - Paste the combined
docker-compose.ymlcontent provided in the "Basic PostgreSQL Setup - Option 1: PostgreSQL in Docker" section above. - Crucially:
- Review and change the default PostgreSQL passwords (
POSTGRES_PASSWORD) to something strong and unique. Record these securely. - Ensure the
volumespaths for both Synapse (/srv/synapse/data:/data) and PostgreSQL (e.g.,postgres_data:/var/lib/postgresql/dataor a bind mount like/srv/synapse/postgres_data:/var/lib/postgresql/data) are correct for your system. If using a bind mount for Postgres, create the host directory (sudo mkdir /srv/synapse/postgres_data) and ensure permissions allow the container (often needs specific user ID or group ownership, or looser permissions initially). Using a Docker named volume (postgres_data:) is often easier. - Verify the
POSTGRES_INITDB_ARGSare set as desired (the example usingCcollation is generally good).
- Review and change the default PostgreSQL passwords (
- Generate
homeserver.yaml(or Modify Existing Clean One): -
Since we are starting fresh, you might need to generate
homeserver.yamlagain if you deleted the old data directory. -
If you kept an old
homeserver.yamlbut want to start fresh with Postgres, edit it now. - Configure
homeserver.yamlfor PostgreSQL: - Edit the
homeserver.yamlfile in your data directory: - Find the
database:section. - Delete or comment out the entire
sqlite3configuration block. - Add the
psycopg2configuration block, ensuring theuser,password,database, andhostmatch exactly what you set in thedocker-compose.ymlfile'spostgresservice environment variables and service name: - Save and close the file.
- Start the Stack:
- Navigate to the directory containing your new
docker-compose.yml. - Run Docker Compose:
- Docker Compose will:
- Create the
matrix_network. - Start the
postgrescontainer. If thepostgres_datavolume is empty, it will initialize the database using the environment variables. - Wait for the
postgreshealthcheck to pass (if configured). - Start the
synapsecontainer.
- Create the
- Check Logs:
- Monitor the logs of both containers, especially Synapse, to ensure it connects successfully to PostgreSQL.
- You should not see errors about database connections. Initial schema setup might take a short while.
- Register a New User:
- Since this is effectively a fresh instance (using an empty database), you will need to register your admin user again. Use the
register_new_matrix_usercommand: - Verify:
- Connect using Element Web (pointing to
https://matrix.yourdomain.comif Nginx is set up) with the newly created admin user. - Everything should work, but your previous rooms and history (from the SQLite instance) will be gone because we started with a fresh database.
Outcome: You have successfully launched a Synapse instance configured to use a PostgreSQL database running in a separate Docker container. This provides a much more robust and performant foundation for your Matrix server compared to SQLite. You are now better prepared for growth and increased load.
7. Federation Deep Dive
Federation is the cornerstone of Matrix's decentralized nature. It allows users on your homeserver (yourdomain.com) to communicate seamlessly with users on other homeservers (anotherdomain.org, matrix.org, etc.) within the same Matrix room. Understanding how federation works is crucial for troubleshooting connectivity issues and managing how your server interacts with the wider Matrix network.
How Federation Works
Federation relies on a Server-Server (S2S) API defined in the Matrix specification. When servers need to interact (e.g., send a message to a room hosted elsewhere, invite a remote user, query user profiles), they communicate directly over HTTPS.
Key components:
- Server Discovery: When
server-A.comneeds to talk toserver-B.org, it first needs to find the actual hostname and port whereserver-B.org's S2S API is listening. It does this using:- SRV Record: Looks for
_matrix-fed._tcp.server-B.org. If found, it yields a target hostname (e.g.,matrix.server-B.org) and port (usually8448).server-A.comthen connects tomatrix.server-B.org:8448. .well-knownDelegation: If the SRV lookup fails or as a primary method,server-A.comfetcheshttps://server-B.org/.well-known/matrix/server. If successful, this returns JSON like{"m.server": "matrix.server-B.org:443"}, tellingserver-A.comto connect tomatrix.server-B.orgon port443.- Direct Connection (Fallback): If both SRV and
.well-knownfail,server-A.commight try connecting directly toserver-B.orgon port8448(less common now).
- SRV Record: Looks for
- TLS Connection: All S2S communication must occur over HTTPS (TLS). Servers validate each other's TLS certificates based on the expected server name (
server-B.orgin the example). Self-signed or expired certificates will cause federation failures. - Server Signing Keys: Each homeserver has a long-term cryptographic signing keypair. All events (messages, state changes) sent over federation are signed by the originating server's key. Receiving servers verify these signatures to ensure authenticity and prevent tampering.
- Servers fetch each other's public signing keys via the S2S API (e.g.,
/_matrix/key/v2/server/<key_id>). These keys are often cached. - Your server's signing key is stored in a file specified by
signing_key_pathinhomeserver.yaml(e.g.,yourdomain.com.signing.key). Protect this key file! Losing it or having it compromised requires complex recovery steps.
- Servers fetch each other's public signing keys via the S2S API (e.g.,
- Event Authorization and State Resolution: When an event arrives via federation, the receiving server checks if the sending server and user are authorized to perform that action in the given room (based on room permissions, power levels, and membership state). Matrix uses a sophisticated state resolution algorithm to ensure all servers eventually agree on the room's state (topic, members, permissions) even if events arrive out of order or concurrently from different servers.
- Persistent Data Units (PDUs): Events are formatted as PDUs, containing the content, sender, origin server, signatures, cryptographic hashes of previous events (forming a DAG - Directed Acyclic Graph), and authorization information.
Essentially, federation involves servers discovering each other, establishing secure connections, exchanging signed events, and verifying the authenticity and authorization of those events based on a shared understanding of room state.
Understanding SRV Records vs. .well-known/matrix/server
As mentioned in the Intermediate section, these are the two primary methods for server discovery:
- SRV (
_matrix-fed._tcp.<server_name>on port 8448):- Pros: The traditional method, clearly separates federation traffic port.
- Cons: Requires opening port 8448 on the firewall. Requires Synapse itself (or a dedicated proxy for 8448) to handle TLS connections on this port, potentially complicating certificate management if another proxy (like Nginx) handles client TLS on 443.
.well-known(https://<server_name>/.well-known/matrix/serverdelegating to<hostname>:<port>usually 443):- Pros: Only requires port 443 to be open, simplifying firewalls. Allows your main reverse proxy (Nginx) to handle all TLS termination and certificate management consistently for both client and federation traffic. Often easier to configure correctly.
- Cons: Relies on the base domain (
<server_name>) being served over HTTPS with a valid certificate, even if only to provide the.well-knownresponse. Requires careful Nginx configuration to serve the correct JSON response with CORS headers.
Recommendation: Using .well-known delegation to port 443 is generally the preferred and simpler approach for modern deployments, especially when using a reverse proxy like Nginx that already handles TLS for client connections on port 443. You can still have the SRV record as a fallback or for compatibility.
Troubleshooting Federation Issues
When federation breaks, users might report being unable to join rooms on other servers, messages not arriving from remote users, or remote users being unable to join your local rooms.
Common Tools and Techniques:
- Matrix Federation Tester: https://federationtester.matrix.org/
- Your first stop. Enter your server name (e.g.,
yourdomain.com). - It simulates how another server would discover and connect to yours.
- Checks DNS (SRV, A/AAAA records).
- Checks
.well-knownendpoint. - Attempts TLS connection to the discovered endpoint(s).
- Validates the TLS certificate (hostname match, trusted CA, expiration).
- Checks if Synapse responds correctly on the API endpoint.
- Interpreting Results: Look for red crosses or warnings. They usually pinpoint the exact problem (DNS misconfiguration, firewall block, incorrect TLS certificate, bad Nginx proxy setup).
- Your first stop. Enter your server name (e.g.,
- Synapse Logs:
- Check your Synapse container logs (
sudo docker compose logs -f synapse). - Increase log verbosity temporarily in
homeserver.yamlif needed (e.g., setlevel: DEBUGin theloggerssection, but be cautious as this generates a lot of logs). - Look for lines containing:
ERROR,WARNINGfederationFailed to connect,Connection refused,TimeoutTLSerrors,certificate verify failedSignature verification failed- Specific remote server names you're having trouble with.
- Logs often provide detailed error messages about outgoing federation attempts (your server trying to reach others) or issues processing incoming requests.
- Check your Synapse container logs (
- Reverse Proxy Logs (Nginx):
- Check Nginx's access log (
/var/log/nginx/access.log) and error log (/var/log/nginx/error.log). - Look for requests to
/_matrix/federation/...endpoints. - Check for HTTP status codes like
4xx(client errors) or5xx(server errors) related to federation requests. - Errors here might indicate problems with the
proxy_passdirective, timeouts connecting to the Synapse backend, or issues with the Nginx configuration itself.
- Check Nginx's access log (
- DNS Tools (
dig):- Manually verify your SRV and A/AAAA records from an external network if possible, or using
dig <type> <name> @<resolver>(e.g.,dig SRV _matrix-fed._tcp.yourdomain.com @8.8.8.8). - Check for correct target hostnames, ports, and IP addresses. Ensure propagation.
- Manually verify your SRV and A/AAAA records from an external network if possible, or using
- TLS/Certificate Tools (
openssl s_client):- Test the TLS connection directly to the port federation is expected on (e.g., 443 if using
.well-known, 8448 if using SRV). openssl s_client -connect matrix.yourdomain.com:443 -servername yourdomain.comopenssl s_client -connect matrix.yourdomain.com:8448 -servername yourdomain.com(adjust host/port/servername)- Examine the output for the certificate chain, verification results (
Verify return code), and supported TLS protocols/ciphers. Ensure the certificate presented matches the expected hostname and is trusted.
- Test the TLS connection directly to the port federation is expected on (e.g., 443 if using
- Firewall Checks: Double-check that your server and network firewalls allow incoming traffic on the necessary ports (443 and/or 8448) from all IP addresses (Any /
0.0.0.0/0).
Common Issues & Fixes:
- Incorrect DNS: SRV target points to wrong host/port; A/AAAA record incorrect;
.well-knownserver name wrong. -> Fix DNS records. - Firewall Blocking: Port 443 or 8448 not open. -> Adjust firewall rules.
- TLS Certificate Invalid: Expired, self-signed, mismatch hostname (e.g., cert for
matrix.yourdomain.combut server name isyourdomain.comand.well-knownisn't set up correctly), incomplete chain. -> Renew/fix certificate (Certbot), ensure Nginx serves the correct cert, configure.well-knownproperly. - Nginx Proxy Error: Incorrect
proxy_pass(wrong port/host), headers not set correctly (Host,X-Forwarded-For), timeouts. -> Review Nginx config, ensure Synapse is running and reachable from Nginx. - Synapse Not Running/Responding: Synapse container stopped or unhealthy. -> Check Synapse logs, restart container.
- Server Signing Key Issues: Key file corrupted or permissions wrong (rare). -> Restore from backup, check file permissions.
Federation Allow/Deny Lists
By default, Synapse will attempt to federate with any other Matrix server it needs to interact with. In some scenarios (e.g., corporate environments, specific community servers), you might want to restrict federation to a predefined list of trusted servers or block communication with specific problematic servers.
This is configured in homeserver.yaml:
federation:
# Block federation with specific servers
# federation_domain_blacklist:
# - bad-server.com
# - malicious.matrix.host
# Allow federation ONLY with specific servers (blocks all others)
# federation_domain_whitelist:
# - trusted-partner.org
# - internal-server.mycorp
federation_domain_blacklist: A list of server names your homeserver will refuse to communicate with. Useful for blocking known spam or malicious servers.federation_domain_whitelist: A list of server names your homeserver is allowed to communicate with. If this list is defined, your server will refuse all federation attempts (incoming and outgoing) with any server not on this list. This creates a closed or restricted federation environment.
Use these options with caution. A whitelist significantly limits your users' ability to interact with the public Matrix network. A blacklist requires ongoing maintenance as problematic servers emerge. Remember to restart Synapse after modifying these settings.
Server Signing Keys Management
Synapse automatically generates a signing keypair when it first starts (e.g., yourdomain.com.signing.key). This key is crucial for proving the authenticity of events originating from your server.
- Backup: Regularly back up your entire Synapse data directory, especially the signing key file.
- Protection: Ensure the signing key file has restrictive file permissions (readable only by the user Synapse runs as).
- Expiration/Revocation: Signing keys have a validity period. Synapse handles rotation and publication of new keys automatically. You can view your server's currently published keys at
https://matrix.yourdomain.com/_matrix/key/v2/server. - Compromise: If you suspect your signing key has been compromised, consult the Synapse documentation for key revocation and replacement procedures. This is an advanced operation.
Workshop Debugging Federation
This workshop provides hands-on experience with diagnosing federation problems using common tools.
Objective: Learn to use the Federation Tester and server logs to identify and understand potential federation issues.
Prerequisites:
- Your Synapse server running with Nginx and HTTPS, configured for federation (ideally via
.well-known). - Access to your server's command line (SSH).
- A web browser.
Steps:
- Baseline Test with Federation Tester:
- Go to https://federationtester.matrix.org/.
- Enter your server name (e.g.,
yourdomain.com). - Click "Test".
- Carefully review the results. Ideally, everything should be green. Note how it discovers your server (via SRV or
.well-known) and the endpoint it successfully connects to (e.g.,matrix.yourdomain.com:443).
- Simulate a
.well-knownFailure:- Intentionally break the Nginx config:
- SSH into your server.
- Edit the Nginx config file for your base domain (the one serving
.well-known): - Find the
location /.well-known/matrix/serverblock. - Temporarily comment out the
return 200 ...;line by adding a#at the beginning. - Save the file.
- Test Nginx syntax:
sudo nginx -t - Reload Nginx:
sudo systemctl reload nginx
- Retest with Federation Tester:
- Run the Federation Tester again for
yourdomain.com. - Observe: The test should now likely fail or show warnings related to fetching the
.well-known/matrix/serverfile (e.g., 404 Not Found). Note how this impacts the ability to discover the server via this method.
- Run the Federation Tester again for
- Fix the Nginx config:
- Edit
/etc/nginx/sites-available/yourdomain.com.confagain. - Uncomment the
return 200 ...;line. - Save, test syntax (
sudo nginx -t), and reload Nginx (sudo systemctl reload nginx). - Retest with the Federation Tester to confirm it's working again.
- Edit
- Intentionally break the Nginx config:
- Simulate a Firewall Block:
- Block Port 443 (Briefly!):
- Identify your firewall tool (e.g.,
ufw). - Temporarily add a rule to deny incoming traffic on port 443. Be careful not to lock yourself out if SSH is also firewalled. If unsure, skip this step.
- Example using
ufw:sudo ufw deny 443/tcp
- Identify your firewall tool (e.g.,
- Retest with Federation Tester:
- Run the Federation Tester.
- Observe: It should now fail with errors related to connection timeouts or refusals when trying to connect to your server on port 443.
- Remove the Block:
- Remove the temporary deny rule.
- Example:
sudo ufw delete deny 443/tcp(Or ensure yourallowrule takes precedence). - Retest to confirm connectivity is restored.
- Block Port 443 (Briefly!):
- Examine Synapse Logs for Federation Traffic:
- Trigger Federation: Use your Matrix client (logged into your server) to try and join a large, well-known public room on another server (e.g.,
#matrix:matrix.orgif you aren't already in it). This action forces your server to communicate withmatrix.org. -
View Logs: Quickly view the Synapse logs:
- Look for lines indicating outgoing requests (
Sending request,Received response) tomatrix.orgor other remote servers. - Look for lines indicating incoming requests (
Received PUT request,IncomingFederationRequest) potentially frommatrix.org(e.g., if they send presence updates or messages back).- (Optional) Try sending a message to the public room and watch the logs again for outgoing PDU transmissions.
- Examine Nginx Logs for Federation Traffic:
- View the Nginx access log:
- Look for
GETorPUTrequests to paths starting with/_matrix/federation/v1/.... These are incoming federation requests from other servers hitting your Nginx proxy. Note the source IPs and status codes (should be mostly2xx).
- Configure a Federation Blacklist:
- Edit
homeserver.yaml: - Add a blacklist section (choose a domain you don't need to federate with for this test, perhaps a temporary domain or one you know is offline):
- Save the file.
- Restart Synapse:
sudo docker compose restart synapse - Verify (Conceptual): Try to join a room hosted on
some-test-domain-to-block.com(if one existed). Your server should now refuse to communicate with it. Check the Synapse logs for messages indicating the block is active when attempting contact. - Cleanup: Remove the
federation_domain_blacklistsection fromhomeserver.yamland restart Synapse again unless you intend to keep it.
- Edit
- Look for lines indicating outgoing requests (
- Trigger Federation: Use your Matrix client (logged into your server) to try and join a large, well-known public room on another server (e.g.,
Outcome: You should now be more comfortable using the Federation Tester to diagnose common connectivity problems related to DNS, TLS, and server availability. You have also seen how intentional misconfigurations affect federation and how to inspect Synapse and Nginx logs for federation-related activity. You understand how to implement basic federation filtering using allow/deny lists.
8. Performance Tuning and Scaling
As your Synapse instance grows in users, rooms, and federation activity, the default single-process setup might struggle to keep up. Performance tuning involves identifying bottlenecks and implementing strategies like monitoring, database optimization, and deploying Synapse workers to distribute the load.
Identifying Bottlenecks
Performance issues in Synapse typically manifest as:
- Slow message sending/receiving.
- Delays in joining rooms.
- Slow loading of room history or initial sync.
- High CPU usage on the server.
- High RAM usage (potentially leading to OOM kills).
- High disk I/O wait times (especially with SQLite or slow disks).
- Federation delays or failures under load.
Common bottlenecks include:
- Database: Often the primary bottleneck, especially with SQLite. Even with PostgreSQL, inefficient queries or insufficient database server resources (CPU, RAM, IOPS) can cause slowdowns.
- CPU: Synapse's main process handles event processing, federation sending/receiving, client requests, etc. High activity can max out available CPU cores. Python's Global Interpreter Lock (GIL) can also limit true parallelism within a single Synapse process.
- RAM: Synapse caches various data (room state, user profiles, event data) in memory. Insufficient RAM leads to cache misses, more database queries, and potentially swapping (which drastically degrades performance).
- Disk I/O: Particularly relevant for the database (reads/writes) and media storage. Slow disks (HDDs vs SSDs) significantly impact database performance.
- Network: Less common, but network latency or bandwidth limitations can affect federation speed and client responsiveness.
Using Monitoring Tools (Prometheus and Grafana)
To effectively identify bottlenecks, you need data. Synapse exposes detailed metrics in Prometheus format, which can be scraped by Prometheus (a time-series database) and visualized using Grafana (a dashboard tool).
Setting up the Stack:
-
Enable Synapse Metrics:
- Edit
homeserver.yaml. -
Ensure the metrics listener is enabled and accessible. It's often enabled by default but might be bound to
localhost. You need Prometheus (which might run in a separate container) to be able to reach it. -
Restart Synapse.
- Verify you can access the metrics endpoint:
curl http://<synapse_host_ip>:9092/metrics(use the host IP, not just localhost, if accessing from outside the Synapse container). You should see a large text output in Prometheus format. - Deploy Prometheus & Grafana (Docker Compose): Add Prometheus and Grafana services to your
docker-compose.yml.
3. Configure Prometheus (# docker-compose.yml (add these services alongside synapse, postgres) prometheus: image: prom/prometheus:latest container_name: matrix_prometheus restart: unless-stopped volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml # Mount config file - prometheus_data:/prometheus # Persist data command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--web.console.libraries=/usr/share/prometheus/console_libraries' - '--web.console.templates=/usr/share/prometheus/consoles' networks: - matrix_network ports: # Expose Prometheus web UI (optional, bind to localhost for security) - "127.0.0.1:9090:9090" depends_on: - synapse grafana: image: grafana/grafana-oss:latest container_name: matrix_grafana restart: unless-stopped volumes: - grafana_data:/var/lib/grafana # Persist Grafana data (dashboards, etc.) # You can also mount provisioning files for datasources/dashboards # - ./grafana/provisioning:/etc/grafana/provisioning networks: - matrix_network environment: # Set Grafana admin user (change password!) - GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_PASSWORD=grafana_strong_password # Optional: configure anonymous access, orgs, etc. # - GF_AUTH_ANONYMOUS_ENABLED=true # - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer ports: # Expose Grafana web UI - "3000:3000" # Access via http://<your_server_ip>:3000 depends_on: - prometheus # Add node-exporter for host metrics (optional but recommended) node_exporter: image: prom/node-exporter:latest container_name: matrix_node_exporter restart: unless-stopped volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro command: - '--path.procfs=/host/proc' - '--path.sysfs=/host/sys' - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)' networks: - matrix_network # No ports needed if Prometheus scrapes it over the Docker network volumes: # Define volumes used by prometheus and grafana prometheus_data: grafana_data: postgres_data: # From previous section # synapse_data: # If using named volume for synapse dataprometheus.yml): Create a file namedprometheus.ymlin the same directory as yourdocker-compose.yml.# prometheus.yml global: scrape_interval: 15s # Default scrape interval scrape_configs: - job_name: 'synapse' static_configs: # Use 'host.docker.internal' if supported on your Docker version # Or the Synapse container's IP (less robust) # Or the Synapse service name if Synapse metrics port is EXPOSED # Best: Use IP of the Docker host machine if Prometheus is in Docker # and Synapse metrics listener binds to 0.0.0.0 # Replace <SYNAPSE_HOST_IP> with the IP of the machine running Docker - targets: ['<SYNAPSE_HOST_IP>:9092'] # Port from homeserver.yaml metrics listener labels: instance: 'synapse-main' # Label for this instance - job_name: 'node_exporter' static_configs: # Target the node_exporter container via service name and default port - targets: ['node_exporter:9100'] labels: instance: 'matrix-host' # Label for the host machine- Replace
<SYNAPSE_HOST_IP>with the actual IP address of the server running Docker that Prometheus can reach. Alternatively, investigate Docker networking options likehost.docker.internalor ensure the Synapse container's metrics port is accessible via its service name on thematrix_network. - Start the Stack:
sudo docker compose up -d - Configure Grafana:
- Access Grafana via
http://<your_server_ip>:3000. - Log in with the admin user/password set in
docker-compose.yml(e.g.,admin/grafana_strong_password). Change the password on first login. - Add Prometheus as a Data Source:
- Configuration (gear icon) -> Data Sources -> Add data source -> Prometheus.
- Set the URL to
http://prometheus:9090(using the service name). - Click "Save & Test".
- Import a Synapse Dashboard:
- Find a community Synapse dashboard for Grafana. Search on https://grafana.com/grafana/dashboards/. A popular one is often simply named "Synapse" (e.g., ID 10711, though check for newer ones).
- In Grafana: Dashboards (grid icon) -> Import -> Paste the dashboard URL or ID -> Load.
- Select your Prometheus data source when prompted.
- Click "Import".
- Edit
Analyzing Metrics: Explore the imported dashboard. Key metrics to watch:
- CPU Usage: Overall host CPU and Synapse process CPU (
process_cpu_seconds_total). - Memory Usage: Synapse process memory (
process_resident_memory_bytes). - Event Processing: Rate of incoming/outgoing events, processing time per event (
synapse_event_processing_duration_seconds_bucket). - Federation: Queue sizes (
synapse_federation_sender_pending_pdus,synapse_federation_sender_pending_edus), success/error rates for outgoing requests (synapse_http_server_requests_received, filtering by job/path). - Database: Connection pool usage (
synapse_storage_engine_connection_pool_active_connections), transaction times (synapse_storage_persisted_events_txn_duration_seconds_bucket). Requires specific metrics often enabled by database exporters (e.g.,postgres_exporter) for deeper DB analysis. - Cache Performance: Cache hit rates (
synapse_util_caches_cache_hits,synapse_util_caches_cache_misses). Low hit rates indicate memory pressure or ineffective caching. - Python GIL: Waiting time (
synapse_python_runtime_gil_wait_seconds_total). High waiting time suggests the GIL is a bottleneck, often addressed by workers.
Synapse Worker Processes
By default, Synapse runs as a single Python process (the "main" process). To overcome the limitations of the GIL and distribute load across multiple CPU cores, Synapse can be configured to run specific parts of its workload in separate processes called workers.
Concept:
- The main process still handles most core logic and coordination.
- Worker processes are configured to handle specific, often resource-intensive tasks. They communicate with the main process (and sometimes directly with the database or clients) via internal replication protocols (usually over HTTP or Redis).
Common Worker Types:
- Federation Sender: Handles sending events to other homeservers. Offloads network I/O and event formatting/signing.
- Federation Receiver: Handles incoming events from other homeservers. Offloads signature verification and initial processing.
- Client Readers: Handle client API requests (sync, messages, profiles). Distributes load from many connected clients.
- Appservice Notifier: Sends events to application services (bridges, bots).
- Event Creators: Handle the creation and persistence of new local events.
- Pusher: Manages sending push notifications.
- Background Task Processor: Runs various cleanup and maintenance tasks.
Benefits:
- Improved CPU Utilization: Multiple processes can run on multiple cores, bypassing the GIL for specific tasks.
- Increased Throughput: Can handle more concurrent federation requests, client connections, or background tasks.
- Better Resilience: An issue in one worker type might not bring down the entire Synapse instance (though dependencies exist).
Configuration:
homeserver.yaml(Main Process): Define which workers exist and how to connect to them.# homeserver.yaml worker_app: null # This identifies the main process # Define how the main process connects to workers worker_replication_host: 127.0.0.1 # Or internal Docker IP/hostname worker_replication_http_port: 9093 # Port for main->worker communication # List the configured workers worker_listeners: - type: http port: 9093 # Internal replication listener bind_addresses: ['0.0.0.0'] # Allow workers to connect resources: - names: [replication] workers: # List the types of workers you intend to run - worker_app: synapse.app.federation_sender worker_name: federation_sender_1 - worker_app: synapse.app.client_reader worker_name: client_reader_1 # Add other worker types as needed- Worker Configuration Files: Each worker needs its own configuration file, often derived from
homeserver.yaml. The key difference is theworker_appsetting, which specifies the worker's type, and potentially different listener configurations.- Example
federation_sender.yaml:# federation_sender.yaml (copy of homeserver.yaml, then modify) worker_app: synapse.app.federation_sender # Identifies this as a federation sender worker_name: federation_sender_1 # Must match name in main homeserver.yaml # Connection info for this worker to reach the main process's replication listener worker_replication_host: synapse # Service name of main process container worker_replication_http_port: 9093 # This worker might have its own metrics port listeners: - port: 9094 # Different metrics port for this worker type: metrics bind_addresses: ['0.0.0.0'] tls: false # Crucially, remove or adjust other listeners (like client http) # that this worker type shouldn't handle. # It often needs the database configuration. database: name: psycopg2 # ... same args as main config ... # Other settings inherited from main config or specifically overridden
- Example
-
Docker Compose: Add new services for each worker process, mounting their specific configuration files and potentially different command arguments.
# docker-compose.yml (simplified example adding a federation sender) services: synapse: # Main process image: matrixdotorg/synapse:latest container_name: matrix_synapse_main restart: unless-stopped volumes: - /srv/synapse/data:/data # Main homeserver.yaml is here networks: - matrix_network depends_on: - postgres command: ["run"] # Default command runs the main process federation_sender: image: matrixdotorg/synapse:latest container_name: matrix_synapse_federation_sender restart: unless-stopped volumes: # Mount the main data dir (for media, keys etc.) - /srv/synapse/data:/data # Mount the WORKER'S config file OVER the default one in the container - /srv/synapse/worker_configs/federation_sender.yaml:/data/homeserver.yaml networks: - matrix_network depends_on: - synapse # Depends on the main process being available - postgres command: ["run"] # Command is the same, config file dictates the worker type
Note: Setting up workers significantly increases configuration complexity. Start with one or two worker types (like federation sender or client reader) based on where monitoring shows bottlenecks. Using Redis for replication communication instead of HTTP can improve performance further but adds another dependency. Refer to the official Synapse worker documentation for detailed setup instructions.
Database Tuning (PostgreSQL)
If PostgreSQL itself is the bottleneck:
- Resource Allocation: Ensure the PostgreSQL container/server has sufficient RAM and CPU allocated.
- Configuration Tuning (
postgresql.conf): Adjust settings like:shared_buffers: Amount of RAM dedicated to caching data (e.g., 25% of system RAM is a common starting point).effective_cache_size: Estimate of total RAM available for caching by both Postgres and the OS (e.g., 75% of system RAM).work_mem: Memory used for sorting/hashing per operation (increase cautiously, affects concurrent operations).maintenance_work_mem: Memory for maintenance tasks likeVACUUM,CREATE INDEX.max_connections: Ensure it's high enough for Synapse's connection pool (cp_max) plus other potential connections.- Use tools like
pgtuneto get baseline recommendations.
- Indexing: Ensure appropriate indexes exist. Synapse creates necessary indexes, but custom queries or specific workload patterns might benefit from additional indexes. Use
EXPLAIN ANALYZEto check query plans. - Vacuuming: PostgreSQL requires periodic
VACUUMoperations to reclaim storage and update statistics. Autovacuum is usually enabled, but may need tuning for very high-traffic databases. - Connection Pooling: Consider external connection poolers like PgBouncer if Synapse's internal pool is insufficient or causing issues (advanced).
Caching Layers (Redis)
For very large instances, Redis can be integrated with Synapse:
- Replication Communication: As mentioned, workers can use Redis Pub/Sub for faster internal communication instead of HTTP replication.
- Caching: Synapse can be configured to use Redis for certain caches, potentially reducing database load further.
This adds another component to manage and is generally considered an advanced scaling technique.
Workshop Setting up Monitoring with Prometheus and Grafana
This workshop guides you through enabling Synapse metrics, deploying Prometheus and Grafana using Docker Compose, and importing a basic Synapse dashboard into Grafana.
Objective: Implement a monitoring stack to visualize key Synapse performance metrics.
Prerequisites:
- Your Synapse and PostgreSQL setup running via Docker Compose (from Workshop 6).
- Nginx reverse proxy configured.
- Access to your server's command line (SSH).
- Web browser.
Steps:
- Enable and Expose Synapse Metrics:
- Edit
homeserver.yaml: - Add or ensure the following configuration exists (adjust port if 9092 is in use):
- Save the file.
- Edit
- Modify
docker-compose.yml:- Edit your
docker-compose.ymlfile. - Add the
prometheus,grafana, andnode_exporterservice definitions from the "Deploy Prometheus & Grafana (Docker Compose)" section above. - Important:
- In the
prometheusservice volume mapping- ./prometheus.yml:/etc/prometheus/prometheus.yml, ensure you will createprometheus.ymllocally. - Change the default Grafana admin password (
GF_SECURITY_ADMIN_PASSWORD). - Ensure all services (
synapse,postgres,prometheus,grafana,node_exporter) are connected to the samematrix_network. - Add the
prometheus_dataandgrafana_datavolume definitions at the bottom.
- In the
- Save the file.
- Edit your
- Create
prometheus.ymlConfiguration:- In the same directory as
docker-compose.yml, createprometheus.yml: - Paste the configuration from the "Configure Prometheus (
prometheus.yml)" section above. - Crucially: Replace
<SYNAPSE_HOST_IP>in thesynapsejob'stargetslist with the actual IP address of your server (the one running Docker). Prometheus needs this to reach the port 9092 exposed by the Synapse container. For example:targets: ['192.168.1.100:9092']ortargets: ['your.server.public.ip:9092']. (Ensure your server's firewall allows access to port 9092 from the Prometheus container - typically from the Docker bridge IP range, e.g., 172.17.0.0/16, or from localhost if Prometheus runs on the host). A simpler alternative if networking allows is to target the Synapse container by service name if the port is exposed within the Docker network:targets: ['synapse:9092']- test connectivity first. - Save the file.
- In the same directory as
- Start the Monitoring Stack:
- Apply the changes and start all services:
- This will pull the Prometheus, Grafana, and Node Exporter images and start the containers.
- Verify Prometheus Targets:
- Access Prometheus UI:
http://<your_server_ip>:9090(orhttp://localhost:9090if you bound it to localhost). - Go to Status -> Targets.
- You should see
synapseandnode_exportertargets listed. Their state should beUP. IfDOWN, check:- The IP/port configuration in
prometheus.yml. - Firewall rules blocking Prometheus from accessing Synapse's port 9092 or Node Exporter's 9100.
- Whether the Synapse/Node Exporter containers are running and exposing the metrics endpoint correctly.
- Docker networking (can Prometheus resolve/reach the target?).
- The IP/port configuration in
- Access Prometheus UI:
- Configure Grafana Data Source:
- Access Grafana UI:
http://<your_server_ip>:3000. - Log in (default:
admin/your_chosen_password). Change password when prompted. - Click the Gear icon (Configuration) -> Data Sources -> Add data source.
- Select Prometheus.
- Set the Name (e.g., "Prometheus").
- Set the URL to
http://prometheus:9090. - Leave other settings as default for now.
- Click "Save & Test". It should report "Data source is working".
- Access Grafana UI:
- Import a Synapse Dashboard:
- Click the Grid icon (Dashboards) -> Import.
- In the "Import via grafana.com" field, enter a known Synapse dashboard ID. Let's try
10711(a common one, though search for others if it doesn't work well). Click "Load". - On the next screen, you can change the dashboard name if desired.
- Crucially, select your configured Prometheus data source ("Prometheus") from the dropdown at the bottom.
- Click "Import".
- Explore the Dashboard:
- You should now see the imported Synapse dashboard, populated with data scraped by Prometheus.
- Spend time exploring the different panels showing CPU, memory, event rates, federation queues, cache statistics, etc. It might take a few minutes for data to fully populate.
- Look at the Node Exporter panels (often included or available in separate dashboards) to see overall host system metrics.
Outcome: You have successfully deployed Prometheus and Grafana alongside your Synapse instance. Synapse is exporting metrics, Prometheus is scraping them, and Grafana is visualizing them through an imported dashboard. You now have a powerful tool for observing your Synapse server's performance and identifying potential bottlenecks. Regularly review these dashboards to understand your server's typical load and spot anomalies.
9. Bridging and Integrations
Matrix is designed for interoperability, not just within its own ecosystem but also with existing communication platforms. Bridging allows you to connect Matrix rooms to channels or conversations on other networks like IRC, Slack, Telegram, WhatsApp, Discord, etc. This is achieved through Application Services (AS).
Concept of Matrix Bridging
A bridge is a specific type of Application Service that acts as a translator between the Matrix protocol and another protocol (e.g., IRC, XMPP, Slack's API).
- Puppeting: Bridges often employ "puppeting," where the bridge logs into the remote network as you (using your credentials for that network) and relays messages back and forth. When you send a message in the Matrix room, the bridge sends it as your user on the other network, and vice-versa. This provides a more seamless experience than simple relay bots. Double-puppeting bridges go further, fully syncing read receipts and typing notifications.
- Relay Bots: Simpler bridges might just act as a relay bot, where messages from the remote network appear as coming from a single bridge bot user in Matrix, and messages from Matrix users might appear as coming from a single bot user on the remote network.
- Room Plumbing: Bridges connect a specific Matrix room to a specific channel/chat on the remote network. This connection is often called "plumbing" a room.
Application Services (AS) API
Application Services are special, trusted Matrix clients that have more privileges than regular users. They register with the homeserver and can:
- Create users and rooms automatically (often used to represent remote users/channels, e.g.,
@irc_Nick:yourdomain.com). - Read and send events in rooms they are invited to (or control).
- Listen for events matching specific criteria (e.g., all messages in rooms matching a certain alias pattern).
Bridges leverage the AS API to monitor Matrix rooms for messages to send to the remote network and to push messages received from the remote network into Matrix.
Popular Bridges
The Matrix ecosystem has bridges for many platforms, developed by various teams (e.g., Matrix.org, Tullio/Mautrix, Sorunome). Some popular examples include:
- IRC:
matrix-appservice-irc,heisenbridge(simpler, user-managed). Connects Matrix rooms to IRC channels. - Telegram:
mautrix-telegram(popular, supports puppeting),matrix-appservice-telegram. Connects Matrix direct chats/groups to Telegram chats. - Slack:
matrix-appservice-slack,mautrix-slack. Connects Matrix rooms to Slack channels. - Discord:
matrix-appservice-discord,mx-puppet-discord. Connects Matrix rooms to Discord channels. - WhatsApp:
mautrix-whatsapp. Requires running an Android VM or using a phone connection. Known for being complex and potentially unstable due to WhatsApp's nature. - Signal:
mautrix-signal. Requires linking a secondary Signal device. - XMPP:
matrix-bifrost(official bridge for XMPP/libpurple). - Email:
matrix-appservice-email. Allows receiving emails in Matrix rooms.
Maturity and Complexity: Bridge maturity varies. Some are very stable and feature-rich, while others (especially those bridging closed platforms like WhatsApp) can be more complex to set up and maintain due to API changes or platform restrictions. Puppeting bridges generally require users to provide their credentials for the remote network to the bridge, which has security implications.
Setting up a Simple Bridge (Example matrix-appservice-irc)
Let's outline the typical steps for setting up a bridge, using the official matrix-appservice-irc bridge as an example. Most bridges follow a similar pattern.
Prerequisites:
- Running Synapse server.
- Docker and Docker Compose.
Steps:
- Generate Bridge Configuration:
- Most bridges need a configuration file (e.g.,
config.yaml) and a registration file (registration.yaml). -
Often, you run the bridge's Docker image once with a generate command:
# Create a directory for the bridge config mkdir -p /srv/synapse/bridge-irc cd /srv/synapse/bridge-irc # Run the bridge image to generate default config and registration files sudo docker run -it --rm \ -v $(pwd):/data \ matrixdotorg/matrix-appservice-irc:latest \ -g -u "http://localhost:9898" -c /data/config.yaml -f /data/registration.yaml --server <YOUR_SYNAPSE_URL> --domain <YOUR_DOMAIN>-g: Generate mode.-u: The URL the bridge will listen on for Synapse to connect to it.-c,-f: Output paths for config and registration files inside the container.--server: Public base URL of your Synapse (e.g.,https://matrix.yourdomain.com).--domain: Your server name (e.g.,yourdomain.com).- This creates
config.yamlandregistration.yamlin/srv/synapse/bridge-irc.
- This creates
- Edit Bridge Configuration (
config.yaml):- Open
config.yaml(e.g.,sudo nano /srv/synapse/bridge-irc/config.yaml). - Configure bridge-specific settings:
- Open
- IRC server(s) to connect to (hostname, port, SSL).
- Username/password templates for IRC connections (if needed).
- Permissions (who can use the bridge).
- Logging levels.
- Refer to the specific bridge's documentation for details.
- Edit Registration File (
registration.yaml):- Open
registration.yaml(e.g.,sudo nano /srv/synapse/bridge-irc/registration.yaml). - This file tells Synapse about the bridge. Key fields:
- Open
id: Unique ID for the bridge.url: The URL Synapse will use to send events to the bridge (e.g.,http://irc-bridge:9898if using Docker networking, matching the-uparameter used during generation but potentially using the service name). Synapse needs to be able to reach this URL.as_token,hs_token: Secret tokens shared between Synapse and the bridge for authentication. These are generated automatically.sender_localpart: The username the bridge will use to act (e.g.,_irc_bot).namespaces: Defines which users and room aliases the bridge controls (e.g.,@irc_*:yourdomain.com,#irc_*:yourdomain.com).-
Configure Synapse (
homeserver.yaml):- Tell Synapse to load the bridge's registration file.
- Edit
homeserver.yaml: - Add or modify the
app_service_config_fileslist:app_service_config_files: # Add the path *as seen from inside the Synapse container* # If you mount the bridge config dir into Synapse container: # - /bridge-irc-config/registration.yaml # Or use an absolute path on the host if Synapse data dir includes it: - /data/bridge-irc/registration.yaml # Assuming bridge dir is inside Synapse data dir
-
Important: The path must be accessible by the Synapse process. If
/srv/synapse/bridge-ircis separate from/srv/synapse/data, you might need to add another volume mount to thesynapseservice indocker-compose.ymlspecifically for the registration file(s). A simple approach is to place the bridge directory inside the main Synapse data directory. - Add Bridge Service to
docker-compose.yml:# docker-compose.yml services: # ... synapse, postgres, prometheus, grafana ... irc_bridge: image: matrixdotorg/matrix-appservice-irc:latest container_name: matrix_irc_bridge restart: unless-stopped networks: - matrix_network volumes: # Mount the bridge config directory - /srv/synapse/bridge-irc:/data # Expose the bridge's listener port *to the Docker network* # Synapse will connect to this port using the 'url' in registration.yaml # ports: # - "9898:9898" # Only expose externally if needed for debugging command: # Command to run the bridge (check bridge docs) - "-c" - "/data/config.yaml" - "-f" - "/data/registration.yaml" - "-p" - "9898" # Port the bridge listens on inside the container depends_on: - synapse # Optional, helps ensure Synapse is available # Modify Synapse service if needed to access registration file synapse: # ... existing synapse config ... volumes: - /srv/synapse/data:/data # Ensure registration.yaml is accessible, e.g., put bridge-irc dir inside data dir # Or add another mount: # - /srv/synapse/bridge-irc:/bridge-irc-config:ro - Restart Synapse and Start Bridge:
- Synapse needs to be restarted to load the new registration file:
- Start the entire stack including the new bridge:
- Using the Bridge:
- Consult the bridge's documentation on how to use it.
- Often involves inviting the bridge bot (
@_irc_bot:yourdomain.com) to a Matrix room and issuing commands (e.g.,!join #irc-channel irc.network.org) or interacting with a dedicated bridge management room.
- Most bridges need a configuration file (e.g.,
Security Implications of Bridges
- Credentials: Puppeting bridges require user credentials for remote networks. The security of these credentials depends entirely on the bridge's implementation and how securely it stores them. Consider the risks before providing credentials.
- Access Control: Ensure the bridge configuration restricts who can use it (e.g., only allow users on your homeserver).
- Attack Surface: Running a bridge adds another piece of software to maintain and secure. Keep bridges updated.
- Information Exposure: Bridging a private Matrix room to a public network (like IRC) exposes the conversation. Be mindful of room permissions and the nature of the networks being bridged.
Workshop Bridging to IRC
This workshop guides you through setting up the matrix-appservice-irc bridge to connect a Matrix room to an IRC channel on the Libera.Chat network.
Objective: Deploy and configure the IRC bridge, allowing communication between a Matrix room and an IRC channel.
Prerequisites:
- Your full Synapse stack running (Synapse, Postgres, Nginx, preferably Monitoring) via Docker Compose.
- Synapse configured and accessible (
https://matrix.yourdomain.com). - Admin access to your Synapse server.
Steps:
- Create Bridge Configuration Directory:
- Place the bridge config within the Synapse data directory for easy access by Synapse.
- ```bash
- sudo mkdir -p /srv/synapse/data/bridge-irc
- cd /srv/synapse/data/bridge-irc
- ```
- Generate Initial Bridge Config & Registration:
-
Run the generation command (adjust domain/URL):
# Run from /srv/synapse/data/bridge-irc sudo docker run -it --rm \ -v $(pwd):/data \ matrixdotorg/matrix-appservice-irc:latest \ -g \ -u "http://irc-bridge:9898" \ -c /data/config.yaml \ -f /data/registration.yaml \ --server https://matrix.yourdomain.com \ --domain yourdomain.com- Note
-u "http://irc-bridge:9898": We assume the bridge service will be namedirc-bridgein Docker Compose. - Review
registration.yaml:- Check the generated
/srv/synapse/data/bridge-irc/registration.yaml. Note theurl,as_token,hs_token,sender_localpart(_irc_), andnamespaces. You don't usually need to change this file.
- Check the generated
- Configure
config.yamlfor Libera.Chat:- Edit the bridge config file:
- Make the following changes (referencing the
matrix-appservice-ircsample config):
- Find the
homeserver:section and ensureurlanddomainmatch your Synapse setup. - Find the
ircService:section. Insideservers:, configure Libera.Chat:servers: irc.libera.chat: # Use the hostname as the key # Optional: Bridge nickname template. $SUFFIX is random. $USERID, $MATRIX_DISPLAYNAME also available. # nick: "M-$MATRIX_DISPLAYNAME" # Optional: Nickname password using NickServ # password: "your-nickserv-password" # Optional: Dynamic channels based on room aliases # dynamicChannels: # enabled: true # aliasTemplate: "#irc_#CHANNEL" # e.g., #irc_#mychannel:yourdomain.com port: 6697 # Libera.Chat SSL port ssl: true # Enable SSL # Optional: Define specific channels to bridge statically # channels: # - "#your-static-channel" # IRC channel name - In the
permissions:section underircService:, restrict who can use the bridge:Start withpermissions: # '*': # Default permissions for all servers # 'users': # Matrix users who are allowed. null = anyone. [] = no one except admin. # - '@youradmin:yourdomain.com' # Example: Only allow admin initially # 'channels': [] # Matrix room IDs allowed. null = any. 'irc.libera.chat': # Server specific permissions override '*' 'users': null # Allow any user on your homeserver to use the Libera bridge 'channels': null # Allow bridging in any Matrix roomnullfor users/channels for easier testing, then restrict later if needed. - Review other options like logging (
level: "debug"can be useful initially).- Save and close the file.
- Configure Synapse (
homeserver.yaml):- Edit Synapse config:
- Add the path to the registration file. Since we put it inside the main data directory, the path inside the container is straightforward:
- Save and close the file.
- Update
docker-compose.yml:- Edit
docker-compose.yml: - Add the
irc_bridgeservice definition (ensure it's at the same indentation level assynapse,postgresetc.):# At the 'services:' level irc_bridge: image: matrixdotorg/matrix-appservice-irc:latest container_name: matrix_irc_bridge restart: unless-stopped networks: - matrix_network volumes: # Mount the specific bridge config directory - /srv/synapse/data/bridge-irc:/data # No external ports needed command: - "-c" - "/data/config.yaml" - "-f" - "/data/registration.yaml" - "-p" - "9898" # Port inside container, matches 'url' in registration.yaml host part depends_on: - synapse - Ensure the
synapseservice definition doesn't need changes (the registration file path should be correct because we put the bridge dir inside the main data dir). - Save and close the file.
- Edit
- Restart Synapse and Start Bridge:
- Restart Synapse first to load the registration file:
Check Synapse logs (
sudo docker compose logs synapse) for confirmation that it loaded the IRC AS. - Bring up the full stack including the bridge:
- Check bridge logs:
sudo docker compose logs -f irc_bridge. Look for successful connection to Synapse and potentially to the IRC network (if configured).
- Restart Synapse first to load the registration file:
Check Synapse logs (
- Bridge a Room:
- Open your Matrix client (e.g., Element).
- Create a new, empty Matrix room (e.g., "IRC Test Room").
- Invite the bridge bot. Its ID is typically
@<sender_localpart>:<your_domain>, so likely@_irc_:yourdomain.com. - Once the bot joins, grant it Moderator or Admin privileges in the Matrix room (required by many bridges to manage plumbing). Go to Room Settings -> Roles & Permissions -> Invite -> Search for
_irc_-> Select -> Set to Moderator/Admin. - Send a command to bridge the room. For
matrix-appservice-irc, the command might be: (Replace#libera-matrix-testwith a real, preferably low-traffic, IRC channel on Libera.Chat for testing, e.g.,#matrix-bridge-testingmight exist or choose a unique one). - The bridge should respond, indicating it's joining the channel and plumbing the room. You might see IRC users join the Matrix room as virtual users (e.g.,
@irc_SomeNick:yourdomain.com).
- Test Communication:
- Send a message from your Matrix client in the bridged room. It should appear in the corresponding IRC channel (from the nick the bridge uses for you).
- Send a message from an IRC client in the bridged channel. It should appear in the Matrix room, posted by the virtual IRC user.
- Note
-
Outcome: You have successfully deployed the matrix-appservice-irc bridge, configured it for Libera.Chat, and bridged a Matrix room to an IRC channel. Messages sent in one place should now appear in the other, demonstrating a functional Matrix bridge setup. Remember to consult the specific bridge's documentation for advanced features and commands.
10. Advanced Security Hardening
While setting up a reverse proxy with TLS is a fundamental security step, further hardening is crucial for protecting your Synapse server, user data, and ensuring long-term stability. This involves regular maintenance, careful configuration, and potentially employing additional security tools.
Regular Updates
Outdated software is a primary vector for security breaches. Regularly update:
- Synapse: Monitor Synapse releases (https://github.com/matrix-org/synapse/releases). Pay attention to security advisories. Update your Docker image tag (
matrixdotorg/synapse:<new_version>) indocker-compose.ymland runsudo docker compose pull && sudo docker compose up -d. Read release notes for potential breaking changes or required configuration updates. - Operating System: Keep your host OS updated with security patches (
sudo apt update && sudo apt upgrade -yon Debian/Ubuntu). Schedule regular reboots if necessary for kernel updates. - Dependencies: Update Docker, Docker Compose, Nginx, PostgreSQL, Certbot, and any bridges or other components you have installed.
- Bridges: Bridge software also needs regular updates, especially those interacting with external services whose APIs might change.
Automating updates where possible (e.g., unattended-upgrades for the OS) is recommended, but be cautious with major Synapse or database upgrades – test them first if possible.
Reviewing homeserver.yaml for Security Settings
Periodically review your homeserver.yaml for security-relevant configurations:
- Rate Limiting: Prevent brute-force attacks and resource exhaustion. Synapse has built-in rate limiting capabilities.
Tune these values based on your server size and observed traffic. Start restrictive and loosen if necessary.
rc_login: # Rate limits for login attempts per_second: 0.166 # Allow ~1 attempt every 6 seconds burst_count: 3 # Allow small bursts rc_registration: # Rate limits for registration (if enabled with tokens/SSO) per_second: 0.016 # Allow ~1 registration per minute burst_count: 3 rc_message: # Rate limits for sending messages per_second: 0.5 # Allow 1 message every 2 seconds per user burst_count: 10 rc_admin_redaction: # Rate limit redaction API usage per_second: 0.1 burst_count: 5 # Limit concurrent requests from a single IP rc_limits: per_second: 10 burst_count: 50 # You can also define limits per-user or per-room # See documentation for 'RatelimitSettings' - IP Range Filtering: Restrict access to certain APIs (like registration, admin APIs) to specific IP addresses or ranges.
# Example: Restrict registration endpoint to internal IPs only ip_range_whitelist: - '127.0.0.1' - '::1' - '192.168.1.0/24' # Your internal network # Apply the whitelist to specific resources limit_registration_to_whitelisted_ips: true limit_login_to_whitelisted_ips: false # Careful, might lock out users - Media Repository:
max_upload_size: Prevent users from uploading excessively large files (set here and also in Nginxclient_max_body_size).media_store_path: Ensure this points to storage with sufficient space and appropriate permissions.
- URL Previews:
url_preview_enabled: trueurl_preview_ip_range_blacklist: Prevent Synapse from making requests to internal/sensitive IP ranges when generating URL previews (e.g.,127.0.0.1/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,fe80::/10). This prevents Server-Side Request Forgery (SSRF) vulnerabilities.url_preview_url_blacklist: Block specific domains from being previewed.
- Federation Allow/Deny Lists: Use
federation_domain_whitelistorfederation_domain_blacklistif needed (as discussed in Section 7). - Public Room Directory:
public_rooms_directory_enabled: true/falseallow_public_rooms_over_federation: true/falseallow_public_rooms_without_auth: true/false- Control whether your server publishes a public room list and allows unauthenticated access. Disable if not needed for a private server.
- Registration: Ensure
enable_registration: falseunless you have a specific controlled method (tokens, SSO) enabled. Double-checkregistration_requires_tokenor SSO settings. - Admin API Access: Be mindful of who has admin access (
register_new_matrix_user -a). Access to the admin API should be protected (e.g., not exposed directly by Nginx, potentially firewalled).
Content Repository Security
The media repository (media_store_path) stores all files uploaded by users.
- Scanning: Consider integrating anti-virus scanning (like ClamAV) for uploaded files. This usually involves custom scripts or external tools that monitor the media directory or hook into the upload process (if supported).
- Size Limits: Enforce
max_upload_sizein both Synapse and Nginx. - Storage: Ensure the media storage volume has adequate space and monitoring to prevent it from filling up, which could cause service denial. Consider separate, potentially cheaper storage (like S3-compatible object storage via Synapse configuration) for large deployments.
- Permissions: Ensure file permissions in the media store are restrictive.
End-to-End Encryption (E2EE) Considerations
Matrix offers strong E2EE, but its security relies on proper key management by users and clients.
- Key Backup (Secure Backup & Restore - SSSS): Encourage users to set up key backup in their clients (Element prompts for this). This encrypts their E2EE keys using a recovery passphrase or key and stores the encrypted backup on the homeserver. This allows restoring messages on new client sessions. Without backup, messages encrypted before a new login might be unreadable.
- Device Verification: Educate users on the importance of verifying other users' devices (and their own new devices) through interactive verification (QR code scan or emoji comparison). This ensures they are talking to whom they think they are and protects against Man-in-the-Middle (MitM) attacks if a homeserver were compromised.
- Server-Side Key Storage: While keys are end-to-end encrypted, the encrypted key backups reside on the homeserver. Protecting the homeserver's overall security remains critical.
- Disabling E2EE (Not Recommended): While technically possible to disable E2EE features via
homeserver.yaml, this significantly weakens privacy and is generally not recommended.
Using Tools like fail2ban
fail2ban is an intrusion prevention software framework that monitors log files for patterns matching failed login attempts or other suspicious activity and updates firewall rules to block the offending IP addresses temporarily or permanently.
Configuring fail2ban for Synapse:
- Install
fail2ban: - Create a Synapse Filter: Define patterns to match failed login attempts in Synapse logs. Synapse logs authentication failures with specific messages. Create a filter file:
sudo nano /etc/fail2ban/filter.d/matrix-synapse.conf-
Paste filter rules (these might need adjustment based on your Synapse version and log format):
# /etc/fail2ban/filter.d/matrix-synapse.conf # Fail2Ban filter for Matrix Synapse [INCLUDES] before = common.conf [Definition] # Example patterns (VERIFY these against your actual Synapse logs): # Look for failed password logins (adjust log format if needed) # Assumes format like: ... [synapse.handlers.login] ... Failed password login for user @user:domain.com from <HOST> ... # Or messages indicating rate limiting due to failed logins failregex = ^.* Failed password login .* from <HOST> .*$ ^.* Login attempt .* from <HOST> failed \d+ times .*$ ^.* Blocked sign in attempt for .* from <HOST>$ # Add more patterns if Synapse logs other relevant failure types ignoreregex = # Author: Your Name # Date: YYYY-MM-DD- Crucially: Examine your actual Synapse logs (
sudo docker compose logs synapse) during a failed login attempt to get the exact log message format and adjust thefailregexpatterns accordingly. The<HOST>part is essential forfail2banto extract the IP address. -
Create a Jail Configuration: Tell
fail2banhow to apply the filter. Create a new file (don't editjail.confdirectly):sudo nano /etc/fail2ban/jail.local- Add a section for Synapse:
# /etc/fail2ban/jail.local [DEFAULT] # Default ban settings (can be overridden per jail) bantime = 10m # Ban duration (e.g., 10 minutes) findtime = 10m # Time window to look for failures (e.g., 10 minutes) maxretry = 5 # Number of failures before banning # Add this section for your Synapse jail [matrix-synapse] enabled = true port = http,https # Check ports 80, 443 (where login requests arrive via Nginx) filter = matrix-synapse # Matches the filter file name (matrix-synapse.conf) logpath = /srv/synapse/data/homeserver.log # !!! ADJUST PATH TO YOUR LOG FILE !!! # Or use journald if Synapse logs there: systemd-journal # Or if using docker logs: /var/lib/docker/containers/*/*-json.log (more complex filtering needed) maxretry = 5 # Ban after 5 failed attempts bantime = 1h # Ban for 1 hour
-
enabled = true: Activates the jail. port = http,https: Specifies the ports associated with the service (login requests come via Nginx on 443).filter = matrix-synapse: Links to the filter definition created earlier.logpath: This is critical. Point it to the actual Synapse log file thatfail2bancan read. If using Docker with default json logging, this is complex. A common workaround is to configure Synapse (via its logging config file, e.g.,yourdomain.com.log.config) to also log to a file inside the data volume (e.g.,/data/homeserver.log) which maps to/srv/synapse/data/homeserver.logon the host. Thenfail2bancan monitor that file. Alternatively, explorefail2banintegration withjournaldif your Docker logs go there.maxretry,bantime: Customize thresholds and ban duration.- Restart
fail2ban: - Verify:
- Check
fail2banstatus:sudo fail2ban-client status - Check the specific jail status:
sudo fail2ban-client status matrix-synapse - Intentionally fail logins from a test IP address and verify that the IP gets banned after
maxretryattempts withinfindtime. Checksudo iptables -L -nor the relevant firewall command to see the ban rulesfail2banadds.
- Check
- Crucially: Examine your actual Synapse logs (
Security Implications of Open Federation vs. Closed Federation
- Open Federation (Default):
- Pros: Allows seamless communication with the entire Matrix network. Maximum interoperability.
- Cons: Exposes your server to potential spam, abuse, or unwanted contact from any other server. Requires robust rate limiting and potentially blacklist management. Higher resource usage due to processing events from many servers.
- Closed Federation (
federation_domain_whitelist):- Pros: Significantly enhanced security and control. Only communicates with explicitly trusted servers. Reduces spam/abuse vectors. Lower resource usage from federation. Ideal for internal corporate or private community deployments.
- Cons: Isolates your users from the public Matrix network. Users cannot join rooms hosted on non-whitelisted servers or communicate with users on those servers. Requires maintaining the whitelist.
The choice depends entirely on the purpose of your homeserver. For team chat within an organization, a whitelist might be appropriate. For a public community server, open federation is usually expected, requiring more attention to moderation and security settings.
Workshop Implementing Rate Limiting and Fail2ban
This workshop guides you through configuring Synapse's built-in rate limiting and setting up fail2ban to protect the login endpoint.
Objective: Harden your Synapse server against brute-force login attacks and resource abuse.
Prerequisites:
- Your full Synapse stack running (Synapse, Postgres, Nginx).
- Fail2ban installed (
sudo apt install fail2ban). - Access to your server's command line (SSH).
- Ability to determine the path to Synapse's log file as seen by the host system.
Part 1: Configure Synapse Rate Limiting
- Edit
homeserver.yaml: - Add/Modify Rate Limiting Section: Add or uncomment and adjust the following settings (place them at the top level of the YAML file, not indented under another section unless merging with an existing
rc_setting block):Feel free to adjust these values based on your expected usage, but start reasonably strict.# Rate Limiting Configuration rc_login: per_second: 0.1 # Approx 1 attempt / 10 seconds burst_count: 5 # Allow short burst rc_message: per_second: 1.0 # Allow 1 message per second / user burst_count: 10 # Allow burst of 10 # Optional: More restrictive registration limit if using tokens/SSO # rc_registration: # per_second: 0.016 # ~1 / minute # burst_count: 3 # General request limits per IP rc_limits: per_second: 5 burst_count: 20 - Restart Synapse:
- Test (Conceptual): Try logging in repeatedly with an incorrect password in quick succession. After a few attempts, your client should receive an error indicating you are being rate-limited (often an M_LIMIT_EXCEEDED error or HTTP 429).
Part 2: Configure Synapse File Logging (for Fail2ban)
Fail2ban needs direct access to log files. Docker's default logging drivers can make this tricky. The most reliable way is often to configure Synapse to log to a file within its persistent data volume.
- Locate Synapse Logging Config: The main
homeserver.yamlusually points to a logging configuration file via thelog_configsetting (e.g.,/data/yourdomain.com.log.config). Find this file.# Look inside homeserver.yaml for the log_config setting grep log_config /srv/synapse/data/homeserver.yaml # Example output: log_config: "/data/yourdomain.com.log.config" LOG_CONFIG_PATH_IN_CONTAINER=$(grep log_config /srv/synapse/data/homeserver.yaml | awk '{print $2}') LOG_CONFIG_PATH_ON_HOST="/srv/synapse/data/${LOG_CONFIG_PATH_IN_CONTAINER#/data/}" echo "Editing Log Config at: $LOG_CONFIG_PATH_ON_HOST" - Edit Logging Config:
-
Add a File Handler: Add a new handler configuration to write logs to a file. Add a
filehandler underhandlers:and include it in the root logger'shandlerslist.# Inside the logging config yaml (e.g., yourdomain.com.log.config) version: 1 formatters: precise: format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s' handlers: console: class: logging.StreamHandler formatter: precise level: INFO # ADD THIS FILE HANDLER SECTION file: class: logging.handlers.RotatingFileHandler formatter: precise filename: /data/homeserver.log # Log file *inside* the container's data volume maxBytes: 104857600 # 100 MB backupCount: 10 # Keep 10 backup logs level: INFO # Log level for the file loggers: synapse: level: INFO synapse.storage.SQL: level: INFO root: level: INFO # ADD 'file' TO THIS LIST handlers: [console, file] # Log to both console and file disable_existing_loggers: false- Ensure
filename:points to a path within the/datavolume mount (e.g.,/data/homeserver.log). - Adjust
maxBytes,backupCount, andlevelas needed. - Restart Synapse:
- Verify Log File Creation: Check that the log file now exists on the host system:
- Ensure
Part 3: Configure Fail2ban
-
Create Filter File:
- Paste the filter content provided in the main text above. Verify the
failregexpatterns by manually failing a login and checking the exact message format in/srv/synapse/data/homeserver.log. Adjust the patterns if necessary. Save the file. -
Create Jail File:
-
Add the
[matrix-synapse]jail configuration provided in the main text above. - Crucially, ensure the
logpathpoints to the host path of the log file you just configured: - Save the file.
- Restart Fail2ban:
- Check Fail2ban Status:
- Test Fail2ban:
- From a different IP address (e.g., your phone off Wi-Fi, another machine), attempt to log in to your Matrix server using Element or another client with an incorrect password multiple times (more than
maxretrywithinfindtime). - Check the status again:
sudo fail2ban-client status matrix-synapse. You should see the offending IP address listed as banned. - Verify the IP is blocked by the firewall (e.g.,
sudo iptables -L f2b-matrix-synapse -n). - Wait for the
bantimeto expire, and the IP should be automatically unbanned. You can also manually unban:sudo fail2ban-client set matrix-synapse unbanip <IP_ADDRESS>.
- Paste the filter content provided in the main text above. Verify the
Outcome: You have implemented rate limiting within Synapse to mitigate resource abuse and brute-force attempts. You have also configured Synapse to log to a file accessible by the host and set up fail2ban to monitor this log file, automatically banning IP addresses that exhibit malicious login behavior. This significantly improves the security posture of your login endpoint.
Conclusion
Congratulations! You have journeyed through the process of setting up, configuring, and hardening your own self-hosted Matrix Synapse server. Starting with the basic concepts and a simple Docker installation, you progressed through essential configurations like Nginx reverse proxy setup with TLS encryption, user management strategies, and the critical performance upgrade to a PostgreSQL database. Finally, you explored advanced topics including the intricacies of federation, performance monitoring and scaling with workers, bridging to other chat networks like IRC, and implementing robust security hardening measures like rate limiting and fail2ban.
By self-hosting Synapse, you've gained:
- Data Sovereignty: Complete control over your communication data and user accounts.
- Enhanced Privacy: Reduced reliance on third-party providers, especially when combined with Matrix's End-to-End Encryption.
- Customization: The ability to fine-tune server behavior, enable specific features, manage federation policies, and integrate with other services via bridging.
- Deep Understanding: A valuable, in-depth understanding of decentralized communication protocols, server administration, security practices, and database management.
However, running a homeserver is an ongoing responsibility. Remember the importance of:
- Regular Maintenance: Keep Synapse, your OS, database, reverse proxy, and all related components updated with security patches and new versions.
- Monitoring: Continuously monitor performance metrics (CPU, RAM, disk, database, federation queues) using tools like Prometheus and Grafana to proactively identify issues and plan for scaling.
- Backups: Implement and regularly test a robust backup strategy for your Synapse configuration files, signing keys, database (PostgreSQL backups using
pg_dump), and media storage. - Security Awareness: Stay informed about potential vulnerabilities and best practices for server hardening. Regularly review your configurations.
The Matrix ecosystem is vast and constantly evolving. Don't hesitate to delve deeper:
- Explore other Homeservers: Investigate lighter-weight alternatives like Dendrite (Go, second-gen) or Conduit (Rust).
- Try different Clients: Experiment with clients like Fluffychat, Nheko, Cinny, or Gomuks to find interfaces that suit different user preferences.
- Dive into the Specification: Read the official Matrix Specification (https://spec.matrix.org/latest/) to truly understand the protocol's inner workings.
- Engage with the Community: Join Matrix rooms like
#synapse:matrix.orgor#matrix:matrix.orgto ask questions, share experiences, and learn from other administrators and developers. - Experiment with more Bridges and Bots: Explore setting up bridges to other platforms relevant to your team or community, or look into helpful bots for moderation, RSS feeds, etc.
Self-hosting is a rewarding experience that provides unparalleled control over your digital communication. We hope this guide has equipped you with the knowledge and practical skills to confidently run and maintain your own secure and efficient Matrix Synapse server. Happy chatting!