Author	Nejat Hakan
eMail	nejat.hakan@outlook.de
PayPal Me	https://paypal.me/nejathakan

Reverse Proxy Traefik

This document provides a comprehensive guide to understanding, configuring, and mastering Traefik as a reverse proxy for your self-hosted services. We will journey from the fundamental concepts to advanced configurations, equipping you with the knowledge to effectively manage and secure your applications. Each section includes theoretical explanations followed by practical workshops to solidify your learning.

Introduction to Reverse Proxies and Traefik

Welcome to the world of reverse proxies! Before we dive into Traefik specifically, it's essential to understand what a reverse proxy is and the problems it solves, especially in a self-hosting context. This foundational knowledge will help you appreciate the power and elegance of tools like Traefik.

What is a Reverse Proxy

Imagine you have a bustling apartment complex (your server) with many different apartments (your web services like a blog, a photo gallery, a cloud storage service, etc.). If every visitor (internet user) had to know the exact apartment number and navigate complex hallways (different ports like 8080, 8081, 9000), it would be chaotic and insecure.

A reverse proxy acts like a friendly and efficient concierge or front desk for your server. It sits in front of your web services and receives all incoming requests from the internet. It then intelligently routes these requests to the correct backend service (your applications) based on rules you define.

Core Concepts and Benefits:

1. Single Point of Entry: Instead of users accessing yourserver.com:8080 for service A and yourserver.com:8081 for service B, they can access serviceA.yourserver.com and serviceB.yourserver.com, both using the standard web ports (80 for HTTP, 443 for HTTPS). The reverse proxy handles the internal port mapping. This simplifies URLs and makes your services look more professional.

2. Load Balancing: If you have multiple instances of the same application running (for high availability or to handle more traffic), a reverse proxy can distribute incoming requests among them. This prevents any single instance from being overwhelmed and improves overall performance and reliability. Traefik can automatically discover these instances when using orchestrators like Docker Swarm or Kubernetes.

3. SSL/TLS Termination: Implementing HTTPS (secure web browsing) can be complex. A reverse proxy can handle all SSL/TLS encryption and decryption at a single point. This means your backend applications don't need to worry about managing SSL certificates; they can communicate internally over HTTP, while all external communication is encrypted. Traefik excels at this with its automatic Let's Encrypt integration.

4. Enhanced Security:

   • Hiding Backend Infrastructure: The reverse proxy shields the identity and characteristics of your backend servers. Attackers only see the reverse proxy, not the individual services behind it.
   • Centralized Access Control: You can implement authentication, IP whitelisting/blacklisting, or rate limiting at the reverse proxy level, protecting all your backend services consistently.
   • Web Application Firewall (WAF) Integration: Some reverse proxies can integrate with WAFs to filter out malicious traffic.

5. Simplified Access and Management:

   • Clean URLs: As mentioned, you can use subdomains (e.g., blog.yourdomain.com) or paths (e.g., yourdomain.com/blog) to access different services, all managed by the reverse proxy.
   • Centralized Logging and Monitoring: You can gather access logs and performance metrics for all your services at the reverse proxy.

6. Serving Static Content / Caching: Some reverse proxies can efficiently serve static content (images, CSS, JavaScript) or cache responses from backend services, reducing the load on those services and speeding up response times for users.

In a self-hosting environment, where you might be running numerous applications on a single server or a small cluster, a reverse proxy is not just a convenience but a near necessity for a well-organized, secure, and maintainable setup.

Why Traefik

While several excellent reverse proxies exist (like Nginx, Apache httpd, Caddy, HAProxy), Traefik has gained immense popularity, especially in environments using containers and microservices. Here's why:

1. Automatic Service Discovery: This is Traefik's flagship feature. Traefik can listen to orchestrators like Docker, Kubernetes, Docker Swarm, Consul, etc. When you deploy a new service (e.g., a Docker container) or scale an existing one, Traefik automatically detects the change and updates its routing rules on the fly. You don't need to manually edit configuration files and restart the reverse proxy every time you add or remove a service. This is incredibly powerful for dynamic environments.

2. Seamless Let's Encrypt Integration: Traefik can automatically obtain and renew SSL/TLS certificates from Let's Encrypt (and other ACME-compatible CAs). You configure it once, and Traefik handles the entire lifecycle of your certificates, ensuring your services are always served over HTTPS.

3. Modern and Cloud-Native Design: Traefik was built from the ground up with containers and microservices in mind. It's lightweight, fast, and easy to configure using various methods including Docker labels, configuration files (YAML/TOML), or Key-Value stores.

4. User-Friendly Dashboard: Traefik comes with a clean web UI that provides a real-time overview of your entrypoints, routers, services, and middlewares. This is invaluable for understanding your current configuration and troubleshooting issues.

5. Extensible through Middlewares: Middlewares are components that can modify requests or responses, or make decisions before a request reaches your service. Traefik offers a rich set of built-in middlewares for tasks like authentication (BasicAuth, ForwardAuth), rate limiting, adding security headers, path manipulation, and more. You can chain them together to create sophisticated request processing pipelines.

6. Simplicity for Common Use Cases: For many standard self-hosting scenarios (e.g., exposing Docker containers with HTTPS), Traefik's configuration is often more concise and intuitive than traditional reverse proxies.

Traefik's Architecture - A Brief Overview:

Understanding these core components is key to working with Traefik:

• Entrypoints: These define the network ports on which Traefik listens for incoming traffic (e.g., port 80 for HTTP, port 443 for HTTPS).
• Routers: Routers analyze incoming requests from entrypoints. They use rules (e.g., based on hostname, path, headers, methods) to determine which service should handle the request. Routers can also be configured to use specific TLS settings or apply middlewares.
• Services: Services define how Traefik reaches your actual backend applications. This includes load balancing strategies and the addresses of your application instances.
• Middlewares: As mentioned, these are pluggable components that can alter the request or response, or perform actions like authentication or rate limiting. They are attached to routers.
• Providers: Providers are Traefik's way of discovering services and their configurations. Examples include the Docker provider (reads container labels), the File provider (reads configuration files), the Kubernetes Ingress provider, etc.

In essence, Traefik simplifies the complexity of routing traffic in modern, dynamic environments, making it an ideal choice for self-hosters who embrace containerization and automation.

Workshop Basic Concepts

Goal:
To visually and conceptually understand the problem that a reverse proxy like Traefik solves, without writing any code yet.

Scenario:
Imagine you are setting up a home server. You want to host three different web applications:

1. personal blog (e.g., running on Ghost, accessible at blog.yourserver.local).
2. file-sharing service (e.g., running on Nextcloud, accessible at cloud.yourserver.local).
3. recipe manager (e.g., running on Mealie, accessible at recipes.yourserver.local).

All these applications typically want to be accessed via a web browser, which defaults to port 80 (HTTP) or port 443 (HTTPS).

Steps:

1. Diagramming the Problem (Without a Reverse Proxy):

   • Take a piece of paper or use a simple diagramming tool.
   • Draw a box representing your "Server" (with a single public IP address).
   • Inside the server box, draw three smaller boxes representing your applications: "Blog," "Cloud," and "Recipes."
   • Each application, when run, will listen on a specific port. Let's say:
     • Blog listens on port 2368.
     • Cloud listens on port 8080.
     • Recipes listens on port 9000.
   • Now, draw an "Internet User" outside your server. How would they access these services?
     • To access the blog, they'd need to go to http://yourserver.local:2368.
     • To access the cloud, http://yourserver.local:8080.
     • To access recipes, http://yourserver.local:9000.
   • Discussion Points:
     • What are the drawbacks of this setup? (Ugly URLs with port numbers, users need to remember ports, only one service can use port 80/443 directly without conflicts, managing SSL for each service individually is painful).
     • What if two services accidentally try to use the same port? (One will fail to start).

2. Diagramming the Solution (With a Reverse Proxy):

   • On a new diagram, again draw your "Server" box and the three application boxes ("Blog," "Cloud," "Recipes") inside it, still with their internal ports (2368, 8080, 9000).
   • Now, add a new box within the "Server" box, positioned between the "Internet User" and your applications. Label this box "Traefik Reverse Proxy."
   • The "Traefik Reverse Proxy" box should be shown listening on standard ports 80 (HTTP) and 443 (HTTPS) from the internet.
   • Draw arrows from the "Internet User" to the "Traefik Reverse Proxy" using only ports 80/443.
     • User requests http://blog.yourserver.local.
     • User requests http://cloud.yourserver.local.
     • User requests http://recipes.yourserver.local.
   • Now, draw arrows from the "Traefik Reverse Proxy" to the respective applications:
     • If Traefik receives a request for blog.yourserver.local, it forwards it to the "Blog" application on its internal port 2368.
     • If Traefik receives a request for cloud.yourserver.local, it forwards it to the "Cloud" application on its internal port 8080.
     • If Traefik receives a request for recipes.yourserver.local, it forwards it to the "Recipes" application on its internal port 9000.
   • Discussion Points:
     • How does this improve the user experience? (Clean URLs, standard ports).
     • How does this simplify service management? (Central point for SSL, access control).
     • What information does Traefik need to do its job? (It needs rules: "if host is blog.yourserver.local, send to internal address X on port Y").

3. Identifying Key Terms on the Diagram:

   • Client: The "Internet User" or their web browser.
   • Internet: The network connecting the client to your server.
   • Entrypoint (for Traefik): The ports Traefik listens on (e.g., 80, 443).
   • Router (conceptually): The logic inside Traefik that inspects the request (e.g., Host: blog.yourserver.local) and decides where it goes.
   • Service (conceptually): The backend application (e.g., Blog, Cloud, Recipes) and how Traefik knows to connect to it (e.g., its internal IP/hostname and port).
   • Backend Services: Your applications (Blog, Cloud, Recipes).

This workshop provides a non-technical, high-level understanding. In the following sections, we'll translate these concepts into actual Traefik configurations.

1. Getting Started with Traefik using Docker

Now that you understand the "why" of reverse proxies and Traefik, let's get our hands dirty. The most common and convenient way to run Traefik, especially for self-hosting, is using Docker. This section will guide you through setting up Traefik with Docker and Docker Compose.

Prerequisites

Before we begin, ensure you have the following:

1. Docker and Docker Compose Installed:

   • Docker: A platform for developing, shipping, and running applications in containers. If you don't have it, follow the official installation guide for your operating system (Linux, macOS, Windows) from docs.docker.com.
   • Docker Compose: A tool for defining and running multi-container Docker applications. It uses a YAML file to configure your application's services. Docker Compose V2 is typically included with Docker Desktop or can be installed as a plugin for Docker Engine on Linux. Verify with docker compose version.
   • Permissions: Ensure your user can run Docker commands, usually by adding your user to the docker group (on Linux: sudo usermod -aG docker $USER, then log out and log back in).

2. Basic Understanding of Docker Concepts:

   • Image: A lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, runtime, libraries, environment variables, and configuration files.
   • Container: A running instance of a Docker image.
   • Volume: A mechanism for persisting data generated by and used by Docker containers. Volumes are managed by Docker and are the preferred way to persist data. We'll use a volume to store Traefik's Let's Encrypt certificates.
   • Network: Docker containers can communicate with each other if they are on the same Docker network. We'll create a dedicated network for Traefik and the services it proxies.
   • Port Mapping: Exposing a container's internal port to a port on the host machine, allowing external access.

3. A Domain Name (Optional for Initial Setup, Recommended for Full Functionality):

   • While you can test Traefik locally using localhost or by modifying your computer's hosts file, for features like automatic HTTPS with Let's Encrypt, you'll need a publicly registered domain name.
   • For testing, you can use free Dynamic DNS (DDNS) services like DuckDNS or No-IP. Ensure you can point this domain (or subdomains) to your server's public IP address.
   • Your server/router/firewall must allow incoming traffic on ports 80 (for HTTP) and 443 (for HTTPS).

Traefik's Core Components Explained

Let's revisit Traefik's architecture with more technical detail, as these terms will appear in our configurations:

• Entrypoints:

  • These are network listeners for Traefik. They define the port (and protocol, though usually inferred) on which Traefik accepts incoming traffic.
  • Example: You'll typically define an entrypoint named web on port 80 for HTTP traffic and an entrypoint named websecure on port 443 for HTTPS traffic.
  • Configuration is done in Traefik's static configuration (via command-line arguments or a traefik.yml file).

• Routers:

  • Routers are responsible for examining incoming requests that arrive on an entrypoint. They connect requests to services.
  • Each router is defined by a rule that specifies which requests it should handle. Rules can match based on:
    • Host: e.g., Host(\app.yourdomain.com`)`
    • PathPrefix: e.g., PathPrefix(\/api`)`
    • Headers: e.g., Headers(\Content-Type`, `application/json`)`
    • Method: e.g., Method(\GET`)`
    • And combinations of these using && (AND) or || (OR).
  • Routers also specify which entrypoint(s) they are associated with, which service to forward traffic to, and can have middlewares attached.
  • Configuration is part of Traefik's dynamic configuration (e.g., via Docker labels, file provider).

• Services:

  • Services in Traefik define how to reach your backend applications (e.g., your Docker containers).
  • A Traefik service will typically include load balancing configuration, specifying one or more servers (your application instances with their IP/hostname and port within the Docker network).
  • Traefik can also health-check these backend servers and automatically remove unhealthy instances from the load balancing pool.
  • Configuration is part of Traefik's dynamic configuration. When using the Docker provider, Traefik often creates services automatically based on container information.

• Middlewares:

  • Middlewares are pieces of logic that can modify requests or responses, or make decisions about them, before they reach your service or after the service responds.
  • Think of them as plugins in a chain. Examples:
    • BasicAuth for password protection.
    • Headers for adding or modifying HTTP headers (e.g., security headers).
    • RateLimit for preventing abuse.
    • StripPrefix for removing a path prefix before forwarding to a backend.
    • RedirectScheme for redirecting HTTP to HTTPS.
  • Middlewares are defined in the dynamic configuration and then attached to routers.

• Providers:

  • Providers are responsible for discovering the available backend services and fetching their routing configurations (routers, services, middlewares). Traefik is "provider-agnostic," meaning it can work with many different infrastructure backends.
  • Docker Provider: Watches the Docker daemon for container start/stop events and reads configuration from container labels. This is what we'll primarily use.
  • File Provider: Reads configuration from local YAML or TOML files. Useful for services not running in Docker or for complex, centralized configurations.
  • Other providers include Kubernetes CRD/Ingress, Consul, etcd, Rancher, etc.
  • Provider configuration is done in Traefik's static configuration.

Understanding this flow is crucial: Incoming Request -> Entrypoint -> Router (matches rule, applies middlewares) -> Service (load balances) -> Your Backend Application

Basic Traefik v2 Configuration with Docker Compose

We'll use Docker Compose to define and run our Traefik service. Docker Compose uses a docker-compose.yml file.

Here's a minimal docker-compose.yml to get Traefik up and running:

version: '3.8' # Specifies the Docker Compose file format version

services:
  traefik:
    image: "traefik:v2.11" # Use a specific, recent stable version of Traefik
    container_name: "traefik" # A friendly name for our Traefik container
    command:
      # Static configuration for Traefik, passed as command-line arguments
      # For detailed explanations, see the workshop steps.

      # (Optional) Enable debug logging for troubleshooting.
      # - "--log.level=DEBUG"

      # Enable the Traefik API and Dashboard (insecurely for now, for easy access).
      # The dashboard will be available on port 8080.
      - "--api.insecure=true"
      - "--api.dashboard=true" # Ensures the dashboard is enabled with api.insecure

      # Enable the Docker provider to discover services running as Docker containers.
      - "--providers.docker=true"

      # Do not expose all Docker containers to Traefik by default.
      # We will explicitly enable Traefik for containers using labels.
      - "--providers.docker.exposedbydefault=false"

      # Define an entrypoint named 'web' for HTTP traffic on port 80.
      - "--entrypoints.web.address=:80"
    ports:
      # Map port 80 on the host to port 80 in the Traefik container (for the 'web' entrypoint).
      - "80:80"

      # Map port 8080 on the host to port 8080 in the Traefik container (for the API/Dashboard).
      - "8080:8080"
    volumes:
      # Mount the Docker socket into the Traefik container (read-only).
      # This allows Traefik to listen to Docker events (e.g., container start/stop)
      # and discover services. This is crucial for the Docker provider.
      # On Linux, the path is /var/run/docker.sock.
      # On macOS/Windows with Docker Desktop, this path also works.
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      # Connect the Traefik container to a custom Docker network.
      # Services that Traefik proxies will also need to be on this network.
      - traefik_public

# Define the custom Docker network.
# 'external: false' means Docker Compose will create it if it doesn't exist.
networks:
  traefik_public:
    name: traefik_public # Explicitly name the network
    driver: bridge      # Use the default bridge driver

Explanation of the docker-compose.yml:

• version: '3.8': Specifies the Docker Compose file syntax version. It's good practice to use a recent, compatible version.
• services:: This block defines the different application services that Docker Compose will manage. Here, we only have one: traefik.
• traefik:: Defines our Traefik service.
  • image: "traefik:v2.11": Tells Docker to use the official traefik image from Docker Hub, specifically version 2.11. Using specific versions is recommended for production to avoid unexpected updates. You can check Docker Hub for the latest stable v2.x version.
  • container_name: "traefik": Assigns a predictable name to the container, making it easier to identify.
  • command:: This section passes command-line arguments to the Traefik process when it starts. These arguments form Traefik's static configuration.
    • --log.level=DEBUG: (Commented out by default) Increases log verbosity. Extremely useful for troubleshooting, but can be noisy for regular operation.
    • --api.insecure=true: Enables Traefik's API. Setting it to insecure means the API (and dashboard) will be accessible without authentication. This is fine for initial local testing but MUST be secured before exposing to the internet. The API runs on port 8080 by default within the container.
    • --api.dashboard=true: Explicitly enables the web dashboard. Paired with api.insecure=true, it makes the dashboard accessible.
    • --providers.docker=true: Enables the Docker configuration provider. This tells Traefik to listen to Docker events and look for container labels to configure routing.
    • --providers.docker.exposedbydefault=false: A crucial security and organization setting. If true, Traefik would try to create routes for every container running on the Docker host, which is usually not desired. By setting it to false, we instruct Traefik to only manage containers that are explicitly "enabled" via a Docker label (which we'll see later).
    • --entrypoints.web.address=:80: Defines an entrypoint named web. It will listen on port 80 for all network interfaces (:) inside the container. This will be our HTTP entrypoint.
  • ports:: This maps ports from your host machine to ports inside the Traefik container.
    • "80:80": Maps port 80 on your host machine to port 80 inside the Traefik container. This allows external HTTP traffic to reach the web entrypoint.
    • "8080:8080": Maps port 8080 on your host machine to port 8080 inside the Traefik container. This makes the Traefik API and dashboard (which listens on 8080 due to --api.insecure=true) accessible via http://your-server-ip:8080.
  • volumes:: This defines volume mounts.
    • "/var/run/docker.sock:/var/run/docker.sock:ro": This is critical for the Docker provider. It mounts the Docker Unix socket from the host machine into the Traefik container in read-only (ro) mode. Traefik uses this socket to communicate with the Docker API, receive events (like container starts/stops), and inspect container metadata (including labels).
  • networks:: Connects the Traefik container to a specific Docker network.
    • - traefik_public: Attaches the traefik service to the traefik_public network.
• networks:: This top-level block defines the Docker networks used by the services.
  • traefik_public:: Defines the network named traefik_public.
    • name: traefik_public: Explicitly names the Docker network. This is useful so that other docker-compose.yml files or manually run containers can easily join this same network.
    • driver: bridge: Uses the standard Docker bridge network driver, which creates a private internal network for the containers.

Workshop Your First Traefik Setup

Goal: Deploy this basic Traefik instance using Docker Compose and access its dashboard.

Steps:

1. Create Project Directory and docker-compose.yml File:

   • Open your terminal or command prompt.
   • Create a new directory for your Traefik project, for example, mkdir my-traefik-setup && cd my-traefik-setup.
   • Inside this directory, create a file named docker-compose.yml.
   • Copy the YAML content provided in the "Basic Traefik v2 Configuration with Docker Compose" section above into your docker-compose.yml file. Save the file.

2. Understanding Each Line (Recap and Elaboration):

   • version: '3.8': We're using a modern Docker Compose file format. This version supports the features we need.
   • services: traefik:: We're defining a single service named traefik.
   • image: "traefik:v2.11": We pull the official Traefik image, version 2.11. Pinning versions ensures reproducibility. For production, you'd monitor for security updates and plan upgrades.
   • container_name: "traefik": Makes it easy to find our container with docker ps.
   • command: block:
     • --api.insecure=true & --api.dashboard=true: These enable the dashboard on port 8080 without any security. This is ONLY for learning and local testing. In later sections, we will secure this. The API allows Traefik to expose its current configuration and metrics, and the dashboard uses this API.
     • --providers.docker=true: Tells Traefik, "Watch Docker! If new containers start or stop, or if their labels change, reconfigure yourself."
     • --providers.docker.exposedbydefault=false: This is a best practice. Without it, if you ran, say, a database container, Traefik might try to make it web-accessible, which is rarely what you want without explicit configuration. We will choose which containers Traefik manages.
     • --entrypoints.web.address=:80: This defines an "entrypoint" (a listening point) named web that listens on port 80 for any incoming IP address within the container. This will be our standard HTTP port.
   • ports: block:
     • "80:80": This maps port 80 on your host machine to port 80 inside the Traefik container. So, when traffic hits your server's IP on port 80, it gets directed to Traefik's web entrypoint.
     • "8080:8080": Maps port 8080 on your host machine to port 8080 inside the Traefik container. This is how we'll access the Traefik dashboard for now.
   • volumes: block:
     • "/var/run/docker.sock:/var/run/docker.sock:ro": This is the magic that lets Traefik talk to Docker. The Docker socket is a file that the Docker daemon listens on for API commands. By mounting it (read-only for security), Traefik can ask Docker "what containers are running?" and "what are their labels?".
   • networks: - traefik_public: This tells Docker to connect our traefik container to a network we're about to define called traefik_public. Services that Traefik will manage must also be on this network so Traefik can route traffic to them.
   • networks: traefik_public: name: traefik_public driver: bridge: This defines the traefik_public network.
     • name: traefik_public: Giving it an explicit name is good because other Docker Compose files or even docker run commands can later join this same network by referencing its name.
     • driver: bridge: This is the standard Docker network type for allowing containers on the same host to communicate.

3. Run Traefik with Docker Compose:

   • In your terminal, make sure you are in the my-traefik-setup directory (or wherever you saved your docker-compose.yml).

   • Execute the command:

     docker compose up -d
     

     • docker compose: The command to interact with Docker Compose (V2 syntax). If you have an older version, it might be docker-compose (with a hyphen).
     • up: This command builds, (re)creates, starts, and attaches to containers for a service.
     • -d: Stands for "detached" mode. It runs the containers in the background and prints the new container names. Without -d, you'd see Traefik's logs directly in your terminal.

   • You should see output similar to this (names might vary slightly):

     [+] Running 2/2
      ⠿ Network my-traefik-setup_traefik_public  Created                                                                           0.1s
      ⠿ Container traefik                        Started                                                                           0.8s
     

     This indicates the traefik_public network was created (or already existed) and the traefik container was started.

4. Verify Traefik is Running:

   • You can check the status of your containers:

     docker ps
     

     You should see the traefik container listed with status "Up".
   • You can also check Traefik's logs (especially if something seems wrong):

     docker logs traefik
     

     Look for messages indicating that Traefik has started successfully and is listening on the configured entrypoints. You should see lines like: Configuration loaded from command line arguments. Traefik version 2.11.0 built on ... Starting provider *docker.Provider ... Starting provider *traefik.Provider ... (for internal API/dashboard)

5. Access the Traefik Dashboard:

   • Open your web browser.
   • Navigate to http://localhost:8080 or http://<your-server-ip>:8080 (replace <your-server-ip> with the actual IP address of the machine where Docker is running).
   • You should see the Traefik Dashboard!

6. Explore the Dashboard:

   • Take a few minutes to look around. You'll see sections for:
     • Entrypoints: You should see your web entrypoint listed, listening on port 80. You might also see a traefik entrypoint (internal, for the API).
     • Routers: Initially, you might only see internal routers related to the API itself (e.g., dashboard@internal, api@internal) if api.insecure=true automatically creates them.
     • Services: Similarly, you might see an api@internal service.
   • At this stage, it's normal for the routers and services lists related to your own applications to be empty, as we haven't told Traefik to proxy any other services yet.

Congratulations! You have successfully deployed Traefik. It's now listening for HTTP traffic on port 80 and is ready to start managing your applications. In the next section, we'll configure it to route traffic to a simple web service.

To stop Traefik and remove the container and network created by this compose file, you can navigate to the directory containing your docker-compose.yml and run:

docker compose down

This is useful for cleaning up. For now, let's keep it running for the next workshop.

2. Exposing Your First Service with Traefik

With Traefik running, its real power comes from dynamically configuring it to route traffic to your backend applications. When using Traefik with the Docker provider, the primary way to do this is by adding specific labels to your application containers. Traefik monitors these labels and automatically creates the necessary routers and services.

Using Docker Labels for Configuration

Docker labels are key-value pairs that you can attach as metadata to Docker objects like containers, images, volumes, and networks. Traefik's Docker provider scans containers for labels prefixed with traefik. to discover how they should be exposed.

This approach is incredibly powerful because the routing configuration lives alongside the application's definition (e.g., in its own docker-compose.yml or when you run docker run ...). When the application container starts, Traefik sees its labels and configures itself. When it stops, Traefik removes the configuration.

Common Traefik Labels (for HTTP routers and services):

• traefik.enable=true:

  • This is the most fundamental label. If you've set providers.docker.exposedbydefault=false (which we did, and is recommended), you must add this label to any container you want Traefik to manage.

• traefik.http.routers.<router_name>.rule=Host(\your.domain.com`)`:

  • Defines an HTTP router.
  • <router_name>: A unique name you choose for this router (e.g., my-app-router).
  • .rule: Specifies the condition for this router to match. Host(\your.domain.com`)is a common rule, meaning this router will handle requests for that specific hostname. Other rules includePathPrefix(`/path`),Headers(`X-My-Header`, `value`)`, etc.

• traefik.http.routers.<router_name>.entrypoints=<entrypoint_name>:

  • Tells the router which Traefik entrypoint(s) to listen on. For HTTP, this would typically be web (our entrypoint on port 80).
  • Example: traefik.http.routers.my-app-router.entrypoints=web

• traefik.http.routers.<router_name>.service=<service_name>:

  • Links the router to a Traefik service. This service will handle requests matched by this router.
  • <service_name>: A name you choose for the Traefik service associated with this application.

• traefik.http.services.<service_name>.loadbalancer.server.port=<port_number>:

  • Defines a Traefik service and how it connects to your application container.
  • <service_name>: Must match the service name used in the router's label.
  • .loadbalancer.server.port: Specifies the internal port that your application container is listening on within the Docker network. This is not a port exposed on the host, but the port the application process itself uses inside its container. For example, if a web server in a container listens on port 8000, you'd put 8000 here. Traefik, being on the same Docker network, can reach this internal port.

There are many more labels for advanced configurations, including TLS, middlewares, multiple services, health checks, etc., which we'll explore later.

Routing to a Simple Web Application

To demonstrate, we'll use a very simple web application container called traefik/whoami. It's a tiny Go server that, when accessed, prints information about the HTTP request it received (headers, host, IP addresses, etc.). This is extremely useful for testing reverse proxy setups.

We will:

1. Add the whoami service to our docker-compose.yml.
2. Ensure it's on the same traefik_public network as Traefik.
3. Add Docker labels to the whoami service to tell Traefik how to route traffic to it.

Let's say we want to access this whoami service when a user browses to http://whoami.localhost. (Note: For *.localhost domains to work, your system should resolve them to 127.0.0.1. This is standard on most modern OSes. If you use a different domain like whoami.yourdomain.com, you'll need to ensure DNS for whoami.yourdomain.com points to your server's IP.)

Workshop Exposing a "Whoami" Service

Goal: Make the traefik/whoami application accessible through Traefik using the hostname whoami.localhost (or a custom domain you control).

Steps:

1. Modify docker-compose.yml:

   • Open your docker-compose.yml file from the previous workshop.
   • We need to add a new service definition for whoami. Append the following to your services: block, making sure the indentation is correct (at the same level as the traefik: service definition):

   # ... (previous traefik service definition) ...
   
     whoami:
       image: "traefik/whoami" # A simple service that outputs request information
       container_name: "whoami_app" # A friendly name for this container
       networks:
         - traefik_public # Crucial: Must be on the same network as Traefik
       labels:
         # Enable Traefik for this container
         - "traefik.enable=true"
   
         # Define a router for HTTP traffic for this service
         # We'll name our router 'whoami-http-router'
         # The rule: match requests where the Host header is 'whoami.localhost'
         - "traefik.http.routers.whoami-http-router.rule=Host(`whoami.localhost`)"
         # (If using a real domain, replace whoami.localhost with e.g., whoami.yourdomain.com)
   
         # Assign this router to the 'web' entrypoint (our HTTP entrypoint on port 80)
         - "traefik.http.routers.whoami-http-router.entrypoints=web"
   
         # Define the service that this router will forward traffic to.
         # We'll name our Traefik service 'whoami-service'
         # This label is implicitly creating a service, but we also need to tell Traefik
         # which port the 'whoami' container is listening on.
         # The 'whoami' container listens on port 80 by default.
         - "traefik.http.services.whoami-service.loadbalancer.server.port=80"
   
         # Link the router to the service explicitly (good practice, though Traefik can infer sometimes)
         # <router_name>.service=<service_name>
         - "traefik.http.routers.whoami-http-router.service=whoami-service"
   

   Your complete docker-compose.yml should now look something like this:

   version: '3.8'
   
   services:
     traefik:
       image: "traefik:v2.11"
       container_name: "traefik"
       command:
         - "--api.insecure=true"
         - "--api.dashboard=true"
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
         - "--entrypoints.web.address=:80"
       ports:
         - "80:80"
         - "8080:8080"
       volumes:
         - "/var/run/docker.sock:/var/run/docker.sock:ro"
       networks:
         - traefik_public
   
     whoami:
       image: "traefik/whoami"
       container_name: "whoami_app"
       networks:
         - traefik_public
       labels:
         - "traefik.enable=true"
         - "traefik.http.routers.whoami-http-router.rule=Host(`whoami.localhost`)"
         - "traefik.http.routers.whoami-http-router.entrypoints=web"
         - "traefik.http.services.whoami-service.loadbalancer.server.port=80"
         - "traefik.http.routers.whoami-http-router.service=whoami-service" # Explicit link
   
   networks:
     traefik_public:
       name: traefik_public
       driver: bridge
   

2. Explain the New whoami Service Definition and Labels:

   • whoami:: Defines a new service managed by Docker Compose, named whoami.
   • image: "traefik/whoami": Specifies the Docker image to use.
   • container_name: "whoami_app": Gives our whoami container a friendly name.
   • networks: - traefik_public: This is VERY IMPORTANT. For Traefik to be able to route traffic to the whoami container, both Traefik and whoami must be part of the same Docker network. Here, we add whoami to the traefik_public network that Traefik is also on. Docker then allows network communication between these containers using their service names or container names as hostnames (e.g., Traefik can reach whoami_app on its internal Docker IP).
   • labels:: This block contains the instructions for Traefik.
     • "traefik.enable=true": This "activates" Traefik for this container. Since we set exposedbydefault=false for Traefik, this label is mandatory.
     • "traefik.http.routers.whoami-http-router.rule=Host(\whoami.localhost`)"`:
       • We're defining an HTTP router named whoami-http-router. The name can be anything descriptive.
       • The rule is Host(\whoami.localhost`). This tells Traefik: "If an incoming HTTP request has aHostheader exactly matchingwhoami.localhost`, this router should handle it."
       • Note on backticks vs quotes in rules: Traefik rule syntax uses backticks (`) around string literals like hostnames or paths. The outer quotes (") are for the YAML syntax of the label value.
     • "traefik.http.routers.whoami-http-router.entrypoints=web":
       • This assigns our whoami-http-router to listen on the web entrypoint, which we configured in Traefik's static config to be port 80.
     • "traefik.http.services.whoami-service.loadbalancer.server.port=80":
       • This defines a Traefik service named whoami-service.
       • It tells Traefik that the actual application inside the whoami_app container is listening on its internal port 80. The traefik/whoami image indeed runs its web server on port 80 by default. Traefik will forward requests to http://whoami_app:80 (using Docker's internal DNS for whoami_app).
     • "traefik.http.routers.whoami-http-router.service=whoami-service":
       • This explicitly links our whoami-http-router to the whoami-service. When the router's rule matches, traffic will be sent to this service. While Traefik can sometimes infer this if router and service names are similar and based on the container name, being explicit is clearer and more robust.

3. Update and Run Docker Compose:

   • Save your modified docker-compose.yml file.
   • In your terminal (still in the my-traefik-setup directory), run:

     docker compose up -d
     

     Docker Compose will notice the changes to the docker-compose.yml file. It will:
     • Leave the traefik container running (as its definition hasn't changed).
     • Pull the traefik/whoami image if you don't have it locally.
     • Create and start the whoami_app container, attaching it to the traefik_public network and applying the labels.
     • You should see output indicating the whoami_app container is created and started.

4. Verify in Traefik Dashboard:

   • Go back to your Traefik dashboard in your browser (e.g., http://localhost:8080).
   • You should now see new entries:
     • Under HTTP Routers: A router named whoami-http-router@docker (or similar, @docker indicates it was discovered via the Docker provider). Its rule should be Host(\whoami.localhost`)and it should be linked to theweb` entrypoint.
     • Under HTTP Services: A service named whoami-service@docker. If you click on it, you'll see it points to the internal IP address of your whoami_app container on port 80.

5. Test Access to the whoami Service:

   • Domain Setup (Crucial for Host rule):

     • If you used Host(\whoami.localhost`): Most modern operating systems (Windows, macOS, many Linux distributions) automatically resolve any*.localhostdomain to127.0.0.1` (your local machine). So, this should work out of the box if you're testing on the same machine where Docker is running.
     • If you used Host(\whoami.yourcustomdomain.com`): You must have DNS configured forwhoami.yourcustomdomain.com` to point to the public IP address of your server running Docker.
     • Alternative for local testing (if *.localhost isn't working or you want a custom name): You can edit your operating system's hosts file.
       • On Linux/macOS: sudo nano /etc/hosts
       • On Windows: C:\Windows\System32\drivers\etc\hosts (edit as Administrator)
       • Add a line like: 127.0.0.1 whoami.localtest (replace 127.0.0.1 with your server's IP if Docker is remote). Then use Host(\whoami.localtest`)in your label and accesshttp://whoami.localtest`. Remember to remove this entry later if it's just for testing.

   • Access the Service: Open a new browser tab and navigate to http://whoami.localhost (or whatever hostname you configured).

     • You should see output from the whoami service, similar to this:

       Hostname: <container_id_of_whoami_app>
       IP: 127.0.0.1
       IP: ::1
       IP: <internal_docker_ip_of_whoami_app>
       RemoteAddr: <internal_docker_ip_of_traefik_container>:port
       GET / HTTP/1.1
       Host: whoami.localhost
       User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0
       Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
       Accept-Encoding: gzip, deflate, br
       Accept-Language: en-US,en;q=0.5
       Connection: keep-alive
       Upgrade-Insecure-Requests: 1
       X-Forwarded-For: <your_actual_ip_or_docker_gateway_ip>
       X-Forwarded-Host: whoami.localhost
       X-Forwarded-Port: 80
       X-Forwarded-Proto: http
       X-Forwarded-Server: <traefik_container_hostname>
       X-Real-Ip: <your_actual_ip_or_docker_gateway_ip>
       

   • Observe the X-Forwarded-* headers. These are added by Traefik and tell the backend application (whoami in this case) about the original request from the client, such as the original host, protocol, and client IP. This is very useful for applications that need to know this information.

You have now successfully exposed your first application through Traefik! Traefik received the request for whoami.localhost on port 80, its whoami-http-router matched the host, and it forwarded the request to the whoami-service, which in turn sent it to your whoami_app container on its internal port 80.

This basic setup forms the foundation for exposing all your other self-hosted services. In the next sections, we'll build upon this by adding HTTPS, more sophisticated routing, and security features.

Introduction to Intermediate Traefik Concepts

You've successfully set up Traefik and exposed your first service using Docker labels. This is a significant first step! The basic level covered the "what" and "why" of Traefik, along with its initial deployment and the fundamental concept of routing based on hostnames.

Recap of Basic Concepts

Let's quickly recap what we've learned:

• Reverse Proxy: A server that sits in front of web applications, forwarding client requests to the appropriate backend service.
• Traefik: A modern, cloud-native reverse proxy designed for ease of use, especially with containerized applications.
• Core Components: Entrypoints (listening ports), Routers (matching rules), Services (backend definitions), and Providers (service discovery mechanisms like Docker).
• Docker Integration: Traefik can dynamically configure itself based on labels applied to Docker containers, making service discovery seamless.
• Basic Setup: We deployed Traefik using Docker Compose, configured an HTTP entrypoint, and exposed a simple whoami service accessible via a hostname.

What's Next HTTPS Middlewares File Provider

Now, we move into the intermediate territory. In this section, we'll focus on enhancing our Traefik setup with critical features for any real-world deployment:

1. Securing Services with HTTPS: HTTP is insecure as data is transmitted in plaintext. We'll learn how to configure Traefik to automatically obtain and renew SSL/TLS certificates from Let's Encrypt, enabling HTTPS for our services. This includes securing the Traefik dashboard itself.
2. Mastering Traefik Middlewares: Middlewares are powerful tools for modifying requests/responses or implementing features like authentication, security headers, rate limiting, and redirects. We'll explore common middlewares and how to apply them.
3. Using the File Provider: While Docker labels are great for containerized services, sometimes you need more complex configurations or want to manage routing rules in a centralized file. The File Provider allows Traefik to read its dynamic configuration (routers, services, middlewares) from YAML or TOML files.

Importance of Security and Customization

As you start exposing more services, security becomes paramount. HTTPS is non-negotiable for any service handling sensitive information or even just for user trust. Middlewares allow you to layer additional security measures and customize the behavior of your reverse proxy to fit your specific needs. The File Provider offers flexibility for more intricate setups or when integrating non-Dockerized applications.

By the end of this intermediate section, you'll have a much more robust, secure, and flexible Traefik deployment, capable of handling a wide variety of self-hosting scenarios.

3. Securing Traefik with HTTPS and Let's Encrypt

In today's web, HTTPS is no longer optional; it's a standard. HTTPS encrypts the communication between a user's browser and your server, protecting data from eavesdropping and tampering. It also helps verify that users are connecting to the authentic server. Let's Encrypt is a free, automated, and open Certificate Authority (CA) that makes obtaining SSL/TLS certificates incredibly easy, and Traefik integrates with it seamlessly.

Understanding HTTPS and SSL TLS Certificates

• HTTPS (HyperText Transfer Protocol Secure): It's the secure version of HTTP. When you see https:// in your browser's address bar and a padlock icon, you're using HTTPS.
• SSL/TLS (Secure Sockets Layer / Transport Layer Security): These are cryptographic protocols that provide security for communications over a network. SSL is the predecessor to TLS, and TLS is the current standard, though "SSL certificate" is still a commonly used term.
• How it Works (Simplified):
  1. When your browser connects to an HTTPS server, the server presents its SSL/TLS certificate.
  2. This certificate contains the server's public key, information about the server's identity (like its domain name), and a digital signature from a Certificate Authority (CA).
  3. Your browser checks if it trusts the CA that signed the certificate. CAs are trusted third parties (like Let's Encrypt, DigiCert, GoDaddy). Browsers and operating systems come with a pre-installed list of trusted CAs.
  4. If the CA is trusted and the certificate is valid (not expired, matches the domain), the browser uses the server's public key to establish a secure, encrypted session with the server. All subsequent data exchanged is encrypted.
• Types of Certificates (based on validation level):
  • Domain Validated (DV): The CA only verifies that the applicant controls the domain name. Let's Encrypt issues DV certificates. These are perfectly fine for most self-hosting needs and provide full encryption.
  • Organization Validated (OV): The CA verifies domain control AND does some vetting of the organization itself.
  • Extended Validation (EV): The CA performs a more thorough vetting of the organization. EV certificates used to trigger a green bar in browsers, but this UI distinction is largely gone. DV certificates from Let's Encrypt are what we'll be using. They are free, automated, and highly trusted.

Configuring Traefik for Automatic SSL with Let's Encrypt

Traefik's integration with Let's Encrypt is one of its most celebrated features. It uses the ACME (Automatic Certificate Management Environment) protocol to request, renew, and manage certificates.

Key Configuration Steps:

1. Define an HTTPS Entrypoint: Just like we defined an HTTP entrypoint (web on port 80), we need an HTTPS entrypoint (e.g., websecure on port 443).
2. Configure a Certificate Resolver: This tells Traefik how to obtain certificates. We'll configure an ACME resolver for Let's Encrypt.
3. Specify the ACME Challenge Type: Let's Encrypt needs to verify that you actually control the domain for which you're requesting a certificate. There are several ways to do this (challenges):
   • HTTP-01 Challenge: You prove domain control by placing a specific file at a specific well-known URL on your domain (e.g., http://yourdomain.com/.well-known/acme-challenge/...). Traefik can automate this if it's handling HTTP traffic on port 80 for that domain. This is often the simplest to set up.
   • DNS-01 Challenge: You prove domain control by creating a specific DNS TXT record for your domain. This method is more complex to set up initially as it requires Traefik to have API access to your DNS provider (e.g., Cloudflare, GoDaddy, AWS Route53). However, it's more flexible, allows for wildcard certificates (e.g., *.yourdomain.com), and doesn't require your server to be directly accessible on port 80 from the internet (though Traefik itself still needs to be reachable).
   • TLS-ALPN-01 Challenge: Similar to HTTP-01 but performed over TLS on port 443. Traefik also supports this. For most basic setups, the HTTP-01 challenge is sufficient and easiest. We'll start with this.
4. Persistent Storage for Certificates: Traefik needs to store the obtained certificates and associated private keys. We'll use a Docker volume for this to ensure they persist across container restarts. Let's Encrypt has rate limits, so losing your certificates and re-requesting them too often can get you temporarily blocked.
5. Tell Routers to Use HTTPS and the Resolver: For each service you want to secure, you'll update its router configuration to use the HTTPS entrypoint and specify the certificate resolver.

Securing the Traefik Dashboard

Our current Traefik dashboard is accessible via HTTP on port 8080 (--api.insecure=true). This is not secure. We need to:

1. Route the dashboard through Traefik itself using a proper hostname (e.g., traefik.yourdomain.com).
2. Secure this route with HTTPS using our Let's Encrypt resolver.
3. Add authentication (e.g., Basic Authentication) to protect it with a username and password.

Traefik has a special internal service api@internal that refers to its own API and dashboard. We can create a router that points to this service.

Workshop Enabling HTTPS for Your Services

Goal:

1. Secure the whoami service with an HTTPS certificate from Let's Encrypt.
2. Set up automatic HTTP to HTTPS redirection.
3. Secure the Traefik dashboard with HTTPS and Basic Authentication.

Prerequisites:

• A publicly registered domain name (e.g., yourdomain.com).
• DNS records for your main domain and any subdomains you plan to use (e.g., A record for whoami.yourdomain.com and traefik.yourdomain.com) pointing to your server's public IP address.
• Your server's firewall and router/NAT must allow incoming traffic on port 80 (for the HTTP-01 challenge and HTTP to HTTPS redirection) and port 443 (for HTTPS traffic).

Steps:

1. Prepare for Certificate Storage:

   • In your project directory (e.g., my-traefik-setup), create a directory to store Let's Encrypt certificates:

     mkdir letsencrypt
     

   • Create an empty JSON file for ACME to store its data. It's critical that this file is writable by the Traefik container user. Traefik's official Docker image runs as root by default (UID 0), but setting strict permissions is good practice.

     touch letsencrypt/acme.json
     chmod 600 letsencrypt/acme.json # Only owner can read/write
     

     (The chmod 600 is especially important if Traefik were running as a non-root user. With the default root user in the container, it will be able to write regardless, but it's a good habit for sensitive files.)

2. Update docker-compose.yml for Traefik: Modify the traefik service definition in your docker-compose.yml file as follows. Pay close attention to the new command arguments, ports, volumes, and labels for Traefik itself.

   version: '3.8'
   
   services:
     traefik:
       image: "traefik:v2.11"
       container_name: "traefik"
       command:
         # --- Core Traefik Settings ---
         # - "--log.level=DEBUG" # Uncomment for detailed logs during setup/troubleshooting
         - "--api.dashboard=true" # Enable the dashboard, will be exposed via a router
   
         # --- Provider Settings ---
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
   
         # --- Entrypoint Definitions ---
         # HTTP Entrypoint
         - "--entrypoints.web.address=:80"
         # HTTPS Entrypoint
         - "--entrypoints.websecure.address=:443"
   
         # --- HTTP to HTTPS Redirection ---
         # Create a global redirection from 'web' (HTTP) to 'websecure' (HTTPS)
         - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
         - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
   
         # --- Let's Encrypt (ACME) Configuration ---
         # Replace 'your-email@example.com' with your actual email address.
         # This email is used by Let's Encrypt for certificate expiration notices and important updates.
         - "--certificatesresolvers.myresolver.acme.email=your-email@example.com"
         # Define where to store the ACME certificates (the acme.json file)
         - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
         # Enable the HTTP-01 challenge for Let's Encrypt.
         # It will use the 'web' entrypoint (port 80) for this challenge.
         - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
         # For testing Let's Encrypt integration without hitting rate limits, you can use the staging server:
         # - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
   
       ports:
         # Map host port 80 to container port 80 (for 'web' entrypoint and HTTP-01 challenge)
         - "80:80"
         # Map host port 443 to container port 443 (for 'websecure' entrypoint)
         - "443:443"
         # We no longer need to expose port 8080 directly, as the dashboard will be routed.
         # - "8080:8080" # This line can be removed or commented out
   
       volumes:
         - "/var/run/docker.sock:/var/run/docker.sock:ro"
         # Mount the local 'letsencrypt' directory into the container at '/letsencrypt'
         # This ensures that your SSL certificates persist even if the container is removed.
         - "./letsencrypt:/letsencrypt"
       networks:
         - traefik_public
       labels:
         # --- Labels to Expose Traefik Dashboard Securely ---
         - "traefik.enable=true" # Enable Traefik to manage itself for the dashboard
   
         # Router for the Traefik Dashboard (HTTPS)
         # Replace 'traefik.yourdomain.com' with your desired domain for the dashboard.
         - "traefik.http.routers.traefik-dashboard.rule=Host(`traefik.yourdomain.com`)"
         - "traefik.http.routers.traefik-dashboard.entrypoints=websecure" # Use the HTTPS entrypoint
         - "traefik.http.routers.traefik-dashboard.tls.certresolver=myresolver" # Use our Let's Encrypt resolver
         - "traefik.http.routers.traefik-dashboard.service=api@internal" # Points to Traefik's internal API/dashboard service
   
         # Middleware for Basic Authentication on the Dashboard
         # Generate your user:hashed_password pair.
         # Example: user 'admin', password 'securepassword'. Use 'htpasswd' to generate the hash.
         # Command: docker run --rm httpd:alpine htpasswd -nb admin securepassword
         # This will output something like: admin:$apr1$j51gI8x...
         # Replace 'user:$$apr1$$yourhashedpassword' with your generated string.
         # The double '$$' is to escape the '$' for Docker Compose interpretation if not quoted properly.
         - "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$apr1$$RcNo82fR$$qc0F4SxX8shegK61FtzL2/" # Replace with your actual htpasswd output
         # Apply the auth middleware to the dashboard router
         - "traefik.http.routers.traefik-dashboard.middlewares=dashboard-auth"
   
   # ... (whoami service definition from previous workshop) ...
   # ... (networks definition from previous workshop) ...
   

   Explanation of Changes to Traefik Service:

   • command: Block:
     • --api.dashboard=true: We still need the dashboard enabled. api.insecure=true is GONE. Security will be handled by a router and middleware.
     • --entrypoints.websecure.address=:443: Defines a new entrypoint named websecure listening on port 443 (standard HTTPS port).
     • --entrypoints.web.http.redirections.entrypoint.to=websecure and scheme=https: These lines set up a global redirection. Any traffic coming to the web entrypoint (HTTP) will be automatically redirected to the websecure entrypoint using the HTTPS scheme.
     • --certificatesresolvers.myresolver.acme.email=your-email@example.com: Defines a certificate resolver named myresolver. Replace your-email@example.com with YOUR email. Let's Encrypt uses this to notify you about certificate issues or upcoming expirations.
     • --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json: Tells the myresolver to store certificate data in the /letsencrypt/acme.json file inside the container.
     • --certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web: Configures myresolver to use the HTTP-01 challenge type, and specifies that challenge requests should be handled by the web entrypoint (port 80). Traefik will automatically serve the necessary challenge files when Let's Encrypt tries to validate your domain.
     • --certificatesresolvers.myresolver.acme.caserver=...staging...: (Commented out) During testing, it's highly recommended to use the Let's Encrypt staging server to avoid hitting rate limits on their production server. Certificates from the staging server are not trusted by browsers (you'll get a warning), but it's perfect for verifying your setup. Once you confirm it works, comment this line out (or remove it) to use the production server and get trusted certificates.
   • ports: Block:
     • "443:443": Added to map port 443 on the host to port 443 in the Traefik container for the websecure entrypoint.
     • "8080:8080": Removed or commented out. The dashboard is no longer exposed directly on this port to the host. It will be accessed via Traefik's routing on port 443.
   • volumes: Block:
     • "./letsencrypt:/letsencrypt": Mounts the letsencrypt directory (which contains acme.json) from your host project directory into /letsencrypt inside the container. This is where Traefik, configured with acme.storage=/letsencrypt/acme.json, will save the certificates.
   • labels: Block (for Traefik's own dashboard):
     • "traefik.enable=true": Yes, Traefik will manage itself to expose its dashboard.
     • "traefik.http.routers.traefik-dashboard.rule=Host(\traefik.yourdomain.com`)": Defines a router for the dashboard. **Replacetraefik.yourdomain.com` with the actual domain you'll use for your dashboard.**
     • "traefik.http.routers.traefik-dashboard.entrypoints=websecure": The dashboard will only be accessible via the websecure (HTTPS) entrypoint.
     • "traefik.http.routers.traefik-dashboard.tls.certresolver=myresolver": Tells this router to obtain and use an SSL certificate from Let's Encrypt using our myresolver configuration.
     • "traefik.http.routers.traefik-dashboard.service=api@internal": This special service name api@internal tells Traefik to route to its own internal API/dashboard.
     • "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$apr1$$...$$...": This defines a middleware named dashboard-auth.
       • It uses basicauth (HTTP Basic Authentication).
       • users: Specifies one or more users. The format is user:hashed_password.
       • Generating htpasswd: To generate the admin:$$apr1$$... string:
         1. Use an online htpasswd generator that supports APR1-MD5 hashing, or use the htpasswd command-line tool. If you have apache2-utils installed on Linux, you can run htpasswd -nb yourusername yourpassword.
         2. A convenient Docker way: docker run --rm httpd:alpine htpasswd -nb admin securepassword (replace admin and securepassword with your desired credentials).
         3. The output will be something like admin:$apr1$j51gI8xJ$5Q5tGOKjHk5rPVJPd1Ape0.
         4. Important for Docker Compose: If your hashed password contains dollar signs ($), you need to escape them by using two dollar signs ($$) in the docker-compose.yml file (e.g., $apr1$ becomes $$apr1$$). This is because Docker Compose uses $ for variable substitution. Or, enclose the entire label value in single quotes instead of double quotes if it simplifies escaping. For the provided example admin:$$apr1$$RcNo82fR$$qc0F4SxX8shegK61FtzL2/, this is already done.
     • "traefik.http.routers.traefik-dashboard.middlewares=dashboard-auth": This applies the dashboard-auth middleware to the traefik-dashboard router. Now, anyone trying to access the dashboard will be prompted for a username and password.

3. Update Labels for the whoami Service: Now, modify the labels section of your whoami service definition to use HTTPS:

   # ... (traefik service definition) ...
   
     whoami:
       image: "traefik/whoami"
       container_name: "whoami_app"
       networks:
         - traefik_public
       labels:
         - "traefik.enable=true"
   
         # Define a NEW router for HTTPS, or update the existing one.
         # It's good practice to use a distinct name for the HTTPS router if you keep the HTTP one (though HTTP will redirect).
         # Let's name it 'whoami-secure-router'.
         # Replace 'whoami.yourdomain.com' with your actual domain for the whoami service.
         - "traefik.http.routers.whoami-secure-router.rule=Host(`whoami.yourdomain.com`)"
   
         # Assign this router to the 'websecure' (HTTPS) entrypoint
         - "traefik.http.routers.whoami-secure-router.entrypoints=websecure"
   
         # Tell this router to use our Let's Encrypt certificate resolver
         - "traefik.http.routers.whoami-secure-router.tls.certresolver=myresolver"
   
         # The service definition remains the same, as it defines how Traefik connects to the backend.
         # The whoami container still listens on port 80 internally.
         - "traefik.http.services.whoami-service.loadbalancer.server.port=80" # Internal port of whoami app
   
         # Link the secure router to the service
         - "traefik.http.routers.whoami-secure-router.service=whoami-service"
   
         # Optional: If you want to keep the old HTTP router for explicit definition before redirect,
         # you could, but the global redirect often makes it unnecessary.
         # If you had an old 'whoami-http-router', you might remove its labels or ensure it doesn't conflict.
         # For this workshop, we focus on the new secure router. The global redirect will handle HTTP requests.
   

   Explanation of Changes to whoami Labels:
   • "traefik.http.routers.whoami-secure-router.rule=Host(\whoami.yourdomain.com`)": **Replacewhoami.yourdomain.comwith your actual domain.** We're defining a router (can be a new name or reuse, but new namewhoami-secure-routeris clear) for thewhoami` service.
   • "traefik.http.routers.whoami-secure-router.entrypoints=websecure": This router now listens on the websecure (HTTPS) entrypoint.
   • "traefik.http.routers.whoami-secure-router.tls.certresolver=myresolver": This crucial line tells Traefik to enable TLS for this router and to use our myresolver (Let's Encrypt) to obtain a certificate for whoami.yourdomain.com.
   • The service part (traefik.http.services.whoami-service...) remains unchanged because the whoami container itself still listens on port 80 internally. Traefik handles the external HTTPS and talks HTTP to the backend service on the private Docker network. This is called SSL termination.

4. Final docker-compose.yml Structure: Your complete docker-compose.yml should look like this (remember to replace placeholders):

   version: '3.8'
   
   services:
     traefik:
       image: "traefik:v2.11"
       container_name: "traefik"
       command:
         - "--log.level=INFO" # Change to DEBUG for troubleshooting Let's Encrypt
         - "--api.dashboard=true"
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
         - "--entrypoints.web.address=:80"
         - "--entrypoints.websecure.address=:443"
         - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
         - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
         - "--certificatesresolvers.myresolver.acme.email=your-real-email@example.com" # !!! REPLACE !!!
         - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
         - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
         # - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory" # Use for testing
       ports:
         - "80:80"
         - "443:443"
       volumes:
         - "/var/run/docker.sock:/var/run/docker.sock:ro"
         - "./letsencrypt:/letsencrypt"
       networks:
         - traefik_public
       labels:
         - "traefik.enable=true"
         - "traefik.http.routers.traefik-dashboard.rule=Host(`traefik.yourdomain.com`)" # !!! REPLACE !!!
         - "traefik.http.routers.traefik-dashboard.entrypoints=websecure"
         - "traefik.http.routers.traefik-dashboard.tls.certresolver=myresolver"
         - "traefik.http.routers.traefik-dashboard.service=api@internal"
         - "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$apr1$$RcNo82fR$$qc0F4SxX8shegK61FtzL2/" # !!! REPLACE with your htpasswd !!!
         - "traefik.http.routers.traefik-dashboard.middlewares=dashboard-auth"
   
     whoami:
       image: "traefik/whoami"
       container_name: "whoami_app"
       networks:
         - traefik_public
       labels:
         - "traefik.enable=true"
         - "traefik.http.routers.whoami-secure-router.rule=Host(`whoami.yourdomain.com`)" # !!! REPLACE !!!
         - "traefik.http.routers.whoami-secure-router.entrypoints=websecure"
         - "traefik.http.routers.whoami-secure-router.tls.certresolver=myresolver"
         - "traefik.http.services.whoami-service.loadbalancer.server.port=80"
         - "traefik.http.routers.whoami-secure-router.service=whoami-service"
   
   networks:
     traefik_public:
       name: traefik_public
       driver: bridge
   

5. Apply Changes and Test:

   • Ensure your DNS records are pointing correctly to your server's public IP. This is critical for Let's Encrypt to verify your domain. Propagation can take time.
   • Ensure ports 80 and 443 are open on your firewall/router and forwarded to the server running Docker.
   • Save the docker-compose.yml file.
   • In your terminal, run:

     docker compose up -d
     

     This will recreate the traefik container with the new configuration and potentially the whoami container if its labels changed significantly.
   • Monitor Traefik Logs (Very Important for Let's Encrypt):

     docker logs -f traefik
     

     Look for messages related to ACME certificate acquisition. You should see Traefik attempting to get certificates for traefik.yourdomain.com and whoami.yourdomain.com.
     • Successful messages might look like: "Successfully Acknowledged ACME order", "Validating challenge for domain...", "Certificate received from ACME...".
     • If you see errors, they often provide clues (e.g., DNS not propagated, port 80 not reachable, rate limits from Let's Encrypt). If using the staging server first, errors are less impactful.
     • Common issue: Timeout during connect (likely firewall problem) or No valid IP addresses found for.... This means Let's Encrypt servers couldn't reach your server on port 80 or resolve your domain correctly.
   • Test whoami service:
     • Open your browser and go to https://whoami.yourdomain.com (use HTTPS!).
     • You should see the whoami output, and your browser should show a valid SSL certificate (lock icon). Click the lock icon to inspect the certificate; it should be issued by Let's Encrypt.
     • Try going to http://whoami.yourdomain.com (HTTP). You should be automatically redirected to the HTTPS version.

   • Test Traefik Dashboard:

     • Go to https://traefik.yourdomain.com (HTTPS, using the domain you configured).
     • You should be prompted for a username and password. Enter the credentials you generated with htpasswd.
     • Upon successful authentication, you should see the Traefik dashboard, now served securely over HTTPS.
     • Check the Routers and Services sections in the dashboard. You'll see your new secure routers (traefik-dashboard@docker, whoami-secure-router@docker) associated with the websecure entrypoint and using TLS.

   • Check acme.json: After successful certificate issuance, the letsencrypt/acme.json file should be populated with data. This file is important; back it up!

Troubleshooting Let's Encrypt:

• Use Staging First: Always uncomment the acme.caserver line for the staging server (https://acme-staging-v02.api.letsencrypt.org/directory) when first setting up or making significant changes. This avoids Let's Encrypt production rate limits.
• Check Logs: docker logs -f traefik (with log.level=DEBUG in Traefik's command for more detail) is your best friend.
• Verify DNS: Use tools like nslookup yourdomain.com or online DNS checkers to ensure your domain resolves to the correct public IP from outside your network.
• Verify Port Accessibility: Use an online port checker tool to see if ports 80 and 443 on your public IP are open and reachable from the internet.
• Firewall/Router: Double-check firewall rules on your server and port forwarding rules on your router.

You have now significantly improved your Traefik setup by implementing HTTPS for your services and dashboard, along with basic authentication for the dashboard. This is a cornerstone of secure self-hosting.

4. Mastering Traefik Middlewares

Middlewares are one of Traefik's most powerful and flexible features. They are intermediate components that can process requests and responses, allowing you to add functionality, enhance security, and modify traffic flow before it reaches your backend services or after your services have responded.

What are Middlewares

Think of middlewares as layers in an onion, or as a pipeline of processing steps. When a request comes into a router and matches its rule, it can then be passed through one or more middlewares before being forwarded to the target service. Similarly, the response from the service can pass back through middlewares before being sent to the client.

Key Characteristics:

• Modular: Each middleware typically performs a specific task (e.g., authentication, adding a header, redirecting).
• Chainable: You can apply multiple middlewares to a single router. The order in which they are defined and executed can be important.
• Configurable: Most middlewares have their own set of configuration options.
• Dynamic: Middlewares are part of Traefik's dynamic configuration, meaning they can be defined via Docker labels or in file provider configuration files.

Order of Execution:
When multiple middlewares are attached to a router, they are generally executed in the order they are specified in the router's configuration. For example, if you have an authentication middleware and a header-adding middleware, you'd typically want authentication to happen first. If authentication fails, the request might not even reach the header-adding middleware or the backend service.

Linking Middlewares to Routers:
You define a middleware with a unique name (e.g., my-auth-middleware, my-headers-middleware). Then, on a router's configuration, you specify which middlewares it should use by referencing their names. Example label for a router: traefik.http.routers.my-app.middlewares=middleware1@docker,middleware2@docker The @docker (or @file if defined in a file) suffix tells Traefik where to find the middleware definition.

Common Use Cases for Middlewares

Traefik comes with a rich set of built-in middlewares. Here are some of the most common and useful ones:

1. Authentication:

   • BasicAuth: Implements HTTP Basic Authentication (username/password prompt). We used this for the dashboard.
     • Labels: traefik.http.middlewares.<middleware_name>.basicauth.users=<user:hashed_password>,... or basicauth.usersfile=<path_to_users_file>
   • DigestAuth: A more secure alternative to Basic Authentication, though less commonly used.
   • ForwardAuth: Delegates authentication to an external service. Traefik forwards the request (or just headers) to an auth server. If the auth server responds with 2xx, the request is allowed. Otherwise, it's denied. This is very powerful for integrating with OAuth2/OIDC providers (like Authelia, Keycloak, Authentik, Google Sign-In).
     • Labels: traefik.http.middlewares.<middleware_name>.forwardauth.address=<auth_server_url>

2. Security Headers: Adding HTTP security headers is crucial for protecting your applications against common web vulnerabilities like Cross-Site Scripting (XSS), clickjacking, etc.

   • Headers Middleware: Allows you to add/modify custom request and response headers.
     • Common security headers to add:
       • Strict-Transport-Security (HSTS): Enforces HTTPS. traefik.http.middlewares.<name>.headers.stsseconds=31536000
       • Content-Security-Policy (CSP): Controls resources the browser is allowed to load. traefik.http.middlewares.<name>.headers.contentsecuritypolicy="your-policy-string" (Can be complex to configure correctly).
       • X-Frame-Options: Prevents clickjacking. traefik.http.middlewares.<name>.headers.framedeny=true or customframeoptionsvalue=SAMEORIGIN
       • X-Content-Type-Options: Prevents MIME-sniffing. traefik.http.middlewares.<name>.headers.contenttypenosniff=true
       • Referrer-Policy: Controls how much referrer information is sent. traefik.http.middlewares.<name>.headers.referrerpolicy="strict-origin-when-cross-origin"
       • Permissions-Policy (formerly Feature-Policy): Controls browser features. traefik.http.middlewares.<name>.headers.permissionspolicy="camera=(), microphone=()"
     • You can also set sslRedirect, stsPreload, forceSTSHeader, etc. within the headers middleware, although global HSTS via entrypoints is also common.

3. Rate Limiting:

   • RateLimit Middleware: Protects your services from brute-force attacks or excessive requests from a single client.
     • Labels: traefik.http.middlewares.<middleware_name>.ratelimit.average=<requests_per_period> and ratelimit.burst=<allowed_burst_size>. You can also set ratelimit.period=<time_period>.
     • Example: traefik.http.middlewares.my-rl.ratelimit.average=10 (10 requests/sec avg), traefik.http.middlewares.my-rl.ratelimit.burst=20 (allow bursts up to 20).

4. Path Manipulation:

   • StripPrefix: Removes a specified prefix from the URL path before forwarding the request to the backend. Useful if your backend app isn't aware of a path prefix used in the public URL.
     • Example: Public URL yourdomain.com/myapp/api, backend expects /api.
     • Labels: traefik.http.middlewares.<middleware_name>.stripprefix.prefixes=/myapp
   • ReplacePath / ReplacePathRegex: Allows more complex path modifications using simple replacement or regular expressions.

5. Redirects:

   • RedirectScheme: Redirects based on URL scheme (e.g., HTTP to HTTPS). Often handled globally at the entrypoint level, but can be used per-router.
   • RedirectRegex: Redirects requests if the URL matches a regular expression, replacing it with another URL.
     • Example: Redirect yourdomain.com/old-path to yourdomain.com/new-path.
     • Labels: traefik.http.middlewares.<name>.redirectregex.regex=^http://yourdomain.com/old/(.*), traefik.http.middlewares.<name>.redirectregex.replacement=http://yourdomain.com/new/$${1} (note $$ for $1 capture group).

6. Compression:

   • Compress Middleware: Enables gzip or brotli compression for responses, reducing bandwidth usage and speeding up load times.
     • Labels: traefik.http.middlewares.<middleware_name>.compress=true (Traefik will then negotiate content encoding with the client).

7. IP Whitelisting/Blacklisting:

   • IPWhiteList / IPAllowList (newer versions): Restricts access to specified IP addresses or CIDR ranges.
     • Labels: traefik.http.middlewares.<middleware_name>.ipallowlist.sourcerange=192.168.1.10, 10.0.0.0/24
   • There's no direct IPBlackList middleware, but it can be achieved with IPWhiteList by allowing all (0.0.0.0/0, ::/0) and then using custom request header checks or a more advanced setup, or by using other tools in front of/with Traefik. Often, firewall rules are better for blacklisting.

Configuring Middlewares via Docker Labels and File Provider

You can define and apply middlewares in two main ways:

1. Directly in Docker Labels (as shown in examples above):

   • Definition: You define the middleware configuration directly using labels on the Traefik service itself or, more commonly, on the application service container if the middleware is specific to that application's router.
     • Example: traefik.http.middlewares.my-app-headers.headers.framedeny=true
   • Application: You then apply this middleware to a router (which is also defined via labels on the application container).
     • Example: traefik.http.routers.my-app-router.middlewares=my-app-headers@docker
   • The @docker suffix tells Traefik that the middleware my-app-headers is defined via Docker labels (Traefik will scan labels on all relevant containers in networks it's attached to).

2. Using the File Provider (Dynamic Configuration Files):

   • You can define middlewares in a separate YAML or TOML file that Traefik's File Provider watches. This is excellent for:
     • Reusable middlewares (define once, use on many routers).
     • Complex middleware configurations that would be unwieldy in labels.
     • Keeping routing logic separate from docker-compose.yml.
   • Definition in file (e.g., dynamic_conf/middlewares.yml):

     http:
       middlewares:
         my-global-secure-headers:
           headers:
             frameDeny: true
             contentTypeNosniff: true
             # ... other headers ...
         another-middleware:
           rateLimit:
             average: 5
             burst: 10
     

   • Application (in Docker labels for a router): traefik.http.routers.my-app-router.middlewares=my-global-secure-headers@file,another-middleware@file The @file suffix tells Traefik to look for these middleware definitions in the configuration loaded by the File Provider.

Workshop Implementing Security Headers and Rate Limiting

Goal:
Enhance the security of our whoami service by:

1. Adding common security headers (HSTS, X-Frame-Options, X-Content-Type-Options).
2. Implementing basic rate limiting.

We will define these middlewares using Docker labels directly on the whoami service for simplicity in this workshop. Later, we'll explore the file provider.

Steps:

1. Plan Your Middlewares:

   • Security Headers Middleware: We'll name it sec-headers.
     • Strict-Transport-Security (STS): max-age=31536000 (1 year). We'll also set stsPreload and stsIncludeSubdomains.
     • X-Frame-Options: DENY (via frameDeny=true).
     • X-Content-Type-Options: nosniff (via contentTypeNosniff=true).
     • Referrer-Policy: no-referrer-when-downgrade
   • Rate Limiting Middleware: We'll name it req-limit.
     • Average: 5 requests per second.
     • Burst: 10 requests.

2. Modify docker-compose.yml for the whoami Service: Open your docker-compose.yml file and add the following labels to your whoami service. The existing labels for routing and TLS will remain, and we'll add new ones for defining and applying the middlewares.

   # ... (traefik service definition) ...
   
     whoami:
       image: "traefik/whoami"
       container_name: "whoami_app"
       networks:
         - traefik_public
       labels:
         - "traefik.enable=true"
   
         # --- Router Definition (from previous workshop) ---
         - "traefik.http.routers.whoami-secure-router.rule=Host(`whoami.yourdomain.com`)" # !!! REPLACE !!!
         - "traefik.http.routers.whoami-secure-router.entrypoints=websecure"
         - "traefik.http.routers.whoami-secure-router.tls.certresolver=myresolver"
         - "traefik.http.routers.whoami-secure-router.service=whoami-service" # Link to the service
   
         # --- Service Definition (from previous workshop) ---
         - "traefik.http.services.whoami-service.loadbalancer.server.port=80"
   
         # --- Middleware Definitions ---
         # 1. Security Headers Middleware named 'whoami-sec-headers'
         - "traefik.http.middlewares.whoami-sec-headers.headers.stsseconds=31536000" # HSTS max-age 1 year
         - "traefik.http.middlewares.whoami-sec-headers.headers.stspreload=true"      # HSTS preload
         - "traefik.http.middlewares.whoami-sec-headers.headers.stsincludesubdomains=true" # HSTS include subdomains
         - "traefik.http.middlewares.whoami-sec-headers.headers.framedeny=true"        # X-Frame-Options: DENY
         - "traefik.http.middlewares.whoami-sec-headers.headers.contenttypenosniff=true" # X-Content-Type-Options: nosniff
         - "traefik.http.middlewares.whoami-sec-headers.headers.browserxssfilter=true" # X-XSS-Protection (legacy, but often still added)
         - "traefik.http.middlewares.whoami-sec-headers.headers.referrerpolicy=no-referrer-when-downgrade"
   
         # 2. Rate Limiting Middleware named 'whoami-req-limit'
         - "traefik.http.middlewares.whoami-req-limit.ratelimit.average=5"  # Allow 5 requests per second on average
         - "traefik.http.middlewares.whoami-req-limit.ratelimit.burst=10"   # Allow bursts of up to 10 requests
         # Default period for rate limit is 1 second.
   
         # --- Apply Middlewares to the Router ---
         # The names must match the middleware definitions above, followed by @docker
         # Middlewares are applied in the order listed.
         - "traefik.http.routers.whoami-secure-router.middlewares=whoami-sec-headers@docker,whoami-req-limit@docker"
   
   # ... (networks definition) ...
   

   Explanation of New Labels for whoami:

   • Middleware Definitions:
     • Lines starting with traefik.http.middlewares.whoami-sec-headers... define our security headers middleware.
       • whoami-sec-headers is the custom name we give this middleware instance.
       • .headers.stsseconds, .headers.stspreload, etc., configure specific security headers and their values.
     • Lines starting with traefik.http.middlewares.whoami-req-limit... define our rate limiting middleware.
       • whoami-req-limit is its custom name.
       • .ratelimit.average=5 and .ratelimit.burst=10 configure the rate limiting parameters.
   • Applying Middlewares:
     • "traefik.http.routers.whoami-secure-router.middlewares=whoami-sec-headers@docker,whoami-req-limit@docker":
       • This crucial line attaches the defined middlewares to our existing whoami-secure-router.
       • whoami-sec-headers@docker refers to the security headers middleware defined via Docker labels.
       • whoami-req-limit@docker refers to the rate limit middleware defined via Docker labels.
       • The middlewares will be processed in this order: first whoami-sec-headers, then whoami-req-limit. For these two, the order might not be critical, but for others (like auth then something else), it would be.

3. Update Docker Stack and Verify:

   • Save your docker-compose.yml file.
   • In your terminal, apply the changes:

     docker compose up -d
     

     Docker Compose will recreate the whoami_app container because its labels have changed. Traefik will automatically detect this and reconfigure itself.
   • Check the Traefik Dashboard (https://traefik.yourdomain.com):
     • Navigate to HTTP -> Routers and find your whoami-secure-router@docker.
     • You should see your middlewares (whoami-sec-headers@docker, whoami-req-limit@docker) listed in its configuration.
     • You can also navigate to HTTP -> Middlewares to see the definitions of whoami-sec-headers@docker and whoami-req-limit@docker.

4. Test the Middlewares:

   • Security Headers:
     1. Open your browser and go to https://whoami.yourdomain.com.
     2. Open your browser's Developer Tools (usually by pressing F12).
     3. Go to the "Network" tab, refresh the page to capture the request to whoami.yourdomain.com.
     4. Click on the request for whoami.yourdomain.com in the list.
     5. Inspect the "Response Headers." You should now see the headers we added:
        • Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
        • X-Frame-Options: DENY
        • X-Content-Type-Options: nosniff
        • X-Xss-Protection: 1; mode=block (or similar from browserxssfilter)
        • Referrer-Policy: no-referrer-when-downgrade (Note: Content-Security-Policy was not added in this example for simplicity, as it requires careful policy definition based on application content.)
   • Rate Limiting:
     1. This is best tested with a command-line tool like curl or a simple script, or by rapidly refreshing your browser.
     2. Open your terminal and try sending requests quickly:

        # Send 6 requests in quick succession
        for i in $(seq 1 6); do curl -s -o /dev/null -w "%{http_code}\n" https://whoami.yourdomain.com; sleep 0.1; done
        

        You should see five 200 (OK) responses, and the sixth one (or subsequent ones if you send more faster than 5/sec) should return 429 Too Many Requests. The exact behavior depends on the burst capacity and how quickly Traefik replenishes the token bucket.
     3. After a short pause (e.g., a second or two), try another request. It should succeed again with a 200 as your quota replenishes.
     4. If you try to hit it very hard (e.g., 20 requests very quickly):

        for i in $(seq 1 20); do curl -s -o /dev/null -w "%{http_code}\n" https://whoami.yourdomain.com; done
        

        You'll see the first few (up to the burst limit) as 200, then many 429s.

You have now successfully defined and applied middlewares to add security headers and rate limiting to your service. This demonstrates the power of Traefik middlewares to enhance your applications without modifying the application code itself. You can explore and implement many other middlewares based on your needs.

5. Using the File Provider for Dynamic Configuration

While Docker labels are excellent for configuring Traefik for services running within Docker, there are scenarios where using a file-based configuration for Traefik's dynamic elements (routers, services, middlewares) is more advantageous. Traefik's "File Provider" allows you to define these elements in YAML or TOML files.

Static vs Dynamic Configuration in Traefik

It's crucial to understand the distinction Traefik makes:

1. Static Configuration:

   • This is the configuration that defines Traefik's fundamental behavior and is set up when Traefik starts. It rarely changes during runtime.
   • Includes:
     • Entrypoints (e.g., port 80, 443 definitions).
     • Providers enablement and their basic settings (e.g., enabling Docker provider, File provider, API/Dashboard settings).
     • Logging settings.
     • Certificate Resolvers (like our Let's Encrypt ACME setup).
     • Metrics settings (e.g., Prometheus).
   • Configured via:
     • Command-line arguments (like we've been using: --entrypoints.web.address=:80).
     • A traefik.yml (or traefik.toml) configuration file.
     • Environment variables.

2. Dynamic Configuration:

   • This is the configuration that defines how traffic is routed and processed. It can change frequently without requiring a Traefik restart (if the provider supports hot reloading, which Docker and File providers do).
   • Includes:
     • Routers (rules, entrypoints they attach to, TLS settings, middlewares they use).
     • Services (load balancer settings, server URLs).
     • Middlewares (definitions of authentication, headers, rate limits, etc.).
   • Loaded by:
     • Providers like Docker (from container labels), File (from YAML/TOML files), Kubernetes Ingress/CRD, etc.

The File Provider is specifically for loading dynamic configuration.

Advantages of the File Provider

1. Centralized Configuration: You can manage routing rules and middleware definitions for multiple services in one place or a structured set of files, rather than scattered across Docker labels in potentially many docker-compose.yml files.
2. Complex Configurations: For very long or intricate middleware chains, or complex router rules, YAML/TOML syntax in a file is often much cleaner and easier to read/maintain than extremely long Docker labels.
3. Version Control: Configuration files are easily version-controlled with Git, allowing you to track changes, revert to previous states, and collaborate more effectively.
4. Non-Dockerized Services: If you have services running outside of Docker (e.g., on the host machine or a VM that Traefik can reach), the File Provider is the primary way to define how Traefik should route traffic to them.
5. Reusable Elements: Define a middleware (like a standard set of security headers) once in a file and reference it from multiple routers, whether those routers are defined in the same file or via Docker labels.
6. Clearer Separation: Separates the Traefik routing logic from the application deployment definitions (like docker-compose.yml), which some find to be a cleaner architectural approach.

Structuring Configuration Files YAML TOML

Traefik supports both YAML and TOML for its dynamic configuration files. YAML is often preferred for its readability with nested structures.

A basic structure for a dynamic configuration file (e.g., dynamic.yml) looks like this:

http: # Root key for HTTP configurations
  routers:
    my-router-from-file: # A unique name for your router
      rule: "Host(`app-from-file.yourdomain.com`)"
      entryPoints:
        - "websecure" # List of entrypoints
      service: "my-service-from-file" # Name of the service this router points to
      tls:
        certResolver: "myresolver" # Your Let's Encrypt resolver
      middlewares:
        - "my-headers@file" # Reference a middleware defined in a file
        - "auth-middleware@docker" # You can even reference middlewares defined by other providers

  services:
    my-service-from-file: # A unique name for your service
      loadBalancer:
        servers:
          - url: "http://192.168.1.100:8080" # URL of your backend application
          # For Dockerized services on the same network as Traefik, you can use Docker's DNS:
          # - url: "http://container_name_or_service_name:internal_port"

  middlewares:
    my-headers: # Define a middleware (note: no @file suffix here, it's implicit)
      headers:
        frameDeny: true
        contentTypeNosniff: true
    # ... other middlewares ...

Key points for File Provider configuration:

• All dynamic elements (routers, services, middlewares) are typically defined under the http: key (or tcp: for TCP routers/services, udp: for UDP).
• When referencing a middleware defined within the same set of dynamic configuration files loaded by the file provider, you use its name directly (e.g., my-headers). If you need to be explicit or avoid ambiguity, you can use middleware-name@file.
• Traefik can watch for changes in these files (or the directory containing them) and reload the configuration automatically if watch=true is set for the file provider in the static configuration.

Workshop Migrating a Service Configuration to the File Provider

Goal:
Move the Traefik configuration for our whoami service from Docker labels to a dynamic configuration file managed by Traefik's File Provider.

Steps:

1. Create a Directory for Dynamic Configuration:

   • In your Traefik project directory (e.g., my-traefik-setup), create a new subdirectory to hold your dynamic configuration files. Let's call it traefik_dynamic_conf:

     mkdir traefik_dynamic_conf
     

2. Create a Dynamic Configuration File:

   • Inside traefik_dynamic_conf, create a new YAML file, for example, whoami-config.yml.
   • Add the following content to traefik_dynamic_conf/whoami-config.yml. This will define the router, service, and middlewares for whoami. We'll use a new hostname whoami-file.yourdomain.com to distinguish it from the label-configured version.

   # traefik_dynamic_conf/whoami-config.yml
   http:
     routers:
       whoami-router-from-file: # Unique name for this router
         rule: "Host(`whoami-file.yourdomain.com`)" # !!! REPLACE with your domain !!!
         entryPoints:
           - "websecure"
         service: "whoami-service-from-file" # Links to the service defined below
         tls:
           certResolver: "myresolver" # Your Let's Encrypt resolver
         middlewares:
           - "whoami-sec-headers-file" # Reference middleware defined below
           - "whoami-req-limit-file"   # Reference middleware defined below
   
     services:
       whoami-service-from-file: # Unique name for this service
         loadBalancer:
           servers:
             # 'whoami_app' is the container_name of our whoami service in docker-compose.yml.
             # Docker's internal DNS will resolve this to the container's IP on the traefik_public network.
             # Port 80 is the internal port the whoami application listens on.
             - url: "http://whoami_app:80"
   
     middlewares:
       whoami-sec-headers-file: # Define the security headers middleware
         headers:
           stsSeconds: 31536000
           stsPreload: true
           stsIncludeSubdomains: true
           frameDeny: true
           contentTypeNosniff: true
           browserXssFilter: true
           referrerPolicy: "no-referrer-when-downgrade"
   
       whoami-req-limit-file: # Define the rate limit middleware
         rateLimit:
           average: 5
           burst: 10
   

   Make sure to replace whoami-file.yourdomain.com with a domain you control and have pointed its DNS A record to your server's IP.

3. Configure Traefik to Use the File Provider:

   • Open your main docker-compose.yml file.
   • Modify the traefik service definition:
     • Add command-line arguments to enable the file provider and tell it where to look for configuration files.
     • Add a volume mount to make the traefik_dynamic_conf directory accessible inside the Traefik container.

   Update the command: and volumes: sections of your traefik service:

   # ... (inside services: traefik:) ...
       command:
         # ... (existing commands: log.level, api.dashboard, providers.docker, entrypoints, certificatesresolvers) ...
   
         # --- File Provider Configuration ---
         - "--providers.file.directory=/etc/traefik/dynamic_conf" # Path INSIDE the container where dynamic config files are located
         - "--providers.file.watch=true"                         # Automatically watch for changes in this directory (and files)
   
       # ... (ports section remains the same) ...
   
       volumes:
         - "/var/run/docker.sock:/var/run/docker.sock:ro"
         - "./letsencrypt:/letsencrypt"
         # --- Mount for Dynamic Configuration Files ---
         # Mounts your local './traefik_dynamic_conf' directory to '/etc/traefik/dynamic_conf' inside the container (read-only).
         - "./traefik_dynamic_conf:/etc/traefik/dynamic_conf:ro"
   
   # ... (rest of traefik service, labels for dashboard, etc. can remain for now) ...
   

   Explanation:
   • --providers.file.directory=/etc/traefik/dynamic_conf: This tells Traefik to load all .yml or .toml files it finds in the /etc/traefik/dynamic_conf directory inside the container.
   • --providers.file.watch=true: This enables hot reloading. If you change any file in the specified directory, Traefik will automatically detect the changes and update its configuration without needing a restart.
   • "./traefik_dynamic_conf:/etc/traefik/dynamic_conf:ro": This volume mount maps the traefik_dynamic_conf directory from your host (where you created whoami-config.yml) to /etc/traefik/dynamic_conf inside the Traefik container. We mount it read-only (ro) because Traefik only needs to read these files.

4. Adjust whoami Service Labels in docker-compose.yml:

   • Since we are now defining the routing for whoami via the file provider, we should remove or disable the Traefik-specific Docker labels from the whoami service in docker-compose.yml to avoid conflicts or redundant configurations.
   • The whoami service still needs to be on the traefik_public network so Traefik can reach it via http://whoami_app:80.
   • The traefik.enable=true label can be removed if providers.docker.exposedbydefault is false and all configuration for this service is in the file provider. If you had other label-based configurations for this service not covered by the file, you might keep it. For this workshop, we are moving everything to the file.

   Modify the whoami service in docker-compose.yml:

   # ... (traefik service definition) ...
   
     whoami:
       image: "traefik/whoami"
       container_name: "whoami_app" # This name is used in the file provider service URL
       networks:
         - traefik_public # Still essential for Traefik to reach the container
       # labels: # All Traefik-specific labels below are removed or commented out
         # - "traefik.enable=true" # No longer strictly needed if all config is in file provider
         # - "traefik.http.routers.whoami-secure-router.rule=Host(`whoami.yourdomain.com`)"
         # - "traefik.http.routers.whoami-secure-router.entrypoints=websecure"
         # - "traefik.http.routers.whoami-secure-router.tls.certresolver=myresolver"
         # - "traefik.http.services.whoami-service.loadbalancer.server.port=80" # This info is now in the file
         # - "traefik.http.routers.whoami-secure-router.service=whoami-service"
         # - "traefik.http.middlewares.whoami-sec-headers.headers.stsseconds=..."
         # - "traefik.http.middlewares.whoami-req-limit.ratelimit.average=..."
         # - "traefik.http.routers.whoami-secure-router.middlewares=whoami-sec-headers@docker,whoami-req-limit@docker"
   

   Note: The whoami container itself doesn't change. It still runs the same application. Only how Traefik is told to route to it changes.

5. Apply Changes and Test:

   • Ensure your DNS for whoami-file.yourdomain.com (or your chosen domain) is pointing to your server's IP.
   • Save both docker-compose.yml and traefik_dynamic_conf/whoami-config.yml.
   • In your terminal, run:

     docker compose up -d
     

     This will recreate the traefik container (due to new command args and volume) and potentially the whoami container (if its labels were significantly changed/removed).
   • Monitor Traefik Logs:

     docker logs -f traefik
     

     Look for messages indicating that the File Provider has started and loaded your configuration. You should see something like: Starting provider *file.Provider {\"directory\":\"/etc/traefik/dynamic_conf\",\"watch\":true} And messages about Traefik processing your whoami-config.yml. If there are syntax errors in your YAML, Traefik will log them.
   • Check Traefik Dashboard:
     • Go to your secure Traefik dashboard (e.g., https://traefik.yourdomain.com).
     • Under HTTP -> Routers, you should now see whoami-router-from-file@file. Its rule should be Host(\whoami-file.yourdomain.com`)and it should be using the middlewareswhoami-sec-headers-file@fileandwhoami-req-limit-file@file`.
     • Under HTTP -> Services, you should see whoami-service-from-file@file pointing to http://whoami_app:80.
     • Under HTTP -> Middlewares, you should see whoami-sec-headers-file@file and whoami-req-limit-file@file.
   • Test Access:
     • Open your browser and go to https://whoami-file.yourdomain.com (using the new hostname).
     • It should work just like before: you should see the whoami output, the connection should be secure with a Let's Encrypt certificate, security headers should be present, and rate limiting should be active.
   • Test Hot Reloading (Optional):
     1. While docker compose up -d is running and Traefik is up, open traefik_dynamic_conf/whoami-config.yml.
     2. Modify something, for example, change the rateLimit average in whoami-req-limit-file to average: 2.
     3. Save the file.
     4. Check docker logs -f traefik. You should see messages indicating Traefik detected a change and reloaded the configuration from the file provider.
     5. The change should take effect almost immediately. Test the rate limiting again; it should now trigger after fewer requests.

You have successfully migrated a service's Traefik configuration to the File Provider! This approach provides greater flexibility and organization for more complex setups and is essential for managing non-Dockerized services. You can now add more .yml files to the traefik_dynamic_conf directory for other services or define shared middlewares.

Introduction to Advanced Traefik Features

Having mastered the intermediate concepts of HTTPS, middlewares, and the file provider, you're now well-equipped to handle most common self-hosting scenarios with Traefik. The advanced level will explore features and strategies that push Traefik further, focusing on reliability, sophisticated traffic management, and observability.

Pushing Traefik to Its Limits

In this section, we'll delve into:

• High Availability (HA) for Traefik: Strategies to avoid Traefik itself becoming a single point of failure, ensuring your services remain accessible even if one Traefik instance goes down. This often involves running multiple Traefik instances and managing shared state, like Let's Encrypt certificates.
• Advanced Middleware Chains and Custom Error Pages: Building more complex request processing pipelines with chained middlewares and providing a better user experience with custom error pages instead of generic browser or Traefik errors. We'll also touch upon Forward Authentication for integrating with external identity providers.
• Traefik Pilot Metrics and Logging: Enhancing observability by integrating Traefik with monitoring tools like Prometheus and Grafana to visualize metrics, and configuring advanced logging for better diagnostics and tracing. (Note: Traefik Pilot as a specific SaaS product might have evolved; we'll focus on open standards like Prometheus metrics).

High Availability Custom Error Pages Advanced Middlewares

These advanced topics are crucial for production-grade self-hosted environments where uptime, robust security, and insightful monitoring are key priorities. While some HA setups can be complex, understanding the principles and available tools will allow you to scale your Traefik deployment effectively. Customizing user-facing elements like error pages and implementing sophisticated authentication flows further refines the professionalism and security of your services. Finally, good metrics and logging are indispensable for troubleshooting and performance optimization.

Let's embark on these advanced explorations to make your Traefik setup truly resilient and insightful.

6. High Availability HA for Traefik

High Availability (HA) refers to designing systems to operate continuously without failure for a long time. In the context of Traefik, it means ensuring that your reverse proxy layer remains operational even if one of its instances fails or needs maintenance. This prevents Traefik from becoming a single point of failure (SPOF) for all the services it fronts.

Why HA is Important

1. Uptime: If you have a single Traefik instance and it crashes or the server it's on goes down, all your services become inaccessible. HA aims to minimize or eliminate this downtime.
2. Maintenance: With an HA setup, you can perform maintenance (like updating Traefik, a host OS, or Docker) on one instance while other instances continue to serve traffic, allowing for zero-downtime updates.
3. Scalability: While Traefik itself is quite performant, an HA setup (often involving multiple Traefik instances behind a load balancer) can also help distribute the load of managing incoming connections, although typically the backend services are the primary scaling concern.

HA Strategies

Achieving true HA for Traefik involves several components and considerations:

1. Multiple Traefik Instances:

   • The core idea is to run at least two (preferably three or more for better fault tolerance) instances of Traefik. These can be on different physical servers, virtual machines, or nodes in a container orchestration cluster (like Kubernetes or Docker Swarm).

2. Shared Configuration and State:

   • Static Configuration: Each Traefik instance generally needs the same static configuration (entrypoints, provider settings, etc.). This can be managed via identical command-line flags, traefik.yml files (deployed to each instance), or configuration management tools.
   • Dynamic Configuration:
     • If using the Docker Provider in a clustered environment (Swarm/Kubernetes), each Traefik instance can independently discover services.
     • If using the File Provider, the dynamic configuration files need to be accessible and identical for all Traefik instances. This might involve a shared network filesystem (e.g., NFS, GlusterFS) or a mechanism to distribute/synchronize these files.
     • Let's Encrypt Certificates (ACME State): This is a critical piece of shared state. If multiple Traefik instances independently try to request certificates for the same domains, they can hit Let's Encrypt rate limits or cause conflicts. The acme.json file (or its equivalent data) must be shared and accessible by all Traefik instances that are configured as certificate resolvers.
       • Shared Volume: The simplest approach for VMs or bare-metal is to store acme.json on a shared, concurrently accessible filesystem.
       • Distributed Key-Value (KV) Store: Traefik supports using KV stores like Consul, etcd, or Zookeeper to store ACME certificates. This is a more robust and scalable solution for clustered environments. Each Traefik instance reads/writes certificate data to the central KV store.

3. Load Balancer in Front of Traefik Instances:

   • You need a mechanism to distribute incoming client traffic (on ports 80 and 443) across your multiple Traefik instances. This "front-end" load balancer also handles health checks of the Traefik instances and directs traffic only to healthy ones.
   • Cloud Load Balancers: If you're in a cloud environment (AWS, GCP, Azure), using their native load balancer services is often the easiest.
   • Hardware Load Balancers: Physical load balancing appliances.
   • Software Load Balancers on-premises:
     • Keepalived + VRRP: For IP address failover. You can have a "floating" virtual IP address that automatically moves to a backup server if the primary server (running a Traefik instance and Keepalived) fails. Each server would run its own Traefik.
     • HAProxy or Nginx: These can act as Layer 4 (TCP) or Layer 7 (HTTP) load balancers in front of your Traefik instances. Often used in conjunction with Keepalived for HA of the load balancer itself.
     • Kubernetes: Uses Services of type LoadBalancer (integrating with cloud LBs) or Ingress Controllers (where Traefik itself can be the Ingress controller, and Kubernetes handles exposing it).
     • Docker Swarm: Uses the built-in routing mesh to distribute traffic to service replicas.

Setting up Traefik with a KV Store e.g. Consul for Certificate Sharing

Using a Key-Value store like Consul is a robust way to manage shared ACME certificate data in an HA Traefik setup.

Overview of Consul:

• Consul is a service mesh solution providing a distributed, highly available, and data-center-aware service discovery and configuration system.
• It includes a distributed Key-Value store that Traefik can leverage.
• You would typically run a small Consul cluster (e.g., 3 nodes for HA of Consul itself).

Configuring Traefik to use Consul for ACME storage: In Traefik's static configuration (command-line arguments or traefik.yml):

# Command-line arguments example:
# ... (other Traefik commands) ...
--certificatesresolvers.myresolver.acme.storage= # This line would be removed or empty if using KV
--certificatesresolvers.myresolver.acme.kvprovider=consul
--certificatesresolvers.myresolver.acme.consul.endpoint=your_consul_address:8500 # e.g., consul-server1:8500
--certificatesresolvers.myresolver.acme.consul.prefix=traefik/acme_certs # Path within Consul KV store

• kvprovider=consul: Specifies Consul as the backend for ACME storage.
• consul.endpoint: The address and port of one or more Consul servers.
• consul.prefix: The key prefix under which Traefik will store ACME data in Consul.

Each Traefik instance in your HA setup would point to the same Consul cluster and prefix. When one instance obtains or renews a certificate, it writes it to Consul, and other instances can then read it from there.

Workshop Setting up Traefik with Consul for Distributed ACME Certificate Management

Goal:
Demonstrate configuring Traefik to use HashiCorp Consul as a distributed Key-Value store for ACME (Let's Encrypt) certificate storage. This is a foundational step towards a full HA Traefik deployment.

Note:
This workshop will set up a single Consul server for simplicity. In a production HA environment, you would run a Consul cluster (typically 3 or 5 servers). We will also run a single Traefik instance connecting to Consul; the benefit is that a second Traefik instance configured identically could share these certificates.

Prerequisites:

• Docker and Docker Compose.
• A public domain name and ability to pass Let's Encrypt challenges (as in the HTTPS workshop).
• Your docker-compose.yml from the HTTPS workshop (or a similar setup).

Steps:

1. Add Consul Service to docker-compose.yml: Open your docker-compose.yml file and add a new service definition for Consul:

   # ... (version and existing services like traefik, whoami) ...
   
     consul:
       image: "consul:1.18" # Use a specific, recent stable version
       container_name: "consul_server"
       ports:
         # Expose Consul's UI and API port (optional for this workshop, but useful for inspection)
         - "8500:8500"
       networks:
         - traefik_public # Traefik needs to reach Consul on this network
       volumes:
         - consul_data:/consul/data # Persist Consul data
       command: "agent -server -bootstrap-expect=1 -ui -client=0.0.0.0"
       # Explanation of Consul command:
       # agent: Runs a Consul agent.
       # -server: Enables server mode.
       # -bootstrap-expect=1: For a single-node "cluster" for dev/testing. In production, this would be >=3.
       # -ui: Enables the web UI.
       # -client=0.0.0.0: Binds client interfaces (HTTP, DNS) to all network interfaces within the container, making it accessible.
   
   # ... (existing networks definition) ...
   
   volumes: # Add a named volume for Consul data
     letsencrypt: # from previous workshop
     consul_data:
   

   Explanation of Consul Service:
   • image: "consul:1.18": Uses the official Consul image.
   • ports: - "8500:8500": Exposes Consul's HTTP API and UI on port 8500 of the host.
   • networks: - traefik_public: Puts Consul on the same network as Traefik.
   • volumes: - consul_data:/consul/data: Persists Consul's data (including KV store data).
   • command: "agent -server -bootstrap-expect=1 -ui -client=0.0.0.0": Starts Consul as a single server node with the UI enabled. client=0.0.0.0 makes the API accessible from other containers on the network (like Traefik).

2. Modify Traefik Configuration to Use Consul for ACME: In your docker-compose.yml, update the command: section for the traefik service:

   # ... (inside services: traefik:) ...
       command:
         - "--log.level=INFO" # Or DEBUG for troubleshooting
         - "--api.dashboard=true"
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
         # - "--providers.file.directory=/etc/traefik/dynamic_conf" # If using file provider
         # - "--providers.file.watch=true"                        # If using file provider
   
         - "--entrypoints.web.address=:80"
         - "--entrypoints.websecure.address=:443"
         - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
         - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
   
         # --- Let's Encrypt (ACME) Configuration with Consul ---
         - "--certificatesresolvers.myresolver.acme.email=your-real-email@example.com" # !!! REPLACE !!!
         # Remove or comment out the acme.storage line for file-based storage:
         # - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
   
         # Add ACME KV Provider configuration for Consul:
         - "--certificatesresolvers.myresolver.acme.kvprovider=consul"
         - "--certificatesresolvers.myresolver.acme.consul.endpoint=consul_server:8500" # 'consul_server' is the container name
         - "--certificatesresolvers.myresolver.acme.consul.prefix=traefik_acme_storage" # Key prefix in Consul
   
         - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
         # - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory" # Use for testing
   
       # The 'letsencrypt' volume for acme.json is no longer strictly needed by Traefik for ACME
       # if Consul is used, but you might keep it if other things use it or for fallback.
       # For this workshop, we can leave the volume mount as is, but Traefik won't use acme.json for this resolver.
       volumes:
         - "/var/run/docker.sock:/var/run/docker.sock:ro"
         - "./letsencrypt:/letsencrypt" # acme.json will likely remain empty or not updated by 'myresolver'
         # - "./traefik_dynamic_conf:/etc/traefik/dynamic_conf:ro" # If using file provider
   # ... (rest of traefik service including ports, networks, labels) ...
   

   Key Changes to Traefik Commands:
   • --certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json: This line is removed or commented out. We are telling Traefik not to use the JSON file for this resolver.
   • --certificatesresolvers.myresolver.acme.kvprovider=consul: Tells Traefik to use a KV store provider, specifically Consul.
   • --certificatesresolvers.myresolver.acme.consul.endpoint=consul_server:8500: Specifies the address of the Consul server. consul_server is the Docker service name of our Consul container, and Docker's internal DNS will resolve it. Port 8500 is Consul's default HTTP API port.
   • --certificatesresolvers.myresolver.acme.consul.prefix=traefik_acme_storage: This defines a "folder" or namespace within Consul's KV store where Traefik will store all ACME-related data for the myresolver.

3. Clear Old Certificates (Optional but Recommended for Clean Test):

   • To ensure Traefik requests new certificates and stores them in Consul, you might want to temporarily remove or rename your existing letsencrypt/acme.json file if it contains valid certificates for your test domains. This forces a new issuance cycle.

     # mv letsencrypt/acme.json letsencrypt/acme.json.backup
     # mkdir letsencrypt # if you removed the directory entirely
     # touch letsencrypt/acme.json
     # chmod 600 letsencrypt/acme.json
     

     Alternatively, use different domain names for this test if you don't want to touch existing certs. For the workshop, using the staging server (acme.caserver) is a good idea initially.

4. Update and Run Docker Compose:

   • Ensure your DNS records for the domains used by Traefik dashboard and whoami service (e.g., traefik.yourdomain.com, whoami.yourdomain.com or whoami-file.yourdomain.com) are still pointing to your server's public IP.
   • Save your docker-compose.yml file.
   • In your terminal, run:

     docker compose up -d
     

     This will start the new consul_server container and recreate the traefik container with the updated configuration.

5. Monitor Logs and Test:

   • Consul Logs: Check that Consul starts correctly:

     docker logs consul_server
     

     You should see messages like agent: Consul agent running! and agent: Synced node info.
   • Traefik Logs: This is crucial for observing the certificate acquisition process:

     docker logs -f traefik
     

     Look for messages indicating Traefik is connecting to Consul and then proceeding with ACME challenges for your domains. If you cleared old certs or are using new domains/staging, it will request new ones. You might see lines like: Provider "consul" provider ACME store is configured" Attempting to obtain ACME certificate for domains "yourdomain.com"..." Successfully Acknowledged ACME order ... Storing certificate for domain(s) ... in KV store
   • Test Service Access:
     • Access your services via HTTPS (e.g., https://traefik.yourdomain.com, https://whoami.yourdomain.com). They should load securely with valid certificates.
     • If you used the Let's Encrypt staging server, your browser will show a warning about the certificate issuer being untrusted – this is expected. Click through to verify the site loads. Once confirmed, switch to the production Let's Encrypt server by commenting out the acme.caserver line in Traefik's command and run docker compose up -d again.

6. Inspect Consul KV Store (Optional):

   • If you exposed port 8500 for Consul, you can access its web UI at http://<your-server-ip>:8500.
   • Navigate to the "Key/Value" section in the UI.
   • You should see keys stored under the prefix you defined (e.g., traefik_acme_storage/). Drilling down, you'll find data related to your ACME account and certificates. This confirms Traefik is using Consul for storage.
     • For example, you might see traefik_acme_storage/myresolver/account/meta.json and traefik_acme_storage/myresolver/certificates/...yourdomain.com.json.

   The acme.json file in your ./letsencrypt volume should now remain empty or unchanged by myresolver, as certificate data is being stored in Consul.

Implications for HA:
With this setup, if you were to launch a second Traefik instance with the exact same ACME resolver configuration (pointing to the same Consul endpoint and prefix), it would:

1. Connect to Consul.
2. Find the existing ACME account and certificates.
3. Use those certificates without needing to request new ones from Let's Encrypt.
4. Participate in renewals if it's the "leader" for that task (Traefik handles this coordination when using a KV store).

This workshop demonstrates a key piece of the HA puzzle: distributed certificate management. A full HA setup would also require a load balancer in front of multiple such Traefik instances and potentially a clustered Consul setup. However, even with a single Traefik instance, using Consul for ACME storage can be beneficial for resilience and easier migration/backup of certificate data.

7. Advanced Middleware Chains and Custom Error Pages

As your Traefik setup grows more sophisticated, you'll often need to combine multiple middlewares to achieve desired behaviors and provide a polished user experience, even when things go wrong. This section covers chaining middlewares, implementing custom error pages, and a brief look at Forward Authentication.

Chaining Middlewares

We've already seen a basic example of chaining middlewares when we applied both security headers and rate limiting to a router: "traefik.http.routers.my-router.middlewares=headers-middleware@docker,auth-middleware@docker"

The key points for chaining are:

• Order Matters: Middlewares are executed in the order they are listed in the middlewares attribute of a router. For example, you usually want an authentication middleware to run before a caching middleware or one that modifies request headers based on user identity. If authentication fails, subsequent middlewares in the chain might not even be processed for that request.
• Naming and Referencing: Each middleware definition needs a unique name (e.g., my-auth, global-headers). When applying them to a router, you list these names, followed by @docker or @file to indicate where Traefik should find their definition.

• Practical Example: Auth + Headers + Redirects Imagine you want to:

  1. Ensure the user is authenticated (e.g., BasicAuth or ForwardAuth).
  2. If authenticated, add specific headers to the request going to the backend.
  3. Perhaps perform a conditional redirect based on some criteria (though redirects are often better handled before heavy processing).

  Your router's middleware configuration might look like: traefik.http.routers.my-app.middlewares=auth-check@file,add-user-headers@file,cache-control@file

  Here, auth-check would run first. If it passes, add-user-headers runs, then cache-control. If auth-check fails (e.g., returns a 401), the request processing typically stops there, and Traefik sends the 401 response to the client.

Custom Error Pages

When an error occurs (e.g., your backend service is down resulting in a 500-range error, or Traefik can't find a route for a request resulting in a 404 Not Found), Traefik by default serves a very plain error page. For a more professional look and better user experience, you can configure Traefik to serve custom HTML error pages.

This is achieved using the errors middleware.

• How it Works: You define an errors middleware that specifies:
  • A range of HTTP status codes it should intercept (e.g., 404, 500-599).
  • A backend service that is responsible for serving the actual error page content.
  • A query parameter that tells the error page service which page to serve (often / followed by the status code, like /{status}.html).
• Error Page Service: This is typically a simple static web server (e.g., an Nginx or Apache container) that serves your custom 404.html, 500.html, 503.html, etc., files.
• Application: You apply this errors middleware to your routers. If a router (or the service it points to) generates an error status code that the errors middleware is configured to handle, Traefik will internally fetch the corresponding error page from your error page service and deliver it to the client with the original error code.

Configuration (Example using File Provider for clarity):

# In your dynamic configuration file (e.g., dynamic_conf/middlewares.yml)
http:
  middlewares:
    my-custom-errors:
      errors:
        status:
          - "400-499" # Catch all 4xx client errors
          - "500-599" # Catch all 5xx server errors
        service: "error-page-service@file" # Service that serves the error pages
        query: "/{status}.html" # Path to the error page, e.g., /404.html, /500.html

  services:
    error-page-service: # Definition of the service that serves error pages
      loadBalancer:
        servers:
          - url: "http://static-error-server" # URL to your error page server container
                                            # (e.g., an Nginx container named 'static-error-server')

Then, in your docker-compose.yml, you'd have a service like static-error-server serving HTML files from a volume.

Forward Authentication ForwardAuth

Forward Authentication is a powerful mechanism for delegating authentication decisions to an external, specialized authentication service. This is commonly used to integrate Traefik with Single Sign-On (SSO) systems, OAuth2/OIDC providers, or custom-built authentication microservices.

How ForwardAuth Works:

1. A request arrives at a Traefik router that has a ForwardAuth middleware configured.
2. Traefik temporarily puts the original request on hold.
3. Traefik sends a new request (often just the headers of the original request, or a full copy depending on configuration) to the specified external authentication address.
4. The external auth service performs its checks (e.g., validates a session cookie, checks an API token, redirects to a login page if no session).
5. The auth service responds to Traefik:
   • Success (e.g., HTTP 2xx status code): Traefik considers the user authenticated. The original request is then forwarded to the backend application. The auth service can also send back specific headers (e.g., X-Forwarded-User, X-Authenticated-Groups) that Traefik can then add to the request going to the backend.
   • Failure (e.g., HTTP 401 Unauthorized, 403 Forbidden): Traefik denies access. It can return the error from the auth service or redirect to a login page URL provided by the auth service (if the authResponseHeaders are configured to pass such a header).
6. The ForwardAuth middleware can be configured to:
   • trustForwardHeader: Trust X-Forwarded-* headers.
   • authResponseHeaders: A list of headers to copy from the auth server's response back to the client (if auth fails) or to the backend application (if auth succeeds).
   • authRequestHeaders: A list of headers from the original client request to not forward to the auth server.

This allows you to centralize authentication logic outside of your individual applications and Traefik itself. Popular self-hosted auth providers that work well with Traefik's ForwardAuth include Authelia, Authentik, and Keycloak.

Workshop Implementing ForwardAuth with a Simple Auth Service and Custom Error Pages

This workshop is in two parts:

1. Part 1: Custom Error Pages: Set up custom 404 and 500 error pages.
2. Part 2: Forward Authentication: Secure the whoami service using a mock ForwardAuth service.

Prerequisites:

• Your Traefik setup from previous workshops (using File Provider for dynamic configuration is recommended here for clarity).
• DNS and HTTPS configured for your services.

Part 1: Custom Error Pages

1. Create Custom Error HTML Files:

   • In your project directory (e.g., my-traefik-setup), create a subdirectory for your custom error pages:

     mkdir custom_error_pages
     

   • Inside custom_error_pages, create at least two files:
     • 404.html:

       <!DOCTYPE html>
       <html lang="en">
       <head>
           <meta charset="UTF-8">
           <title>404 Not Found</title>
           <style>body { font-family: sans-serif; text-align: center; padding-top: 50px; background-color: #f0f0f0; color: #333; } h1 { font-size: 3em; } p { font-size: 1.2em; }</style>
       </head>
       <body>
           <h1>Oops! 404 - Page Not Found</h1>
           <p>Sorry, the page you are looking for does not exist.</p>
           <p><a href="/">Go to Homepage</a></p>
       </body>
       </html>
       

     • 500.html:

       <!DOCTYPE html>
       <html lang="en">
       <head>
           <meta charset="UTF-8">
           <title>500 Internal Server Error</title>
           <style>body { font-family: sans-serif; text-align: center; padding-top: 50px; background-color: #f0f0f0; color: #333; } h1 { font-size: 3em; } p { font-size: 1.2em; }</style>
       </head>
       <body>
           <h1>Oops! 500 - Internal Server Error</h1>
           <p>Sorry, something went wrong on our end. Please try again later.</p>
       </body>
       </html>
       

       You can create other files like 401.html, 403.html, 503.html as needed.

2. Add an Nginx Service to Serve Error Pages (in docker-compose.yml):

   # ... (inside services:) ...
     errorpages_server:
       image: "nginx:alpine"
       container_name: "error_pages_static_server"
       volumes:
         - ./custom_error_pages:/usr/share/nginx/html:ro # Mount your HTML files
       networks:
         - traefik_public # So Traefik can reach it
       # No ports exposed to host, Traefik accesses it internally
   

3. Define Error Middleware and Service in Dynamic Configuration:

   • Open your dynamic configuration file (e.g., traefik_dynamic_conf/whoami-config.yml or create a new traefik_dynamic_conf/common-middlewares.yml). Add:

   # In traefik_dynamic_conf/common-middlewares.yml (or similar)
   http:
     middlewares:
       app-custom-errors: # Name of the error middleware
         errors:
           status:
             - "400-599" # Handle this range of status codes
           service: "errorpages-service-def@file" # Points to the service defined below
           query: "/{status}.html" # Tells error server to serve /404.html, /500.html etc.
   
     services:
       errorpages-service-def: # Name of the Traefik service for error pages
         loadBalancer:
           servers:
             # 'errorpages_server' is the container_name from docker-compose.yml
             - url: "http://errorpages_server" # Nginx listens on port 80 by default
   

4. Apply the Error Middleware to a Router:

   • For example, let's apply it to the whoami-router-from-file (if you're using the file provider setup for whoami). Modify its middleware chain in traefik_dynamic_conf/whoami-config.yml:

   # In traefik_dynamic_conf/whoami-config.yml
   # ... (router definition for whoami-router-from-file) ...
         middlewares:
           - "app-custom-errors@file" # Add this FIRST, so it can catch errors from subsequent processes/backend
           - "whoami-sec-headers-file@file"
           - "whoami-req-limit-file@file"
   # ...
   

   Note on order: Placing the error middleware early in the chain allows it to catch errors generated by Traefik itself (like a 404 if no route matches deeper rules within a complex router, though this specific 404 is often global) or by the backend service.

5. Deploy and Test:

   • Run docker compose up -d. Traefik will pick up the new Nginx container and the new middleware/service definitions.
   • Test 404: Try to access a non-existent path on your whoami service, e.g., https://whoami-file.yourdomain.com/this-page-does-not-exist. You should see your custom 404.html page.
   • Test 500 (Harder to simulate on whoami): If whoami were to crash and return a 500, you'd see 500.html. You can test by temporarily pointing a router to a non-existent backend service to trigger a 502/503, which should also be caught if your status range includes them.

Part 2: Forward Authentication with a Mock Auth Service

We'll create a very simple "mock" auth service using Nginx. This service will simply return HTTP 200 (OK) for a specific path /auth-ok (simulating successful auth) and 401 (Unauthorized) for /auth-fail (simulating failed auth).

1. Create Nginx Configuration for Mock Auth Service:

   • Create a file named nginx_mock_auth.conf in your project root:

     # nginx_mock_auth.conf
     server {
         listen 80;
         server_name mock_auth_server;
     
         location = /auth-ok {
             return 200; # Simulate successful authentication
             add_header X-Authenticated-User "mockuser" always; # Pass a user header
         }
     
         location = /auth-fail {
             return 401; # Simulate failed authentication
             add_header WWW-Authenticate "Mock realm=\"Access denied\"" always;
         }
     
         location / {
             return 404; # Default deny
         }
     }
     

2. Add Mock Auth Service to docker-compose.yml:

   # ... (inside services:) ...
     mock_auth_service:
       image: "nginx:alpine"
       container_name: "mock_auth_nginx"
       volumes:
         - ./nginx_mock_auth.conf:/etc/nginx/conf.d/default.conf:ro
       networks:
         - traefik_public
       # No ports exposed to host
   

3. Define ForwardAuth Middleware in Dynamic Configuration:

   • Add to traefik_dynamic_conf/common-middlewares.yml (or your main dynamic config file):

   # In traefik_dynamic_conf/common-middlewares.yml
   # ... (http: middlewares: section) ...
       whoami-forward-auth:
         forwardAuth:
           # 'mock_auth_service' is the Docker service name, Nginx listens on port 80
           # We'll point to the /auth-ok path to simulate success for now
           address: "http://mock_auth_service/auth-ok"
           trustForwardHeader: true # Standard practice
           authResponseHeaders: # Headers from auth server to pass to backend if auth is successful
             - "X-Authenticated-User"
   

   To test failure, you could change the address to http://mock_auth_service/auth-fail.

4. Apply ForwardAuth Middleware to whoami Router:

   • Modify the middlewares chain for whoami-router-from-file in traefik_dynamic_conf/whoami-config.yml. ForwardAuth should typically run early, often before other processing middlewares but potentially after error pages if you want custom auth error pages. For this example, let's put it after error pages but before headers/rate limits.

   # In traefik_dynamic_conf/whoami-config.yml
   # ... (router definition for whoami-router-from-file) ...
         middlewares:
           - "app-custom-errors@file"
           - "whoami-forward-auth@file" # Add ForwardAuth
           - "whoami-sec-headers-file@file"
           - "whoami-req-limit-file@file"
   # ...
   

5. Deploy and Test ForwardAuth:

   • Run docker compose up -d.
   • Test Successful Auth:
     • Access https://whoami-file.yourdomain.com. Since the ForwardAuth middleware is configured with /auth-ok, which returns 200, you should be able to access the whoami page.
     • Inspect the whoami output. You should see the X-Authenticated-User: mockuser header, which was added by our mock auth service and passed through by Traefik.
   • Test Failed Auth (by reconfiguring):
     1. Modify traefik_dynamic_conf/common-middlewares.yml. Change the address in whoami-forward-auth to http://mock_auth_service/auth-fail.

        # ...
        whoami-forward-auth:
          forwardAuth:
            address: "http://mock_auth_service/auth-fail" # Changed to simulate failure
        # ...
        

     2. Save the file. Traefik (with watch=true for file provider) should reload the configuration automatically.
     3. Try accessing https://whoami-file.yourdomain.com again. This time, you should get a 401 Unauthorized error (either Traefik's default 401 page or, if your custom error page middleware is configured for 401s and you have 401.html, your custom one). You will not reach the whoami service.

This workshop illustrates how to chain middlewares for custom error handling and how ForwardAuth can delegate authentication decisions. Real ForwardAuth setups with services like Authelia or Keycloak are more complex but follow the same principles, offering robust, centralized authentication for your self-hosted applications.

8. Traefik Pilot Metrics and Logging

Observability is key to maintaining a healthy and performant self-hosted environment. Traefik provides several mechanisms for insight into its operations: metrics for monitoring performance and request patterns, and detailed logging for troubleshooting and auditing. (Note: Traefik Pilot was a SaaS offering from Traefik Labs; its features and availability may change. We will focus on the open standards-based metrics and logging capabilities of Traefik itself.)

Traefik Pilot

Traefik Pilot was designed as a service by Traefik Labs to provide enhanced observability features, such as centralized dashboards, alerts, and plugin management for Traefik instances. Connecting Traefik to Pilot typically involved registering your instance and obtaining a token.

If you are interested in features that were part of Traefik Pilot, it's best to check the current offerings on the official Traefik Labs website, as cloud services evolve. For robust self-hosted observability, integrating with Prometheus and Grafana is a very common and powerful approach.

Enabling Metrics Prometheus

Traefik can expose detailed metrics in a format compatible with Prometheus, a popular open-source monitoring and alerting toolkit. Prometheus scrapes (collects) these metrics periodically, stores them in a time-series database, and allows you to query them using its query language (PromQL). Grafana is then often used to visualize these metrics in dashboards.

Traefik Metrics Include:

• Request counts, latencies (total and per quantile).
• Response counts by status code.
• Open connections.
• Configuration load successes/failures.
• Entrypoint, router, and service level metrics.
• TLS handshake information.

Configuration:
To enable Prometheus metrics in Traefik's static configuration:

# Command-line arguments:
--metrics.prometheus=true

# Optional: Expose metrics on a dedicated entrypoint (recommended for security/isolation)
--metrics.prometheus.entryPoint=metrics # Default is 'traefik' if not specified
--entrypoints.metrics.address=:8082     # Define a new entrypoint for metrics on a different port

# Optional: Add router/service labels to metrics for finer-grained filtering/aggregation
--metrics.prometheus.addrouterslabels=true
--metrics.prometheus.addserviceslabels=true

If you expose metrics on a dedicated entrypoint (e.g., metrics on port 8082), this port is typically not exposed to the internet but only accessible internally by Prometheus. Alternatively, you can expose the /metrics path via a regular Traefik router, secured with HTTPS and authentication, using the special prometheus@internal service:

# In labels for Traefik itself, or in a file provider config for Traefik's internal services
# Assuming 'traefik.yourdomain.com' is your dashboard host
- "traefik.http.routers.prometheus-metrics.rule=Host(`traefik.yourdomain.com`) && Path(`/metrics`)"
- "traefik.http.routers.prometheus-metrics.entrypoints=websecure"
- "traefik.http.routers.prometheus-metrics.service=prometheus@internal" # Special internal service
- "traefik.http.routers.prometheus-metrics.tls.certresolver=myresolver"
# Add auth middleware, e.g., the same one used for the dashboard
- "traefik.http.routers.prometheus-metrics.middlewares=dashboard-auth@docker"

This makes metrics available at https://traefik.yourdomain.com/metrics.

Advanced Logging and Tracing

Logging:
Traefik provides comprehensive logging capabilities:

• Traefik Logs: General operational logs about Traefik itself (startup, configuration reloads, errors).
  • Configured via static configuration:
    • --log.level=<DEBUG|INFO|WARN|ERROR> (Default is ERROR)
    • --log.filepath=<path_to_log_file> (Default is stdout)
    • --log.format=<common|json> (Default is common)
• Access Logs: Detailed logs for each HTTP request processed by Traefik. Highly valuable for traffic analysis, debugging, and security auditing.
  • Configured via static configuration:
    • --accesslog=true
    • --accesslog.filepath=<path_to_access_log_file> (Default is stdout)
    • --accesslog.format=<common|json> (Default is common)
    • --accesslog.fields.names: Customize fields included (e.g., ClientHost, RequestMethod, RequestPath, DownstreamStatus, Duration).
    • --accesslog.bufferingSize=<number_of_lines>: Buffer logs before writing.
    • --accesslog.filters.statuscodes="400-499,500-599": Only log requests with specific status codes. JSON format is often preferred for easier parsing by log management systems (e.g., ELK Stack, Loki).

Distributed Tracing:
For microservice architectures, understanding the full lifecycle of a request as it passes through multiple services can be complex. Distributed tracing helps visualize these flows. Traefik supports several tracing backends:

• Jaeger
• Zipkin
• Datadog
• OpenTracing (via various implementations)

Configuration involves enabling a tracing backend in Traefik's static configuration and providing connection details to the tracing collector. Traefik then injects trace context headers into requests and reports spans to the configured backend.

Example for Jaeger:

# Command-line arguments:
--tracing.jaeger=true
--tracing.jaeger.samplingServerURL=http://jaeger-agent:5778/sampling
--tracing.jaeger.localAgentHostPort=jaeger-agent:6831
# ... other Jaeger options

Workshop Setting up Prometheus and Grafana for Traefik Metrics

Goal:
Configure Traefik to expose Prometheus metrics, deploy Prometheus to scrape these metrics, and deploy Grafana to visualize them using a pre-built dashboard.

Prerequisites:

• Your existing Traefik setup (Docker Compose).
• Ensure your Traefik dashboard is accessible, as we might expose metrics similarly or on a dedicated internal port.

Steps:

1. Enable Prometheus Metrics in Traefik: Modify the command: section of your traefik service in docker-compose.yml. We'll expose metrics on a dedicated internal port 8082 for simplicity, making it directly accessible to Prometheus on the same Docker network.

   # ... (inside services: traefik: command:) ...
         # --- Metrics Configuration ---
         - "--metrics.prometheus=true"
         # Define a dedicated entrypoint for metrics (internal access)
         - "--entrypoints.metrics.address=:8082"
         # Tell Prometheus metrics system to use this 'metrics' entrypoint
         - "--metrics.prometheus.entryPoint=metrics"
         # Optional: Add labels for more detailed metrics
         - "--metrics.prometheus.addrouterslabels=true"
         - "--metrics.prometheus.addserviceslabels=true"
   

   No new port mapping to the host is needed for 8082 because Prometheus will run in another container on the same Docker network (traefik_public) and can access traefik:8082 directly.

2. Create Prometheus Configuration File (prometheus.yml): In your project root (e.g., my-traefik-setup), create a file named prometheus.yml:

   # prometheus.yml
   global:
     scrape_interval: 15s # How frequently to scrape targets
   
   scrape_configs:
     - job_name: 'traefik'
       static_configs:
         # 'traefik' is the service name of our Traefik container in docker-compose.yml.
         # Port 8082 is where Traefik's 'metrics' entrypoint is listening.
         - targets: ['traefik:8082']
   

3. Add Prometheus Service to docker-compose.yml:

   # ... (inside services:) ...
     prometheus:
       image: "prom/prometheus:v2.51.0" # Use a specific, recent stable version
       container_name: "prometheus_server"
       volumes:
         - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro # Mount config file
         - prometheus_data:/prometheus # Persistent storage for Prometheus data
       ports:
         - "9090:9090" # Expose Prometheus UI on host port 9090
       networks:
         - traefik_public # So it can reach traefik:8082
       command:
         - '--config.file=/etc/prometheus/prometheus.yml'
         - '--storage.tsdb.path=/prometheus'
         - '--web.console.libraries=/usr/share/prometheus/console_libraries'
         - '--web.console.templates=/usr/share/prometheus/consoles'
   

   Also, define the prometheus_data volume at the top-level volumes: section in your docker-compose.yml:

   # ... (top-level volumes:)
   volumes:
     letsencrypt: # from previous workshops
     consul_data: # if using Consul
     prometheus_data: # For Prometheus
     grafana_data: # For Grafana (we'll add this next)
   

4. Add Grafana Service to docker-compose.yml:

   # ... (inside services:) ...
     grafana:
       image: "grafana/grafana:10.4.2" # Use a specific, recent stable version
       container_name: "grafana_server"
       ports:
         - "3000:3000" # Expose Grafana UI on host port 3000
       volumes:
         - grafana_data:/var/lib/grafana # Persistent storage for Grafana data and dashboards
       networks:
         - traefik_public # So it can reach Prometheus
       environment:
         # You can set other GF_ environment variables for Grafana config
         - GF_SECURITY_ADMIN_PASSWORD=YourSecureGrafanaPassword # !!! CHANGE THIS !!!
         # Optional: anonymous access for viewing dashboards
         # - GF_AUTH_ANONYMOUS_ENABLED=true
         # - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer
   

   Remember to change YourSecureGrafanaPassword!

5. Update and Run Docker Compose:

   • Save your docker-compose.yml file.
   • Run:

     docker compose up -d
     

     This will create/recreate Traefik (with new metrics commands), and start Prometheus and Grafana containers.

6. Configure Grafana Data Source and Import Dashboard:

   • Access Grafana: Open your browser to http://<your-server-ip>:3000.
   • Log in with username admin and the password you set (YourSecureGrafanaPassword). You might be prompted to change the password on first login.
   • Add Prometheus Data Source:
     1. In Grafana, go to "Connections" (or "Configuration" -> "Data Sources" in older versions).
     2. Click "Add new data source."
     3. Select "Prometheus."
     4. Set the Name to something like "MyPrometheus".
     5. For Prometheus server URL, enter http://prometheus:9090.
        • prometheus is the Docker service name of your Prometheus container. Grafana, being on the same traefik_public network, can resolve this.
     6. Leave other settings as default for now.
     7. Click "Save & test." You should see a message like "Data source is working."
   • Import a Traefik Dashboard:
     1. Go to "Dashboards" (usually a four-square icon on the left).
     2. Click "New" and then "Import" (or "+", then "Import dashboard" in older versions).
     3. There are many community Traefik dashboards available on Grafana.com/grafana/dashboards/. A popular one for Traefik v2+ is ID 12Traefik (or search for "Traefik 2"). Often ID 4475 (older but comprehensive) or 12460 are mentioned. Let's try 12460 (Traefik 2 by JRECODE) or 16951 (Traefik v2 K показатели). You might need to try a few to find one that works well with your Traefik version and metrics exposure.
        • For this workshop, let's try dashboard ID 12460. In the "Import via grafana.com" box, enter 12460 and click "Load."
     4. On the next screen, you might need to select your Prometheus data source ("MyPrometheus").
     5. Click "Import."
   • View the Dashboard: You should now see a dashboard populated with metrics from your Traefik instance! It might take a few minutes for data to start appearing and graphs to populate. Explore the different panels showing request rates, response codes, latencies, etc.

7. Verify Prometheus (Optional):

   • Access Prometheus UI at http://<your-server-ip>:9090.
   • Go to "Status" -> "Targets." You should see your traefik job listed with a state of "UP."
   • In the "Expression" bar, you can try querying Traefik metrics, e.g., traefik_entrypoint_requests_total.

You now have a powerful monitoring setup for Traefik using Prometheus and Grafana. This allows you to observe traffic patterns, diagnose issues, and understand the performance of your reverse proxy and the services behind it. Remember to secure your Grafana instance appropriately, especially if it's exposed externally.

Conclusion

Throughout this guide, we've journeyed from the fundamental concepts of reverse proxies to advanced configurations and observability for Traefik. You've learned how to deploy Traefik with Docker, expose services, secure them with HTTPS, leverage powerful middlewares, manage configurations with the File Provider, understand HA considerations, and monitor Traefik's performance.

Recap of Key Learnings

• Basic Understanding:

  • We defined what a reverse proxy is and why it's essential for self-hosting, highlighting Traefik's strengths in dynamic, containerized environments.
  • You successfully deployed Traefik using Docker Compose, explored its dashboard, and exposed your first simple web service (whoami) using Docker labels for configuration.

• Intermediate Skills:

  • HTTPS and Let's Encrypt: You learned to secure your services and the Traefik dashboard itself with automatic SSL/TLS certificates from Let's Encrypt, including setting up HTTP-to-HTTPS redirection.
  • Traefik Middlewares: We explored the concept of middlewares and implemented practical examples like security headers and rate limiting, understanding how they can modify requests and enhance application behavior without code changes.
  • File Provider: You discovered the benefits of using the File Provider for managing Traefik's dynamic configuration, offering better organization and flexibility for complex setups or non-Dockerized services.

• Advanced Capabilities:

  • High Availability (HA): We discussed strategies for making Traefik highly available, focusing on the critical aspect of shared ACME certificate storage using a Key-Value store like Consul.
  • Advanced Middlewares and Customization: You saw how to chain middlewares for more sophisticated request processing and learned to implement custom error pages for a better user experience. We also introduced the concept of Forward Authentication for integrating with external auth systems.
  • Observability (Metrics and Logging): You set up Prometheus and Grafana to collect and visualize detailed metrics from Traefik, providing crucial insights into its performance and traffic patterns. We also touched upon advanced logging configurations.

Best Practices for Self-Hosting Traefik

As you continue your self-hosting journey with Traefik, keep these best practices in mind:

1. Keep Traefik Updated: Regularly check for new Traefik releases and update your instances to benefit from the latest features, performance improvements, and security patches.
2. Secure Everything:
   • Always secure your Traefik dashboard with strong authentication and HTTPS.
   • Use HTTPS (via Let's Encrypt) for all your exposed services.
   • Implement appropriate security headers using middlewares.
   • Restrict access where necessary (e.g., IP allow-listing for sensitive internal tools).
3. Use exposedByDefault=false: For the Docker provider, always set providers.docker.exposedbydefault=false in your static configuration and explicitly enable Traefik for containers using the traefik.enable=true label. This prevents accidental exposure of services.
4. Leverage Let's Encrypt Wisely:
   • Use the Let's Encrypt staging server for testing to avoid rate limits.
   • Ensure your ACME certificate storage (e.g., acme.json or KV store) is persistent and backed up.
5. Monitor Your Traefik Instances: Utilize Traefik's metrics with Prometheus and Grafana (or other monitoring tools) to keep an eye on request volumes, error rates, and performance. Set up alerts for critical issues.
6. Centralize and Version Control Configurations: For anything beyond a few simple services, consider using the File Provider for dynamic configuration. Store your docker-compose.yml files and dynamic configuration files (e.g., in traefik_dynamic_conf) in a Git repository for version control and easier management.
7. Understand Traefik's Logs: Configure appropriate log levels (INFO for general operation, DEBUG for troubleshooting) and access logs. Consider sending logs to a centralized log management system for easier analysis.
8. Network Segmentation: Use dedicated Docker networks (like our traefik_public) for Traefik and the services it proxies. Only connect services to this network that need to be exposed.
9. Resource Limits: For containerized deployments, consider setting resource limits (CPU, memory) for your Traefik containers, especially in resource-constrained environments.
10. Read the Docs: The official Traefik documentation is an invaluable resource. Refer to it often for detailed information on specific features, middlewares, and providers.

Further Learning Resources

The world of reverse proxying and service networking is vast. To continue deepening your knowledge:

• Official Traefik Documentation: doc.traefik.io/traefik/ - This should always be your primary reference.
• Traefik Community Forum: community.traefik.io - A great place to ask questions, share solutions, and learn from other users.
• Awesome Traefik: Check GitHub for "awesome-traefik" lists, which often compile useful resources, articles, and tools related to Traefik.
• Blogs and Tutorials: Many self-hosting and DevOps blogs feature detailed Traefik tutorials and use cases (e.g., search for "Traefik tutorial," "Traefik Docker self-hosting").
• Prometheus and Grafana Documentation: If you delve deeper into monitoring, the official docs for these tools are essential.

By applying the knowledge gained here and continuing to explore, you'll be well on your way to mastering Traefik and building a robust, secure, and efficient self-hosted infrastructure. Happy self-hosting!