Skip to content

Podman Quadlets & Systemd: Declarative Container Orchestration

Created: 2026-11-10 Updated: 2026-11-10

Management Summary: Traditional container deployments rely heavily on shell scripts or docker-compose files operating outside the standard Linux operating system lifecycle. This frequently introduces unmanaged states, boot race conditions, and broken dependency chains during host reboots, process crashes, and patch cycles. Podman Quadlets fundamentally transform single-host container orchestration: rather than brittle wrappers, a native systemd generator dynamically compiles declarative .container, .volume, .network, and .pod specifications into native systemd units during boot or daemon reload. This enables deterministic dependency resolution, system-wide resource supervision, standardized journald telemetry, and zero-touch auto-updates with automated health-check rollback. This guide covers the generator architecture, rootless deployment models, multi-tier stacks replacing compose, auto-updates, and operational runbooks.

  • Quadlet Architecture

    How the native systemd generator parses declarative .container files and compiles them into transient service units at boot.

  • File Types & Storage Paths

    Architectural separation between root units (/etc/containers/systemd/) and unprivileged rootless units (~/.config/containers/systemd/).

  • Multi-Container Stacks

    Connecting complex stacks declaratively using .network and .volume units—the modern replacement for docker-compose.yml.

  • Podman Auto-Update & Rollback

    Zero-touch updates using registry polling, systemd timers, and automated rollbacks upon failing health checks.

  • Secrets & Access Controls

    Secure secret injection via Podman Secrets, user namespace isolation (subuid), and SELinux volume relabeling flags (:Z,U).

  • Operations & Troubleshooting

    Generator dry-run diagnostics, inspecting compiled units under /run/, and isolating failed service startup sequences.

1. Architectural Paradigm: Why Compose Fails in OS Operations

Across legacy container workflows, docker-compose (or podman-compose) was long regarded as the default toolchain for defining multi-container services. In production host environments, however, this model introduces fundamental architectural liabilities:

  1. Absence of Native OS Supervision: Compose files operate as standalone client utilities. When a server reboots, external wrapper scripts or opaque container restart flags (restart: always) must intervene. If the container runtime or host process terminates unexpectedly, containers are frequently left orphaned without supervisor awareness.
  2. Missing Integration with Linux Subsystems: cgroups v2 resource quotas, kernel audit telemetry, and strict host ordering dependencies (e.g., waiting for persistent mounts under /mnt/data or network stack readiness) cannot be natively harmonized through Compose.
  3. The Dead End of podman generate systemd: Earlier Podman releases introduced podman generate systemd. However, this utility generated bloated, static service files containing hardcoded CLI flags and container hashes. Every container update necessitated manual regeneration and deletion of outdated units. Upstream has formally deprecated this approach.

The Solution: The Native Quadlet Generator

With Quadlet (introduced in Podman 4.4 and steadily expanded), container lifecycle management was anchored directly inside the operating system. Quadlet is not an additional background daemon; it is a systemd generator (podman-system-generator) embedded into systemd's native boot and reload execution cycles:

%%{init: {'flowchart': {'wrappingWidth': 350}}}%%
flowchart TB
    Quadlet_File["Declarative Quadlet File<br/>(~/.config/containers/systemd/web.container)"] ==> Systemd_Reload["systemctl --user daemon-reload<br/>(or Host System Boot)"]
    Systemd_Reload --> Generator["podman-system-generator<br/>(Parses OCI Spec)"]
    Generator --> Transiente_Unit["Transient systemd Unit<br/>(/run/user/1000/systemd/generator/web.service)"]
    Transiente_Unit --> Systemd_Engine["systemd Service Manager<br/>(Dependencies, cgroups, Journal)"]
    Systemd_Engine ==> Conmon["conmon Runtime Monitor<br/>(Unprivileged Child Process)"]
    Conmon --> Container["OCI Rootless Container<br/>(Isolated Application Process)"]

The decisive advantage: operators maintain concise, readable declarative configuration files. Systemd compiles these definitions on-the-fly into production-ready systemd services equipped with robust startup, shutdown, healthcheck, and logging mechanisms.

2. File Types, Storage Locations & Rootless Prerequisites

Quadlet supports distinct declarative file extensions, each mapped to a specific architectural layer of container infrastructure.

The 6 Core Quadlet File Types

File Extension Target Infrastructure Generated systemd Unit
.container Defines an individual OCI container workload <name>.service
.volume Provisions and manages a persistent Podman named volume <name>-volume.service
.network Declares an isolated bridge or internal network <name>-network.service
.pod Bundles multiple containers inside a shared network/IPC namespace <name>-pod.service
.kube Executes Kubernetes-compliant pod manifests via podman kube play <name>.service
.image Manages pre-fetching and caching of OCI images <name>-image.service

Storage Locations: Root vs. Rootless

Quadlets execute within two separate scopes:

  1. System-Wide (Root):
  2. Location: /etc/containers/systemd/
  3. Executed by root under the system instance (systemctl).
  4. Reserved for low-level infrastructure requiring direct host privileges.
  5. Rootless (Unprivileged – Recommended Enterprise Standard):
  6. Location: ~/.config/containers/systemd/ (or /etc/containers/systemd/users/ for fleet-wide user drop-ins).
  7. Executed entirely within the userspace of a dedicated service account via systemctl --user.
  8. Security Posture: Even in the event of a total container breakout, the compromised process possesses zero administrative permissions on the underlying host!

Mandatory Prerequisite: Enable User Lingering

By default, Linux terminates all background processes belonging to an unprivileged user session as soon as the last SSH connection closes. To ensure continuous background operation for service accounts, lingering must be explicitly enabled:

# Enable persistent systemd instance for service account (e.g., "deploy")
sudo loginctl enable-linger deploy

# Verify lingering status (must return "Linger=yes")
loginctl show-user deploy --property=Linger

3. Anatomy of a .container File in Detail

A Quadlet specification combines standard systemd sections ([Unit], [Service], [Install]) with the Quadlet-specific [Container] section.

Turnkey Example: Hardened Nginx Microservice

Create the configuration directory and file at ~/.config/containers/systemd/webserver.container:

# /home/deploy/.config/containers/systemd/webserver.container
# ==============================================================================
# Enterprise Rootless Nginx Container Unit via Podman Quadlet
# ==============================================================================

[Unit]
Description=Production Web Server (Rootless OCI Nginx)
Documentation=https://nejat-hakan.de/Linux/linux_containers__systemd_quadlets/
After=network-online.target
Wants=network-online.target

[Container]
# OCI Image Source
Image=docker.io/library/nginx:alpine

# Deterministic container identifier for logging and CLI queries
ContainerName=production-webserver

# Port mapping: host loopback port 8080 to unprivileged container port 80
PublishPort=127.0.0.1:8080:80

# Persistent mounts: :Z applies SELinux context, :U remaps subUID ownership
Volume=%h/web_content:/usr/share/nginx/html:ro,Z
Volume=%h/web_logs:/var/log/nginx:Z,U

# Environment configuration
Environment=TZ=UTC

# Enable automatic registry updates
AutoUpdate=registry

# Hardening directives: drop kernel capabilities, enforce read-only rootfs
SecurityLabelDisable=false
NoNewPrivileges=true
ReadOnly=true
DropCapability=ALL
AddCapability=CHOWN SETGID SETUID NET_BIND_SERVICE

[Service]
# Automatic crash recovery
Restart=always
TimeoutStartSec=300

[Install]
# Automatic initialization on user session startup
WantedBy=default.target

Essential [Container] Directives

  • Image=: Fully qualified container image reference (always include the registry domain, e.g., docker.io/library/nginx:alpine or quay.io/..., avoiding ambiguous short names).
  • PublishPort=: Syntax [Host-IP:]Host-Port:Container-Port. On rootless services, bind containers to 127.0.0.1 and terminate TLS/HTTP traffic through an edge reverse proxy (such as Nginx or Caddy).
  • Volume=: Supports dynamic systemd specifiers like %h (user home directory). The flags :Z (SELinux container boundary) and :U (ownership adjustment to match container namespace subUIDs) are mandatory for error-free storage binding.
  • AutoUpdate=registry: Flags this container for scheduled evaluation by the systemd auto-update timer against remote registry digests.
  • NoNewPrivileges=true & DropCapability=ALL: Enforces in-kernel privilege mitigation, blocking setuid escalation pathways.

4. Multi-Container Stacks & Networks: Replacing docker-compose

Enterprise applications rarely run in isolation; standard workloads require a frontend web server, an application runtime, a database backend, and private interconnects.

Quadlet fulfills this requirement natively through declarative orchestration across .network, .volume, and multiple .container files.

1. Isolated Network: app-net.network

Create ~/.config/containers/systemd/app-net.network:

# /home/deploy/.config/containers/systemd/app-net.network
# ==============================================================================
# Isolated Bridge Network for Production Application Stack
# ==============================================================================

[Network]
# Internal network identifier
NetworkName=production-app-net

# Custom subnet configuration avoiding VPN conflicts
Subnet=10.89.10.0/24
Gateway=10.89.10.1

# Restrict external outbound internet routing if required
Internal=false

2. Persistent Storage Volume: db-data.volume

Create ~/.config/containers/systemd/db-data.volume:

# /home/deploy/.config/containers/systemd/db-data.volume
# ==============================================================================
# Dedicated Persistent Volume for PostgreSQL Storage
# ==============================================================================

[Volume]
VolumeName=production-postgres-data

3. Database Backend: postgres.container

Create ~/.config/containers/systemd/postgres.container:

# /home/deploy/.config/containers/systemd/postgres.container
# ==============================================================================
# PostgreSQL 16 Enterprise Database Unit
# ==============================================================================

[Unit]
Description=PostgreSQL Database Service
After=app-net-network.service
Requires=app-net-network.service

[Container]
Image=docker.io/library/postgres:16-alpine
ContainerName=app-database

# Connect to declarative Quadlet network
Network=app-net.network

# Attach persistent declarative Quadlet volume
Volume=db-data.volume:/var/lib/postgresql/data:Z

# Secure configuration
Environment=POSTGRES_DB=app_db
Environment=POSTGRES_USER=app_user
EnvironmentFile=%h/.config/containers/postgres.env

# No host port publication required: accessed internally via DNS "app-database:5432"

[Service]
Restart=always
TimeoutStartSec=180

[Install]
WantedBy=default.target

4. Web Application Frontend: app.container

Create ~/.config/containers/systemd/app.container:

# /home/deploy/.config/containers/systemd/app.container
# ==============================================================================
# Web Application Service Dependent on Database Backend
# ==============================================================================

[Unit]
Description=Application Backend Runtime
# Ensure database initializes before application launches
After=postgres.service
Requires=postgres.service

[Container]
Image=quay.io/enterprise/web-app:latest
ContainerName=app-backend

# Attach to identical private bridge network
Network=app-net.network

# Inter-container DNS resolution using container name "app-database"
Environment=DB_HOST=app-database
Environment=DB_PORT=5432
Environment=DB_NAME=app_db

# Expose service solely on local loopback interface
PublishPort=127.0.0.1:3000:3000

AutoUpdate=registry

[Service]
Restart=always

[Install]
WantedBy=default.target

Dependency Governance via systemd

Unlike docker-compose, systemd monitors real OS process states: * Requires=postgres.service: If the database crashes or stops, systemd cleanly shuts down or restarts app.service in tandem. * After=postgres.service: Enforces deterministic startup ordering during system initialization.

5. Zero-Touch Auto-Updates & Health-Check Rollback

One of Quadlet's most formidable capabilities is native, autonomous image updating with health-check validation.

Mechanics of Podman Auto-Updates

  1. The .container unit defines AutoUpdate=registry.
  2. Podman checks the OCI label io.containers.autoupdate=registry or the Quadlet directive.
  3. The built-in systemd timer podman-auto-update.timer is activated.
  4. Once daily (typically at midnight), the timer invokes podman-auto-update.service, which compares local SHA256 image digests against remote registry tags.
  5. If an update exists, the image is pulled, and the unit is automatically restarted.
%%{init: {'flowchart': {'wrappingWidth': 350}}}%%
flowchart TB
    Timer["podman-auto-update.timer<br/>(Daily systemd Trigger)"] ==> AutoUpdate_Service["podman-auto-update.service<br/>(Compares Remote Digests)"]
    AutoUpdate_Service --> Pull{"New Digest Available?"}
    Pull -->|"No"| Done["No Action Required"]
    Pull -->|"Yes"| Download["Pull Updated OCI Image<br/>(podman pull)"]
    Download --> Restart["systemctl restart <unit><br/>(New Container Launches)"]
    Restart --> Healthcheck{"Health Check Successful?<br/>(Executes HealthCmd)"}
    Healthcheck -->|"Pass"| Stable["Update Finalized<br/>(Active Production State)"]
    Healthcheck -->|"Fail"| Rollback["Automated Rollback Triggered!<br/>(Prior Image Restored)"]

Enabling the Auto-Update Timer

# Enable and start the automatic update timer for the service user
systemctl --user enable --now podman-auto-update.timer

# Inspect scheduled execution intervals
systemctl --user list-timers podman-auto-update.timer

Defining Health Checks for Safe Rollbacks

Without a health check, Podman cannot assess whether an updated container is operating correctly. Health checks guarantee production continuity:

[Container]
Image=docker.io/library/nginx:alpine
AutoUpdate=registry

# Health check: probe HTTP 200 on port 80 every 30s
HealthCmd=curl -f http://localhost:80/ || exit 1
HealthInterval=30s
HealthRetries=3
HealthTimeout=5s
HealthStartupInterval=5s

If the updated container fails its health check three times consecutively, Podman terminates the bad container and automatically rolls back to the previously working image!

6. Enterprise Operations, Troubleshooting & Best Practices

1. Activating and Compiling Quadlet Units

Modifications to .container, .network, or .volume files require a systemd generator re-run:

# 1. Trigger systemd generator execution
systemctl --user daemon-reload

# 2. Start and enable the compiled service
systemctl --user enable --now webserver.service

# 3. Verify runtime status
systemctl --user status webserver.service

2. Dry-Run Diagnostics: Pre-Flight Syntax Validation

If a unit fails to appear following daemon-reload, a syntax error in the Quadlet file is the probable cause. Execute the generator in dry-run mode:

# Manually execute generator to inspect parsing diagnostics
/usr/lib/systemd/system-generators/podman-system-generator --dryrun --user

Any unknown keys or malformed directives will be clearly logged to the terminal.

3. Inspecting the Compiled Transient Service Unit

To audit the underlying systemd directives synthesized by Quadlet:

# Display generated unit file from transient runtime directory
cat /run/user/$(id -u)/systemd/generator/webserver.service

This reveals the synthesized ExecStart, ExecStop, and logging configuration injected automatically by Podman.

4. Forensic Logging via journalctl

Because Quadlet containers run as native systemd units, all stdout and stderr streams are ingested by the journal:

# Stream live logs with precise timestamps
journalctl --user -u webserver.service -f

# Review logs since last host boot
journalctl --user -u webserver.service -b

# Filter for errors and critical messages
journalctl --user -u webserver.service -p err

5. Troubleshooting Matrix

Symptom Root Cause Remediation Workflow
Service terminates on SSH logout User lingering disabled Execute sudo loginctl enable-linger $USER.
Permission denied on volume paths Missing SELinux context or unmapped subUID ownership Append :Z,U to the volume directive: Volume=/host/dir:/cont/dir:Z,U.
Unit not found after reload Syntax error in .container or missing daemon-reload Run systemctl --user daemon-reload and verify output using the generator dry-run tool.
Cannot bind port < 1024 Linux kernel security boundary for unprivileged processes Bind to unprivileged ports (> 1024) or configure sudo sysctl net.ipv4.ip_unprivileged_port_start=80.
Auto-update fails silently Unauthenticated registry or ambiguous image tag Execute podman login <registry> as the service user and avoid pinned immutable digests.

Podman Quadlets integrate container workloads seamlessly with the foundational strengths of Enterprise Linux: operational stability, deterministic dependency trees, granular resource auditing, and autonomous self-healing without third-party daemon layers.