Zum Inhalt

Task Automation & Service Orchestration: Systemd Units, .timer Schedules, Sandboxing & Cron

Executive Summary: Legacy cron jobs run with a stripped environment (PATH=/usr/bin:/bin), silently discard stderr if local MTA mail delivery is unconfigured, cannot declare network/mount dependencies (After=network-online.target), and cannot enforce cgroup resource caps or filesystem sandboxing. This architectural guide covers production .service and .timer systemd units, systemd-analyze calendar validation, cgroups v2 resource controls, systemd security sandboxing (systemd-analyze security), and defensive cron + flock patterns where POSIX cron remains required.

  • Systemd .timer Over Legacy Cron

    Replace silent cron failures with persistent, jitter-aware (RandomizedDelaySec), journald-logged systemd timers and OnFailure= alert hooks.

  • Custom .service Lifecycle Design

    Architect Type=simple, Type=notify, and Type=oneshot services with automatic crash recovery (Restart=on-failure) and dependency ordering.

  • Systemd Unit Sandboxing

    Isolate services using DynamicUser=yes, ProtectSystem=strict, PrivateTmp=yes, NoNewPrivileges=yes, and CapabilityBoundingSet=.

  • Overlap-Safe Cron (flock)

    Prevent overlapping cron stampedes using kernel advisory locks (flock -n) and explicit PATH / SHELL headers in /etc/cron.d/.

1. Architecture: Systemd .timer + .service vs. Legacy crond

In systemd, scheduling is cleanly decoupled into two paired unit files in /etc/systemd/system/: 1. <name>.timer: Defines when the job triggers (OnCalendar=, OnBootSec=, Persistent=true). 2. <name>.service: Defines what executes (ExecStart=), under which user/cgroup (User=, MemoryMax=), inside what security sandbox (ProtectSystem=strict), and what happens if it fails (OnFailure=notify-failure@%n.service):

%%{init: {'flowchart': {'wrappingWidth': 600, 'subGraphTitleMargin': {'top': 15, 'bottom': 15}}}}%%
flowchart TB
    subgraph TimerUnit["db-backup.timer"]
        Cal["OnCalendar=*-*-* 02:00:00<br/>Persistent=true<br/>RandomizedDelaySec=15m"]
    end

    subgraph ServiceUnit["db-backup.service (Type=oneshot)"]
        Sandbox["Cgroup v2 & Namespace Sandbox<br/>MemoryMax=2G, ProtectSystem=strict"]
        Exec["ExecStart=/usr/local/bin/backup.sh<br/>Logs -> systemd-journald"]
    end

    subgraph AlertUnit["OnFailure=notify-ops@%n.service"]
        Webhook["Dispatch Matrix / ntfy / Kuma<br/>Alert with Exit Status"]
    end

    Cal -->|"Triggers Unit"| Sandbox
    Sandbox --> Exec
    Exec -.->|"Non-Zero Exit Code"| Webhook

Systemd Timers vs. Legacy Cron Comparison

Engineering Feature Systemd .timer + .service Legacy crontab / /etc/cron.d
Logging & Observability Every line of stdout/stderr structured in journalctl -u <unit> with exit code & runtime metrics. Silently discarded unless local postfix/msmtp MTA is configured to deliver local mail.
Overlap Prevention Built-in Native: A .timer will never spawn a second instance of its .service if the previous run is still active. Spawns duplicate overlapping processes every minute unless manually wrapped in flock -n.
Missed Run Catch-Up Built-in via Persistent=true (if server was powered off at 02:00, runs immediately upon boot). Requires external anacron daemon (daily/weekly granularity only).
Manual Test Execution Run sudo systemctl start <name>.service at any time in the exact production environment. Must copy-paste command manually into an interactive shell (different environment variables!).

2. Authoring a Hardened Production .service and .timer Pair

Always place custom or overridden unit files inside /etc/systemd/system/ (never edit vendor files inside /lib/systemd/system/, as package upgrades will overwrite them).

/etc/systemd/system/restic-sync.service
[Unit]
Description=Encrypted Off-Site Restic Backup Synchronization
Documentation=https://nejat-hakan.de/
After=network-online.target local-fs.target
Wants=network-online.target
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
User=root
Group=root
EnvironmentFile=/etc/restic/env.conf
ExecStart=/usr/local/bin/restic-backup-job.sh

# --- Resource Scheduling & Cgroup v2 Caps ---
Nice=15
IOSchedulingClass=best-effort
IOSchedulingPriority=6
CPUQuota=150%
MemoryMax=2G

# --- Systemd Namespace & Filesystem Sandboxing ---
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/cache/restic /var/log
ProtectKernelTunables=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
/etc/systemd/system/restic-sync.timer
[Unit]
Description=Nightly Schedule for Encrypted Off-Site Restic Backup

[Timer]
# Run daily at 02:15:00 with up to 10 minutes of random jitter to prevent I/O storms
OnCalendar=*-*-* 02:15:00
RandomizedDelaySec=10m
Persistent=true
Unit=restic-sync.service

[Install]
WantedBy=timers.target
/etc/systemd/system/notify-failure@.service
[Unit]
Description=Out-of-Band Alert Notification for Failed Unit %i

[Service]
Type=oneshot
ExecStart=/usr/bin/curl -fsS -H "Title: Systemd Failure on %H" -H "Priority: high" \
  -d "Unit %i failed on host %H. Inspect via: journalctl -u %i -n 50" \
  https://ntfy.nejat-hakan.de/infra-alerts

3. Validating Calendar Expressions, Sandboxing & Activating Units

Before enabling a timer, verify your OnCalendar= expression using systemd-analyze calendar (which calculates the exact next 5 trigger timestamps) and audit the unit's security exposure score:

Validating Calendar Syntax, Security Score & Activating the Timer
# 1. Verify unit file syntax and test OnCalendar schedule calculation
systemd-analyze verify /etc/systemd/system/restic-sync.*
systemd-analyze calendar --iterations=5 "*-*-* 02:15:00"

# 2. Audit the security sandboxing score of any service (Target: < 2.5 SAFE)
systemd-analyze security restic-sync.service

# 3. Reload systemd manager configuration, enable the .timer, and test-run the .service once
sudo systemctl daemon-reload
sudo systemctl enable --now restic-sync.timer
sudo systemctl start restic-sync.service

# 4. Inspect all active timers with next/last execution timestamps
systemctl list-timers --all

4. Defensive Cron Engineering (flock, PATH & /etc/cron.d/)

When working on legacy appliances or POSIX systems where cron is mandatory, avoid user crontab -e for infrastructure tasks. Instead, deploy version-controlled drop-in files in /etc/cron.d/ (chmod 0644, owned by root:root) with an explicit SHELL, PATH, and flock -n lockfile wrapper so a slow job can never spawn overlapping runaway instances:

/etc/cron.d/cert-sync (Hardened Cron Drop-In)
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=""

# <min> <hour> <dom> <month> <dow> <user> <flock wrapper + command + syslog logger>
15 3 * * * root /usr/bin/flock -n /run/cert-sync.lock /usr/local/bin/sync-certs.sh 2>&1 | /usr/bin/logger -t cert-sync