Zum Inhalt

Synthetic Availability Monitoring: Uptime Kuma Probes, Docker Socket Proxy, Push Heartbeats & Prometheus Metrics

Executive Summary: Discovering that a self-hosted service, TLS certificate, or backup cron job silently failed weeks ago destroys infrastructure reliability. This architectural guide details the production deployment of Uptime Kuma as a synthetic black-box availability monitor, implementing multi-protocol active probes (HTTP(s) JSONPath, TCP, DNSSEC, gRPC, PostgreSQL), dead-man's-switch Push Heartbeats for Borg/Restic backup jobs, read-only container state monitoring, and native /metrics exposition for Prometheus and Grafana.

  • Multi-Protocol Black-Box Probes

    Execute synthetic checks across HTTP/2/3 status codes, JSONPath payload assertions, TCP ports, DNS record resolution, and database handshakes.

  • Dead-Man's-Switch Push URLs

    Monitor asynchronous cron jobs, systemd timers, and air-gapped nodes that cannot be reached inbound by requiring periodic outbound webhook pings.

  • Proactive X.509 Expiry Alerts

    Parse remote TLS certificate chains on every HTTPS/TCP-TLS probe and dispatch multi-channel alerts 21, 14, and 7 days before expiration.

  • Native Prometheus Exporter

    Scrape monitor_status, monitor_response_time, and monitor_cert_days_remaining at /metrics for long-term Grafana SLA dashboards.

1. Architecture: Active Synthetic Probes vs. Passive Push Heartbeats

Uptime Kuma combines two complementary monitoring paradigms inside a single Node.js/Vue.js + SQLite (WAL) application:

  1. Active Pull Probes: Uptime Kuma initiates outbound connections at fixed intervals (e.g., every 60 seconds) to verify network reachability, TLS chain validity, and application-layer JSON health payloads.
  2. Passive Push Heartbeats (Dead-Man's Switch): Scheduled batch tasks (such as nightly borgmatic or restic backups) call a unique Uptime Kuma webhook URL upon completion (?status=up&msg=OK&ping=1420). If no heartbeat arrives within the configured grace window, Uptime Kuma triggers an alert.
%%{init: {'flowchart': {'wrappingWidth': 600, 'subGraphTitleMargin': {'top': 15, 'bottom': 15}}}}%%
flowchart TB
    subgraph Targets["Monitored Infrastructure"]
        WebApps["HTTPS / JSON APIs<br/>TLS Expiry + Keyword"]
        DNSNode["Unbound / Pi-hole<br/>DNS A / DNSSEC Probe"]
        CronJob["Nightly BorgBackup Timer<br/>Outbound curl Push"]
    end

    subgraph KumaEngine["Uptime Kuma Core (SQLite WAL)"]
        ProbeLoop["Async Probe Scheduler<br/>Retry / Flapping Filter"]
        PushEndpoint["Push Webhook Receiver<br/>/api/push/<token>"]
        PromExporter["Prometheus /metrics<br/>SLA & Latency Gauges"]
    end

    subgraph AlertChannels["Out-of-Band Alerting"]
        Notify["ntfy / Matrix / Telegram<br/> Gotify / SMTP"]
        Grafana["Prometheus + Grafana<br/>Long-Term SLA Trends"]
    end

    ProbeLoop -->|"60s Synthetic Poll"| WebApps
    ProbeLoop -->|"UDP/TCP 53 Query"| DNSNode
    CronJob -->|"HTTPS GET Heartbeat"| PushEndpoint
    ProbeLoop --> Notify
    PushEndpoint --> Notify
    PromExporter -->|"Scrape :3001/metrics"| Grafana

2. Production Container Deployment with Read-Only Socket Proxy

To monitor local Docker containers without exposing /var/run/docker.sock directly to the Uptime Kuma process, deploy a dedicated read-only socket-proxy sidecar on an isolated internal network (kuma_socket_net).

/opt/uptime-kuma/compose.yaml
services:
  kuma-socket-proxy:
    image: lscr.io/linuxserver/socket-proxy:latest
    container_name: kuma-socket-proxy
    restart: unless-stopped
    read_only: true
    tmpfs:
      - /run
    environment:
      CONTAINERS: 1
      POST: 0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - kuma_socket_net

  uptime-kuma:
    image: louislam/uptime-kuma:1
    container_name: uptime-kuma
    restart: unless-stopped
    depends_on:
      - kuma-socket-proxy
    ports:
      - "127.0.0.1:3001:3001/tcp"
    environment:
      TZ: "Europe/Berlin"
      UPTIME_KUMA_DISABLE_FRAME_SAMEORIGIN: "false"
    volumes:
      - ./data:/app/data:Z
      # Optional: Mount custom Internal Root CA so Kuma trusts .lan certificates
      - /usr/local/share/ca-certificates/caddy-internal-root-ca.crt:/etc/ssl/certs/caddy-internal-root-ca.pem:ro
    environment:
      NODE_EXTRA_CA_CERTS: "/etc/ssl/certs/caddy-internal-root-ca.pem"
    networks:
      - kuma_socket_net
      - proxy_net

networks:
  kuma_socket_net:
    internal: true
  proxy_net:
    external: true

Inside the Uptime Kuma UI (Settings → Docker Hosts), add a Docker Host with Connection Type TCP / HTTP and Docker Daemon URL http://kuma-socket-proxy:2375.

3. Eliminating Alert Fatigue: Retry & Flapping Mathematics

Single-packet network jitter or a 2-second container restart should never trigger a midnight wake-up page. Configure Uptime Kuma's two-stage state machine for every production probe:

Parameter Production Value State Machine Behavior
Heartbeat Interval 60s Normal polling cadence when the service is UP.
Retries 3 Number of consecutive failed checks required before transitioning from PENDING to DOWN and dispatching an alert.
Heartbeat Retry Interval 20s Accelerated polling cadence once the first failure is detected (PENDING state). Total time to alert: $60\text{s} + (3 \times 20\text{s}) = 120\text{s}$.
Resend Notification 0 (or every 30 times) Prevents inbox flooding during extended outages while still reminding on-call engineers every 30 minutes if desired.

Reverse Proxy WebSocket Requirement (Upgrade: websocket)

Uptime Kuma's reactive UI communicates exclusively over persistent Socket.IO WebSockets. If you place Uptime Kuma behind Nginx, Nginx Proxy Manager, or Traefik without enabling WebSocket support (proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade";), the dashboard will get stuck on "Connecting to socket server...".

4. Integrating Dead-Man's-Switch Push Probes into Systemd / Cron

Create a Push monitor in Uptime Kuma with a 86400s (24-hour) interval and 1 retry. Wrap your nightly backup script or systemd service so it reports execution duration (ping in milliseconds) and exit status directly to the Push endpoint:

/usr/local/bin/kuma-heartbeat-wrapper.sh
#!/usr/bin/env bash
set -euo pipefail

KUMA_PUSH_URL="https://status.nejat-hakan.de/api/push/x9Km2pL8qR"
START_MS=$(date +%s%3N)

if "$@"; then
    END_MS=$(date +%s%3N)
    DURATION_MS=$((END_MS - START_MS))
    curl -fsS --retry 3 "${KUMA_PUSH_URL}?status=up&msg=OK&ping=${DURATION_MS}" >/dev/null
else
    EXIT_CODE=$?
    curl -fsS --retry 3 "${KUMA_PUSH_URL}?status=down&msg=Failed_Exit_${EXIT_CODE}" >/dev/null
    exit "${EXIT_CODE}"
fi

5. Scraping Uptime Kuma Metrics in Prometheus

Uptime Kuma exposes a full Prometheus metrics endpoint at /metrics authenticated via HTTP Basic Auth (use any username and your Uptime Kuma API Key as the password):

Prometheus Scrape Job Configuration (prometheus.yml)
scrape_configs:
  - job_name: "uptime_kuma"
    scrape_interval: 30s
    metrics_path: "/metrics"
    scheme: "http"
    basic_auth:
      username: "prometheus"
      password_file: "/etc/prometheus/secrets/kuma_api_key"
    static_configs:
      - targets: ["127.0.0.1:3001"]

Verify the exported metrics directly via curl:

Verifying Prometheus Metrics Output
curl -s -u ":uk1_your_api_key_here" http://127.0.0.1:3001/metrics | grep -E "^monitor_(status|response_time|cert_days_remaining)"