fix(config): take the rollback backup BEFORE writing the new config

generate_config() wrote /etc/haproxy/haproxy.cfg and only then called
reload_haproxy_safely(), which called create_backup(). The "backup" was
therefore a copy of the config that had just been written, so on a validation
failure restore_backup() restored the identical broken bytes: the advertised
rollback was a no-op and a fatal haproxy.cfg stayed on disk, where
start_haproxy() refuses to launch. Same shape as the June 2026 incident where
a missing template produced a fatal config and took an edge down.

Reproduced end to end before the fix (invalid config generated -> "Backups
created successfully" -> "Backups restored successfully" -> haproxy.cfg on
disk still invalid, `haproxy -c` rc=1).

Changes:

* create_backup() is now called by generate_config() BEFORE the first write,
  which also covers blocked_ips.map (rewritten early in generate_config) and
  coraza-spoe.cfg - both previously written before the backup and, for the
  SPOE file, never backed up at all even though `haproxy -c` parses it.
* create_backup() refuses to promote a config HAProxy already rejects, so a
  broken file on disk cannot overwrite a known-good backup ("rollback" must
  not mean "restore a different broken config"). It returns (ok, status) so
  the caller knows whether a rollback target exists.
* promote_current_config_to_backup() records the config as known-good only
  after it has validated AND loaded, so a box whose first generation succeeded
  has a rollback target immediately, and a config that never loaded is never
  promoted.
* restore_backup() returns (restored, message) and distinguishes "no backup
  available" from "restored". Every caller now surfaces the difference; a
  failed rollback is logged CRITICAL and reported as ROLLBACK FAILED in the
  API error message instead of silently looking like a successful recovery.
* reload_haproxy_safely(backup_status=...) no longer takes its own backup - it
  runs after the write, where a backup is meaningless. Called without a status
  it logs the contract violation rather than overwriting a good backup.
* validate_config_file() separates "config is invalid" from "validator could
  not run" so a missing haproxy binary is not read as a bad config.
* Config writes are atomic (temp file + fsync + os.replace, mode preserved);
  a truncated haproxy.cfg is as fatal as an invalid one. Removes the dead
  temp_config_path variable whose comment claimed this already happened.
* Fast path: if the live config set is already byte-identical to the backup
  (the normal case after a successful reload), skip the re-validation and the
  copy, so this adds no `haproxy -c` latency to customer-facing API calls.

Tests: scripts/test-config-rollback.py - 17 self-contained stdlib-unittest
tests, no new dependencies (the repo has no Python test framework; the
existing scripts/test-*.sh are curl integration scripts). A stub `haproxy`
binary stands in for the validator. 12 of the 17 fail against the previous
code; every assertion was mutation-proven (9 mutations, each reddening only
the tests that cover it).

No template, QUIC or HTTP/3 changes.

.claude/skills/haproxy-manager-deploy/SKILL.md

@@ -0,0 +1,215 @@
+---
+name: haproxy-manager-deploy
+description: Use when shipping a haproxy-manager-base code change — editing templates, the Dockerfile, the Python manager, the coraza-spoa subdir, or static assets like errors/, then getting it onto whp01 or staging. Trigger eagerly on phrases like "deploy haproxy", "ship the haproxy change", "rebuild haproxy-manager", "update the WAF block page", "recreate haproxy-manager", or any time the next step would involve `git push` from this repo, `docker pull` on the image, or `container-manager.sh recreate`. Walks the Gitea-CI-auto-build + recreate flow, surfaces the named-volume shadowing foot-gun, and includes post-deploy verification.
+---
+
+# haproxy-manager-base commit / build / deploy
+
+This is procedural discipline for changes to `haproxy-manager-base`. The repository builds via Gitea Actions on push, not via a local build script (the WHP flow uses `build-release.sh`; this one doesn't — don't conflate them, see the `whp-deploy` skill in the whp repo for that one). Each step has caught a real foot-gun.
+
+## The pipeline at a glance
+
+```
+edit code (local)
+   └─> commit + push
+         └─> Gitea Actions auto-build (build-push.yaml / build-push-coraza.yaml)
+               ├─> publishes :latest tag to repo.anhonesthost.net
+               └─> wait for image (~2-4 min)
+                     └─> recreate container on target server
+                           └─> verify
+```
+
+Do not skip the verify step. The container can come up "healthy" while still serving stale config or missing a baked-in file (see Step 5).
+
+---
+
+## Step 0a — Resolve the target host (never hardcoded)
+
+This skill deliberately does **not** bake in a server hostname — this repo is mirrored to a public remote, so a real FQDN in the skill would leak into commits. Instead, resolve the deploy target into a `DEPLOY_HOST` shell variable that every `ssh` command below uses.
+
+```bash
+HOST_FILE=".claude/skills/haproxy-manager-deploy/target-host.local"
+DEPLOY_HOST="$(cat "$HOST_FILE" 2>/dev/null)"
+```
+
+- **If `$DEPLOY_HOST` is non-empty**, use it — that's the user's saved target. The file is gitignored, so the real hostname never lands in a commit.
+- **If it's empty**, ask the user which server this deploy targets (e.g. production vs. staging) and what its hostname or SSH alias is. Then offer to save it so future deploys don't have to ask:
+
+  ```bash
+  echo 'the-host-they-gave.example' > "$HOST_FILE"   # gitignored — safe to store the real FQDN here
+  ```
+
+Confirm `$DEPLOY_HOST` is set before running any `ssh` step:
+
+```bash
+[ -n "$DEPLOY_HOST" ] || echo "DEPLOY_HOST not set — ask the user for the target server"
+```
+
+All commands below assume the variable is set in the same shell session (`ssh root@"$DEPLOY_HOST" ...`).
+
+---
+
+## Step 0 — Confirm before pushing
+
+If the user just said "deploy" or "ship the haproxy fix", confirm what's actually changing: a template, the Python manager, the coraza-spoa subdir (separate image, separate workflow), or a static asset. Look at `git status` and `git diff` and read the diff back to the user if it's non-trivial.
+
+Anything that affects the customer-facing block path (e.g. `templates/hap_listener.tpl`, `errors/403-waf.html`) is **visible to every visitor on every site**. Authorization is per-deploy, not standing.
+
+---
+
+## Step 1 — Know which workflow your change triggers
+
+- `build-push.yaml` builds `haproxy-manager-base:latest` (the main image). Triggered by changes anywhere outside `coraza-spoa/`.
+- `build-push-coraza.yaml` builds `coraza-spoa:latest`. Triggered by changes inside `coraza-spoa/`.
+- `mirror-base-image.yaml` is a scheduled job mirroring upstream base images; unrelated to feature deploys.
+
+If you've changed both subtrees in one push, both workflows fire — note that the order they finish isn't guaranteed.
+
+---
+
+## Step 2 — Beware the `/etc/haproxy` named volume shadow
+
+If your change adds a NEW file that the running container needs (a baked-in asset, an errorfile, a new config snippet), **do not place it under `/etc/haproxy/` in the Dockerfile**. That path is a Docker named volume in deployed containers — image content only seeds the volume on first creation, so existing deployments will not see your new file even after a recreate.
+
+Safe paths for baked-in assets:
+- `/haproxy/...` (the image's WORKDIR — not volumed)
+- Anywhere outside `/etc/haproxy`, `/etc/letsencrypt`
+
+Reference the asset by absolute path from the haproxy config templates (e.g. `lf-file /haproxy/errors/403-waf.html`).
+
+See `feedback-haproxy-named-volume` memory for the full pattern.
+
+---
+
+## Step 3 — Commit + push
+
+Standard commit format with trailing `Co-Authored-By:` line. Match the recent commit message style (`git log --oneline -5`). Stage files explicitly by name.
+
+```bash
+git push origin main
+```
+
+Pushing immediately triggers the Gitea Actions build.
+
+---
+
+## Step 4 — Wait for the build
+
+The Go build inside coraza-spoa takes ~2-3 minutes; the haproxy-manager-base build is faster (~1-2 min). Don't bother polling the runs UI — just pull on the target server until the digest changes:
+
+```bash
+ssh root@"$DEPLOY_HOST" 'until docker pull -q repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest 2>&1 | tail -1 | grep -qE "Image is up to date|Status: Downloaded"; do sleep 15; done'
+```
+
+`-q` suppresses the noisy layer progress so the grep can match cleanly. If you started this command before the CI build finished, it'll loop until the new image lands; once the digest matches, it exits.
+
+To confirm you got the new image, check the image-creation time vs your push:
+
+```bash
+ssh root@"$DEPLOY_HOST" 'docker images repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base --format "{{.CreatedSince}}"'
+```
+
+It should say "X minutes ago" matching the build wait, not "yesterday".
+
+---
+
+## Step 5 — Verify the new image has what you think it has, BEFORE recreating
+
+The image is `gcr.io/distroless/static-debian12:nonroot`-based, no shell. To peek inside, run a one-shot with a sh entrypoint override (only works if you put one in the image — coraza-spoa is distroless and won't have sh; haproxy-manager-base is Python-based and does):
+
+```bash
+# haproxy-manager-base (has sh):
+ssh root@"$DEPLOY_HOST" 'docker run --rm --entrypoint sh repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest -c "ls /haproxy/errors/ && head -5 /haproxy/errors/403-waf.html"'
+
+# coraza-spoa (distroless, no sh) — use docker create + docker cp instead:
+ssh root@"$DEPLOY_HOST" 'docker create --name _peek repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest && docker cp _peek:/etc/coraza/config.yaml - | tar xO; docker rm _peek'
+```
+
+This step exists because the CI build can succeed but ship the wrong file (wrong commit pulled, build cache issue, etc.). Catching it here is one step earlier than catching it from a customer report.
+
+---
+
+## Step 6 — Recreate the container
+
+```bash
+ssh root@"$DEPLOY_HOST" '/root/whp/scripts/container-manager.sh recreate haproxy-manager'
+```
+
+For coraza-spoa changes:
+```bash
+ssh root@"$DEPLOY_HOST" '/root/whp/scripts/container-manager.sh recreate coraza-spoa'
+```
+
+`container-manager.sh recreate` does: stop, remove, docker pull (idempotent if already pulled), start with the right flags from settings.json. **It reads `/docker/whp/settings.json` for things like `coraza_waf.mode`**, so if the user has toggled mode while you were building, the recreated container reflects the current setting — not whatever it was when you started.
+
+---
+
+## Step 7 — Verify the deploy
+
+For haproxy-manager:
+
+```bash
+ssh root@"$DEPLOY_HOST" '
+echo "=== container ==="
+docker ps --filter name=haproxy-manager --format "image: {{.Image}} status: {{.Status}}"
+echo "=== healthy ==="
+docker inspect haproxy-manager --format "{{.State.Health.Status}}"
+echo "=== haproxy config valid ==="
+docker exec haproxy-manager haproxy -c -f /etc/haproxy/haproxy.cfg 2>&1 | tail -3
+echo "=== new asset reachable inside container ==="
+docker exec haproxy-manager ls -la /haproxy/errors/ 2>&1 | tail -3
+echo "=== panel health ==="
+curl -fsS -m 5 -o /dev/null -w "PANEL=%{http_code}\n" http://127.0.0.1:8000/health
+'
+```
+
+Pass criteria:
+- Container status = healthy
+- haproxy config validates (warnings OK, errors not)
+- Your new asset (if any) is at the expected path inside the running container
+- Panel returns 200
+
+If any check fails, the change still went out — diagnose immediately. Don't say "deploy complete" before this clears.
+
+---
+
+## Step 8 — End-to-end test if customer-visible
+
+If your change affects what a visitor sees (block pages, redirects, security responses), do a synthetic test that exercises the actual path. For WAF block-page changes, the recipe is:
+
+```bash
+# Inject a temporary ACL that forces the WAF deny path on a custom header,
+# fire one request, observe the rendered response, then revert + reload.
+ssh root@"$DEPLOY_HOST" '
+docker exec haproxy-manager cp /etc/haproxy/haproxy.cfg /tmp/cfg-bak
+docker exec haproxy-manager sh -c "sed -i \"/http-request send-spoe-group coraza coraza-req/a\\\\    http-request set-var(txn.coraza.action) str(deny) if { req.hdr(x-force-waf-block) -m str yes }\" /etc/haproxy/haproxy.cfg"
+docker exec haproxy-manager sh -c "echo reload | socat stdio /tmp/haproxy-cli" >/dev/null
+sleep 1
+curl -sSk -D - -H "x-force-waf-block: yes" -H "Host: <live-vhost>" "https://localhost/" | head -40
+# revert
+docker exec haproxy-manager cp /tmp/cfg-bak /etc/haproxy/haproxy.cfg
+docker exec haproxy-manager sh -c "echo reload | socat stdio /tmp/haproxy-cli" >/dev/null
+'
+```
+
+**Pick a real `<live-vhost>`.** The `Host:` header must match a domain currently served by this haproxy-manager, or the request won't route to the WAF path. Don't hardcode a customer hostname in this skill — pull a live one at test time (any entry from the panel's domain list, or `docker exec haproxy-manager ls /etc/letsencrypt/live`) and substitute it.
+
+**The injection point matters.** Insert AFTER `http-request send-spoe-group coraza coraza-req`, because the SPOE call overwrites `txn.coraza.action` based on the real Coraza verdict — if you inject before it, your override is wiped.
+
+**The reload mechanism matters.** Use `echo reload | socat stdio /tmp/haproxy-cli` — the container is python-based but doesn't have `kill` in PATH, and `docker kill --signal=HUP` signals the python manager (PID 1), not haproxy.
+
+---
+
+## Recovery hints
+
+- **`docker pull` exits "Image is up to date" but your change isn't there** — CI hasn't finished yet. Check `https://repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base/actions` for in-progress runs.
+- **Container recreates but new file is missing inside** — you put the file under `/etc/haproxy/` and the named volume shadows it. See Step 2. Move the file under `/haproxy/` (or another non-volumed path) and rebuild.
+- **HAProxy `lf-file` page renders but CSS is broken / percentages stripped** — literal `%` in the file body must be doubled (`100%%`). HAProxy log-format expansion eats single `%`. See `haproxy-lf-file-percent-escape` memory.
+- **Synthetic test returns 200 from gunicorn instead of the block page** — your test ACL is being overwritten by the SPOE call. Inject after `send-spoe-group`, not before.
+- **`docker exec haproxy-manager kill -HUP 1` fails** — the python-based container doesn't have `kill` in PATH. Use the haproxy admin socket: `echo reload | socat stdio /tmp/haproxy-cli`.
+
+---
+
+## Why this skill is rigid
+
+The pipeline is short, but the volume-shadowing trap and the SPOE-overwrite trap during testing each cost a 5-10 minute debugging detour during the session this skill was authored from. Both are silent failures — your change goes out, the container is healthy, and you only notice the bug when a customer report (or a careful synthetic test) surfaces it. The verification steps exist to catch them before that happens.

.gitea/workflows/build-push-coraza.yaml

@@ -34,6 +34,15 @@ jobs:
           username: ${{ secrets.CI_USER }}
           password: ${{ secrets.CI_TOKEN }}
 
+      # Mirror to GitHub Container Registry — see build-push.yaml for the
+      # secret/username convention.
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: shadowdao
+          password: ${{ secrets.GHCR_TOKEN }}
+
       - name: Build Image
         uses: docker/build-push-action@v6
         with:
@@ -42,3 +51,4 @@ jobs:
           push: true
           tags: |
             repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest
+            ghcr.io/shadowdao/coraza-spoa:latest

.gitea/workflows/build-push.yaml

@@ -25,10 +25,37 @@ jobs:
           username: ${{ secrets.CI_USER }}
           password: ${{ secrets.CI_TOKEN }}
 
+      # Second push target so the image is also available from GitHub Container
+      # Registry under the user's account. The PAT only needs write:packages
+      # (and read:packages if the package is private). Stored in Gitea as
+      # secrets.GHCR_TOKEN; username is the literal GitHub login.
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: shadowdao
+          password: ${{ secrets.GHCR_TOKEN }}
+
+      # Read the human-readable release version from the VERSION file so every
+      # build is pinnable for rollback (alongside the immutable git SHA). Bump
+      # VERSION (YYYY.MM.N) in the same commit as a release-worthy change.
+      - name: Read version
+        id: ver
+        run: echo "version=$(cat VERSION)" >> "$GITHUB_OUTPUT"
+
       - name: Build Image
         uses: docker/build-push-action@v6
         with:
           platforms: linux/amd64
           push: true
+          build-args: |
+            VERSION=${{ steps.ver.outputs.version }}
+          # Three tags per registry: :latest (moving), :<version> (human-readable
+          # release), :<sha> (immutable, guaranteed-unique rollback target).
           tags: |
             repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+            repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ steps.ver.outputs.version }}
+            repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ gitea.sha }}
+            ghcr.io/shadowdao/haproxy-manager-base:latest
+            ghcr.io/shadowdao/haproxy-manager-base:${{ steps.ver.outputs.version }}
+            ghcr.io/shadowdao/haproxy-manager-base:${{ gitea.sha }}

.gitignore

@@ -37,3 +37,7 @@ ENV/
 # OS
 .DS_Store
 Thumbs.db
+
+# Local-only deploy config (never commit real hostnames)
+.claude/skills/haproxy-manager-deploy/target-host.local
+*.local

CLAUDE.md

@@ -94,7 +94,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - `trusted_ips.list` — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
 - `trusted_ips.map` — Real IP whitelist for proxy-header matching (format: `<IP> 1`)
 - Both files are baked into the Docker image via `COPY` in the Dockerfile
-- Currently contains phone system IP `172.116.197.166`
+- Ship as comment-only templates (no real IPs). Add trusted IPs locally and do **not** commit them — this repo is mirrored publicly. Entries persist in the `/etc/haproxy` named volume across recreates
 
 ### Timeout Hardening (hap_header.tpl)
 

Dockerfile

@@ -9,6 +9,20 @@
 #     docker push repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
 # Future improvement: a scheduled Gitea Action that does the above automatically.
 FROM repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
+
+# image.source is what ghcr.io uses to link the package to a GitHub repo
+# sidebar; pointing at the public GitHub mirror enables that linking. The
+# canonical source-of-truth git remote is still Gitea, but Gitea's registry
+# doesn't consume this label, so there's no contention.
+# Stamped from the VERSION file by CI (build-arg) so `docker inspect` reports
+# what's running on any host. Defaults to "dev" for local/manual builds.
+ARG VERSION=dev
+LABEL org.opencontainers.image.title="haproxy-manager-base" \
+      org.opencontainers.image.description="HAProxy management API with Let's Encrypt automation, Coraza WAF integration, and template-driven config" \
+      org.opencontainers.image.source="https://github.com/shadowdao/haproxy-manager-base" \
+      org.opencontainers.image.version="${VERSION}" \
+      org.opencontainers.image.licenses="MIT"
+
 RUN apt update -y && apt dist-upgrade -y && apt install socat haproxy cron certbot curl jq net-tools -y && apt clean && rm -rf /var/lib/apt/lists/*
 WORKDIR /haproxy
 COPY ./templates /haproxy/templates
@@ -17,6 +31,11 @@ COPY haproxy_manager.py /haproxy/
 COPY scripts /haproxy/scripts
 COPY trusted_ips.list /etc/haproxy/trusted_ips.list
 COPY trusted_ips.map /etc/haproxy/trusted_ips.map
+# /etc/haproxy is a named volume in deployed containers, so baked-in files
+# under that path get shadowed by the volume on existing deployments.
+# Place errorfiles outside the volumed path; the HAProxy config references
+# them by absolute path.
+COPY errors /haproxy/errors
 RUN chmod +x /haproxy/scripts/*
 RUN pip install -r requirements.txt
 # Create log directories
@@ -28,7 +47,9 @@ RUN mkdir -p /var/spool/cron/crontabs && \
     echo '0 */12 * * * /haproxy/scripts/renew-certificates.sh >> /var/log/haproxy-manager.log 2>&1' >> /var/spool/cron/crontabs/root && \
     chmod 600 /var/spool/cron/crontabs/root && \
     chown root:crontab /var/spool/cron/crontabs/root
-EXPOSE 80 443 8000
+# 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
+# must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
+EXPOSE 80 443 443/udp 8000
 # Add health check
 HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
     CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1

README.md

@@ -6,10 +6,10 @@ A Flask-based API service for managing HAProxy configurations with dynamic SSL c
 To run the container:
 ```bash
 # Without API key authentication (default)
-docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 
 # With API key authentication (recommended for production)
-docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 ```
 
 ## Features
@@ -394,7 +394,7 @@ You can customize the default page by setting environment variables:
 
 ```bash
 docker run -d \
-  -p 80:80 -p 443:443 -p 8000:8000 \
+  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
   -v lets-encrypt:/etc/letsencrypt \
   -v haproxy:/etc/haproxy \
   -e HAPROXY_API_KEY=your-secure-api-key-here \
@@ -402,7 +402,7 @@ docker run -d \
   -e HAPROXY_DEFAULT_MAIN_MESSAGE="This website is currently under construction and will be available soon." \
   -e HAPROXY_DEFAULT_SECONDARY_MESSAGE="Please check back later or contact us for more information." \
   --name haproxy-manager \
-  repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 ```
 
 ## Example Usage
@@ -411,12 +411,12 @@ docker run -d \
 ```bash
 # Start container with API key
 docker run -d \
-  -p 80:80 -p 443:443 -p 8000:8000 \
+  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
   -v lets-encrypt:/etc/letsencrypt \
   -v haproxy:/etc/haproxy \
   -e HAPROXY_API_KEY=your-secure-api-key-here \
   --name haproxy-manager \
-  repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 
 # Add a domain
 curl -X POST http://localhost:8000/api/domain \

VERSION

@@ -0,0 +1 @@
+2026.08.1

coraza-spoa/Dockerfile

@@ -27,6 +27,16 @@ RUN git clone --depth 1 --branch "${CORAZA_SPOA_VERSION}" \
     && go mod download \
     && CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa .
 
+# Catalog extractor: walks the bundled CRS at build time and emits
+# rules-catalog.json so WHP's UI can render rule metadata without parsing
+# .conf files at runtime. Uses the SAME coraza-coreruleset version pin as
+# the coraza-spoa binary above (drift between the two would mislabel rules).
+FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog
+WORKDIR /src
+COPY catalog-extractor/ .
+RUN go build -trimpath -o /out/catalog-extractor . \
+    && /out/catalog-extractor > /out/rules-catalog.json
+
 # Distroless runtime: no shell, no package manager, no /tmp by default —
 # smallest attack surface for an exposed service. Audit log directory is
 # bind-mounted; coraza-spoa writes to it via direct file I/O (no shell needed).
@@ -34,11 +44,16 @@ FROM gcr.io/distroless/static-debian12:nonroot
 
 LABEL org.opencontainers.image.title="coraza-spoa-whp" \
       org.opencontainers.image.description="Coraza WAF SPOA agent configured for WHP haproxy-manager integration" \
-      org.opencontainers.image.source="https://repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base"
+      org.opencontainers.image.source="https://github.com/shadowdao/haproxy-manager-base" \
+      org.opencontainers.image.licenses="MIT"
 
 COPY --from=build /out/coraza-spoa /coraza-spoa
 COPY config.yaml /etc/coraza-spoa/config.yaml
 COPY overrides.conf /etc/coraza/overrides.conf
+COPY pre-overrides.conf /etc/coraza/pre-overrides.conf
+COPY local-overrides.conf /etc/coraza/local-overrides.conf
+COPY host-exceptions/ /etc/coraza/host-exceptions/
+COPY --from=catalog /out/rules-catalog.json /etc/coraza/rules-catalog.json
 
 # Audit log directory — bind-mount /var/log/coraza:/var/log/coraza from host
 # so logs persist across container restarts and AI Monitor can tail them.

coraza-spoa/README.md

@@ -32,7 +32,7 @@ docker run -d \
     --network client-net \
     --restart unless-stopped \
     -v /var/log/coraza:/var/log/coraza \
-    repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest
+    your-registry.example.com/cloud-hosting-platform/coraza-spoa:latest
 ```
 
 Then on the `haproxy-manager` container, add the env var:

coraza-spoa/catalog-extractor/go.mod

@@ -0,0 +1,7 @@
+module catalog-extractor
+
+go 1.23
+
+require github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
+
+require github.com/magefile/mage v1.17.0 // indirect

coraza-spoa/catalog-extractor/go.sum

@@ -0,0 +1,12 @@
+github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 h1:tqFO1lfVpTiyWtlN618OXpZMfw+nnN0Q4///W5W+/HM=
+github.com/corazawaf/coraza-coreruleset/v4 v4.25.0/go.mod h1:nRuGXITxOPvsLF2VxaTB7pYok8QB8BitX3ZenXcUryY=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM=
+github.com/magefile/mage v1.17.0/go.mod h1:Yj51kqllmsgFpvvSzgrZPK9WtluG3kUhFaBUVLo4feA=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

coraza-spoa/catalog-extractor/main.go

@@ -0,0 +1,80 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/fs"
+	"os"
+	"regexp"
+	"strconv"
+	"strings"
+
+	crs "github.com/corazawaf/coraza-coreruleset/v4"
+)
+
+type Rule struct {
+	ID       int      `json:"id"`
+	Msg      string   `json:"msg"`
+	Severity string   `json:"severity"`
+	Tags     []string `json:"tags"`
+	File     string   `json:"file"`
+}
+
+var (
+	idRe       = regexp.MustCompile(`(?i)\bid:'?(\d+)'?`)
+	msgRe      = regexp.MustCompile(`(?i)\bmsg:'([^']+)'`)
+	severityRe = regexp.MustCompile(`(?i)\bseverity:'?([A-Z]+)'?`)
+	tagRe      = regexp.MustCompile(`(?i)\btag:'([^']+)'`)
+)
+
+func main() {
+	out := []Rule{}
+	err := fs.WalkDir(crs.FS, ".", func(path string, d fs.DirEntry, err error) error {
+		if err != nil || d.IsDir() {
+			return err
+		}
+		if !strings.HasSuffix(path, ".conf") {
+			return nil
+		}
+		b, err := fs.ReadFile(crs.FS, path)
+		if err != nil {
+			return err
+		}
+		// Coalesce backslash-continuation lines so id/msg/etc on the same
+		// logical rule are visible to the per-line scanner.
+		text := regexp.MustCompile(`\\\s*\n\s*`).ReplaceAllString(string(b), " ")
+		for _, line := range strings.Split(text, "\n") {
+			line = strings.TrimSpace(line)
+			if !strings.HasPrefix(line, "SecRule") && !strings.HasPrefix(line, "SecAction") {
+				continue
+			}
+			m := idRe.FindStringSubmatch(line)
+			if m == nil {
+				continue
+			}
+			id, _ := strconv.Atoi(m[1])
+			r := Rule{ID: id, File: path}
+			if mm := msgRe.FindStringSubmatch(line); mm != nil {
+				r.Msg = mm[1]
+			}
+			if mm := severityRe.FindStringSubmatch(line); mm != nil {
+				r.Severity = strings.ToLower(mm[1])
+			}
+			for _, mm := range tagRe.FindAllStringSubmatch(line, -1) {
+				r.Tags = append(r.Tags, mm[1])
+			}
+			out = append(out, r)
+		}
+		return nil
+	})
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	enc := json.NewEncoder(os.Stdout)
+	enc.SetIndent("", "  ")
+	if err := enc.Encode(out); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}

coraza-spoa/config.yaml

@@ -27,6 +27,14 @@ applications:
       # the rule pack itself (~16 MB of rules embedded in the binary).
       Include @coraza.conf-recommended
       Include @crs-setup.conf.example
+
+      # Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.
+      # Loaded BEFORE the CRS rules so per-host ctl:ruleRemoveById exemptions
+      # fire in phase:1 BEFORE the CRS rule they're trying to exempt would
+      # otherwise match. Server-wide overrides live in local-overrides.conf
+      # (loaded after CRS) instead.
+      Include /etc/coraza/pre-overrides.conf
+
       Include @owasp_crs/*.conf
 
       # WHP-specific overrides — day-one enforce list, plus tuning for
@@ -34,6 +42,9 @@ applications:
       # to see exactly what blocks vs what's detect-only.
       Include /etc/coraza/overrides.conf
 
+      # Runtime-managed POST-CRS overrides written by WHP UI. Empty by default.
+      Include /etc/coraza/local-overrides.conf
+
       # Global mode: log all alerts, block only what overrides.conf
       # explicitly promotes via ctl:ruleEngine=On.
       SecRuleEngine DetectionOnly

coraza-spoa/local-overrides.conf

@@ -0,0 +1,3 @@
+# AUTOGENERATED by WHP — do not hand-edit.
+# Source of truth: whp.security_db coraza_rule_overrides table.
+# Empty file = no runtime overrides; baked-in overrides.conf governs.

coraza-spoa/overrides.conf

@@ -94,3 +94,12 @@ SecRuleUpdateActionById 930130 "ctl:ruleEngine=On"
 #                         (`session_start` literal appearing in billing form data)
 #   950xxx-953xxx       — Data leakage / backup-file disclosure (mixed FP)
 # ---------------------------------------------------------------------------
+
+# ---------------------------------------------------------------------------
+# RESERVED RULE-ID RANGE: 990000000 – 990999999
+# WHP's coraza_rule_manager generates per-host-exception rules in this range
+# (rule ID = 990000000 + target_rule_id). Do NOT add new rules in this range
+# from any other source. When bumping the coraza-spoa pin, check the CRS
+# changelog for new rules with 9-digit IDs (rare but possible) and re-namespace
+# if collision risk emerges.
+# ---------------------------------------------------------------------------

coraza-spoa/pre-overrides.conf

@@ -0,0 +1,3 @@
+# AUTOGENERATED by WHP — do not hand-edit.
+# Source of truth: whp.security_db coraza_rule_host_exceptions table.
+# Loaded BEFORE the CRS rules. Empty file = no per-host exemptions active.

errors/403-waf.html

@@ -0,0 +1,109 @@
+<!DOCTYPE html>
+<!--
+  Served by HAProxy via `lf-file` on Coraza WAF deny.
+  IMPORTANT: HAProxy's lf-file expansion treats `%` as the start of a
+  log-format expression. Literal percent signs (CSS 100%, gradient stops,
+  url-encoded data, etc.) MUST be doubled as `%%` or HAProxy will silently
+  swallow them. Expressions like `%[unique-id]` / `%[req.hdr(host)]` stay
+  single-`%` — those are the substitutions we want.
+-->
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="robots" content="noindex, nofollow">
+<title>Request blocked &middot; %[req.hdr(host)]</title>
+<style>
+  *, *::before, *::after { box-sizing: border-box; }
+  html, body { margin: 0; padding: 0; height: 100%%; }
+  body {
+    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+    color: #1f2937;
+    background: linear-gradient(135deg, #f9fafb 0%%, #eef2f7 100%%);
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    padding: 24px;
+    line-height: 1.5;
+  }
+  .card {
+    background: #fff;
+    border-radius: 12px;
+    box-shadow: 0 1px 3px rgba(0,0,0,0.05), 0 12px 32px rgba(31,41,55,0.08);
+    max-width: 560px;
+    width: 100%%;
+    padding: 36px 40px;
+  }
+  .badge {
+    display: inline-block;
+    background: #fef3c7;
+    color: #92400e;
+    font-size: 12px;
+    font-weight: 600;
+    letter-spacing: 0.04em;
+    text-transform: uppercase;
+    padding: 4px 10px;
+    border-radius: 999px;
+    margin-bottom: 16px;
+  }
+  h1 { font-size: 22px; margin: 0 0 12px; color: #111827; }
+  p { margin: 0 0 14px; color: #374151; }
+  .ref {
+    background: #f3f4f6;
+    border: 1px solid #e5e7eb;
+    border-radius: 6px;
+    padding: 12px 14px;
+    margin: 20px 0;
+    font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
+    font-size: 13px;
+    color: #111827;
+    word-break: break-all;
+  }
+  .ref-label {
+    display: block;
+    color: #6b7280;
+    font-size: 11px;
+    font-weight: 600;
+    letter-spacing: 0.05em;
+    text-transform: uppercase;
+    margin-bottom: 4px;
+    font-family: inherit;
+  }
+  .owner {
+    border-top: 1px solid #e5e7eb;
+    margin-top: 24px;
+    padding-top: 20px;
+    color: #4b5563;
+    font-size: 14px;
+  }
+  .owner h2 { font-size: 14px; font-weight: 600; color: #111827; margin: 0 0 8px; }
+  a {
+    color: #1d4ed8;
+    text-decoration: none;
+    border-bottom: 1px solid transparent;
+  }
+  a:hover, a:focus { border-bottom-color: #1d4ed8; }
+  .small { font-size: 12px; color: #6b7280; margin-top: 16px; }
+</style>
+</head>
+<body>
+  <main class="card" role="main">
+    <span class="badge">Access blocked</span>
+    <h1>Your request was blocked by our security filter</h1>
+    <p>The request to <strong>%[req.hdr(host)]</strong> looked suspicious to our web application firewall and was not delivered to the site.</p>
+    <p>This is automated. No one has reviewed the request yet.</p>
+
+    <div class="ref">
+      <span class="ref-label">Request reference</span>
+      %[unique-id]
+    </div>
+
+    <div class="owner">
+      <h2>Site owner?</h2>
+      <p>If you operate this site and believe this block is incorrect, please contact your hosting provider's support team and include the request reference above. They can look up exactly which rule fired and adjust it if it's a false positive.</p>
+    </div>
+
+    <p class="small">Reference IDs expire from our active logs after 14 days, so please open a ticket promptly if you'd like this investigated.</p>
+  </main>
+</body>
+</html>

haproxy_manager.py

@@ -12,12 +12,45 @@ from datetime import datetime, timedelta
 import json
 import ipaddress
 import shutil
+import stat
 import tempfile
 import threading
 import time
 import re
 import fcntl
 
+# ---------------------------------------------------------------------------
+# Bounded subprocess execution (incident 2026-07-07)
+# ---------------------------------------------------------------------------
+# Every external command this manager runs — certbot ACME issuance/renewal,
+# `socat` reloads over the haproxy admin socket, `haproxy -c` validation — is a
+# potential hang. The management API runs under gunicorn gthread workers, and a
+# subprocess.run() with NO timeout blocks its worker thread forever if the
+# command stalls (e.g. an ACME/upstream that stops responding mid-read).
+# gunicorn's --timeout does not rescue this: for gthread it only kills a worker
+# whose *main* thread stops heart-beating, but the main thread keeps polling
+# while pool threads are wedged. Enough stalled calls exhaust the 4-thread pool
+# and the whole API stops responding — "healthy" health-check, every request
+# 30s-timeouts — which is exactly what stalled WHP site updates on 2026-07-07.
+#
+# Fix: give EVERY subprocess.run() a default timeout unless the caller passes
+# one explicitly. On expiry Python kills the child and raises
+# subprocess.TimeoutExpired (a subclass of Exception); the existing per-endpoint
+# try/except turns that into a clean error AND releases the worker thread.
+# Bounding by default (instead of editing ~30 call sites) means no site can be
+# missed and any future call is protected automatically.
+DEFAULT_SUBPROCESS_TIMEOUT = int(os.environ.get('HAPROXY_MGR_SUBPROCESS_TIMEOUT', '180'))
+_unbounded_subprocess_run = subprocess.run
+
+
+def _bounded_subprocess_run(*args, **kwargs):
+    if kwargs.get('timeout') is None:
+        kwargs['timeout'] = DEFAULT_SUBPROCESS_TIMEOUT
+    return _unbounded_subprocess_run(*args, **kwargs)
+
+
+subprocess.run = _bounded_subprocess_run
+
 app = Flask(__name__)
 
 # Default page server (port 8080) — served to HAProxy clients whose request hit
@@ -73,8 +106,18 @@ HAPROXY_CONFIG_PATH = '/etc/haproxy/haproxy.cfg'
 HAPROXY_BACKUP_PATH = '/etc/haproxy/haproxy.cfg.backup'
 BLOCKED_IPS_MAP_PATH = '/etc/haproxy/blocked_ips.map'
 BLOCKED_IPS_MAP_BACKUP_PATH = '/etc/haproxy/blocked_ips.map.backup'
+# Coraza SPOE engine file. `haproxy -c` parses this too (the frontend's
+# `filter spoe engine coraza config <path>` line points at it), so it is part
+# of the same restorable config set as haproxy.cfg — rolling back haproxy.cfg
+# while leaving a broken coraza-spoe.cfg behind still fails validation.
+CORAZA_SPOE_CONFIG_PATH = '/etc/haproxy/coraza-spoe.cfg'
+CORAZA_SPOE_BACKUP_PATH = '/etc/haproxy/coraza-spoe.cfg.backup'
 HAPROXY_SOCKET_PATH = '/var/run/haproxy.sock'
 SSL_CERTS_DIR = '/etc/haproxy/certs'
+# Stable per-host secret for QUIC Retry/address-validation tokens. Lives in the
+# /etc/haproxy named volume so it survives container recreates; self-healed on
+# first config render. See get_or_create_cluster_secret().
+CLUSTER_SECRET_PATH = '/etc/haproxy/cluster-secret'
 API_KEY = os.environ.get('HAPROXY_API_KEY')  # Optional API key for authentication
 
 # Setup logging
@@ -807,10 +850,12 @@ def renew_certificates():
         # Defensive: clear any stale lock left by a SIGKILLed prior run.
         clear_stale_certbot_locks()
 
-        # Run certbot renew
+        # Run certbot renew. Explicit long timeout (overrides the module
+        # default): `renew` walks every lineage and can legitimately make many
+        # ACME round-trips when several certs are actually due.
         result = subprocess.run([
             'certbot', 'renew', '--quiet'
-        ], capture_output=True, text=True)
+        ], capture_output=True, text=True, timeout=900)
         
         if result.returncode == 0:
             # Check if any certificates were renewed
@@ -1687,6 +1732,45 @@ def dns_challenge_verify():
         log_operation('dns_challenge_verify', False, str(e))
         return jsonify({'success': False, 'error': str(e)}), 500
 
+def get_or_create_cluster_secret():
+    """Return a stable secret for QUIC token derivation, generating it once.
+
+    HAProxy uses `cluster-secret` to key QUIC Retry/address-validation tokens.
+    Without a stable value it picks a random one each (re)start and logs a
+    notice; tokens then don't survive reloads. We persist one in the
+    /etc/haproxy named volume so it's stable across container recreates.
+    Exclusive-create avoids a race if two renders run concurrently. Failure to
+    read/write is non-fatal: we fall back to an empty string and the template
+    simply omits the directive (HAProxy reverts to its random-per-process
+    behaviour), so QUIC still works.
+    """
+    try:
+        if os.path.exists(CLUSTER_SECRET_PATH):
+            with open(CLUSTER_SECRET_PATH, 'r') as f:
+                secret = f.read().strip()
+                if secret:
+                    return secret
+        # Generate and persist exclusively (0600). hex => config-safe charset.
+        secret = os.urandom(32).hex()
+        fd = os.open(CLUSTER_SECRET_PATH, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+        try:
+            os.write(fd, secret.encode())
+        finally:
+            os.close(fd)
+        logger.info("Generated new QUIC cluster-secret at %s", CLUSTER_SECRET_PATH)
+        return secret
+    except FileExistsError:
+        # Lost the create race — another render just wrote it; read it back.
+        try:
+            with open(CLUSTER_SECRET_PATH, 'r') as f:
+                return f.read().strip()
+        except Exception as e:
+            logger.error("Failed to read cluster-secret after race: %s", e)
+            return ''
+    except Exception as e:
+        logger.error("Failed to get/create cluster-secret: %s", e)
+        return ''
+
 def generate_config():
     try:
         conn = sqlite3.connect(DB_FILE)
@@ -1718,6 +1802,21 @@ def generate_config():
         
         config_parts = []
 
+        # Snapshot the last-known-good config BEFORE anything below touches a
+        # file in /etc/haproxy. Everything this function writes (haproxy.cfg,
+        # blocked_ips.map, coraza-spoe.cfg) is validated as one set by
+        # `haproxy -c`, so the rollback point has to predate the first of them.
+        # Taking it here (rather than inside reload_haproxy_safely(), which runs
+        # after the writes) is what makes rollback real - see create_backup().
+        backup_ok, backup_status = create_backup()
+        if not backup_ok:
+            # Could not even attempt a snapshot (I/O error). Writing a new
+            # config now would leave us with no way back, so refuse.
+            raise Exception(
+                "Refusing to regenerate config: failed to back up the current "
+                "configuration, so a failed change could not be rolled back"
+            )
+
         # Optional Coraza WAF integration. When HAPROXY_CORAZA_SPOE_BACKEND is
         # set on the haproxy-manager container, we render an extra TCP backend
         # pointing at a coraza-spoa sidecar AND inject a `filter spoe ...` line
@@ -1749,7 +1848,9 @@ def generate_config():
                     logger.error(f"Failed to create {suspended_list_path}: {e}")
 
         # Add Haproxy Default Headers
-        default_headers = template_env.get_template('hap_header.tpl').render()
+        default_headers = template_env.get_template('hap_header.tpl').render(
+            cluster_secret = get_or_create_cluster_secret(),
+        )
         config_parts.append(default_headers)
 
         # Update blocked IPs map file first
@@ -1810,7 +1911,11 @@ def generate_config():
         # First pass: exact domain ACLs (higher priority - evaluated first)
         for domain in exact_domains:
             if not domain['backend_name']:
-                logger.warning(f"Skipping domain {domain['domain']} - no backend name")
+                # Expected for domains registered without a proxy backend (e.g. the
+                # panel's own hostname, present only for certificate management).
+                # Log at INFO — not WARNING — so it doesn't trip log monitors as an
+                # error; it recurs on every generate_config by design.
+                logger.info(f"Skipping domain {domain['domain']} - no proxy backend (cert/management-only)")
                 continue
 
             try:
@@ -1829,7 +1934,8 @@ def generate_config():
         # Second pass: wildcard domain ACLs (lower priority - evaluated after exact matches)
         for domain in wildcard_domains:
             if not domain['backend_name']:
-                logger.warning(f"Skipping wildcard domain {domain['domain']} - no backend name")
+                # See note above — INFO, not WARNING; expected for cert/management-only domains.
+                logger.info(f"Skipping wildcard domain {domain['domain']} - no proxy backend (cert/management-only)")
                 continue
 
             try:
@@ -1900,25 +2006,21 @@ backend default-backend
             # how the file was authored.
             if not coraza_spoe_cfg.endswith('\n'):
                 coraza_spoe_cfg += '\n'
-            coraza_spoe_path = '/etc/haproxy/coraza-spoe.cfg'
+            write_config_atomically(CORAZA_SPOE_CONFIG_PATH, coraza_spoe_cfg)
-            with open(coraza_spoe_path, 'w') as f:
+            logger.info(f"Coraza SPOE engine config written to "
-                f.write(coraza_spoe_cfg)
+                        f"{CORAZA_SPOE_CONFIG_PATH} "
-            logger.info(f"Coraza SPOE engine config written to {coraza_spoe_path} "
                         f"(SPOA target: {coraza_spoe_backend})")
 
-        # Write complete configuration to tmp
-        temp_config_path = "/etc/haproxy/haproxy.cfg"
-
         config_content = '\n'.join(config_parts)
         logger.debug("Generated HAProxy configuration")
 
-        # Write complete configuration to tmp
+        # Write new configuration to file (atomically - a truncated haproxy.cfg
-        # Write new configuration to file
+        # is as fatal as an invalid one). The rollback point was taken above,
-        with open(HAPROXY_CONFIG_PATH, 'w') as f:
+        # before this write.
-            f.write(config_content)
+        write_config_atomically(HAPROXY_CONFIG_PATH, config_content)
 
         # Use safe reload with validation and rollback
-        success, message = reload_haproxy_safely()
+        success, message = reload_haproxy_safely(backup_status=backup_status)
         if success:
             logger.info("Configuration generated and HAProxy reloaded safely")
             log_operation('generate_config', True, 'Configuration generated and HAProxy reloaded safely')
@@ -1935,61 +2037,304 @@ backend default-backend
         traceback.print_exc()
         raise
 
-def create_backup():
+# ---------------------------------------------------------------------------
-    """Create backup of current config and map files"""
+# Config backup / rollback
+# ---------------------------------------------------------------------------
+# Rollback only works if the backup predates the write it is supposed to undo.
+# Until 2026-08 create_backup() ran from inside reload_haproxy_safely(), i.e.
+# AFTER generate_config() had already overwritten haproxy.cfg — so the "backup"
+# was a copy of the new (possibly broken) config and restore_backup() restored
+# the same broken bytes. The advertised rollback was a no-op and a fatal
+# haproxy.cfg persisted on disk, where start_haproxy() refuses to launch (the
+# June 2026 missing-template incident). create_backup() must now be called by
+# the writer, BEFORE the first byte is written.
+
+# Statuses returned by create_backup() that mean a rollback target exists.
+_ROLLBACK_AVAILABLE_STATUSES = ('created', 'kept_previous')
+
+
+def _files_identical(path_a, path_b):
+    """Byte-compare two files.
+
+    Deliberately not filecmp.cmp(): it memoises on (size, mtime), and
+    shutil.copy2() preserves mtime, so a stale cache entry could report a
+    changed config as unchanged. These files are small; read them.
+    """
     try:
-        if os.path.exists(HAPROXY_CONFIG_PATH):
+        if os.path.getsize(path_a) != os.path.getsize(path_b):
-            shutil.copy2(HAPROXY_CONFIG_PATH, HAPROXY_BACKUP_PATH)
+            return False
-        if os.path.exists(BLOCKED_IPS_MAP_PATH):
+        with open(path_a, 'rb') as fa, open(path_b, 'rb') as fb:
-            shutil.copy2(BLOCKED_IPS_MAP_PATH, BLOCKED_IPS_MAP_BACKUP_PATH)
+            while True:
-        logger.info("Backups created successfully")
+                chunk_a = fa.read(65536)
+                chunk_b = fb.read(65536)
+                if chunk_a != chunk_b:
+                    return False
+                if not chunk_a:
                     return True
+    except OSError:
+        return False
+
+
+def _config_set_matches_backup():
+    """True if every live config file is byte-identical to its backup copy.
+
+    After a successful reload the live set has already been recorded as
+    known-good (see promote_current_config_to_backup()), which is the common
+    case at the start of the next generation. Recognising it lets create_backup()
+    skip both the re-validation and the copy - worth doing because
+    `haproxy -c` on an edge with hundreds of certificates is not free and
+    generate_config() runs synchronously inside customer-facing API calls.
+    """
+    for live_path, backup_path in _config_backup_pairs():
+        if os.path.exists(live_path) != os.path.exists(backup_path):
+            return False
+        if (os.path.exists(live_path)
+                and not _files_identical(live_path, backup_path)):
+            return False
+    return True
+
+
+def _config_backup_pairs():
+    """(live, backup) pairs forming one restorable config set.
+
+    Built at call time rather than at import so the module-level path constants
+    stay patchable (tests, alternate deployments).
+    """
+    return (
+        (HAPROXY_CONFIG_PATH, HAPROXY_BACKUP_PATH),
+        (BLOCKED_IPS_MAP_PATH, BLOCKED_IPS_MAP_BACKUP_PATH),
+        (CORAZA_SPOE_CONFIG_PATH, CORAZA_SPOE_BACKUP_PATH),
+    )
+
+
+def write_config_atomically(path, content):
+    """Write content to path via temp file + rename.
+
+    A half-written haproxy.cfg (disk full, container killed mid-write) is just
+    as fatal as an invalid one and is invisible to the caller. os.replace() is
+    atomic within a filesystem, so the file on disk is always either the whole
+    old config or the whole new one — never a truncated hybrid. This also keeps
+    the "existing config is already broken" case from being self-inflicted.
+    """
+    directory = os.path.dirname(path) or '.'
+    # Preserve the mode of the file we are replacing; mkstemp defaults to 0600
+    # and HAProxy config files are conventionally 0644.
+    try:
+        mode = stat.S_IMODE(os.stat(path).st_mode)
+    except OSError:
+        mode = 0o644
+    fd, tmp_path = tempfile.mkstemp(
+        dir=directory, prefix=os.path.basename(path) + '.', suffix='.tmp'
+    )
+    try:
+        with os.fdopen(fd, 'w') as f:
+            f.write(content)
+            f.flush()
+            os.fsync(f.fileno())
+        os.chmod(tmp_path, mode)
+        os.replace(tmp_path, path)
+    except Exception:
+        try:
+            os.unlink(tmp_path)
+        except OSError:
+            pass
+        raise
+
+
+def create_backup(require_valid=True):
+    """Snapshot the CURRENT on-disk config set as the rollback point.
+
+    MUST be called BEFORE the new configuration is written — see the module
+    comment above. Calling it afterwards silently disarms rollback.
+
+    require_valid=True (default) refuses to promote a config that HAProxy
+    already rejects. Backing up a broken config would make "rollback" mean
+    "restore a different broken config"; keeping the older, validated backup
+    instead means a rollback always lands on something HAProxy will actually
+    start with. Cost is one `haproxy -c` run per config generation.
+
+    Returns (ok, status):
+      ok=False, status='error'      - the copy itself failed; caller decides.
+      status='created'              - backup now holds the current config.
+      status='kept_previous'        - current config missing or invalid; the
+                                      existing (older, good) backup was kept.
+      status='unavailable'          - nothing to roll back to at all (first
+                                      run, or broken config and no prior
+                                      backup). Rollback is NOT possible.
+    """
+    try:
+        snapshot_ok = True
+        reason = None
+
+        if not os.path.exists(HAPROXY_CONFIG_PATH):
+            snapshot_ok = False
+            reason = 'no existing HAProxy config on disk (first run?)'
+        elif _config_set_matches_backup():
+            # The backup already IS the current config, recorded when it last
+            # loaded successfully. Nothing to copy and nothing to re-validate.
+            logger.debug("Config backup already matches the live config")
+            return True, 'created'
+        elif require_valid:
+            status, msg = validate_config_file(HAPROXY_CONFIG_PATH)
+            if status == 'invalid':
+                snapshot_ok = False
+                reason = f'current config on disk does not validate: {msg}'
+            elif status == 'unavailable':
+                # The validator itself could not run (no haproxy binary, etc).
+                # That is NOT evidence the config is bad, and refusing to back
+                # up would leave us with no rollback target at all, so fall
+                # back to last-written semantics and say so loudly.
+                logger.warning(
+                    f"Could not verify current config before backup ({msg}); "
+                    "backing it up unverified"
+                )
+
+        if not snapshot_ok:
+            if os.path.exists(HAPROXY_BACKUP_PATH):
+                logger.warning(
+                    f"Not refreshing config backup: {reason}. Keeping the "
+                    f"existing backup at {HAPROXY_BACKUP_PATH} as the rollback "
+                    "target."
+                )
+                return True, 'kept_previous'
+            logger.error(
+                f"No config backup could be taken: {reason}, and no previous "
+                f"backup exists at {HAPROXY_BACKUP_PATH}. ROLLBACK IS NOT "
+                "AVAILABLE for this configuration change."
+            )
+            return True, 'unavailable'
+
+        for live_path, backup_path in _config_backup_pairs():
+            if os.path.exists(live_path):
+                shutil.copy2(live_path, backup_path)
+        logger.info("Backup of last-known-good config created successfully")
+        return True, 'created'
     except Exception as e:
         logger.error(f"Failed to create backup: {e}")
-        return False
+        return False, 'error'
 
-def restore_backup():
+def promote_current_config_to_backup():
-    """Restore from backup files"""
+    """Record the live config as the known-good rollback target.
+
+    Called ONLY after the config has both validated and been loaded by HAProxy,
+    so "backup" really means "the last configuration this box was running".
+    Must never be called before a reload attempt: doing so would make the
+    backup a copy of the config we may still have to roll back from - the same
+    class of bug as backing up after the write.
+
+    Without this, a box whose very first generation succeeded has no rollback
+    target at all until its second successful generation, and any corruption of
+    haproxy.cfg in between leaves nothing to recover to.
+    """
     try:
-        if os.path.exists(HAPROXY_BACKUP_PATH):
+        for live_path, backup_path in _config_backup_pairs():
-            shutil.copy2(HAPROXY_BACKUP_PATH, HAPROXY_CONFIG_PATH)
+            if os.path.exists(live_path):
-        if os.path.exists(BLOCKED_IPS_MAP_BACKUP_PATH):
+                shutil.copy2(live_path, backup_path)
-            shutil.copy2(BLOCKED_IPS_MAP_BACKUP_PATH, BLOCKED_IPS_MAP_PATH)
+        logger.debug("Known-good config backup updated after successful reload")
-        logger.info("Backups restored successfully")
         return True
     except Exception as e:
-        logger.error(f"Failed to restore backup: {e}")
+        # Non-fatal: the config is live and working, we just failed to record
+        # it. Loud, because the next change now has a staler rollback target.
+        logger.error(f"Failed to record known-good config backup: {e}")
         return False
 
-def validate_haproxy_config():
+
-    """Validate HAProxy configuration file"""
+def restore_backup():
+    """Restore the backed-up config set over the live files.
+
+    Returns (restored, message). restored=False means NOTHING was rolled back
+    and the live config is still whatever the failed change left on disk —
+    callers MUST surface that difference, it is the difference between "we
+    recovered" and "this edge is sitting on a config HAProxy will not load".
+    """
+    if not os.path.exists(HAPROXY_BACKUP_PATH):
+        msg = (f"No config backup at {HAPROXY_BACKUP_PATH} - cannot roll back; "
+               f"{HAPROXY_CONFIG_PATH} still holds the failed configuration")
+        logger.critical(msg)
+        return False, msg
     try:
-        result = subprocess.run(['haproxy', '-c', '-f', HAPROXY_CONFIG_PATH], 
+        for live_path, backup_path in _config_backup_pairs():
+            if os.path.exists(backup_path):
+                shutil.copy2(backup_path, live_path)
+        msg = f"Configuration restored from backup ({HAPROXY_BACKUP_PATH})"
+        logger.info(msg)
+        return True, msg
+    except Exception as e:
+        msg = (f"Failed to restore backup: {e} - {HAPROXY_CONFIG_PATH} may hold "
+               "a broken configuration")
+        logger.critical(msg)
+        return False, msg
+
+
+def validate_config_file(config_path):
+    """Run `haproxy -c` against config_path.
+
+    Returns (status, message) with status one of:
+      'valid'       - HAProxy parsed the file successfully
+      'invalid'     - HAProxy rejected it (message carries stderr)
+      'unavailable' - the validator could not be run at all (binary missing,
+                      timeout, ...). Deliberately distinct from 'invalid':
+                      it tells us nothing about the config.
+    """
+    try:
+        result = subprocess.run(['haproxy', '-c', '-f', config_path],
                                 capture_output=True, text=True)
+    except Exception as e:
+        return 'unavailable', f"Error validating HAProxy config: {e}"
     if result.returncode == 0:
+        return 'valid', None
+    return 'invalid', f"HAProxy configuration validation failed: {result.stderr}"
+
+
+def validate_haproxy_config():
+    """Validate the live HAProxy configuration file. Returns (is_valid, error)."""
+    status, message = validate_config_file(HAPROXY_CONFIG_PATH)
+    if status == 'valid':
         logger.info("HAProxy configuration validation passed")
         return True, None
-        else:
+    logger.error(message)
-            error_msg = f"HAProxy configuration validation failed: {result.stderr}"
+    return False, message
-            logger.error(error_msg)
-            return False, error_msg
-    except Exception as e:
-        error_msg = f"Error validating HAProxy config: {e}"
-        logger.error(error_msg)
-        return False, error_msg
 
-def reload_haproxy_safely():
+def reload_haproxy_safely(backup_status=None):
-    """Safely reload HAProxy with validation and rollback"""
+    """Safely reload HAProxy with validation and rollback.
+
+    PRECONDITION: the caller must already have called create_backup() BEFORE
+    writing the new config, and pass the status it returned. This function runs
+    after the new config is on disk, so it cannot take a meaningful backup
+    itself — doing so is exactly the bug this contract exists to prevent.
+
+    backup_status=None means the caller did not take a pre-write backup. We do
+    NOT create one here (that would overwrite a genuinely good backup with the
+    unverified new config); we log it and fall back to whatever backup already
+    exists on disk.
+    """
     try:
-        # Create backup before changes
+        if backup_status is None:
-        if not create_backup():
+            logger.error(
-            return False, "Failed to create backup"
+                "reload_haproxy_safely() called without a pre-write backup "
+                "status - rollback will fall back to whatever backup already "
+                "exists on disk. Callers must call create_backup() BEFORE "
+                "writing the new configuration."
+            )
+        elif backup_status not in _ROLLBACK_AVAILABLE_STATUSES:
+            logger.warning(
+                f"Proceeding with reload without a rollback target "
+                f"(backup status: {backup_status})"
+            )
 
         # Validate new configuration
         is_valid, error_msg = validate_haproxy_config()
         if not is_valid:
             # Restore backup on validation failure
-            restore_backup()
+            restored, restore_msg = restore_backup()
+            if not restored:
+                logger.critical(
+                    "Config validation failed AND rollback was not possible - "
+                    f"{HAPROXY_CONFIG_PATH} holds an invalid configuration that "
+                    "HAProxy will refuse to start with"
+                )
+                return False, (f"Config validation failed: {error_msg} | "
+                               f"ROLLBACK FAILED: {restore_msg}")
             return False, f"Config validation failed: {error_msg}"
 
         # Attempt reload
@@ -2010,20 +2355,28 @@ def reload_haproxy_safely():
                 
                 if reload_result.returncode == 0:
                     logger.info("HAProxy reloaded successfully")
+                    # Now - and only now - is this config known good.
+                    promote_current_config_to_backup()
                     return True, "HAProxy reloaded successfully"
                 else:
                     # Reload failed, restore backup
-                    restore_backup()
+                    restored, restore_msg = restore_backup()
-                    # Try to reload with backup config
+                    if restored:
-                    subprocess.run('echo "reload" | socat stdio /tmp/haproxy-cli', 
+                        # Try to reload with the restored (known-good) config
+                        subprocess.run(
+                            'echo "reload" | socat stdio /tmp/haproxy-cli',
                             shell=True, capture_output=True)
                     error_msg = f"HAProxy reload failed: {reload_result.stderr}"
+                    if not restored:
+                        error_msg += f" | ROLLBACK FAILED: {restore_msg}"
                     logger.error(error_msg)
                     return False, error_msg
             except Exception as e:
                 # Critical error during reload, restore backup
-                restore_backup()
+                restored, restore_msg = restore_backup()
                 error_msg = f"Critical error during reload: {e}"
+                if not restored:
+                    error_msg += f" | ROLLBACK FAILED: {restore_msg}"
                 logger.error(error_msg)
                 return False, error_msg
         else:
@@ -2034,11 +2387,15 @@ def reload_haproxy_safely():
                     check=True, capture_output=True, text=True
                 )
                 logger.info("HAProxy started successfully")
+                # Now - and only now - is this config known good.
+                promote_current_config_to_backup()
                 return True, "HAProxy started successfully"
             except subprocess.CalledProcessError as e:
                 # Start failed, restore backup
-                restore_backup()
+                restored, restore_msg = restore_backup()
                 error_msg = f"Failed to start HAProxy: {e.stderr}"
+                if not restored:
+                    error_msg += f" | ROLLBACK FAILED: {restore_msg}"
                 logger.error(error_msg)
                 return False, error_msg
     except Exception as e:

scripts/ensure_haproxy.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python3
+"""Idempotent haproxy liveness check — driven by the in-container supervisor loop.
+
+Why this exists
+---------------
+haproxy runs as a *background child of PID 1* (gunicorn) — it is started once at
+container init (scripts/init.py -> do_initial_setup -> start_haproxy) and then
+left running. Nothing supervises it after that. If the haproxy master process
+dies mid-life (SIGABRT -> exit 134, segfault, or an OOM of the haproxy master),
+the container stays "up" because gunicorn is still PID 1, so Docker's
+`--restart` policy never fires. haproxy then stays down until the *external*
+host watchdog (haproxy-watchdog.sh) notices port 80 is dead for ~3 minutes and
+does a full `docker restart` — which drops every in-flight connection.
+
+This script closes that gap: called on a short interval by the supervisor loop
+in start-up.sh, it re-launches haproxy *in place* within one interval.
+
+Safety
+------
+start_haproxy() is guarded by `is_process_running('haproxy')` (psutil-based, so
+it works in this container which has no `ps`), so calling this while haproxy is
+healthy is a cheap no-op. It only ever acts when haproxy is genuinely gone.
+"""
+import sys
+
+sys.path.insert(0, '/haproxy')
+import haproxy_manager  # noqa: E402  (sys.path manipulation must come first)
+
+
+def main():
+    if haproxy_manager.is_process_running('haproxy'):
+        return 0
+
+    haproxy_manager.logger.warning(
+        "[haproxy-supervisor] haproxy process not found — attempting in-place restart"
+    )
+    # start_haproxy() validates the config (and regenerates it if invalid)
+    # before launching, and swallows its own errors, so it will not raise here.
+    haproxy_manager.start_haproxy()
+
+    if haproxy_manager.is_process_running('haproxy'):
+        haproxy_manager.logger.info("[haproxy-supervisor] haproxy restarted in place")
+        return 0
+
+    haproxy_manager.logger.error(
+        "[haproxy-supervisor] haproxy restart FAILED — still not running after start_haproxy()"
+    )
+    return 1
+
+
+if __name__ == '__main__':
+    sys.exit(main())

scripts/manage-blocked-ips.sh

@@ -6,22 +6,40 @@
 SOCKET="/tmp/haproxy-cli"
 MAP_FILE="/etc/haproxy/blocked_ips.map"
 
+# HAProxy runs in master-worker mode here, and /tmp/haproxy-cli is the MASTER
+# socket. Data-plane commands (map/table manipulation) are NOT accepted on the
+# master socket — they must be routed to a worker with the "@<n>" prefix. "@1"
+# targets the current active worker. (A bare "add map ..." on the master socket
+# fails with "Unknown command: 'add'".)
+cli() { printf '@1 %s\n' "$*" | socat stdio "$SOCKET"; }
+
+# Map lookup in haproxy.cfg is `map_ip(...,0) -m int gt 0`, so each entry MUST be
+# "<ip_or_cidr> 1" — a bare IP yields an empty value (0) and is NOT blocked once
+# the map file is re-read on reload. The runtime map and the file must agree.
+MAP_VALUE=1
+
 # Ensure map file exists
 if [ ! -f "$MAP_FILE" ]; then
-    touch "$MAP_FILE"
+    echo "# Blocked IPs - Format: <ip_or_cidr> 1 (one per line)" > "$MAP_FILE"
-    echo "# Blocked IPs - Format: IP_ADDRESS" > "$MAP_FILE"
 fi
 
+# Escape regex metacharacters (notably dots) in an IP/CIDR for anchored matching.
+esc_re() { printf '%s' "$1" | sed 's/[.[\*^$/]/\\&/g'; }
+
 case "$1" in
     block)
         if [ -z "$2" ]; then
             echo "Usage: $0 block IP_ADDRESS"
             exit 1
         fi
-        # Add IP to map file
+        re="$(esc_re "$2")"
-        grep -q "^$2" "$MAP_FILE" || echo "$2" >> "$MAP_FILE"
+        # Persist (idempotent, anchored so 1.2.3.4 doesn't match 1.2.3.45),
-        # Add to runtime map
+        # always with the trailing value so the block survives a reload.
-        echo "add map /etc/haproxy/blocked_ips.map $2 1" | socat stdio "$SOCKET"
+        if ! grep -qE "^${re}([[:space:]]|$)" "$MAP_FILE"; then
+            echo "$2 $MAP_VALUE" >> "$MAP_FILE"
+        fi
+        # Apply at runtime immediately (no reload).
+        cli "add map $MAP_FILE $2 $MAP_VALUE"
         echo "Blocked IP: $2"
         ;;
 
@@ -30,31 +48,33 @@ case "$1" in
             echo "Usage: $0 unblock IP_ADDRESS"
             exit 1
         fi
-        # Remove from map file
+        re="$(esc_re "$2")"
-        sed -i "/^$2$/d" "$MAP_FILE"
+        # Remove from map file (match "<ip>" optionally followed by a value).
-        # Remove from runtime map
+        sed -i -E "/^${re}([[:space:]]|$)/d" "$MAP_FILE"
-        echo "del map /etc/haproxy/blocked_ips.map $2" | socat stdio "$SOCKET"
+        # Remove from runtime map.
+        cli "del map $MAP_FILE $2"
         echo "Unblocked IP: $2"
         ;;
 
     list)
         echo "Currently blocked IPs:"
-        echo "show map /etc/haproxy/blocked_ips.map" | socat stdio "$SOCKET" | awk '{print $1}'
+        # `show map` output is "<ptr> <key> <value>" — the IP is field 2.
+        cli "show map $MAP_FILE" | awk 'NF>=2 {print $2}'
         ;;
 
     clear)
         echo "Clearing all blocked IPs..."
-        echo "clear map /etc/haproxy/blocked_ips.map" | socat stdio "$SOCKET"
+        cli "clear map $MAP_FILE"
-        echo "# Blocked IPs - Format: IP_ADDRESS" > "$MAP_FILE"
+        echo "# Blocked IPs - Format: <ip_or_cidr> 1 (one per line)" > "$MAP_FILE"
         echo "All IPs unblocked"
         ;;
 
     stats)
         echo "=== HAProxy 3.0.11 Threat Intelligence Dashboard ==="
-        echo "show table web" | socat stdio "$SOCKET" | awk 'NR<=21'
+        cli "show table web" | awk 'NR<=21'
         echo ""
         echo "=== Top Threat Scores ==="
-        echo "show table web" | socat stdio "$SOCKET" | awk '
+        cli "show table web" | awk '
         NR>1 {
             ip = $1
             auth_fail = 0
@@ -84,7 +104,7 @@ case "$1" in
             exit 1
         fi
         # Add to manual blacklist using GPC(13)
-        echo "set table web key $2 data.gpc(13) 1" | socat stdio "$SOCKET"
+        cli "set table web key $2 data.gpc(13) 1"
         echo "Manually blacklisted IP: $2 (GPC(13) = 1)"
         ;;
 
@@ -94,7 +114,7 @@ case "$1" in
             exit 1
         fi
         # Clear manual blacklist flag
-        echo "set table web key $2 data.gpc(13) 0" | socat stdio "$SOCKET"
+        cli "set table web key $2 data.gpc(13) 0"
         echo "Removed manual blacklist for IP: $2"
         ;;
 
@@ -104,7 +124,7 @@ case "$1" in
             exit 1
         fi
         # Add to auto-blacklist using GPC(14)
-        echo "set table web key $2 data.gpc(14) 1" | socat stdio "$SOCKET"
+        cli "set table web key $2 data.gpc(14) 1"
         echo "Auto-blacklisted IP: $2 (GPC(14) = 1)"
         ;;
 
@@ -115,14 +135,14 @@ case "$1" in
         fi
         # Show detailed threat breakdown for specific IP
         echo "Threat analysis for $2:"
-        echo "show table web key $2" | socat stdio "$SOCKET"
+        cli "show table web key $2"
         ;;
 
     *)
         echo "Usage: $0 {block|unblock|list|clear|blacklist|unblacklist|auto-blacklist|threat-score|stats} [IP_ADDRESS]"
         echo ""
         echo "HAProxy 3.0.11 Enhanced Security Commands:"
-        echo "  block IP         - Block IP via map file (immediate)"
+        echo "  block IP         - Block IP via map file (immediate + persisted)"
         echo "  unblock IP       - Unblock IP from map file"
         echo "  blacklist IP     - Manual blacklist via GPC(13) array"
         echo "  unblacklist IP   - Remove manual blacklist flag"

scripts/start-up.sh

@@ -27,11 +27,33 @@ cron &
 # Phase 1: container init
 python /haproxy/scripts/init.py
 
+# Phase 1.5: in-container haproxy supervisor.
+# haproxy runs as a background child of PID 1 (gunicorn) with NOTHING watching
+# it after init. If the haproxy master dies mid-life (e.g. SIGABRT -> exit 134,
+# segfault), the container stays "up" (gunicorn is PID 1), Docker's --restart
+# policy never fires, and haproxy is down until the external host watchdog
+# full-restarts the whole container minutes later (dropping every connection).
+# This loop revives haproxy in place within one interval. ensure_haproxy.py is
+# idempotent — a cheap no-op whenever haproxy is already running.
+HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
+(
+    while true; do
+        sleep "${HAPROXY_SUPERVISOR_INTERVAL}"
+        python /haproxy/scripts/ensure_haproxy.py 2>&1 || true
+    done
+) &
+
 # Phase 2: WSGI servers
 # Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
 # (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
 # 1000 — worker recycle frequency).
-API_WORKERS="${HAPROXY_MGR_API_WORKERS:-1}"
+#
+# API_WORKERS default is 2 (was 1). A single worker is a single point of
+# failure: if its gthread pool ever wedges (see the 2026-07-07 subprocess-hang
+# incident — now bounded by DEFAULT_SUBPROCESS_TIMEOUT in haproxy_manager.py),
+# the entire management API goes dark. A second worker keeps the API answering
+# (config regenerate, health, SSL) while the other recycles via --max-requests.
+API_WORKERS="${HAPROXY_MGR_API_WORKERS:-2}"
 API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
 MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
 MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"

scripts/test-config-rollback.py

@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+"""Regression tests for HAProxy config backup / rollback ordering.
+
+Why this file exists
+--------------------
+generate_config() used to write the new haproxy.cfg and only THEN call
+reload_haproxy_safely() -> create_backup(), so the "backup" was a copy of the
+config that had just been written. On a validation failure restore_backup()
+restored the identical broken bytes: the advertised rollback was a no-op and a
+fatal haproxy.cfg stayed on disk, where start_haproxy() refuses to launch.
+
+These tests pin the ordering invariant (backup predates the write) and the
+observable end-to-end behaviour (after a failed validation the file on disk is
+the previous working config and HAProxy will start with it).
+
+Running
+-------
+    python3 scripts/test-config-rollback.py           # tests the repo checkout
+    HAPROXY_MANAGER_DIR=/some/other/tree \
+        python3 scripts/test-config-rollback.py       # tests another tree
+
+The repo has no Python test framework (scripts/test-*.sh are curl-based
+integration scripts against a running API), so this is a self-contained
+stdlib-unittest script - no pytest, no venv, no new dependencies beyond the
+application's own requirements.txt (Flask/Jinja2/psutil), which are already
+present in the container image.
+
+No HAProxy binary is required: a stub `haproxy` is put on PATH that mimics
+`haproxy -c -f <file>` by rejecting any config containing the token
+__BROKEN__, which is how the tests inject an invalid configuration.
+"""
+
+import os
+import sys
+import shutil
+import sqlite3
+import logging
+import tempfile
+import textwrap
+import unittest
+
+BROKEN_TOKEN = '__BROKEN__'
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+
+# haproxy_manager builds its Jinja2 environment from the relative path
+# Path('templates'), so it has to be imported with the module dir as cwd.
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+# The module opens /var/log/haproxy-manager.log at import time via
+# logging.FileHandler. Redirect that one call so the suite runs unprivileged.
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda fn, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(fn)), *a, **kw)
+)
+try:
+    import haproxy_manager as hm
+except ImportError as exc:  # pragma: no cover - environment problem, not a failure
+    sys.stderr.write(
+        f"SKIP: cannot import haproxy_manager ({exc}).\n"
+        "Install the application requirements first: pip install -r requirements.txt\n"
+    )
+    raise SystemExit(77)
+finally:
+    logging.FileHandler = _real_file_handler
+
+logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
+
+FAKE_HAPROXY = textwrap.dedent(f"""\
+    #!/bin/sh
+    # Test stub for the haproxy binary.
+    #   haproxy -c -f FILE   -> exit 1 if FILE contains {BROKEN_TOKEN}, else 0
+    #   haproxy -W -S ... -f FILE (start) -> same validation, then exit 0
+    cfg=""
+    while [ $# -gt 0 ]; do
+      case "$1" in -f) cfg="$2"; shift ;; esac
+      shift
+    done
+    if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then
+      echo "[ALERT] parsing [$cfg:1] : unknown keyword '{BROKEN_TOKEN}'" >&2
+      exit 1
+    fi
+    exit 0
+""")
+
+
+class RollbackTestCase(unittest.TestCase):
+    """Base fixture: an isolated fake /etc/haproxy plus a stub haproxy binary."""
+
+    def setUp(self):
+        self.tmp = tempfile.mkdtemp(prefix='haproxy-rollback-test-')
+        self.addCleanup(shutil.rmtree, self.tmp, True)
+
+        bindir = os.path.join(self.tmp, 'bin')
+        os.makedirs(bindir)
+        stub = os.path.join(bindir, 'haproxy')
+        with open(stub, 'w') as fh:
+            fh.write(FAKE_HAPROXY)
+        os.chmod(stub, 0o755)
+        self._old_path = os.environ['PATH']
+        os.environ['PATH'] = bindir + os.pathsep + self._old_path
+        self.addCleanup(lambda: os.environ.__setitem__('PATH', self._old_path))
+
+        self.etc = os.path.join(self.tmp, 'etc')
+        os.makedirs(self.etc)
+
+        overrides = {
+            'DB_FILE': os.path.join(self.etc, 'haproxy_config.db'),
+            'HAPROXY_CONFIG_PATH': os.path.join(self.etc, 'haproxy.cfg'),
+            'HAPROXY_BACKUP_PATH': os.path.join(self.etc, 'haproxy.cfg.backup'),
+            'BLOCKED_IPS_MAP_PATH': os.path.join(self.etc, 'blocked_ips.map'),
+            'BLOCKED_IPS_MAP_BACKUP_PATH': os.path.join(self.etc, 'blocked_ips.map.backup'),
+            'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'),
+            'SSL_CERTS_DIR': os.path.join(self.etc, 'certs'),
+            'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'),
+            # Added by the rollback fix; older trees do not have it.
+            'CORAZA_SPOE_CONFIG_PATH': os.path.join(self.etc, 'coraza-spoe.cfg'),
+            'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'),
+        }
+        self._saved = {}
+        for name, value in overrides.items():
+            self._saved[name] = getattr(hm, name, None)
+            setattr(hm, name, value)
+        self.addCleanup(self._restore_globals)
+        os.makedirs(hm.SSL_CERTS_DIR)
+
+        # log_operation() appends to a hardcoded /var/log path. Injecting `open`
+        # into the module namespace shadows the builtin for that module only
+        # (module globals are searched before builtins), so the real
+        # log_operation code still runs.
+        real_open = open
+        log_dir = self.tmp
+
+        def _redirecting_open(path, *args, **kwargs):
+            if isinstance(path, str) and path.startswith('/var/log/'):
+                path = os.path.join(log_dir, os.path.basename(path))
+            return real_open(path, *args, **kwargs)
+
+        hm.open = _redirecting_open
+        self.addCleanup(lambda: hm.__dict__.pop('open', None))
+
+        hm.init_db()
+
+    def _restore_globals(self):
+        for name, value in self._saved.items():
+            if value is None:
+                hm.__dict__.pop(name, None)
+            else:
+                setattr(hm, name, value)
+
+    # -- helpers ---------------------------------------------------------
+    def add_domain(self, domain, backend_name, address='10.0.0.1'):
+        with sqlite3.connect(hm.DB_FILE) as conn:
+            cur = conn.cursor()
+            cur.execute('INSERT INTO domains (domain, ssl_enabled) VALUES (?, 0)',
+                        (domain,))
+            domain_id = cur.lastrowid
+            cur.execute('INSERT INTO backends (name, domain_id) VALUES (?, ?)',
+                        (backend_name, domain_id))
+            backend_id = cur.lastrowid
+            cur.execute(
+                'INSERT INTO backend_servers '
+                '(backend_id, server_name, server_address, server_port) '
+                'VALUES (?, ?, ?, ?)',
+                (backend_id, 'srv1', address, 8080))
+            conn.commit()
+
+    def block_ip(self, ip):
+        with sqlite3.connect(hm.DB_FILE) as conn:
+            conn.execute('INSERT INTO blocked_ips (ip_address, reason) VALUES (?, ?)',
+                         (ip, 'test'))
+            conn.commit()
+
+    def read(self, path):
+        with open(path) as fh:
+            return fh.read()
+
+    def config_is_loadable(self):
+        """True if HAProxy would accept the config currently on disk."""
+        import subprocess
+        return subprocess.run(
+            ['haproxy', '-c', '-f', hm.HAPROXY_CONFIG_PATH],
+            capture_output=True).returncode == 0
+
+    def generate_good_config(self):
+        self.add_domain('good.example.com', 'good_backend')
+        hm.generate_config()
+        self.assertTrue(self.config_is_loadable(),
+                        'fixture precondition: first generated config must be valid')
+        return self.read(hm.HAPROXY_CONFIG_PATH)
+
+    def break_the_config(self):
+        """Queue a domain whose rendered backend the validator rejects."""
+        self.add_domain('bad.example.com', BROKEN_TOKEN + '_backend', '10.0.0.2')
+
+
+class TestBackupOrdering(RollbackTestCase):
+
+    def test_backup_is_taken_before_the_new_config_is_written(self):
+        """The ordering invariant, asserted directly.
+
+        Whatever create_backup() sees on disk must be the OLD config; if the
+        write happens first the backup is a copy of the new config and rollback
+        is meaningless.
+        """
+        good = self.generate_good_config()
+
+        seen = {}
+        real_create_backup = hm.create_backup
+
+        def spy(*args, **kwargs):
+            seen['config_on_disk'] = self.read(hm.HAPROXY_CONFIG_PATH)
+            return real_create_backup(*args, **kwargs)
+
+        hm.create_backup = spy
+        self.addCleanup(setattr, hm, 'create_backup', real_create_backup)
+
+        self.add_domain('second.example.com', 'second_backend', '10.0.0.3')
+        hm.generate_config()
+
+        self.assertIn('config_on_disk', seen,
+                      'create_backup() was never called during generate_config()')
+        self.assertEqual(
+            seen['config_on_disk'], good,
+            'create_backup() ran AFTER the new config was written - the backup '
+            'is a copy of the new config, so rollback cannot undo anything')
+
+    def test_backup_tracks_the_last_known_good_config(self):
+        """After a change that validated AND loaded, the backup is that config.
+
+        The rollback target is "the last configuration HAProxy actually ran",
+        not "the file that happened to be there last time".
+        """
+        good = self.generate_good_config()
+        self.add_domain('second.example.com', 'second_backend', '10.0.0.3')
+        hm.generate_config()
+
+        live = self.read(hm.HAPROXY_CONFIG_PATH)
+        self.assertNotEqual(live, good, 'fixture sanity: the new config should differ')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), live,
+                         'the successful config was not recorded as known-good')
+
+    def test_backup_is_not_promoted_when_the_change_fails(self):
+        """A config that never loaded must not become the rollback target."""
+        good = self.generate_good_config()
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'a config that failed validation was promoted to backup')
+
+
+class TestRollbackEndToEnd(RollbackTestCase):
+
+    def test_failed_validation_leaves_the_last_good_config_on_disk(self):
+        good = self.generate_good_config()
+
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        on_disk = self.read(hm.HAPROXY_CONFIG_PATH)
+        self.assertNotIn(BROKEN_TOKEN, on_disk,
+                         'the rejected config is still on disk - rollback was a no-op')
+        self.assertEqual(on_disk, good,
+                         'on-disk config is not byte-identical to the last good one')
+
+    def test_haproxy_would_still_start_after_a_failed_change(self):
+        """The operational consequence: the edge can still come up."""
+        self.generate_good_config()
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        self.assertTrue(self.config_is_loadable(),
+                        'HAProxy would refuse to start with the config left on disk')
+        with self.assertLogs('haproxy_manager', level='INFO') as captured:
+            hm.start_haproxy()
+        self.assertTrue(
+            any('HAProxy started successfully' in line for line in captured.output),
+            f'start_haproxy() did not succeed after rollback: {captured.output}')
+
+    def test_blocked_ips_map_is_rolled_back_too(self):
+        """generate_config() rewrites the map file before writing haproxy.cfg."""
+        self.block_ip('192.0.2.10')
+        self.generate_good_config()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        self.block_ip('198.51.100.20')
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        self.assertEqual(self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+                         'blocked IPs map was not rolled back with the config')
+
+    def test_first_run_failure_reports_that_rollback_was_impossible(self):
+        """No prior config: there is nothing to restore, and that must be said.
+
+        A missing backup must never be reported as a successful restore, and it
+        must never be turned into "restore an empty file".
+        """
+        self.break_the_config()
+        with self.assertRaises(Exception) as ctx:
+            hm.generate_config()
+
+        self.assertIn('ROLLBACK FAILED', str(ctx.exception),
+                      'a failed change with no backup was not reported as such')
+        self.assertFalse(os.path.exists(hm.HAPROXY_BACKUP_PATH),
+                         'a backup was fabricated from the broken config')
+        # The broken config is deliberately left in place: start_haproxy() can
+        # then detect it and try to regenerate. It must not be blanked.
+        self.assertGreater(os.path.getsize(hm.HAPROXY_CONFIG_PATH), 0,
+                           'config file was emptied instead of left for diagnosis')
+
+
+class TestBackupPrimitives(RollbackTestCase):
+
+    def test_restore_backup_distinguishes_missing_backup_from_success(self):
+        restored, message = hm.restore_backup()
+        self.assertFalse(restored,
+                         'restore_backup() reported success with no backup present')
+        self.assertIn('cannot roll back', message.lower())
+
+        good = self.generate_good_config()
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write('scribbled over\n')
+
+        restored, message = hm.restore_backup()
+        self.assertTrue(restored, message)
+        self.assertEqual(self.read(hm.HAPROXY_CONFIG_PATH), good)
+
+    def test_a_successful_generation_records_a_rollback_target(self):
+        """Even the first-ever generation must leave something to roll back to."""
+        good = self.generate_good_config()
+        self.assertTrue(
+            os.path.exists(hm.HAPROXY_BACKUP_PATH),
+            'after a successful reload there is still no known-good backup')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
+
+    def test_a_broken_current_config_does_not_replace_a_good_backup(self):
+        """The known-good marker.
+
+        If the config already on disk is broken (previous failed write, manual
+        edit), snapshotting it would make "rollback" mean "restore a different
+        broken config". The older validated backup must survive.
+        """
+        good = self.generate_good_config()
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'fixture: a good backup should exist by now')
+
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write(f'garbage {BROKEN_TOKEN} config\n')
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'kept_previous')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'a broken config overwrote the known-good backup')
+
+    def test_reload_does_not_take_its_own_backup(self):
+        """reload_haproxy_safely() runs after the write, so it must not back up."""
+        good = self.generate_good_config()
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write(f'broken {BROKEN_TOKEN}\n')
+
+        success, message = hm.reload_haproxy_safely(backup_status='created')
+
+        self.assertFalse(success)
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'reload_haproxy_safely() overwrote the good backup')
+        self.assertEqual(self.read(hm.HAPROXY_CONFIG_PATH), good,
+                         'reload_haproxy_safely() did not roll the config back')
+
+    def test_unchanged_config_is_not_revalidated(self):
+        """Fast path: if the backup already is the live config, do no work.
+
+        generate_config() runs inside customer-facing API calls and
+        `haproxy -c` is expensive on an edge with hundreds of certificates.
+        """
+        self.generate_good_config()
+
+        calls = []
+        real_validate = hm.validate_config_file
+        hm.validate_config_file = lambda path: (calls.append(path),
+                                                real_validate(path))[1]
+        self.addCleanup(setattr, hm, 'validate_config_file', real_validate)
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'created')
+        self.assertEqual(calls, [],
+                         'the unchanged live config was re-validated needlessly')
+
+    def test_fast_path_does_not_hide_a_drifted_broken_config(self):
+        """If the live config drifted from the backup, the gate must still run."""
+        good = self.generate_good_config()
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write(f'hand edited {BROKEN_TOKEN}\n')
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'kept_previous',
+                         'a drifted broken config was silently accepted')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
+
+    def test_backup_set_covers_every_file_generate_config_writes(self):
+        pairs = dict(hm._config_backup_pairs())
+        for path in (hm.HAPROXY_CONFIG_PATH, hm.BLOCKED_IPS_MAP_PATH,
+                     hm.CORAZA_SPOE_CONFIG_PATH):
+            self.assertIn(path, pairs,
+                          f'{path} is written by generate_config() but is not '
+                          'part of the backed-up config set')
+
+    def test_coraza_spoe_config_round_trips(self):
+        self.generate_good_config()
+        with open(hm.CORAZA_SPOE_CONFIG_PATH, 'w') as fh:
+            fh.write('spoe-good\n')
+        hm.create_backup()
+        with open(hm.CORAZA_SPOE_CONFIG_PATH, 'w') as fh:
+            fh.write('spoe-broken\n')
+        restored, message = hm.restore_backup()
+        self.assertTrue(restored, message)
+        self.assertEqual(self.read(hm.CORAZA_SPOE_CONFIG_PATH), 'spoe-good\n')
+
+
+class TestAtomicWrite(RollbackTestCase):
+
+    def test_write_is_atomic_and_preserves_mode(self):
+        path = os.path.join(self.etc, 'atomic.cfg')
+        with open(path, 'w') as fh:
+            fh.write('old')
+        os.chmod(path, 0o644)
+
+        hm.write_config_atomically(path, 'new content\n')
+
+        self.assertEqual(self.read(path), 'new content\n')
+        self.assertEqual(oct(os.stat(path).st_mode & 0o777), oct(0o644))
+        leftovers = [n for n in os.listdir(self.etc) if n.endswith('.tmp')]
+        self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
+
+    def test_failed_write_leaves_the_previous_file_intact(self):
+        path = os.path.join(self.etc, 'atomic.cfg')
+        with open(path, 'w') as fh:
+            fh.write('old content\n')
+
+        # Anything that makes f.write() blow up mid-flight stands in for a full
+        # disk / killed container.
+        with self.assertRaises(Exception):
+            hm.write_config_atomically(path, object())
+
+        self.assertEqual(self.read(path), 'old content\n',
+                         'a failed write clobbered the previous config')
+        leftovers = [n for n in os.listdir(self.etc) if n.endswith('.tmp')]
+        self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
+
+
+if __name__ == '__main__':
+    print(f"testing haproxy_manager from: {MODULE_DIR}")
+    unittest.main(verbosity=2)

templates/hap_backend_longlived.tpl

@@ -0,0 +1,41 @@
+# Long-lived backend for {{ name }} (template_override='hap_backend_longlived').
+# Use for apps whose PRIMARY traffic holds connections open: media streaming,
+# large up/downloads, or persistent viewer/streaming sessions. Both the primary
+# and the SSE backend are tuned long-lived here (no http-server-close,
+# http-no-delay, 6h server/tunnel/keep-alive timeouts).
+#
+# Compare hap_backend_websocket.tpl, which keeps the PRIMARY backend standard
+# and only makes the -sse-backend long-lived. Pick this one when the main path
+# itself needs long-lived connections, not just an SSE side-channel.
+backend {{ name }}-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}
+
+# SSE variant (Accept: text/event-stream / ?action=stream auto-routes here)
+backend {{ name }}-sse-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}

templates/hap_backend_websocket.tpl

@@ -0,0 +1,34 @@
+# Long-lived / websocket-safe backend for {{ name }} (template_override)
+# For apps with persistent WebSocket/streaming connections (e.g. Jitsi /xmpp-websocket, /colibri-ws).
+backend {{ name }}-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}
+
+# SSE variant (Accept: text/event-stream / ?action=stream auto-routes here)
+backend {{ name }}-sse-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}

templates/hap_coraza_spoe_engine.tpl

@@ -44,8 +44,14 @@ spoe-agent coraza
 # Arg order/names are mandatory: Coraza-SPOA parses positionally and renames
 # break the agent. `app=str(haproxy)` is the literal application name from
 # coraza-spoa's config.yaml `applications:` block.
+#
+# src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
+# top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
+# only honoring those headers from trusted proxies). Falls back to `src`
+# when no proxy headers are present. This is what shows up as `client_ip`
+# in /var/log/coraza/audit.log and the rule manager's view-matches panel.
 spoe-message coraza-req
-    args app=str(haproxy) src-ip=src src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
+    args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
 
 # Group binding for send-spoe-group invocation in the frontend. One group,
 # one message; could add more in the future (e.g. coraza-res for response

templates/hap_header.tpl

@@ -27,6 +27,23 @@ global
     # SSL and Performance
     tune.ssl.default-dh-param 2048
 
+    # HTTP/3 over QUIC. The Debian haproxy package is built against system
+    # OpenSSL via the compatibility shim (USE_QUIC_OPENSSL_COMPAT), which is
+    # not a native QUIC TLS stack. HAProxy therefore rejects `quic*@` binds
+    # unless this opt-in is set. `limited-quic` enables QUIC through the compat
+    # layer (no 0-RTT — that needs quictls/aws-lc or native OpenSSL 3.5 QUIC).
+    # Without this, the quic bind in the frontend fails to start: "this SSL
+    # library does not support the QUIC protocol".
+    limited-quic
+{%- if cluster_secret %}
+
+    # Stable secret keying QUIC Retry/address-validation tokens. Self-healed
+    # to /etc/haproxy/cluster-secret (named volume) by the manager so it
+    # survives recreates; without it haproxy picks a random one per process
+    # and tokens don't survive reloads (benign, just a startup notice).
+    cluster-secret "{{ cluster_secret }}"
+{%- endif %}
+
     # HTTP/2 protection against Rapid Reset (CVE-2023-44487) and stream abuse
     tune.h2.fe.max-total-streams 2000
     tune.h2.fe.glitches-threshold 50
@@ -74,3 +91,14 @@ defaults
     timeout tarpit          10s  # Tarpit delay for low-level scanners (before silent-drop)
     maxconn                 3000
 
+    # Per-request unique reference, used:
+    #   - in the log line (httplog includes %ID)
+    #   - echoed to clients in the X-Request-Reference response header on
+    #     WAF blocks so a customer can quote it when opening a support ticket
+    #   - embedded in /etc/haproxy/errors/403-waf.html so a blocked visitor
+    #     sees it on the rendered 403 page
+    # Support correlates ref → /var/log/haproxy.log line → timestamp+client+host
+    # → /var/log/coraza/audit.log entry → rule_id.
+    unique-id-format        %[uuid()]
+    unique-id-header        X-Request-Reference
+    

templates/hap_listener.tpl

@@ -4,6 +4,21 @@ frontend web
     # crt can now be a path, so it will load all .pem files in the path
     bind 0.0.0.0:443 ssl crt {{ crt_path }} alpn h2,http/1.1
 
+    # HTTP/3 over QUIC (UDP/443). Same cert path as the TCP listener above.
+    # The Debian haproxy package is built +QUIC (QUIC_OPENSSL_COMPAT), so this
+    # is config-only — no source build. Requires UDP/443 published on the
+    # container (`-p 443:443/udp`) and open at the host firewall. `h3` is the
+    # only ALPN QUIC negotiates; h2/http1 stay on the TCP bind above. Sharing
+    # the frontend means all the real-IP, rate-limit, IP-block and Coraza
+    # rules below apply identically to H3 traffic.
+    bind quic4@0.0.0.0:443 ssl crt {{ crt_path }} alpn h3
+
+    # Advertise H3 so browsers upgrade their existing TCP (h2) connection to
+    # QUIC on the next request. `ma` is how long (seconds) the client may
+    # cache the advertisement. http-after-response applies it to every
+    # response, including haproxy-generated ones (blocks, default page).
+    http-after-response set-header alt-svc "h3=\":443\"; ma=86400"
+
     # Capture Host header so it appears in httplog output (in %hr field)
     http-request capture req.hdr(Host) len 64
 
@@ -13,10 +28,14 @@ frontend web
     acl has_x_real_ip req.hdr(X-Real-IP) -m found
     acl has_x_forwarded_for req.hdr(X-Forwarded-For) -m found
 
-    # Set the real IP based on available headers
+    # Set the real IP based on available headers. Use hdr_ip (not hdr) so the
-    http-request set-var(txn.real_ip) req.hdr(CF-Connecting-IP) if has_cf_connecting_ip
+    # variable is typed as IP — required by the Coraza SPOE arg `src-ip` which
-    http-request set-var(txn.real_ip) req.hdr(X-Real-IP) if !has_cf_connecting_ip has_x_real_ip
+    # decodes binary IP bytes (passing a string IP panics the SPOA goroutine).
-    http-request set-var(txn.real_ip) req.hdr(X-Forwarded-For) if !has_cf_connecting_ip !has_x_real_ip has_x_forwarded_for
+    # `hdr_ip(X-Forwarded-For,1)` extracts the FIRST address from a possibly
+    # comma-separated chain (original client, not intermediate proxies).
+    http-request set-var(txn.real_ip) req.hdr_ip(CF-Connecting-IP) if has_cf_connecting_ip
+    http-request set-var(txn.real_ip) req.hdr_ip(X-Real-IP) if !has_cf_connecting_ip has_x_real_ip
+    http-request set-var(txn.real_ip) req.hdr_ip(X-Forwarded-For,1) if !has_cf_connecting_ip !has_x_real_ip has_x_forwarded_for
     http-request set-var(txn.real_ip) src if !has_cf_connecting_ip !has_x_real_ip !has_x_forwarded_for
 
     # --- Connection & rate tracking ---
@@ -45,6 +64,67 @@ frontend web
     # High error rate: >100 errors in 30s (scanner/fuzzer behavior)
     http-request tarpit deny_status 403 if { sc_http_err_rate(0) gt 100 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
 
+    # --- WordPress wp-login.php brute-force protection ---
+    # The generic limits above are deliberately high (media-heavy sites), so a
+    # slow credential-stuffing run (dozens of login POSTs/min) slips under them.
+    # Track POSTs to wp-login.php per real client IP in a DEDICATED 60s table
+    # (sc1 / backend wp_bruteforce, defined in hap_security_tables.tpl) and
+    # tarpit once an IP exceeds 30/min. Only login POSTs are counted — GETs of
+    # the login form, normal browsing, and the handful of POSTs a legit user
+    # makes are unaffected; an offending IP can still browse, just not keep
+    # hammering login. path_end also covers subdirectory WP installs. Honors the
+    # same whitelist (RFC1918 / trusted_ips.list / trusted_ips.map).
+    acl wp_login_path path_end /wp-login.php
+    http-request track-sc1 var(txn.real_ip) table wp_bruteforce if METH_POST wp_login_path
+    http-request tarpit deny_status 429 if METH_POST wp_login_path { sc_http_req_rate(1) gt 30 } !is_local !is_trusted_ip !is_whitelisted
+
+    # --- WordPress wp-login.php "must-load-the-form-first" cookie challenge ---
+    # Defeats DISTRIBUTED credential-stuffing (hundreds of thousands of unique
+    # IPs, each low-and-slow, so the per-IP rule above can't see them). Such
+    # bots POST straight to /wp-login.php without ever GETting the form — on
+    # these sites the login POST:GET ratio is ~15:1. We hand out a cookie when
+    # the form is actually fetched (GET) and require it on POST; direct-POST
+    # bots lack it and are denied AT THE EDGE before reaching PHP. Real logins
+    # are unaffected — WordPress login already requires loading the page and
+    # accepting cookies. Immediate deny (NOT tarpit) — under a 300k-POST flood,
+    # holding tarpit connections would exhaust HAProxy. Honors the whitelist.
+    # Mark login-form GETs at REQUEST time (method/path are reliably evaluable
+    # here; in the response phase they are not) so the cookie is emitted on the
+    # form's own response.
+    http-request set-var(txn.wp_login_form) int(1) if METH_GET wp_login_path
+    http-after-response add-header set-cookie "whplc=1; Path=/; Max-Age=1800; HttpOnly; Secure; SameSite=Lax" if { var(txn.wp_login_form) -m found }
+    acl has_login_cookie req.cook(whplc) -m found
+    http-request deny deny_status 403 if METH_POST wp_login_path !has_login_cookie !is_local !is_trusted_ip !is_whitelisted
+
+    # WordPress REST batch endpoint lockdown ("wp2shell": CVE-2026-63030 +
+    # CVE-2026-60137). Chaining a core SQL injection with REST batch-route
+    # confusion gives unauthenticated RCE on WP 6.9.0-6.9.4 and 7.0.0-7.0.1
+    # (fixed in 6.9.5 / 7.0.2). Exploits are public and were used against this
+    # fleet on 2026-07-19/20; one site was compromised via this path before
+    # patching. This is a virtual patch: it does not repair the vulnerable
+    # application logic, it only removes reachability, so it stays until every
+    # site is confirmed on a fixed release.
+    #
+    # Both routing forms must be covered -- a rule matching only the pretty
+    # permalink path leaves the ?rest_route= fallback wide open, and urlp()
+    # does not URL-decode, hence the third ACL for the %2F spelling.
+    #
+    # Anonymous-only. batch/v1 is used legitimately by the block editor for
+    # multi-entity saves, so a blanket deny would break wp-admin for real
+    # users; requiring a wordpress_logged_in_* cookie costs them nothing.
+    # req.cook() needs an exact name and WordPress suffixes a per-site hash,
+    # so this substring-matches the raw Cookie header instead.
+    #
+    # Immediate deny, not tarpit -- holding connections open helps an attacker
+    # who is already scripting this. Honors the same whitelist as above.
+    acl wp_batch_path path_beg /wp-json/batch/v1
+    acl wp_batch_route urlp(rest_route) -i -m beg /batch/v1
+    acl wp_batch_route_enc query -i -m sub rest_route=%2Fbatch%2Fv1
+    acl has_wp_logged_in req.hdr(Cookie) -i -m sub wordpress_logged_in_
+    http-request deny deny_status 403 if wp_batch_path !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
+    http-request deny deny_status 403 if wp_batch_route !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
+    http-request deny deny_status 403 if wp_batch_route_enc !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
+
     # IP blocking using map file (manual blocks only)
     # Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
     # Runtime updates: echo "add map #0 IP_ADDRESS 1" | socat stdio /var/run/haproxy.sock
@@ -81,8 +161,17 @@ frontend web
     # disruptive action fires (depends on SecRuleEngine mode + per-rule
     # ctl:ruleEngine overrides). Without these rules, Coraza would inspect
     # but never block.
-    http-request deny deny_status 403 hdr waf-block "request"  if { var(txn.coraza.action) -m str deny }
+    #
-    http-response deny deny_status 403 hdr waf-block "response" if { var(txn.coraza.action) -m str deny }
+    # On request-phase deny we return a rendered HTML page that surfaces the
+    # request reference (the unique-id) so a customer who's been blocked
+    # incorrectly can open a support ticket and quote it. lf-file expands
+    # log-format expressions inside the file at response time, so
+    # %[unique-id] / %[req.hdr(host)] / etc. get substituted live.
+    # Response-phase deny stays as a bare 403 — outbound blocks are rare in
+    # our config (Coraza response inspection is disabled by default) and
+    # an HTML body on a 403 generated mid-response could land mid-stream.
+    http-request return status 403 content-type "text/html; charset=utf-8" hdr waf-block "request" hdr x-request-reference "%[unique-id]" lf-file /haproxy/errors/403-waf.html if { var(txn.coraza.action) -m str deny }
+    http-response deny deny_status 403 hdr waf-block "response" hdr x-request-reference "%[unique-id]" if { var(txn.coraza.action) -m str deny }
     http-request silent-drop if { var(txn.coraza.action) -m str drop }
     http-response silent-drop if { var(txn.coraza.action) -m str drop }
     http-request redirect code 302 location %[var(txn.coraza.data)] if { var(txn.coraza.action) -m str redirect }

templates/hap_security_tables.tpl

@@ -6,3 +6,11 @@ frontend stats
     stats refresh 30s
     stats show-legends
     stats show-node
+
+# Dedicated stick-table for WordPress wp-login.php brute-force tracking.
+# Tracked via track-sc1 from the `web` frontend (hap_listener.tpl); counts only
+# login POSTs per real client IP over a 60s window. Separate from the generic
+# sc0 connection/rate table so the login-attempt threshold is independent of
+# the (much higher) flood thresholds.
+backend wp_bruteforce
+    stick-table type ip size 100k expire 30m store http_req_rate(60s)

trusted_ips.list

@@ -1 +1,9 @@
-172.116.197.166
+# Source-IP whitelist — exempt from HAProxy rate limits (one IP or CIDR per line).
+# Referenced by templates/hap_listener.tpl:
+#   acl is_trusted_ip src -f /etc/haproxy/trusted_ips.list
+#
+# Add trusted source IPs below. Do NOT commit real/personal IPs to this repo —
+# it is mirrored publicly. Keep real entries in an untracked local copy, or add
+# them directly on the server (the file lives in the /etc/haproxy named volume
+# and persists across container recreates).
+127.0.0.1

trusted_ips.map

@@ -1 +1,9 @@
-172.116.197.166 1
+# Real-IP whitelist for proxy-header matching — exempt from HAProxy rate limits.
+# Format: "<IP> 1" (one per line). Referenced by templates/hap_listener.tpl:
+#   acl is_whitelisted var(txn.real_ip),map_ip(/etc/haproxy/trusted_ips.map,0) -m int gt 0
+#
+# Add trusted real IPs below. Do NOT commit real/personal IPs to this repo —
+# it is mirrored publicly. Keep real entries in an untracked local copy, or add
+# them directly on the server (the file lives in the /etc/haproxy named volume
+# and persists across container recreates).
+127.0.0.1 1