Merge pull request 'fix(haproxy): set X-CLIENT-IP instead of adding it' (#11) from fix/x-client-ip-set-not-add into main

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.claude/skills/haproxy-manager-deploy/SKILL.md

@@ -0,0 +1,215 @@
+---
+name: haproxy-manager-deploy
+description: Use when shipping a haproxy-manager-base code change — editing templates, the Dockerfile, the Python manager, the coraza-spoa subdir, or static assets like errors/, then getting it onto whp01 or staging. Trigger eagerly on phrases like "deploy haproxy", "ship the haproxy change", "rebuild haproxy-manager", "update the WAF block page", "recreate haproxy-manager", or any time the next step would involve `git push` from this repo, `docker pull` on the image, or `container-manager.sh recreate`. Walks the Gitea-CI-auto-build + recreate flow, surfaces the named-volume shadowing foot-gun, and includes post-deploy verification.
+---
+
+# haproxy-manager-base commit / build / deploy
+
+This is procedural discipline for changes to `haproxy-manager-base`. The repository builds via Gitea Actions on push, not via a local build script (the WHP flow uses `build-release.sh`; this one doesn't — don't conflate them, see the `whp-deploy` skill in the whp repo for that one). Each step has caught a real foot-gun.
+
+## The pipeline at a glance
+
+```
+edit code (local)
+   └─> commit + push
+         └─> Gitea Actions auto-build (build-push.yaml / build-push-coraza.yaml)
+               ├─> publishes :latest tag to repo.anhonesthost.net
+               └─> wait for image (~2-4 min)
+                     └─> recreate container on target server
+                           └─> verify
+```
+
+Do not skip the verify step. The container can come up "healthy" while still serving stale config or missing a baked-in file (see Step 5).
+
+---
+
+## Step 0a — Resolve the target host (never hardcoded)
+
+This skill deliberately does **not** bake in a server hostname — this repo is mirrored to a public remote, so a real FQDN in the skill would leak into commits. Instead, resolve the deploy target into a `DEPLOY_HOST` shell variable that every `ssh` command below uses.
+
+```bash
+HOST_FILE=".claude/skills/haproxy-manager-deploy/target-host.local"
+DEPLOY_HOST="$(cat "$HOST_FILE" 2>/dev/null)"
+```
+
+- **If `$DEPLOY_HOST` is non-empty**, use it — that's the user's saved target. The file is gitignored, so the real hostname never lands in a commit.
+- **If it's empty**, ask the user which server this deploy targets (e.g. production vs. staging) and what its hostname or SSH alias is. Then offer to save it so future deploys don't have to ask:
+
+  ```bash
+  echo 'the-host-they-gave.example' > "$HOST_FILE"   # gitignored — safe to store the real FQDN here
+  ```
+
+Confirm `$DEPLOY_HOST` is set before running any `ssh` step:
+
+```bash
+[ -n "$DEPLOY_HOST" ] || echo "DEPLOY_HOST not set — ask the user for the target server"
+```
+
+All commands below assume the variable is set in the same shell session (`ssh root@"$DEPLOY_HOST" ...`).
+
+---
+
+## Step 0 — Confirm before pushing
+
+If the user just said "deploy" or "ship the haproxy fix", confirm what's actually changing: a template, the Python manager, the coraza-spoa subdir (separate image, separate workflow), or a static asset. Look at `git status` and `git diff` and read the diff back to the user if it's non-trivial.
+
+Anything that affects the customer-facing block path (e.g. `templates/hap_listener.tpl`, `errors/403-waf.html`) is **visible to every visitor on every site**. Authorization is per-deploy, not standing.
+
+---
+
+## Step 1 — Know which workflow your change triggers
+
+- `build-push.yaml` builds `haproxy-manager-base:latest` (the main image). Triggered by changes anywhere outside `coraza-spoa/`.
+- `build-push-coraza.yaml` builds `coraza-spoa:latest`. Triggered by changes inside `coraza-spoa/`.
+- `mirror-base-image.yaml` is a scheduled job mirroring upstream base images; unrelated to feature deploys.
+
+If you've changed both subtrees in one push, both workflows fire — note that the order they finish isn't guaranteed.
+
+---
+
+## Step 2 — Beware the `/etc/haproxy` named volume shadow
+
+If your change adds a NEW file that the running container needs (a baked-in asset, an errorfile, a new config snippet), **do not place it under `/etc/haproxy/` in the Dockerfile**. That path is a Docker named volume in deployed containers — image content only seeds the volume on first creation, so existing deployments will not see your new file even after a recreate.
+
+Safe paths for baked-in assets:
+- `/haproxy/...` (the image's WORKDIR — not volumed)
+- Anywhere outside `/etc/haproxy`, `/etc/letsencrypt`
+
+Reference the asset by absolute path from the haproxy config templates (e.g. `lf-file /haproxy/errors/403-waf.html`).
+
+See `feedback-haproxy-named-volume` memory for the full pattern.
+
+---
+
+## Step 3 — Commit + push
+
+Standard commit format with trailing `Co-Authored-By:` line. Match the recent commit message style (`git log --oneline -5`). Stage files explicitly by name.
+
+```bash
+git push origin main
+```
+
+Pushing immediately triggers the Gitea Actions build.
+
+---
+
+## Step 4 — Wait for the build
+
+The Go build inside coraza-spoa takes ~2-3 minutes; the haproxy-manager-base build is faster (~1-2 min). Don't bother polling the runs UI — just pull on the target server until the digest changes:
+
+```bash
+ssh root@"$DEPLOY_HOST" 'until docker pull -q repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest 2>&1 | tail -1 | grep -qE "Image is up to date|Status: Downloaded"; do sleep 15; done'
+```
+
+`-q` suppresses the noisy layer progress so the grep can match cleanly. If you started this command before the CI build finished, it'll loop until the new image lands; once the digest matches, it exits.
+
+To confirm you got the new image, check the image-creation time vs your push:
+
+```bash
+ssh root@"$DEPLOY_HOST" 'docker images repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base --format "{{.CreatedSince}}"'
+```
+
+It should say "X minutes ago" matching the build wait, not "yesterday".
+
+---
+
+## Step 5 — Verify the new image has what you think it has, BEFORE recreating
+
+The image is `gcr.io/distroless/static-debian12:nonroot`-based, no shell. To peek inside, run a one-shot with a sh entrypoint override (only works if you put one in the image — coraza-spoa is distroless and won't have sh; haproxy-manager-base is Python-based and does):
+
+```bash
+# haproxy-manager-base (has sh):
+ssh root@"$DEPLOY_HOST" 'docker run --rm --entrypoint sh repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest -c "ls /haproxy/errors/ && head -5 /haproxy/errors/403-waf.html"'
+
+# coraza-spoa (distroless, no sh) — use docker create + docker cp instead:
+ssh root@"$DEPLOY_HOST" 'docker create --name _peek repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest && docker cp _peek:/etc/coraza/config.yaml - | tar xO; docker rm _peek'
+```
+
+This step exists because the CI build can succeed but ship the wrong file (wrong commit pulled, build cache issue, etc.). Catching it here is one step earlier than catching it from a customer report.
+
+---
+
+## Step 6 — Recreate the container
+
+```bash
+ssh root@"$DEPLOY_HOST" '/root/whp/scripts/container-manager.sh recreate haproxy-manager'
+```
+
+For coraza-spoa changes:
+```bash
+ssh root@"$DEPLOY_HOST" '/root/whp/scripts/container-manager.sh recreate coraza-spoa'
+```
+
+`container-manager.sh recreate` does: stop, remove, docker pull (idempotent if already pulled), start with the right flags from settings.json. **It reads `/docker/whp/settings.json` for things like `coraza_waf.mode`**, so if the user has toggled mode while you were building, the recreated container reflects the current setting — not whatever it was when you started.
+
+---
+
+## Step 7 — Verify the deploy
+
+For haproxy-manager:
+
+```bash
+ssh root@"$DEPLOY_HOST" '
+echo "=== container ==="
+docker ps --filter name=haproxy-manager --format "image: {{.Image}} status: {{.Status}}"
+echo "=== healthy ==="
+docker inspect haproxy-manager --format "{{.State.Health.Status}}"
+echo "=== haproxy config valid ==="
+docker exec haproxy-manager haproxy -c -f /etc/haproxy/haproxy.cfg 2>&1 | tail -3
+echo "=== new asset reachable inside container ==="
+docker exec haproxy-manager ls -la /haproxy/errors/ 2>&1 | tail -3
+echo "=== panel health ==="
+curl -fsS -m 5 -o /dev/null -w "PANEL=%{http_code}\n" http://127.0.0.1:8000/health
+'
+```
+
+Pass criteria:
+- Container status = healthy
+- haproxy config validates (warnings OK, errors not)
+- Your new asset (if any) is at the expected path inside the running container
+- Panel returns 200
+
+If any check fails, the change still went out — diagnose immediately. Don't say "deploy complete" before this clears.
+
+---
+
+## Step 8 — End-to-end test if customer-visible
+
+If your change affects what a visitor sees (block pages, redirects, security responses), do a synthetic test that exercises the actual path. For WAF block-page changes, the recipe is:
+
+```bash
+# Inject a temporary ACL that forces the WAF deny path on a custom header,
+# fire one request, observe the rendered response, then revert + reload.
+ssh root@"$DEPLOY_HOST" '
+docker exec haproxy-manager cp /etc/haproxy/haproxy.cfg /tmp/cfg-bak
+docker exec haproxy-manager sh -c "sed -i \"/http-request send-spoe-group coraza coraza-req/a\\\\    http-request set-var(txn.coraza.action) str(deny) if { req.hdr(x-force-waf-block) -m str yes }\" /etc/haproxy/haproxy.cfg"
+docker exec haproxy-manager sh -c "echo reload | socat stdio /tmp/haproxy-cli" >/dev/null
+sleep 1
+curl -sSk -D - -H "x-force-waf-block: yes" -H "Host: <live-vhost>" "https://localhost/" | head -40
+# revert
+docker exec haproxy-manager cp /tmp/cfg-bak /etc/haproxy/haproxy.cfg
+docker exec haproxy-manager sh -c "echo reload | socat stdio /tmp/haproxy-cli" >/dev/null
+'
+```
+
+**Pick a real `<live-vhost>`.** The `Host:` header must match a domain currently served by this haproxy-manager, or the request won't route to the WAF path. Don't hardcode a customer hostname in this skill — pull a live one at test time (any entry from the panel's domain list, or `docker exec haproxy-manager ls /etc/letsencrypt/live`) and substitute it.
+
+**The injection point matters.** Insert AFTER `http-request send-spoe-group coraza coraza-req`, because the SPOE call overwrites `txn.coraza.action` based on the real Coraza verdict — if you inject before it, your override is wiped.
+
+**The reload mechanism matters.** Use `echo reload | socat stdio /tmp/haproxy-cli` — the container is python-based but doesn't have `kill` in PATH, and `docker kill --signal=HUP` signals the python manager (PID 1), not haproxy.
+
+---
+
+## Recovery hints
+
+- **`docker pull` exits "Image is up to date" but your change isn't there** — CI hasn't finished yet. Check `https://repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base/actions` for in-progress runs.
+- **Container recreates but new file is missing inside** — you put the file under `/etc/haproxy/` and the named volume shadows it. See Step 2. Move the file under `/haproxy/` (or another non-volumed path) and rebuild.
+- **HAProxy `lf-file` page renders but CSS is broken / percentages stripped** — literal `%` in the file body must be doubled (`100%%`). HAProxy log-format expansion eats single `%`. See `haproxy-lf-file-percent-escape` memory.
+- **Synthetic test returns 200 from gunicorn instead of the block page** — your test ACL is being overwritten by the SPOE call. Inject after `send-spoe-group`, not before.
+- **`docker exec haproxy-manager kill -HUP 1` fails** — the python-based container doesn't have `kill` in PATH. Use the haproxy admin socket: `echo reload | socat stdio /tmp/haproxy-cli`.
+
+---
+
+## Why this skill is rigid
+
+The pipeline is short, but the volume-shadowing trap and the SPOE-overwrite trap during testing each cost a 5-10 minute debugging detour during the session this skill was authored from. Both are silent failures — your change goes out, the container is healthy, and you only notice the bug when a customer report (or a careful synthetic test) surfaces it. The verification steps exist to catch them before that happens.

.gitea/workflows/build-push-coraza.yaml

@@ -0,0 +1,54 @@
+name: Build and push coraza-spoa
+run-name: ${{ gitea.actor }} pushed a change to coraza-spoa/
+
+# Triggers only on changes to the coraza-spoa subdirectory or this workflow
+# file itself — keeps the main haproxy-manager-base build and the coraza-spoa
+# build independent. workflow_dispatch lets us trigger manually after bumping
+# the upstream coraza-spoa version pin.
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'coraza-spoa/**'
+      - '.gitea/workflows/build-push-coraza.yaml'
+  workflow_dispatch:
+
+jobs:
+  Build-and-Push:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: https://github.com/docker/setup-buildx-action@v3
+
+      - name: Login to Gitea
+        uses: docker/login-action@v3
+        with:
+          registry: repo.anhonesthost.net
+          username: ${{ secrets.CI_USER }}
+          password: ${{ secrets.CI_TOKEN }}
+
+      # Mirror to GitHub Container Registry — see build-push.yaml for the
+      # secret/username convention.
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: shadowdao
+          password: ${{ secrets.GHCR_TOKEN }}
+
+      - name: Build Image
+        uses: docker/build-push-action@v6
+        with:
+          context: ./coraza-spoa
+          platforms: linux/amd64
+          push: true
+          tags: |
+            repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest
+            ghcr.io/shadowdao/coraza-spoa:latest

.gitea/workflows/build-push.yaml

@@ -24,11 +24,38 @@ jobs:
           registry: repo.anhonesthost.net
           username: ${{ secrets.CI_USER }}
           password: ${{ secrets.CI_TOKEN }}
-      
+
+      # Second push target so the image is also available from GitHub Container
+      # Registry under the user's account. The PAT only needs write:packages
+      # (and read:packages if the package is private). Stored in Gitea as
+      # secrets.GHCR_TOKEN; username is the literal GitHub login.
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: shadowdao
+          password: ${{ secrets.GHCR_TOKEN }}
+
+      # Read the human-readable release version from the VERSION file so every
+      # build is pinnable for rollback (alongside the immutable git SHA). Bump
+      # VERSION (YYYY.MM.N) in the same commit as a release-worthy change.
+      - name: Read version
+        id: ver
+        run: echo "version=$(cat VERSION)" >> "$GITHUB_OUTPUT"
+
       - name: Build Image
         uses: docker/build-push-action@v6
         with:
           platforms: linux/amd64
           push: true
+          build-args: |
+            VERSION=${{ steps.ver.outputs.version }}
+          # Three tags per registry: :latest (moving), :<version> (human-readable
+          # release), :<sha> (immutable, guaranteed-unique rollback target).
           tags: |
             repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+            repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ steps.ver.outputs.version }}
+            repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ gitea.sha }}
+            ghcr.io/shadowdao/haproxy-manager-base:latest
+            ghcr.io/shadowdao/haproxy-manager-base:${{ steps.ver.outputs.version }}
+            ghcr.io/shadowdao/haproxy-manager-base:${{ gitea.sha }}

.gitea/workflows/mirror-base-image.yaml

@@ -0,0 +1,65 @@
+name: Mirror base images
+run-name: weekly base-image mirror
+
+# Pulls each declared base image from upstream and re-pushes to the in-house
+# registry, so any of our images that FROM these don't depend on docker.io's
+# Cloudflare R2 blob storage being reachable. The 2026-05-12 Cloudflare
+# incident motivated this for python:3.12-slim and again for golang:1.25
+# when the coraza-spoa build hit the same blob-fetch failure.
+#
+# Adding a new mirror = add one entry to the matrix below. The destination
+# tag is always cloud-hosting-platform/<image>:<tag>, matching upstream.
+
+on:
+  schedule:
+    # Mondays 06:00 UTC — outside customer peak hours and well before the
+    # typical Tuesday/Thursday push cycles. workflow_dispatch lets us trigger
+    # manually from the Gitea UI when upstream publishes patches.
+    - cron: '0 6 * * 1'
+  workflow_dispatch:
+
+jobs:
+  Mirror-Base:
+    runs-on: ubuntu-latest
+    strategy:
+      # fail-fast=false so one image's upstream being down doesn't block the
+      # others from refreshing.
+      fail-fast: false
+      matrix:
+        image:
+          - { src: 'docker.io/library/python:3.12-slim', dst_path: 'cloud-hosting-platform/python', tag: '3.12-slim' }
+          - { src: 'docker.io/library/golang:1.25',      dst_path: 'cloud-hosting-platform/golang', tag: '1.25' }
+
+    steps:
+      - name: Login to in-house registry
+        uses: docker/login-action@v3
+        with:
+          registry: repo.anhonesthost.net
+          username: ${{ secrets.CI_USER }}
+          password: ${{ secrets.CI_TOKEN }}
+
+      - name: Pull, retag, push ${{ matrix.image.src }}
+        run: |
+          set -euo pipefail
+          SRC="${{ matrix.image.src }}"
+          DST="repo.anhonesthost.net/${{ matrix.image.dst_path }}:${{ matrix.image.tag }}"
+
+          echo "::group::Pulling ${SRC}"
+          docker pull "${SRC}"
+          echo "::endgroup::"
+
+          # Capture the upstream digest so the workflow log shows what we
+          # actually pushed. Helps diagnose "did the mirror really update"
+          # questions later.
+          SRC_DIGEST=$(docker image inspect "${SRC}" -f '{{index .RepoDigests 0}}')
+          echo "upstream digest: ${SRC_DIGEST}"
+
+          docker tag "${SRC}" "${DST}"
+
+          echo "::group::Pushing ${DST}"
+          docker push "${DST}"
+          echo "::endgroup::"
+
+          # Sanity: the in-house tag should now resolve to the same content.
+          DST_DIGEST=$(docker image inspect "${DST}" -f '{{index .RepoDigests 0}}')
+          echo "mirror digest:   ${DST_DIGEST}"

.gitignore

@@ -36,4 +36,8 @@ ENV/
 
 # OS
 .DS_Store
-Thumbs.db
+Thumbs.db
+
+# Local-only deploy config (never commit real hostnames)
+.claude/skills/haproxy-manager-deploy/target-host.local
+*.local

CLAUDE.md

@@ -7,8 +7,55 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ### Testing
 - **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
 - **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
+- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
+- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
+- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
 - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
 
+### Reading stick tables (and why it is easy to get silently wrong)
+
+`/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Worker commands
+(`show table`, `show map`, `add map`, ...) need an `@1` prefix. Without it
+HAProxy answers `Unknown command: 'show', ...` **and socat still exits 0** — so
+an exit-status check passes and the help text gets parsed as data. Always use
+`haproxy_cli(cmd, worker=True)` in Python, which inspects the response body.
+
+Stick-table entries are `name=value` / `name(window_ms)=value` pairs, not fixed
+columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
+by NAME, and treat a missing field as an ERROR — never default it to `0`. The
+`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
+`http_err_rate`; it holds **no history and no counter of past blocks**. What was
+actually denied/tarpitted is in the edge access log on the **host** at
+`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table.
+
+This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
+reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
+stick table has ever stored — for their entire existence. See the header of
+`haproxy_tarpit_config.txt` and the contract test.
+
+### Changing a runtime map (`add map` / `del map`)
+
+Same socket, two more ways to fail silently — and both were live in
+`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
+existence:
+
+* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
+  config-parse time and move on every config regeneration (on whp01
+  `blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
+  `add map /etc/haproxy/blocked_ips.map <ip> 1`.
+* **A mutation answers NOTHING on success**, so an empty body is the only
+  success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
+  *also* answers nothing and adds nothing, so the body cannot prove an add
+  worked. **Read it back** with `@1 get map <path> <key>`.
+* Entries must carry the value `1`; haproxy.cfg matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
+
+In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
+and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
+only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
+re-reads it on reload, so a failed runtime command must degrade to
+"enforced on reload" and be reported, never swallowed.
+
 ### Running the Application
 - **Docker Build**: `docker build -t haproxy-manager .`
 - **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)
@@ -39,10 +86,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
    - `backend_servers` - Individual servers within backend groups
 
 3. **Template System** - Jinja2 templates for HAProxy configuration generation:
-   - `hap_header.tpl` - Global HAProxy settings and defaults
+   - `hap_header.tpl` - Global HAProxy settings, defaults, and HTTP/2 tuning
    - `hap_backend.tpl` - Backend server definitions
-   - `hap_listener.tpl` - Frontend listener configurations
+   - `hap_listener.tpl` - Frontend listener configurations with rate limiting
    - `hap_letsencrypt.tpl` - SSL certificate configurations
+   - `hap_security_tables.tpl` - Stats frontend and security stick tables
    - Template override support for custom backend configurations
 
 4. **Certificate Management** - Automated SSL certificate handling:
@@ -73,10 +121,51 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - Certificate private keys combined with certificates in HAProxy-compatible format
 - Default backend page for unmatched domains instead of exposing HAProxy errors
 
+### Rate Limiting & Connection Limits (hap_listener.tpl)
+
+- **Stick table**: `type ip size 200k expire 10m` tracking `conn_cur`, `conn_rate(10s)`, `http_req_rate(10s)`, `http_err_rate(30s)`
+- Tracks real client IP via `var(txn.real_ip)` to work correctly behind Cloudflare/proxies
+- **Rate limit thresholds**:
+  - Tarpit at 3000 req/10s (300 req/s)
+  - Hard block (deny) at 5000 req/10s (500 req/s)
+  - Connection rate limit: 500/10s
+  - Concurrent connection limit: 500
+  - Error rate limit: 100/30s
+- **Whitelist bypasses** (exempt from rate limits):
+  - `is_local` — RFC1918 private address ranges
+  - `is_trusted_ip` — source IPs listed in `trusted_ips.list`
+  - `is_whitelisted` — real IPs (from proxy headers) matched in `trusted_ips.map`
+
+### Trusted IP Whitelist Files
+
+- `trusted_ips.list` — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
+- `trusted_ips.map` — Real IP whitelist for proxy-header matching (format: `<IP> 1`)
+- Both files are baked into the Docker image via `COPY` in the Dockerfile
+- Ship as comment-only templates (no real IPs). Add trusted IPs locally and do **not** commit them — this repo is mirrored publicly. Entries persist in the `/etc/haproxy` named volume across recreates
+
+### Timeout Hardening (hap_header.tpl)
+
+- `timeout http-request`: 300s -> 30s (slowloris protection)
+- `timeout connect`: 120s -> 10s
+- `timeout client`: 10m -> 5m
+- `timeout http-keep-alive`: 120s -> 30s
+
+### HTTP/2 Protection (hap_header.tpl)
+
+- `tune.h2.fe.max-total-streams 2000` — limits total streams per HTTP/2 connection
+- `tune.h2.fe.glitches-threshold 50` — CVE-2023-44487 Rapid Reset protection
+
+### Stats Frontend (hap_security_tables.tpl)
+
+- HAProxy stats page bound to `127.0.0.1:8404` (localhost only, accessible inside container)
+- Template: `templates/hap_security_tables.tpl`
+
 ### Deployment Context
 
 - Designed to run as Docker container with persistent volumes for certificates and configurations
 - Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
+- Stats page on port 8404 (localhost only inside container)
 - Management interface on port 8000 should be firewall-protected in production
+- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s`
 - Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
 - HAProxy is version 3.0.11

Dockerfile

@@ -1,12 +1,98 @@
-FROM python:3.12-slim
-RUN apt update -y && apt dist-upgrade -y && apt install socat haproxy cron certbot curl jq net-tools -y && apt clean && rm -rf /var/lib/apt/lists/*
+# Base image mirrored into the in-house registry to remove docker.io
+# (Cloudflare R2) as a single point of failure for CI builds. The 2026-05-12
+# Cloudflare incident took down docker.io blob pulls and broke this image's CI.
+# Refresh procedure (run on a workstation that can reach docker.io, e.g.
+# monthly or when Python patches drop):
+#     docker pull docker.io/library/python:3.12-slim
+#     docker tag  docker.io/library/python:3.12-slim \
+#                 repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
+#     docker push repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
+# Future improvement: a scheduled Gitea Action that does the above automatically.
+FROM repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
+
+# image.source is what ghcr.io uses to link the package to a GitHub repo
+# sidebar; pointing at the public GitHub mirror enables that linking. The
+# canonical source-of-truth git remote is still Gitea, but Gitea's registry
+# doesn't consume this label, so there's no contention.
+# Stamped from the VERSION file by CI (build-arg) so `docker inspect` reports
+# what's running on any host. Defaults to "dev" for local/manual builds.
+ARG VERSION=dev
+LABEL org.opencontainers.image.title="haproxy-manager-base" \
+      org.opencontainers.image.description="HAProxy management API with Let's Encrypt automation, Coraza WAF integration, and template-driven config" \
+      org.opencontainers.image.source="https://github.com/shadowdao/haproxy-manager-base" \
+      org.opencontainers.image.version="${VERSION}" \
+      org.opencontainers.image.licenses="MIT"
+
+# haproxy is PINNED. It was previously unpinned, so the binary could move under
+# us at Debian's timing — an upstream release that rejected our config would have
+# broken an unrelated commit's build, or worse, shipped an edge that refuses to
+# start (see the `haproxy -c` gate below for why that matters: a config HAProxy
+# rejects leaves the container Up with 80/443 unbound and /health still 200).
+#
+# Pinning does NOT make the gate redundant, and the gate does NOT make pinning
+# unnecessary — they compose. Pinned means the version moves deliberately; the
+# gate then answers immediately whether the new binary still accepts fleet config.
+# It also makes the image reproducible, which it previously was not.
+#
+# To move it: bump the version here, rebuild, and let the gate verify. If Debian
+# security-updates the package (e.g. -1+deb13u4) the build FAILS until this pin is
+# updated — that failure is the point, not a bug. Check availability with:
+#   apt-cache policy haproxy
+ARG HAPROXY_VERSION=3.0.11-1+deb13u3
+RUN apt update -y && apt dist-upgrade -y && apt install socat "haproxy=${HAPROXY_VERSION}" cron certbot curl jq net-tools -y && apt-mark hold haproxy && apt clean && rm -rf /var/lib/apt/lists/*
 WORKDIR /haproxy
 COPY ./templates /haproxy/templates
 COPY requirements.txt /haproxy/
 COPY haproxy_manager.py /haproxy/
 COPY scripts /haproxy/scripts
+COPY trusted_ips.list /etc/haproxy/trusted_ips.list
+COPY trusted_ips.map /etc/haproxy/trusted_ips.map
+# /etc/haproxy is a named volume in deployed containers, so baked-in files
+# under that path get shadowed by the volume on existing deployments. The
+# trusted_ips.* pair above predates that discovery and is handled by the
+# older start-up.sh guard (out of scope here). cloudflare_ips.list and
+# trusted_proxies.list are staged under /haproxy/defaults instead, so
+# start-up.sh can always read the image's baked copy regardless of what the
+# volume shadows /etc/haproxy with.
+COPY cloudflare_ips.list /haproxy/defaults/cloudflare_ips.list
+COPY trusted_proxies.list /haproxy/defaults/trusted_proxies.list
+COPY wpadmin_gate_exempt.list /haproxy/defaults/wpadmin_gate_exempt.list
+# Place errorfiles outside the volumed path; the HAProxy config references
+# them by absolute path.
+COPY errors /haproxy/errors
 RUN chmod +x /haproxy/scripts/*
 RUN pip install -r requirements.txt
+# ---------------------------------------------------------------------------
+# Build gate: no image ships unless the real haproxy binary accepts the config
+# this image's templates actually produce.
+#
+# On 2026-08-14 a template change rendered fine, passed all 13 unit tests, and
+# was rejected by HAProxy ("invalid arg 2 in converter 'regsub'"). It was only
+# caught because someone built an image by hand and ran `haproxy -c`. Nothing
+# in the build or in CI would have stopped it: .gitea/workflows/build-push.yaml
+# is checkout -> build -> push, and test-config-rollback.py's "haproxy" is a
+# shell stub that only rejects a sentinel token. In production an invalid
+# haproxy.cfg means init.py refuses to start HAProxy while the container stays
+# Up - ports 80/443 unbound, every site on the host down, /health still 200.
+#
+# This lives in the Dockerfile rather than in the workflow deliberately:
+#   * it cannot be skipped, and it protects local `docker build` too;
+#   * no workflow restructuring (build-push-action builds and pushes in one
+#     step, so gating in CI would mean splitting build from push);
+#   * it validates against the EXACT haproxy binary in this image. Line 26
+#     installs haproxy unpinned, so that binary moves between builds - this
+#     turns "the new haproxy rejects our config" from a silent production
+#     risk into a build failure.
+#
+# The unit suites run here too. They had never run anywhere automated either,
+# and they cost a few seconds.
+RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
+ && python3 /haproxy/scripts/test-trusted-proxy-gate.py \
+ && python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \
+ && python3 /haproxy/scripts/test-config-rollback.py \
+ && python3 /haproxy/scripts/test-cert-write-safety.py \
+ && python3 /haproxy/scripts/test-cert-scripts.py \
+ && python3 /haproxy/scripts/validate-rendered-config.py
 # Create log directories
 RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
 RUN chmod 755 /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
@@ -16,7 +102,9 @@ RUN mkdir -p /var/spool/cron/crontabs && \
     echo '0 */12 * * * /haproxy/scripts/renew-certificates.sh >> /var/log/haproxy-manager.log 2>&1' >> /var/spool/cron/crontabs/root && \
     chmod 600 /var/spool/cron/crontabs/root && \
     chown root:crontab /var/spool/cron/crontabs/root
-EXPOSE 80 443 8000
+# 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
+# must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
+EXPOSE 80 443 443/udp 8000
 # Add health check
 HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
     CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1

IP_BLOCKING_API.md

@@ -508,20 +508,45 @@ curl -X POST http://localhost:8000/api/blocked-ips/sync \
 
 For advanced users, you can interact directly with HAProxy's runtime API:
 
+Three things about these commands are easy to get wrong, and each one fails
+**silently** (socat exits 0 either way — the rejection, if any, is only in the
+response body):
+
+* `/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Map commands are
+  worker commands and need the `@1` prefix. Without it the reply is
+  `Unknown command: 'add', ...`.
+* Reference the map by its **file path**, never by `#<id>`. Ids are assigned at
+  config-parse time and move on every config regeneration (on a live edge,
+  `blocked_ips.map` is id 37, `trusted_ips.map` is 10 — there is no id 0).
+  Worse, `@1 add map #0 <ip> 1` returns an **empty** reply and adds nothing.
+* Entries must carry the value `1`. `haproxy.cfg` matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block. (`add map`
+  with no value is rejected: `'add map' expects three parameters ...`.)
+
 ```bash
+MAP=/etc/haproxy/blocked_ips.map
+
 # Add IP to runtime (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map $MAP 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP from runtime
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
+
+# Confirm what actually happened (do not trust the exit status)
+echo "@1 get map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
 
 # Clear all blocked IPs from runtime
-echo "clear map #0" | socat stdio /var/run/haproxy.sock
+echo "@1 clear map $MAP" | socat stdio /tmp/haproxy-cli
 
-# Show all runtime map entries
-echo "show map #0" | socat stdio /var/run/haproxy.sock
+# Show all runtime map entries, and the map ids currently in use
+echo "@1 show map $MAP" | socat stdio /tmp/haproxy-cli
+echo "@1 show map" | socat stdio /tmp/haproxy-cli
 ```
 
+The runtime map is a **fast path only**. `/etc/haproxy/blocked_ips.map` is
+authoritative: HAProxy re-reads it on reload, so a failed runtime command
+delays a block until the next reload rather than losing it.
+
 ## Migration from ACL Method
 
 If you're upgrading from the old ACL-based method:

MIGRATION_GUIDE.md

@@ -50,14 +50,22 @@ http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }
 - **Graceful error handling**
 
 ### 2. Runtime IP Management
+Map commands go to a **worker** (`@1`), reference the map by **file path**
+(ids move between config regenerations, and `#0` silently adds nothing), and
+carry the value `1` that `map_ip(...,0) -m int gt 0` matches on:
+
 ```bash
 # Add IP without reload (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map /etc/haproxy/blocked_ips.map 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP without reload
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map /etc/haproxy/blocked_ips.map 192.168.1.100" | socat stdio /tmp/haproxy-cli
 ```
 
+socat exits 0 even when HAProxy rejects the command, so read the response body
+(or read the entry back with `@1 get map ...`) rather than the exit status.
+See IP_BLOCKING_API.md for the full set.
+
 ### 3. New API Endpoints
 
 #### Safe Config Reload

README.md

@@ -6,10 +6,10 @@ A Flask-based API service for managing HAProxy configurations with dynamic SSL c
 To run the container:
 ```bash
 # Without API key authentication (default)
-docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 
 # With API key authentication (recommended for production)
-docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 ```
 
 ## Features
@@ -394,7 +394,7 @@ You can customize the default page by setting environment variables:
 
 ```bash
 docker run -d \
-  -p 80:80 -p 443:443 -p 8000:8000 \
+  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
   -v lets-encrypt:/etc/letsencrypt \
   -v haproxy:/etc/haproxy \
   -e HAPROXY_API_KEY=your-secure-api-key-here \
@@ -402,7 +402,7 @@ docker run -d \
   -e HAPROXY_DEFAULT_MAIN_MESSAGE="This website is currently under construction and will be available soon." \
   -e HAPROXY_DEFAULT_SECONDARY_MESSAGE="Please check back later or contact us for more information." \
   --name haproxy-manager \
-  repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 ```
 
 ## Example Usage
@@ -411,12 +411,12 @@ docker run -d \
 ```bash
 # Start container with API key
 docker run -d \
-  -p 80:80 -p 443:443 -p 8000:8000 \
+  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
   -v lets-encrypt:/etc/letsencrypt \
   -v haproxy:/etc/haproxy \
   -e HAPROXY_API_KEY=your-secure-api-key-here \
   --name haproxy-manager \
-  repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 
 # Add a domain
 curl -X POST http://localhost:8000/api/domain \

VERSION

@@ -0,0 +1 @@
+2026.09.1

cloudflare_ips.list

@@ -0,0 +1,34 @@
+# Cloudflare edge ranges — peers allowed to set CF-Connecting-IP / X-Real-IP /
+# X-Forwarded-For. Referenced by templates/hap_listener.tpl:
+#   acl from_trusted_proxy src -f /etc/haproxy/cloudflare_ips.list -f /etc/haproxy/trusted_proxies.list
+#
+# PUBLIC DATA — safe to commit. Source: https://www.cloudflare.com/ips-v4
+# and https://www.cloudflare.com/ips-v6. Keep in sync with the IPv4 snapshot in
+# WHP's ssl_renewal_orchestrator.php::isCloudflareIP() (that one is IPv4-only).
+# Refresh at release time; Cloudflare has not added a range since 2021.
+#
+# IPv4
+173.245.48.0/20
+103.21.244.0/22
+103.22.200.0/22
+103.31.4.0/22
+141.101.64.0/18
+108.162.192.0/18
+190.93.240.0/20
+188.114.96.0/20
+197.234.240.0/22
+198.41.128.0/17
+162.158.0.0/15
+104.16.0.0/13
+104.24.0.0/14
+172.64.0.0/13
+131.0.72.0/22
+#
+# IPv6
+2400:cb00::/32
+2606:4700::/32
+2803:f800::/32
+2405:b500::/32
+2405:8100::/32
+2a06:98c0::/29
+2c0f:f248::/32

coraza-spoa/Dockerfile

@@ -0,0 +1,70 @@
+# Coraza-SPOA sidecar for haproxy-manager.
+#
+# Layout: built from upstream source. main.go is at the repo root; CRS rules
+# are bundled into the binary at build time (referenced as @owasp_crs/), so
+# the CRS version is whatever ships with the pinned coraza-spoa tag.
+#
+# Pin: review the upstream CHANGELOG (https://github.com/corazawaf/coraza-spoa/releases)
+# before bumping. New tags can ship newer CRS, which can introduce new rules
+# whose IDs fall into the "enforce day-one" ranges in overrides.conf — verify
+# those are still high-confidence before promoting a new tag to prod.
+
+ARG CORAZA_SPOA_VERSION=v0.7.1
+
+# golang:1.25 from the in-house mirror. The 2026-05-12 Cloudflare incident
+# took out docker.io blob pulls TWICE in one day (first for python:3.12-slim,
+# then for this image's golang:1.25), so both are mirrored at
+# repo.anhonesthost.net via the .gitea/workflows/mirror-base-image.yaml
+# weekly job.
+FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS build
+ARG CORAZA_SPOA_VERSION
+WORKDIR /src
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git \
+    && rm -rf /var/lib/apt/lists/*
+RUN git clone --depth 1 --branch "${CORAZA_SPOA_VERSION}" \
+        https://github.com/corazawaf/coraza-spoa.git . \
+    && go mod download \
+    && CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa .
+
+# Catalog extractor: walks the bundled CRS at build time and emits
+# rules-catalog.json so WHP's UI can render rule metadata without parsing
+# .conf files at runtime. Uses the SAME coraza-coreruleset version pin as
+# the coraza-spoa binary above (drift between the two would mislabel rules).
+FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog
+WORKDIR /src
+COPY catalog-extractor/ .
+RUN go build -trimpath -o /out/catalog-extractor . \
+    && /out/catalog-extractor > /out/rules-catalog.json
+
+# Distroless runtime: no shell, no package manager, no /tmp by default —
+# smallest attack surface for an exposed service. Audit log directory is
+# bind-mounted; coraza-spoa writes to it via direct file I/O (no shell needed).
+FROM gcr.io/distroless/static-debian12:nonroot
+
+LABEL org.opencontainers.image.title="coraza-spoa-whp" \
+      org.opencontainers.image.description="Coraza WAF SPOA agent configured for WHP haproxy-manager integration" \
+      org.opencontainers.image.source="https://github.com/shadowdao/haproxy-manager-base" \
+      org.opencontainers.image.licenses="MIT"
+
+COPY --from=build /out/coraza-spoa /coraza-spoa
+COPY config.yaml /etc/coraza-spoa/config.yaml
+COPY overrides.conf /etc/coraza/overrides.conf
+COPY pre-overrides.conf /etc/coraza/pre-overrides.conf
+COPY local-overrides.conf /etc/coraza/local-overrides.conf
+COPY host-exceptions/ /etc/coraza/host-exceptions/
+COPY --from=catalog /out/rules-catalog.json /etc/coraza/rules-catalog.json
+
+# Audit log directory — bind-mount /var/log/coraza:/var/log/coraza from host
+# so logs persist across container restarts and AI Monitor can tail them.
+# Distroless nonroot user has UID 65532; the host directory must be writable
+# by that UID (install script will chown it appropriately).
+VOLUME ["/var/log/coraza"]
+
+# SPOE TCP port — bound on 0.0.0.0:9000 inside the container. The host-side
+# port mapping is controlled by `docker run -p` (typically not exposed beyond
+# the internal docker network, since haproxy-manager reaches it by container
+# name on client-net).
+EXPOSE 9000
+
+ENTRYPOINT ["/coraza-spoa", "--config", "/etc/coraza-spoa/config.yaml"]

coraza-spoa/README.md

@@ -0,0 +1,78 @@
+# coraza-spoa sidecar
+
+A sidecar container that runs [Coraza-SPOA](https://github.com/corazawaf/coraza-spoa) as a WAF engine for `haproxy-manager`. HAProxy consults it per-request via the SPOE/SPOP protocol; Coraza evaluates the request against OWASP CRS rules and tells HAProxy whether to allow or block.
+
+## Design constraints
+
+- **`haproxy-manager` does NOT depend on this sidecar.** The base image works standalone (used in other projects and home networks) without WAF. SPOE config in the generated `haproxy.cfg` is opt-in via an env var on `haproxy-manager`.
+- **Fail-open when the sidecar is unhealthy.** `option set-on-error continue` in the HAProxy SPOE config means request flow continues uninspected if coraza-spoa is unreachable, rather than 503-ing customer traffic.
+- **Detect-only globally; enforce explicitly.** See `overrides.conf` for the day-one enforce list. Most CRS rules log without blocking until we've tuned per-customer false positives.
+
+## Deployment shape
+
+Two containers per host, both on the `client-net` docker network:
+
+```
+haproxy-manager        (existing) — ports 80, 443, 8000
+   │ SPOE TCP/9000 → reach coraza-spoa by container DNS
+   ▼
+coraza-spoa            (this image)
+   port 9000 (SPOE) — NOT exposed on host; internal network only
+   /var/log/coraza    — bind-mounted to host for AI Monitor consumption
+```
+
+Typical `docker run`:
+
+```bash
+mkdir -p /var/log/coraza
+chown 65532:65532 /var/log/coraza   # distroless nonroot UID
+
+docker run -d \
+    --name coraza-spoa \
+    --network client-net \
+    --restart unless-stopped \
+    -v /var/log/coraza:/var/log/coraza \
+    your-registry.example.com/cloud-hosting-platform/coraza-spoa:latest
+```
+
+Then on the `haproxy-manager` container, add the env var:
+
+```
+-e HAPROXY_CORAZA_SPOE_BACKEND=coraza-spoa:9000
+```
+
+The haproxy-manager template engine sees the env var and renders the SPOE config block pointing at this sidecar. Without the env var, no SPOE blocks render — the haproxy-manager image's behavior is unchanged.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `Dockerfile` | Multi-stage build (golang:1.25 → distroless), pinned to upstream coraza-spoa tag |
+| `config.yaml` | SPOA listener config + one named application `haproxy` |
+| `overrides.conf` | Day-one enforce list (`ctl:ruleEngine=On` for high-confidence rule IDs) |
+| `README.md` | This file |
+
+## Audit log
+
+`/var/log/coraza/audit.log` — JSON, one event per line, RelevantOnly (only requests that triggered ≥1 rule are logged). AI Monitor should be configured to tail this on each host.
+
+Entries include rule IDs, matched patterns, request metadata, and action taken (`log` for detect-only, `deny` for enforced). Use the JSON `action` field to filter blocked vs. observed.
+
+## Upgrading the pin
+
+CRS rules are bundled into the coraza-spoa binary at build time, so the CRS version is whatever ships with the pinned coraza-spoa tag. To upgrade:
+
+1. Check upstream releases: <https://github.com/corazawaf/coraza-spoa/releases>
+2. Skim the CHANGELOG for new/changed rules in the `overrides.conf` ID ranges.
+3. Bump `ARG CORAZA_SPOA_VERSION` in the Dockerfile.
+4. Push to `main` — the Gitea workflow at `.gitea/workflows/build-push-coraza.yaml` rebuilds + pushes `:latest`.
+5. On each host, run `container-manager.sh recreate coraza-spoa` to pull the new image.
+
+## Tuning false positives
+
+When a legitimate request triggers a blocked rule, the audit log shows the rule ID. Two ways to silence it:
+
+1. **Per-rule exception** in `overrides.conf`: `SecRuleRemoveById <id>` (full disable) or `SecRuleRemoveTargetById <id> "<target>"` (targeted exception).
+2. **Drop from the enforce list**: remove the rule's ID range from the `ctl:ruleEngine=On` overrides; it falls back to detect-only.
+
+After tuning, push the change — CI rebuilds, then `recreate coraza-spoa` on each host to apply.

coraza-spoa/catalog-extractor/go.mod

@@ -0,0 +1,7 @@
+module catalog-extractor
+
+go 1.23
+
+require github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
+
+require github.com/magefile/mage v1.17.0 // indirect

coraza-spoa/catalog-extractor/go.sum

@@ -0,0 +1,12 @@
+github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 h1:tqFO1lfVpTiyWtlN618OXpZMfw+nnN0Q4///W5W+/HM=
+github.com/corazawaf/coraza-coreruleset/v4 v4.25.0/go.mod h1:nRuGXITxOPvsLF2VxaTB7pYok8QB8BitX3ZenXcUryY=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM=
+github.com/magefile/mage v1.17.0/go.mod h1:Yj51kqllmsgFpvvSzgrZPK9WtluG3kUhFaBUVLo4feA=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

coraza-spoa/catalog-extractor/main.go

@@ -0,0 +1,80 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/fs"
+	"os"
+	"regexp"
+	"strconv"
+	"strings"
+
+	crs "github.com/corazawaf/coraza-coreruleset/v4"
+)
+
+type Rule struct {
+	ID       int      `json:"id"`
+	Msg      string   `json:"msg"`
+	Severity string   `json:"severity"`
+	Tags     []string `json:"tags"`
+	File     string   `json:"file"`
+}
+
+var (
+	idRe       = regexp.MustCompile(`(?i)\bid:'?(\d+)'?`)
+	msgRe      = regexp.MustCompile(`(?i)\bmsg:'([^']+)'`)
+	severityRe = regexp.MustCompile(`(?i)\bseverity:'?([A-Z]+)'?`)
+	tagRe      = regexp.MustCompile(`(?i)\btag:'([^']+)'`)
+)
+
+func main() {
+	out := []Rule{}
+	err := fs.WalkDir(crs.FS, ".", func(path string, d fs.DirEntry, err error) error {
+		if err != nil || d.IsDir() {
+			return err
+		}
+		if !strings.HasSuffix(path, ".conf") {
+			return nil
+		}
+		b, err := fs.ReadFile(crs.FS, path)
+		if err != nil {
+			return err
+		}
+		// Coalesce backslash-continuation lines so id/msg/etc on the same
+		// logical rule are visible to the per-line scanner.
+		text := regexp.MustCompile(`\\\s*\n\s*`).ReplaceAllString(string(b), " ")
+		for _, line := range strings.Split(text, "\n") {
+			line = strings.TrimSpace(line)
+			if !strings.HasPrefix(line, "SecRule") && !strings.HasPrefix(line, "SecAction") {
+				continue
+			}
+			m := idRe.FindStringSubmatch(line)
+			if m == nil {
+				continue
+			}
+			id, _ := strconv.Atoi(m[1])
+			r := Rule{ID: id, File: path}
+			if mm := msgRe.FindStringSubmatch(line); mm != nil {
+				r.Msg = mm[1]
+			}
+			if mm := severityRe.FindStringSubmatch(line); mm != nil {
+				r.Severity = strings.ToLower(mm[1])
+			}
+			for _, mm := range tagRe.FindAllStringSubmatch(line, -1) {
+				r.Tags = append(r.Tags, mm[1])
+			}
+			out = append(out, r)
+		}
+		return nil
+	})
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	enc := json.NewEncoder(os.Stdout)
+	enc.SetIndent("", "  ")
+	if err := enc.Encode(out); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}

coraza-spoa/config.yaml

@@ -0,0 +1,72 @@
+# Coraza-SPOA configuration for WHP haproxy-manager integration.
+#
+# One named application "haproxy" — the haproxy-manager spoe template
+# references this same name in its spoe-agent block, so the SPOA knows
+# which rules to apply when HAProxy dispatches a request.
+#
+# Mode: SecRuleEngine DetectionOnly globally; overrides.conf promotes
+# specific high-confidence rule ID ranges to enforcement individually.
+# This is the safest posture for v1 — every rule logs, but only the
+# unambiguous ones (scanner UAs, RCE, LFI, webshells, Log4Shell) block.
+
+bind: 0.0.0.0:9000
+
+# Process-level logging (separate from per-request audit logging below)
+log_level: info
+log_file: /dev/stdout
+log_format: json
+
+# Fallback when the request doesn't match a named application — we only
+# have one, so it's also the default.
+default_application: haproxy
+
+applications:
+  - name: haproxy
+    directives: |
+      # CRS-bundled defaults: recommended Coraza settings + CRS setup +
+      # the rule pack itself (~16 MB of rules embedded in the binary).
+      Include @coraza.conf-recommended
+      Include @crs-setup.conf.example
+
+      # Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.
+      # Loaded BEFORE the CRS rules so per-host ctl:ruleRemoveById exemptions
+      # fire in phase:1 BEFORE the CRS rule they're trying to exempt would
+      # otherwise match. Server-wide overrides live in local-overrides.conf
+      # (loaded after CRS) instead.
+      Include /etc/coraza/pre-overrides.conf
+
+      Include @owasp_crs/*.conf
+
+      # WHP-specific overrides — day-one enforce list, plus tuning for
+      # the customer mix (WordPress, WooCommerce, Divi). Read this file
+      # to see exactly what blocks vs what's detect-only.
+      Include /etc/coraza/overrides.conf
+
+      # Runtime-managed POST-CRS overrides written by WHP UI. Empty by default.
+      Include /etc/coraza/local-overrides.conf
+
+      # Global mode: log all alerts, block only what overrides.conf
+      # explicitly promotes via ctl:ruleEngine=On.
+      SecRuleEngine DetectionOnly
+
+      # Audit log: JSON to a bind-mounted file so AI Monitor + log
+      # rotation can pick it up. RelevantOnly means we don't log every
+      # passing request, only ones that triggered at least one rule.
+      SecAuditEngine RelevantOnly
+      SecAuditLog /var/log/coraza/audit.log
+      SecAuditLogFormat JSON
+      SecAuditLogParts ABIJDEFHKZ
+
+    # HAProxy sends request-only events for v1. Response inspection adds
+    # latency on every page render with marginal additional protection
+    # for our customer mix; can be turned on later if we want it.
+    response_check: false
+
+    # Transactions cache for 60s. SPOE protocol is fire-and-forget per
+    # request, so this is just how long Coraza holds context for any
+    # multi-stage processing.
+    transaction_ttl_ms: 60000
+
+    log_level: info
+    log_file: /var/log/coraza/spoa.log
+    log_format: json

coraza-spoa/local-overrides.conf

@@ -0,0 +1,3 @@
+# AUTOGENERATED by WHP — do not hand-edit.
+# Source of truth: whp.security_db coraza_rule_overrides table.
+# Empty file = no runtime overrides; baked-in overrides.conf governs.

coraza-spoa/overrides.conf

@@ -0,0 +1,105 @@
+# WHP day-one enforce overrides for coraza-spoa.
+#
+# Global mode in config.yaml is SecRuleEngine DetectionOnly. The rule ID
+# ranges below are promoted to enforcement individually, chosen for very
+# low false-positive rate on the kinds of customer traffic seen on WHP
+# (WordPress, WooCommerce, Divi page builders).
+#
+# When bumping the upstream coraza-spoa pin (and thus the bundled CRS):
+#  1. Skim the CRS CHANGELOG for new/changed rules in these ID ranges.
+#  2. Verify they're still high-confidence before promoting the new image.
+#  3. Smoke-test in staging detect-only mode for 24h before flipping enforce.
+#
+# Per-customer false-positive tuning lives in a future per-customer
+# override mechanism; v1 is server-wide.
+
+# ---------------------------------------------------------------------------
+# 930120 — LFI: explicit traversal to sensitive system files
+# (/etc/passwd, /proc/self/, /.ssh/, /etc/shadow, /etc/group, etc.)
+# Unambiguous probe pattern; no legitimate site path leads here.
+# Note: 930xxx as a whole includes broader traversal patterns that can FP
+# on legitimate relative-path file browsers — keep those detect-only.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 930120 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 932100-932160 — RCE: Unix shell command injection
+# Patterns like `; cat /etc/passwd`, `|whoami`, backtick `\`uname\``,
+# $(...) substitution, &&/|| chaining with shell builtins.
+# Don't appear in normal POST bodies, URL params, or headers. Targeting
+# these is unambiguous attempted command execution.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 932100-932160 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 933170-933200 — PHP Webshell access patterns
+# Direct requests to known webshell paths: c99.php, r57.php, b374k.php,
+# wso.php, alfa.php, mini.php, etc. Almost universally reconnaissance
+# scanning for post-exploitation. Even legitimate WordPress installs
+# never serve these paths.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 933170-933200 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 944100-944300 — Log4Shell / JNDI injection
+# `${jndi:ldap://}`, `${jndi:rmi://}`, and obfuscated variants thereof
+# in headers, query strings, or bodies. Even our PHP/Node stack isn't
+# vulnerable, but blocking at the edge keeps logs clean and protects
+# any future Java workloads.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 920440 — URL file extension restricted by policy
+# Catches probes for backup / config / dump files: .bak, .old, .save,
+# .swp, .sql, .dist, .backup. Promoted to enforce after empirical
+# observation on whp01 (2026-05-12, first ~30 min of detect-only):
+# 124 events, all backup-file recon — `/wp-config.php.old`,
+# `/db_backup.sql`, `/.env.save`, `/releases.sql`, etc. — from a
+# single GCP-hosted scanner. Zero false positives observed; standard
+# WP/WooCommerce/Divi/HPR URLs do not end in these extensions.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 930130 — Restricted File Access Attempt
+# Catches dotfile / VCS / config-disclosure probes: .env (and .env.local /
+# .env.bak / .env.save variants), .git/config, config.php at root or under
+# /admin /backend, etc. Distinct from 930120 (system file paths like
+# /etc/passwd); this targets application secret files.
+#
+# Promoted to enforce on the same observation pass that justified 920440:
+# 117 events split across joshuaknapp.net (136), cgdannyb.com (51),
+# onlinesupplements.net (23) — all `.env`-class disclosure probes.
+# Zero false positives observed. Notably, HPR's `/ccdn.php?filename=...`
+# audio delivery path does NOT trigger this rule — verified empirically.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 930130 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# Rule families intentionally kept at DETECT-ONLY for v1 — high FP rate
+# on customer mix. Promote individually after observation:
+#
+#   913xxx (Scanner UAs)— matches legitimate ActivityPub federation
+#                         (Mastodon's "...Bot" UA) and SiteLockSpider (a
+#                         paid customer-security service some sites use).
+#                         Observed on whp01 burn-in 2026-05-13:
+#                         20/185 hits = ~11% FP rate on HPR + greggfranklin
+#                         + suchascream. Detection adds anomaly score
+#                         either way; enforce upside is low.
+#   941xxx (XSS)        — Divi rich-text editor saves, TinyMCE submissions
+#   942xxx (SQLi)       — WP admin queries reflected in params
+#   920xxx (other)      — most 920xxx rules; 920440 specifically promoted above
+#   933150              — PHP injection FP on WooCommerce checkout
+#                         (`session_start` literal appearing in billing form data)
+#   950xxx-953xxx       — Data leakage / backup-file disclosure (mixed FP)
+# ---------------------------------------------------------------------------
+
+# ---------------------------------------------------------------------------
+# RESERVED RULE-ID RANGE: 990000000 – 990999999
+# WHP's coraza_rule_manager generates per-host-exception rules in this range
+# (rule ID = 990000000 + target_rule_id). Do NOT add new rules in this range
+# from any other source. When bumping the coraza-spoa pin, check the CRS
+# changelog for new rules with 9-digit IDs (rare but possible) and re-namespace
+# if collision risk emerges.
+# ---------------------------------------------------------------------------

coraza-spoa/pre-overrides.conf

@@ -0,0 +1,3 @@
+# AUTOGENERATED by WHP — do not hand-edit.
+# Source of truth: whp.security_db coraza_rule_host_exceptions table.
+# Loaded BEFORE the CRS rules. Empty file = no per-host exemptions active.

errors/403-waf.html

@@ -0,0 +1,109 @@
+<!DOCTYPE html>
+<!--
+  Served by HAProxy via `lf-file` on Coraza WAF deny.
+  IMPORTANT: HAProxy's lf-file expansion treats `%` as the start of a
+  log-format expression. Literal percent signs (CSS 100%, gradient stops,
+  url-encoded data, etc.) MUST be doubled as `%%` or HAProxy will silently
+  swallow them. Expressions like `%[unique-id]` / `%[req.hdr(host)]` stay
+  single-`%` — those are the substitutions we want.
+-->
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="robots" content="noindex, nofollow">
+<title>Request blocked &middot; %[req.hdr(host)]</title>
+<style>
+  *, *::before, *::after { box-sizing: border-box; }
+  html, body { margin: 0; padding: 0; height: 100%%; }
+  body {
+    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+    color: #1f2937;
+    background: linear-gradient(135deg, #f9fafb 0%%, #eef2f7 100%%);
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    padding: 24px;
+    line-height: 1.5;
+  }
+  .card {
+    background: #fff;
+    border-radius: 12px;
+    box-shadow: 0 1px 3px rgba(0,0,0,0.05), 0 12px 32px rgba(31,41,55,0.08);
+    max-width: 560px;
+    width: 100%%;
+    padding: 36px 40px;
+  }
+  .badge {
+    display: inline-block;
+    background: #fef3c7;
+    color: #92400e;
+    font-size: 12px;
+    font-weight: 600;
+    letter-spacing: 0.04em;
+    text-transform: uppercase;
+    padding: 4px 10px;
+    border-radius: 999px;
+    margin-bottom: 16px;
+  }
+  h1 { font-size: 22px; margin: 0 0 12px; color: #111827; }
+  p { margin: 0 0 14px; color: #374151; }
+  .ref {
+    background: #f3f4f6;
+    border: 1px solid #e5e7eb;
+    border-radius: 6px;
+    padding: 12px 14px;
+    margin: 20px 0;
+    font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
+    font-size: 13px;
+    color: #111827;
+    word-break: break-all;
+  }
+  .ref-label {
+    display: block;
+    color: #6b7280;
+    font-size: 11px;
+    font-weight: 600;
+    letter-spacing: 0.05em;
+    text-transform: uppercase;
+    margin-bottom: 4px;
+    font-family: inherit;
+  }
+  .owner {
+    border-top: 1px solid #e5e7eb;
+    margin-top: 24px;
+    padding-top: 20px;
+    color: #4b5563;
+    font-size: 14px;
+  }
+  .owner h2 { font-size: 14px; font-weight: 600; color: #111827; margin: 0 0 8px; }
+  a {
+    color: #1d4ed8;
+    text-decoration: none;
+    border-bottom: 1px solid transparent;
+  }
+  a:hover, a:focus { border-bottom-color: #1d4ed8; }
+  .small { font-size: 12px; color: #6b7280; margin-top: 16px; }
+</style>
+</head>
+<body>
+  <main class="card" role="main">
+    <span class="badge">Access blocked</span>
+    <h1>Your request was blocked by our security filter</h1>
+    <p>The request to <strong>%[req.hdr(host)]</strong> looked suspicious to our web application firewall and was not delivered to the site.</p>
+    <p>This is automated. No one has reviewed the request yet.</p>
+
+    <div class="ref">
+      <span class="ref-label">Request reference</span>
+      %[unique-id]
+    </div>
+
+    <div class="owner">
+      <h2>Site owner?</h2>
+      <p>If you operate this site and believe this block is incorrect, please contact your hosting provider's support team and include the request reference above. They can look up exactly which rule fired and adjust it if it's a false positive.</p>
+    </div>
+
+    <p class="small">Reference IDs expire from our active logs after 14 days, so please open a ticket promptly if you'd like this investigated.</p>
+  </main>
+</body>
+</html>

haproxy_tarpit_config.txt

@@ -1,3 +1,37 @@
+# =============================================================================
+# NOT IMPLEMENTED. THIS FILE IS A DESIGN SKETCH THAT WAS NEVER SHIPPED.
+# =============================================================================
+#
+# Nothing here is deployed, has ever been deployed, or is rendered into
+# haproxy.cfg. The real edge config is generated from templates/*.tpl. Compare:
+#
+#   THIS FILE proposes:  store gpc0,gpc1,gpc2,http_err_rate(30s),...
+#                        plus sc-inc-gpc0/1/2 scan-escalation rules
+#   templates/hap_listener.tpl ACTUALLY has:
+#                        store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(30s)
+#
+# There is no gpc0, no gpc1, no gpc2, and no scan-escalation state anywhere on
+# the edge, and never has been.
+#
+# WHY THE BANNER. This file was mistaken for the shipped config. Two consumers
+# -- /api/security/stats in haproxy_manager.py and scripts/show-tarpit-ips.sh --
+# were written to parse gpc0/gpc1 out of `show table web`, defaulting missing
+# fields to 0. The result was a "Scan Count" column and BLOCKED/TARPITTED
+# statuses that were pure fabrication, presented to an operator as fact, for
+# the entire life of both tools. Fixed 2026-08-22; scripts/test-stick-table-
+# contract.py now fails if any consumer's field expectations and the templates'
+# `store` clauses ever drift apart again.
+#
+# IF YOU WANT TO REVIVE ANY OF THIS: the counters must be added to a template
+# `store` clause and to STICK_TABLE_FIELD_CONTRACT in haproxy_manager.py first.
+# Adding them to a consumer alone produces confident zeros, not data. Note also
+# that sc0/sc1/sc2 are all in use and HAProxy's tune.stick-counters defaults to
+# 3, and that since 2026.08.8 the edge has real per-request access logging on
+# the host at /var/log/haproxy.log -- which records what was actually denied,
+# tarpitted and rate-limited, with request references. That log is a better
+# source for most of what this sketch was reaching for.
+# =============================================================================
+
 global
     daemon
     log stdout local0 info

requirements.txt

@@ -1,3 +1,7 @@
 Flask==2.3.3
 Jinja2==3.1.2
-psutil
+psutil
+# Production WSGI server. Replaces Flask's built-in werkzeug dev server, which
+# is single-threaded and leaks workers over long uptimes (root cause of the
+# 2026-05 haproxy-manager "healthy but stalled" incidents).
+gunicorn==23.0.0

scripts/cert-publish-lib.sh

@@ -0,0 +1,371 @@
+# shellcheck shell=bash
+# cert-publish-lib.sh - safe publication of HAProxy certificate bundles.
+#
+# This file is SOURCED, never executed (hence no shebang / no exec bit).
+#
+# Why this exists
+# ---------------
+# renew-certificates.sh and sync-certificates.sh used to publish a bundle with
+#
+#     cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"
+#
+# where $COMBINED_FILE is the LIVE pem HAProxy is serving right now. The shell
+# truncates the destination to zero bytes when it sets up the redirect, BEFORE
+# cat ever runs, so any failure after that point (unreadable source, ENOSPC,
+# container killed mid-write) leaves a zero-length or key-less pem in place.
+# Checking cat's exit status does not help: the damage is already done.
+#
+# That matters more here than for an ordinary file because HAProxy loads
+# $SSL_CERTS_DIR as a DIRECTORY:
+#
+#     bind 0.0.0.0:443 ssl crt /etc/haproxy/certs
+#
+# It tries to load *every* file in that directory, and one unloadable file
+# fails the whole bind - i.e. HTTPS goes down for every customer on the host.
+#
+# Two consequences drive the design below:
+#   1. Assemble somewhere else and rename into place, so the live pem is either
+#      the old bundle or the new one and never a half-written one.
+#   2. NEVER create a temp file, .tmp, .backup or any other non-final file
+#      inside $SSL_CERTS_DIR. Staging and backups live in SIBLING directories.
+#
+# Directory layout (kept identical to the Python half in haproxy_manager.py):
+#   staging: $(dirname $SSL_CERTS_DIR)/cert-staging   [$CERT_STAGING_DIR]
+#   backups: $(dirname $SSL_CERTS_DIR)/cert-backups   [$CERT_BACKUP_DIR]
+# Both siblings of the certs dir, so they are on the same filesystem and the
+# final mv is a rename(2) - atomic. There is deliberately no "just write it
+# directly into the certs dir" fallback path.
+#
+# The same-filesystem property is CHECKED, not assumed (see cert_publish step
+# (c)). An earlier revision of this comment claimed a cross-device mv would
+# "fail loudly and leave the live pem alone". It does not: GNU mv falls back to
+# copy-then-unlink across filesystems, so it OPENS THE DESTINATION FOR WRITING
+# and only then discovers it cannot finish - e.g. with a full destination
+# filesystem the live pem is already overwritten when mv reports failure. That
+# is precisely the truncation this library exists to prevent, so the device
+# numbers of the staging dir and the certs dir are compared with stat(1) before
+# anything is written, and a mismatch aborts the publish. Both directories are
+# env-overridable ($CERT_STAGING_DIR / $SSL_CERTS_DIR), so "they are siblings"
+# is not something the code can take on faith. This mirrors the explicit
+# st_dev check in publish_pem_bundle() on the Python side.
+
+# Logging: the callers define their own log_info/log_error. Only provide
+# fallbacks so this library is usable standalone (e.g. from a test or a shell).
+declare -F log_info >/dev/null || log_info() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [INFO] $*"
+}
+declare -F log_error >/dev/null || log_error() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" >&2
+}
+# log_warn is not part of the callers' vocabulary; route it through log_info
+# with a loud prefix so it lands in the main log without tripping the
+# error-log monitors (scripts/monitor-errors.sh) for non-fatal conditions.
+declare -F log_warn >/dev/null || log_warn() {
+    log_info "WARNING: $*"
+}
+
+cert_staging_dir() {
+    if [ -n "${CERT_STAGING_DIR:-}" ]; then
+        echo "$CERT_STAGING_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-staging"
+    fi
+}
+
+cert_backup_dir() {
+    if [ -n "${CERT_BACKUP_DIR:-}" ]; then
+        echo "$CERT_BACKUP_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-backups"
+    fi
+}
+
+# cert_bundle_valid FILE
+#
+# Returns 0 if FILE is publishable as an HAProxy pem bundle.
+#
+# Layer 1 (structure, pure shell/grep): covers the truncation / partial-write /
+#         key-less failure modes this library exists to prevent.
+# Layer 2 (cryptographic pairing via the openssl CLI): covers what layer 1
+#         cannot see. Structural checks are weak on their own - a bundle of
+#         EMPTY pem blocks ("-----BEGIN CERTIFICATE-----" immediately followed
+#         by "-----END CERTIFICATE-----") satisfies every grep below and is
+#         caught only by openssl.
+#
+# BOTH LAYERS ARE MANDATORY. An absent openssl binary is a hard failure, not a
+# skip.
+#
+# The previous revision made layer 2 best-effort and justified it with "the
+# container image installs haproxy, certbot and socat, but not necessarily the
+# openssl CLI". That premise is false. openssl 3.x is present in the image: it
+# is a dependency of ca-certificates, which certbot needs, and
+# generate_self_signed_cert() in haproxy_manager.py shells out to `openssl req`
+# with check=True during first-run setup, so a container that reached the point
+# of publishing a bundle has always had it. The "unavailable" branch therefore
+# never fired in production, which means the fail-open was safe only by
+# accident - and a comment that justifies a decision on a false premise is
+# worse than no comment, because the next person extends the reasoning.
+#
+# Making it mandatory does mean a hypothetical image without openssl stops
+# publishing renewals. That is the right trade: it fails immediately and
+# loudly, into the monitored error log, on the first renewal run, whereas
+# publishing an unpaired or empty-block bundle takes the whole :443 bind (i.e.
+# every site on the host) down at the next reload. There is deliberately no
+# python `cryptography` fallback: the app runs on /usr/local/bin/python3 (the
+# base image's 3.12), where cryptography is NOT importable - it is installed
+# for Debian's /usr/bin/python3 as a certbot dependency. Coding to a hardcoded
+# /usr/bin/python3 would just be a second unverified premise.
+cert_bundle_valid() {
+    local file="$1"
+
+    if [ -z "$file" ]; then
+        log_error "cert_bundle_valid: no file given"
+        return 1
+    fi
+    if [ ! -f "$file" ]; then
+        log_error "Certificate bundle $file does not exist (or is not a regular file)"
+        return 1
+    fi
+
+    # openssl is checked BEFORE any content check so a broken image is reported
+    # as a broken image rather than as a bad certificate.
+    if ! command -v openssl >/dev/null 2>&1; then
+        log_error "openssl binary not found - REFUSING to publish $file." \
+                  "The cert/key pairing check (openssl x509 -pubkey vs openssl pkey -pubout)" \
+                  "is mandatory; structural checks alone cannot tell a real bundle from" \
+                  "empty pem blocks. Install openssl in this image."
+        return 1
+    fi
+
+    # Read the file ONCE and run every check against that snapshot.
+    #
+    # This used to open $file six times (two [ ] tests, three greps, two
+    # openssl invocations). cert_bundle_valid() is called on the LIVE pem in
+    # cert_publish() step (d), where a concurrent publisher can replace it
+    # between two of those opens - each open then sees a different file. The
+    # observable symptom was a spurious "private key does not match the
+    # certificate" ERROR in the monitored error log for a pair that was fine:
+    # openssl x509 read the old bundle and openssl pkey the new one.
+    local content
+    if ! content="$(cat -- "$file" 2>/dev/null)"; then
+        log_error "Certificate bundle $file could not be read"
+        return 1
+    fi
+    if [ -z "$content" ]; then
+        log_error "Certificate bundle $file is empty"
+        return 1
+    fi
+
+    # --- layer 1: structure -------------------------------------------------
+    if ! grep -qF -- '-----BEGIN CERTIFICATE-----' <<< "$content"; then
+        log_error "Certificate bundle $file contains no certificate block"
+        return 1
+    fi
+    if ! grep -qF -- '-----END CERTIFICATE-----' <<< "$content"; then
+        log_error "Certificate bundle $file has an unterminated certificate block (truncated?)"
+        return 1
+    fi
+
+    local key_begin key_end
+    key_begin="$(grep -m1 -oE -- '-----BEGIN (RSA |EC )?PRIVATE KEY-----' <<< "$content")"
+    if [ -z "$key_begin" ]; then
+        log_error "Certificate bundle $file contains no private key block"
+        return 1
+    fi
+    key_end="${key_begin/BEGIN/END}"
+    if ! grep -qF -- "$key_end" <<< "$content"; then
+        log_error "Certificate bundle $file has an unterminated private key block (truncated?)"
+        return 1
+    fi
+
+    # --- layer 2: cert/key pairing ------------------------------------------
+    # Fed from the same snapshot on stdin (openssl reads stdin when -in is
+    # omitted) rather than re-opening $file, so layer 2 judges exactly the
+    # bytes layer 1 judged. -passin pass: means an encrypted key fails fast
+    # instead of prompting - a passphrase prompt in a cron job is a hang, not
+    # an error.
+    local cert_pub key_pub
+    if ! cert_pub="$(openssl x509 -noout -pubkey 2>/dev/null <<< "$content")" \
+       || [ -z "$cert_pub" ]; then
+        log_error "Certificate bundle $file: openssl could not read the certificate"
+        return 1
+    fi
+    if ! key_pub="$(openssl pkey -pubout -passin pass: 2>/dev/null <<< "$content")" \
+       || [ -z "$key_pub" ]; then
+        log_error "Certificate bundle $file: openssl could not read the private key"
+        return 1
+    fi
+    if [ "$cert_pub" != "$key_pub" ]; then
+        log_error "Certificate bundle $file: private key does not match the certificate"
+        return 1
+    fi
+
+    return 0
+}
+
+# cert_publish CERT_FILE KEY_FILE DEST_FILE
+#
+# Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
+# partially written DEST_FILE to HAProxy. Returns 0 on success.
+#
+# On ANY failure DEST_FILE is left exactly as it was.
+cert_publish() {
+    if [ $# -ne 3 ]; then
+        log_error "cert_publish: expected 3 arguments (cert key dest), got $#"
+        return 1
+    fi
+
+    local cert_file="$1" key_file="$2" dest_file="$3"
+    local staging_dir backup_dir dest_dir tmp base
+
+    # (a) sources must exist and be non-empty before we touch anything.
+    if [ ! -s "$cert_file" ]; then
+        log_error "cert_publish: certificate $cert_file is missing or empty"
+        return 1
+    fi
+    if [ ! -s "$key_file" ]; then
+        log_error "cert_publish: private key $key_file is missing or empty"
+        return 1
+    fi
+
+    # (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
+    #     scans wholesale.
+    staging_dir="$(cert_staging_dir)"
+    dest_dir="$(dirname "$dest_file")"
+    if ! mkdir -p "$staging_dir" || ! mkdir -p "$dest_dir"; then
+        log_error "cert_publish: cannot create staging directory $staging_dir or destination directory $dest_dir"
+        return 1
+    fi
+
+    # (c) the final swap is `mv`, which is only a rename(2) - and therefore only
+    #     atomic - within one filesystem. Across filesystems GNU mv copies:
+    #     it truncates and writes the DESTINATION, then unlinks the source, so a
+    #     failure part-way through (ENOSPC is the realistic one) leaves exactly
+    #     the half-written live pem this library exists to prevent. Both paths
+    #     are env-overridable, so check instead of assuming. Same check as the
+    #     st_dev comparison in publish_pem_bundle() on the Python side.
+    local staging_dev dest_dev
+    staging_dev="$(stat -Lc '%d' "$staging_dir" 2>/dev/null)"
+    dest_dev="$(stat -Lc '%d' "$dest_dir" 2>/dev/null)"
+    if [ -z "$staging_dev" ] || [ -z "$dest_dev" ]; then
+        log_error "cert_publish: cannot stat $staging_dir and/or $dest_dir - refusing to publish $dest_file"
+        return 1
+    fi
+    if [ "$staging_dev" != "$dest_dev" ]; then
+        log_error "cert_publish: staging dir $staging_dir and destination dir $dest_dir" \
+                  "are on different filesystems, so the bundle cannot be swapped in atomically." \
+                  "Refusing to publish $dest_file (live file left untouched);" \
+                  "point CERT_STAGING_DIR at a directory on the same filesystem as $dest_dir"
+        return 1
+    fi
+
+    # Sweep temps orphaned by a kill -9 / OOM in an earlier run. Two shapes,
+    # because two publishers share this directory: mktemp's six-X suffix from
+    # this library, and `<name>.<random>.tmp` from write_config_atomically() on
+    # the Python side (tempfile.mkstemp(prefix=name + '.', suffix='.tmp')).
+    # Matching only the mktemp shape - as this did - left every Python-side
+    # temp behind forever.
+    find "$staging_dir" -maxdepth 1 -type f \
+        \( -name '*.??????' -o -name '*.tmp' \) -mmin +1440 -delete 2>/dev/null
+
+    base="$(basename "$dest_file")"
+    tmp="$(mktemp "${staging_dir}/${base}.XXXXXX" 2>/dev/null)"
+    if [ -z "$tmp" ] || [ ! -f "$tmp" ]; then
+        log_error "cert_publish: cannot create a staging file in $staging_dir"
+        return 1
+    fi
+    # 0600 while the temp file holds a private key; the final mode is matched
+    # to the file being replaced just before the swap (see below).
+    chmod 600 "$tmp" 2>/dev/null
+
+    if ! cat "$cert_file" "$key_file" > "$tmp"; then
+        log_error "cert_publish: failed to assemble $cert_file + $key_file (live $dest_file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+    if [ ! -s "$tmp" ]; then
+        log_error "cert_publish: assembled bundle for $dest_file is empty (live file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    # (d) never promote something HAProxy would choke on.
+    if ! cert_bundle_valid "$tmp"; then
+        log_error "cert_publish: assembled bundle for $dest_file failed validation (live file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    # (e) back up the bundle we are about to replace - but only if it is itself
+    #     valid. Overwriting a good backup with garbage would turn "restore the
+    #     backup" into "restore a different broken file". Same semantics as
+    #     create_backup(require_valid=True) in haproxy_manager.py.
+    if [ -e "$dest_file" ]; then
+        backup_dir="$(cert_backup_dir)"
+        if cert_bundle_valid "$dest_file"; then
+            if mkdir -p "$backup_dir"; then
+                if ! cp -p "$dest_file" "${backup_dir}/${base}"; then
+                    log_warn "could not back up $dest_file to ${backup_dir}/${base}; publishing anyway"
+                fi
+            else
+                log_warn "could not create backup directory $backup_dir; publishing without a backup"
+            fi
+        else
+            log_warn "existing $dest_file is not a valid bundle - KEEPING the previous backup" \
+                     "in $backup_dir rather than overwriting it with an unusable one"
+        fi
+    fi
+
+    # Preserve the mode of the bundle being replaced (0644 by default, which is
+    # what `cat > file` produced under the standard umask). mktemp gives 0600,
+    # and mv carries the temp file's mode onto the destination, so without this
+    # every publish would silently tighten the live pem's permissions. Changing
+    # who can read these files is not something a write-safety fix should do as
+    # a side effect - and it must match write_config_atomically() on the Python
+    # side, which preserves the mode the same way.
+    #
+    # -L (follow symlinks) matters: stat without it reports the mode of the
+    # SYMLINK, which is 0777 on Linux and is not a permission at all. A live
+    # pem that is a symlink therefore produced a world-WRITABLE 0777 private
+    # key sitting in the crt directory. With -L we copy the mode of the file
+    # the link points at, which is the mode an operator actually chose.
+    local mode
+    mode="$(stat -Lc '%a' "$dest_file" 2>/dev/null)"
+    [ -n "$mode" ] || mode=644
+    chmod "$mode" "$tmp" 2>/dev/null
+
+    # (f) atomic swap. Same filesystem, verified in (c); if it still fails,
+    #     stop - do not fall back to writing into the certs dir.
+    if ! mv -f "$tmp" "$dest_file"; then
+        log_error "cert_publish: failed to move $tmp into place as $dest_file" \
+                  "(live file left untouched; NOT falling back to a direct write)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    return 0
+}
+
+# haproxy_config_ok
+#
+# Gate a reload on `haproxy -c`. Returns 0 if the config validates, or if we
+# cannot check (no haproxy binary) - a missing checker must not block a reload
+# that is otherwise needed, but a checker that says "no" always wins.
+haproxy_config_ok() {
+    local cfg="${HAPROXY_CONFIG:-/etc/haproxy/haproxy.cfg}"
+    local out rc
+
+    if ! command -v haproxy >/dev/null 2>&1; then
+        log_warn "haproxy binary not found - skipping 'haproxy -c' validation before reload"
+        return 0
+    fi
+
+    out="$(haproxy -c -f "$cfg" 2>&1 </dev/null)"
+    rc=$?
+    if [ $rc -eq 0 ]; then
+        return 0
+    fi
+
+    log_error "haproxy -c -f $cfg failed (exit $rc): $out"
+    return 1
+}

scripts/ensure_haproxy.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python3
+"""Idempotent haproxy liveness check — driven by the in-container supervisor loop.
+
+Why this exists
+---------------
+haproxy runs as a *background child of PID 1* (gunicorn) — it is started once at
+container init (scripts/init.py -> do_initial_setup -> start_haproxy) and then
+left running. Nothing supervises it after that. If the haproxy master process
+dies mid-life (SIGABRT -> exit 134, segfault, or an OOM of the haproxy master),
+the container stays "up" because gunicorn is still PID 1, so Docker's
+`--restart` policy never fires. haproxy then stays down until the *external*
+host watchdog (haproxy-watchdog.sh) notices port 80 is dead for ~3 minutes and
+does a full `docker restart` — which drops every in-flight connection.
+
+This script closes that gap: called on a short interval by the supervisor loop
+in start-up.sh, it re-launches haproxy *in place* within one interval.
+
+Safety
+------
+start_haproxy() is guarded by `is_process_running('haproxy')` (psutil-based, so
+it works in this container which has no `ps`), so calling this while haproxy is
+healthy is a cheap no-op. It only ever acts when haproxy is genuinely gone.
+"""
+import sys
+
+sys.path.insert(0, '/haproxy')
+import haproxy_manager  # noqa: E402  (sys.path manipulation must come first)
+
+
+def main():
+    if haproxy_manager.is_process_running('haproxy'):
+        return 0
+
+    haproxy_manager.logger.warning(
+        "[haproxy-supervisor] haproxy process not found — attempting in-place restart"
+    )
+    # start_haproxy() validates the config (and regenerates it if invalid)
+    # before launching, and swallows its own errors, so it will not raise here.
+    haproxy_manager.start_haproxy()
+
+    if haproxy_manager.is_process_running('haproxy'):
+        haproxy_manager.logger.info("[haproxy-supervisor] haproxy restarted in place")
+        return 0
+
+    haproxy_manager.logger.error(
+        "[haproxy-supervisor] haproxy restart FAILED — still not running after start_haproxy()"
+    )
+    return 1
+
+
+if __name__ == '__main__':
+    sys.exit(main())

scripts/init.py

@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+"""Container init: DB schema, certbot account, config generation, HAProxy start.
+
+Runs once per container start, BEFORE gunicorn workers spawn. Keeping init out
+of the WSGI app's module-load path avoids fork-time races (multiple workers
+attempting to start_haproxy() simultaneously, certbot lock contention, etc.).
+"""
+import sys
+
+sys.path.insert(0, '/haproxy')
+import haproxy_manager  # noqa: E402  (sys.path manipulation must come first)
+
+haproxy_manager.do_initial_setup()

scripts/manage-blocked-ips.sh

@@ -6,22 +6,40 @@
 SOCKET="/tmp/haproxy-cli"
 MAP_FILE="/etc/haproxy/blocked_ips.map"
 
+# HAProxy runs in master-worker mode here, and /tmp/haproxy-cli is the MASTER
+# socket. Data-plane commands (map/table manipulation) are NOT accepted on the
+# master socket — they must be routed to a worker with the "@<n>" prefix. "@1"
+# targets the current active worker. (A bare "add map ..." on the master socket
+# fails with "Unknown command: 'add'".)
+cli() { printf '@1 %s\n' "$*" | socat stdio "$SOCKET"; }
+
+# Map lookup in haproxy.cfg is `map_ip(...,0) -m int gt 0`, so each entry MUST be
+# "<ip_or_cidr> 1" — a bare IP yields an empty value (0) and is NOT blocked once
+# the map file is re-read on reload. The runtime map and the file must agree.
+MAP_VALUE=1
+
 # Ensure map file exists
 if [ ! -f "$MAP_FILE" ]; then
-    touch "$MAP_FILE"
-    echo "# Blocked IPs - Format: IP_ADDRESS" > "$MAP_FILE"
+    echo "# Blocked IPs - Format: <ip_or_cidr> 1 (one per line)" > "$MAP_FILE"
 fi
 
+# Escape regex metacharacters (notably dots) in an IP/CIDR for anchored matching.
+esc_re() { printf '%s' "$1" | sed 's/[.[\*^$/]/\\&/g'; }
+
 case "$1" in
     block)
         if [ -z "$2" ]; then
             echo "Usage: $0 block IP_ADDRESS"
             exit 1
         fi
-        # Add IP to map file
-        grep -q "^$2" "$MAP_FILE" || echo "$2" >> "$MAP_FILE"
-        # Add to runtime map
-        echo "add map /etc/haproxy/blocked_ips.map $2 1" | socat stdio "$SOCKET"
+        re="$(esc_re "$2")"
+        # Persist (idempotent, anchored so 1.2.3.4 doesn't match 1.2.3.45),
+        # always with the trailing value so the block survives a reload.
+        if ! grep -qE "^${re}([[:space:]]|$)" "$MAP_FILE"; then
+            echo "$2 $MAP_VALUE" >> "$MAP_FILE"
+        fi
+        # Apply at runtime immediately (no reload).
+        cli "add map $MAP_FILE $2 $MAP_VALUE"
         echo "Blocked IP: $2"
         ;;
 
@@ -30,31 +48,33 @@ case "$1" in
             echo "Usage: $0 unblock IP_ADDRESS"
             exit 1
         fi
-        # Remove from map file
-        sed -i "/^$2$/d" "$MAP_FILE"
-        # Remove from runtime map
-        echo "del map /etc/haproxy/blocked_ips.map $2" | socat stdio "$SOCKET"
+        re="$(esc_re "$2")"
+        # Remove from map file (match "<ip>" optionally followed by a value).
+        sed -i -E "/^${re}([[:space:]]|$)/d" "$MAP_FILE"
+        # Remove from runtime map.
+        cli "del map $MAP_FILE $2"
         echo "Unblocked IP: $2"
         ;;
 
     list)
         echo "Currently blocked IPs:"
-        echo "show map /etc/haproxy/blocked_ips.map" | socat stdio "$SOCKET" | awk '{print $1}'
+        # `show map` output is "<ptr> <key> <value>" — the IP is field 2.
+        cli "show map $MAP_FILE" | awk 'NF>=2 {print $2}'
         ;;
 
     clear)
         echo "Clearing all blocked IPs..."
-        echo "clear map /etc/haproxy/blocked_ips.map" | socat stdio "$SOCKET"
-        echo "# Blocked IPs - Format: IP_ADDRESS" > "$MAP_FILE"
+        cli "clear map $MAP_FILE"
+        echo "# Blocked IPs - Format: <ip_or_cidr> 1 (one per line)" > "$MAP_FILE"
         echo "All IPs unblocked"
         ;;
 
     stats)
         echo "=== HAProxy 3.0.11 Threat Intelligence Dashboard ==="
-        echo "show table web" | socat stdio "$SOCKET" | awk 'NR<=21'
+        cli "show table web" | awk 'NR<=21'
         echo ""
         echo "=== Top Threat Scores ==="
-        echo "show table web" | socat stdio "$SOCKET" | awk '
+        cli "show table web" | awk '
         NR>1 {
             ip = $1
             auth_fail = 0
@@ -84,7 +104,7 @@ case "$1" in
             exit 1
         fi
         # Add to manual blacklist using GPC(13)
-        echo "set table web key $2 data.gpc(13) 1" | socat stdio "$SOCKET"
+        cli "set table web key $2 data.gpc(13) 1"
         echo "Manually blacklisted IP: $2 (GPC(13) = 1)"
         ;;
 
@@ -94,7 +114,7 @@ case "$1" in
             exit 1
         fi
         # Clear manual blacklist flag
-        echo "set table web key $2 data.gpc(13) 0" | socat stdio "$SOCKET"
+        cli "set table web key $2 data.gpc(13) 0"
         echo "Removed manual blacklist for IP: $2"
         ;;
 
@@ -104,7 +124,7 @@ case "$1" in
             exit 1
         fi
         # Add to auto-blacklist using GPC(14)
-        echo "set table web key $2 data.gpc(14) 1" | socat stdio "$SOCKET"
+        cli "set table web key $2 data.gpc(14) 1"
         echo "Auto-blacklisted IP: $2 (GPC(14) = 1)"
         ;;
 
@@ -115,14 +135,14 @@ case "$1" in
         fi
         # Show detailed threat breakdown for specific IP
         echo "Threat analysis for $2:"
-        echo "show table web key $2" | socat stdio "$SOCKET"
+        cli "show table web key $2"
         ;;
 
     *)
         echo "Usage: $0 {block|unblock|list|clear|blacklist|unblacklist|auto-blacklist|threat-score|stats} [IP_ADDRESS]"
         echo ""
         echo "HAProxy 3.0.11 Enhanced Security Commands:"
-        echo "  block IP         - Block IP via map file (immediate)"
+        echo "  block IP         - Block IP via map file (immediate + persisted)"
         echo "  unblock IP       - Unblock IP from map file"
         echo "  blacklist IP     - Manual blacklist via GPC(13) array"
         echo "  unblacklist IP   - Remove manual blacklist flag"
@@ -141,4 +161,4 @@ case "$1" in
         echo "  gpc(14): Auto-blacklist candidate           × 50"
         exit 1
         ;;
-esac
+esac

scripts/monitor-attacks.sh

@@ -1,136 +1,148 @@
-#!/bin/bash
+#!/usr/bin/env bash
+#
+# monitor-attacks.sh — HAProxy edge activity monitor.
+#
+# Two sections, both fed from real data:
+#   1. Current per-IP rates, from the `web` stick table (delegated to
+#      show-edge-ip-rates.sh — there is exactly one stick-table parser).
+#   2. Recent enforcement events, from the HAProxy access log.
+#
+# HISTORY / WHY THIS IS SHORTER THAN IT USED TO BE
+#   The previous version printed a "Threat Intelligence Dashboard" with
+#   fourteen categories (auth_fail, authz_fail, scanner, sql_inj, traversal,
+#   wp_brute, admin_scan, shell_att, repeat_off, manual_bl, auto_bl,
+#   glitch_rate, ...) and a composite "threat score", all parsed out of
+#   gpc(0), gpc(1), gpc(3), gpc(12), gpc(13) and glitch_rate(300s). NONE of
+#   those fields exist: the `web` table stores only conn_cur, conn_rate,
+#   http_req_rate and http_err_rate. Every category was permanently 0 and the
+#   whole dashboard printed nothing while implying it was watching. All of it
+#   has been deleted rather than "fixed" — there was no data source to fix it
+#   against.
+#
+# Usage: monitor-attacks.sh [live]
+# Env:   LOG_FILE=<path>  access log to read (default /var/log/haproxy.log)
+#        LOG_LINES=<n>    how many trailing log lines to scan (default 500)
 
-# Real-time attack monitoring for HAProxy
-# Shows blocked requests and suspicious activity
+set -uo pipefail
 
-LOG_FILE="/var/log/haproxy.log"
-SOCKET="/tmp/haproxy-cli"
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+LOG_FILE="${LOG_FILE:-/var/log/haproxy.log}"
+LOG_LINES="${LOG_LINES:-500}"
 
-echo "==================================================="
-echo "HAProxy Security Monitor - Real-time Attack Detection"
-echo "==================================================="
-echo ""
-
-# Function to show current threats with HAProxy 3.0.11 metrics
-show_threats() {
-    echo "HAProxy 3.0.11 Threat Intelligence Dashboard:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 {
-            # Parse the stick table output for array-based GPC values
-            ip = $1
-            # Look for GPC array values in the data
-            auth_fail = 0
-            authz_fail = 0
-            rate_viol = 0
-            scanner = 0
-            sql_inj = 0
-            traversal = 0
-            wp_brute = 0
-            admin_scan = 0
-            shell_att = 0
-            repeat_off = 0
-            manual_bl = 0
-            auto_bl = 0
-            glitch_rate = 0
-            threat_score = 0
-
-            # Extract relevant metrics (simplified parsing)
-            if ($0 ~ /gpc\(0\)=([0-9]+)/) {
-                match($0, /gpc\(0\)=([0-9]+)/, arr); auth_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(1\)=([0-9]+)/) {
-                match($0, /gpc\(1\)=([0-9]+)/, arr); authz_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(3\)=([0-9]+)/) {
-                match($0, /gpc\(3\)=([0-9]+)/, arr); scanner = arr[1]
-            }
-            if ($0 ~ /gpc\(12\)=([0-9]+)/) {
-                match($0, /gpc\(12\)=([0-9]+)/, arr); repeat_off = arr[1]
-            }
-            if ($0 ~ /gpc\(13\)=([0-9]+)/) {
-                match($0, /gpc\(13\)=([0-9]+)/, arr); manual_bl = arr[1]
-            }
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr); glitch_rate = arr[1]
-            }
-
-            # Calculate composite threat score (simplified)
-            threat_score = auth_fail*10 + authz_fail*8 + scanner*12 + repeat_off*25 + manual_bl*100
-
-            # Only show IPs with significant threat indicators
-            if (auth_fail > 0 || authz_fail > 0 || scanner > 0 || repeat_off > 0 || manual_bl > 0 || glitch_rate > 0) {
-                threat_level = "LOW"
-                if (threat_score >= 100) threat_level = "CRITICAL"
-                else if (threat_score >= 50) threat_level = "HIGH"
-                else if (threat_score >= 20) threat_level = "MEDIUM"
-
-                printf "%-15s [%8s] Score:%-3d Auth:%-2d Authz:%-2d Scanner:%-1d Repeat:%-1d Glitch:%-2d\n",
-                       ip, threat_level, threat_score, auth_fail, authz_fail, scanner, repeat_off, glitch_rate
-            }
-        }' | head -15
-
-    echo ""
-    echo "Top HTTP/2 Protocol Violators:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 && $0 ~ /glitch/ {
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr)
-                if (arr[1] > 2) {
-                    printf "%-15s glitch_rate:%-3s\n", $1, arr[1]
-                }
-            }
-        }' | head -5
-    echo "---------------------------------------------------"
+# --- Section 1: current rates (real stick-table data) -----------------------
+show_rates() {
+    "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
 }
 
-# Function to show recent blocks
+# --- Section 2: recent enforcement events (real access-log data) ------------
 show_recent_blocks() {
-    echo "Recent Blocked Requests:"
-    tail -100 "$LOG_FILE" 2>/dev/null | \
-        grep -E "(bot_scanner|scan_admin|scan_shells|sql_injection|directory_traversal|rate_abuse|tarpit|denied|403)" | \
-        tail -10 | \
-        awk '{
-            if (match($0, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+/)) {
-                ip = substr($0, RSTART, RLENGTH)
-                gsub(/:.*/, "", ip)
-                reason = ""
-                if ($0 ~ /bot_scanner/) reason = "BOT_SCANNER"
-                else if ($0 ~ /scan_admin/) reason = "ADMIN_SCAN"
-                else if ($0 ~ /scan_shells/) reason = "SHELL_SCAN"
-                else if ($0 ~ /sql_injection/) reason = "SQL_INJECTION"
-                else if ($0 ~ /directory_traversal/) reason = "DIR_TRAVERSAL"
-                else if ($0 ~ /rate_abuse/) reason = "RATE_ABUSE"
-                else if ($0 ~ /tarpit/) reason = "TARPIT"
-                else if ($0 ~ /denied/) reason = "DENIED"
-                else if ($0 ~ /403/) reason = "BLOCKED"
-                printf "[%s] %-15s %s\n", strftime("%H:%M:%S"), ip, reason
+    echo "Recent enforcement events (last $LOG_LINES log lines):"
+    echo
+
+    if [ ! -f "$LOG_FILE" ] || [ ! -r "$LOG_FILE" ]; then
+        cat <<MSG
+  Access log not readable at: $LOG_FILE
+
+  This is expected INSIDE the haproxy-manager container: HAProxy logs to
+  syslog on the DOCKER HOST, and the file lives on the host, not in here.
+  Read it from the host instead:
+
+    grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+    grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limited / blocked
+    grep -a 'cip=<IP>'  /var/log/haproxy.log | tail -50     # one client IP
+    grep -a 'id=<uuid>' /var/log/haproxy.log                # one request reference
+                                                            # (the UUID on the block page)
+    tail -f /var/log/haproxy.log | grep -aE ' (PT|PR)--'    # live
+
+  Or point this script at a copy:  LOG_FILE=/path/to/haproxy.log $0
+MSG
+        echo
+        return 0
+    fi
+
+    printf "%-15s %-16s %-4s %-5s %-28s %s\n" "TIME" "CLIENT IP" "CODE" "TERM" "HOST" "REQUEST / REQUEST-ID"
+    printf "%s\n" "-----------------------------------------------------------------------------------------------------"
+
+    local found
+    found=$(tail -n "$LOG_LINES" "$LOG_FILE" 2>/dev/null | awk '
+    {
+        status = ""; term = ""; cip = ""; host = ""; id = ""; ts = ""; req = ""
+
+        # %tr is bracketed: [22/Aug/2026:10:11:12.345] -> keep HH:MM:SS
+        # No {n} interval expressions here: not every awk in a slim Debian
+        # image supports them. Spelled out instead.
+        if (match($0, /\[[0-9][0-9]\/[A-Za-z][A-Za-z][A-Za-z]\/[0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/)) {
+            ts = substr($0, RSTART + 13, 8)
+        }
+
+        # Anchor on the %TR/%Tw/%Tc/%Tr/%Ta timers block: %ST follows it, and
+        # the termination state (%tsc) is 4 fields further on (%B %CC %CS %tsc).
+        for (i = 1; i <= NF; i++) {
+            if ($i ~ /^[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+$/) {
+                status = $(i + 1)
+                term = $(i + 5)
+                break
             }
-        }'
-    echo ""
+        }
+
+        # Only enforcement outcomes: tarpit (PT--), deny (PR--), 403, 429.
+        if (!(term ~ /^PT/ || term ~ /^PR/ || status == "403" || status == "429")) next
+
+        for (i = 1; i <= NF; i++) {
+            if (substr($i, 1, 4) == "cip=")  cip  = substr($i, 5)
+            if (substr($i, 1, 5) == "host=") host = substr($i, 6)
+            if (substr($i, 1, 3) == "id=")   id   = substr($i, 4)
+        }
+
+        if (match($0, /"[A-Z]+ [^"]*"/)) {
+            req = substr($0, RSTART + 1, RLENGTH - 2)
+            if (length(req) > 42) req = substr(req, 1, 41) "..."
+        }
+
+        if (cip == "") cip = "-"
+        if (host == "") host = "-"
+        if (term == "") term = "-"
+        if (status == "") status = "-"
+        printf "%-15s %-16s %-4s %-5s %-28s %s\n", ts, cip, status, term, host, req
+        if (id != "" && id != "-") printf "%-15s %s\n", "", "  id=" id
+        n++
+    }
+    END { if (n == 0) print "(no tarpit/deny/403/429 events in the scanned window)" }
+    ')
+    printf '%s\n' "$found"
+    echo
+    echo "TERM = HAProxy termination state: PT-- tarpit, PR-- deny (incl. WAF/rate limit)."
+    echo "id=  = request reference; it is printed on the block page and is how a"
+    echo "       customer support ticket correlates to an exact request here."
 }
 
-# Monitor mode selection
-if [ "$1" == "live" ]; then
-    echo "Live monitoring mode - Press Ctrl+C to exit"
-    echo ""
+banner() {
+    echo "==================================================="
+    echo "HAProxy Edge Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
+    echo "==================================================="
+    echo
+}
 
+if [ "${1:-}" = "live" ]; then
+    echo "Live monitoring mode - Press Ctrl+C to exit"
     while true; do
         clear
-        echo "==================================================="
-        echo "HAProxy Security Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
-        echo "==================================================="
-        echo ""
-        show_threats
-        echo ""
+        banner
+        show_rates || true
+        echo
         show_recent_blocks
         sleep 5
     done
 else
-    # Single run mode
-    show_threats
-    echo ""
+    banner
+    rc=0
+    show_rates || rc=$?
+    echo
     show_recent_blocks
-    echo ""
-    echo "Tip: Run with 'live' parameter for continuous monitoring"
+    echo
+    echo "Tip: run with 'live' for a refreshing view."
     echo "Usage: $0 [live]"
-fi
+    # Propagate a stick-table read failure: if the rates section could not be
+    # produced, this run did NOT report what it claims to report.
+    exit "$rc"
+fi

scripts/renew-certificates.sh

@@ -8,6 +8,7 @@ LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
 ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
 DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
 SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
+LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
 
 # Logging functions
 log_info() {
@@ -18,14 +19,35 @@ log_error() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
 }
 
+# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
+# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
+# uses this script's logging rather than its own fallbacks.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=cert-publish-lib.sh
+if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
+    . "${SCRIPT_DIR}/cert-publish-lib.sh"
+else
+    log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
+    exit 1
+fi
+
 log_info "Starting certificate renewal process"
 
-# Run certbot renewal
-if certbot renew --quiet --no-random-sleep-on-renew; then
-    log_info "Certbot renewal completed"
+# Run certbot renewal — don't exit on failure, some certs may have
+# renewed successfully even if others failed (e.g., domain no longer
+# pointed here). Continue to copy/combine whatever succeeded.
+CERTBOT_OUTPUT=$(certbot renew --no-random-sleep-on-renew 2>&1)
+CERTBOT_EXIT=$?
+
+if [ $CERTBOT_EXIT -eq 0 ]; then
+    log_info "Certbot renewal completed successfully"
 else
-    log_error "Certbot renewal failed with exit code $?"
-    exit 1
+    log_error "Certbot renewal had failures (exit code $CERTBOT_EXIT):"
+    # Log the specific failures
+    echo "$CERTBOT_OUTPUT" | grep -E "Failed to renew|failure" | while read -r line; do
+        log_error "  $line"
+    done
+    log_info "Continuing to process successfully renewed certificates..."
 fi
 
 # Copy all certificates to HAProxy format
@@ -33,7 +55,7 @@ fi
 mkdir -p "$SSL_CERTS_DIR"
 
 # Get all SSL-enabled domains from database
-DOMAINS=$(find /etc/letsencrypt/live/ -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
+DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
 
 if [ -z "$DOMAINS" ]; then
     log_info "No SSL-enabled domains found"
@@ -45,13 +67,16 @@ UPDATED=0
 FAILED=0
 
 while read -r domain; do
-    CERT_FILE="/etc/letsencrypt/live/${domain}/fullchain.pem"
-    KEY_FILE="/etc/letsencrypt/live/${domain}/privkey.pem"
+    CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
+    KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
     if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
-        # Combine cert and key into single file for HAProxy
-        if cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"; then
+        # Assemble in a staging dir and rename into place. NEVER redirect into
+        # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
+        # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
+        # takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
+        if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
             log_info "Updated certificate for $domain"
             UPDATED=$((UPDATED + 1))
         else
@@ -68,6 +93,13 @@ log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
 
 # Reload HAProxy if any certificates were updated
 if [ $UPDATED -gt 0 ]; then
+    # Never reload onto unvalidated material: a reload that fails to load the
+    # certs directory drops HTTPS for every site on this host.
+    if ! haproxy_config_ok; then
+        log_error "HAProxy configuration does not validate - refusing to reload after certificate renewal"
+        exit 1
+    fi
+
     if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
         log_info "HAProxy reloaded successfully"
     else
@@ -76,5 +108,18 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# A per-domain publication failure is a real failure and must be reported as
+# one. This script used to `exit 0` no matter how many domains failed, so
+# "0 updated, 12 failed" - a host that has completely stopped publishing
+# renewals - looked identical to a clean run to cron, to
+# host-renew-certificates.sh (which branches on this exit code) and to any
+# external monitoring. The first visible symptom would have been certificates
+# expiring. The loop above deliberately continues past a failed domain so the
+# others still get published; the status is reported here instead.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate renewal process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate renewal process completed"
 exit 0

scripts/show-edge-ip-rates.sh

@@ -0,0 +1,314 @@
+#!/usr/bin/env bash
+#
+# show-edge-ip-rates.sh — real, current per-IP rate counters from the HAProxy
+# `web` stick table.
+#
+# WHAT THIS CAN TELL YOU
+#   The `web` stick table (templates/hap_listener.tpl) stores exactly four
+#   counters per client IP:
+#       conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
+#   Those are INSTANTANEOUS values — the current concurrency and the current
+#   sliding-window rates. This script prints them, and nothing else.
+#
+# WHAT THIS CANNOT TELL YOU
+#   * Who has been tarpitted, denied, or rate-limited. The stick table stores
+#     NO history and NO counter of past enforcement actions. It has no gpc0 /
+#     gpc1 / gpc(N) / gpc_rate / glitch_rate columns at all — any tool that
+#     claims to read them from this table is fabricating numbers.
+#   * Anything about an IP that has gone quiet: entries expire after 10m.
+#
+#   Real enforcement events live in the HAProxy ACCESS LOG, which is on the
+#   DOCKER HOST at /var/log/haproxy.log (it does NOT exist inside this
+#   container). The log-format carries the HAProxy termination state plus
+#   cip= (real client IP), host=, ua= and id= (the request UUID shown on the
+#   block page, which correlates with customer support tickets).
+#
+#   Ready to run ON THE HOST:
+#       # last 20 tarpitted (PT--) or denied (PR--) requests
+#       grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+#       # everything HAProxy answered 429/403 to, newest last
+#       grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20
+#       # everything for one client IP
+#       grep -a 'cip=203.0.113.7' /var/log/haproxy.log | tail -50
+#       # look up one request reference from a support ticket
+#       grep -a 'id=<uuid-from-the-block-page>' /var/log/haproxy.log
+#
+# USAGE
+#   show-edge-ip-rates.sh [-a|--all]
+#     -a, --all   also show rows whose counters are all zero (off by default:
+#                 a table with hundreds of idle entries is pure noise)
+#
+# ENVIRONMENT
+#   SHOW_ALL=1               same as --all
+#   HAPROXY_SOCKET=<path>    override the CLI socket (default /tmp/haproxy-cli)
+#   HAPROXY_TABLE_DUMP=<file>
+#                            parse a previously captured `show table web` dump
+#                            from a file instead of talking to the socket.
+#                            Supported seam for offline analysis of a captured
+#                            support bundle, and for testing this parser.
+#
+# NOTE ON THE SOCKET
+#   /tmp/haproxy-cli is HAProxy's MASTER CLI socket, so worker commands need an
+#   `@1` prefix. Without it HAProxy answers "Unknown command: 'show' ..." AND
+#   socat still exits 0 — so exit status is worthless here and this script
+#   inspects the RESPONSE BODY instead.
+
+set -euo pipefail
+
+SOCKET="${HAPROXY_SOCKET:-/tmp/haproxy-cli}"
+TABLE="web"
+
+# Fields this script expects the `web` stick table to store. Keep on ONE line
+# in this exact NAME=(a b c d) shape — the contract test greps for it, and the
+# parser below is driven entirely by it.
+EXPECTED_FIELDS=(conn_cur conn_rate http_req_rate http_err_rate)
+
+# Which of EXPECTED_FIELDS to sort on (descending). Falls back to the first
+# field if this name is not in the list.
+SORT_FIELD="http_req_rate"
+
+SHOW_ALL="${SHOW_ALL:-0}"
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -a|--all) SHOW_ALL=1 ;;
+        -h|--help) sed -n '2,60p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
+        *) echo "Unknown argument: $1" >&2; echo "Usage: $0 [-a|--all]" >&2; exit 2 ;;
+    esac
+    shift
+done
+
+die() { echo "ERROR: $*" >&2; exit 1; }
+
+# First non-blank line of a blob.
+#
+# Deliberately NOT `printf ... | sed -n '/./{p;q;}'`. sed quits after the first
+# match and closes the pipe; on a real 550-entry table dump printf is still
+# writing and takes SIGPIPE, so under `set -o pipefail` the whole command
+# substitution returns 141 and `set -e` kills the script -- silently, with no
+# output at all. That is the same class of failure this script exists to stop
+# hiding, so it does not get to happen here. A plain read loop has no pipeline
+# and no early close.
+first_nonblank() {
+    local line
+    while IFS= read -r line || [ -n "$line" ]; do
+        case "$line" in
+            *[![:space:]]*) printf '%s\n' "$line"; return 0 ;;
+        esac
+    done <<EOF
+$1
+EOF
+    return 0
+}
+
+# Return 0 if the CLI response body is a rejection rather than table data.
+# Checked on the body because socat's exit status is 0 either way.
+body_is_rejected() {
+    local first
+    first=$(first_nonblank "$1")
+    case "$first" in
+        "Unknown command"*|"No such table"*|"Permission denied"*) return 0 ;;
+        *) return 1 ;;
+    esac
+}
+
+send_cmd() {
+    printf '%s\n' "$1" | socat stdio "$SOCKET" 2>/dev/null
+}
+
+# ---------------------------------------------------------------- fetch data
+BODY=""
+SOURCE=""
+if [ -n "${HAPROXY_TABLE_DUMP:-}" ]; then
+    [ -r "$HAPROXY_TABLE_DUMP" ] || die "HAPROXY_TABLE_DUMP is set but '$HAPROXY_TABLE_DUMP' is not readable."
+    BODY=$(cat "$HAPROXY_TABLE_DUMP")
+    SOURCE="file $HAPROXY_TABLE_DUMP"
+else
+    [ -S "$SOCKET" ] || die "HAProxy CLI socket not found at $SOCKET (is HAProxy running, and are you inside the haproxy-manager container?)"
+    command -v socat >/dev/null 2>&1 || die "socat is not installed; cannot talk to $SOCKET"
+
+    # Master socket form first, then the plain stats-socket form.
+    BODY=$(send_cmd "@1 show table $TABLE" || true)
+    SOURCE="socket $SOCKET (@1 show table $TABLE)"
+    if [ -z "${BODY//[[:space:]]/}" ] || body_is_rejected "$BODY"; then
+        FALLBACK=$(send_cmd "show table $TABLE" || true)
+        if [ -n "${FALLBACK//[[:space:]]/}" ] && ! body_is_rejected "$FALLBACK"; then
+            BODY="$FALLBACK"
+            SOURCE="socket $SOCKET (show table $TABLE)"
+        else
+            echo "ERROR: HAProxy rejected BOTH '@1 show table $TABLE' and 'show table $TABLE'." >&2
+            echo "  @1 response : $(first_nonblank "$BODY")" >&2
+            echo "  bare response: $(first_nonblank "$FALLBACK")" >&2
+            echo "  Check the socket is HAProxy's CLI and that the table '$TABLE' exists" >&2
+            echo "  (a config reload without the frontend would drop it)." >&2
+            exit 1
+        fi
+    fi
+fi
+
+# ------------------------------------------------------------- header checks
+HEADER=$(first_nonblank "$BODY")
+case "$HEADER" in
+    "# table: $TABLE,"*) : ;;
+    *)
+        echo "ERROR: unexpected first line from '$SOURCE'." >&2
+        echo "  expected it to start with: # table: $TABLE," >&2
+        echo "  got                      : $HEADER" >&2
+        exit 1
+        ;;
+esac
+
+TBL_SIZE=$(printf '%s\n' "$HEADER" | sed -n 's/.*size:\([0-9]*\).*/\1/p')
+TBL_USED=$(printf '%s\n' "$HEADER" | sed -n 's/.*used:\([0-9]*\).*/\1/p')
+[ -n "$TBL_SIZE" ] || TBL_SIZE="?"
+[ -n "$TBL_USED" ] || TBL_USED="?"
+
+# ------------------------------------------------------------------- parsing
+# awk emits:
+#   W \t <field>:<window-seconds-or-dash> ...       (one line, from first row)
+#   R \t <sortkey> \t <ip> \t <value per EXPECTED_FIELDS in order>
+# and exits 1 after reporting any row missing an expected field.
+PARSED=""
+if ! PARSED=$(printf '%s\n' "$BODY" | awk -v fieldlist="${EXPECTED_FIELDS[*]}" -v sortfield="$SORT_FIELD" '
+BEGIN {
+    nf = split(fieldlist, F, " ")
+    sortidx = 1
+    for (i = 1; i <= nf; i++) if (F[i] == sortfield) sortidx = i
+    wprinted = 0
+}
+/^#/ { next }
+!/key=/ { next }
+{
+    split("", val, " "); split("", win, " ")
+    for (i = 1; i <= NF; i++) {
+        tok = $i
+        p = index(tok, "=")
+        if (p == 0) continue
+        lhs = substr(tok, 1, p - 1)
+        rhs = substr(tok, p + 1)
+        b = index(lhs, "(")
+        if (b > 0) {
+            nm = substr(lhs, 1, b - 1)
+            win[nm] = substr(lhs, b + 1, length(lhs) - b - 1)
+        } else {
+            nm = lhs
+            win[nm] = ""
+        }
+        val[nm] = rhs
+    }
+
+    missing = ""
+    for (i = 1; i <= nf; i++) if (!(F[i] in val)) missing = missing (missing == "" ? "" : ", ") F[i]
+    if (missing != "") {
+        printf "ERROR: stick table row is missing expected field(s): %s\n", missing > "/dev/stderr"
+        printf "  offending row: %s\n", $0 > "/dev/stderr"
+        printf "  this script expects the web table to store: %s\n", fieldlist > "/dev/stderr"
+        print  "  Those expectations and the templates/hap_listener.tpl `store` clause have DRIFTED." > "/dev/stderr"
+        print  "  Fix one or the other; refusing to print 0 for a counter HAProxy never reported." > "/dev/stderr"
+        exit 1
+    }
+    if (!("key" in val)) {
+        printf "ERROR: stick table row has no key= field: %s\n", $0 > "/dev/stderr"
+        exit 1
+    }
+
+    if (!wprinted) {
+        line = "W"
+        for (i = 1; i <= nf; i++) {
+            w = win[F[i]]
+            if (w ~ /^[0-9]+$/) w = sprintf("%g", w / 1000); else w = "-"
+            line = line "\t" F[i] ":" w
+        }
+        print line
+        wprinted = 1
+    }
+
+    nonzero = 0
+    row = ""
+    for (i = 1; i <= nf; i++) {
+        v = val[F[i]]
+        if (v + 0 != 0) nonzero = 1
+        row = row "\t" v
+    }
+    printf "R\t%s\t%s\t%d%s\n", val[F[sortidx]] + 0, val["key"], nonzero, row
+}
+'); then
+    exit 1
+fi
+
+# ------------------------------------------------------------------ printing
+WINSPEC=$(printf '%s\n' "$PARSED" | sed -n 's/^W\t//p' || true)
+
+echo "==================================================================="
+echo "  HAProxy edge IP rates — table '$TABLE' (current values only)"
+echo "==================================================================="
+echo "Source     : $SOURCE"
+echo "Tracked    : ${TBL_USED} of ${TBL_SIZE} slots in use"
+if [ "$SHOW_ALL" = "1" ]; then
+    echo "Filter     : showing ALL tracked IPs"
+else
+    echo "Filter     : showing only IPs with a non-zero counter (use --all for every row)"
+fi
+echo
+
+# Column headers, with each counter's window rendered in SECONDS (HAProxy
+# reports the window in milliseconds, e.g. conn_rate(10000) = 10s).
+HDR=$(printf "%-18s" "IP Address")
+i=0
+for f in "${EXPECTED_FIELDS[@]}"; do
+    w=$(printf '%s\n' "$WINSPEC" | tr '\t' '\n' | sed -n "s/^${f}://p")
+    if [ -n "$w" ] && [ "$w" != "-" ]; then
+        label="${f}/${w}s"
+    else
+        label="$f"
+    fi
+    HDR="$HDR $(printf '%18s' "$label")"
+    i=$((i + 1))
+done
+echo "$HDR"
+printf '%s\n' "$HDR" | sed 's/./-/g'
+
+ROWS=$(printf '%s\n' "$PARSED" | sed -n 's/^R\t//p' || true)
+shown=0
+if [ -n "$ROWS" ]; then
+    while IFS=$'\t' read -r sortkey ip nonzero rest; do
+        [ -n "${ip:-}" ] || continue
+        if [ "$SHOW_ALL" != "1" ] && [ "$nonzero" = "0" ]; then
+            continue
+        fi
+        line=$(printf "%-18s" "$ip")
+        oldifs="$IFS"; IFS=$'\t'
+        # shellcheck disable=SC2086
+        set -- $rest
+        IFS="$oldifs"
+        for v in "$@"; do
+            line="$line $(printf '%18s' "$v")"
+        done
+        echo "$line"
+        shown=$((shown + 1))
+    done < <(printf '%s\n' "$ROWS" | sort -t"$(printf '\t')" -k1,1nr)
+fi
+
+if [ "$shown" -eq 0 ]; then
+    if [ "$SHOW_ALL" = "1" ]; then
+        echo "(no IPs currently tracked)"
+    else
+        echo "(no IP currently has a non-zero counter — re-run with --all to list idle entries)"
+    fi
+fi
+
+echo
+echo "==================================================================="
+echo "These are CURRENT values. The table keeps no history and no record of"
+echo "past tarpits/denials. For actual enforcement events, read the access"
+echo "log ON THE DOCKER HOST (it does not exist in this container):"
+echo "  grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20     # tarpit / deny"
+echo "  grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20    # rate-limit / block"
+echo "  grep -a 'cip=<IP>' /var/log/haproxy.log | tail -50        # one client"
+echo "  grep -a 'id=<uuid>' /var/log/haproxy.log                  # one request reference"
+echo
+echo "Operator actions (via the MASTER CLI socket — the @1 prefix is required):"
+echo "  printf '@1 show table $TABLE key <IP>\\n'  | socat stdio $SOCKET"
+echo "  printf '@1 set table $TABLE key <IP> data.http_req_rate 0\\n' | socat stdio $SOCKET"
+echo "  printf '@1 clear table $TABLE key <IP>\\n' | socat stdio $SOCKET   # drop one entry"
+echo "  printf '@1 clear table $TABLE\\n'          | socat stdio $SOCKET   # drop ALL entries"
+echo "==================================================================="

scripts/show-tarpit-ips.sh

@@ -1,123 +1,35 @@
-#!/bin/bash
-
-# Script to display IPs that have been tarpitted by HAProxy 3.0
-# Uses HAProxy stats socket to query stick-table data
+#!/usr/bin/env bash
 #
-# Usage in Docker container:
-#   docker exec -it haproxy-manager /haproxy/scripts/show-tarpit-ips.sh
+# DEPRECATED SHIM — kept so existing docs/runbooks/muscle memory keep working.
+#
+# This script used to print a "Tarpitted IPs Report" with a "Scan Count" and a
+# BLOCKED / SILENT-DROP / TARPIT status per IP, all derived from gpc0 and gpc1
+# stick-table columns. Those columns DO NOT EXIST: the `web` table
+# (templates/hap_listener.tpl) stores only conn_cur, conn_rate, http_req_rate
+# and http_err_rate. The old parser defaulted every missing field to 0, so the
+# whole report was fabricated — every IP showed "Scan Count 0 / Normal"
+# regardless of what it was actually doing.
+#
+# The stick table also keeps NO history, so nothing in it can identify who was
+# tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST
+# at /var/log/haproxy.log (it does not exist inside this container):
+#   grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+#   grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limit / block
+#
+# What IS knowable from the stick table — the current per-IP rates — is printed
+# by show-edge-ip-rates.sh, which this shim now runs.
 
-SOCKET="/tmp/haproxy-cli"
+set -euo pipefail
 
-# Check if socket exists
-if [ ! -S "$SOCKET" ]; then
-    echo "Error: HAProxy socket not found at $SOCKET"
-    echo "Make sure HAProxy is running with stats socket enabled"
-    exit 1
-fi
+cat >&2 <<'NOTE'
+NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits.
+      The HAProxy stick table stores no history and no gpc0/gpc1 counters, so
+      the old "Scan Count"/"BLOCKED" columns were fabricated numbers.
+      Actual tarpit/deny events are in /var/log/haproxy.log ON THE HOST:
+        grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+      Running show-edge-ip-rates.sh instead (current rates, real values):
 
-echo "==================================================================="
-echo "                    HAProxy Tarpitted IPs Report                   "
-echo "==================================================================="
-echo
-echo "Showing IPs tracked in the stick-table with scan detection counters:"
-echo "(gpc0 = total scan attempts, gpc1 = escalation level)"
-echo
+NOTE
 
-# In HAProxy 3.0, we need to use the proper process prefix
-# The web frontend table is in the worker process, not master
-# First check which process has the table
-# Note: grep for actual worker line, not the header
-PROCESS_ID=$(echo "show proc" | socat stdio "$SOCKET" 2>/dev/null | grep -E '^[0-9]+.*worker' | awk '{print $1}' | head -1)
-
-if [ -z "$PROCESS_ID" ]; then
-    echo "Error: Could not find HAProxy worker process"
-    echo "Try: echo 'show proc' | socat stdio $SOCKET"
-    exit 1
-fi
-
-# Show stick-table entries from the web frontend using the worker process
-# Use printf to avoid bash history expansion issues with !
-printf "@!%s show table web\n" "${PROCESS_ID}" | socat stdio "$SOCKET" 2>/dev/null | {
-    # Skip the header line
-    read header
-    
-    # Check if we got an error or empty response
-    if echo "$header" | grep -q "No such table"; then
-        echo "Error: Table 'web' not found. HAProxy may need to be reloaded."
-        exit 1
-    fi
-    
-    has_data=false
-    echo "IP Address           | Scan Count | Level | HTTP Err Rate | Status"
-    echo "---------------------|------------|-------|---------------|------------------"
-    
-    # Process each line
-    while IFS= read -r line; do
-        # Skip empty lines and comments
-        if [ -z "$line" ] || echo "$line" | grep -q "^#"; then
-            continue
-        fi
-        
-        # HAProxy 3.0 format: 0x... key=<ip> use=... exp=... gpc0=... gpc1=... http_err_rate(10s)=...
-        if echo "$line" | grep -q "key="; then
-            has_data=true
-            
-            # Extract IP and counters
-            ip=$(echo "$line" | grep -o 'key=[^ ]*' | cut -d'=' -f2)
-            gpc0=$(echo "$line" | grep -o 'gpc0=[0-9]*' | cut -d'=' -f2)
-            gpc1=$(echo "$line" | grep -o 'gpc1=[0-9]*' | cut -d'=' -f2)
-            err_rate=$(echo "$line" | grep -o 'http_err_rate([^)]*=[0-9]*' | grep -o '[0-9]*$')
-            
-            # Set defaults if values are empty
-            gpc0=${gpc0:-0}
-            gpc1=${gpc1:-0}
-            err_rate=${err_rate:-0}
-            
-            # Determine status based on scan count and escalation
-            status=""
-            if [ "$gpc0" -ge 100 ]; then
-                status="BLOCKED (429)"
-            elif [ "$gpc0" -ge 60 ]; then
-                status="SILENT-DROP"
-            elif [ "$gpc0" -ge 40 ]; then
-                if [ "$gpc1" -ge 2 ]; then
-                    status="SILENT-DROP (repeat)"
-                else
-                    status="TARPIT 10s"
-                fi
-            elif [ "$gpc0" -ge 25 ]; then
-                status="TARPIT 10s"
-            else
-                status="Normal"
-            fi
-            
-            # Format output
-            printf "%-20s | %10s | %5s | %13s | %s\n" "$ip" "$gpc0" "$gpc1" "$err_rate/10s" "$status"
-        fi
-    done
-    
-    if [ "$has_data" = false ]; then
-        echo "(No IPs currently tracked - table is empty)"
-    fi
-}
-
-echo
-echo "==================================================================="
-echo "Legend:"
-echo "  - Scan Count 25-39: Low scanner → TARPIT 10s delay"
-echo "  - Scan Count 40-59: Medium scanner → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "  - Scan Count 60-99: High scanner → SILENT-DROP (immediate disconnect)"
-echo "  - Scan Count 100+:  Critical scanner → BLOCKED (429 response)"
-echo "  - Burst (5+ in 10s): → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "==================================================================="
-echo "Note: Only counts suspicious scripts/configs, NOT missing images/fonts/CSS"
-echo "Note: IPs are tracked for 1 hour since last activity"
-echo
-echo "To clear a specific IP from the table:"
-echo "  printf '@!${PROCESS_ID} del table web key <IP>\\n' | socat stdio $SOCKET"
-echo
-echo "To clear all entries:"
-echo "  printf '@!${PROCESS_ID} clear table web\\n' | socat stdio $SOCKET"
-echo
-echo "Debug: Worker PID is ${PROCESS_ID}"
-echo
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+exec "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"

scripts/start-up.sh

@@ -1,6 +1,116 @@
 #!/usr/bin/env bash
+# Container entrypoint. Two-phase startup:
+#   1. One-shot init (init.py): DB schema, certbot register, config gen, start HAProxy.
+#      Runs synchronously and to completion so haproxy is up before the API binds.
+#   2. WSGI serving via gunicorn (replacing the Flask dev server). Two gunicorn
+#      instances:
+#        - port 8080 -> default_app  (default page + blocked-ip page; HAProxy
+#          proxies unmatched / blocked traffic here)
+#        - port 8000 -> app          (management API)
+#
+# Why gunicorn:
+#   Flask's built-in werkzeug "development server" is single-threaded and leaks
+#   workers under sustained load. It carried haproxy-manager for a long time but
+#   stalled out around 24-48h uptime ("healthy" health-check, but every request
+#   queued behind a stuck worker). Gunicorn with --max-requests cycles workers
+#   periodically, which prevents the slow-leak failure mode entirely.
 
-# Exit on error
 set -eo pipefail
+
+# Ensure trusted IP whitelist files exist (volume-mounted /etc/haproxy may shadow image defaults)
+mkdir -p /etc/haproxy
+[ -f /etc/haproxy/trusted_ips.list ] || : > /etc/haproxy/trusted_ips.list
+[ -f /etc/haproxy/trusted_ips.map ]  || : > /etc/haproxy/trusted_ips.map
+
+# cloudflare_ips.list is SHIPPED DATA: it must always match what this image
+# bakes in (/haproxy/defaults), so a Cloudflare range refresh actually reaches
+# existing hosts instead of being permanently shadowed by the volume.
+# Overwrite it from the baked copy on every start.
+#
+# trusted_proxies.list and wpadmin_gate_exempt.list are OPERATOR DATA:
+# operators add entries directly on the server and those must survive
+# restarts/recreates. Seed each from the baked copy only when it's missing;
+# never overwrite an existing one.
+#
+# All branches fall back to an empty file if the baked default is somehow
+# absent, because "acl ... -f <missing file>" is a fatal HAProxy config
+# error -- the list files must exist unconditionally by the time HAProxy starts.
+if [ -f /haproxy/defaults/cloudflare_ips.list ]; then
+    cp /haproxy/defaults/cloudflare_ips.list /etc/haproxy/cloudflare_ips.list
+else
+    [ -f /etc/haproxy/cloudflare_ips.list ] || : > /etc/haproxy/cloudflare_ips.list
+fi
+if [ ! -f /etc/haproxy/trusted_proxies.list ]; then
+    if [ -f /haproxy/defaults/trusted_proxies.list ]; then
+        cp /haproxy/defaults/trusted_proxies.list /etc/haproxy/trusted_proxies.list
+    else
+        : > /etc/haproxy/trusted_proxies.list
+    fi
+fi
+if [ ! -f /etc/haproxy/wpadmin_gate_exempt.list ]; then
+    if [ -f /haproxy/defaults/wpadmin_gate_exempt.list ]; then
+        cp /haproxy/defaults/wpadmin_gate_exempt.list /etc/haproxy/wpadmin_gate_exempt.list
+    else
+        : > /etc/haproxy/wpadmin_gate_exempt.list
+    fi
+fi
+
 cron &
-python /haproxy/haproxy_manager.py
+
+# Phase 1: container init
+python /haproxy/scripts/init.py
+
+# Phase 1.5: in-container haproxy supervisor.
+# haproxy runs as a background child of PID 1 (gunicorn) with NOTHING watching
+# it after init. If the haproxy master dies mid-life (e.g. SIGABRT -> exit 134,
+# segfault), the container stays "up" (gunicorn is PID 1), Docker's --restart
+# policy never fires, and haproxy is down until the external host watchdog
+# full-restarts the whole container minutes later (dropping every connection).
+# This loop revives haproxy in place within one interval. ensure_haproxy.py is
+# idempotent — a cheap no-op whenever haproxy is already running.
+HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
+(
+    while true; do
+        sleep "${HAPROXY_SUPERVISOR_INTERVAL}"
+        python /haproxy/scripts/ensure_haproxy.py 2>&1 || true
+    done
+) &
+
+# Phase 2: WSGI servers
+# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
+# (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
+# 1000 — worker recycle frequency).
+#
+# API_WORKERS default is 2 (was 1). A single worker is a single point of
+# failure: if its gthread pool ever wedges (see the 2026-07-07 subprocess-hang
+# incident — now bounded by DEFAULT_SUBPROCESS_TIMEOUT in haproxy_manager.py),
+# the entire management API goes dark. A second worker keeps the API answering
+# (config regenerate, health, SSL) while the other recycles via --max-requests.
+API_WORKERS="${HAPROXY_MGR_API_WORKERS:-2}"
+API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
+MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
+MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
+
+# Default page server on :8080. Stays in the background.
+# --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
+# without forking. --max-requests recycles the worker to bound memory drift.
+gunicorn \
+    --bind 0.0.0.0:8080 \
+    --workers 1 --threads 4 --worker-class gthread \
+    --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
+    --timeout 30 \
+    --access-logfile - --error-logfile - --log-level info \
+    --pythonpath /haproxy \
+    'haproxy_manager:default_app' &
+
+# Main API server on :8000 in the foreground. exec so signals propagate
+# correctly and the container exits if the API dies (docker --restart picks it
+# up). Longer --timeout because cert issuance hits ACME and can take a while.
+exec gunicorn \
+    --bind 0.0.0.0:8000 \
+    --workers "${API_WORKERS}" --threads 4 --worker-class gthread \
+    --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
+    --timeout "${API_TIMEOUT}" \
+    --access-logfile - --error-logfile - --log-level info \
+    --pythonpath /haproxy \
+    'haproxy_manager:app'

scripts/sync-certificates.sh

@@ -8,6 +8,7 @@ LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
 ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
 DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
 SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
+LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
 
 # Logging functions
 log_info() {
@@ -18,13 +19,25 @@ log_error() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
 }
 
+# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
+# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
+# uses this script's logging rather than its own fallbacks.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=cert-publish-lib.sh
+if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
+    . "${SCRIPT_DIR}/cert-publish-lib.sh"
+else
+    log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
+    exit 1
+fi
+
 log_info "Starting certificate sync process"
 
 # Ensure SSL certs directory exists
 mkdir -p "$SSL_CERTS_DIR"
 
 # Get all SSL-enabled domains from database
-DOMAINS=$(find /etc/letsencrypt/live/ -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
+DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
 
 if [ -z "$DOMAINS" ]; then
     log_info "No SSL-enabled domains found"
@@ -36,13 +49,16 @@ UPDATED=0
 FAILED=0
 
 while read -r domain; do
-    CERT_FILE="/etc/letsencrypt/live/${domain}/fullchain.pem"
-    KEY_FILE="/etc/letsencrypt/live/${domain}/privkey.pem"
+    CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
+    KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
     if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
-        # Combine cert and key into single file for HAProxy
-        if cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"; then
+        # Assemble in a staging dir and rename into place. NEVER redirect into
+        # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
+        # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
+        # takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
+        if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
             log_info "Updated certificate for $domain"
             UPDATED=$((UPDATED + 1))
         else
@@ -59,6 +75,13 @@ log_info "Certificate sync completed: $UPDATED updated, $FAILED failed"
 
 # Reload HAProxy if any certificates were updated
 if [ $UPDATED -gt 0 ]; then
+    # Never reload onto unvalidated material: a reload that fails to load the
+    # certs directory drops HTTPS for every site on this host.
+    if ! haproxy_config_ok; then
+        log_error "HAProxy configuration does not validate - refusing to reload after certificate sync"
+        exit 1
+    fi
+
     if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
         log_info "HAProxy reloaded successfully"
     else
@@ -67,5 +90,12 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# See the matching block in renew-certificates.sh: a run in which every domain
+# failed to publish must not look like a clean run to its caller.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate sync process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate sync process completed"
 exit 0

scripts/test-cert-scripts.py

@@ -0,0 +1,958 @@
+#!/usr/bin/env python3
+"""Regression tests for the certificate publication shell scripts.
+
+Why this file exists
+--------------------
+renew-certificates.sh and sync-certificates.sh published a bundle with
+
+    cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"
+
+where $COMBINED_FILE is the pem HAProxy is serving *right now*. The shell
+truncates the destination when it opens the redirect, before cat runs, so any
+failure after that point - unreadable source key, ENOSPC, container killed
+mid-write - left a zero-length or key-less pem behind. The exit status of cat
+was checked, but by then the live file was already destroyed.
+
+HAProxy loads $SSL_CERTS_DIR as a DIRECTORY (`bind :443 ssl crt /etc/haproxy/
+certs`) and tries to load every file in it, so a single unloadable file fails
+the whole bind: HTTPS down for every customer on the host.
+
+The fix (scripts/cert-publish-lib.sh) assembles into a sibling staging dir,
+validates, backs up the outgoing bundle into a sibling backup dir, and renames
+into place. These tests pin the observable guarantees:
+
+  * a successful publish replaces the live pem and archives the old one;
+  * a FAILED publish leaves the previous, still-valid pem byte-for-byte intact;
+  * nothing that is not a final *.pem ever appears in the certs directory;
+  * HAProxy is not reloaded when `haproxy -c` rejects the configuration.
+
+Running
+-------
+    python3 scripts/test-cert-scripts.py            # tests the repo checkout
+    HAPROXY_MANAGER_DIR=/some/other/tree \
+        python3 scripts/test-cert-scripts.py        # tests another tree
+
+Self-contained stdlib unittest - no pytest, no venv, no bats, and nothing is
+imported from the application. The scripts are driven as subprocesses with
+every path they touch redirected by environment variable (SSL_CERTS_DIR,
+LETSENCRYPT_LIVE_DIR, CERT_STAGING_DIR, CERT_BACKUP_DIR, LOG_FILE,
+ERROR_LOG_FILE, HAPROXY_CONFIG) and stub `certbot`, `socat` and `haproxy`
+binaries on PATH.
+
+The certificate material below is a real self-signed test certificate with its
+matching key (plus a second, unrelated key for the mismatch case), embedded as
+constants so the tests need no openssl to *create* material. openssl IS needed
+to run them: the library's cert/key pairing check is mandatory (it is the only
+layer that can reject a bundle of empty pem blocks), so without the binary
+every publish is refused by design. The image ships openssl 3.x.
+
+The acceptance bar for this file is a green run INSIDE the built image, as
+root, which is where these scripts actually execute - not on a workstation.
+Several failure modes are invisible outside the container (root ignores the
+directory permissions one test used to rely on) and one was actively
+destructive there; see _cleanup_tmp().
+"""
+
+import os
+import re
+import shutil
+import stat
+import subprocess
+import tempfile
+import textwrap
+import time
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+SCRIPTS_DIR = os.path.join(MODULE_DIR, 'scripts')
+LIB = os.path.join(SCRIPTS_DIR, 'cert-publish-lib.sh')
+
+BROKEN_TOKEN = '__BROKEN__'
+DOMAIN = 'test.example.com'
+
+# --- test key material -------------------------------------------------------
+# openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \
+#         -subj /CN=test.example.com
+TEST_CERT = """\
+-----BEGIN CERTIFICATE-----
+MIIDFzCCAf+gAwIBAgIUeaz/lNOESOTHsB3Y97+Xja7Fy4gwDQYJKoZIhvcNAQEL
+BQAwGzEZMBcGA1UEAwwQdGVzdC5leGFtcGxlLmNvbTAeFw0yNjA4MDYxNTQyMTda
+Fw0zNjA4MDMxNTQyMTdaMBsxGTAXBgNVBAMMEHRlc3QuZXhhbXBsZS5jb20wggEi
+MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDF5GL7Gjn+UnPFy5sqP2k4XHth
+mkWFZj+mjK6cDhbBXYt60NrwVdrrgOFydMC75VeUceFxG/5GD7wrXZP23xzbnWKm
+7FxfOSmr4y+1rVEZwi8IeWEz3W6C6y5rjZsCI+pBgdna+aJSpTQZHPfDpNtQm5vl
+enj5BfizYixinORxm9kvXMGXV+Cw1CJkqB3mzScwWt40EtQoVxekebf8B7i4ZHyx
+xT6/xwF+WY8OliZkY1pdqncoTLUAYcaE/HR/ojJKmSVIq1GswZE/y3E56LIwq+wJ
+eGbgH46a+86z+VO2UX1jbad1kWKBCsRoOpaybZDEWAYTOqahW7kOH7umTUsRAgMB
+AAGjUzBRMB0GA1UdDgQWBBTIqa3BNEjjcxkhXqnRwInrLM9yijAfBgNVHSMEGDAW
+gBTIqa3BNEjjcxkhXqnRwInrLM9yijAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
+DQEBCwUAA4IBAQB/lYzXb5PI3magMz/IXmwTsMCrVSdaUYEIKLEJggmbGxqpwO1a
+iYagWZ/5H3B9KDvNQA+L4FkMJ726ZkdGEH/vkwvTAuhwU2NSWcbRJ8DK5u3Q4rnJ
+VswPcW5njUF9mQq0NPX/PMCeOoFDEI8+RrgQZxtHhopwuKOgVA6HRBINKdEZJlrp
+oLLQrHDNVLMYTclNHG6kBg0lOHUV31TgkJQ8kMgtq0WQX7RseKR10QKgN5iOBomU
+3+y713Ibpac5B1zw5l3LjE/59xFteFbDENr2+A5VGhVNVZC+bs+YTYTzeAsGB0e3
+MQ+XJMJq3kaJmQ+QcTrRaKMtoMz2h1AIRbLd
+-----END CERTIFICATE-----
+"""
+
+# The private key that matches TEST_CERT.
+TEST_KEY = """\
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDF5GL7Gjn+UnPF
+y5sqP2k4XHthmkWFZj+mjK6cDhbBXYt60NrwVdrrgOFydMC75VeUceFxG/5GD7wr
+XZP23xzbnWKm7FxfOSmr4y+1rVEZwi8IeWEz3W6C6y5rjZsCI+pBgdna+aJSpTQZ
+HPfDpNtQm5vlenj5BfizYixinORxm9kvXMGXV+Cw1CJkqB3mzScwWt40EtQoVxek
+ebf8B7i4ZHyxxT6/xwF+WY8OliZkY1pdqncoTLUAYcaE/HR/ojJKmSVIq1GswZE/
+y3E56LIwq+wJeGbgH46a+86z+VO2UX1jbad1kWKBCsRoOpaybZDEWAYTOqahW7kO
+H7umTUsRAgMBAAECggEAA9SnFFqGfR1yO4XUlfmmgyZqJoJmnl2TdZlDT4bHyrwx
+dSIKHO0iiNzEsHMhYHnA62EVdruunUNUdofwE28v9zHZnSbV5mt8OqUyERue5mdf
+gvPbjXYXu63LBx61fZH9qME3WwFqUpx7UNGiW62LJ8ktWEC5ywNCNFG+D3YfR3Iu
+v3PdIFXwkpJMaXO+42JSoSVoiqxlONqNiqcdQj64iYgCYdNxcgdLy+mHGrf1hAZs
+dom2Jp0oEM4ZdOQu3Z6uyEyPsECiz1PdQjzagaEfPWtKCQk0kY8DDCn5X2xQif9w
+3xeaISqho7bQgSo4IEgWHTP6V0v7brTP7VcK+gyMyQKBgQD15FrE4XAL89NBQ6iX
+m2l6quZv5tIUtreuYxsOpFIMU22zfsRZOvA7sJR3ufOJ+b7tFHuJ8DAJUpK08Vvg
+A930/LW9wUsY42d54XKIO/8DTsIrmjppoGdshs3axJQkqfN0zQkhqfbwaXRJ9X2k
+Fvax+5jftaIaw0hQdWLnfTmGmQKBgQDOBue89D8THYl/LvDGg6WoVyK3ljBMK2s3
+4BljeZCJdWAjtido13Mltubc6YVHScmVoIZKTmx+fCjdQ3y1t92vZZSBeLsXhfFr
+N+IOGZu3ZmJu64x3OukSYQ7x6agi5yP3+7k0siZgxOMXJRQDYZUcHxIHDrSGLeLZ
+sj7LnbvLOQKBgE8murE1gEPYsOAJT3O96y45ZQQQYP+Z8XaJIGSOMHsXP/DPlZTD
+jCEqrh/8E5EOe48FUN8OGehmVCM6rkBl/kSmNDpoxiu0x9JL5/pClcwSxh4S/0qQ
+/7nHiuwo6ycCLgQjHBViCMNKrsw/4bm4SqDwRD1+0jebNOPxZWzuul3BAoGAZij0
+ZjSyxhbCZEdxau5CiYvTkjct8cch3k4IKNRRwGdsaajcN9eFqHDeXzKIPQYwqDo1
+/MiQcdO9K6JYR39JtLxo/B5Sn2JyiJjoRdea6EEjlB7GwyR6B/wKvhf/oHb+1euD
+NccU0q0ucf6XwulzV8NsXAWFrHc6YnpJOwwW37kCgYAkEyjUc73jImiTyf/IXVOD
+UHlRXZPvwtZUuPGe4RI0Gds97tKnvXnvFsPIRCOGfVzZ8z79DGiQ8TR2a0hgZec1
+Mo3J2dCjlv4Q6ACjHkCA1cmi13OHUPnpaeesrOk+SpEVJfR2k4qRWH4z3oxbPO/q
+TDFnjWHSgjkDMrxVHZ8wPg==
+-----END PRIVATE KEY-----
+"""
+
+# A perfectly valid RSA key that has nothing to do with TEST_CERT.
+UNRELATED_KEY = """\
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDE7mosrsdBhdn1
+ZIErYmMuPU69ws51JdyqwtRZlV2uLLby0dOfJpFKZPuBPEqTqDimd9N4FSL0CGjG
+zXG23RYd9AbdSoorFORrUxeiPzFnbz4v38srGHckOS9ozKAbPLACUEjWMX1NwAZ+
+wGlSz3cWYTWVztFCUIxpvLVR87PSpTnpdCXIj0EABOc6WoLBLE+v2knZOa63LKg6
+GryInD43CnWFBKpH0gdgWqh+ie3NFMumLR8M3lZq2Mk0EFgVWrxPnJobYvOmavNp
+NBR3ZugR4X57c1YFyLkYXQIdhxuYV1ZAY6NgmAIsaiCKFqdksF5AJSV7FZNBM43r
+33ae/R5hAgMBAAECggEAIgNYsL9+OEWtU8ooWi0r2q5plXJaVNb1gkPUx+U5sS3V
+amKNxbj0UrBW1Scr7U1afXwIOP8TkqkKKb4NpCsS2RkO/3USoKbC3fuTwyjdeFM5
+Hy0sytR2rXm4A8aF57ZnYvrpXZ9eGEnwhT9n4Y7mL2YaSnXWZDkDy3Z1rcIk/p5P
+jUo4UQyzD/9Gab1stcBbGv+66B3mlVdRVJor5+tGn6zmr4TjvqupAexuXd+Q6+1L
+OG4e2bAUmuOVOa+w8Xo4vwkiXSDLVEsl1z1x4Bcrv61bIg0rbZAeqcZ4EUPyrEyR
+RpaaOLAiwdzjOj5pQvHAF2q/++IlY6xPhotTr11tvQKBgQD37C2UQRGRo622+2kx
+qgsdA6jPM9vCE9S4aS4qaOj8XSMMCu6bQW5lU48DaONvXxDmT6x9IrgR3QN1Iz/t
+17kUCCghuviEP1RjnahBRQqZR38KdDzKhaWt9jgdjCun9MzUXIaQRTFlFGA1z1pz
+apZ9a8/ehYPSk4pv9h06/B3IjQKBgQDLWPBuhj203QOmwVF1v1k4yyuJ7UBqtRaY
+I9jMW93sB2hPs6Se10UroQHlF95IHeRXvNKH/UXuAILIJMN8oTI4uU7t60shJHvI
+o14RzEZoUQjES7BBVhePglndJmYIKoKKAX5lSFe9Lk0ei9B+1Dle5Q+9ZeP09cp4
+vVcGvOgqJQKBgQCtxe1srO8TlhZ821uwY+/GNnpsQX0XW68OUyr4rvAfc2jNWBxG
+1mX6v8bOLQa9WXUO+Wl9jIhYfQGfaUW2AC7Jy63VdqgaigksiaUVmr8DEQoK2c6C
+ZYrrlFlg3I78+qlXcEMhfF5S6yVEkkJkA6HX52mcHxl2z9OJBokWfwChQQKBgAzH
+xzy7FS/D4FHfvpXu89Wc91yQ28aZIRVo01xsvbLy+DxiJwuQrhlC4lKawG656jsV
+dAn2AiomQBICNYMkwnpMM0jCzBMGLv16PxRRSW+PAEUOGMLSfWKYp7s9iZYjzdaM
+p3wIIvOR8GjmErGV9xEexnF58OzZceNKyyhyQQk9AoGAbvjJvYLOhQpAbbqErNNv
+LK1P+TngKpukRHXjiUpPVEGNhN6krBBJCBWzY7ucrIy6jz8UBy7SbITBy7qIKhxZ
+PVP7WVATMWEeW1AfdBfCYDI4jFKAD8SLECby45nRBuBllYdQnW1gBzLcCulCwB+w
+FV0RvuQPDYkqsx8ibqpSv7c=
+-----END PRIVATE KEY-----
+"""
+
+# A SECOND, unrelated but internally consistent pair. Needed by the
+# concurrency test: two bundles that are each perfectly valid but whose keys
+# differ, so a validator that reads the certificate from one and the key from
+# the other reports a mismatch. Two bundles sharing key material - which is
+# what PREVIOUS_BUNDLE and NEW_BUNDLE are - cannot detect that at all.
+# openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
+#         -subj /CN=second.example.com
+TEST_CERT_2 = """\
+-----BEGIN CERTIFICATE-----
+MIIDGzCCAgOgAwIBAgIUKQAAvrVkden7gIg2zYMLb6dO1c8wDQYJKoZIhvcNAQEL
+BQAwHTEbMBkGA1UEAwwSc2Vjb25kLmV4YW1wbGUuY29tMB4XDTI2MDgwNjE2NTcz
+NVoXDTM2MDgwMzE2NTczNVowHTEbMBkGA1UEAwwSc2Vjb25kLmV4YW1wbGUuY29t
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv2pfTOvN2zrZJ6d+biiD
+VOczMqanuHPBCy2CnTBif+7VWf8AaywRzQ3ShfhmalVRNEFMn0MSO+GocmH71Ve1
+nl89oGiJPmvX1lpncbgM692ddhP9ez4xUeNj+QAWp9VBZhInNuM4Pawv5BPngtpj
+2MGXf3ZlBSli8Ng7jBo1fTMT3bh8GcE1rIPRvmUuQwFIt2eGnLR8jQd+xGelhAjG
+nnXtlc+ebo4r2OjljNgvtdUknBZdpiZXmjdFzyClYTeMuEen2uwMpJNc0wLbRjcU
+khVF3nw4jUnkOhWH3JYGAoWslJyEqZSANwt/eOHwXgyVuxg31bCl297iskW0IRZL
+wQIDAQABo1MwUTAdBgNVHQ4EFgQU7HH8GacJzc2j9s2UJCPwykgrIckwHwYDVR0j
+BBgwFoAU7HH8GacJzc2j9s2UJCPwykgrIckwDwYDVR0TAQH/BAUwAwEB/zANBgkq
+hkiG9w0BAQsFAAOCAQEAYE5jHX1dK091jVsFSZDdiw9AU5rrk8XpF1yuPmDisRnE
+dJ4QQq3dzWXRnp0bzZnq7fdfiEz1m39zVixov7WFp24QhenD2n5K7/wew7RpXTnA
+pAGBEdsGvBJ+3MgkRYklXCM9f9f4z21xXRNZ+BwBcM25D+gR4b+PRMQR6BhZx5R+
+y2jQsoM68cFjRApFWgmji4pBjg/eOaZMBCfVTjP+npVyqG7UtV5EyYXwPgPa/rm2
+FoP+eftzP6dszBEonkIVyyvkdscI4Wkr8hw3S0R/TP8l9lTnvN1HC3o7Es5VY53R
+swNAXWBlgm0N7A96ISLtQjgvOfeMRTCSjxW9pm0wJA==
+-----END CERTIFICATE-----
+"""
+
+TEST_KEY_2 = """\
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC/al9M683bOtkn
+p35uKINU5zMypqe4c8ELLYKdMGJ/7tVZ/wBrLBHNDdKF+GZqVVE0QUyfQxI74ahy
+YfvVV7WeXz2gaIk+a9fWWmdxuAzr3Z12E/17PjFR42P5ABan1UFmEic24zg9rC/k
+E+eC2mPYwZd/dmUFKWLw2DuMGjV9MxPduHwZwTWsg9G+ZS5DAUi3Z4actHyNB37E
+Z6WECMaede2Vz55ujivY6OWM2C+11SScFl2mJleaN0XPIKVhN4y4R6fa7Aykk1zT
+AttGNxSSFUXefDiNSeQ6FYfclgYChayUnISplIA3C3944fBeDJW7GDfVsKXb3uKy
+RbQhFkvBAgMBAAECggEABQLqy6eedRPt31sLOMFEDkzbZdFHOMpZeuqThsNmjo7z
+t9diokgeD4ZQXimbEQqsZYDAtmFGdiEp86I9JQBk/4TUiqFhHrOADBe1jQAptjfs
+iyIb51gIPnjZ1RBA5Al8zohy28T9h1+Z7+/OeCvcgLyVAixf/U9pU8D1El9cu9zv
+4q4WJPB5Tgkq+YwcmeuT8LzsKoSDmPQjFVY9v+gz6hoVUVyP6gswnlFnKjNcmfZU
+0CKP90sCAc5mKZv7RyGG920LDU4u2ggnQoK05GhXK8R3amJmoAF3i+xGeTRvNEKe
+wDC7NinTG2WDVo6y/FCvFKs0+qqlKww1u39fq166rQKBgQDtjPjvkDkSYFcWFznn
+sfsxN5R1cLpxdutLZhfvtpHIj5NkQbmOk0r3LbDAwU+UNVQ6F8jWQVHUsni9jAY6
+3RNRF4ZabC51aqg/Ssj7d5mE4kj7y6Ch/nnSXIogRxLG4bWo2rtG1M2Dt3Ef08R4
+6gjcp9ZmELyVx7H4yXFJGIaDdQKBgQDOSCHFK/ggZzzy4iC7DOq1lrbR+jyk95rh
+F5EGzyAkgJ4uYc9TCPkUDxXjWL17r61/obfbW4znaV9fHEKpz3R54osDXln3oLea
+BBWkJI3ANe8iNrGDE4FN9to4DdUMFsWX/WEiRyBPIqDy9DTyadblMLTNZjNPuLmg
+ZJOB3tRZnQKBgQCjeglya9Eq2Uv1MuSxk2VnmHU9YOed4BXLHKZKXFz1JgFr1GNL
+QAguFK536FDIkO62z9lxwR/8fRnkb7F13uBFRSg7oAlU2qKQc/nePI9UyJkrVxXj
+hYn2f6K61c6ROZFXc7e/5gDMrXhXS9gA0iZpG8PLF6eAeB39NTwV7p/bZQKBgQCV
+v5eEY58FJu0ABVhtcbsRiA+/70EHIRi2Pz1xC/vxg81RLoArb2AiR7FEEa+8kpQJ
+C4VFIPjxJXWuvf1G+Os9cFAqadw1/95JWJ29QywEVSL8W2gSF57O0l0oRCJdXEql
+Q7O4BppV2HWu6clmEZ+HUgxu77pgLWHUJi9PIExXoQKBgEq6rYBXmsSQermnPTOn
+YFx7c2ns97hsjYIbs497+gPW4/xQWwsN76t60SWqjXV4DHJCpi5Tnjo2fk5/OzK+
+HfshC9CKFDk8T0KGQWwaPwqP/OYbqOA88IlJ7xbPdSuJkjefHCtVVEtao6+HNgzm
+lniLtDMpU0MLPgB98ClQ4HDA
+-----END PRIVATE KEY-----
+"""
+
+# The bundle already on disk when a run starts. Same key material, plus a
+# trailing marker so "the live file was replaced" and "the old file was
+# archived" can be told apart byte-for-byte.
+PREVIOUS_BUNDLE = TEST_CERT + TEST_KEY + '# previous bundle\n'
+NEW_BUNDLE = TEST_CERT + TEST_KEY
+OTHER_BUNDLE = TEST_CERT_2 + TEST_KEY_2
+
+# --- stub binaries -----------------------------------------------------------
+STUB_CERTBOT = """\
+#!/bin/sh
+# Test stub for certbot: pretend there was nothing to renew.
+echo "No renewals were attempted."
+exit 0
+"""
+
+STUB_SOCAT = """\
+#!/bin/sh
+# Test stub for socat: record that a reload was attempted, and what was sent.
+{ printf 'socat %s <<' "$*"; cat; printf '>>\\n'; } >> "$SOCAT_LOG"
+exit 0
+"""
+
+STUB_HAPROXY = """\
+#!/bin/sh
+# Test stub for the haproxy binary: `haproxy -c -f FILE` rejects any config
+# containing %(token)s, which is how the tests inject an invalid config.
+cfg=""
+while [ $# -gt 0 ]; do
+  case "$1" in -f) cfg="$2"; shift ;; esac
+  shift
+done
+if [ -n "$cfg" ] && grep -q '%(token)s' "$cfg" 2>/dev/null; then
+  echo "[ALERT] parsing [$cfg:1] : unknown keyword '%(token)s'" >&2
+  exit 1
+fi
+exit 0
+""" % {'token': BROKEN_TOKEN}
+
+GOOD_HAPROXY_CFG = textwrap.dedent("""\
+    global
+        daemon
+    defaults
+        mode http
+    frontend fe
+        bind 0.0.0.0:443 ssl crt /etc/haproxy/certs
+""")
+
+
+def write(path, content, mode=None):
+    with open(path, 'w') as fh:
+        fh.write(content)
+    if mode is not None:
+        os.chmod(path, mode)
+    return path
+
+
+def read(path):
+    with open(path) as fh:
+        return fh.read()
+
+
+class CertScriptFixture(unittest.TestCase):
+    """An isolated fake /etc/haproxy + /etc/letsencrypt plus stub binaries."""
+
+    def setUp(self):
+        # Without the library every "the library rejects X" assertion in this
+        # file can be satisfied by bash exiting 127, so make its absence a
+        # failure of every test rather than a silent pass of several.
+        self.assertTrue(os.path.isfile(LIB),
+                        f'{LIB} is missing - nothing below tests anything')
+
+        self.tmp = tempfile.mkdtemp(prefix='haproxy-cert-test-')
+        self.addCleanup(self._cleanup_tmp)
+
+        self.bindir = os.path.join(self.tmp, 'bin')
+        os.makedirs(self.bindir)
+        write(os.path.join(self.bindir, 'certbot'), STUB_CERTBOT, 0o755)
+        write(os.path.join(self.bindir, 'socat'), STUB_SOCAT, 0o755)
+        write(os.path.join(self.bindir, 'haproxy'), STUB_HAPROXY, 0o755)
+        self.socat_log = os.path.join(self.tmp, 'socat-invocations.log')
+
+        # Mirrors the real layout: certs dir, with staging/backups as SIBLINGS.
+        self.haproxy_dir = os.path.join(self.tmp, 'etc', 'haproxy')
+        self.certs_dir = os.path.join(self.haproxy_dir, 'certs')
+        self.staging_dir = os.path.join(self.haproxy_dir, 'cert-staging')
+        self.backup_dir = os.path.join(self.haproxy_dir, 'cert-backups')
+        os.makedirs(self.certs_dir)
+
+        self.le_live = os.path.join(self.tmp, 'etc', 'letsencrypt', 'live')
+        self.domain_dir = os.path.join(self.le_live, DOMAIN)
+        os.makedirs(self.domain_dir)
+        self.src_cert = write(os.path.join(self.domain_dir, 'fullchain.pem'), TEST_CERT)
+        self.src_key = write(os.path.join(self.domain_dir, 'privkey.pem'), TEST_KEY)
+
+        self.live_pem = os.path.join(self.certs_dir, DOMAIN + '.pem')
+        self.haproxy_cfg = write(os.path.join(self.haproxy_dir, 'haproxy.cfg'),
+                                 GOOD_HAPROXY_CFG)
+        self.log_file = os.path.join(self.tmp, 'haproxy-manager.log')
+        self.error_log = os.path.join(self.tmp, 'haproxy-manager-errors.log')
+
+    def _cleanup_tmp(self):
+        # A test may have chmod 000'd a fixture file, which would stop rmtree.
+        #
+        # SYMLINKS ARE SKIPPED, and that is not a nicety. os.chmod() FOLLOWS
+        # symlinks, and the openssl-availability tests below build a stripped
+        # PATH directory out of symlinks to real system binaries (/usr/bin/cat,
+        # /usr/bin/chmod, ...). Walking those with chmod 0600 as root - which is
+        # how this container runs - stripped the exec bit from a dozen core
+        # binaries of the machine running the tests, chmod itself included, so
+        # it could not even be undone from inside the container: every later
+        # test failed with "/usr/bin/grep: Permission denied" and certificate
+        # publishing stayed dead until the container was recreated. It never
+        # showed up on a workstation because an unprivileged chmod of a
+        # root-owned file fails EPERM and was swallowed by `except OSError`.
+        # This file ships in the image (COPY scripts /haproxy/scripts), so
+        # running it in place is a thing an operator will do.
+        for root, dirs, files in os.walk(self.tmp):
+            for name in files:
+                path = os.path.join(root, name)
+                if os.path.islink(path):
+                    continue
+                try:
+                    os.chmod(path, 0o600)
+                except OSError:
+                    pass
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    # -- helpers ---------------------------------------------------------
+    def env(self, **overrides):
+        env = dict(os.environ)
+        env.update({
+            'PATH': self.bindir + os.pathsep + os.environ['PATH'],
+            'SSL_CERTS_DIR': self.certs_dir,
+            'LETSENCRYPT_LIVE_DIR': self.le_live,
+            'CERT_STAGING_DIR': self.staging_dir,
+            'CERT_BACKUP_DIR': self.backup_dir,
+            'LOG_FILE': self.log_file,
+            'ERROR_LOG_FILE': self.error_log,
+            'HAPROXY_CONFIG': self.haproxy_cfg,
+            'SOCAT_LOG': self.socat_log,
+        })
+        env.update(overrides)
+        return env
+
+    def run_script(self, name, **env_overrides):
+        path = os.path.join(SCRIPTS_DIR, name)
+        self.assertTrue(os.path.exists(path), f'{path} does not exist')
+        return subprocess.run(['bash', path], env=self.env(**env_overrides),
+                              capture_output=True, text=True, timeout=120)
+
+    def bundle_is_valid(self, path):
+        """Ask the shipped library whether HAProxy could use this bundle."""
+        return subprocess.run(
+            ['bash', '-c', '. "$1"; cert_bundle_valid "$2"', '_', LIB, path],
+            env=self.env(), capture_output=True, text=True).returncode == 0
+
+    def reload_attempted(self):
+        return os.path.exists(self.socat_log) and os.path.getsize(self.socat_log) > 0
+
+    def logs(self):
+        text = ''
+        for path in (self.log_file, self.error_log):
+            if os.path.exists(path):
+                text += read(path)
+        return text
+
+    def seed_previous_bundle(self, content=PREVIOUS_BUNDLE):
+        return write(self.live_pem, content)
+
+    def assert_certs_dir_is_clean(self):
+        """HAProxy loads this directory wholesale: only final *.pem may be here."""
+        entries = sorted(os.listdir(self.certs_dir))
+        strays = [e for e in entries if not e.endswith('.pem')]
+        self.assertEqual(strays, [],
+                         f'non-.pem files left in the certs directory HAProxy '
+                         f'loads wholesale: {strays} (dir: {entries})')
+
+    def assert_no_staging_leftovers(self):
+        if os.path.isdir(self.staging_dir):
+            self.assertEqual(sorted(os.listdir(self.staging_dir)), [],
+                             'staging file was not cleaned up')
+
+    def assert_rejected(self, result, because):
+        """The library refused, FOR THE STATED REASON.
+
+        `assertNotEqual(rc, 0)` on its own proves nothing about this library.
+        Delete cert-publish-lib.sh and `. "$1"` fails, cert_bundle_valid is
+        never defined, bash exits 127 - and a bare rc!=0 assertion passes. Four
+        tests in TestCertPublishLibrary were doing exactly that; they were
+        pinning "some bash pipeline failed", not "the bundle was rejected".
+        """
+        output = result.stdout + result.stderr
+        self.assertNotIn('command not found', output,
+                         'the shell could not find the function under test - '
+                         'this asserts nothing about the library')
+        self.assertNotEqual(127, result.returncode,
+                            f'exit 127 means "no such command", not "rejected": {output}')
+        self.assertNotEqual(0, result.returncode,
+                            f'expected a rejection, got success: {output}')
+        self.assertIn(because, output,
+                      f'rejected, but not for the expected reason '
+                      f'({because!r} not in output): {output}')
+
+
+class CertScriptBehaviour:
+    """Behaviour shared by renew-certificates.sh and sync-certificates.sh.
+
+    A mixin rather than a TestCase so the cases are collected once per concrete
+    script, not a third time for the base class.
+    """
+
+    SCRIPT = None
+
+    def run_it(self, **env_overrides):
+        return self.run_script(self.SCRIPT, **env_overrides)
+
+    # -- happy path ------------------------------------------------------
+    def test_publish_replaces_live_pem_and_archives_the_previous_one(self):
+        self.seed_previous_bundle()
+        result = self.run_it()
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE,
+                         'live pem is not the newly assembled cert+key')
+        backup = os.path.join(self.backup_dir, DOMAIN + '.pem')
+        self.assertTrue(os.path.exists(backup),
+                        f'previous bundle was not archived to {backup}')
+        self.assertEqual(read(backup), PREVIOUS_BUNDLE,
+                         'the archived bundle is not the one that was replaced')
+        self.assertIn('1 updated, 0 failed', self.logs())
+        self.assertTrue(self.reload_attempted(),
+                        'HAProxy was never reloaded after a successful update')
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+
+    def test_first_publish_works_with_no_previous_bundle(self):
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+        self.assert_certs_dir_is_clean()
+
+    # -- THE HEADLINE ----------------------------------------------------
+    def test_unreadable_source_key_leaves_the_previous_bundle_intact(self):
+        """The bug this whole change exists for.
+
+        Pre-fix: `cat cert key > live.pem` truncated live.pem before cat ran,
+        so an unreadable key left a key-less (or empty) pem in the directory
+        HAProxy loads wholesale -> the :443 bind fails -> every site on the
+        host loses HTTPS. Checking cat's exit status did not undo that.
+        """
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        self.assertTrue(self.bundle_is_valid(self.live_pem),
+                        'fixture precondition: the seeded bundle must be valid')
+
+        how = self.break_source_key()
+
+        result = self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         f'the live pem was damaged by a failed publish ({how}); '
+                         f'HAProxy would fail to load the certs directory')
+        self.assertTrue(self.bundle_is_valid(self.live_pem),
+                        'the pem left on disk is no longer a usable bundle')
+        logs = self.logs()
+        self.assertIn(f'Failed to combine certificate for {DOMAIN}', logs)
+        self.assertIn('0 updated, 1 failed', logs)
+        self.assertFalse(self.reload_attempted(),
+                         'HAProxy was reloaded even though nothing was updated')
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+        # This used to assert returncode == 0 with the comment "a per-domain
+        # failure should not change the exit code", codifying the script's
+        # `exit 0`. That is wrong and it is the dangerous kind of wrong: a host
+        # where EVERY domain fails to publish looked, to cron and to
+        # host-renew-certificates.sh (which branches on this exit code),
+        # exactly like a clean run. Nothing would notice until the certificates
+        # expired. Continuing past a failed domain so the others still get
+        # published is right; reporting success afterwards is not.
+        self.assertNotEqual(result.returncode, 0,
+                            'a domain that failed to publish must be reported '
+                            'in the exit code, not just in the log')
+
+    def break_source_key(self):
+        """Make reading the source key fail, however this environment allows.
+
+        chmod 000 is the faithful reproduction (file present, `-f` true, cat
+        fails), but it is a no-op for root, so as root we truncate the key
+        instead: pre-fix that is even nastier, because `cat` then *succeeds*
+        and silently publishes a key-less pem.
+        """
+        if os.geteuid() == 0:
+            write(self.src_key, '')
+            return 'zero-length source key (running as root)'
+        os.chmod(self.src_key, 0o000)
+        return 'unreadable source key (chmod 000)'
+
+    # -- other ways to end up with an unusable bundle --------------------
+    def test_cert_only_bundle_is_rejected(self):
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(self.src_key, TEST_CERT)  # no private key block at all
+
+        self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         'a key-less bundle was published over the live pem')
+        self.assertIn('0 updated, 1 failed', self.logs())
+        self.assertFalse(self.reload_attempted())
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+
+    def test_truncated_certificate_block_is_rejected(self):
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(self.src_cert, TEST_CERT.split('\n')[0] + '\nMIIDFzCCAf+gAwIBA\n')
+
+        self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         'a truncated certificate was published over the live pem')
+        self.assertIn('0 updated, 1 failed', self.logs())
+        self.assert_certs_dir_is_clean()
+
+    def test_mismatched_key_is_rejected(self):
+        if shutil.which('openssl') is None:
+            self.skipTest('openssl CLI not available: without it every publish '
+                          'is refused, so this test could not tell a pairing '
+                          'rejection from a missing-checker rejection')
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(self.src_key, UNRELATED_KEY)
+
+        self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         'a bundle whose key does not match the cert was published')
+        self.assertIn('does not match the certificate', self.logs())
+        self.assertIn('0 updated, 1 failed', self.logs())
+        self.assert_certs_dir_is_clean()
+
+    # -- the certs directory is HAProxy's, not ours ----------------------
+    def test_no_stray_files_in_certs_dir_after_success_or_failure(self):
+        self.seed_previous_bundle()
+        self.run_it()
+        self.assert_certs_dir_is_clean()
+        self.assertEqual(sorted(os.listdir(self.certs_dir)), [DOMAIN + '.pem'])
+
+        self.break_source_key()
+        self.run_it()
+        self.assert_certs_dir_is_clean()
+        self.assertEqual(sorted(os.listdir(self.certs_dir)), [DOMAIN + '.pem'])
+        self.assert_no_staging_leftovers()
+
+    def test_staging_and_backup_dirs_are_outside_the_certs_dir(self):
+        """Belt and braces: even with the defaults, nothing lands under certs/."""
+        self.seed_previous_bundle()
+        # Drop the explicit overrides so the derived defaults are exercised.
+        env = {'CERT_STAGING_DIR': '', 'CERT_BACKUP_DIR': ''}
+        self.run_it(**env)
+
+        self.assert_certs_dir_is_clean()
+        self.assertEqual(sorted(os.listdir(self.certs_dir)), [DOMAIN + '.pem'])
+        self.assertTrue(
+            os.path.exists(os.path.join(self.haproxy_dir, 'cert-staging')),
+            'default staging dir is not the documented sibling of the certs dir')
+        self.assertTrue(
+            os.path.exists(os.path.join(self.haproxy_dir, 'cert-backups',
+                                        DOMAIN + '.pem')),
+            'default backup dir is not the documented sibling of the certs dir')
+
+    # -- reload gating ---------------------------------------------------
+    def test_reload_is_not_attempted_when_haproxy_config_is_invalid(self):
+        write(self.haproxy_cfg, GOOD_HAPROXY_CFG + BROKEN_TOKEN + '\n')
+
+        result = self.run_it()
+
+        self.assertFalse(self.reload_attempted(),
+                         'HAProxy was reloaded with a configuration that '
+                         '`haproxy -c` rejects')
+        self.assertNotEqual(result.returncode, 0,
+                            'refusing to reload must be a loud, non-zero exit')
+        self.assertIn('does not validate', self.logs())
+
+    def test_reload_happens_when_the_config_validates(self):
+        self.run_it()
+        self.assertTrue(self.reload_attempted())
+        self.assertIn('reload', read(self.socat_log))
+
+    def test_reload_is_not_attempted_when_nothing_was_updated(self):
+        shutil.rmtree(self.domain_dir)
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0)
+        self.assertFalse(self.reload_attempted())
+
+
+class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture):
+    SCRIPT = 'renew-certificates.sh'
+
+
+class TestSyncCertificates(CertScriptBehaviour, CertScriptFixture):
+    SCRIPT = 'sync-certificates.sh'
+
+
+class TestCertPublishLibrary(CertScriptFixture):
+    """Unit-level checks on cert-publish-lib.sh itself."""
+
+    def call(self, snippet, *args, **env_overrides):
+        return subprocess.run(
+            ['bash', '-c', '. "$1"; shift; ' + snippet, '_', LIB, *args],
+            env=self.env(**env_overrides), capture_output=True, text=True)
+
+    def test_valid_bundle_accepted(self):
+        path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
+        self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0)
+
+    def test_empty_and_missing_bundles_rejected(self):
+        empty = write(os.path.join(self.tmp, 'empty.pem'), '')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),
+                             'is empty')
+        missing = os.path.join(self.tmp, 'nope.pem')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', missing),
+                             'does not exist')
+
+    def test_key_without_end_marker_rejected(self):
+        truncated = write(os.path.join(self.tmp, 'cut.pem'),
+                          TEST_CERT + '-----BEGIN PRIVATE KEY-----\nMIIEvAIB\n')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', truncated),
+                             'unterminated private key block')
+
+    def test_empty_pem_blocks_are_rejected(self):
+        """Why the pairing check is mandatory rather than best-effort.
+
+        Every structural check in the library passes on this file: a complete
+        CERTIFICATE block and a complete PRIVATE KEY block, both with nothing
+        between BEGIN and END. Only openssl can tell it is not a certificate.
+        """
+        hollow = write(os.path.join(self.tmp, 'hollow.pem'),
+                       '-----BEGIN CERTIFICATE-----\n'
+                       '-----END CERTIFICATE-----\n'
+                       '-----BEGIN PRIVATE KEY-----\n'
+                       '-----END PRIVATE KEY-----\n')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', hollow),
+                             'openssl could not read the certificate')
+
+    def test_a_broken_live_pem_does_not_overwrite_a_good_backup(self):
+        """Mirrors create_backup(require_valid=True) in haproxy_manager.py.
+
+        If the pem currently on disk is already garbage, archiving it would
+        replace a restorable backup with an unusable one.
+        """
+        os.makedirs(self.backup_dir)
+        good_backup = write(os.path.join(self.backup_dir, DOMAIN + '.pem'),
+                            PREVIOUS_BUNDLE)
+        write(self.live_pem, 'garbage, not a pem at all\n')
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+        self.assertEqual(read(good_backup), PREVIOUS_BUNDLE,
+                         'a good backup was overwritten with an unusable pem')
+
+    def test_publish_fails_loudly_when_the_rename_cannot_happen(self):
+        """No silent fallback to writing straight into the certs dir.
+
+        The failure is injected with a stub `mv` that refuses, rather than by
+        chmod 0500 on the certs dir: the container these scripts run in is
+        root, root ignores directory permissions, so the chmod version skipped
+        itself exactly where it matters and only ever ran on a workstation.
+        """
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(os.path.join(self.bindir, 'mv'),
+              "#!/bin/sh\n"
+              "echo \"mv: cannot move '$2': Permission denied\" >&2\n"
+              "exit 1\n", 0o755)
+        self.addCleanup(os.unlink, os.path.join(self.bindir, 'mv'))
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+
+        self.assert_rejected(result, 'NOT falling back to a direct write')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was damaged by a failed rename')
+        self.assert_no_staging_leftovers()
+
+    def test_published_pem_keeps_the_mode_of_the_file_it_replaces(self):
+        """A write-safety fix must not silently re-permission private keys.
+
+        Both directions matter: mktemp stages at 0600, so without the explicit
+        chmod every publish would tighten a 0644 bundle; and the mode must not
+        be copied from a symlink (see the next test).
+        """
+        for mode in (0o644, 0o640, 0o600):
+            with self.subTest(oct(mode)):
+                write(self.live_pem, PREVIOUS_BUNDLE, mode)
+                result = self.call('cert_publish "$1" "$2" "$3"',
+                                   self.src_cert, self.src_key, self.live_pem)
+                self.assertEqual(result.returncode, 0,
+                                 result.stdout + result.stderr)
+                self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+                self.assertEqual(
+                    stat.S_IMODE(os.stat(self.live_pem).st_mode), mode,
+                    'publishing changed who can read the private key')
+
+    def test_symlinked_live_pem_does_not_become_world_writable(self):
+        """`stat -c %a` on a symlink reports 0777 - the LINK's mode, not a
+        permission. Copying that onto the staged bundle put a world-writable
+        private key in the directory HAProxy serves from. -L is what makes the
+        preserved mode the mode of the file an operator actually chose.
+        """
+        target = write(os.path.join(self.tmp, 'real-bundle.pem'),
+                       PREVIOUS_BUNDLE, 0o640)
+        os.symlink(target, self.live_pem)
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        mode = stat.S_IMODE(os.stat(self.live_pem).st_mode)
+        self.assertEqual(
+            0, mode & 0o022,
+            f'published bundle is group/world writable ({oct(mode)}) - the '
+            f'symlink mode was copied onto a real private key')
+        self.assertEqual(0o640, mode)
+
+    def test_publish_refuses_when_staging_is_on_another_filesystem(self):
+        """The header used to claim a cross-device mv "fails loudly and leaves
+        the live pem alone". GNU mv does no such thing: across filesystems it
+        copies, so it truncates and writes the DESTINATION first and only then
+        discovers it cannot finish (ENOSPC being the realistic case) - the very
+        truncation this library exists to prevent. Both directories are
+        env-overridable, so the device numbers have to be checked up front.
+        """
+        staging = os.path.join(self.a_dir_on_another_filesystem(),
+                               'cert-staging')
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem,
+                           CERT_STAGING_DIR=staging)
+
+        self.assert_rejected(result, 'different filesystems')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was disturbed by a refused publish')
+        self.assert_certs_dir_is_clean()
+
+    def test_stale_python_side_staging_temps_are_reaped(self):
+        """The staging dir has two writers.
+
+        write_config_atomically() on the Python side stages as
+        `<name>.<random>.tmp` (tempfile.mkstemp(prefix=name + '.',
+        suffix='.tmp')). The reaper matched only mktemp's `*.??????` shape, so
+        every temp leaked by a SIGKILL on the Python side stayed there forever.
+        """
+        os.makedirs(self.staging_dir, exist_ok=True)
+        stale_py = write(os.path.join(self.staging_dir,
+                                      DOMAIN + '.pem.ab12cd34.tmp'), 'stale\n')
+        stale_sh = write(os.path.join(self.staging_dir,
+                                      DOMAIN + '.pem.AbCdEf'), 'stale\n')
+        fresh = write(os.path.join(self.staging_dir,
+                                   'recent.pem.zz99yy.tmp'), 'fresh\n')
+        old = time.time() - 3 * 24 * 3600
+        for path in (stale_py, stale_sh):
+            os.utime(path, (old, old))
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+
+        self.assertFalse(os.path.exists(stale_py),
+                         'a stale Python-side staging temp was never reaped')
+        self.assertFalse(os.path.exists(stale_sh),
+                         'a stale shell-side staging temp was never reaped')
+        self.assertTrue(os.path.exists(fresh),
+                        'the reaper deleted a temp a concurrent publisher may '
+                        'still be writing')
+
+    def test_validation_of_a_live_pem_being_republished_is_not_spurious(self):
+        """cert_bundle_valid must judge ONE snapshot of the file.
+
+        cert_publish() calls cert_bundle_valid() on the LIVE pem (step (e), to
+        decide whether it is worth backing up) while another publisher may be
+        renaming a new bundle over it. The function used to open the file six
+        times, so `openssl x509` could read the outgoing bundle and `openssl
+        pkey` the incoming one - and report "private key does not match the
+        certificate" about two files that were each perfectly fine. That ERROR
+        goes into the log monitor-errors.sh watches, which makes it a page.
+
+        The two bundles alternated below are each internally valid but carry
+        DIFFERENT key material. That matters: NEW_BUNDLE and PREVIOUS_BUNDLE
+        share a cert and a key, so alternating those two could never produce a
+        mismatch no matter how badly the reads were interleaved - the test
+        would model a world in which the bug cannot happen and pass forever.
+        """
+        write(self.live_pem, NEW_BUNDLE)
+        a = write(os.path.join(self.tmp, 'churn-a.pem'), NEW_BUNDLE)
+        b = write(os.path.join(self.tmp, 'churn-b.pem'), OTHER_BUNDLE)
+
+        # A publisher renaming over the live pem as fast as it can. Staged
+        # outside the certs dir, then renamed, exactly like cert_publish().
+        churn = subprocess.Popen(
+            ['bash', '-c',
+             'end=$((SECONDS+8)); s="$4"; while [ $SECONDS -lt $end ]; do '
+             '  cp "$1" "$s"; mv -f "$s" "$2"; '
+             '  cp "$3" "$s"; mv -f "$s" "$2"; '
+             'done', '_', a, self.live_pem, b,
+             os.path.join(self.tmp, 'churn-staged.pem')],
+            env=self.env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+        def _stop():
+            churn.kill()
+            churn.wait()
+        self.addCleanup(_stop)
+
+        result = subprocess.run(
+            ['bash', '-c',
+             '. "$1"; for i in $(seq 1 200); do cert_bundle_valid "$2" || exit 1; done',
+             '_', LIB, self.live_pem],
+            env=self.env(), capture_output=True, text=True, timeout=120)
+        _stop()
+
+        output = result.stdout + result.stderr
+        self.assertNotIn('does not match the certificate', output,
+                         'two valid bundles were reported as a mismatched pair '
+                         'because the checks read different files')
+        self.assertEqual(0, result.returncode,
+                         f'a concurrent republish made validation fail: {output}')
+
+    def a_dir_on_another_filesystem(self):
+        certs_dev = os.stat(self.certs_dir).st_dev
+        for candidate in ('/dev/shm', '/run', '/var/tmp', '/tmp', '/'):
+            try:
+                if (os.path.isdir(candidate)
+                        and os.access(candidate, os.W_OK)
+                        and os.stat(candidate).st_dev != certs_dev):
+                    path = tempfile.mkdtemp(prefix='cert-xdev-', dir=candidate)
+                    self.addCleanup(shutil.rmtree, path, True)
+                    return path
+            except OSError:
+                continue
+        self.skipTest('no writable directory on a second filesystem available')
+
+    def test_haproxy_config_ok_follows_the_validator(self):
+        self.assertEqual(self.call('haproxy_config_ok').returncode, 0)
+        write(self.haproxy_cfg, GOOD_HAPROXY_CFG + BROKEN_TOKEN + '\n')
+        self.assertNotEqual(self.call('haproxy_config_ok').returncode, 0)
+
+    def _openssl_free_path(self, name):
+        """A PATH directory with the library's tools but no openssl.
+
+        Symlinks, so this directory must never be walked with a chmod that
+        follows them - see _cleanup_tmp().
+        """
+        fake_path = os.path.join(self.tmp, name)
+        os.makedirs(fake_path)
+        for tool in ('cat', 'grep', 'mktemp', 'mv', 'cp', 'rm', 'mkdir',
+                     'basename', 'dirname', 'find', 'date', 'chmod', 'stat'):
+            real = shutil.which(tool)
+            if real:
+                os.symlink(real, os.path.join(fake_path, tool))
+        return fake_path
+
+    def test_missing_openssl_is_a_hard_failure(self):
+        """The pairing check is MANDATORY: no openssl, no publication.
+
+        This used to assert the opposite - that a missing openssl warns and
+        publishes anyway - justified by "the image does not necessarily install
+        the openssl CLI". The image does: openssl 3.x arrives with
+        ca-certificates, which certbot needs, and generate_self_signed_cert()
+        already runs `openssl req` with check=True at first-run setup. So the
+        fail-open never actually fired, and structural checks alone accept a
+        bundle of empty pem blocks (see test_empty_pem_blocks_are_rejected).
+        """
+        fake_path = self._openssl_free_path('no-openssl-bin')
+        path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
+
+        # bash by absolute path: the stripped PATH cannot resolve it.
+        result = subprocess.run(
+            [shutil.which('bash'), '-c', '. "$1"; cert_bundle_valid "$2"',
+             '_', LIB, path],
+            env=self.env(PATH=fake_path), capture_output=True, text=True)
+
+        self.assert_rejected(result, 'openssl binary not found')
+        self.assertIn('REFUSING', result.stdout + result.stderr,
+                      'a broken image must be reported as a broken image')
+
+    def test_missing_openssl_stops_a_publish_rather_than_weakening_it(self):
+        """cert_publish must inherit the refusal, and not touch the live pem."""
+        fake_path = self._openssl_free_path('no-openssl-bin2')
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+
+        result = subprocess.run(
+            [shutil.which('bash'), '-c',
+             '. "$1"; cert_publish "$2" "$3" "$4"',
+             '_', LIB, self.src_cert, self.src_key, self.live_pem],
+            env=self.env(PATH=fake_path), capture_output=True, text=True)
+
+        self.assert_rejected(result, 'openssl binary not found')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was disturbed by a refused publish')
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+
+
+class TestScriptsAreSane(unittest.TestCase):
+    """Cheap static guards against the failure mode coming back."""
+
+    SHELL_FILES = ('renew-certificates.sh', 'sync-certificates.sh',
+                   'cert-publish-lib.sh')
+
+    def test_shell_files_parse(self):
+        for name in self.SHELL_FILES:
+            path = os.path.join(SCRIPTS_DIR, name)
+            result = subprocess.run(['bash', '-n', path],
+                                    capture_output=True, text=True)
+            self.assertEqual(result.returncode, 0,
+                             f'{name}: {result.stderr}')
+
+    def test_no_script_redirects_into_the_live_pem(self):
+        pattern = re.compile(r'>\s*"?\$\{?COMBINED_FILE')
+        for name in ('renew-certificates.sh', 'sync-certificates.sh'):
+            body = read(os.path.join(SCRIPTS_DIR, name))
+            self.assertIsNone(pattern.search(body),
+                              f'{name} still redirects output straight into the '
+                              f'live pem HAProxy is serving')
+
+
+if __name__ == '__main__':
+    print(f'testing scripts from: {SCRIPTS_DIR}')
+    unittest.main(verbosity=2)

scripts/test-config-rollback.py

@@ -0,0 +1,775 @@
+#!/usr/bin/env python3
+"""Regression tests for HAProxy config backup / rollback ordering.
+
+Why this file exists
+--------------------
+generate_config() used to write the new haproxy.cfg and only THEN call
+reload_haproxy_safely() -> create_backup(), so the "backup" was a copy of the
+config that had just been written. On a validation failure restore_backup()
+restored the identical broken bytes: the advertised rollback was a no-op and a
+fatal haproxy.cfg stayed on disk, where start_haproxy() refuses to launch.
+
+These tests pin the ordering invariant (backup predates the write) and the
+observable end-to-end behaviour (after a failed validation the file on disk is
+the previous working config and HAProxy will start with it).
+
+Running
+-------
+    python3 scripts/test-config-rollback.py           # tests the repo checkout
+    HAPROXY_MANAGER_DIR=/some/other/tree \
+        python3 scripts/test-config-rollback.py       # tests another tree
+
+The repo has no Python test framework (scripts/test-*.sh are curl-based
+integration scripts against a running API), so this is a self-contained
+stdlib-unittest script - no pytest, no venv, no new dependencies beyond the
+application's own requirements.txt (Flask/Jinja2/psutil), which are already
+present in the container image.
+
+No HAProxy binary is required: a stub `haproxy` is put on PATH that mimics
+`haproxy -c -f <file>` by rejecting any config containing the token
+__BROKEN__, which is how the tests inject an invalid configuration.
+"""
+
+import os
+import re
+import sys
+import shutil
+import inspect
+import sqlite3
+import logging
+import tempfile
+import textwrap
+import unittest
+
+BROKEN_TOKEN = '__BROKEN__'
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+
+# haproxy_manager builds its Jinja2 environment from the relative path
+# Path('templates'), so it has to be imported with the module dir as cwd.
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+# The module opens /var/log/haproxy-manager.log at import time via
+# logging.FileHandler. Redirect that one call so the suite runs unprivileged.
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda fn, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(fn)), *a, **kw)
+)
+try:
+    import haproxy_manager as hm
+except ImportError as exc:  # pragma: no cover - environment problem, not a failure
+    sys.stderr.write(
+        f"SKIP: cannot import haproxy_manager ({exc}).\n"
+        "Install the application requirements first: pip install -r requirements.txt\n"
+    )
+    raise SystemExit(77)
+finally:
+    logging.FileHandler = _real_file_handler
+
+logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
+
+FAKE_HAPROXY = textwrap.dedent(f"""\
+    #!/bin/sh
+    # Test stub for the haproxy binary.
+    #   haproxy -c -f FILE   -> exit 1 if FILE contains {BROKEN_TOKEN}, else 0
+    #   haproxy -W -S ... -f FILE (start) -> same validation, then exit 0
+    cfg=""
+    while [ $# -gt 0 ]; do
+      case "$1" in -f) cfg="$2"; shift ;; esac
+      shift
+    done
+    if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then
+      echo "[ALERT] parsing [$cfg:1] : unknown keyword '{BROKEN_TOKEN}'" >&2
+      exit 1
+    fi
+    exit 0
+""")
+
+
+class RollbackTestCase(unittest.TestCase):
+    """Base fixture: an isolated fake /etc/haproxy plus a stub haproxy binary."""
+
+    def setUp(self):
+        self.tmp = tempfile.mkdtemp(prefix='haproxy-rollback-test-')
+        self.addCleanup(shutil.rmtree, self.tmp, True)
+
+        bindir = os.path.join(self.tmp, 'bin')
+        os.makedirs(bindir)
+        stub = os.path.join(bindir, 'haproxy')
+        with open(stub, 'w') as fh:
+            fh.write(FAKE_HAPROXY)
+        os.chmod(stub, 0o755)
+        self._old_path = os.environ['PATH']
+        os.environ['PATH'] = bindir + os.pathsep + self._old_path
+        self.addCleanup(lambda: os.environ.__setitem__('PATH', self._old_path))
+
+        self.etc = os.path.join(self.tmp, 'etc')
+        os.makedirs(self.etc)
+
+        overrides = {
+            'DB_FILE': os.path.join(self.etc, 'haproxy_config.db'),
+            'HAPROXY_CONFIG_PATH': os.path.join(self.etc, 'haproxy.cfg'),
+            'HAPROXY_BACKUP_PATH': os.path.join(self.etc, 'haproxy.cfg.backup'),
+            'BLOCKED_IPS_MAP_PATH': os.path.join(self.etc, 'blocked_ips.map'),
+            'BLOCKED_IPS_MAP_BACKUP_PATH': os.path.join(self.etc, 'blocked_ips.map.backup'),
+            'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'),
+            'SSL_CERTS_DIR': os.path.join(self.etc, 'certs'),
+            'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'),
+            # Added by the rollback fix; older trees do not have it.
+            'CORAZA_SPOE_CONFIG_PATH': os.path.join(self.etc, 'coraza-spoe.cfg'),
+            'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'),
+        }
+        self._saved = {}
+        for name, value in overrides.items():
+            self._saved[name] = getattr(hm, name, None)
+            setattr(hm, name, value)
+        self.addCleanup(self._restore_globals)
+        os.makedirs(hm.SSL_CERTS_DIR)
+
+        # log_operation() appends to a hardcoded /var/log path. Injecting `open`
+        # into the module namespace shadows the builtin for that module only
+        # (module globals are searched before builtins), so the real
+        # log_operation code still runs.
+        real_open = open
+        log_dir = self.tmp
+
+        def _redirecting_open(path, *args, **kwargs):
+            if isinstance(path, str) and path.startswith('/var/log/'):
+                path = os.path.join(log_dir, os.path.basename(path))
+            return real_open(path, *args, **kwargs)
+
+        hm.open = _redirecting_open
+        self.addCleanup(lambda: hm.__dict__.pop('open', None))
+
+        hm.init_db()
+
+    def _restore_globals(self):
+        for name, value in self._saved.items():
+            if value is None:
+                hm.__dict__.pop(name, None)
+            else:
+                setattr(hm, name, value)
+
+    # -- helpers ---------------------------------------------------------
+    def add_domain(self, domain, backend_name, address='10.0.0.1'):
+        with sqlite3.connect(hm.DB_FILE) as conn:
+            cur = conn.cursor()
+            cur.execute('INSERT INTO domains (domain, ssl_enabled) VALUES (?, 0)',
+                        (domain,))
+            domain_id = cur.lastrowid
+            cur.execute('INSERT INTO backends (name, domain_id) VALUES (?, ?)',
+                        (backend_name, domain_id))
+            backend_id = cur.lastrowid
+            cur.execute(
+                'INSERT INTO backend_servers '
+                '(backend_id, server_name, server_address, server_port) '
+                'VALUES (?, ?, ?, ?)',
+                (backend_id, 'srv1', address, 8080))
+            conn.commit()
+
+    def block_ip(self, ip):
+        with sqlite3.connect(hm.DB_FILE) as conn:
+            conn.execute('INSERT INTO blocked_ips (ip_address, reason) VALUES (?, ?)',
+                         (ip, 'test'))
+            conn.commit()
+
+    def read(self, path):
+        with open(path) as fh:
+            return fh.read()
+
+    def config_is_loadable(self):
+        """True if HAProxy would accept the config currently on disk."""
+        import subprocess
+        return subprocess.run(
+            ['haproxy', '-c', '-f', hm.HAPROXY_CONFIG_PATH],
+            capture_output=True).returncode == 0
+
+    def generate_good_config(self):
+        self.add_domain('good.example.com', 'good_backend')
+        hm.generate_config()
+        self.assertTrue(self.config_is_loadable(),
+                        'fixture precondition: first generated config must be valid')
+        return self.read(hm.HAPROXY_CONFIG_PATH)
+
+    def break_the_config(self):
+        """Queue a domain whose rendered backend the validator rejects."""
+        self.add_domain('bad.example.com', BROKEN_TOKEN + '_backend', '10.0.0.2')
+
+
+class TestBackupOrdering(RollbackTestCase):
+
+    def test_backup_is_taken_before_the_new_config_is_written(self):
+        """The ordering invariant, asserted directly.
+
+        Whatever create_backup() sees on disk must be the OLD config; if the
+        write happens first the backup is a copy of the new config and rollback
+        is meaningless.
+        """
+        good = self.generate_good_config()
+
+        seen = {}
+        real_create_backup = hm.create_backup
+
+        def spy(*args, **kwargs):
+            seen['config_on_disk'] = self.read(hm.HAPROXY_CONFIG_PATH)
+            return real_create_backup(*args, **kwargs)
+
+        hm.create_backup = spy
+        self.addCleanup(setattr, hm, 'create_backup', real_create_backup)
+
+        self.add_domain('second.example.com', 'second_backend', '10.0.0.3')
+        hm.generate_config()
+
+        self.assertIn('config_on_disk', seen,
+                      'create_backup() was never called during generate_config()')
+        self.assertEqual(
+            seen['config_on_disk'], good,
+            'create_backup() ran AFTER the new config was written - the backup '
+            'is a copy of the new config, so rollback cannot undo anything')
+
+    def test_backup_tracks_the_last_known_good_config(self):
+        """After a change that validated AND loaded, the backup is that config.
+
+        The rollback target is "the last configuration HAProxy actually ran",
+        not "the file that happened to be there last time".
+        """
+        good = self.generate_good_config()
+        self.add_domain('second.example.com', 'second_backend', '10.0.0.3')
+        hm.generate_config()
+
+        live = self.read(hm.HAPROXY_CONFIG_PATH)
+        self.assertNotEqual(live, good, 'fixture sanity: the new config should differ')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), live,
+                         'the successful config was not recorded as known-good')
+
+    def test_backup_is_not_promoted_when_the_change_fails(self):
+        """A config that never loaded must not become the rollback target."""
+        good = self.generate_good_config()
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'a config that failed validation was promoted to backup')
+
+
+class TestRollbackEndToEnd(RollbackTestCase):
+
+    def test_failed_validation_leaves_the_last_good_config_on_disk(self):
+        good = self.generate_good_config()
+
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        on_disk = self.read(hm.HAPROXY_CONFIG_PATH)
+        self.assertNotIn(BROKEN_TOKEN, on_disk,
+                         'the rejected config is still on disk - rollback was a no-op')
+        self.assertEqual(on_disk, good,
+                         'on-disk config is not byte-identical to the last good one')
+
+    def test_haproxy_would_still_start_after_a_failed_change(self):
+        """The operational consequence: the edge can still come up."""
+        self.generate_good_config()
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        self.assertTrue(self.config_is_loadable(),
+                        'HAProxy would refuse to start with the config left on disk')
+        with self.assertLogs('haproxy_manager', level='INFO') as captured:
+            hm.start_haproxy()
+        self.assertTrue(
+            any('HAProxy started successfully' in line for line in captured.output),
+            f'start_haproxy() did not succeed after rollback: {captured.output}')
+
+    def test_blocked_ips_map_is_rolled_back_too(self):
+        """generate_config() rewrites the map file before writing haproxy.cfg."""
+        self.block_ip('192.0.2.10')
+        self.generate_good_config()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        self.block_ip('198.51.100.20')
+        self.break_the_config()
+        with self.assertRaises(Exception):
+            hm.generate_config()
+
+        self.assertEqual(self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+                         'blocked IPs map was not rolled back with the config')
+
+    def test_first_run_failure_reports_that_rollback_was_impossible(self):
+        """No prior config: there is nothing to restore, and that must be said.
+
+        A missing backup must never be reported as a successful restore, and it
+        must never be turned into "restore an empty file".
+        """
+        self.break_the_config()
+        with self.assertRaises(Exception) as ctx:
+            hm.generate_config()
+
+        self.assertIn('ROLLBACK FAILED', str(ctx.exception),
+                      'a failed change with no backup was not reported as such')
+        self.assertFalse(os.path.exists(hm.HAPROXY_BACKUP_PATH),
+                         'a backup was fabricated from the broken config')
+        # The broken config is deliberately left in place: start_haproxy() can
+        # then detect it and try to regenerate. It must not be blanked.
+        self.assertGreater(os.path.getsize(hm.HAPROXY_CONFIG_PATH), 0,
+                           'config file was emptied instead of left for diagnosis')
+
+
+class TestBackupPrimitives(RollbackTestCase):
+
+    def test_restore_backup_distinguishes_missing_backup_from_success(self):
+        restored, message = hm.restore_backup()
+        self.assertFalse(restored,
+                         'restore_backup() reported success with no backup present')
+        self.assertIn('cannot roll back', message.lower())
+
+        good = self.generate_good_config()
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write('scribbled over\n')
+
+        restored, message = hm.restore_backup()
+        self.assertTrue(restored, message)
+        self.assertEqual(self.read(hm.HAPROXY_CONFIG_PATH), good)
+
+    def test_a_successful_generation_records_a_rollback_target(self):
+        """Even the first-ever generation must leave something to roll back to."""
+        good = self.generate_good_config()
+        self.assertTrue(
+            os.path.exists(hm.HAPROXY_BACKUP_PATH),
+            'after a successful reload there is still no known-good backup')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
+
+    def test_a_broken_current_config_does_not_replace_a_good_backup(self):
+        """The known-good marker.
+
+        If the config already on disk is broken (previous failed write, manual
+        edit), snapshotting it would make "rollback" mean "restore a different
+        broken config". The older validated backup must survive.
+        """
+        good = self.generate_good_config()
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'fixture: a good backup should exist by now')
+
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write(f'garbage {BROKEN_TOKEN} config\n')
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'kept_previous')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'a broken config overwrote the known-good backup')
+
+    def test_reload_does_not_take_its_own_backup(self):
+        """reload_haproxy_safely() runs after the write, so it must not back up."""
+        good = self.generate_good_config()
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write(f'broken {BROKEN_TOKEN}\n')
+
+        success, message = hm.reload_haproxy_safely(backup_status='created')
+
+        self.assertFalse(success)
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'reload_haproxy_safely() overwrote the good backup')
+        self.assertEqual(self.read(hm.HAPROXY_CONFIG_PATH), good,
+                         'reload_haproxy_safely() did not roll the config back')
+
+    def test_unchanged_config_is_not_revalidated(self):
+        """Fast path: if the backup already is the live config, do no work.
+
+        generate_config() runs inside customer-facing API calls and
+        `haproxy -c` is expensive on an edge with hundreds of certificates.
+        """
+        self.generate_good_config()
+
+        calls = []
+        real_validate = hm.validate_config_file
+        hm.validate_config_file = lambda path: (calls.append(path),
+                                                real_validate(path))[1]
+        self.addCleanup(setattr, hm, 'validate_config_file', real_validate)
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'created')
+        self.assertEqual(calls, [],
+                         'the unchanged live config was re-validated needlessly')
+
+    def test_fast_path_does_not_hide_a_drifted_broken_config(self):
+        """If the live config drifted from the backup, the gate must still run."""
+        good = self.generate_good_config()
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write(f'hand edited {BROKEN_TOKEN}\n')
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'kept_previous',
+                         'a drifted broken config was silently accepted')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
+
+    def test_backup_set_covers_every_file_generate_config_writes(self):
+        """Derived, not restated.
+
+        An earlier version of this test listed the three files it expected and
+        checked they were in the backup set, so it could never have noticed a
+        FOURTH file being added. Here the set of files generate_config() writes
+        is observed (and, for env-gated branches this fixture cannot safely
+        execute, read out of the source), and anything not backed up has to be
+        on the documented exclusion list below.
+        """
+        # Written by generate_config() but deliberately NOT restorable, with
+        # the reason. Everything else must be in the backup set: `haproxy -c`
+        # validates the config as a set, so a file it loads that is not
+        # restored alongside haproxy.cfg breaks rollback.
+        excluded = {
+            # Only ever created empty-when-missing (haproxy refuses to start
+            # with an ACL -f pointing at a missing file); its contents are
+            # owned by the /suspended API, not by generate_config(), so there
+            # is nothing here for a config rollback to undo.
+            'suspended_domains.list',
+            # Generated once and then read, never rewritten with new content.
+            # Its value is rendered INTO haproxy.cfg, so restoring an older
+            # haproxy.cfg alongside the current secret file is consistent.
+            'cluster-secret',
+        }
+        backed_up = {os.path.basename(p) for p, _ in hm._config_backup_pairs()}
+
+        # 1. Observed: run a generation with every patchable optional branch on
+        #    and see what actually changed on disk.
+        self.add_domain('derive.example.com', 'derive_backend')
+        os.environ['HAPROXY_CORAZA_SPOE_BACKEND'] = '127.0.0.1:9000'
+        self.addCleanup(os.environ.pop, 'HAPROXY_CORAZA_SPOE_BACKEND', None)
+        before = self._snapshot_etc()
+        hm.generate_config()
+        after = self._snapshot_etc()
+        touched = {name for name, blob in after.items()
+                   if before.get(name) != blob}
+        self.assertIn('coraza-spoe.cfg', touched,
+                      'fixture precondition: the Coraza branch did not run')
+
+        # 2. Read out of the source: branches this fixture must not execute
+        #    (suspension writes a hardcoded /etc/haproxy path that no test
+        #    global can redirect) still have to be accounted for.
+        touched |= {
+            os.path.basename(m)
+            for m in re.findall(r"'(/etc/haproxy/[\w.+-]+)'",
+                                inspect.getsource(hm.generate_config))
+        }
+
+        unaccounted = touched - backed_up - excluded
+        self.assertEqual(
+            unaccounted, set(),
+            f'generate_config() writes {sorted(unaccounted)}, which is neither '
+            'in the backup set nor on the documented exclusion list - a '
+            'rollback would restore a mixed-vintage config set')
+
+    def _snapshot_etc(self):
+        """Contents of every plain file in the fake /etc/haproxy.
+
+        Skips the backup halves (they are the thing being maintained), the
+        SQLite database and its journals, and the stats socket.
+        """
+        skip_prefixes = (os.path.basename(hm.DB_FILE),
+                         os.path.basename(hm.HAPROXY_SOCKET_PATH))
+        state = {}
+        for name in os.listdir(self.etc):
+            path = os.path.join(self.etc, name)
+            if not os.path.isfile(path) or name.endswith('.backup'):
+                continue
+            if name.startswith(skip_prefixes):
+                continue
+            with open(path, 'rb') as fh:
+                state[name] = fh.read()
+        return state
+
+    def test_coraza_spoe_config_round_trips(self):
+        self.generate_good_config()
+        with open(hm.CORAZA_SPOE_CONFIG_PATH, 'w') as fh:
+            fh.write('spoe-good\n')
+        hm.create_backup()
+        with open(hm.CORAZA_SPOE_CONFIG_PATH, 'w') as fh:
+            fh.write('spoe-broken\n')
+        restored, message = hm.restore_backup()
+        self.assertTrue(restored, message)
+        self.assertEqual(self.read(hm.CORAZA_SPOE_CONFIG_PATH), 'spoe-good\n')
+
+
+class TestAtomicWrite(RollbackTestCase):
+
+    def test_write_is_atomic_and_preserves_mode(self):
+        path = os.path.join(self.etc, 'atomic.cfg')
+        with open(path, 'w') as fh:
+            fh.write('old')
+        os.chmod(path, 0o644)
+
+        hm.write_config_atomically(path, 'new content\n')
+
+        self.assertEqual(self.read(path), 'new content\n')
+        self.assertEqual(oct(os.stat(path).st_mode & 0o777), oct(0o644))
+        leftovers = [n for n in os.listdir(self.etc) if n.endswith('.tmp')]
+        self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
+
+    def test_failed_write_leaves_the_previous_file_intact(self):
+        path = os.path.join(self.etc, 'atomic.cfg')
+        with open(path, 'w') as fh:
+            fh.write('old content\n')
+
+        # Anything that makes f.write() blow up mid-flight stands in for a full
+        # disk / killed container. TypeError specifically, not Exception: a
+        # bare assertRaises(Exception) also swallows the AttributeError raised
+        # when write_config_atomically does not exist at all, so this test
+        # passed against the pre-fix tree and would keep passing if the
+        # function were deleted.
+        with self.assertRaises(TypeError):
+            hm.write_config_atomically(path, object())
+
+        self.assertEqual(self.read(path), 'old content\n',
+                         'a failed write clobbered the previous config')
+        leftovers = [n for n in os.listdir(self.etc) if n.endswith('.tmp')]
+        self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
+
+
+class TestBackupFailureGuard(RollbackTestCase):
+    """generate_config() refuses to write when no rollback target could be taken."""
+
+    def test_a_failed_backup_stops_the_config_from_being_written(self):
+        good = self.generate_good_config()
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        real_create_backup = hm.create_backup
+        hm.create_backup = lambda *a, **kw: (False, 'error')
+        self.addCleanup(setattr, hm, 'create_backup', real_create_backup)
+
+        self.add_domain('second.example.com', 'second_backend', '10.0.0.3')
+        with self.assertRaises(Exception) as ctx:
+            hm.generate_config()
+
+        self.assertIn('Refusing to regenerate', str(ctx.exception),
+                      'a backup failure was not reported as a refusal')
+        self.assertEqual(
+            self.read(hm.HAPROXY_CONFIG_PATH), good,
+            'a new config was written even though the snapshot failed - a bad '
+            'change could not have been rolled back')
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+            'the blocked IPs map was rewritten even though the snapshot failed')
+
+
+class TestValidatorAvailability(RollbackTestCase):
+    """'the validator could not run' is not the same as 'the config is bad'."""
+
+    def _hide_the_haproxy_binary(self):
+        empty = os.path.join(self.tmp, 'empty-bin')
+        os.makedirs(empty, exist_ok=True)
+        os.environ['PATH'] = empty
+        # setUp's cleanup restores the original PATH.
+
+    def test_a_missing_validator_is_unavailable_not_invalid(self):
+        good = self.generate_good_config()
+        self._hide_the_haproxy_binary()
+
+        status, message = hm.validate_config_file(hm.HAPROXY_CONFIG_PATH)
+        self.assertEqual(status, 'unavailable',
+                         'a validator that could not run was reported as a '
+                         f'verdict on the config ({status}: {message})')
+
+        # And the consequence create_backup() draws from it: a config it could
+        # not check is still snapshotted, because refusing would leave the box
+        # with no rollback target at all. Contrast
+        # test_a_broken_current_config_does_not_replace_a_good_backup, where a
+        # real 'invalid' verdict yields 'kept_previous'.
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write('hand written, unverifiable\n')
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'created',
+                         'an unverifiable config was treated as a rejected one')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH),
+                         'hand written, unverifiable\n')
+        self.assertNotEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
+
+
+class TestFileComparison(RollbackTestCase):
+    """The fast path compares bytes, not sizes."""
+
+    def test_same_size_different_content_is_not_identical(self):
+        a = os.path.join(self.etc, 'a')
+        b = os.path.join(self.etc, 'b')
+        with open(a, 'w') as fh:
+            fh.write('aaaa\n')
+        with open(b, 'w') as fh:
+            fh.write('aaba\n')
+        self.assertEqual(os.path.getsize(a), os.path.getsize(b),
+                         'fixture: the two files must be the same size')
+        self.assertFalse(hm._files_identical(a, b),
+                         'two same-size files with different bytes compared equal')
+
+    def test_a_same_size_drifted_config_still_hits_the_validation_gate(self):
+        """The case the byte-compare exists for.
+
+        A config edited in place without changing its length (one character
+        swapped, a hostname replaced by another of the same width) must not be
+        mistaken for the known-good backup and waved through.
+        """
+        good = self.generate_good_config()
+        # Sized in BYTES, not characters: the rendered config contains
+        # non-ASCII (em dashes in template comments), so len(str) would be
+        # smaller than the file and this test would pass for the wrong reason.
+        size = os.path.getsize(hm.HAPROXY_CONFIG_PATH)
+        broken = f'# {BROKEN_TOKEN}\n'.encode()
+        broken += b'#' * (size - len(broken) - 1) + b'\n'
+        with open(hm.HAPROXY_CONFIG_PATH, 'wb') as fh:
+            fh.write(broken)
+        self.assertEqual(os.path.getsize(hm.HAPROXY_CONFIG_PATH), size,
+                         'fixture: the drifted config must be the same size')
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(
+            status, 'kept_previous',
+            'a same-size broken config was accepted as unchanged and skipped '
+            'the validation gate')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'the known-good backup was overwritten')
+
+
+class TestBlockedIpsMapWrites(RollbackTestCase):
+    """blocked_ips.map is loaded by `haproxy -c`, so it gets the same care.
+
+    Verified against HAProxy 2.8: with the map referenced by
+    map_ip(/etc/haproxy/blocked_ips.map,0), a half-written final line makes the
+    WHOLE configuration invalid ("'198.51.10' is not a valid IPv4 or IPv6
+    address at line 2 of file ..."), not merely a dropped entry.
+    """
+
+    def test_the_map_goes_through_the_atomic_writer(self):
+        seen = []
+        real_write = hm.write_config_atomically
+
+        def spy(path, content, *args, **kwargs):
+            seen.append(path)
+            return real_write(path, content, *args, **kwargs)
+
+        hm.write_config_atomically = spy
+        self.addCleanup(setattr, hm, 'write_config_atomically', real_write)
+
+        self.block_ip('192.0.2.10')
+        self.assertTrue(hm.update_blocked_ips_map())
+        self.assertIn(hm.BLOCKED_IPS_MAP_PATH, seen,
+                      'the blocked IPs map was written without the atomic '
+                      'writer - a truncated map is a fatal config')
+
+    def test_a_crash_before_the_rename_leaves_the_old_map_intact(self):
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        real_replace = os.replace
+
+        def boom(src, dst, *args, **kwargs):
+            if dst == hm.BLOCKED_IPS_MAP_PATH:
+                raise OSError('simulated crash between write and rename')
+            return real_replace(src, dst, *args, **kwargs)
+
+        os.replace = boom
+        self.addCleanup(setattr, os, 'replace', real_replace)
+
+        self.block_ip('198.51.100.20')
+        self.assertFalse(hm.update_blocked_ips_map(),
+                         'a failed map write was reported as success')
+        os.replace = real_replace
+
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+            'an interrupted map write clobbered the map HAProxy is running')
+        leftovers = [n for n in os.listdir(self.etc) if n.endswith('.tmp')]
+        self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
+
+    def test_a_malformed_map_is_not_recorded_as_known_good(self):
+        """The map backup must stay something HAProxy would actually load."""
+        self.generate_good_config()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_BACKUP_PATH)
+
+        # Straight into the table, the way a bad row gets there in the first
+        # place - the API route is not the only writer.
+        self.block_ip('not-an-ip')
+        hm.update_blocked_ips_map()
+
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_BACKUP_PATH), good_map,
+            'a map HAProxy cannot parse was promoted to the rollback target')
+
+    def test_no_map_backup_is_fabricated_before_a_config_exists(self):
+        """Nothing to stay in step with means nothing to write."""
+        self.block_ip('192.0.2.10')
+        self.assertTrue(hm.update_blocked_ips_map())
+        self.assertFalse(
+            os.path.exists(hm.BLOCKED_IPS_MAP_BACKUP_PATH),
+            'a rollback target was invented out of a map write alone')
+
+
+class TestValidationCost(RollbackTestCase):
+    """`haproxy -c` runs are the customer-facing cost of a config change.
+
+    generate_config() runs synchronously inside the API call that adds a
+    domain, and on an edge with hundreds of certificates `haproxy -c` is the
+    expensive part. These counts are the contract; changing them should be a
+    deliberate decision, not a side effect.
+    """
+
+    def _count_validations(self, action):
+        calls = []
+        real_validate = hm.validate_config_file
+
+        def spy(path):
+            calls.append(path)
+            return real_validate(path)
+
+        hm.validate_config_file = spy
+        try:
+            action()
+        finally:
+            hm.validate_config_file = real_validate
+        return len(calls)
+
+    def test_blocking_an_ip_does_not_add_a_validation_to_the_next_change(self):
+        self.generate_good_config()
+
+        def add(domain, backend, address):
+            self.add_domain(domain, backend, address)
+            hm.generate_config()
+
+        steady = self._count_validations(
+            lambda: add('a.example.com', 'a_backend', '10.0.0.4'))
+        self.assertEqual(
+            steady, 1,
+            'a steady-state config change should cost exactly one `haproxy -c` '
+            '(the pre-reload gate); the known-good fast path should skip the '
+            f'other one, but {steady} ran')
+
+        # What POST /api/blocked-ips does: rewrite the map outside
+        # generate_config(). This fleet blocks IPs automatically, so it happens
+        # between most config changes.
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+
+        after_block = self._count_validations(
+            lambda: add('b.example.com', 'b_backend', '10.0.0.5'))
+        self.assertEqual(
+            after_block, steady,
+            'an IP block left the map out of step with its backup, so the next '
+            f'domain add paid {after_block} `haproxy -c` runs instead of '
+            f'{steady} - on the customer-facing call')
+
+
+if __name__ == '__main__':
+    print(f"testing haproxy_manager from: {MODULE_DIR}")
+    unittest.main(verbosity=2)

scripts/test-runtime-map-contract.py

@@ -0,0 +1,397 @@
+#!/usr/bin/env python3
+"""Contract test: the runtime-map fast path (blocked IPs).
+
+Why this file exists
+--------------------
+`add_ip_to_runtime_map()` and `remove_ip_from_runtime_map()` spent their whole
+existence sending
+
+    add map #0 <ip> 1
+    del map #0 <ip>
+
+to `/tmp/haproxy-cli` and returning True whenever socat exited 0. Neither
+command has ever worked. Two independent defects:
+
+  * **No `@1` prefix.** `/tmp/haproxy-cli` is HAProxy's MASTER CLI socket; map
+    commands are worker commands. The master answers `Unknown command: 'add',
+    but maybe one of the following ones is a better match: ...` -- and **socat
+    still exits 0**, so `result.returncode == 0` was true and the function
+    logged "Added IP x to runtime map".
+  * **`#0` is not a valid map id.** Ids are assigned at config-parse time and
+    move on every config regeneration; on the live edge `blocked_ips.map` is
+    id 37 and `trusted_ips.map` is 10. There is no id 0. Any hardcoded number
+    is wrong -- the map must be referenced by its FILE PATH, which is what
+    haproxy.cfg itself names in `map_ip(/etc/haproxy/blocked_ips.map,0)`.
+
+And a third silence that makes a response-body check alone insufficient:
+`@1 add map #0 <ip> 1` returns an **empty body**, exit 0, and adds nothing
+anywhere -- while `@1 del map #0 <ip>` answers `Unknown map identifier.`. The
+add path can therefore only be trusted after reading the entry back.
+
+IP blocking still worked, because `update_blocked_ips_map()` rewrites
+`/etc/haproxy/blocked_ips.map` and the callers reload HAProxy, which re-reads
+it. The FILE is authoritative; this is the no-reload fast path, and it has
+never once run while reporting that it did.
+
+What it enforces
+----------------
+  1. The command strings actually sent: `@1` prefix first, map referenced by
+     PATH and never by `#<id>`, and the value `1` that
+     `map_ip(...,0) -m int gt 0` requires.
+  2. Every captured rejection is classified as FAILURE (returns False), not
+     success -- including the two that carry no error text at all.
+  3. Success is only reported when the entry reads back in the state asked
+     for. Not the exit status, not an empty reply.
+  4. No source in this repo builds a map command with a `#<id>` reference.
+     Comments may describe the old form; code may not use it.
+
+Runs fully offline: `_cli_send()` is replaced, so no socket, no socat, no
+HAProxy, no network.
+
+Running
+-------
+    python3 scripts/test-runtime-map-contract.py
+"""
+
+import ast
+import io
+import os
+import re
+import sys
+import glob
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+logging.getLogger().setLevel(logging.CRITICAL)
+
+MAP = haproxy_manager.BLOCKED_IPS_MAP_PATH
+IP = '192.0.2.77'
+
+# ---------------------------------------------------------------------------
+# Responses captured VERBATIM from the haproxy-manager container on the live
+# edge (HAProxy 3.0.11, 2026-08-22). socat exited 0 for every single one of
+# them, which is the entire reason none of this is decided on exit status.
+# ---------------------------------------------------------------------------
+
+# What the MASTER socket answers to an unprefixed `add map ...` -- i.e. the
+# reply the old code read as success.
+MASTER_REJECTS_ADD = """\
+Unknown command: 'add', but maybe one of the following ones is a better match:
+  @!<pid>                                 : send a command to the <pid> process
+  @master                                 : send a command to the master process
+  hard-reload                             : achieve a hard-reload (-st) of haproxy
+  reload                                  : achieve a soft-reload (-sf) of haproxy
+  user                                    : lower the level of the current CLI session to user
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+MASTER_REJECTS_DEL = MASTER_REJECTS_ADD.replace("'add'", "'del'")
+
+# `@1 del map #0 <ip>` / `@1 get map /etc/haproxy/nope.map <ip>`.
+UNKNOWN_MAP_IDENTIFIER = 'Unknown map identifier. Please use #<id> or <file>.\n'
+
+# `@1 add map <map> <ip>` with the value omitted (as the old docs showed).
+ADD_MAP_MISSING_VALUE = (
+    "'add map' expects three parameters (map identifier, key and value) or one "
+    "parameter (map identifier) and a payload\n")
+
+# `@1 del map <map> <ip>` for a key the runtime map does not hold.
+KEY_NOT_FOUND = 'Key not found.\n'
+
+# A successful mutation. THIS IS THE WHOLE PROBLEM: it is byte-for-byte what
+# `@1 add map #0 <ip> 1` also returns while adding nothing at all.
+MUTATION_OK = ''
+
+GET_FOUND = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+             'value="1", type="str"\n' % IP)
+GET_NOT_FOUND = 'type=ip, case=sensitive, found=no\n'
+# `@1 get map #0 <ip>` -- "found", but with no value. haproxy.cfg matches with
+# `-m int gt 0`, so an entry like this does NOT block.
+GET_FOUND_NO_VALUE = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+                      'value=none\n' % IP)
+
+SHOW_MAP_SAMPLE = (
+    '0x7f6e5a788700 101.36.109.130 1\n'
+    '0x7f6e5a788780 101.47.140.218 1\n'
+    '0x7f6e5a1fd500 %s 1\n' % IP)
+
+
+class FakeSocket(object):
+    """Replaces _cli_send(). Records every command, answers from a script.
+
+    `script` is a list of (substring, response) pairs, consulted in order; the
+    first whose substring appears in the command wins. Anything unmatched is
+    an explicit test bug, not a silent default.
+    """
+
+    def __init__(self, script):
+        self.script = script
+        self.sent = []
+
+    def __call__(self, command, socket_path, timeout):
+        self.sent.append(command)
+        for needle, response in self.script:
+            if needle in command:
+                return response
+        raise AssertionError('test script has no response for %r' % command)
+
+
+def with_socket(script):
+    """Install a FakeSocket for the duration of a `with` block."""
+    class _Ctx(object):
+        def __enter__(self):
+            self.fake = FakeSocket(script)
+            self._real = haproxy_manager._cli_send
+            haproxy_manager._cli_send = self.fake
+            return self.fake
+
+        def __exit__(self, *exc):
+            haproxy_manager._cli_send = self._real
+            return False
+    return _Ctx()
+
+
+# Every command the happy path needs, in the order the code issues them.
+HAPPY_ADD = [('get map', GET_FOUND), ('add map', MUTATION_OK)]
+HAPPY_DEL = [('get map', GET_NOT_FOUND), ('del map', MUTATION_OK)]
+
+
+class CommandsAreWellFormed(unittest.TestCase):
+    """Guard 1: the exact bytes on the wire.
+
+    Both original defects are visible here and nowhere else -- a `#0` map
+    reference and a missing `@1` are perfectly ordinary-looking Python.
+    """
+
+    def test_add_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_ADD) as fake:
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 add map %s %s 1' % (MAP, IP))
+
+    def test_del_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_DEL) as fake:
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 del map %s %s' % (MAP, IP))
+
+    def test_every_command_is_tried_on_the_worker_first(self):
+        """`/tmp/haproxy-cli` is the MASTER socket; bare map commands 404."""
+        for run in (lambda: haproxy_manager.add_ip_to_runtime_map(IP),
+                    lambda: haproxy_manager.remove_ip_from_runtime_map(IP)):
+            with with_socket(HAPPY_ADD + HAPPY_DEL) as fake:
+                run()
+            for command in fake.sent:
+                self.assertTrue(command.startswith('@1 '),
+                                '%r is missing the @1 worker prefix' % command)
+
+    def test_no_command_references_a_map_by_id(self):
+        """Map ids move on every config regeneration. Path, always."""
+        script = HAPPY_ADD + HAPPY_DEL + [('show map', SHOW_MAP_SAMPLE)]
+        with with_socket(script) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+            haproxy_manager.remove_ip_from_runtime_map(IP)
+            haproxy_manager.runtime_map_keys(MAP)
+        for command in fake.sent:
+            self.assertNotRegex(
+                command, r'\bmap\s+#',
+                '%r references a map by id; ids are not stable' % command)
+            self.assertIn(MAP, command,
+                          '%r does not name the map file' % command)
+
+    def test_add_carries_the_value_the_config_matches_on(self):
+        """`map_ip(...,0) -m int gt 0`: a valueless entry does not block."""
+        self.assertEqual(haproxy_manager.BLOCKED_IPS_MAP_VALUE, '1')
+        with with_socket(HAPPY_ADD) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+        self.assertTrue(fake.sent[0].endswith(' %s 1' % IP),
+                        '%r has no value; HAProxy rejects it' % fake.sent[0])
+
+
+class RejectionIsFailure(unittest.TestCase):
+    """Guard 2+3: nothing may report success unless the map really changed.
+
+    Every response below was returned by the live socket with **exit code 0**.
+    The old code returned True for all of them.
+    """
+
+    def test_master_socket_rejection_of_add_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_master_socket_rejection_of_del_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MASTER_REJECTS_DEL)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_unknown_map_identifier_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket([('get map', GET_FOUND), ('del map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_missing_value_rejection_is_failure(self):
+        """Carries no marker word at all -- caught by 'a mutation says nothing'."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', ADD_MAP_MISSING_VALUE)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_silent_noop_add_is_failure(self):
+        """The `#0` failure mode: accepted, empty reply, nothing added.
+
+        Nothing in the response distinguishes this from success. Only the
+        read-back does -- which is why the read-back is not optional.
+        """
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_add_that_lands_without_a_value_is_failure(self):
+        with with_socket([('get map', GET_FOUND_NO_VALUE), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_del_that_leaves_the_key_behind_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_key_not_found_on_del_is_the_requested_end_state(self):
+        """Not a failure: the runtime map already lacks the key."""
+        with with_socket([('get map', GET_NOT_FOUND), ('del map', KEY_NOT_FOUND)]):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_verified_success_is_reported_as_success(self):
+        with with_socket(HAPPY_ADD):
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket(HAPPY_DEL):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_a_runtime_failure_never_raises_into_the_request_handler(self):
+        """The map file + reload still enforces the block; degrade, don't 500."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertIs(haproxy_manager.add_ip_to_runtime_map(IP), False)
+
+    def test_mutations_are_checked_for_an_empty_body_not_a_marker_list(self):
+        """_HAPROXY_CLI_ERROR_MARKERS can only know rejections already seen."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            with with_socket([('add map', 'something nobody has ever seen\n')]):
+                haproxy_manager.haproxy_cli('add map %s x 1' % MAP,
+                                            worker=True, expect_empty=True)
+
+    def test_the_new_error_markers_are_recognised(self):
+        for response in (UNKNOWN_MAP_IDENTIFIER, KEY_NOT_FOUND,
+                         MASTER_REJECTS_ADD):
+            self.assertTrue(haproxy_manager._cli_response_is_error(response),
+                            '%r must be classified as an error' % response[:40])
+        for response in (GET_FOUND, GET_NOT_FOUND, SHOW_MAP_SAMPLE):
+            self.assertFalse(haproxy_manager._cli_response_is_error(response),
+                             '%r is data, not an error' % response[:40])
+
+    def test_socat_exit_zero_carries_no_information(self):
+        """FakeSocket never signals failure any other way, and neither did socat."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]) as fake:
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertTrue(fake.sent, 'the command was sent and "succeeded" at the '
+                                   'process level; only the body says otherwise')
+
+
+class ReadBack(unittest.TestCase):
+    """The read-back primitives the guarantees above rest on."""
+
+    def test_lookup_reports_found_with_value(self):
+        with with_socket([('get map', GET_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (True, '1'))
+
+    def test_lookup_reports_not_found(self):
+        with with_socket([('get map', GET_NOT_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (False, None))
+
+    def test_lookup_raises_on_a_rejected_reference(self):
+        with with_socket([('get map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_lookup(MAP, IP)
+
+    def test_keys_parses_show_map_output(self):
+        with with_socket([('show map', SHOW_MAP_SAMPLE)]):
+            self.assertEqual(
+                haproxy_manager.runtime_map_keys(MAP),
+                {'101.36.109.130', '101.47.140.218', IP})
+
+    def test_empty_map_is_not_a_rejection(self):
+        with with_socket([('show map', '')]):
+            self.assertEqual(haproxy_manager.runtime_map_keys(MAP), set())
+
+    def test_rejected_show_map_still_raises(self):
+        with with_socket([('show map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_keys(MAP)
+
+
+MAP_BY_ID_RE = re.compile(r'\b(?:add|del|clear|show|get)\s+map\s+#')
+
+
+class NoSourceBuildsAMapIdCommand(unittest.TestCase):
+    """Guard 4: `map #<id>` may be described in comments, never executed.
+
+    Scanning string literals rather than raw text is deliberate -- the whole
+    reason this bug is documented at length in the source is so the next reader
+    does not reintroduce it, and a plain grep would fail on those comments.
+    """
+
+    def test_no_python_string_literal_builds_a_map_id_command(self):
+        tree = ast.parse(io.open('haproxy_manager.py', encoding='utf-8').read())
+        offenders = [
+            node.value for node in ast.walk(tree)
+            if isinstance(node, ast.Constant) and isinstance(node.value, str)
+            and MAP_BY_ID_RE.search(node.value)
+            and not (ast.get_docstring(tree) == node.value)
+        ]
+        # Docstrings are string literals too; exclude any literal that is a
+        # docstring of a module/class/function.
+        docstrings = set()
+        for node in ast.walk(tree):
+            if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef,
+                                 ast.AsyncFunctionDef)):
+                doc = ast.get_docstring(node, clean=False)
+                if doc:
+                    docstrings.add(doc)
+        offenders = [o for o in offenders if o not in docstrings]
+        self.assertEqual(offenders, [],
+                         'these string literals build a map command with an '
+                         'unstable #<id> reference')
+
+    def test_no_shell_or_template_code_line_uses_a_map_id(self):
+        targets = (glob.glob('scripts/*.sh') + glob.glob('templates/*.tpl')
+                   + ['Dockerfile'])
+        offenders = []
+        for path in targets:
+            if not os.path.exists(path):
+                continue
+            for lineno, line in enumerate(
+                    io.open(path, encoding='utf-8').read().splitlines(), 1):
+                if line.lstrip().startswith('#'):
+                    continue  # a comment describing the old form is fine
+                if MAP_BY_ID_RE.search(line):
+                    offenders.append('%s:%d: %s' % (path, lineno, line.strip()))
+        self.assertEqual(offenders, [],
+                         'map ids are assigned at config-parse time and move; '
+                         'reference the map file by path')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-stick-table-contract.py

@@ -0,0 +1,475 @@
+#!/usr/bin/env python3
+"""Contract test: what the stick tables STORE vs what the consumers READ.
+
+Why this file exists
+--------------------
+`/api/security/stats` and `scripts/show-tarpit-ips.sh` spent their whole
+existence reporting "Scan Count", "offense count" and "BLOCKED" figures parsed
+out of `gpc0` and `gpc1`. **No stick table in this repo has ever stored a
+general-purpose counter.** Every one of those numbers was fabricated, and an
+operator was making decisions on them.
+
+Nothing caught it, because each layer failed silently in a different way:
+
+  * `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
+    and the loop `continue`d -- so the endpoint answered `active_threats: 0`
+    with an empty list. "No threats" and "the parser is broken" looked
+    identical.
+  * The command went to the MASTER CLI socket without the `@1` worker prefix.
+    HAProxy answered `Unknown command: 'show', but maybe one of the following
+    ones is a better match: ...` and **socat still exited 0**, so the
+    `returncode != 0` guard never fired. The reported `total_tracked_ips` was
+    the line count of that help text (8) while the real table held 388 entries.
+  * The shell consumers wrote `gpc0=${gpc0:-0}`, so a field that does not exist
+    rendered as a confident zero.
+
+The durable fix is not "parse better" -- it is making the template and its
+consumers unable to drift apart without something going red. That is this file.
+
+What it enforces
+----------------
+  1. `STICK_TABLE_FIELD_CONTRACT` in haproxy_manager.py equals, exactly and in
+     both directions, the `store` clauses in the rendered templates.
+  2. Every shell consumer's `EXPECTED_FIELDS=(...)` array equals the contract.
+  3. A captured sample of REAL `show table` output from the live edge parses to
+     exactly the contract's fields plus the entry metadata -- so the contract
+     describes reality, not just itself.
+  4. The loud-failure behaviour: `read_stick_table()` RAISES on a rejected
+     command, on a non-table response, and on a row missing a contract field.
+     It must never answer zeros. Guard 4 is the one that would have caught the
+     original bug on day one.
+  5. No `store` clause names a general-purpose counter, and no template tracks
+     one -- the state this repo is actually in, asserted rather than assumed.
+
+Assertions about the TEMPLATES go through `rule_lines()`, which strips comments
+before matching. These templates quote their own rules in prose at length; a
+bare `assertIn` over the rendered text passes just as happily against a rule
+that has been commented out. Same lesson, and same helper, as
+scripts/test-wpadmin-gate.py.
+
+Runs fully offline -- no HAProxy, no socket, no network.
+
+Running
+-------
+    python3 scripts/test-stick-table-contract.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# A captured sample of REAL output, so this test can assert against reality
+# without a live socket.
+#
+# Provenance: `echo "@1 show table web" | socat stdio /tmp/haproxy-cli` inside
+# the haproxy-manager container on whp01, 2026-08-22, HAProxy 3.0.11. Rows
+# trimmed for length; format byte-for-byte as emitted.
+#
+# Note what is and is NOT here: no gpc0, no gpc1, no gpc(N), no gpc_rate, no
+# glitch_rate. Note also that field windows come back in MILLISECONDS
+# (`conn_rate(10000)`), not the `10s` the template writes -- a consumer that
+# labels the raw number "10s" is off by a factor of 1000.
+# ---------------------------------------------------------------------------
+LIVE_TABLE_SAMPLE = """\
+# table: web, type: ip, size:204800, used:388
+0x7f6e5447e728: key=17.58.57.102 use=0 exp=368842 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e541cf7c8: key=43.173.182.9 use=0 exp=171404 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e5547f528: key=95.129.255.180 use=0 exp=590639 shard=0 conn_rate(10000)=1 conn_cur=0 http_req_rate(10000)=1 http_err_rate(30000)=0
+0x7f6e5447e488: key=5.9.105.254 use=0 exp=556277 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=1
+"""
+
+# The verbatim reply the MASTER CLI socket gives to an unprefixed worker
+# command. Captured the same way. socat exits 0 on this -- which is the entire
+# reason haproxy_cli() inspects the body.
+MASTER_SOCKET_REJECTION = """\
+Unknown command: 'show', but maybe one of the following ones is a better match:
+  show cli level                          : display the level of the current CLI session
+  show cli sockets                        : dump list of cli sockets
+  show proc                               : show processes status
+  show startup-logs                       : report logs emitted during HAProxy startup
+  show version                            : show version of the current process
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+# Anything matching these in a `store` clause is a general-purpose counter.
+GPC_PATTERN = re.compile(r'\bgpc|glitch')
+
+# Shell consumers that must declare their field expectations as a single
+# EXPECTED_FIELDS array, and the table each one reads.
+SHELL_CONSUMERS = {
+    'scripts/show-edge-ip-rates.sh': 'web',
+}
+
+EXPECTED_FIELDS_RE = re.compile(r'^\s*EXPECTED_FIELDS=\(([^)]*)\)\s*$', re.M)
+
+
+def rule_lines(cfg, needle):
+    """Comment-stripped config lines containing `needle`.
+
+    A line that is entirely a comment is dropped; a line mixing config with a
+    trailing comment is truncated at the first ' #' before matching. Without
+    this, every assertion below would pass against a rule that had been
+    commented out but whose text survived in the surrounding prose -- and these
+    templates quote their own rules in prose constantly. See
+    scripts/test-wpadmin-gate.py, where a mutation audit proved the point.
+    """
+    out = []
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if stripped and not stripped.startswith('#'):
+            code = stripped.split(' #', 1)[0].rstrip()
+            if code and needle in code:
+                out.append(code)
+    return out
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_security_tables():
+    return haproxy_manager.template_env.get_template('hap_security_tables.tpl').render()
+
+
+def stick_tables_from(cfg):
+    """{table name: (field, ...)} for every stick-table declared in `cfg`.
+
+    A stick-table takes the name of the frontend/backend/listen section that
+    declares it -- that name is what `show table <name>` wants, so the section
+    header is part of the contract, not incidental. Section headers and
+    stick-table lines are both read comment-stripped.
+    """
+    tables = {}
+    section = None
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if not stripped or stripped.startswith('#'):
+            continue
+        code = stripped.split(' #', 1)[0].rstrip()
+        header = re.match(r'^(frontend|backend|listen)\s+(\S+)', code)
+        if header:
+            section = header.group(2)
+            continue
+        if code.startswith('stick-table'):
+            store = re.search(r'\bstore\s+(\S+)', code)
+            if not store:
+                raise AssertionError(
+                    'stick-table in section %r has no `store` clause: %r'
+                    % (section, code))
+            if section is None:
+                raise AssertionError(
+                    'stick-table declared outside any section: %r' % code)
+            # store is a comma-separated list; each item is name or name(window)
+            fields = tuple(item.split('(')[0]
+                           for item in store.group(1).split(','))
+            tables[section] = fields
+    return tables
+
+
+def all_template_stick_tables():
+    tables = {}
+    for cfg in (render_listener(), render_security_tables()):
+        for name, fields in stick_tables_from(cfg).items():
+            if name in tables:
+                raise AssertionError('stick table %r declared twice' % name)
+            tables[name] = fields
+    return tables
+
+
+class StickTableContract(unittest.TestCase):
+    """Guard 1 + 5: the templates and the Python contract, held together."""
+
+    def setUp(self):
+        self.templates = all_template_stick_tables()
+        self.contract = haproxy_manager.STICK_TABLE_FIELD_CONTRACT
+
+    def test_templates_actually_declare_stick_tables(self):
+        """Guard the guard: an empty parse would make every other check vacuous."""
+        self.assertTrue(self.templates,
+                        'parsed no stick tables out of the templates at all -- '
+                        'stick_tables_from() is broken, not the templates')
+
+    def test_same_table_names(self):
+        self.assertEqual(
+            sorted(self.templates), sorted(self.contract),
+            'STICK_TABLE_FIELD_CONTRACT and the templates disagree on WHICH '
+            'stick tables exist. Add/remove the table in both places.')
+
+    def test_same_fields_per_table(self):
+        for table in sorted(self.templates):
+            with self.subTest(table=table):
+                self.assertEqual(
+                    sorted(self.templates[table]),
+                    sorted(self.contract.get(table, ())),
+                    "stick table %r stores %s but STICK_TABLE_FIELD_CONTRACT "
+                    "claims %s. Whichever is wrong, a consumer is about to read "
+                    "a field that is never populated -- which is the bug this "
+                    "test exists for." % (table,
+                                          list(self.templates[table]),
+                                          list(self.contract.get(table, ()))))
+
+    def test_web_table_is_the_one_the_api_reads(self):
+        self.assertIn('web', self.contract)
+        self.assertIn('web', self.templates)
+
+    def test_no_general_purpose_counters_are_stored(self):
+        """The state of the world today, asserted rather than assumed.
+
+        If a gpc/glitch counter is ever genuinely added to a template, this
+        test is the place to update -- and updating it forces whoever does so
+        to also add the field to STICK_TABLE_FIELD_CONTRACT (test_same_fields_
+        per_table) and to the shell consumers (test_shell_consumers_match_
+        contract). That chain is the point: a counter cannot appear in a
+        consumer without existing in the table, and cannot appear in the table
+        without the consumers being updated.
+        """
+        for table, fields in self.templates.items():
+            for field in fields:
+                self.assertIsNone(
+                    GPC_PATTERN.search(field),
+                    'stick table %r now stores %r. Update this test, '
+                    'STICK_TABLE_FIELD_CONTRACT, and every consumer.'
+                    % (table, field))
+
+    def test_track_sc_counters_have_a_table_each(self):
+        """Every `track-scN ... table X` names a table that really exists.
+
+        A typo here is invisible to `haproxy -c` only in the sense that it is
+        NOT -- but it is invisible to the consumers, which would query a table
+        that is never written.
+        """
+        cfg = render_listener()
+        for line in rule_lines(cfg, 'track-sc'):
+            named = re.search(r'\btable\s+(\S+)', line)
+            if named:
+                self.assertIn(
+                    named.group(1), self.templates,
+                    'track-sc rule references undeclared table %r: %r'
+                    % (named.group(1), line))
+
+    def test_sc_counter_indices_fit_haproxys_limit(self):
+        """sc0/sc1/sc2 are all HAProxy gives us by default.
+
+        `tune.stick-counters` defaults to 3. A `track-sc3` without raising it
+        is a config-time failure, and the templates' own comments assume the
+        limit -- so assert it rather than leaving it as folklore.
+        """
+        cfg = render_listener() + '\n' + render_security_tables()
+        raised = rule_lines(cfg, 'tune.stick-counters')
+        limit = 3
+        if raised:
+            limit = int(re.search(r'(\d+)', raised[-1]).group(1))
+        for line in rule_lines(cfg, 'track-sc'):
+            idx = int(re.search(r'track-sc(\d+)', line).group(1))
+            self.assertLess(
+                idx, limit,
+                'track-sc%d exceeds tune.stick-counters (%d): %r'
+                % (idx, limit, line))
+
+
+class ShellConsumerContract(unittest.TestCase):
+    """Guard 2: the shell consumers cannot drift from the contract."""
+
+    def test_shell_consumers_match_contract(self):
+        for path, table in sorted(SHELL_CONSUMERS.items()):
+            with self.subTest(script=path):
+                full = os.path.join(MODULE_DIR, path)
+                self.assertTrue(os.path.exists(full),
+                                '%s is missing; it is a declared consumer of '
+                                'stick table %r' % (path, table))
+                with open(full) as fh:
+                    src = fh.read()
+                m = EXPECTED_FIELDS_RE.search(src)
+                self.assertIsNotNone(
+                    m, '%s must declare its field expectations once as a '
+                       'single-line `EXPECTED_FIELDS=(a b c)` array so this '
+                       'test can hold it to the template' % path)
+                declared = sorted(m.group(1).split())
+                self.assertEqual(
+                    declared,
+                    sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table]),
+                    '%s reads %s but stick table %r stores %s'
+                    % (path, declared, table,
+                       sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table])))
+
+    def test_retired_script_no_longer_parses_phantom_counters(self):
+        """show-tarpit-ips.sh may explain gpc0/gpc1; it may not extract them.
+
+        The shim is allowed -- encouraged -- to name the fields in prose so an
+        operator who runs it learns why its numbers went away. What it must not
+        do is go back to pulling values out of them.
+        """
+        path = os.path.join(MODULE_DIR, 'scripts/show-tarpit-ips.sh')
+        if not os.path.exists(path):
+            self.skipTest('show-tarpit-ips.sh has been removed outright')
+        with open(path) as fh:
+            lines = fh.readlines()
+        for raw in lines:
+            stripped = raw.strip()
+            if not stripped or stripped.startswith('#'):
+                continue
+            code = stripped.split(' #', 1)[0]
+            self.assertIsNone(
+                re.search(r"grep -o ['\"]?gpc|gpc[0-9]*=\$|sc_get_gpc|sc-inc-gpc", code),
+                'show-tarpit-ips.sh is extracting a general-purpose counter '
+                'again: %r' % stripped)
+
+
+class LiveSampleParses(unittest.TestCase):
+    """Guard 3: the contract describes real HAProxy output, not just itself."""
+
+    def test_header_parses(self):
+        header, entries = self._read()
+        self.assertEqual(header['name'], 'web')
+        self.assertEqual(header['type'], 'ip')
+        self.assertEqual(header['size'], 204800)
+        self.assertEqual(header['used'], 388)
+        self.assertEqual(len(entries), 4)
+
+    def test_sample_fields_are_exactly_contract_plus_metadata(self):
+        expected = set(haproxy_manager.STICK_TABLE_FIELD_CONTRACT['web'])
+        expected |= set(haproxy_manager.STICK_TABLE_ENTRY_META)
+        _, entries = self._read()
+        for line, fields in entries:
+            self.assertEqual(
+                set(fields), expected,
+                'real `show table web` output carries %s, contract+metadata '
+                'expects %s. Row: %r'
+                % (sorted(fields), sorted(expected), line))
+
+    def test_key_is_the_ip_not_the_allocation_pointer(self):
+        """The original bug read parts[0] -- the `0x...:` pointer -- as the IP."""
+        _, entries = self._read()
+        ips = [f['key']['value'] for _, f in entries]
+        self.assertIn('95.129.255.180', ips)
+        for ip in ips:
+            self.assertFalse(ip.startswith('0x'),
+                             'parsed a memory address as an IP: %r' % ip)
+
+    def test_windows_are_milliseconds(self):
+        """HAProxy reports `conn_rate(10000)` for a `conn_rate(10s)` store.
+
+        Asserted because labelling that raw 10000 as "10s" (or as seconds) is
+        an easy and completely silent way to be wrong by 1000x in the panel.
+        """
+        _, entries = self._read()
+        _, fields = entries[0]
+        self.assertEqual(fields['conn_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_req_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_err_rate']['window_ms'], 30000)
+        self.assertIsNone(fields['conn_cur']['window_ms'],
+                          'conn_cur is a gauge, not a rate; it has no window')
+
+    def test_values_are_the_real_ones(self):
+        _, entries = self._read()
+        by_ip = {f['key']['value']: f for _, f in entries}
+        self.assertEqual(by_ip['95.129.255.180']['http_req_rate']['value'], '1')
+        self.assertEqual(by_ip['5.9.105.254']['http_err_rate']['value'], '1')
+        self.assertEqual(by_ip['17.58.57.102']['http_req_rate']['value'], '0')
+
+    def _read(self):
+        return _read_table_from(LIVE_TABLE_SAMPLE)
+
+
+class FailsLoudly(unittest.TestCase):
+    """Guard 4: every way this can go wrong must raise, never return zeros.
+
+    This is the guard that would have caught the original bug immediately. Each
+    case below is a real response the old code accepted silently.
+    """
+
+    def test_master_socket_rejection_is_not_data(self):
+        """The exact reply that used to be reported as `total_tracked_ips: 8`."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(MASTER_SOCKET_REJECTION)
+        self.assertIn('not a stick-table dump', str(ctx.exception))
+
+    def test_socat_exit_zero_does_not_mean_success(self):
+        """haproxy_cli() must reject on the BODY, not the exit status.
+
+        socat returns 0 for every response above -- the rejection is only ever
+        visible in the text.
+        """
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error(MASTER_SOCKET_REJECTION))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('No such table\n'))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('Permission denied\n'))
+        self.assertFalse(
+            haproxy_manager._cli_response_is_error(LIVE_TABLE_SAMPLE))
+
+    def test_missing_contract_field_raises_and_names_it(self):
+        """A field the table stopped storing must not silently become 0."""
+        degraded = LIVE_TABLE_SAMPLE.replace(' http_err_rate(30000)=0', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(degraded)
+        msg = str(ctx.exception)
+        self.assertIn('http_err_rate', msg,
+                      'the error must name the missing field')
+        self.assertIn('drifted', msg,
+                      'the error must say what actually went wrong')
+
+    def test_empty_response_raises(self):
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            _read_table_from('')
+
+    def test_row_without_key_raises(self):
+        broken = LIVE_TABLE_SAMPLE.replace('key=17.58.57.102 ', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(broken)
+        self.assertIn('no key=', str(ctx.exception))
+
+    def test_unknown_table_raises_before_touching_the_socket(self):
+        """Querying a table with no contract is a programming error, not a 0."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            haproxy_manager.read_stick_table('does_not_exist')
+        self.assertIn('no field contract', str(ctx.exception))
+
+    def test_empty_table_is_not_an_error(self):
+        """A table with zero entries is a legitimate, distinguishable result."""
+        header, entries = _read_table_from(
+            '# table: web, type: ip, size:204800, used:0\n')
+        self.assertEqual(header['used'], 0)
+        self.assertEqual(entries, [])
+
+
+def _read_table_from(response, table='web'):
+    """Run read_stick_table() against a canned response instead of a socket."""
+    real = haproxy_manager.haproxy_cli
+    haproxy_manager.haproxy_cli = lambda cmd, worker=False, timeout=None: response
+    try:
+        return haproxy_manager.read_stick_table(table)
+    finally:
+        haproxy_manager.haproxy_cli = real
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-trusted-proxy-gate.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Regression tests for the trusted-proxy header gate in hap_listener.tpl.
+
+Why this file exists
+--------------------
+txn.real_ip was derived from CF-Connecting-IP / X-Real-IP / X-Forwarded-For
+with no check on the peer, so any direct client could assert any client IP.
+That variable drives rate limiting, the trusted-IP whitelist, the wp-login
+brute-force table and cookie challenge, the wp-json/batch/v1 virtual patch,
+IP blocking, and Coraza's src-ip -- so a spoofed header bypassed all of them.
+
+These tests pin the invariant that the header strip is rendered BEFORE any
+real-IP resolution. Ordering is the whole fix: a del-header emitted after the
+set-var chain would parse fine, validate fine, and do nothing.
+
+Running
+-------
+    python3 scripts/test-trusted-proxy-gate.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+GATED_HEADERS = ('CF-Connecting-IP', 'X-Real-IP', 'X-Forwarded-For')
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+class TrustedProxyGate(unittest.TestCase):
+
+    def setUp(self):
+        self.cfg = render_listener()
+
+    def test_trusted_proxy_acl_is_defined(self):
+        self.assertIn('acl from_trusted_proxy src', self.cfg)
+        self.assertIn('/etc/haproxy/cloudflare_ips.list', self.cfg)
+        self.assertIn('/etc/haproxy/trusted_proxies.list', self.cfg)
+
+    def test_each_header_is_deleted_for_untrusted_peers(self):
+        for header in GATED_HEADERS:
+            with self.subTest(header=header):
+                pattern = (r'http-request\s+del-header\s+%s\s+if\s+!from_trusted_proxy'
+                           % re.escape(header))
+                self.assertRegex(self.cfg, pattern)
+
+    def test_strip_precedes_real_ip_resolution(self):
+        """The ordering invariant. A strip after the set-var chain is a no-op."""
+        last_strip = max(
+            self.cfg.index('del-header %s' % h) for h in GATED_HEADERS
+        )
+        first_setvar = self.cfg.index('set-var(txn.real_ip)')
+        self.assertLess(
+            last_strip, first_setvar,
+            'del-header rules must be rendered before set-var(txn.real_ip)')
+
+    def test_src_fallback_still_present(self):
+        """Direct clients must fall through to the real TCP peer."""
+        self.assertIn('set-var(txn.real_ip) src', self.cfg)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-wpadmin-gate.py

@@ -0,0 +1,530 @@
+#!/usr/bin/env python3
+"""Regression tests for the WordPress admin edge gate in hap_listener.tpl.
+
+Why this file exists
+--------------------
+Unauthenticated GETs to /wp-admin/* were reaching PHP, booting WordPress just to
+produce a login redirect and exhausting lsphp pools under a distributed
+low-and-slow attack. The gate redirects them at the edge instead.
+
+Two properties are easy to get wrong and invisible to `haproxy -c`:
+
+  * ORDERING. `is_whitelisted` reads var(txn.real_ip). If the rule renders before
+    the set-var chain, the whitelist evaluates against an unset variable.
+  * THE ALLOWLIST. wp-login.php loads its OWN css/js from /wp-admin/. Dropping
+    those entries leaves every login page on the fleet unstyled, while still
+    returning 200 -- a silent regression.
+
+A THIRD property, added after an adversarial mutation audit: every assertion
+here must be scoped to the CODE, not the surrounding prose. This file's own
+comment blocks quote ACL names, rule fragments and even whole rules to explain
+them -- which means a bare `assertIn` / `re.search` / `str.index` run over the
+raw rendered config passes just as happily when the real rule has been deleted
+(or merely commented out) and only its explanation survives. The audit proved
+this concretely: commenting out the entire redirect rule, or the
+`acl wp_admin_allowed` line, or all five normalizers, left the previous version
+of this file at 26/26 PASS. See `rule_lines()` below, and use it (or one of the
+guarded helpers built on it) for every assertion about whether a rule exists,
+what it says, or where it sits relative to another rule. Do not add a new
+`self.cfg.index(...)`, `self.assertIn(x, self.cfg)`, or `re.search(pattern,
+self.cfg)` to this file -- none of them can tell code from comment.
+
+Running
+-------
+    python3 scripts/test-wpadmin-gate.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+ALLOWLIST = ('/admin-ajax.php', '/admin-post.php',
+             '/load-styles.php', '/load-scripts.php')
+EXCLUSIONS = ('!wp_admin_allowed', '!wp_admin_asset', '!has_wp_logged_in',
+              '!wp_gate_exempt', '!is_local', '!is_trusted_ip', '!is_whitelisted')
+
+# The normalizer set, in the order it MUST render. Decoding has to precede the
+# path walkers or "%2e%2e" is decoded to ".." only after path-strip-dotdot has
+# already run, leaving the ".." unresolved -- measured against real HAProxy
+# 3.0.11, both orders side by side.
+NORMALIZERS = ('percent-to-uppercase',
+               'percent-decode-unreserved',
+               'path-merge-slashes',
+               'path-strip-dot',
+               'path-strip-dotdot full')
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_header():
+    return haproxy_manager.template_env.get_template('hap_header.tpl').render(
+        cluster_secret=None,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Comment-safe config inspection
+#
+# Every helper below operates on `rule_positions()`'s output, never on the raw
+# rendered string. That is the one rule this whole module exists to enforce on
+# itself: a mutation audit found that commenting out a real rule (prefixing it
+# with '#', or -- more slyly -- deleting it and appending its own text as a
+# TRAILING comment on the line above) left the previous version of these tests
+# fully green, because plain `str.index` / `assertIn` / `re.search` over
+# `self.cfg` cannot distinguish code from a comment that merely quotes it.
+# ---------------------------------------------------------------------------
+
+def rule_positions(cfg, needle):
+    """[(comment-stripped line, char offset in cfg)] for every non-comment
+    line containing `needle`, in document order.
+
+    Two things a bare substring/regex search over `self.cfg` gets wrong, both
+    fixed here:
+
+      1. A line that is ENTIRELY a comment (starts with '#' once stripped) is
+         dropped. This is necessary but not sufficient -- see (2).
+      2. A line that MIXES real config with a trailing comment
+         (`live-code  # note`, or the decoy `live-code  # was: <the other
+         rule's exact text>`) is truncated at the first ' #' before matching,
+         so text stuffed into a trailing comment cannot masquerade as the
+         rule itself. A real HAProxy comment always starts at a '#' preceded
+         by whitespace here -- none of these templates use bare '#' as a
+         value character -- so this truncation does not clip real rules.
+
+    Returning (line, position) pairs together -- rather than making callers
+    re-derive one from the other with a second `cfg.index(line)` -- also
+    avoids a subtler bug: if the same comment-stripped line occurs twice
+    (e.g. a duplicated rule), re-deriving the position with `str.index` always
+    finds the FIRST copy regardless of which one you meant. Walking the file
+    once and recording positions as we go keeps first/last unambiguous.
+    """
+    out = []
+    pos = 0
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if stripped and not stripped.startswith('#'):
+            code = stripped.split(' #', 1)[0].rstrip()
+            if code and needle in code:
+                out.append((code, pos))
+        pos += len(raw) + 1  # +1 for the '\n' split() consumed
+    return out
+
+
+def rule_lines(cfg, needle):
+    """Comment-stripped rule lines containing `needle` (text only, no
+    position). See `rule_positions()` for what this guards against. Every
+    assertion about a RULE's presence or content must go through this (or
+    `rule_positions`/the guarded helpers below) -- never a bare
+    `needle in cfg` or `re.search(pattern, cfg)`.
+    """
+    return [line for line, _ in rule_positions(cfg, needle)]
+
+
+def require_rule(cfg, needle, what=None):
+    """The single rule line containing `needle`.
+
+    Raises a plain AssertionError naming what was being looked for -- not
+    IndexError from an unguarded `rule_lines(...)[0]`, and not
+    `ValueError: substring not found` from a bare `cfg.index(...)` -- when
+    the rule is missing. A missing rule and a broken test harness must not
+    look identical in a failure report.
+
+    Raises the same way if `needle` is ambiguous (matches more than one rule
+    line): silently taking the first match in that case would hide the
+    ambiguity instead of surfacing it.
+    """
+    label = what or needle
+    lines = rule_lines(cfg, needle)
+    if not lines:
+        raise AssertionError('no rule found for %r (expected: %s)' % (needle, label))
+    if len(lines) > 1:
+        raise AssertionError(
+            '%r matched %d rule lines, expected exactly one (%s): %r'
+            % (needle, len(lines), label, lines))
+    return lines[0]
+
+
+def require_position(cfg, needle, what=None, last=False):
+    """(line, char offset) for an ordering assertion, guarded the same way as
+    `require_rule` -- but tolerant of the needle matching multiple lines
+    (e.g. a multi-line set-var "chain"), since ordering checks often want the
+    first or last of several. Pass last=True for the last occurrence.
+    """
+    label = what or needle
+    positions = rule_positions(cfg, needle)
+    if not positions:
+        raise AssertionError(
+            'no rule found for %r, cannot check ordering (expected: %s)' % (needle, label))
+    return positions[-1] if last else positions[0]
+
+
+class WpAdminGate(unittest.TestCase):
+
+    def setUp(self):
+        self.cfg = render_listener()
+
+    def test_wp_admin_path_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_path', 'wp_admin_path ACL')
+        self.assertIn('path_reg', line)
+
+    def test_wp_admin_asset_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertIn('path_reg', line)
+
+    def test_wp_admin_allowed_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        self.assertIn('path_end', line)
+
+    def test_wp_gate_exempt_acl_declared(self):
+        """Scoped to the ACL line itself, not `self.cfg` as a whole -- the
+        surrounding prose (see hap_listener.tpl's "per-site opt-out" comment)
+        also spells out /etc/haproxy/wpadmin_gate_exempt.list verbatim, so an
+        unscoped `assertIn` would still pass with the real ACL deleted.
+        """
+        line = require_rule(self.cfg, 'acl wp_gate_exempt', 'wp_gate_exempt ACL')
+        self.assertIn('/etc/haproxy/wpadmin_gate_exempt.list', line)
+
+    def test_allowlist_entries_present(self):
+        """wp-login.php loads its own css/js from /wp-admin/ -- see module docstring."""
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        for entry in ALLOWLIST:
+            with self.subTest(entry=entry):
+                self.assertIn(entry, line)
+
+    def test_static_asset_dirs_allowed(self):
+        line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertRegex(line, r'path_reg.*\(css\|js\|images\)')
+
+    def test_install_php_is_NOT_allowlisted(self):
+        """install.php is deliberately gated -- a takeover vector on abandoned installs."""
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        self.assertNotIn('install.php', line)
+
+    def test_redirect_rule_has_all_exclusions(self):
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        for excl in EXCLUSIONS:
+            with self.subTest(exclusion=excl):
+                self.assertIn(excl, rule)
+
+    def test_rule_renders_after_real_ip_resolution(self):
+        """is_whitelisted reads txn.real_ip; before the set-var chain it is unset."""
+        _, setvar_pos = require_position(self.cfg, 'set-var(txn.real_ip)',
+                                          'real_ip set-var chain')
+        _, rule_pos = require_position(self.cfg, 'wp_admin_path', 'wp_admin_path ACL/rule')
+        self.assertLess(setvar_pos, rule_pos,
+                         'wp_admin_path renders before txn.real_ip is resolved')
+
+    def test_rule_renders_after_has_wp_logged_in_declared(self):
+        """HAProxy resolves ACLs as it parses; use-before-declare fails."""
+        _, decl_pos = require_position(self.cfg, 'acl has_wp_logged_in',
+                                        'has_wp_logged_in ACL declaration')
+        _, rule_pos = require_position(self.cfg, 'wp_admin_path', 'wp_admin_path ACL/rule')
+        self.assertLess(decl_pos, rule_pos,
+                         'wp_admin_path renders before has_wp_logged_in is declared')
+
+    def test_only_one_has_wp_logged_in_declaration(self):
+        self.assertEqual(len(rule_lines(self.cfg, 'acl has_wp_logged_in')), 1)
+
+    def test_allowlist_entries_are_anchored_to_wp_admin(self):
+        """Bare `path_end /admin-ajax.php` also matches
+        /wp-admin/evil/admin-ajax.php, which ALSO matches wp_admin_path
+        (path_reg only requires /wp-admin/ to appear somewhere) -- an
+        attacker-inserted path segment would then sail through the
+        allowlist ungated. Entries must be anchored to sit directly under
+        wp-admin/. Scoped to the captured ACL line only, since the
+        surrounding comment block also mentions these bare filenames.
+        """
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        for entry in ALLOWLIST:
+            with self.subTest(entry=entry):
+                self.assertIn('/wp-admin' + entry, line)
+                self.assertNotRegex(
+                    line, r'(?<!wp-admin)' + re.escape(entry) + r'(?!\S)',
+                    'found a bare, unanchored allowlist entry: ' + entry)
+
+    def test_wp_login_url_setvar_renders_in_correct_order(self):
+        """The inline regsub-in-`location` form is rejected by real HAProxy
+        3.0.11 (invalid arg 2 in converter 'regsub'), so the regsub is
+        computed in its own set-var line instead. That set-var must render
+        after the set-var(txn.real_ip) chain (it must not disturb that
+        load-bearing chain) and before the redirect rule that consumes it.
+        """
+        _, last_real_ip_pos = require_position(self.cfg, 'set-var(txn.real_ip)',
+                                                'real_ip set-var chain', last=True)
+        _, wp_login_pos = require_position(self.cfg, 'set-var(txn.wp_login_url)',
+                                            'wp_login_url set-var')
+        _, redirect_pos = require_position(
+            self.cfg, 'http-request redirect code 302 location %[var(txn.wp_login_url)]',
+            'wp-admin redirect rule')
+        self.assertLess(last_real_ip_pos, wp_login_pos,
+                         'wp_login_url set-var must render after the real_ip set-var chain')
+        self.assertLess(wp_login_pos, redirect_pos,
+                         'wp_login_url set-var must render before the redirect rule that uses it')
+
+    def test_redirect_rule_uses_setvar_not_inline_regsub(self):
+        """Guards against reintroducing the rejected inline form."""
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        self.assertIn('%[var(txn.wp_login_url)]', rule)
+        self.assertNotIn('regsub', rule)
+
+    def test_redirect_rule_requires_safe_path(self):
+        """OPEN REDIRECT guard. The redirect target is built by rewriting
+        `path` with regsub, which only replaces the matched substring --
+        everything before "/wp-admin/" survives untouched in the output.
+        Three concrete requests turn that into an off-site `Location:`
+        header: "//evil.example.com/wp-admin/x.php" (protocol-relative,
+        browsers resolve "//host/path" to "https://host/path"),
+        "/\\evil.example.com/wp-admin/x.php" (browsers normalise a leading
+        "/\\" the same as "//"), and an RFC 7230 absolute-form request
+        target ("https://evil.example.com/wp-admin/x.php") which can make
+        HAProxy's `path` fetch return a full URI. wp_admin_safe_path
+        (requiring a well-formed absolute path) must be a POSITIVE
+        condition on the redirect rule -- scoped to the captured rule line
+        only, since the surrounding comment block also mentions this ACL
+        name and a bare substring match would pass even if the condition
+        were dropped from the rule itself.
+        """
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        self.assertIn('wp_admin_safe_path', rule)
+        self.assertNotIn('!wp_admin_safe_path', rule,
+                          'wp_admin_safe_path must be a positive condition, not negated')
+
+    def test_wp_admin_safe_path_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_safe_path', 'wp_admin_safe_path ACL')
+        self.assertIn('path_reg', line)
+
+    def test_unsafe_wp_admin_path_is_denied_not_passed_through(self):
+        """wp_admin_safe_path being a POSITIVE condition on the redirect means
+        a path that fails it is simply not redirected -- which used to mean it
+        fell through to the backend UNGATED, i.e. exactly the PHP-booting
+        request the gate exists to stop. Normalisation removes the "//"
+        spelling of that, but not "/\\", so the fall-through must be closed
+        with an explicit deny rather than left implicit.
+        """
+        denies = rule_lines(self.cfg, '!wp_admin_safe_path')
+        self.assertTrue(denies,
+                        'no rule denies a wp-admin path that fails wp_admin_safe_path')
+        self.assertTrue(any(d.startswith('http-request deny') for d in denies),
+                        'the !wp_admin_safe_path rule must be a deny: %r' % denies)
+
+
+def require_rule_by_predicate(cfg, needle, predicate, what):
+    """Like require_rule(), but for rules identified by needle + a predicate
+    over the comment-stripped line (e.g. "the `http-request redirect` line
+    that also mentions wp_admin_path", since the frontend has more than one
+    `http-request redirect`). Raises a clear AssertionError, not IndexError
+    or a silently-empty match, if no line satisfies both.
+    """
+    candidates = [ln for ln in rule_lines(cfg, needle) if predicate(ln)]
+    if not candidates:
+        raise AssertionError('no rule found matching %s' % what)
+    if len(candidates) > 1:
+        raise AssertionError('%s matched more than one rule line: %r' % (what, candidates))
+    return candidates[0]
+
+
+class UriNormalisation(unittest.TestCase):
+    """The gate matches the RAW path; the backend normalises and decodes it.
+    Every gap between those is a bypass -- five were found this way. These
+    tests pin the normalisation that closes the gap as a class.
+
+    NOTE: these are config-TEXT assertions. They are necessary but NOT
+    sufficient: the previous revision of this file passed while five live
+    bypasses shipped. The real evidence is the behavioural matrix run against
+    real haproxy 3.0.11 with raw sockets -- see
+    .superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-4-normalize-report.md.
+    """
+
+    def setUp(self):
+        self.cfg = render_listener()
+        self.header = render_header()
+
+    def test_experimental_directives_exposed_in_global(self):
+        """normalize-uri is experimental in 3.0; without this HAProxy refuses
+        to start (`haproxy -c` exits 1, ALERT). That does NOT crash-loop the
+        container, though -- see hap_header.tpl's comment and
+        haproxy_manager.py's start_haproxy()/do_initial_setup(): the failure
+        is swallowed, and the container comes up with haproxy simply never
+        running. This test exists so that silent-outage mode is never
+        reintroduced by dropping this line.
+        """
+        lines = rule_lines(self.header, 'expose-experimental-directives')
+        matches = [ln for ln in lines if ln == 'expose-experimental-directives']
+        self.assertTrue(
+            matches, 'expose-experimental-directives missing from the global section')
+
+    def test_all_normalizers_render(self):
+        for norm in NORMALIZERS:
+            with self.subTest(normalizer=norm):
+                self.assertTrue(
+                    rule_lines(self.cfg, 'normalize-uri ' + norm),
+                    'missing normalizer: ' + norm)
+
+    def test_normalizer_order_decode_before_path_walkers(self):
+        """Reverse this order and /wp-admin/js/%2e%2e/plugins.php reaches the
+        ACLs as /wp-admin/js/../plugins.php -- decoded but unresolved.
+        """
+        lines = [ln for ln in rule_lines(self.cfg, 'http-request normalize-uri')
+                 if ln.startswith('http-request normalize-uri')]
+        names = [ln.split('normalize-uri ', 1)[1] for ln in lines]
+        self.assertEqual(
+            names, list(NORMALIZERS),
+            'normalize-uri directives are missing, reordered, or duplicated: %r' % names)
+
+    def test_normalisation_precedes_every_path_based_rule(self):
+        """A normalizer placed after a path rule normalises nothing for it."""
+        norm_positions = []
+        for n in NORMALIZERS:
+            _, pos = require_position(self.cfg, 'http-request normalize-uri ' + n,
+                                       'normalizer: ' + n)
+            norm_positions.append(pos)
+        last_norm = max(norm_positions)
+        for marker in ('acl is_health_check', 'acl wp_login_path',
+                       'acl xmlrpc_path', 'acl wp_batch_path',
+                       'acl wp_admin_path', 'http-request set-path'):
+            with self.subTest(rule=marker):
+                _, marker_pos = require_position(self.cfg, marker, marker)
+                self.assertLess(last_norm, marker_pos,
+                                 marker + ' renders before URI normalisation')
+
+    def test_query_sort_by_name_is_not_enabled(self):
+        """query-sort-by-name reorders query parameters, which would break
+        anything that signs or caches on the exact query string. This is NOT
+        because the enabled normalizers already leave the query alone --
+        percent-to-uppercase and percent-decode-unreserved rewrite the WHOLE
+        request-target, query string included (see hap_listener.tpl's BLAST
+        RADIUS comment) -- it is a deliberate line between "case-fold /
+        decode" (no-ops under RFC 3986) and "reorder" (not a no-op for a
+        signed/cached query string).
+        """
+        self.assertFalse(rule_lines(self.cfg, 'normalize-uri query-sort-by-name'))
+
+    def test_dotdot_normalizer_uses_full(self):
+        """Without "full", ".." segments that climb above the root are left in
+        place and /../../wp-admin/plugins.php survives -- measured.
+        """
+        lines = rule_lines(self.cfg, 'normalize-uri path-strip-dotdot')
+        self.assertTrue(lines, 'missing normalizer: path-strip-dotdot')
+        for ln in lines:
+            self.assertTrue(ln.endswith('path-strip-dotdot full'), ln)
+
+    def test_encoded_separator_on_wp_admin_is_denied(self):
+        """percent-decode-unreserved deliberately leaves %2F encoded ("/" is
+        reserved), but OpenLiteSpeed decodes it and serves the file --
+        /wp-admin%2Fplugins.php was measured booting PHP on the OLS tier while
+        matching no wp-admin ACL. Normalisation cannot close this; it needs its
+        own rule.
+        """
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        acl_line = require_rule(self.cfg, 'acl path_has_encoded_sep', 'path_has_encoded_sep ACL')
+        self.assertIn('%2f', acl_line.lower())
+
+    def test_encoded_separator_deny_is_scoped_to_wp_admin(self):
+        """A blanket "deny any %2F in any path" would break non-WordPress
+        customer apps that legitimately pass an encoded slash in a path
+        parameter. The deny must be conditioned on the path mentioning
+        wp-admin.
+        """
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        for d in denies:
+            self.assertIn('wp_admin_word', d)
+
+    def test_encoded_separator_deny_honors_the_same_whitelist(self):
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        for d in denies:
+            for excl in ('!has_wp_logged_in', '!wp_gate_exempt', '!is_local',
+                         '!is_trusted_ip', '!is_whitelisted'):
+                with self.subTest(rule=d, exclusion=excl):
+                    self.assertIn(excl, d)
+
+    def test_encoded_separator_acl_matches_a_substring_not_a_prefix(self):
+        """/blog%2Fwp-admin/plugins.php hides the separator BEFORE "wp-admin",
+        where an anchored pattern never matches, and OLS still resolves it.
+        """
+        acl_line = require_rule(self.cfg, 'acl path_has_encoded_sep', 'path_has_encoded_sep ACL')
+        self.assertIn('-m sub', acl_line)
+
+    def test_wp_admin_asset_bypass_cannot_cover_a_php_entrypoint(self):
+        """The asset bypass anchored its prefix but not its suffix, so
+        /wp-admin/css/../plugins.php took it and the backend then resolved
+        ".." and booted plugins.php. path-strip-dotdot is the real fix; this
+        keeps the bypass structurally incapable of covering PHP.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertIn('.php', acl_line,
+                      'wp_admin_asset must exclude .php explicitly')
+
+    def test_wp_admin_asset_pattern_is_end_of_flags_guarded(self):
+        """A pattern starting with "(" makes HAProxy warn on EVERY load/reload:
+
+            parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
+            '(^|/)wp-admin/...' is likely a mistake ... Maybe you need to
+            remove the extraneous space before '('.
+
+        "--" is the end-of-flags marker HAProxy itself names as the fix. It is
+        cosmetic to matching but not to operations: an unsilenced warning on
+        every reload on every host trains people to skim past warnings, which
+        is how a real one gets missed. Assert the pattern is still the one we
+        think it is, so this can never pass by the pattern having been changed.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertRegex(
+            acl_line, r'path_reg\s+--\s+\(',
+            'wp_admin_asset pattern begins with "(" and MUST be preceded by "--"')
+        self.assertIn('(^|/)wp-admin/(css|js|images)/', acl_line)
+
+    def test_case_insensitive_acl_and_regsub_are_kept_in_sync(self):
+        """A case-insensitive wp_admin_path with a case-sensitive regsub is an
+        INFINITE REDIRECT LOOP: regsub finds no "/wp-admin/" in
+        "/WP-ADMIN/plugins.php", returns `path` unchanged, and the Location
+        then points at the request's own URL.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_path', 'wp_admin_path ACL')
+        setvar_line = require_rule(self.cfg, 'set-var(txn.wp_login_url)', 'wp_login_url set-var')
+        acl_ci = bool(re.search(r'path_reg\s+-i\s', acl_line))
+        regsub_ci = bool(re.search(r'regsub\([^)]*,\s*i\)', setvar_line))
+        self.assertEqual(
+            acl_ci, regsub_ci,
+            'wp_admin_path case-sensitivity (%s) and regsub flags (%s) disagree'
+            % (acl_line, setvar_line))
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-xmlrpc-rate-limit.py

@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""Regression tests for per-client-IP rate limiting on POST /xmlrpc.php.
+
+Why this file exists
+--------------------
+POST /xmlrpc.php floods were unthrottled fleet-wide. The generic frontend
+rate limits (hap_listener.tpl) trigger at 3000/5000 req/10s -- i.e. 300-500
+req/s -- but the observed floods run at a few req/s for hours, well under
+that ceiling. The existing wp_bruteforce mechanism (dedicated stick-table,
+60s window, per real client IP) solves exactly this shape of problem for
+POST /wp-login.php; this change adds an equivalent dedicated table/rule pair
+for POST /xmlrpc.php.
+
+This is only safe to key on var(txn.real_ip) because of the trusted-proxy
+gate added earlier (release 2026.08.3, see test-trusted-proxy-gate.py) --
+before that fix, a direct client could spoof any client IP via
+X-Forwarded-For and evade all per-IP tracking.
+
+These tests pin:
+  - a dedicated stick-table for xmlrpc tracking exists in
+    hap_security_tables.tpl (own sc slot / own counter -- not sharing the
+    wp_bruteforce counter, so a wp-login brute-force run and an xmlrpc flood
+    from the same IP don't inflate each other's rate)
+  - the tracking rule only fires on POST /xmlrpc.php (path_end, so
+    subdirectory WP installs are covered)
+  - the limiting rule tarpits over the chosen threshold
+  - the limiting rule honors the same whitelist as every other rule in the
+    file (!is_local !is_trusted_ip !is_whitelisted)
+  - xmlrpc is not blocked outright -- only the rate-limit ACL is present,
+    there's no blanket deny of the path
+
+Running
+-------
+    python3 scripts/test-xmlrpc-rate-limit.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_security_tables():
+    return haproxy_manager.template_env.get_template(
+        'hap_security_tables.tpl').render()
+
+
+class XmlrpcRateLimit(unittest.TestCase):
+
+    def setUp(self):
+        self.listener_cfg = render_listener()
+        self.tables_cfg = render_security_tables()
+
+    def test_dedicated_stick_table_defined(self):
+        """A dedicated table (not wp_bruteforce) tracks xmlrpc requests, so a
+        wp-login brute-force run and an xmlrpc flood from the same IP can't
+        inflate each other's rate counter."""
+        self.assertRegex(
+            self.tables_cfg,
+            r'backend\s+xmlrpc_bruteforce\s*\n\s*stick-table\s+type\s+ip\b.*store.*http_req_rate',
+        )
+        # Must not be the same table wp-login already uses.
+        self.assertNotIn('backend wp_bruteforce\n    stick-table type ip size 100k expire 30m store http_req_rate(60s)\nbackend xmlrpc_bruteforce', self.tables_cfg)
+
+    def test_xmlrpc_path_acl_uses_path_end(self):
+        """path_end (not path_beg) so subdirectory WP installs are covered,
+        matching the wp-login rule's reasoning."""
+        self.assertRegex(
+            self.listener_cfg,
+            r'acl\s+xmlrpc_path\s+path_end\s+/xmlrpc\.php',
+        )
+
+    def test_tracking_rule_only_fires_on_post_xmlrpc(self):
+        self.assertRegex(
+            self.listener_cfg,
+            r'http-request\s+track-sc2\s+var\(txn\.real_ip\)\s+table\s+xmlrpc_bruteforce\s+if\s+METH_POST\s+xmlrpc_path',
+        )
+
+    def test_limiting_rule_tarpits_over_threshold_with_whitelist(self):
+        pattern = (
+            r'http-request\s+tarpit\s+deny_status\s+429\s+if\s+METH_POST\s+xmlrpc_path\s+'
+            r'\{\s*sc_http_req_rate\(2\)\s+gt\s+(\d+)\s*\}\s+'
+            r'!is_local\s+!is_trusted_ip\s+!is_whitelisted'
+        )
+        match = re.search(pattern, self.listener_cfg)
+        self.assertIsNotNone(
+            match, 'expected a tarpit rule tracking sc2 with the full whitelist')
+        threshold = int(match.group(1))
+        self.assertGreater(threshold, 0)
+
+    def test_xmlrpc_not_blocked_outright(self):
+        """The endpoint must remain functional for clients under the
+        threshold -- only a rate-limit ACL, no blanket deny of the path."""
+        self.assertNotRegex(
+            self.listener_cfg,
+            r'http-request\s+deny\s+deny_status\s+\d+\s+if\s+(?:METH_POST\s+)?xmlrpc_path\s*(?:!is_local|\n)',
+        )
+
+    def test_rule_order_after_wp_login_block(self):
+        """Not load-bearing for correctness (mutually exclusive paths), but
+        keep the new block grouped with the other WordPress-specific rules
+        rather than scattered elsewhere in the file."""
+        wp_login_idx = self.listener_cfg.index('wp_login_path')
+        xmlrpc_idx = self.listener_cfg.index('xmlrpc_path')
+        self.assertLess(wp_login_idx, xmlrpc_idx)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/validate-rendered-config.py

@@ -0,0 +1,423 @@
+#!/usr/bin/env python3
+"""Build gate: render the real HAProxy config and hand it to the real `haproxy -c`.
+
+Why this file exists
+--------------------
+On 2026-08-14 a change to hap_listener.tpl rendered perfectly, passed every
+unit test in scripts/ (13 green), and was then rejected outright by HAProxy:
+
+    [ALERT] config : parsing [/etc/haproxy/haproxy.cfg:...] :
+            invalid arg 2 in converter 'regsub' : ... unexpected empty
+            replacement string
+
+Nothing between "commit" and "running in production" would have caught it.
+The unit tests assert on the *text* of the rendered config with regexes, which
+tells you what the template says, never whether HAProxy will accept it. And
+scripts/test-config-rollback.py stubs the `haproxy` binary with a shell script
+that only rejects a literal sentinel token, so its "validation" has never
+parsed a single line of real HAProxy syntax.
+
+The failure mode this guards is not cosmetic. When haproxy.cfg is invalid,
+scripts/init.py refuses to start HAProxy but the container still comes up:
+ports 80/443 are unbound, every site on the host is down, and /health keeps
+answering 200 because the Flask API is fine.
+
+So: render the config through the SAME code path production uses
+(haproxy_manager.generate_config(), templates and all), then run the actual
+`haproxy -c` against the result and gate on its exit code.
+
+This runs as a RUN step in the Dockerfile, which means it also validates
+against the exact haproxy binary that ships in the image being built - note
+that the Dockerfile installs haproxy UNPINNED, so that binary can move under
+us between builds. Any syntax the new binary rejects now fails the build
+instead of failing at 3am on an edge node.
+
+What it covers
+--------------
+  * every template generate_config() touches, assembled in the real order
+  * a domain with SSL + a backend, a wildcard domain, a cert-only domain with
+    no backend, and two template_override backends
+  * blocked-IP map entries (single IP and CIDR)
+  * BOTH sides of the two conditional blocks in hap_listener.tpl -
+    {%- if suspension_enabled %} and {%- if coraza_spoe_backend %} - because a
+    syntax error inside a conditional ships undetected otherwise. Scenario
+    "full" turns both on; scenario "default" leaves both off, which is the
+    byte-identical-to-standalone shape.
+
+Warnings vs failures
+--------------------
+`haproxy -c` emits warnings on a clean config here (at minimum "Can't load
+stats file" because /var/lib/haproxy/stats.dat doesn't exist at build time,
+plus assorted path_reg/ACL advisories). Those are NOT failures. This gate keys
+on the process EXIT CODE only, and dumps the full output when it is non-zero.
+
+Running
+-------
+    python3 scripts/validate-rendered-config.py
+
+Needs the real `haproxy` binary, the application's Python dependencies, and
+write access to /etc/haproxy (several templates reference files there by
+absolute path - see _REAL_PATH_NOTE below). Inside the image build all three
+hold. On a workstation, run it in the container instead.
+"""
+
+import logging
+import os
+import re
+import shutil
+import sqlite3
+import subprocess
+import sys
+import tempfile
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+# Same trick the other suites use: haproxy_manager configures logging at import
+# time against /var/log/haproxy-manager.log. Redirect the handlers so this runs
+# without root and without polluting the image's log files.
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-validate-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+try:
+    import haproxy_manager as hm  # noqa: E402
+finally:
+    logging.FileHandler = _real_file_handler
+
+# The application logs a lot at INFO during a render, and legitimately logs at
+# ERROR about things that are only true in this harness ("no existing HAProxy
+# config on disk ... ROLLBACK IS NOT AVAILABLE" - correct, there is no live
+# config during a build). Silence it so the build log carries the gate's own
+# verdict and haproxy's output, which is what matters.
+logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
+
+
+# _REAL_PATH_NOTE
+# ---------------
+# Most paths haproxy_manager writes to are module-level constants and are
+# redirected into a temp dir below. Two cannot be:
+#
+#   /etc/haproxy/blocked_ips.map   - hardcoded inside hap_listener.tpl's
+#                                    map_ip() converter
+#   /etc/haproxy/coraza-spoe.cfg   - hardcoded in the `filter spoe engine`
+#                                    line, and parsed by haproxy -c
+#
+# Redirecting the constants without editing the templates would just make
+# haproxy read a different (missing) file, so those two are left at their real
+# paths. Everything this script creates under /etc/haproxy is removed again on
+# exit; pre-existing files (the baked trusted_ips.*) are never touched.
+ETC_HAPROXY = '/etc/haproxy'
+
+# Files referenced with `-f` / map_ip() from the templates. A missing `-f` file
+# is a FATAL haproxy error, so a gate that didn't create these would fail for
+# reasons that have nothing to do with the config being tested.
+STUB_FILES = {
+    os.path.join(ETC_HAPROXY, 'trusted_ips.list'): '# validation stub\n203.0.113.10\n',
+    os.path.join(ETC_HAPROXY, 'trusted_ips.map'): '# validation stub\n203.0.113.11 1\n',
+    os.path.join(ETC_HAPROXY, 'cloudflare_ips.list'): '# validation stub\n198.51.100.0/24\n',
+    os.path.join(ETC_HAPROXY, 'trusted_proxies.list'): '# validation stub\n192.0.2.0/24\n',
+    os.path.join(ETC_HAPROXY, 'wpadmin_gate_exempt.list'): '# validation stub\nexempt.example.test\n',
+    os.path.join(ETC_HAPROXY, 'suspended_domains.list'): 'suspended.example.test\n',
+    # `lf-file` on the Coraza deny rule; loaded at parse time. Present in the
+    # image (COPY errors /haproxy/errors), stubbed for anything else.
+    '/haproxy/errors/403-waf.html': '<html><body>blocked %[unique-id]</body></html>\n',
+}
+
+# (suspension_enabled, coraza_spoe_backend) combinations to render + validate.
+SCENARIOS = (
+    ('default', {}),
+    ('full', {
+        'HAPROXY_SUSPENSION_ENABLED': 'true',
+        'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
+    }),
+)
+
+
+def log(msg):
+    sys.stdout.write(f'[validate-config] {msg}\n')
+    sys.stdout.flush()
+
+
+def fail(msg):
+    sys.stderr.write(f'[validate-config] FAIL: {msg}\n')
+    sys.stderr.flush()
+    raise SystemExit(1)
+
+
+class CreatedFiles:
+    """Tracks what we put on disk outside the temp dir so it can be removed.
+
+    Two sources: files we create explicitly, and files generate_config() itself
+    writes into /etc/haproxy (blocked_ips.map, coraza-spoe.cfg and their
+    .backup copies). The latter are caught by diffing the directory listing,
+    which also picks up anything a future change starts writing there.
+    """
+
+    def __init__(self):
+        self.explicit = []
+        self.etc_before = self._listdir(ETC_HAPROXY)
+
+    @staticmethod
+    def _listdir(path):
+        try:
+            return set(os.listdir(path))
+        except OSError:
+            return set()
+
+    def ensure(self, path, content):
+        """Create path with content if it does not already exist."""
+        if os.path.exists(path):
+            return
+        os.makedirs(os.path.dirname(path), exist_ok=True)
+        with open(path, 'w') as fh:
+            fh.write(content)
+        os.chmod(path, 0o644)
+        self.explicit.append(path)
+
+    def cleanup(self):
+        for path in self.explicit:
+            try:
+                os.unlink(path)
+            except OSError:
+                pass
+        for name in self._listdir(ETC_HAPROXY) - self.etc_before:
+            try:
+                os.unlink(os.path.join(ETC_HAPROXY, name))
+            except OSError:
+                pass
+
+
+def require_haproxy_binary():
+    """Fail closed. A gate that skips itself when the binary is missing is a
+    gate that would have let the 2026-08-14 change through."""
+    path = shutil.which('haproxy')
+    if not path:
+        fail('no `haproxy` binary on PATH - this gate cannot validate anything. '
+             'Run it inside the image (the Dockerfile installs haproxy).')
+    version = subprocess.run([path, '-v'], capture_output=True, text=True)
+    log(f'using {path}: {version.stdout.strip().splitlines()[0] if version.stdout else "unknown version"}')
+
+
+def make_self_signed_cert(certs_dir):
+    """HAProxy loads every file in the `bind ... ssl crt <dir>` directory at
+    parse time, so the directory has to hold a real, loadable bundle."""
+    os.makedirs(certs_dir, exist_ok=True)
+    cert = os.path.join(certs_dir, 'cert.tmp')
+    key = os.path.join(certs_dir, 'key.tmp')
+    subprocess.run(
+        ['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes',
+         '-keyout', key, '-out', cert, '-days', '1',
+         '-subj', '/CN=validate.example.test'],
+        check=True, capture_output=True,
+    )
+    bundle = os.path.join(certs_dir, 'validate.example.test.pem')
+    with open(bundle, 'w') as out:
+        for part in (cert, key):
+            with open(part) as fh:
+                out.write(fh.read())
+    os.unlink(cert)
+    os.unlink(key)
+    os.chmod(bundle, 0o600)
+
+
+def seed_database(db_path):
+    """A representative fleet: SSL + backend, wildcard, cert-only, overrides."""
+    hm.DB_FILE = db_path
+    hm.init_db()
+    with sqlite3.connect(db_path) as conn:
+        cur = conn.cursor()
+
+        def add_site(domain, backend, ssl_enabled=1, wildcard=0, override=None,
+                     servers=(('web1', '10.0.0.10', 8080, 'check'),)):
+            cur.execute(
+                'INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
+                'template_override, is_wildcard) VALUES (?, ?, ?, ?, ?)',
+                (domain, ssl_enabled, f'/etc/haproxy/certs/{domain}.pem',
+                 override, wildcard),
+            )
+            domain_id = cur.lastrowid
+            cur.execute('INSERT INTO backends (name, domain_id, settings) '
+                        'VALUES (?, ?, ?)', (backend, domain_id, None))
+            backend_id = cur.lastrowid
+            for name, addr, port, opts in servers:
+                cur.execute(
+                    'INSERT INTO backend_servers (backend_id, server_name, '
+                    'server_address, server_port, server_options) '
+                    'VALUES (?, ?, ?, ?, ?)',
+                    (backend_id, name, addr, port, opts),
+                )
+
+        add_site('site-one.example.test', 'site-one',
+                 servers=(('web1', '10.0.0.10', 8080, 'check'),
+                          ('web2', '10.0.0.11', 8080, 'check backup')))
+        add_site('site-two.example.test', 'site-two', ssl_enabled=0)
+        add_site('*.wildcard.example.test', 'wildcard-site', wildcard=1)
+        add_site('ws.example.test', 'ws-site', override='hap_backend_websocket')
+        add_site('sse.example.test', 'sse-site', override='hap_backend_longlived')
+
+        # Cert/management-only domain: registered for certificates, no backend.
+        # generate_config() has an explicit branch for this.
+        cur.execute(
+            'INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
+            'template_override, is_wildcard) VALUES (?, ?, ?, ?, ?)',
+            ('panel.example.test', 1, '/etc/haproxy/certs/panel.pem', None, 0),
+        )
+
+        # Both map_ip() shapes: a single address and a CIDR.
+        for ip in ('203.0.113.66', '198.51.100.0/24'):
+            cur.execute('INSERT INTO blocked_ips (ip_address, reason, blocked_by) '
+                        'VALUES (?, ?, ?)', (ip, 'validation fixture', 'gate'))
+        conn.commit()
+
+
+def render(scenario_name, env_overrides, workdir):
+    """Render via generate_config() and return the path to the assembled config.
+
+    generate_config() is the real production entry point: it takes the rollback
+    snapshot, writes the blocked-IP map, renders every template, and writes
+    haproxy.cfg. Only the reload is stubbed - there is no HAProxy process to
+    reload during a build, and validating the file is the whole point.
+    """
+    scenario_dir = os.path.join(workdir, scenario_name)
+    certs_dir = os.path.join(scenario_dir, 'certs')
+    os.makedirs(scenario_dir)
+    make_self_signed_cert(certs_dir)
+
+    hm.HAPROXY_CONFIG_PATH = os.path.join(scenario_dir, 'haproxy.cfg')
+    hm.HAPROXY_BACKUP_PATH = os.path.join(scenario_dir, 'haproxy.cfg.backup')
+    hm.CLUSTER_SECRET_PATH = os.path.join(scenario_dir, 'cluster-secret')
+    hm.HAPROXY_SOCKET_PATH = os.path.join(scenario_dir, 'haproxy.sock')
+    hm.SSL_CERTS_DIR = certs_dir
+    seed_database(os.path.join(scenario_dir, 'haproxy_config.db'))
+
+    saved_env = {}
+    for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND'):
+        saved_env[key] = os.environ.pop(key, None)
+    os.environ.update(env_overrides)
+
+    real_reload = hm.reload_haproxy_safely
+    hm.reload_haproxy_safely = lambda *a, **kw: (True, 'reload skipped: build-time validation')
+    try:
+        hm.generate_config()
+    finally:
+        hm.reload_haproxy_safely = real_reload
+        for key, value in saved_env.items():
+            os.environ.pop(key, None)
+            if value is not None:
+                os.environ[key] = value
+
+    return hm.HAPROXY_CONFIG_PATH
+
+
+def assert_scenario_branches(scenario_name, config_path, env_overrides):
+    """Cheap sanity check that the conditional blocks actually rendered.
+
+    Without this, a template refactor that silently stopped emitting the
+    suspension or Coraza block would leave the gate passing while covering
+    less than it claims to.
+    """
+    with open(config_path) as fh:
+        text = fh.read()
+    expected = {
+        'suspension': ('acl is_suspended_domain',
+                       'HAPROXY_SUSPENSION_ENABLED' in env_overrides),
+        'coraza': ('filter spoe engine coraza',
+                   'HAPROXY_CORAZA_SPOE_BACKEND' in env_overrides),
+    }
+    for label, (needle, should_be_present) in expected.items():
+        present = needle in text
+        if present != should_be_present:
+            fail(f'[{scenario_name}] {label} block {"missing" if should_be_present else "unexpectedly present"} '
+                 f'in the rendered config (looked for {needle!r}). The gate is '
+                 f'not covering what it thinks it is.')
+
+
+def _dump_context(config_path, output):
+    """Print the rendered lines HAProxy complained about.
+
+    The temp dir is deleted on the way out, so the build log has to carry the
+    evidence. HAProxy reports `parsing [<file>:<line>]`; show a window around
+    each reported line rather than dumping ~1500 lines of config.
+    """
+    line_numbers = sorted({
+        int(n) for n in re.findall(
+            r'parsing \[' + re.escape(config_path) + r':(\d+)\]', output)
+    })
+    if not line_numbers:
+        return
+    with open(config_path) as fh:
+        lines = fh.read().splitlines()
+    sys.stderr.write('[validate-config] --- rendered config around the error ---\n')
+    for number in line_numbers:
+        start = max(1, number - 6)
+        end = min(len(lines), number + 4)
+        for index in range(start, end + 1):
+            marker = '>>' if index == number else '  '
+            sys.stderr.write(f'{marker}{index:6d}| {lines[index - 1]}\n')
+        sys.stderr.write('[validate-config] ---\n')
+
+
+def validate(scenario_name, config_path):
+    result = subprocess.run(['haproxy', '-c', '-f', config_path],
+                            capture_output=True, text=True)
+    if result.returncode != 0:
+        output = result.stdout + result.stderr
+        sys.stderr.write(
+            f'\n[validate-config] ===== {scenario_name}: haproxy REJECTED the '
+            f'rendered configuration (exit {result.returncode}) =====\n')
+        sys.stderr.write(output if output.endswith('\n') else output + '\n')
+        _dump_context(config_path, output)
+        sys.stderr.write(
+            '[validate-config] This is a real HAProxy parse failure. Shipping it '
+            'would leave the container Up with ports 80/443 unbound and every '
+            'site on the host down, while /health still returns 200.\n')
+        sys.stderr.flush()
+        raise SystemExit(1)
+
+    # Exit code 0 is the verdict. Warnings are expected and are NOT failures:
+    # "Can't load stats file" always fires at build time, and HAProxy emits
+    # path_reg/ACL advisories on a perfectly valid config.
+    noise = (result.stdout + result.stderr).strip()
+    log(f'{scenario_name}: haproxy -c OK (exit 0)')
+    if noise:
+        for line in noise.splitlines():
+            log(f'  {scenario_name}: haproxy said: {line}')
+
+
+def main():
+    require_haproxy_binary()
+
+    if not os.path.isdir(ETC_HAPROXY) or not os.access(ETC_HAPROXY, os.W_OK):
+        fail(f'{ETC_HAPROXY} must exist and be writable - several templates '
+             'reference files there by absolute path. Run this inside the image.')
+
+    created = CreatedFiles()
+    workdir = tempfile.mkdtemp(prefix='haproxy-validate-')
+    try:
+        for path, content in STUB_FILES.items():
+            created.ensure(path, content)
+
+        for scenario_name, env_overrides in SCENARIOS:
+            log(f'rendering scenario "{scenario_name}" '
+                f'({env_overrides or "no optional features"})')
+            config_path = render(scenario_name, env_overrides, workdir)
+            assert_scenario_branches(scenario_name, config_path, env_overrides)
+            validate(scenario_name, config_path)
+    finally:
+        created.cleanup()
+        shutil.rmtree(workdir, ignore_errors=True)
+        shutil.rmtree(_LOG_DIR, ignore_errors=True)
+
+    log('all scenarios accepted by the real haproxy binary')
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

templates/hap_backend.tpl

@@ -4,14 +4,14 @@ backend {{ name }}-backend
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
     http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
 
     {% for server in servers %}
-    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
     {% endfor %}
 
 # SSE-specific backend - optimized for Server-Sent Events long-lived connections
@@ -29,12 +29,12 @@ backend {{ name }}-sse-backend
 
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
     http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
 
     {% for server in servers %}
-    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
     {% endfor %}

templates/hap_backend_http_check.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option httpchk
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     {% if ssl_enabled %}http-request set-header X-Forwarded-Proto https if { ssl_fc }{% endif %}

templates/hap_backend_longlived.tpl

@@ -0,0 +1,41 @@
+# Long-lived backend for {{ name }} (template_override='hap_backend_longlived').
+# Use for apps whose PRIMARY traffic holds connections open: media streaming,
+# large up/downloads, or persistent viewer/streaming sessions. Both the primary
+# and the SSE backend are tuned long-lived here (no http-server-close,
+# http-no-delay, 6h server/tunnel/keep-alive timeouts).
+#
+# Compare hap_backend_websocket.tpl, which keeps the PRIMARY backend standard
+# and only makes the -sse-backend long-lived. Pick this one when the main path
+# itself needs long-lived connections, not just an SSE side-channel.
+backend {{ name }}-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}
+
+# SSE variant (Accept: text/event-stream / ?action=stream auto-routes here)
+backend {{ name }}-sse-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}

templates/hap_backend_websocket.tpl

@@ -0,0 +1,34 @@
+# Long-lived / websocket-safe backend for {{ name }} (template_override)
+# For apps with persistent WebSocket/streaming connections (e.g. Jitsi /xmpp-websocket, /colibri-ws).
+backend {{ name }}-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}
+
+# SSE variant (Accept: text/event-stream / ?action=stream auto-routes here)
+backend {{ name }}-sse-backend
+    no option http-server-close
+    option http-no-delay
+    timeout server 6h
+    timeout tunnel 6h
+    timeout http-keep-alive 6h
+    option forwardfor
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-Real-IP %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-For %[var(txn.real_ip)]
+    http-request set-header X-Forwarded-Proto https if { ssl_fc }
+    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
+    {% for server in servers %}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
+    {% endfor %}

templates/hap_coraza_spoa_backend.tpl

@@ -0,0 +1,16 @@
+# Coraza-SPOA backend.
+# Only rendered into haproxy.cfg when HAPROXY_CORAZA_SPOE_BACKEND env var is
+# set on the haproxy-manager container. SPOE traffic to this backend is TCP,
+# not HTTP. The agent target comes from the env var so a single image can be
+# deployed against different sidecar host:port pairs (typically the sidecar
+# container's name + 9000 inside the shared docker network).
+backend coraza-spoa-backend
+    mode tcp
+    # spop-check actually speaks the SPOE protocol against the agent —
+    # confirms the agent can negotiate a session, not just that the TCP
+    # port is open. Required to detect a half-broken SPOA that's listening
+    # but not actually processing.
+    option spop-check
+    timeout connect 5s
+    timeout server  30s
+    server coraza-spoa {{ agent_target }} check

templates/hap_coraza_spoe_engine.tpl

@@ -0,0 +1,75 @@
+# Coraza SPOE engine configuration.
+#
+# Written to /etc/haproxy/coraza-spoe.cfg by haproxy_manager.generate_config()
+# when HAPROXY_CORAZA_SPOE_BACKEND env var is set. Referenced from haproxy.cfg
+# via `filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg`.
+#
+# Engine name "coraza" must match the engine name in the filter line in the
+# main config; group name "coraza-req" must match the send-spoe-group action.
+# Application name "haproxy" must match the application block in coraza-spoa's
+# config.yaml.
+#
+# Reference: this config follows the shape from coraza-spoa's upstream
+# example/haproxy/coraza.cfg (v0.7.1). Arg names + ordering are required by
+# Coraza-SPOA exactly as specified — DO NOT reorder or rename without
+# coordinating with the agent.
+
+[coraza]
+
+spoe-agent coraza
+    # `groups` (not `messages`) lists the spoe-group names this engine offers
+    # via `send-spoe-group` actions. The same group name appears below in a
+    # spoe-group block, which in turn references the actual message.
+    groups      coraza-req
+
+    # Prefix for variables the agent sets back on the request transaction —
+    # e.g. var(txn.coraza.error) when set-on-error triggers.
+    option      var-prefix coraza
+
+    # On agent error/timeout, set var(txn.coraza.error). We DON'T add a
+    # corresponding `http-request deny if { var(txn.coraza.error) -m bool }`
+    # in the frontend, so the request continues uninspected. This is the
+    # fail-open posture: WAF outage shouldn't 503 customer traffic.
+    option      set-on-error error
+
+    timeout     hello       2s
+    timeout     idle        2m
+    timeout     processing  100ms
+
+    use-backend coraza-spoa-backend
+
+    # NO `log global` here, deliberately.
+    #
+    # `log global` in a spoe-agent emits one line PER INSPECTED REQUEST, e.g.
+    #   SPOE: [coraza] <GROUP:coraza-req> sid=537 st=0 0/0/0/0/0 32/32 0/0 0/467
+    # Measured on whp01 immediately after access logging started working:
+    # 618 SPOE lines vs 669 real access lines -- it was ~48% of the log volume,
+    # i.e. it would roughly DOUBLE the edge's log footprint (~400 MB/day extra)
+    # to record `st=0` over and over.
+    #
+    # It carries nothing incident response needs: the WAF's verdict is already
+    # visible in the access log line (status 403 + the `id=` UUID, which joins
+    # to /var/log/coraza/audit.log for the rule_id), and per-transaction WAF
+    # detail is written by the SPOA itself to /var/log/coraza/spoa.log.
+    # Agent-level failures still surface via `option set-on-error error` ->
+    # var(txn.coraza.error) and the fail-open path in hap_listener.tpl.
+
+# Per-request inspection message. No `event` directive — fires only when
+# explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.
+# Arg order/names are mandatory: Coraza-SPOA parses positionally and renames
+# break the agent. `app=str(haproxy)` is the literal application name from
+# coraza-spoa's config.yaml `applications:` block.
+#
+# src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
+# top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
+# only honoring those headers from trusted proxies). Falls back to `src`
+# when no proxy headers are present. This is what shows up as `client_ip`
+# in /var/log/coraza/audit.log and the rule manager's view-matches panel.
+spoe-message coraza-req
+    args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
+
+# Group binding for send-spoe-group invocation in the frontend. One group,
+# one message; could add more in the future (e.g. coraza-res for response
+# inspection — currently disabled in coraza-spoa's config.yaml).
+spoe-group coraza-req
+    messages coraza-req

templates/hap_header.tpl

@@ -2,20 +2,46 @@
 # Global settings
 #---------------------------------------------------------------------
 global
-    # to have these messages end up in /var/log/haproxy.log you will
-    # need to:
+    # ACCESS LOG DESTINATION.
     #
-    # 1) configure syslog to accept network log events.  This is done
-    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
-    #    /etc/sysconfig/syslog
+    # This used to be `log 127.0.0.1 local2`, which was a silent black hole:
+    # 127.0.0.1 is the CONTAINER's own loopback, nothing has ever listened on
+    # udp/514 in the container netns, and there is no /dev/log in the image.
+    # Every access log line -- ~1.5M/day across the whole edge -- was written
+    # to a socket with no receiver and dropped. Nothing errored, nothing
+    # warned, and `haproxy -c` was perfectly happy. The cost only shows up
+    # during an incident: per-IP 429s, tarpits, wp-admin gate redirects,
+    # WAF 403s and `silent-drop`s left no record anywhere, so the edge could
+    # not be asked what it had rejected. Only aggregate stick-table counters
+    # survived.
     #
-    # 2) configure local2 events to go to the /var/log/haproxy.log
-    #   file. A line like the following can be added to
-    #   /etc/sysconfig/syslog
+    # Now points at the DOCKER BRIDGE GATEWAY, where the host's rsyslog has an
+    # imudp listener bound (installed idempotently by WHP's
+    # setup-haproxy-logrotate.sh, which also writes the logrotate stanza).
+    # The host writes local2 to /var/log/haproxy.log and stops it there, so it
+    # does not also flood /var/log/messages or the Graylog forwarder.
     #
-    #    local2.*                       /var/log/haproxy.log
+    # WHY NOT `log stdout format raw local0`: it is INCOMPATIBLE with the
+    # `daemon` keyword below, and incompatible SILENTLY. Verified on the
+    # pinned 3.0.11 binary: with `daemon` set, a `log stdout` config serves
+    # traffic normally and emits ZERO log lines, and `haproxy -c` returns 0
+    # with no error and no warning -- so the CI config gate
+    # (scripts/validate-rendered-config.py) cannot catch it either. Making it
+    # work means dropping `daemon` / adding -db so haproxy stays in the
+    # foreground, which in turn breaks the three synchronous
+    # `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites in
+    # haproxy_manager.py (they would block until the 180s timeout and then be
+    # killed). That is a change to the exact code path whose failure mode is
+    # "container Up, ports 80/443 never bound, every site down, /health still
+    # 200". Not worth it for a logging change.
     #
-    log         127.0.0.1 local2
+    # UDP means a dead listener degrades to dropped log lines, never to a
+    # stalled or failing request path -- the correct failure direction for an
+    # edge fronting ~60 customer sites.
+    #
+    # len 2048 accommodates the enriched log-format in hap_listener.tpl
+    # (URL + User-Agent + UUID); the default 1024 would truncate long ones.
+    log         {{ syslog_target }} len 2048 format rfc5424 local2 info
 
     chroot      /var/lib/haproxy
     pidfile     /var/run/haproxy.pid
@@ -27,12 +53,77 @@ global
     # SSL and Performance
     tune.ssl.default-dh-param 2048
 
+    # Required by the `http-request normalize-uri` chain at the top of the
+    # `web` frontend (hap_listener.tpl). normalize-uri is still flagged
+    # EXPERIMENTAL in HAProxy 3.0, and HAProxy REFUSES TO START without this
+    # opt-in -- not a warning, a fatal:
+    #   [ALERT] config : parsing [...] : 'normalize-uri' action is
+    #   experimental, must be allowed via a global
+    #   'expose-experimental-directives'
+    # (verified against real haproxy 3.0.11-9e587df: `haproxy -c` exits 1).
+    # So this line and the normalize-uri rules must be added/removed together.
+    #
+    # Dropping this one alone does NOT crash-loop the container -- the truth
+    # is worse: it is a SILENT TOTAL OUTAGE that nothing escalates. Container
+    # init (scripts/init.py -> haproxy_manager.do_initial_setup()) calls
+    # generate_config() (which still succeeds -- Jinja doesn't validate
+    # HAProxy semantics) and then start_haproxy(), which runs `haproxy -c`,
+    # sees it fail, logs an error, and RETURNS WITHOUT RAISING. init.py exits
+    # 0. scripts/start-up.sh then execs gunicorn as PID 1 regardless. Result:
+    # the container stays "Up", ports 80/443 are never bound, EVERY SITE ON
+    # THE HOST IS DOWN, and the in-container supervisor loop
+    # (ensure_haproxy.py, every HAPROXY_SUPERVISOR_INTERVAL seconds) retries
+    # the identical failing render forever without ever escalating. Worse
+    # still, GET /health keeps returning HTTP 200 -- health_check() only
+    # answers 500 on a database error; a dead haproxy just flips the JSON
+    # body's "haproxy_status" to "stopped" while the status code a naive
+    # monitor checks never changes. Do not trust /health alone to catch this.
+    #
+    # This exposes ONLY the experimental directives that are actually used --
+    # it does not change the behaviour of anything else in this file.
+    expose-experimental-directives
+
+    # HTTP/3 over QUIC. The Debian haproxy package is built against system
+    # OpenSSL via the compatibility shim (USE_QUIC_OPENSSL_COMPAT), which is
+    # not a native QUIC TLS stack. HAProxy therefore rejects `quic*@` binds
+    # unless this opt-in is set. `limited-quic` enables QUIC through the compat
+    # layer (no 0-RTT — that needs quictls/aws-lc or native OpenSSL 3.5 QUIC).
+    # Without this, the quic bind in the frontend fails to start: "this SSL
+    # library does not support the QUIC protocol".
+    limited-quic
+{%- if cluster_secret %}
+
+    # Stable secret keying QUIC Retry/address-validation tokens. Self-healed
+    # to /etc/haproxy/cluster-secret (named volume) by the manager so it
+    # survives recreates; without it haproxy picks a random one per process
+    # and tokens don't survive reloads (benign, just a startup notice).
+    cluster-secret "{{ cluster_secret }}"
+{%- endif %}
+
     # HTTP/2 protection against Rapid Reset (CVE-2023-44487) and stream abuse
     tune.h2.fe.max-total-streams 2000
     tune.h2.fe.glitches-threshold 50
 
     # Stats persistence for zero-downtime reloads
     stats-file /var/lib/haproxy/stats.dat
+
+#---------------------------------------------------------------------
+# DNS resolver for Docker container name resolution
+# Re-resolves backend server addresses so container IP changes
+# (from restarts, recreations, scaling) are picked up automatically
+#---------------------------------------------------------------------
+resolvers docker_dns
+    nameserver dns1 127.0.0.11:53
+    resolve_retries 3
+    timeout resolve 1s
+    timeout retry 1s
+    hold valid 10s
+    hold other 10s
+    hold refused 10s
+    hold nx 10s
+    hold timeout 10s
+    hold obsolete 10s
+
 #---------------------------------------------------------------------
 # common defaults that all the 'listen' and 'backend' sections will
 # use if not designated in their block
@@ -55,4 +146,19 @@ defaults
     timeout check           10s
     timeout tarpit          10s  # Tarpit delay for low-level scanners (before silent-drop)
     maxconn                 3000
+
+    # Per-request unique reference, used:
+    #   - in the access log line, as the `id=` field of the custom log-format
+    #     in hap_listener.tpl. NOTE: `option httplog` does NOT include %ID
+    #     (verified against haproxy 3.0.11) -- this comment used to claim it
+    #     did, which made the support workflow below look supported when it
+    #     was not. The explicit log-format is what actually carries it.
+    #   - echoed to clients in the X-Request-Reference response header on
+    #     WAF blocks so a customer can quote it when opening a support ticket
+    #   - embedded in /etc/haproxy/errors/403-waf.html so a blocked visitor
+    #     sees it on the rendered 403 page
+    # Support correlates ref → /var/log/haproxy.log line → timestamp+client+host
+    # → /var/log/coraza/audit.log entry → rule_id.
+    unique-id-format        %[uuid()]
+    unique-id-header        X-Request-Reference
     

templates/hap_listener.tpl

@@ -4,46 +4,638 @@ frontend web
     # crt can now be a path, so it will load all .pem files in the path
     bind 0.0.0.0:443 ssl crt {{ crt_path }} alpn h2,http/1.1
 
-    # Capture Host header so it appears in httplog output (in %hr field)
+    # HTTP/3 over QUIC (UDP/443). Same cert path as the TCP listener above.
+    # The Debian haproxy package is built +QUIC (QUIC_OPENSSL_COMPAT), so this
+    # is config-only — no source build. Requires UDP/443 published on the
+    # container (`-p 443:443/udp`) and open at the host firewall. `h3` is the
+    # only ALPN QUIC negotiates; h2/http1 stay on the TCP bind above. Sharing
+    # the frontend means all the real-IP, rate-limit, IP-block and Coraza
+    # rules below apply identically to H3 traffic.
+    bind quic4@0.0.0.0:443 ssl crt {{ crt_path }} alpn h3
+
+    # Advertise H3 so browsers upgrade their existing TCP (h2) connection to
+    # QUIC on the next request. `ma` is how long (seconds) the client may
+    # cache the advertisement. http-after-response applies it to every
+    # response, including haproxy-generated ones (blocks, default page).
+    http-after-response set-header alt-svc "h3=\":443\"; ma=86400"
+
+    # Capture Host header so it appears in httplog output (in %hr field).
+    # ORDER IS LOAD-BEARING: this is capture slot 0, referenced by the
+    # access log-format below as %[capture.req.hdr(0)].
     http-request capture req.hdr(Host) len 64
 
+    # Capture slot 1 = User-Agent. Incident response needs it to tell a
+    # scanner from a browser, and it is not in `option httplog` output.
+    # Any new capture MUST be appended AFTER this line, never inserted
+    # above it, or the slot indices in the log-format silently shift and
+    # the access log starts attributing the wrong string to the wrong field.
+    # req.fhdr(), NOT req.hdr(): req.hdr() treats the header as a comma-
+    # separated list and returns only the LAST element. Real User-Agent strings
+    # contain commas -- "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
+    # AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
+    # captured with req.hdr() logs as just "like Gecko) Chrome/131.0.0.0
+    # Safari/537.36", silently losing the platform half -- which is exactly the
+    # half you need to tell a spoofed crawler from a real browser.
+    # Observed in production on whp01 before this was corrected.
+    http-request capture req.fhdr(User-Agent) len 200
+
+    # --- Access logging -----------------------------------------------------
+    # Scoped to THIS frontend on purpose: it references capture slots and
+    # var(txn.real_ip), which only exist here. Putting it in `defaults` would
+    # apply it to the stats frontend and every backend too, where those
+    # samples are undefined.
+    #
+    # `log-format` overrides `option httplog` for this proxy (haproxy emits a
+    # harmless warning saying so). The first 16 fields are byte-identical to
+    # the 3.0 httplog default, so anything that already parses httplog keeps
+    # working; the `key=value` tail is additive.
+    #
+    # Why the default httplog is not enough for incident response:
+    #   %ci  is the PROXY's address for Cloudflare-fronted sites, not the
+    #        visitor. The real client is var(txn.real_ip), resolved further
+    #        down from CF-Connecting-IP / X-Real-IP / X-Forwarded-For and only
+    #        honoured from trusted proxies. BOTH are logged: cip= is who to
+    #        rate-limit or block, %ci is which edge it arrived through.
+    #   %ID  is NOT included by `option httplog` (verified against
+    #        haproxy 3.0.11). Without it the documented support workflow
+    #        -- X-Request-Reference -> access log -> coraza audit.log -> rule_id
+    #        -- cannot be completed. id= is what makes that join possible.
+    #   host=/ua= identify the vhost and client; %ST/%B/%tsc give status,
+    #        bytes and the termination state that distinguishes a rate-limit
+    #        deny (PR--) from a tarpit (PT--) from a normal close.
+    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major]"
+
+    # --- URI normalisation (MUST be the first path-touching block here) ---
+    # Every path-based control in this frontend (the ACME health-check bypass,
+    # wp-login, xmlrpc, the wp-json/batch virtual patch, the wp-admin gate, the
+    # blocked-IP and suspension set-path rules, and everything Coraza inspects)
+    # matched the RAW request-target while the backend NORMALISED and DECODED
+    # it before resolving a file. Every gap between those two behaviours is a
+    # bypass, and each one had to be patched individually. Five were found in
+    # the wp-admin gate alone, all the same class:
+    #
+    #   //wp-admin/plugins.php            raw path starts "//" -> the safe-path
+    #                                     guard failed -> request fell through
+    #                                     ungated, backend served it
+    #   /wp-admin/css/../plugins.php      matched the css/js/images asset
+    #                                     bypass; backend resolved ".." and
+    #                                     booted plugins.php
+    #   /wp-admin/js/%2e%2e/plugins.php   same, with the ".." percent-encoded
+    #   /wp%2Dadmin/plugins.php           "wp-admin" spelled with %2D never
+    #                                     matched any wp-admin ACL at all
+    #   /wp-admin%2Fplugins.php           encoded separator; see the dedicated
+    #                                     rule in the wp-admin gate below
+    #
+    # Rather than keep bolting a counter-pattern onto each rule, normalise the
+    # URI once, here, so every rule below matches the SAME string the backend
+    # will resolve. HAProxy rewrites the request-target in place, so the
+    # backend receives the normalised form too.
+    #
+    # ORDER IS LOAD-BEARING and was determined empirically against real
+    # haproxy 3.0.11, not from the docs. The decoders must run BEFORE the path
+    # walkers: with the reverse order, /wp-admin/js/%2e%2e/plugins.php ends up
+    # as /wp-admin/js/../plugins.php -- decoded, but the ".." left unresolved,
+    # because path-strip-dotdot had already run by the time the "%2e%2e"
+    # became "..". Verified both directions side by side.
+    #
+    #   percent-to-uppercase       %2f -> %2F. Canonicalises the spelling of
+    #                              whatever stays encoded, so downstream rules
+    #                              need one case of each escape, not two.
+    #   percent-decode-unreserved  Decodes ONLY RFC 3986 unreserved chars
+    #                              (A-Za-z0-9-._~). This is what turns %2e%2e
+    #                              into .. and wp%2Dadmin into wp-admin.
+    #                              Reserved escapes are deliberately left
+    #                              alone -- %2F in particular, which is why
+    #                              the wp-admin gate needs its own encoded-
+    #                              slash rule (see below).
+    #   path-merge-slashes         //x -> /x. Also removes the entire class of
+    #                              "leading // defeats an anchored regex".
+    #   path-strip-dot             /a/./b -> /a/b.
+    #   path-strip-dotdot full     /a/b/../c -> /a/c. "full" additionally
+    #                              resolves ".." segments that would climb
+    #                              above the root (/../../wp-admin/x.php ->
+    #                              /wp-admin/x.php); without "full" HAProxy
+    #                              leaves those in place and the vector
+    #                              survives -- measured, both forms tested.
+    #
+    # DELIBERATELY NOT ENABLED: query-sort-by-name. Reordering query-string
+    # parameters would silently break anything that signs or caches on the
+    # exact query string (signed asset URLs, HMAC'd callbacks, CDN cache
+    # keys). This is NOT because the enabled normalizers already leave the
+    # query alone -- see BLAST RADIUS just below, they don't -- it is a
+    # deliberate line drawn between "case-fold / decode", which RFC 3986
+    # defines as no-ops, and "reorder", which is not a no-op for a caller
+    # treating the query as an opaque signed string.
+    #
+    # BLAST RADIUS: this block applies to EVERY request for EVERY site on
+    # EVERY tier, so the decoding was kept minimal on purpose -- and it is
+    # NOT scoped to the path. percent-to-uppercase and percent-decode-
+    # unreserved normalise the WHOLE request-target as HAProxy parses it --
+    # query string included, not just the path component the ACLs below
+    # match on -- and the BACKEND receives the rewritten query on the wire,
+    # not just an internal haproxy view of it. Measured against real HAProxy
+    # 3.0.11:
+    #   /a?sig=%2babc%2fdef  -> /a?sig=%2Babc%2Fdef   (percent-hex upper-cased)
+    #   /a?b=%41%42%43       -> /a?b=ABC               (unreserved chars decoded)
+    #   /a?tok=%7e%2d%5f%2e  -> /a?tok=~-_.            (unreserved chars decoded)
+    # Only parameter ORDER is preserved -- that guarantee is exactly why
+    # query-sort-by-name above is the one normalizer in this family left
+    # disabled. Per RFC 3986 both enabled rewrites are defined as the same
+    # URI (case in a percent-escape, and an unreserved character vs. its
+    # escape, carry no distinct meaning), but "the same URI" is not "the
+    # same bytes": an application that HMACs or otherwise signs the RAW
+    # query string, rather than parsing it first, could see a mutated value
+    # and fail to verify an otherwise-legitimate request. Checked against
+    # this fleet: no .NET backends (.NET's UrlEncode emits lowercase
+    # percent-hex, which percent-to-uppercase would rewrite) and no
+    # URL-in-path proxies, so there is no known victim today -- but do not
+    # assume "path only" from this block; that was the actual bug in an
+    # earlier draft of this comment.
+    #
+    # normalize-uri is EXPERIMENTAL in 3.0 and requires
+    # `expose-experimental-directives` in the global section
+    # (hap_header.tpl). Without it HAProxy does not start. Remove one and you
+    # must remove the other.
+    http-request normalize-uri percent-to-uppercase
+    http-request normalize-uri percent-decode-unreserved
+    http-request normalize-uri path-merge-slashes
+    http-request normalize-uri path-strip-dot
+    http-request normalize-uri path-strip-dotdot full
+
+    # --- Trusted-proxy gate (MUST precede real-IP resolution below) ---
+    # CF-Connecting-IP / X-Real-IP / X-Forwarded-For are client-supplied. Any
+    # peer that is not a known reverse proxy gets them stripped, so the
+    # set-var chain below falls through to `src` -- the real TCP peer.
+    #
+    # Without this, a direct client dictates txn.real_ip, and every control
+    # keyed on that variable trusts the attacker's own claim: rate limiting
+    # (track-sc0), the trusted-IP whitelist, the wp-login brute-force table and
+    # cookie challenge, the wp-json/batch/v1 virtual patch, IP blocking, and
+    # Coraza's src-ip. Spoofing a whitelisted IP bypassed all of them.
+    #
+    # ORDER MATTERS: these must come before the set-var lines. HAProxy applies
+    # http-request rules in file order, so a strip placed afterwards would
+    # validate cleanly and accomplish nothing.
+    #
+    # Cloudflare-fronted domains keep working: CF's edge matches
+    # from_trusted_proxy, so its CF-Connecting-IP survives.
+    acl from_trusted_proxy src -f /etc/haproxy/cloudflare_ips.list -f /etc/haproxy/trusted_proxies.list
+    http-request del-header CF-Connecting-IP if !from_trusted_proxy
+    http-request del-header X-Real-IP        if !from_trusted_proxy
+    http-request del-header X-Forwarded-For  if !from_trusted_proxy
+
     # Detect real client IP from proxy headers if they exist
     # Priority: CF-Connecting-IP (Cloudflare) > X-Real-IP > X-Forwarded-For > src
     acl has_cf_connecting_ip req.hdr(CF-Connecting-IP) -m found
     acl has_x_real_ip req.hdr(X-Real-IP) -m found
     acl has_x_forwarded_for req.hdr(X-Forwarded-For) -m found
 
-    # Set the real IP based on available headers
-    http-request set-var(txn.real_ip) req.hdr(CF-Connecting-IP) if has_cf_connecting_ip
-    http-request set-var(txn.real_ip) req.hdr(X-Real-IP) if !has_cf_connecting_ip has_x_real_ip
-    http-request set-var(txn.real_ip) req.hdr(X-Forwarded-For) if !has_cf_connecting_ip !has_x_real_ip has_x_forwarded_for
+    # Set the real IP based on available headers. Use hdr_ip (not hdr) so the
+    # variable is typed as IP — required by the Coraza SPOE arg `src-ip` which
+    # decodes binary IP bytes (passing a string IP panics the SPOA goroutine).
+    # `hdr_ip(X-Forwarded-For,1)` extracts the FIRST address from a possibly
+    # comma-separated chain (original client, not intermediate proxies).
+    http-request set-var(txn.real_ip) req.hdr_ip(CF-Connecting-IP) if has_cf_connecting_ip
+    http-request set-var(txn.real_ip) req.hdr_ip(X-Real-IP) if !has_cf_connecting_ip has_x_real_ip
+    http-request set-var(txn.real_ip) req.hdr_ip(X-Forwarded-For,1) if !has_cf_connecting_ip !has_x_real_ip has_x_forwarded_for
     http-request set-var(txn.real_ip) src if !has_cf_connecting_ip !has_x_real_ip !has_x_forwarded_for
 
     # --- Connection & rate tracking ---
     stick-table type ip size 200k expire 10m store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(30s)
     http-request track-sc0 var(txn.real_ip)
 
-    # Whitelist: let health checks and local traffic bypass rate limits
+    # Whitelist: let health checks, local, and trusted traffic bypass rate limits
     acl is_local src 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
+    acl is_trusted_ip src -f /etc/haproxy/trusted_ips.list
     acl is_health_check path_beg /.well-known/acme-challenge
+    acl is_whitelisted var(txn.real_ip),map_ip(/etc/haproxy/trusted_ips.map,0) -m int gt 0
 
     # --- Rate limit rules (applied in order, first match wins) ---
-    # Hard block: >500 req/10s per IP (sustained flood)
-    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 500 } !is_local !is_health_check
-    # Tarpit: >200 req/10s per IP (aggressive scraping / light flood)
-    http-request tarpit deny_status 429 if { sc_http_req_rate(0) gt 200 } !is_local !is_health_check
-    # Connection rate limit: >150 new connections per 10s per IP
-    http-request deny deny_status 429 if { sc_conn_rate(0) gt 150 } !is_local !is_health_check
-    # Concurrent connection limit: >100 simultaneous connections per IP
-    http-request deny deny_status 429 if { sc_conn_cur(0) gt 100 } !is_local !is_health_check
-    # High error rate: >20 errors in 30s (scanner/fuzzer behavior)
-    http-request tarpit deny_status 403 if { sc_http_err_rate(0) gt 20 } !is_local !is_health_check
+    # Thresholds are generous to accommodate media-heavy sites where a
+    # single page can load 100+ images/assets. These only trigger on
+    # obvious automated abuse, not real users.
+    #
+    # Hard block: >5000 req/10s per IP (500 req/s — sustained flood)
+    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 5000 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # Tarpit: >3000 req/10s per IP (300 req/s — aggressive bot/scraper)
+    http-request tarpit deny_status 429 if { sc_http_req_rate(0) gt 3000 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # Connection rate limit: >500 new connections per 10s per IP
+    http-request deny deny_status 429 if { sc_conn_rate(0) gt 500 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # Concurrent connection limit: >500 simultaneous connections per IP
+    http-request deny deny_status 429 if { sc_conn_cur(0) gt 500 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # High error rate: >100 errors in 30s (scanner/fuzzer behavior)
+    http-request tarpit deny_status 403 if { sc_http_err_rate(0) gt 100 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+
+    # --- WordPress wp-login.php brute-force protection ---
+    # The generic limits above are deliberately high (media-heavy sites), so a
+    # slow credential-stuffing run (dozens of login POSTs/min) slips under them.
+    # Track POSTs to wp-login.php per real client IP in a DEDICATED 60s table
+    # (sc1 / backend wp_bruteforce, defined in hap_security_tables.tpl) and
+    # tarpit once an IP exceeds 30/min. Only login POSTs are counted — GETs of
+    # the login form, normal browsing, and the handful of POSTs a legit user
+    # makes are unaffected; an offending IP can still browse, just not keep
+    # hammering login. path_end also covers subdirectory WP installs. Honors the
+    # same whitelist (RFC1918 / trusted_ips.list / trusted_ips.map).
+    acl wp_login_path path_end /wp-login.php
+    http-request track-sc1 var(txn.real_ip) table wp_bruteforce if METH_POST wp_login_path
+    http-request tarpit deny_status 429 if METH_POST wp_login_path { sc_http_req_rate(1) gt 30 } !is_local !is_trusted_ip !is_whitelisted
+
+    # --- WordPress wp-login.php "must-load-the-form-first" cookie challenge ---
+    # Defeats DISTRIBUTED credential-stuffing (hundreds of thousands of unique
+    # IPs, each low-and-slow, so the per-IP rule above can't see them). Such
+    # bots POST straight to /wp-login.php without ever GETting the form — on
+    # these sites the login POST:GET ratio is ~15:1. We hand out a cookie when
+    # the form is actually fetched (GET) and require it on POST; direct-POST
+    # bots lack it and are denied AT THE EDGE before reaching PHP. Real logins
+    # are unaffected — WordPress login already requires loading the page and
+    # accepting cookies. Immediate deny (NOT tarpit) — under a 300k-POST flood,
+    # holding tarpit connections would exhaust HAProxy. Honors the whitelist.
+    # Mark login-form GETs at REQUEST time (method/path are reliably evaluable
+    # here; in the response phase they are not) so the cookie is emitted on the
+    # form's own response.
+    http-request set-var(txn.wp_login_form) int(1) if METH_GET wp_login_path
+    http-after-response add-header set-cookie "whplc=1; Path=/; Max-Age=1800; HttpOnly; Secure; SameSite=Lax" if { var(txn.wp_login_form) -m found }
+    acl has_login_cookie req.cook(whplc) -m found
+    http-request deny deny_status 403 if METH_POST wp_login_path !has_login_cookie !is_local !is_trusted_ip !is_whitelisted
+
+    # --- WordPress xmlrpc.php flood protection ---
+    # xmlrpc.php floods are a common, sustained abuse pattern that the generic
+    # limits above don't catch: those trigger at 3000/5000 req/10s (300-500
+    # req/s, sized for media-heavy pageloads), while observed xmlrpc floods run
+    # at just a few req/s for hours -- comfortably under that ceiling but still
+    # enough to pin PHP-FPM workers and show up as 503s for the rest of the
+    # site. Same shape of problem as wp-login credential stuffing, so it gets
+    # the same fix: track POSTs to xmlrpc.php per real client IP in a DEDICATED
+    # 60s table (sc2 / backend xmlrpc_bruteforce, defined in
+    # hap_security_tables.tpl -- kept separate from wp_bruteforce so the two
+    # endpoints' traffic can't inflate each other's counter, see that file for
+    # the reasoning) and tarpit once an IP exceeds the threshold.
+    #
+    # Threshold is 60/min (double wp-login's 30/min), not because xmlrpc abuse
+    # is less severe but because legitimate traffic here is machine-to-machine
+    # rather than a human filling out a form: Jetpack sync, the WordPress
+    # mobile app, and remote-publishing clients (e.g. an offline blog editor)
+    # can legitimately burst several xmlrpc calls in quick succession. 60/min
+    # (1 req/s average over the window) comfortably absorbs that burst while
+    # still tripping well before an hours-long few-req/s flood does real
+    # damage -- at 2 req/s sustained the 60s counter clears the threshold in
+    # under a minute.
+    #
+    # Tarpit (not deny), matching the wp-login rule: this is per-IP tracking
+    # of a bounded set of offenders, not the wp-login cookie challenge's
+    # distributed hundreds-of-thousands-of-IPs scenario where holding
+    # connections would exhaust HAProxy itself, so tying up the flooding IP's
+    # connections is the cheaper and more effective response. path_end (not
+    # path_beg) covers subdirectory WP installs, same reasoning as wp-login.
+    # Honors the same whitelist (RFC1918 / trusted_ips.list / trusted_ips.map)
+    # so health checks and trusted infrastructure are unaffected, and legit
+    # clients under the threshold are never blocked outright.
+    acl xmlrpc_path path_end /xmlrpc.php
+    http-request track-sc2 var(txn.real_ip) table xmlrpc_bruteforce if METH_POST xmlrpc_path
+    http-request tarpit deny_status 429 if METH_POST xmlrpc_path { sc_http_req_rate(2) gt 60 } !is_local !is_trusted_ip !is_whitelisted
+
+    # WordPress REST batch endpoint lockdown ("wp2shell": CVE-2026-63030 +
+    # CVE-2026-60137). Chaining a core SQL injection with REST batch-route
+    # confusion gives unauthenticated RCE on WP 6.9.0-6.9.4 and 7.0.0-7.0.1
+    # (fixed in 6.9.5 / 7.0.2). Exploits are public and were used against this
+    # fleet on 2026-07-19/20; one site was compromised via this path before
+    # patching. This is a virtual patch: it does not repair the vulnerable
+    # application logic, it only removes reachability, so it stays until every
+    # site is confirmed on a fixed release.
+    #
+    # Both routing forms must be covered -- a rule matching only the pretty
+    # permalink path leaves the ?rest_route= fallback wide open, and urlp()
+    # does not URL-decode, hence the third ACL for the %2F spelling.
+    #
+    # Anonymous-only. batch/v1 is used legitimately by the block editor for
+    # multi-entity saves, so a blanket deny would break wp-admin for real
+    # users; requiring a wordpress_logged_in_* cookie costs them nothing.
+    # req.cook() needs an exact name and WordPress suffixes a per-site hash,
+    # so this substring-matches the raw Cookie header instead.
+    #
+    # Immediate deny, not tarpit -- holding connections open helps an attacker
+    # who is already scripting this. Honors the same whitelist as above.
+    acl wp_batch_path path_beg /wp-json/batch/v1
+    acl wp_batch_route urlp(rest_route) -i -m beg /batch/v1
+    acl wp_batch_route_enc query -i -m sub rest_route=%2Fbatch%2Fv1
+    acl has_wp_logged_in req.hdr(Cookie) -i -m sub wordpress_logged_in_
+    http-request deny deny_status 403 if wp_batch_path !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
+    http-request deny deny_status 403 if wp_batch_route !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
+    http-request deny deny_status 403 if wp_batch_route_enc !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
+
+    # --- WordPress admin edge gate ---
+    # Measured on whp02, 2026-08-14: distributed unauthenticated GETs booting
+    # WordPress just to bounce back a login redirect --
+    #   2801 GET /wp-admin/profile.php   2781 GET /wp-admin/edit.php   2761 GET /wp-admin/plugins.php
+    # -- spread across many source IPs at roughly 4 req/min per IP, each hit
+    # burning a PHP-FPM/lsphp worker. Two sites absorbed 1,243 resulting 503s
+    # in nine hours as their pools saturated.
+    #
+    # IDENTITY, NOT RATE. This is the one rule in this file that tracks
+    # nothing and has no stick-table counter or threshold. Every rate-based
+    # control above (the generic limits, wp_bruteforce, xmlrpc_bruteforce) is
+    # per-IP, and per-IP rate is exactly what this attack is engineered to
+    # stay under: ~4 req/min from any single IP is indistinguishable from a
+    # slow human, and the source set is large enough that no threshold can be
+    # lowered to catch it without also catching real visitors. There is also
+    # no free stick-table slot left to try anyway -- sc0/sc1/sc2 are already
+    # used above and HAProxy's default tune.stick-counters is 3, so a fourth
+    # tracked counter is not an option here. DO NOT "simplify" this into a
+    # rate/threshold rule later: the whole point is that a threshold cannot
+    # see this traffic. Instead we gate on identity -- a real logged-in
+    # WordPress user always carries a wordpress_logged_in_* cookie (the same
+    # ACL the wp2shell block above already declares), and an unauthenticated
+    # request to a wp-admin page has no legitimate reason to boot PHP at all.
+    #
+    # 302, not 403. WordPress itself redirects an unauthenticated /wp-admin/
+    # request to wp-login.php, so replicating that at the edge means an admin
+    # whose session merely expired lands on the normal login screen instead
+    # of an error page -- we are not trading a bot problem for a support
+    # ticket. Bots get a cheap redirect they ignore.
+    #
+    # path_reg, not path_beg. A subdirectory install at /blog/wp-admin/ slips
+    # past a prefix match; path_reg with an optional leading "/" catches both
+    # root and subdirectory installs, same reasoning as the wp-login and
+    # xmlrpc path_end rules above.
+    #
+    # regsub rewrites the redirect target itself, so /blog/wp-admin/x.php
+    # redirects to /blog/wp-login.php rather than 404ing at the site root.
+    #
+    # DO NOT write regsub's regex argument with a capturing group / literal
+    # parentheses, e.g. regsub((^|/)wp-admin/.*,\1wp-login.php) -- neither
+    # inlined into the redirect's `location` nor in a standalone set-var.
+    # HAProxy 3.0.11's converter-argument parser counts parens to find the
+    # end of the regsub(...) call itself, so the *inner* "(^|/)" grouping
+    # parens are misread as closing the outer call early -- it does not
+    # matter whether the argument is quoted ("...": still fails) or the
+    # parens are backslash-escaped (\(...\): still fails). Every such form
+    # was verified against real HAProxy 3.0.11-1+deb13u3 and all produce the
+    # same ALERT: "invalid arg 2 in converter 'regsub' : missing arguments
+    # (got 1/2)". This is a converter-argument-parsing limitation, not a
+    # log-format/`%[...]` issue -- the identical failure reproduces in a
+    # plain set-var (outside any log-format string), which rules out the
+    # `location` value's log-format context as the cause.
+    #
+    # The fix sidesteps groups/backreferences entirely: HTTP paths always
+    # start with "/", so the leading "(^|/)" alternation is redundant --
+    # matching the literal substring "/wp-admin/" (both slashes, no group)
+    # is sufficient to anchor to a real path segment (a false match like
+    # "/somewp-admin/" doesn't contain "/wp-admin/" as a substring, since
+    # there's no "/" directly before "wp-admin"). No backreference is
+    # needed either: regsub only replaces the matched substring, so
+    # replacing "/wp-admin/.*" with a literal "/wp-login.php" leaves
+    # whatever precedes it (the subdirectory-install prefix, if any)
+    # untouched. Computed in its own set-var so it is a plain sample
+    # expression, not something baked into the redirect's log-format
+    # string. Behaviorally verified live against real HAProxy 3.0.11:
+    # /wp-admin/edit.php -> /wp-login.php and
+    # /blog/wp-admin/plugins.php -> /blog/wp-login.php, both with
+    # redirect_to preserved. See
+    # .superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-3b-report.md.
+    #
+    # redirect_to carries the path only (%[path,url_enc]), not the query
+    # string -- deliberate, see design spec. An admin bounced off
+    # post.php?post=123&action=edit lands back on a blank post.php rather
+    # than that exact post. Capturing the full URI needs capture.req.uri
+    # (extra config, and the captured value is length-capped) for a benefit
+    # that only matters on session expiry, so it was not worth it.
+    #
+    # THE ALLOWLIST IS MEASURED, NOT GUESSED -- taken from actual fleet
+    # traffic returning 200 on /wp-admin/*. admin-ajax.php and admin-post.php
+    # are the standard front-end AJAX/form-handler endpoints real themes and
+    # plugins call while logged out. Critically, wp-login.php loads its OWN
+    # css/js FROM /wp-admin/ (load-styles.php, load-scripts.php, and the
+    # static wp_admin_asset dirs below) -- miss those and every login page on
+    # the fleet renders unstyled with a broken password-strength meter, while
+    # wp-login.php itself still returns 200, making it a silent regression
+    # that "looks like" the gate is working.
+    #
+    # Each allowlist entry is anchored to /wp-admin/<file>, not a bare
+    # filename suffix. A bare `path_end /admin-ajax.php` also matches
+    # /wp-admin/evil/admin-ajax.php -- which ALSO matches wp_admin_path
+    # (path_reg only requires /wp-admin/ to appear somewhere), so an
+    # attacker-inserted path segment would sail through this allowlist
+    # ungated and boot full WordPress, exactly the resource exhaustion this
+    # gate exists to stop. Anchoring still covers subdirectory installs via
+    # suffix matching (/blog/wp-admin/admin-ajax.php ends with
+    # /wp-admin/admin-ajax.php) while rejecting an inserted directory.
+    #
+    # install.php is DELIBERATELY NOT allowlisted. It is legitimately
+    # reachable without a cookie during a fresh install, but it is also a
+    # standing scanner target and a real takeover vector on a site that was
+    # half-installed and then abandoned. Anyone genuinely installing uses the
+    # per-site exempt-list opt-out below instead.
+    #
+    # Honors the same whitelist as every other rule in this frontend
+    # (RFC1918 / trusted_ips.list / trusted_ips.map), and a per-site opt-out
+    # via /etc/haproxy/wpadmin_gate_exempt.list (operator-managed, seeded
+    # empty by start-up.sh) for sites where a plugin legitimately serves
+    # unauthenticated visitors from a /wp-admin/ URL outside this allowlist.
+    # EVERY ACL BELOW MATCHES THE NORMALISED PATH. The normalize-uri chain at
+    # the top of this frontend has already merged duplicate slashes, resolved
+    # "." / ".." segments (including percent-encoded ones) and decoded
+    # unreserved escapes by the time these run, so these patterns only have to
+    # describe the ONE canonical spelling the backend will resolve -- they do
+    # not have to anticipate every encoding of it. That is the whole point of
+    # the normalisation block; do not "harden" these regexes by re-adding
+    # encoding variants, fix the normalisation instead.
+    #
+    # wp_admin_safe_path guards against an OPEN REDIRECT this gate would
+    # otherwise introduce. The redirect target below is built by rewriting
+    # `path` with regsub -- regsub only replaces the matched substring, so
+    # everything BEFORE the matched "/wp-admin/" survives untouched in the
+    # output. `path` is not guaranteed to be a clean site-relative string;
+    # three concrete requests turn that survival into an off-site
+    # `Location:` header:
+    #   //evil.example.com/wp-admin/x.php        -> //evil.example.com/wp-login.php
+    #     (protocol-relative -- browsers resolve "//host/path" to
+    #     "https://host/path", so this redirects off-site with no scheme
+    #     needed). NOW NEUTRALISED UPSTREAM: path-merge-slashes rewrites this
+    #     to /evil.example.com/wp-admin/x.php before any ACL sees it, so the
+    #     Location becomes the same-origin /evil.example.com/wp-login.php.
+    #     Verified live.
+    #   /\evil.example.com/wp-admin/x.php        -> /\evil.example.com/wp-login.php
+    #     (browsers normalise a leading "/\" the same as "//"). STILL LIVE
+    #     after normalisation -- a backslash is not a slash, so no normalizer
+    #     touches it. This ACL is the only thing that stops it.
+    #   https://evil.example.com/wp-admin/x.php  -> https://evil.example.com/wp-login.php
+    #     (RFC 7230 absolute-form request targets can make HAProxy's `path`
+    #     fetch return a full URI, not just the path component). HAProxy's own
+    #     H1 parser answers 400 on this frontend; this ACL is the backstop.
+    # So wp_admin_safe_path is NOT redundant with the normalisation and must
+    # not be deleted as such -- one of its three vectors survives normalisation
+    # untouched.
+    #
+    # It is used TWO ways, and the pair matters:
+    #   - as a POSITIVE condition on the redirect, so a pathological path can
+    #     never produce a `Location:` header at all; and
+    #   - as an explicit deny, so such a path is not merely un-redirected.
+    # The deny is what closes the failure mode the positive-condition form
+    # introduced on its own: "not redirected" used to mean "falls through to
+    # the backend UNGATED", i.e. the exact PHP-booting request this gate
+    # exists to stop, reachable by prefixing "//" (that specific spelling is
+    # now normalised away, but "/\" is not). Post-normalisation the only
+    # paths that reach the deny are "/\..." ones, which cannot resolve to a
+    # real file on any tier, so nothing legitimate is denied.
+    # -i (case-insensitive) and the matching ",i" flag on the regsub below are
+    # a PAIR -- adding one without the other produces an infinite redirect
+    # loop, because a case-sensitive regsub finds no "/wp-admin/" in
+    # "/WP-ADMIN/plugins.php", returns `path` UNCHANGED, and the Location then
+    # points at the request's own URL. Verified live that the pair is correct:
+    # /WP-ADMIN/plugins.php -> /wp-login.php and /blog/WP-Admin/plugins.php ->
+    # /blog/wp-login.php.
+    #
+    # On this fleet's Linux backends /WP-ADMIN/plugins.php 404s without booting
+    # PHP, so this is hardening rather than a live-bypass fix; it matters if a
+    # docroot ever sits on a case-insensitive mount, where that same request
+    # WOULD boot PHP. The cost is that a site with a real directory literally
+    # named e.g. /docs/WP-Admin/ now gets gated -- the same false positive the
+    # lowercase pattern already has, which is what the per-site exempt list
+    # exists to resolve.
+    acl wp_admin_path      path_reg -i (^|/)wp-admin/
+    # Four literal backslashes here is NOT a typo. HAProxy's config-line word
+    # parser treats backslash as its OWN escape character before the value
+    # ever reaches the regex engine: "\\" (two backslashes) in the config
+    # collapses to one literal backslash by the time PCRE compiles it, which
+    # leaves an unterminated character class ("[^/\]") and fails with
+    # "missing terminating ] for character class" -- verified against real
+    # HAProxy 3.0.11. Four backslashes ("\\\\") collapse to two ("\\"),
+    # which PCRE then reads as a single escaped-backslash class member --
+    # the intended "reject a literal backslash" semantics.
+    acl wp_admin_safe_path path_reg ^/[^/\\\\]
+    acl wp_admin_allowed   path_end /wp-admin/admin-ajax.php /wp-admin/admin-post.php /wp-admin/load-styles.php /wp-admin/load-scripts.php
+    # The (?!.*\.php) lookahead is DEFENCE IN DEPTH, not the primary fix. This
+    # ACL grants an un-gated bypass to everything under wp-admin/css|js|images,
+    # and it used to anchor its prefix but not its suffix, so
+    # /wp-admin/css/../plugins.php took the bypass and the backend then
+    # resolved ".." and booted plugins.php. path-strip-dotdot now rewrites that
+    # to /wp-admin/plugins.php before this ACL runs, which is the real fix; the
+    # lookahead additionally makes the bypass structurally incapable of
+    # covering a PHP entrypoint even if a future encoding trick survives
+    # normalisation. It excludes ".php" ONLY -- no static asset contains that
+    # substring, so it cannot cause the silent "login page renders unstyled"
+    # regression that an extension allowlist would risk. Requires PCRE2, which
+    # both the Debian (deployed) and Alpine haproxy builds have (+PCRE2).
+    #
+    # THE "--" IS LOAD-BEARING, DO NOT DELETE IT. HAProxy warns on any pattern
+    # whose first character is "(", because it cannot tell an intended regex
+    # from a fetch-argument list someone typo'd a space into:
+    #   parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
+    #   '(^|/)wp-admin/...' is likely a mistake and probably not what you want.
+    #   Maybe you need to remove the extraneous space before '('.
+    # "--" is HAProxy's documented end-of-flags marker and is the fix it names
+    # itself ("please insert '--' between the match and the pattern"). It changes
+    # no matching semantics -- verified live on whp02: /wp-admin/css/login.min.css
+    # still passes and /wp-admin/css/x.php is still gated, before and after.
+    # Left unsilenced this fires on EVERY config load and reload on every host,
+    # where it trains operators to skim past warnings and can bury a real one.
+    # wp_admin_path escapes the warning only because its "-i" flag happens to
+    # consume the flag slot first; it is not otherwise special.
+    acl wp_admin_asset     path_reg -- (^|/)wp-admin/(css|js|images)/(?!.*\.php).*$
+    acl wp_gate_exempt     hdr(host),lower -f /etc/haproxy/wpadmin_gate_exempt.list
+    # ENCODED SEPARATOR. percent-decode-unreserved deliberately does NOT decode
+    # %2F -- "/" is a reserved character, and decoding it in the normalizer
+    # would change the path's structure (it would invent new segments), which
+    # is precisely why HAProxy refuses to. But OpenLiteSpeed DOES decode it and
+    # then serves the file: /wp-admin%2Fplugins.php was measured returning 302
+    # from a real WordPress site on the OLS tier, i.e. full PHP boot, while
+    # matching none of the ACLs above. Apache returns 404 for the same request
+    # (AllowEncodedSlashes Off), so this is an OLS-tier defect -- and OLS is the
+    # tier currently saturating.
+    #
+    # DENY, not "treat it as a wp-admin path and redirect". Two reasons:
+    #   1. The redirect target is computed by regsub(/wp-admin/.*) which finds
+    #      no "/wp-admin/" in "/wp-admin%2Fplugins.php", so `path` would come
+    #      back UNCHANGED and the Location would point at the request's own
+    #      URL -- an infinite redirect loop, not a gate.
+    #   2. Nothing legitimate emits it. A path segment cannot contain a literal
+    #      "/", so %2F inside a path is always either a probe or a proxy-
+    #      confusion attempt, and the Apache tier has been 404ing it all along,
+    #      so no site on the fleet can depend on it.
+    #
+    # SCOPED to paths that mention wp-admin, not all paths. A blanket "deny any
+    # %2F in any path" would also hit REST/API-style routes on non-WordPress
+    # customer apps that legitimately pass an encoded slash inside a path
+    # parameter. Scoping keeps the blast radius inside the attack surface this
+    # gate owns.
+    #
+    # Matching is on the SUBSTRING, not an anchored pattern, on purpose:
+    # /blog%2Fwp-admin/plugins.php hides the separator BEFORE "wp-admin", where
+    # an anchored (^|/)wp-admin/ never matches, and OLS still resolves it to
+    # /blog/wp-admin/plugins.php. Substring matching catches the separator
+    # wherever it is. percent-to-uppercase has already folded %2f into %2F;
+    # the -i is belt and braces so this rule stands on its own if the
+    # normalizer is ever reordered.
+    #
+    # %5C (encoded backslash) is denied on the same terms. On this fleet's
+    # Linux backends a backslash is an ordinary filename character, so
+    # /wp-admin%5Cplugins.php 404s rather than booting PHP -- measured, it is
+    # not a live bypass today. It is included because it is the same
+    # encoded-separator trick against a backend that happens to treat "\" as
+    # one, it costs nothing, and no legitimate path contains it.
+    acl wp_admin_word         path -i -m sub wp-admin
+    acl path_has_encoded_sep  path -i -m sub %2f %5c
+    http-request deny deny_status 403 if wp_admin_word path_has_encoded_sep !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+    http-request deny deny_status 403 if wp_admin_path !wp_admin_safe_path !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+    http-request set-var(txn.wp_login_url) path,regsub(/wp-admin/.*,/wp-login.php,i) if wp_admin_path
+    http-request redirect code 302 location %[var(txn.wp_login_url)]?redirect_to=%[path,url_enc] if wp_admin_path wp_admin_safe_path !wp_admin_allowed !wp_admin_asset !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
 
     # IP blocking using map file (manual blocks only)
     # Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
-    # Runtime updates: echo "add map #0 IP_ADDRESS 1" | socat stdio /var/run/haproxy.sock
+    # Runtime updates (worker command, map referenced by PATH -- "#<id>" ids
+    # move on every config regeneration and "#0" silently adds nothing):
+    #   echo "@1 add map /etc/haproxy/blocked_ips.map IP_ADDRESS 1" | socat stdio /tmp/haproxy-cli
     # Checks the real client IP (from headers if present, otherwise src)
     # map_ip() converter supports both single IPs and CIDR ranges (e.g., 192.168.1.0/24)
     acl is_blocked_ip var(txn.real_ip),map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0
     http-request set-path /blocked-ip if is_blocked_ip
     use_backend default-backend if is_blocked_ip
+{%- if suspension_enabled %}
+
+    # Site suspension routing. Any Host header listed in
+    # /etc/haproxy/suspended_domains.list is rewritten to /suspended and
+    # routed through default-backend, which is the same Flask app that
+    # serves the default page and blocked-ip page (port 8080 inside this
+    # container). The `/suspended` route returns HTTP 503 with a static
+    # suspension page. External tooling (e.g. WHP's site_disable.php)
+    # maintains the list file via `docker cp`. An empty list is safe —
+    # the ACL simply doesn't match. Sits after IP-blocking so 429/403
+    # still trigger first.
+    acl is_suspended_domain hdr(host),lower -f /etc/haproxy/suspended_domains.list
+    http-request set-path /suspended if is_suspended_domain
+    use_backend default-backend if is_suspended_domain
+{%- endif %}
+{%- if coraza_spoe_backend %}
+
+    # Coraza WAF inspection via SPOE. Runs AFTER rate-limit and IP-block
+    # guards (no point asking the WAF about requests we're already dropping)
+    # and AFTER the real-client-IP resolution (so Coraza sees the right src).
+    filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
+    http-request send-spoe-group coraza coraza-req
+
+    # Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to
+    # "deny" / "drop" / "redirect" when a rule with the corresponding
+    # disruptive action fires (depends on SecRuleEngine mode + per-rule
+    # ctl:ruleEngine overrides). Without these rules, Coraza would inspect
+    # but never block.
+    #
+    # On request-phase deny we return a rendered HTML page that surfaces the
+    # request reference (the unique-id) so a customer who's been blocked
+    # incorrectly can open a support ticket and quote it. lf-file expands
+    # log-format expressions inside the file at response time, so
+    # %[unique-id] / %[req.hdr(host)] / etc. get substituted live.
+    # Response-phase deny stays as a bare 403 — outbound blocks are rare in
+    # our config (Coraza response inspection is disabled by default) and
+    # an HTML body on a 403 generated mid-response could land mid-stream.
+    http-request return status 403 content-type "text/html; charset=utf-8" hdr waf-block "request" hdr x-request-reference "%[unique-id]" lf-file /haproxy/errors/403-waf.html if { var(txn.coraza.action) -m str deny }
+    http-response deny deny_status 403 hdr waf-block "response" hdr x-request-reference "%[unique-id]" if { var(txn.coraza.action) -m str deny }
+    http-request silent-drop if { var(txn.coraza.action) -m str drop }
+    http-response silent-drop if { var(txn.coraza.action) -m str drop }
+    http-request redirect code 302 location %[var(txn.coraza.data)] if { var(txn.coraza.action) -m str redirect }
+    http-response redirect code 302 location %[var(txn.coraza.data)] if { var(txn.coraza.action) -m str redirect }
+
+    # FAIL-OPEN on SPOA error. Upstream's example does the opposite — denies
+    # 500 if var(txn.coraza.error) is set — but for a hosting platform we'd
+    # rather lose WAF coverage briefly than 503 customer sites. The error
+    # variable still gets set, so monitoring can observe it.
+{%- endif %}

templates/hap_security_tables.tpl

@@ -5,4 +5,27 @@ frontend stats
     stats uri /stats
     stats refresh 30s
     stats show-legends
-    stats show-node
+    stats show-node
+
+# Dedicated stick-table for WordPress wp-login.php brute-force tracking.
+# Tracked via track-sc1 from the `web` frontend (hap_listener.tpl); counts only
+# login POSTs per real client IP over a 60s window. Separate from the generic
+# sc0 connection/rate table so the login-attempt threshold is independent of
+# the (much higher) flood thresholds.
+backend wp_bruteforce
+    stick-table type ip size 100k expire 30m store http_req_rate(60s)
+
+# Dedicated stick-table for POST /xmlrpc.php flood tracking.
+# Tracked via track-sc2 from the `web` frontend (hap_listener.tpl); counts
+# only xmlrpc POSTs per real client IP over a 60s window. This is a SEPARATE
+# table/counter from wp_bruteforce (sc1) rather than a shared one: both are
+# machine-to-machine WordPress endpoints an attacker could hit from the same
+# IP, and sharing a counter would let one endpoint's traffic inflate the
+# other's rate -- an IP credential-stuffing wp-login while also flooding
+# xmlrpc would trip the wp-login threshold early on xmlrpc volume alone (or
+# vice versa). track-sc1 (wp-login) and track-sc2 (xmlrpc) are each gated on
+# mutually exclusive path ACLs, so at most one of them ever fires per
+# request -- HAProxy's "one track-sc<N> per counter per request" limit is
+# never in play here since they're different counters anyway.
+backend xmlrpc_bruteforce
+    stick-table type ip size 100k expire 30m store http_req_rate(60s)

templates/suspended_page.html

@@ -0,0 +1,58 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <meta name="robots" content="noindex,nofollow">
+    <title>Site temporarily unavailable</title>
+    <style>
+        body {
+            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
+            text-align: center;
+            padding: 50px 20px;
+            background: linear-gradient(135deg, #1e293b 0%, #0f172a 100%);
+            margin: 0;
+            min-height: 100vh;
+            display: flex;
+            align-items: center;
+            justify-content: center;
+            color: #e2e8f0;
+        }
+        .container {
+            background: #1e293b;
+            border: 1px solid #334155;
+            padding: 40px;
+            border-radius: 12px;
+            box-shadow: 0 10px 30px rgba(0,0,0,0.4);
+            max-width: 560px;
+            width: 100%;
+        }
+        h1 {
+            color: #f1f5f9;
+            margin: 0 0 20px;
+            font-size: 1.75em;
+            font-weight: 600;
+        }
+        p {
+            color: #cbd5e1;
+            line-height: 1.7;
+            margin: 0 0 12px;
+            font-size: 1.05em;
+        }
+        .note {
+            color: #94a3b8;
+            font-size: 0.9em;
+            margin-top: 24px;
+            padding-top: 24px;
+            border-top: 1px solid #334155;
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <h1>This site is temporarily unavailable.</h1>
+        <p>The site you are trying to reach is currently offline.</p>
+        <p class="note">Site owners: please contact support to restore service.</p>
+    </div>
+</body>
+</html>

trusted_ips.list

@@ -0,0 +1,9 @@
+# Source-IP whitelist — exempt from HAProxy rate limits (one IP or CIDR per line).
+# Referenced by templates/hap_listener.tpl:
+#   acl is_trusted_ip src -f /etc/haproxy/trusted_ips.list
+#
+# Add trusted source IPs below. Do NOT commit real/personal IPs to this repo —
+# it is mirrored publicly. Keep real entries in an untracked local copy, or add
+# them directly on the server (the file lives in the /etc/haproxy named volume
+# and persists across container recreates).
+127.0.0.1

trusted_ips.map

@@ -0,0 +1,9 @@
+# Real-IP whitelist for proxy-header matching — exempt from HAProxy rate limits.
+# Format: "<IP> 1" (one per line). Referenced by templates/hap_listener.tpl:
+#   acl is_whitelisted var(txn.real_ip),map_ip(/etc/haproxy/trusted_ips.map,0) -m int gt 0
+#
+# Add trusted real IPs below. Do NOT commit real/personal IPs to this repo —
+# it is mirrored publicly. Keep real entries in an untracked local copy, or add
+# them directly on the server (the file lives in the /etc/haproxy named volume
+# and persists across container recreates).
+127.0.0.1 1

trusted_proxies.list

@@ -0,0 +1,13 @@
+# Additional trusted reverse proxies — peers permitted to set CF-Connecting-IP,
+# X-Real-IP and X-Forwarded-For. Anything NOT matched here or in
+# cloudflare_ips.list has those headers stripped before real-IP resolution.
+#
+# Referenced by templates/hap_listener.tpl.
+#
+# Leave EMPTY unless a real proxy sits in front of HAProxy on this host. Adding
+# a range here lets that peer assert any client identity, which bypasses rate
+# limits, IP blocks and the WAF for it.
+#
+# Do NOT commit real IPs — this repo is mirrored publicly. Add entries directly
+# on the server; the file lives in the /etc/haproxy named volume and persists
+# across container recreates.

wpadmin_gate_exempt.list

@@ -0,0 +1,16 @@
+# Per-site opt-out from the WordPress admin edge gate.
+#
+# Hostnames listed here are EXEMPT: unauthenticated /wp-admin/* requests for
+# these sites pass through to PHP instead of being redirected to wp-login.php.
+# One hostname per line, lowercase. Matched against the Host header.
+#
+# Referenced by templates/hap_listener.tpl:
+#   acl wp_gate_exempt hdr(host),lower -f /etc/haproxy/wpadmin_gate_exempt.list
+#
+# Add a site here when a plugin legitimately serves unauthenticated visitors
+# from a /wp-admin/ URL that is not in the rule's allowlist. Symptom: "my
+# plugin's admin page redirects to login".
+#
+# Do NOT commit real customer domains — this repo is mirrored publicly. Add
+# entries directly on the server; the file lives in the /etc/haproxy named
+# volume and persists across container recreates.