sanitize public mirror: drop personal IP and infra/customer hostnames

- trusted_ips.{list,map}: replace home IP with 127.0.0.1 + usage notes - skill: resolve deploy host from gitignored target-host.local, ask if unset (no hardcoded server FQDN); customer host in WAF test -> <live-vhost> - README / coraza README: registry FQDN in run examples -> placeholder - 403 block page: drop hardcoded support link -> contact provider support - CLAUDE.md: note whitelist files ship without real IPs - .gitignore: ignore target-host.local and *.local Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ci: mirror image pushes to ghcr.io/shadowdao (#3 )
2026-06-04 06:32:15 -07:00 · 2026-06-03 17:08:35 +00:00 · 2026-06-03 10:08:19 -07:00 · 2026-05-15 06:02:56 -07:00 · 2026-05-15 05:48:14 -07:00 · 2026-05-15 05:41:16 -07:00
32 changed files with 1742 additions and 71 deletions
@@ -0,0 +1,215 @@
+---
+name: haproxy-manager-deploy
+description: Use when shipping a haproxy-manager-base code change — editing templates, the Dockerfile, the Python manager, the coraza-spoa subdir, or static assets like errors/, then getting it onto whp01 or staging. Trigger eagerly on phrases like "deploy haproxy", "ship the haproxy change", "rebuild haproxy-manager", "update the WAF block page", "recreate haproxy-manager", or any time the next step would involve `git push` from this repo, `docker pull` on the image, or `container-manager.sh recreate`. Walks the Gitea-CI-auto-build + recreate flow, surfaces the named-volume shadowing foot-gun, and includes post-deploy verification.
+---
+
+# haproxy-manager-base commit / build / deploy
+
+This is procedural discipline for changes to `haproxy-manager-base`. The repository builds via Gitea Actions on push, not via a local build script (the WHP flow uses `build-release.sh`; this one doesn't — don't conflate them, see the `whp-deploy` skill in the whp repo for that one). Each step has caught a real foot-gun.
+
+## The pipeline at a glance
+
+```
+edit code (local)
+   └─> commit + push
+         └─> Gitea Actions auto-build (build-push.yaml / build-push-coraza.yaml)
+               ├─> publishes :latest tag to repo.anhonesthost.net
+               └─> wait for image (~2-4 min)
+                     └─> recreate container on target server
+                           └─> verify
+```
+
+Do not skip the verify step. The container can come up "healthy" while still serving stale config or missing a baked-in file (see Step 5).
+
+---
+
+## Step 0a — Resolve the target host (never hardcoded)
+
+This skill deliberately does **not** bake in a server hostname — this repo is mirrored to a public remote, so a real FQDN in the skill would leak into commits. Instead, resolve the deploy target into a `DEPLOY_HOST` shell variable that every `ssh` command below uses.
+
+```bash
+HOST_FILE=".claude/skills/haproxy-manager-deploy/target-host.local"
+DEPLOY_HOST="$(cat "$HOST_FILE" 2>/dev/null)"
+```
+
+- **If `$DEPLOY_HOST` is non-empty**, use it — that's the user's saved target. The file is gitignored, so the real hostname never lands in a commit.
+- **If it's empty**, ask the user which server this deploy targets (e.g. production vs. staging) and what its hostname or SSH alias is. Then offer to save it so future deploys don't have to ask:
+
+  ```bash
+  echo 'the-host-they-gave.example' > "$HOST_FILE"   # gitignored — safe to store the real FQDN here
+  ```
+
+Confirm `$DEPLOY_HOST` is set before running any `ssh` step:
+
+```bash
+[ -n "$DEPLOY_HOST" ] || echo "DEPLOY_HOST not set — ask the user for the target server"
+```
+
+All commands below assume the variable is set in the same shell session (`ssh root@"$DEPLOY_HOST" ...`).
+
+---
+
+## Step 0 — Confirm before pushing
+
+If the user just said "deploy" or "ship the haproxy fix", confirm what's actually changing: a template, the Python manager, the coraza-spoa subdir (separate image, separate workflow), or a static asset. Look at `git status` and `git diff` and read the diff back to the user if it's non-trivial.
+
+Anything that affects the customer-facing block path (e.g. `templates/hap_listener.tpl`, `errors/403-waf.html`) is **visible to every visitor on every site**. Authorization is per-deploy, not standing.
+
+---
+
+## Step 1 — Know which workflow your change triggers
+
+- `build-push.yaml` builds `haproxy-manager-base:latest` (the main image). Triggered by changes anywhere outside `coraza-spoa/`.
+- `build-push-coraza.yaml` builds `coraza-spoa:latest`. Triggered by changes inside `coraza-spoa/`.
+- `mirror-base-image.yaml` is a scheduled job mirroring upstream base images; unrelated to feature deploys.
+
+If you've changed both subtrees in one push, both workflows fire — note that the order they finish isn't guaranteed.
+
+---
+
+## Step 2 — Beware the `/etc/haproxy` named volume shadow
+
+If your change adds a NEW file that the running container needs (a baked-in asset, an errorfile, a new config snippet), **do not place it under `/etc/haproxy/` in the Dockerfile**. That path is a Docker named volume in deployed containers — image content only seeds the volume on first creation, so existing deployments will not see your new file even after a recreate.
+
+Safe paths for baked-in assets:
+- `/haproxy/...` (the image's WORKDIR — not volumed)
+- Anywhere outside `/etc/haproxy`, `/etc/letsencrypt`
+
+Reference the asset by absolute path from the haproxy config templates (e.g. `lf-file /haproxy/errors/403-waf.html`).
+
+See `feedback-haproxy-named-volume` memory for the full pattern.
+
+---
+
+## Step 3 — Commit + push
+
+Standard commit format with trailing `Co-Authored-By:` line. Match the recent commit message style (`git log --oneline -5`). Stage files explicitly by name.
+
+```bash
+git push origin main
+```
+
+Pushing immediately triggers the Gitea Actions build.
+
+---
+
+## Step 4 — Wait for the build
+
+The Go build inside coraza-spoa takes ~2-3 minutes; the haproxy-manager-base build is faster (~1-2 min). Don't bother polling the runs UI — just pull on the target server until the digest changes:
+
+```bash
+ssh root@"$DEPLOY_HOST" 'until docker pull -q repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest 2>&1 | tail -1 | grep -qE "Image is up to date|Status: Downloaded"; do sleep 15; done'
+```
+
+`-q` suppresses the noisy layer progress so the grep can match cleanly. If you started this command before the CI build finished, it'll loop until the new image lands; once the digest matches, it exits.
+
+To confirm you got the new image, check the image-creation time vs your push:
+
+```bash
+ssh root@"$DEPLOY_HOST" 'docker images repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base --format "{{.CreatedSince}}"'
+```
+
+It should say "X minutes ago" matching the build wait, not "yesterday".
+
+---
+
+## Step 5 — Verify the new image has what you think it has, BEFORE recreating
+
+The image is `gcr.io/distroless/static-debian12:nonroot`-based, no shell. To peek inside, run a one-shot with a sh entrypoint override (only works if you put one in the image — coraza-spoa is distroless and won't have sh; haproxy-manager-base is Python-based and does):
+
+```bash
+# haproxy-manager-base (has sh):
+ssh root@"$DEPLOY_HOST" 'docker run --rm --entrypoint sh repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest -c "ls /haproxy/errors/ && head -5 /haproxy/errors/403-waf.html"'
+
+# coraza-spoa (distroless, no sh) — use docker create + docker cp instead:
+ssh root@"$DEPLOY_HOST" 'docker create --name _peek repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest && docker cp _peek:/etc/coraza/config.yaml - | tar xO; docker rm _peek'
+```
+
+This step exists because the CI build can succeed but ship the wrong file (wrong commit pulled, build cache issue, etc.). Catching it here is one step earlier than catching it from a customer report.
+
+---
+
+## Step 6 — Recreate the container
+
+```bash
+ssh root@"$DEPLOY_HOST" '/root/whp/scripts/container-manager.sh recreate haproxy-manager'
+```
+
+For coraza-spoa changes:
+```bash
+ssh root@"$DEPLOY_HOST" '/root/whp/scripts/container-manager.sh recreate coraza-spoa'
+```
+
+`container-manager.sh recreate` does: stop, remove, docker pull (idempotent if already pulled), start with the right flags from settings.json. **It reads `/docker/whp/settings.json` for things like `coraza_waf.mode`**, so if the user has toggled mode while you were building, the recreated container reflects the current setting — not whatever it was when you started.
+
+---
+
+## Step 7 — Verify the deploy
+
+For haproxy-manager:
+
+```bash
+ssh root@"$DEPLOY_HOST" '
+echo "=== container ==="
+docker ps --filter name=haproxy-manager --format "image: {{.Image}} status: {{.Status}}"
+echo "=== healthy ==="
+docker inspect haproxy-manager --format "{{.State.Health.Status}}"
+echo "=== haproxy config valid ==="
+docker exec haproxy-manager haproxy -c -f /etc/haproxy/haproxy.cfg 2>&1 | tail -3
+echo "=== new asset reachable inside container ==="
+docker exec haproxy-manager ls -la /haproxy/errors/ 2>&1 | tail -3
+echo "=== panel health ==="
+curl -fsS -m 5 -o /dev/null -w "PANEL=%{http_code}\n" http://127.0.0.1:8000/health
+'
+```
+
+Pass criteria:
+- Container status = healthy
+- haproxy config validates (warnings OK, errors not)
+- Your new asset (if any) is at the expected path inside the running container
+- Panel returns 200
+
+If any check fails, the change still went out — diagnose immediately. Don't say "deploy complete" before this clears.
+
+---
+
+## Step 8 — End-to-end test if customer-visible
+
+If your change affects what a visitor sees (block pages, redirects, security responses), do a synthetic test that exercises the actual path. For WAF block-page changes, the recipe is:
+
+```bash
+# Inject a temporary ACL that forces the WAF deny path on a custom header,
+# fire one request, observe the rendered response, then revert + reload.
+ssh root@"$DEPLOY_HOST" '
+docker exec haproxy-manager cp /etc/haproxy/haproxy.cfg /tmp/cfg-bak
+docker exec haproxy-manager sh -c "sed -i \"/http-request send-spoe-group coraza coraza-req/a\\\\    http-request set-var(txn.coraza.action) str(deny) if { req.hdr(x-force-waf-block) -m str yes }\" /etc/haproxy/haproxy.cfg"
+docker exec haproxy-manager sh -c "echo reload | socat stdio /tmp/haproxy-cli" >/dev/null
+sleep 1
+curl -sSk -D - -H "x-force-waf-block: yes" -H "Host: <live-vhost>" "https://localhost/" | head -40
+# revert
+docker exec haproxy-manager cp /tmp/cfg-bak /etc/haproxy/haproxy.cfg
+docker exec haproxy-manager sh -c "echo reload | socat stdio /tmp/haproxy-cli" >/dev/null
+'
+```
+
+**Pick a real `<live-vhost>`.** The `Host:` header must match a domain currently served by this haproxy-manager, or the request won't route to the WAF path. Don't hardcode a customer hostname in this skill — pull a live one at test time (any entry from the panel's domain list, or `docker exec haproxy-manager ls /etc/letsencrypt/live`) and substitute it.
+
+**The injection point matters.** Insert AFTER `http-request send-spoe-group coraza coraza-req`, because the SPOE call overwrites `txn.coraza.action` based on the real Coraza verdict — if you inject before it, your override is wiped.
+
+**The reload mechanism matters.** Use `echo reload | socat stdio /tmp/haproxy-cli` — the container is python-based but doesn't have `kill` in PATH, and `docker kill --signal=HUP` signals the python manager (PID 1), not haproxy.
+
+---
+
+## Recovery hints
+
+- **`docker pull` exits "Image is up to date" but your change isn't there** — CI hasn't finished yet. Check `https://repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base/actions` for in-progress runs.
+- **Container recreates but new file is missing inside** — you put the file under `/etc/haproxy/` and the named volume shadows it. See Step 2. Move the file under `/haproxy/` (or another non-volumed path) and rebuild.
+- **HAProxy `lf-file` page renders but CSS is broken / percentages stripped** — literal `%` in the file body must be doubled (`100%%`). HAProxy log-format expansion eats single `%`. See `haproxy-lf-file-percent-escape` memory.
+- **Synthetic test returns 200 from gunicorn instead of the block page** — your test ACL is being overwritten by the SPOE call. Inject after `send-spoe-group`, not before.
+- **`docker exec haproxy-manager kill -HUP 1` fails** — the python-based container doesn't have `kill` in PATH. Use the haproxy admin socket: `echo reload | socat stdio /tmp/haproxy-cli`.
+
+---
+
+## Why this skill is rigid
+
+The pipeline is short, but the volume-shadowing trap and the SPOE-overwrite trap during testing each cost a 5-10 minute debugging detour during the session this skill was authored from. Both are silent failures — your change goes out, the container is healthy, and you only notice the bug when a customer report (or a careful synthetic test) surfaces it. The verification steps exist to catch them before that happens.
@@ -0,0 +1,54 @@
+name: Build and push coraza-spoa
+run-name: ${{ gitea.actor }} pushed a change to coraza-spoa/
+
+# Triggers only on changes to the coraza-spoa subdirectory or this workflow
+# file itself — keeps the main haproxy-manager-base build and the coraza-spoa
+# build independent. workflow_dispatch lets us trigger manually after bumping
+# the upstream coraza-spoa version pin.
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'coraza-spoa/**'
+      - '.gitea/workflows/build-push-coraza.yaml'
+  workflow_dispatch:
+
+jobs:
+  Build-and-Push:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: https://github.com/docker/setup-buildx-action@v3
+
+      - name: Login to Gitea
+        uses: docker/login-action@v3
+        with:
+          registry: repo.anhonesthost.net
+          username: ${{ secrets.CI_USER }}
+          password: ${{ secrets.CI_TOKEN }}
+
+      # Mirror to GitHub Container Registry — see build-push.yaml for the
+      # secret/username convention.
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: shadowdao
+          password: ${{ secrets.GHCR_TOKEN }}
+
+      - name: Build Image
+        uses: docker/build-push-action@v6
+        with:
+          context: ./coraza-spoa
+          platforms: linux/amd64
+          push: true
+          tags: |
+            repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest
+            ghcr.io/shadowdao/coraza-spoa:latest
@@ -25,6 +25,17 @@ jobs:
          username: ${{ secrets.CI_USER }}
          password: ${{ secrets.CI_TOKEN }}

+      # Second push target so the image is also available from GitHub Container
+      # Registry under the user's account. The PAT only needs write:packages
+      # (and read:packages if the package is private). Stored in Gitea as
+      # secrets.GHCR_TOKEN; username is the literal GitHub login.
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: shadowdao
+          password: ${{ secrets.GHCR_TOKEN }}
+
      - name: Build Image
        uses: docker/build-push-action@v6
        with:
@@ -32,3 +43,4 @@ jobs:
          push: true
          tags: |
            repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+            ghcr.io/shadowdao/haproxy-manager-base:latest
@@ -0,0 +1,65 @@
+name: Mirror base images
+run-name: weekly base-image mirror
+
+# Pulls each declared base image from upstream and re-pushes to the in-house
+# registry, so any of our images that FROM these don't depend on docker.io's
+# Cloudflare R2 blob storage being reachable. The 2026-05-12 Cloudflare
+# incident motivated this for python:3.12-slim and again for golang:1.25
+# when the coraza-spoa build hit the same blob-fetch failure.
+#
+# Adding a new mirror = add one entry to the matrix below. The destination
+# tag is always cloud-hosting-platform/<image>:<tag>, matching upstream.
+
+on:
+  schedule:
+    # Mondays 06:00 UTC — outside customer peak hours and well before the
+    # typical Tuesday/Thursday push cycles. workflow_dispatch lets us trigger
+    # manually from the Gitea UI when upstream publishes patches.
+    - cron: '0 6 * * 1'
+  workflow_dispatch:
+
+jobs:
+  Mirror-Base:
+    runs-on: ubuntu-latest
+    strategy:
+      # fail-fast=false so one image's upstream being down doesn't block the
+      # others from refreshing.
+      fail-fast: false
+      matrix:
+        image:
+          - { src: 'docker.io/library/python:3.12-slim', dst_path: 'cloud-hosting-platform/python', tag: '3.12-slim' }
+          - { src: 'docker.io/library/golang:1.25',      dst_path: 'cloud-hosting-platform/golang', tag: '1.25' }
+
+    steps:
+      - name: Login to in-house registry
+        uses: docker/login-action@v3
+        with:
+          registry: repo.anhonesthost.net
+          username: ${{ secrets.CI_USER }}
+          password: ${{ secrets.CI_TOKEN }}
+
+      - name: Pull, retag, push ${{ matrix.image.src }}
+        run: |
+          set -euo pipefail
+          SRC="${{ matrix.image.src }}"
+          DST="repo.anhonesthost.net/${{ matrix.image.dst_path }}:${{ matrix.image.tag }}"
+
+          echo "::group::Pulling ${SRC}"
+          docker pull "${SRC}"
+          echo "::endgroup::"
+
+          # Capture the upstream digest so the workflow log shows what we
+          # actually pushed. Helps diagnose "did the mirror really update"
+          # questions later.
+          SRC_DIGEST=$(docker image inspect "${SRC}" -f '{{index .RepoDigests 0}}')
+          echo "upstream digest: ${SRC_DIGEST}"
+
+          docker tag "${SRC}" "${DST}"
+
+          echo "::group::Pushing ${DST}"
+          docker push "${DST}"
+          echo "::endgroup::"
+
+          # Sanity: the in-house tag should now resolve to the same content.
+          DST_DIGEST=$(docker image inspect "${DST}" -f '{{index .RepoDigests 0}}')
+          echo "mirror digest:   ${DST_DIGEST}"
@@ -37,3 +37,7 @@ ENV/
 # OS
 .DS_Store
 Thumbs.db
+
+# Local-only deploy config (never commit real hostnames)
+.claude/skills/haproxy-manager-deploy/target-host.local
+*.local
@@ -39,10 +39,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
   - `backend_servers` - Individual servers within backend groups

 3. **Template System** - Jinja2 templates for HAProxy configuration generation:
-   - `hap_header.tpl` - Global HAProxy settings and defaults
+   - `hap_header.tpl` - Global HAProxy settings, defaults, and HTTP/2 tuning
   - `hap_backend.tpl` - Backend server definitions
-   - `hap_listener.tpl` - Frontend listener configurations
+   - `hap_listener.tpl` - Frontend listener configurations with rate limiting
   - `hap_letsencrypt.tpl` - SSL certificate configurations
+   - `hap_security_tables.tpl` - Stats frontend and security stick tables
   - Template override support for custom backend configurations

 4. **Certificate Management** - Automated SSL certificate handling:
@@ -73,10 +74,51 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - Certificate private keys combined with certificates in HAProxy-compatible format
 - Default backend page for unmatched domains instead of exposing HAProxy errors

+### Rate Limiting & Connection Limits (hap_listener.tpl)
+
+- **Stick table**: `type ip size 200k expire 10m` tracking `conn_cur`, `conn_rate(10s)`, `http_req_rate(10s)`, `http_err_rate(30s)`
+- Tracks real client IP via `var(txn.real_ip)` to work correctly behind Cloudflare/proxies
+- **Rate limit thresholds**:
+  - Tarpit at 3000 req/10s (300 req/s)
+  - Hard block (deny) at 5000 req/10s (500 req/s)
+  - Connection rate limit: 500/10s
+  - Concurrent connection limit: 500
+  - Error rate limit: 100/30s
+- **Whitelist bypasses** (exempt from rate limits):
+  - `is_local` — RFC1918 private address ranges
+  - `is_trusted_ip` — source IPs listed in `trusted_ips.list`
+  - `is_whitelisted` — real IPs (from proxy headers) matched in `trusted_ips.map`
+
+### Trusted IP Whitelist Files
+
+- `trusted_ips.list` — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
+- `trusted_ips.map` — Real IP whitelist for proxy-header matching (format: `<IP> 1`)
+- Both files are baked into the Docker image via `COPY` in the Dockerfile
+- Ship as comment-only templates (no real IPs). Add trusted IPs locally and do **not** commit them — this repo is mirrored publicly. Entries persist in the `/etc/haproxy` named volume across recreates
+
+### Timeout Hardening (hap_header.tpl)
+
+- `timeout http-request`: 300s -> 30s (slowloris protection)
+- `timeout connect`: 120s -> 10s
+- `timeout client`: 10m -> 5m
+- `timeout http-keep-alive`: 120s -> 30s
+
+### HTTP/2 Protection (hap_header.tpl)
+
+- `tune.h2.fe.max-total-streams 2000` — limits total streams per HTTP/2 connection
+- `tune.h2.fe.glitches-threshold 50` — CVE-2023-44487 Rapid Reset protection
+
+### Stats Frontend (hap_security_tables.tpl)
+
+- HAProxy stats page bound to `127.0.0.1:8404` (localhost only, accessible inside container)
+- Template: `templates/hap_security_tables.tpl`
+
 ### Deployment Context

 - Designed to run as Docker container with persistent volumes for certificates and configurations
 - Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
+- Stats page on port 8404 (localhost only inside container)
 - Management interface on port 8000 should be firewall-protected in production
+- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s`
 - Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
 - HAProxy is version 3.0.11
@@ -1,10 +1,27 @@
-FROM python:3.12-slim
+# Base image mirrored into the in-house registry to remove docker.io
+# (Cloudflare R2) as a single point of failure for CI builds. The 2026-05-12
+# Cloudflare incident took down docker.io blob pulls and broke this image's CI.
+# Refresh procedure (run on a workstation that can reach docker.io, e.g.
+# monthly or when Python patches drop):
+#     docker pull docker.io/library/python:3.12-slim
+#     docker tag  docker.io/library/python:3.12-slim \
+#                 repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
+#     docker push repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
+# Future improvement: a scheduled Gitea Action that does the above automatically.
+FROM repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim
 RUN apt update -y && apt dist-upgrade -y && apt install socat haproxy cron certbot curl jq net-tools -y && apt clean && rm -rf /var/lib/apt/lists/*
 WORKDIR /haproxy
 COPY ./templates /haproxy/templates
 COPY requirements.txt /haproxy/
 COPY haproxy_manager.py /haproxy/
 COPY scripts /haproxy/scripts
+COPY trusted_ips.list /etc/haproxy/trusted_ips.list
+COPY trusted_ips.map /etc/haproxy/trusted_ips.map
+# /etc/haproxy is a named volume in deployed containers, so baked-in files
+# under that path get shadowed by the volume on existing deployments.
+# Place errorfiles outside the volumed path; the HAProxy config references
+# them by absolute path.
+COPY errors /haproxy/errors
 RUN chmod +x /haproxy/scripts/*
 RUN pip install -r requirements.txt
 # Create log directories
@@ -6,10 +6,10 @@ A Flask-based API service for managing HAProxy configurations with dynamic SSL c
 To run the container:
 ```bash
 # Without API key authentication (default)
-docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

 # With API key authentication (recommended for production)
-docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+docker run -d -p 80:80 -p 443:443 -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 ```

 ## Features
@@ -402,7 +402,7 @@ docker run -d \
  -e HAPROXY_DEFAULT_MAIN_MESSAGE="This website is currently under construction and will be available soon." \
  -e HAPROXY_DEFAULT_SECONDARY_MESSAGE="Please check back later or contact us for more information." \
  --name haproxy-manager \
-  repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
 ```

 ## Example Usage
@@ -416,7 +416,7 @@ docker run -d \
  -v haproxy:/etc/haproxy \
  -e HAPROXY_API_KEY=your-secure-api-key-here \
  --name haproxy-manager \
-  repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
+  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

 # Add a domain
 curl -X POST http://localhost:8000/api/domain \
@@ -0,0 +1,69 @@
+# Coraza-SPOA sidecar for haproxy-manager.
+#
+# Layout: built from upstream source. main.go is at the repo root; CRS rules
+# are bundled into the binary at build time (referenced as @owasp_crs/), so
+# the CRS version is whatever ships with the pinned coraza-spoa tag.
+#
+# Pin: review the upstream CHANGELOG (https://github.com/corazawaf/coraza-spoa/releases)
+# before bumping. New tags can ship newer CRS, which can introduce new rules
+# whose IDs fall into the "enforce day-one" ranges in overrides.conf — verify
+# those are still high-confidence before promoting a new tag to prod.
+
+ARG CORAZA_SPOA_VERSION=v0.7.1
+
+# golang:1.25 from the in-house mirror. The 2026-05-12 Cloudflare incident
+# took out docker.io blob pulls TWICE in one day (first for python:3.12-slim,
+# then for this image's golang:1.25), so both are mirrored at
+# repo.anhonesthost.net via the .gitea/workflows/mirror-base-image.yaml
+# weekly job.
+FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS build
+ARG CORAZA_SPOA_VERSION
+WORKDIR /src
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git \
+    && rm -rf /var/lib/apt/lists/*
+RUN git clone --depth 1 --branch "${CORAZA_SPOA_VERSION}" \
+        https://github.com/corazawaf/coraza-spoa.git . \
+    && go mod download \
+    && CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa .
+
+# Catalog extractor: walks the bundled CRS at build time and emits
+# rules-catalog.json so WHP's UI can render rule metadata without parsing
+# .conf files at runtime. Uses the SAME coraza-coreruleset version pin as
+# the coraza-spoa binary above (drift between the two would mislabel rules).
+FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog
+WORKDIR /src
+COPY catalog-extractor/ .
+RUN go build -trimpath -o /out/catalog-extractor . \
+    && /out/catalog-extractor > /out/rules-catalog.json
+
+# Distroless runtime: no shell, no package manager, no /tmp by default —
+# smallest attack surface for an exposed service. Audit log directory is
+# bind-mounted; coraza-spoa writes to it via direct file I/O (no shell needed).
+FROM gcr.io/distroless/static-debian12:nonroot
+
+LABEL org.opencontainers.image.title="coraza-spoa-whp" \
+      org.opencontainers.image.description="Coraza WAF SPOA agent configured for WHP haproxy-manager integration" \
+      org.opencontainers.image.source="https://repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base"
+
+COPY --from=build /out/coraza-spoa /coraza-spoa
+COPY config.yaml /etc/coraza-spoa/config.yaml
+COPY overrides.conf /etc/coraza/overrides.conf
+COPY pre-overrides.conf /etc/coraza/pre-overrides.conf
+COPY local-overrides.conf /etc/coraza/local-overrides.conf
+COPY host-exceptions/ /etc/coraza/host-exceptions/
+COPY --from=catalog /out/rules-catalog.json /etc/coraza/rules-catalog.json
+
+# Audit log directory — bind-mount /var/log/coraza:/var/log/coraza from host
+# so logs persist across container restarts and AI Monitor can tail them.
+# Distroless nonroot user has UID 65532; the host directory must be writable
+# by that UID (install script will chown it appropriately).
+VOLUME ["/var/log/coraza"]
+
+# SPOE TCP port — bound on 0.0.0.0:9000 inside the container. The host-side
+# port mapping is controlled by `docker run -p` (typically not exposed beyond
+# the internal docker network, since haproxy-manager reaches it by container
+# name on client-net).
+EXPOSE 9000
+
+ENTRYPOINT ["/coraza-spoa", "--config", "/etc/coraza-spoa/config.yaml"]
@@ -0,0 +1,78 @@
+# coraza-spoa sidecar
+
+A sidecar container that runs [Coraza-SPOA](https://github.com/corazawaf/coraza-spoa) as a WAF engine for `haproxy-manager`. HAProxy consults it per-request via the SPOE/SPOP protocol; Coraza evaluates the request against OWASP CRS rules and tells HAProxy whether to allow or block.
+
+## Design constraints
+
+- **`haproxy-manager` does NOT depend on this sidecar.** The base image works standalone (used in other projects and home networks) without WAF. SPOE config in the generated `haproxy.cfg` is opt-in via an env var on `haproxy-manager`.
+- **Fail-open when the sidecar is unhealthy.** `option set-on-error continue` in the HAProxy SPOE config means request flow continues uninspected if coraza-spoa is unreachable, rather than 503-ing customer traffic.
+- **Detect-only globally; enforce explicitly.** See `overrides.conf` for the day-one enforce list. Most CRS rules log without blocking until we've tuned per-customer false positives.
+
+## Deployment shape
+
+Two containers per host, both on the `client-net` docker network:
+
+```
+haproxy-manager        (existing) — ports 80, 443, 8000
+   │ SPOE TCP/9000 → reach coraza-spoa by container DNS
+   ▼
+coraza-spoa            (this image)
+   port 9000 (SPOE) — NOT exposed on host; internal network only
+   /var/log/coraza    — bind-mounted to host for AI Monitor consumption
+```
+
+Typical `docker run`:
+
+```bash
+mkdir -p /var/log/coraza
+chown 65532:65532 /var/log/coraza   # distroless nonroot UID
+
+docker run -d \
+    --name coraza-spoa \
+    --network client-net \
+    --restart unless-stopped \
+    -v /var/log/coraza:/var/log/coraza \
+    your-registry.example.com/cloud-hosting-platform/coraza-spoa:latest
+```
+
+Then on the `haproxy-manager` container, add the env var:
+
+```
+-e HAPROXY_CORAZA_SPOE_BACKEND=coraza-spoa:9000
+```
+
+The haproxy-manager template engine sees the env var and renders the SPOE config block pointing at this sidecar. Without the env var, no SPOE blocks render — the haproxy-manager image's behavior is unchanged.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `Dockerfile` | Multi-stage build (golang:1.25 → distroless), pinned to upstream coraza-spoa tag |
+| `config.yaml` | SPOA listener config + one named application `haproxy` |
+| `overrides.conf` | Day-one enforce list (`ctl:ruleEngine=On` for high-confidence rule IDs) |
+| `README.md` | This file |
+
+## Audit log
+
+`/var/log/coraza/audit.log` — JSON, one event per line, RelevantOnly (only requests that triggered ≥1 rule are logged). AI Monitor should be configured to tail this on each host.
+
+Entries include rule IDs, matched patterns, request metadata, and action taken (`log` for detect-only, `deny` for enforced). Use the JSON `action` field to filter blocked vs. observed.
+
+## Upgrading the pin
+
+CRS rules are bundled into the coraza-spoa binary at build time, so the CRS version is whatever ships with the pinned coraza-spoa tag. To upgrade:
+
+1. Check upstream releases: <https://github.com/corazawaf/coraza-spoa/releases>
+2. Skim the CHANGELOG for new/changed rules in the `overrides.conf` ID ranges.
+3. Bump `ARG CORAZA_SPOA_VERSION` in the Dockerfile.
+4. Push to `main` — the Gitea workflow at `.gitea/workflows/build-push-coraza.yaml` rebuilds + pushes `:latest`.
+5. On each host, run `container-manager.sh recreate coraza-spoa` to pull the new image.
+
+## Tuning false positives
+
+When a legitimate request triggers a blocked rule, the audit log shows the rule ID. Two ways to silence it:
+
+1. **Per-rule exception** in `overrides.conf`: `SecRuleRemoveById <id>` (full disable) or `SecRuleRemoveTargetById <id> "<target>"` (targeted exception).
+2. **Drop from the enforce list**: remove the rule's ID range from the `ctl:ruleEngine=On` overrides; it falls back to detect-only.
+
+After tuning, push the change — CI rebuilds, then `recreate coraza-spoa` on each host to apply.
@@ -0,0 +1,7 @@
+module catalog-extractor
+
+go 1.23
+
+require github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
+
+require github.com/magefile/mage v1.17.0 // indirect
@@ -0,0 +1,12 @@
+github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 h1:tqFO1lfVpTiyWtlN618OXpZMfw+nnN0Q4///W5W+/HM=
+github.com/corazawaf/coraza-coreruleset/v4 v4.25.0/go.mod h1:nRuGXITxOPvsLF2VxaTB7pYok8QB8BitX3ZenXcUryY=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM=
+github.com/magefile/mage v1.17.0/go.mod h1:Yj51kqllmsgFpvvSzgrZPK9WtluG3kUhFaBUVLo4feA=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
@@ -0,0 +1,80 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/fs"
+	"os"
+	"regexp"
+	"strconv"
+	"strings"
+
+	crs "github.com/corazawaf/coraza-coreruleset/v4"
+)
+
+type Rule struct {
+	ID       int      `json:"id"`
+	Msg      string   `json:"msg"`
+	Severity string   `json:"severity"`
+	Tags     []string `json:"tags"`
+	File     string   `json:"file"`
+}
+
+var (
+	idRe       = regexp.MustCompile(`(?i)\bid:'?(\d+)'?`)
+	msgRe      = regexp.MustCompile(`(?i)\bmsg:'([^']+)'`)
+	severityRe = regexp.MustCompile(`(?i)\bseverity:'?([A-Z]+)'?`)
+	tagRe      = regexp.MustCompile(`(?i)\btag:'([^']+)'`)
+)
+
+func main() {
+	out := []Rule{}
+	err := fs.WalkDir(crs.FS, ".", func(path string, d fs.DirEntry, err error) error {
+		if err != nil || d.IsDir() {
+			return err
+		}
+		if !strings.HasSuffix(path, ".conf") {
+			return nil
+		}
+		b, err := fs.ReadFile(crs.FS, path)
+		if err != nil {
+			return err
+		}
+		// Coalesce backslash-continuation lines so id/msg/etc on the same
+		// logical rule are visible to the per-line scanner.
+		text := regexp.MustCompile(`\\\s*\n\s*`).ReplaceAllString(string(b), " ")
+		for _, line := range strings.Split(text, "\n") {
+			line = strings.TrimSpace(line)
+			if !strings.HasPrefix(line, "SecRule") && !strings.HasPrefix(line, "SecAction") {
+				continue
+			}
+			m := idRe.FindStringSubmatch(line)
+			if m == nil {
+				continue
+			}
+			id, _ := strconv.Atoi(m[1])
+			r := Rule{ID: id, File: path}
+			if mm := msgRe.FindStringSubmatch(line); mm != nil {
+				r.Msg = mm[1]
+			}
+			if mm := severityRe.FindStringSubmatch(line); mm != nil {
+				r.Severity = strings.ToLower(mm[1])
+			}
+			for _, mm := range tagRe.FindAllStringSubmatch(line, -1) {
+				r.Tags = append(r.Tags, mm[1])
+			}
+			out = append(out, r)
+		}
+		return nil
+	})
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+	enc := json.NewEncoder(os.Stdout)
+	enc.SetIndent("", "  ")
+	if err := enc.Encode(out); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}
@@ -0,0 +1,72 @@
+# Coraza-SPOA configuration for WHP haproxy-manager integration.
+#
+# One named application "haproxy" — the haproxy-manager spoe template
+# references this same name in its spoe-agent block, so the SPOA knows
+# which rules to apply when HAProxy dispatches a request.
+#
+# Mode: SecRuleEngine DetectionOnly globally; overrides.conf promotes
+# specific high-confidence rule ID ranges to enforcement individually.
+# This is the safest posture for v1 — every rule logs, but only the
+# unambiguous ones (scanner UAs, RCE, LFI, webshells, Log4Shell) block.
+
+bind: 0.0.0.0:9000
+
+# Process-level logging (separate from per-request audit logging below)
+log_level: info
+log_file: /dev/stdout
+log_format: json
+
+# Fallback when the request doesn't match a named application — we only
+# have one, so it's also the default.
+default_application: haproxy
+
+applications:
+  - name: haproxy
+    directives: |
+      # CRS-bundled defaults: recommended Coraza settings + CRS setup +
+      # the rule pack itself (~16 MB of rules embedded in the binary).
+      Include @coraza.conf-recommended
+      Include @crs-setup.conf.example
+
+      # Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.
+      # Loaded BEFORE the CRS rules so per-host ctl:ruleRemoveById exemptions
+      # fire in phase:1 BEFORE the CRS rule they're trying to exempt would
+      # otherwise match. Server-wide overrides live in local-overrides.conf
+      # (loaded after CRS) instead.
+      Include /etc/coraza/pre-overrides.conf
+
+      Include @owasp_crs/*.conf
+
+      # WHP-specific overrides — day-one enforce list, plus tuning for
+      # the customer mix (WordPress, WooCommerce, Divi). Read this file
+      # to see exactly what blocks vs what's detect-only.
+      Include /etc/coraza/overrides.conf
+
+      # Runtime-managed POST-CRS overrides written by WHP UI. Empty by default.
+      Include /etc/coraza/local-overrides.conf
+
+      # Global mode: log all alerts, block only what overrides.conf
+      # explicitly promotes via ctl:ruleEngine=On.
+      SecRuleEngine DetectionOnly
+
+      # Audit log: JSON to a bind-mounted file so AI Monitor + log
+      # rotation can pick it up. RelevantOnly means we don't log every
+      # passing request, only ones that triggered at least one rule.
+      SecAuditEngine RelevantOnly
+      SecAuditLog /var/log/coraza/audit.log
+      SecAuditLogFormat JSON
+      SecAuditLogParts ABIJDEFHKZ
+
+    # HAProxy sends request-only events for v1. Response inspection adds
+    # latency on every page render with marginal additional protection
+    # for our customer mix; can be turned on later if we want it.
+    response_check: false
+
+    # Transactions cache for 60s. SPOE protocol is fire-and-forget per
+    # request, so this is just how long Coraza holds context for any
+    # multi-stage processing.
+    transaction_ttl_ms: 60000
+
+    log_level: info
+    log_file: /var/log/coraza/spoa.log
+    log_format: json
@@ -0,0 +1,3 @@
+# AUTOGENERATED by WHP — do not hand-edit.
+# Source of truth: whp.security_db coraza_rule_overrides table.
+# Empty file = no runtime overrides; baked-in overrides.conf governs.
@@ -0,0 +1,105 @@
+# WHP day-one enforce overrides for coraza-spoa.
+#
+# Global mode in config.yaml is SecRuleEngine DetectionOnly. The rule ID
+# ranges below are promoted to enforcement individually, chosen for very
+# low false-positive rate on the kinds of customer traffic seen on WHP
+# (WordPress, WooCommerce, Divi page builders).
+#
+# When bumping the upstream coraza-spoa pin (and thus the bundled CRS):
+#  1. Skim the CRS CHANGELOG for new/changed rules in these ID ranges.
+#  2. Verify they're still high-confidence before promoting the new image.
+#  3. Smoke-test in staging detect-only mode for 24h before flipping enforce.
+#
+# Per-customer false-positive tuning lives in a future per-customer
+# override mechanism; v1 is server-wide.
+
+# ---------------------------------------------------------------------------
+# 930120 — LFI: explicit traversal to sensitive system files
+# (/etc/passwd, /proc/self/, /.ssh/, /etc/shadow, /etc/group, etc.)
+# Unambiguous probe pattern; no legitimate site path leads here.
+# Note: 930xxx as a whole includes broader traversal patterns that can FP
+# on legitimate relative-path file browsers — keep those detect-only.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 930120 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 932100-932160 — RCE: Unix shell command injection
+# Patterns like `; cat /etc/passwd`, `|whoami`, backtick `\`uname\``,
+# $(...) substitution, &&/|| chaining with shell builtins.
+# Don't appear in normal POST bodies, URL params, or headers. Targeting
+# these is unambiguous attempted command execution.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 932100-932160 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 933170-933200 — PHP Webshell access patterns
+# Direct requests to known webshell paths: c99.php, r57.php, b374k.php,
+# wso.php, alfa.php, mini.php, etc. Almost universally reconnaissance
+# scanning for post-exploitation. Even legitimate WordPress installs
+# never serve these paths.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 933170-933200 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 944100-944300 — Log4Shell / JNDI injection
+# `${jndi:ldap://}`, `${jndi:rmi://}`, and obfuscated variants thereof
+# in headers, query strings, or bodies. Even our PHP/Node stack isn't
+# vulnerable, but blocking at the edge keeps logs clean and protects
+# any future Java workloads.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 920440 — URL file extension restricted by policy
+# Catches probes for backup / config / dump files: .bak, .old, .save,
+# .swp, .sql, .dist, .backup. Promoted to enforce after empirical
+# observation on whp01 (2026-05-12, first ~30 min of detect-only):
+# 124 events, all backup-file recon — `/wp-config.php.old`,
+# `/db_backup.sql`, `/.env.save`, `/releases.sql`, etc. — from a
+# single GCP-hosted scanner. Zero false positives observed; standard
+# WP/WooCommerce/Divi/HPR URLs do not end in these extensions.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# 930130 — Restricted File Access Attempt
+# Catches dotfile / VCS / config-disclosure probes: .env (and .env.local /
+# .env.bak / .env.save variants), .git/config, config.php at root or under
+# /admin /backend, etc. Distinct from 930120 (system file paths like
+# /etc/passwd); this targets application secret files.
+#
+# Promoted to enforce on the same observation pass that justified 920440:
+# 117 events split across joshuaknapp.net (136), cgdannyb.com (51),
+# onlinesupplements.net (23) — all `.env`-class disclosure probes.
+# Zero false positives observed. Notably, HPR's `/ccdn.php?filename=...`
+# audio delivery path does NOT trigger this rule — verified empirically.
+# ---------------------------------------------------------------------------
+SecRuleUpdateActionById 930130 "ctl:ruleEngine=On"
+
+# ---------------------------------------------------------------------------
+# Rule families intentionally kept at DETECT-ONLY for v1 — high FP rate
+# on customer mix. Promote individually after observation:
+#
+#   913xxx (Scanner UAs)— matches legitimate ActivityPub federation
+#                         (Mastodon's "...Bot" UA) and SiteLockSpider (a
+#                         paid customer-security service some sites use).
+#                         Observed on whp01 burn-in 2026-05-13:
+#                         20/185 hits = ~11% FP rate on HPR + greggfranklin
+#                         + suchascream. Detection adds anomaly score
+#                         either way; enforce upside is low.
+#   941xxx (XSS)        — Divi rich-text editor saves, TinyMCE submissions
+#   942xxx (SQLi)       — WP admin queries reflected in params
+#   920xxx (other)      — most 920xxx rules; 920440 specifically promoted above
+#   933150              — PHP injection FP on WooCommerce checkout
+#                         (`session_start` literal appearing in billing form data)
+#   950xxx-953xxx       — Data leakage / backup-file disclosure (mixed FP)
+# ---------------------------------------------------------------------------
+
+# ---------------------------------------------------------------------------
+# RESERVED RULE-ID RANGE: 990000000 – 990999999
+# WHP's coraza_rule_manager generates per-host-exception rules in this range
+# (rule ID = 990000000 + target_rule_id). Do NOT add new rules in this range
+# from any other source. When bumping the coraza-spoa pin, check the CRS
+# changelog for new rules with 9-digit IDs (rare but possible) and re-namespace
+# if collision risk emerges.
+# ---------------------------------------------------------------------------
@@ -0,0 +1,3 @@
+# AUTOGENERATED by WHP — do not hand-edit.
+# Source of truth: whp.security_db coraza_rule_host_exceptions table.
+# Loaded BEFORE the CRS rules. Empty file = no per-host exemptions active.
@@ -0,0 +1,109 @@
+<!DOCTYPE html>
+<!--
+  Served by HAProxy via `lf-file` on Coraza WAF deny.
+  IMPORTANT: HAProxy's lf-file expansion treats `%` as the start of a
+  log-format expression. Literal percent signs (CSS 100%, gradient stops,
+  url-encoded data, etc.) MUST be doubled as `%%` or HAProxy will silently
+  swallow them. Expressions like `%[unique-id]` / `%[req.hdr(host)]` stay
+  single-`%` — those are the substitutions we want.
+-->
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="robots" content="noindex, nofollow">
+<title>Request blocked &middot; %[req.hdr(host)]</title>
+<style>
+  *, *::before, *::after { box-sizing: border-box; }
+  html, body { margin: 0; padding: 0; height: 100%%; }
+  body {
+    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+    color: #1f2937;
+    background: linear-gradient(135deg, #f9fafb 0%%, #eef2f7 100%%);
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    padding: 24px;
+    line-height: 1.5;
+  }
+  .card {
+    background: #fff;
+    border-radius: 12px;
+    box-shadow: 0 1px 3px rgba(0,0,0,0.05), 0 12px 32px rgba(31,41,55,0.08);
+    max-width: 560px;
+    width: 100%%;
+    padding: 36px 40px;
+  }
+  .badge {
+    display: inline-block;
+    background: #fef3c7;
+    color: #92400e;
+    font-size: 12px;
+    font-weight: 600;
+    letter-spacing: 0.04em;
+    text-transform: uppercase;
+    padding: 4px 10px;
+    border-radius: 999px;
+    margin-bottom: 16px;
+  }
+  h1 { font-size: 22px; margin: 0 0 12px; color: #111827; }
+  p { margin: 0 0 14px; color: #374151; }
+  .ref {
+    background: #f3f4f6;
+    border: 1px solid #e5e7eb;
+    border-radius: 6px;
+    padding: 12px 14px;
+    margin: 20px 0;
+    font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
+    font-size: 13px;
+    color: #111827;
+    word-break: break-all;
+  }
+  .ref-label {
+    display: block;
+    color: #6b7280;
+    font-size: 11px;
+    font-weight: 600;
+    letter-spacing: 0.05em;
+    text-transform: uppercase;
+    margin-bottom: 4px;
+    font-family: inherit;
+  }
+  .owner {
+    border-top: 1px solid #e5e7eb;
+    margin-top: 24px;
+    padding-top: 20px;
+    color: #4b5563;
+    font-size: 14px;
+  }
+  .owner h2 { font-size: 14px; font-weight: 600; color: #111827; margin: 0 0 8px; }
+  a {
+    color: #1d4ed8;
+    text-decoration: none;
+    border-bottom: 1px solid transparent;
+  }
+  a:hover, a:focus { border-bottom-color: #1d4ed8; }
+  .small { font-size: 12px; color: #6b7280; margin-top: 16px; }
+</style>
+</head>
+<body>
+  <main class="card" role="main">
+    <span class="badge">Access blocked</span>
+    <h1>Your request was blocked by our security filter</h1>
+    <p>The request to <strong>%[req.hdr(host)]</strong> looked suspicious to our web application firewall and was not delivered to the site.</p>
+    <p>This is automated. No one has reviewed the request yet.</p>
+
+    <div class="ref">
+      <span class="ref-label">Request reference</span>
+      %[unique-id]
+    </div>
+
+    <div class="owner">
+      <h2>Site owner?</h2>
+      <p>If you operate this site and believe this block is incorrect, please contact your hosting provider's support team and include the request reference above. They can look up exactly which rule fired and adjust it if it's a false positive.</p>
+    </div>
+
+    <p class="small">Reference IDs expire from our active logs after 14 days, so please open a ticket promptly if you'd like this investigated.</p>
+  </main>
+</body>
+</html>
@@ -16,9 +16,56 @@ import tempfile
 import threading
 import time
 import re
+import fcntl

 app = Flask(__name__)

+# Default page server (port 8080) — served to HAProxy clients whose request hit
+# an unconfigured domain OR whose IP is blocked. Defined at module level so
+# gunicorn can import it from start-up.sh; previously this was created inside
+# the __main__ block, which prevented out-of-process WSGI servers from reaching
+# it. Routes accept ALL HTTP methods because HAProxy proxies the original
+# request verb unchanged — a POST to a blocked domain would otherwise 405,
+# which is just log noise.
+default_app = Flask('haproxy_default')
+default_app.template_folder = 'templates'
+
+_ANY_METHOD = ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'HEAD', 'OPTIONS']
+
+
+@default_app.route('/', methods=_ANY_METHOD)
+def default_page():
+    """Serve the default page for unmatched domains."""
+    return render_template(
+        'default_page.html',
+        page_title=os.environ.get('HAPROXY_DEFAULT_PAGE_TITLE', 'Site Not Configured'),
+        main_message=os.environ.get(
+            'HAPROXY_DEFAULT_MAIN_MESSAGE',
+            'This domain has not been configured yet. Please contact your '
+            'system administrator to set up this website.'
+        ),
+        secondary_message=os.environ.get(
+            'HAPROXY_DEFAULT_SECONDARY_MESSAGE',
+            'If you believe this is an error, please check the domain name '
+            'and try again.'
+        ),
+    )
+
+
+@default_app.route('/blocked-ip', methods=_ANY_METHOD)
+def blocked_ip_page():
+    """Serve the blocked IP page for blocked clients (HTTP 403)."""
+    return render_template('blocked_ip_page.html'), 403
+
+
+@default_app.route('/suspended', methods=_ANY_METHOD)
+def suspended_page():
+    """Serve the suspended-site page (HTTP 503) for hosts listed in
+    /etc/haproxy/suspended_domains.list. Routed here via the frontend
+    path-rewrite ACL when HAPROXY_SUSPENSION_ENABLED=true."""
+    return render_template('suspended_page.html'), 503
+
+
 # Configuration
 DB_FILE = '/etc/haproxy/haproxy_config.db'
 TEMPLATE_DIR = Path('templates')
@@ -137,6 +184,69 @@ def validate_ip_address(ip_string):
    except ValueError:
        return False

+# Certbot uses fasteners (fcntl-based) to serialize concurrent invocations.
+# When a previous certbot run is SIGKILLed mid-execution (container restart,
+# OOM, manual kill), the kernel releases the fcntl lock automatically — but
+# the LOCK FILE on disk persists. Subsequent runs sometimes report
+# "Another instance of Certbot is already running" anyway, blocking SSL
+# issuance until someone manually clears the files.
+#
+# Our hung-process scenario (observed 2026-05-09 during the bundling rollout):
+# certbot from a previous attempt sat in defunct state holding the lock fd.
+# Once the process eventually exited, the locks were physically removable but
+# the symptoms persisted across multiple subsequent attempts.
+#
+# This helper probes each known lock path with fcntl.LOCK_NB. If we get the
+# lock, no real process holds it and the file is stale — we delete it. If we
+# DON'T get the lock, a real certbot is running and we leave it alone (so we
+# never accidentally trigger concurrent certbot runs).
+CERTBOT_LOCK_PATHS = (
+    '/etc/letsencrypt/.certbot.lock',
+    '/var/lib/letsencrypt/.certbot.lock',
+    '/var/log/letsencrypt/.certbot.lock',
+)
+
+def clear_stale_certbot_locks():
+    """Remove stale certbot lock files. Safe to call before any ACME run.
+    Returns {'cleared': [paths...], 'held': [paths...]} for logging.
+    """
+    cleared, held = [], []
+    for path in CERTBOT_LOCK_PATHS:
+        if not os.path.exists(path):
+            continue
+        try:
+            fd = os.open(path, os.O_RDWR)
+        except FileNotFoundError:
+            continue
+        except Exception as e:
+            held.append(f'{path} (open: {e})')
+            continue
+        try:
+            fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            # A real process holds it; do not touch.
+            os.close(fd)
+            held.append(path)
+            continue
+        try:
+            # We hold the lock now. Release before unlinking so the lock
+            # state is clean if someone races us.
+            fcntl.flock(fd, fcntl.LOCK_UN)
+        except Exception:
+            pass
+        try:
+            os.close(fd)
+        except Exception:
+            pass
+        try:
+            os.remove(path)
+            cleared.append(path)
+        except FileNotFoundError:
+            cleared.append(path)
+        except Exception as e:
+            held.append(f'{path} (unlink: {e})')
+    return {'cleared': cleared, 'held': held}
+
 def find_certbot_live_dir(base_domain):
    """Find the most recent certbot live directory for a domain.
    Certbot creates -NNNN suffixed dirs for repeated requests."""
@@ -403,6 +513,9 @@ def request_ssl():
        return jsonify({'status': 'error', 'message': 'Domain is required'}), 400

    try:
+        # Defensive: clear any stale lock left by a SIGKILLed prior run.
+        clear_stale_certbot_locks()
+
        # Request Let's Encrypt certificate
        result = subprocess.run([
            'certbot', 'certonly', '-n', '--standalone',
@@ -455,11 +568,245 @@ def request_ssl():
        log_operation('request_ssl', False, str(e))
        return jsonify({'status': 'error', 'message': str(e)}), 500

+def _cleanup_superseded_lineages(keep_path, keep_lineage, bundle_names):
+    """Remove cert files + certbot lineages that the just-issued bundle supersedes.
+
+    A `.pem` in /etc/haproxy/certs/ is "superseded" iff its certificate's CN
+    is one of the bundle's names AND the file isn't the bundle's own combined
+    file. We don't look at SANs of the OLD certs — being the CN is enough,
+    since that's what HAProxy SNI-matches against and what the file
+    convention names it after.
+
+    Also drops the corresponding certbot renewal config so `certbot renew`
+    stops trying to renew the dead lineage on its next 12h cron tick.
+
+    Returns a small summary dict for logging / API response.
+    """
+    summary = {'removed': [], 'errors': [], 'skipped': []}
+
+    if not os.path.isdir(SSL_CERTS_DIR):
+        return summary
+
+    keep_basename = os.path.basename(keep_path)
+
+    for fname in sorted(os.listdir(SSL_CERTS_DIR)):
+        if not fname.endswith('.pem'):
+            continue
+        if fname == keep_basename:
+            continue
+        fpath = os.path.join(SSL_CERTS_DIR, fname)
+        try:
+            cn_proc = subprocess.run(
+                ['openssl', 'x509', '-in', fpath, '-noout', '-subject', '-nameopt', 'multiline'],
+                capture_output=True, text=True
+            )
+            if cn_proc.returncode != 0:
+                summary['skipped'].append({'file': fname, 'reason': 'openssl read failed'})
+                continue
+            # `-nameopt multiline` lays out the subject one RDN per line; CN is
+            # the row matching `commonName`. Robust against unusual subject orderings.
+            cn = None
+            for line in cn_proc.stdout.splitlines():
+                line = line.strip()
+                if line.startswith('commonName'):
+                    # format: "commonName                = example.com"
+                    parts = line.split('=', 1)
+                    if len(parts) == 2:
+                        cn = parts[1].strip()
+                    break
+            if not cn:
+                summary['skipped'].append({'file': fname, 'reason': 'no CN found'})
+                continue
+        except Exception as e:
+            summary['skipped'].append({'file': fname, 'reason': f'inspect failed: {e}'})
+            continue
+
+        if cn not in bundle_names:
+            continue  # not superseded — different domain group
+
+        # This file's CN is now part of our new bundle — supersede it.
+        lineage_name = fname[:-len('.pem')]
+        if lineage_name == keep_lineage:
+            # Defensive: shouldn't happen because of keep_basename check, but
+            # don't accidentally drop the lineage we just wrote.
+            continue
+
+        try:
+            os.remove(fpath)
+            removed_entry = {'file': fname, 'cn': cn, 'lineage_deleted': False}
+            # Best-effort certbot lineage delete. Some files may not have a
+            # corresponding lineage (e.g. self-signed dev certs); ignore those.
+            try:
+                cb_proc = subprocess.run(
+                    ['certbot', 'delete', '--cert-name', lineage_name, '-n'],
+                    capture_output=True, text=True
+                )
+                removed_entry['lineage_deleted'] = (cb_proc.returncode == 0)
+                if cb_proc.returncode != 0:
+                    removed_entry['certbot_stderr'] = (cb_proc.stderr or '').strip()[:200]
+            except Exception as e:
+                removed_entry['certbot_error'] = str(e)
+            summary['removed'].append(removed_entry)
+        except Exception as e:
+            summary['errors'].append({'file': fname, 'error': str(e)})
+
+    return summary
+
+@app.route('/api/ssl/bundle', methods=['POST'])
+@require_api_key
+def request_ssl_bundle():
+    """Issue a single Let's Encrypt cert covering multiple SANs.
+
+    Used by WHP's per-site bundling: one ACME order, one combined .pem,
+    one DB row update per included name. Replaces N separate single-domain
+    /api/ssl calls when a site has multiple domains.
+
+    Body:
+      {"primary": "example.com", "sans": ["www.example.com", ...]}
+
+    The cert lineage uses --cert-name <primary>, so renewal under the same
+    name doesn't proliferate -0001/-0002 dirs (the issue we hit with the
+    legacy single-domain flow). The combined PEM is written to
+    /etc/haproxy/certs/<primary>.pem; HAProxy matches SNI against the cert's
+    SAN list, so this single file serves all included names.
+    """
+    data = request.get_json() or {}
+    primary = (data.get('primary') or '').strip()
+    sans = data.get('sans') or []
+
+    if not primary:
+        log_operation('request_ssl_bundle', False, 'primary not provided')
+        return jsonify({'status': 'error', 'message': '"primary" is required'}), 400
+
+    # Basic shape validation. certbot will hard-validate the rest.
+    domain_re = re.compile(
+        r'^(?:\*\.)?(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,}$',
+        re.IGNORECASE,
+    )
+    if not domain_re.match(primary):
+        return jsonify({'status': 'error', 'message': f'invalid primary: {primary!r}'}), 400
+
+    # Build the unique ordered name list — primary first, then de-duped SANs.
+    if not isinstance(sans, list):
+        return jsonify({'status': 'error', 'message': '"sans" must be a list'}), 400
+    cleaned_sans = []
+    for s in sans:
+        if not isinstance(s, str):
+            return jsonify({'status': 'error', 'message': f'invalid SAN entry: {s!r}'}), 400
+        s = s.strip()
+        if not s:
+            continue
+        if not domain_re.match(s):
+            return jsonify({'status': 'error', 'message': f'invalid SAN: {s!r}'}), 400
+        cleaned_sans.append(s)
+
+    seen = {primary}
+    names = [primary]
+    for s in cleaned_sans:
+        if s not in seen:
+            names.append(s)
+            seen.add(s)
+
+    # Let's Encrypt allows up to 100 names per cert.
+    if len(names) > 100:
+        return jsonify({
+            'status': 'error',
+            'message': f'Too many SANs ({len(names)}); Let\'s Encrypt limit is 100',
+        }), 400
+
+    cmd = [
+        'certbot', 'certonly', '-n', '--standalone',
+        '--preferred-challenges', 'http', '--http-01-port=8688',
+        '--cert-name', primary,
+    ]
+    for n in names:
+        cmd.extend(['-d', n])
+
+    try:
+        # Defensive: clear any stale lock left by a SIGKILLed prior run.
+        clear_stale_certbot_locks()
+
+        result = subprocess.run(cmd, capture_output=True, text=True)
+        if result.returncode != 0:
+            stderr_excerpt = (result.stderr or '').strip()[:800]
+            error_msg = f'Failed to obtain SSL bundle for {primary}: {stderr_excerpt}'
+            log_operation('request_ssl_bundle', False, error_msg)
+            return jsonify({
+                'status': 'error',
+                'message': error_msg,
+                'primary': primary,
+                'attempted_names': names,
+            }), 500
+
+        # Locate the lineage. With --cert-name primary, this should be a
+        # stable directory name (no -NNNN suffix on the first issuance).
+        live_dir = find_certbot_live_dir(primary)
+        if not live_dir:
+            error_msg = f'Bundle issued but live dir not found for {primary}'
+            log_operation('request_ssl_bundle', False, error_msg)
+            return jsonify({'status': 'error', 'message': error_msg}), 500
+
+        cert_path = os.path.join(live_dir, 'fullchain.pem')
+        key_path = os.path.join(live_dir, 'privkey.pem')
+        combined_path = f'{SSL_CERTS_DIR}/{primary}.pem'
+
+        os.makedirs(SSL_CERTS_DIR, exist_ok=True)
+        with open(combined_path, 'w') as combined:
+            subprocess.run(['cat', cert_path, key_path], stdout=combined)
+
+        # Mark every name in the bundle as ssl_enabled, all pointing at the
+        # same combined .pem. HAProxy serves one file for many SNI hostnames.
+        with sqlite3.connect(DB_FILE) as conn:
+            cursor = conn.cursor()
+            for n in names:
+                cursor.execute('''
+                    UPDATE domains
+                    SET ssl_enabled = 1, ssl_cert_path = ?
+                    WHERE domain = ?
+                ''', (combined_path, n))
+            conn.commit()
+            cursor.close()
+
+        # Clean up superseded lineages. When the bundle covers names that were
+        # previously each in their own single-SAN -0001/-0002 lineage, those
+        # older .pem files coexist in /etc/haproxy/certs/ and get loaded by the
+        # `bind ... ssl crt /etc/haproxy/certs` directive. HAProxy then picks
+        # one of them by alphabetical/load order — frequently the older
+        # single-SAN file — and the new bundle has no effect on what's served.
+        # This block deletes those superseded files (and their certbot lineage)
+        # before the generate_config() reload so HAProxy picks up the bundle.
+        cleanup_summary = _cleanup_superseded_lineages(
+            keep_path=combined_path,
+            keep_lineage=primary,
+            bundle_names=set(names),
+        )
+
+        generate_config()
+        log_operation(
+            'request_ssl_bundle', True,
+            f'SSL bundle issued for {primary} covering {len(names)} names; '
+            f'cleaned up {len(cleanup_summary["removed"])} superseded lineage(s)'
+        )
+        return jsonify({
+            'status': 'success',
+            'primary': primary,
+            'names': names,
+            'cert_path': combined_path,
+            'cleanup': cleanup_summary,
+            'message': f'Bundled certificate obtained for {len(names)} names',
+        })
+    except Exception as e:
+        log_operation('request_ssl_bundle', False, str(e))
+        return jsonify({'status': 'error', 'message': str(e)}), 500
+
@app.route('/api/certificates/renew', methods=['POST'])
@require_api_key
 def renew_certificates():
    """Renew all certificates and reload HAProxy"""
    try:
+        # Defensive: clear any stale lock left by a SIGKILLed prior run.
+        clear_stale_certbot_locks()
+
        # Run certbot renew
        result = subprocess.run([
            'certbot', 'renew', '--quiet'
@@ -1371,6 +1718,36 @@ def generate_config():
        
        config_parts = []

+        # Optional Coraza WAF integration. When HAPROXY_CORAZA_SPOE_BACKEND is
+        # set on the haproxy-manager container, we render an extra TCP backend
+        # pointing at a coraza-spoa sidecar AND inject a `filter spoe ...` line
+        # into the frontend via hap_listener.tpl. Unset (the default for
+        # standalone deployments, home networks, and any non-WHP use of this
+        # image) -> the generated haproxy.cfg is byte-identical to today's.
+        coraza_spoe_backend = os.environ.get('HAPROXY_CORAZA_SPOE_BACKEND')
+
+        # Optional site-suspension routing. When HAPROXY_SUSPENSION_ENABLED is
+        # set (any truthy value), the frontend gets an ACL that rewrites the
+        # path to /suspended and routes through default-backend for any host
+        # listed in /etc/haproxy/suspended_domains.list. The /suspended Flask
+        # route in this same process returns HTTP 503 + a static page — no
+        # separate container needed (mirrors the existing /blocked-ip pattern).
+        # Same opt-in shape as Coraza: unset -> config byte-identical to today.
+        # We just ensure the list file exists (haproxy refuses to start with
+        # `-f` pointing at a missing file).
+        suspension_raw = os.environ.get('HAPROXY_SUSPENSION_ENABLED', '').strip().lower()
+        suspension_enabled = suspension_raw in ('1', 'true', 'yes', 'on')
+        if suspension_enabled:
+            suspended_list_path = '/etc/haproxy/suspended_domains.list'
+            if not os.path.exists(suspended_list_path):
+                try:
+                    with open(suspended_list_path, 'w') as f:
+                        f.write('')
+                    os.chmod(suspended_list_path, 0o644)
+                    logger.info(f"Created empty {suspended_list_path}")
+                except Exception as e:
+                    logger.error(f"Failed to create {suspended_list_path}: {e}")
+
        # Add Haproxy Default Headers
        default_headers = template_env.get_template('hap_header.tpl').render()
        config_parts.append(default_headers)
@@ -1380,7 +1757,9 @@ def generate_config():

        # Add Listener Block
        listener_block = template_env.get_template('hap_listener.tpl').render(
-            crt_path = SSL_CERTS_DIR
+            crt_path = SSL_CERTS_DIR,
+            coraza_spoe_backend = coraza_spoe_backend,
+            suspension_enabled = suspension_enabled,
        )
        config_parts.append(listener_block)

@@ -1500,6 +1879,33 @@ backend default-backend
            config_parts.append(fallback_backend)
        # Add Backends
        config_parts.append('\n' .join(config_backends) + '\n')
+
+        # Coraza WAF backend + SPOE engine config file (only when env var set).
+        # Writing /etc/haproxy/coraza-spoe.cfg here keeps it in sync with the
+        # filter line that hap_listener.tpl just rendered into the frontend.
+        # Explicit trailing '\n' because this is now the LAST config_part —
+        # HAProxy fails parse with "Missing LF on last line" otherwise.
+        if coraza_spoe_backend:
+            coraza_backend_block = template_env.get_template(
+                'hap_coraza_spoa_backend.tpl'
+            ).render(agent_target=coraza_spoe_backend)
+            config_parts.append(coraza_backend_block + '\n')
+
+            coraza_spoe_cfg = template_env.get_template(
+                'hap_coraza_spoe_engine.tpl'
+            ).render()
+            # HAProxy also rejects this file without a trailing LF
+            # ("Missing LF on last line"). Belt-and-suspenders — even if the
+            # template ends with a newline, Jinja2 can trim it depending on
+            # how the file was authored.
+            if not coraza_spoe_cfg.endswith('\n'):
+                coraza_spoe_cfg += '\n'
+            coraza_spoe_path = '/etc/haproxy/coraza-spoe.cfg'
+            with open(coraza_spoe_path, 'w') as f:
+                f.write(coraza_spoe_cfg)
+            logger.info(f"Coraza SPOE engine config written to {coraza_spoe_path} "
+                        f"(SPOA target: {coraza_spoe_backend})")
+
        # Write complete configuration to tmp
        temp_config_path = "/etc/haproxy/haproxy.cfg"

@@ -1776,8 +2182,22 @@ def start_haproxy():
            log_operation('start_haproxy', False, error_msg)
            logger.warning("Container will continue without HAProxy running")

-if __name__ == '__main__':
+def do_initial_setup():
+    """One-time container-startup setup: DB schema, certbot account, fresh
+    self-signed cert, config generation, and HAProxy launch. Idempotent;
+    safe to re-run, but in prod it should run exactly once per container
+    instance (via scripts/init.py before gunicorn workers spawn) so that
+    start_haproxy() doesn't race with itself across forks.
+    """
    init_db()
+    # Clear any stale certbot locks left from a previous container instance
+    # that didn't shut down cleanly. Safe — only removes locks that no live
+    # process holds (verified via fcntl probe).
+    _stale = clear_stale_certbot_locks()
+    if _stale['cleared']:
+        logger.info(f"Cleared stale certbot lock(s) at startup: {_stale['cleared']}")
+    if _stale['held']:
+        logger.warning(f"certbot lock(s) actively held at startup: {_stale['held']}")
    certbot_register()
    generate_self_signed_cert(SSL_CERTS_DIR)

@@ -1792,36 +2212,19 @@ if __name__ == '__main__':
    start_haproxy()
    certbot_register()

-    # Run Flask app on port 8000 for API and port 8080 for default page
+
+if __name__ == '__main__':
+    # Direct-invocation path: `python haproxy_manager.py`. Used for local dev
+    # and as a fallback. In the container this runs only when scripts/start-up.sh
+    # is bypassed; production uses gunicorn after scripts/init.py.
+    do_initial_setup()
+
+    # Run both Flask apps on the werkzeug dev server. Acceptable for local
+    # development but NOT production — gunicorn is the prod server, invoked
+    # from scripts/start-up.sh.
    from threading import Thread
-    
-    def run_default_page_server():
-        """Run a separate Flask app on port 8080 for the default page"""
-        from flask import Flask, render_template
-        default_app = Flask(__name__)
-        default_app.template_folder = 'templates'
-        
-        @default_app.route('/')
-        def default_page():
-            """Serve the default page for unmatched domains"""
-            admin_email = os.environ.get('HAPROXY_ADMIN_EMAIL', 'admin@example.com')
-            
-            return render_template('default_page.html',
-                page_title=os.environ.get('HAPROXY_DEFAULT_PAGE_TITLE', 'Site Not Configured'),
-                main_message=os.environ.get('HAPROXY_DEFAULT_MAIN_MESSAGE', 'This domain has not been configured yet. Please contact your system administrator to set up this website.'),
-                secondary_message=os.environ.get('HAPROXY_DEFAULT_SECONDARY_MESSAGE', 'If you believe this is an error, please check the domain name and try again.')
-            )
-        
-        @default_app.route('/blocked-ip')
-        def blocked_ip_page():
-            """Serve the blocked IP page for blocked clients"""
-            return render_template('blocked_ip_page.html'), 403
-        
-        default_app.run(host='0.0.0.0', port=8080)
-    
-    # Start the default page server in a separate thread
-    default_server_thread = Thread(target=run_default_page_server, daemon=True)
-    default_server_thread.start()
-    
-    # Run the main API server
+    Thread(
+        target=lambda: default_app.run(host='0.0.0.0', port=8080),
+        daemon=True,
+    ).start()
    app.run(host='0.0.0.0', port=8000)
@@ -1,3 +1,7 @@
 Flask==2.3.3
 Jinja2==3.1.2
 psutil
+# Production WSGI server. Replaces Flask's built-in werkzeug dev server, which
+# is single-threaded and leaks workers over long uptimes (root cause of the
+# 2026-05 haproxy-manager "healthy but stalled" incidents).
+gunicorn==23.0.0
@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+"""Container init: DB schema, certbot account, config generation, HAProxy start.
+
+Runs once per container start, BEFORE gunicorn workers spawn. Keeping init out
+of the WSGI app's module-load path avoids fork-time races (multiple workers
+attempting to start_haproxy() simultaneously, certbot lock contention, etc.).
+"""
+import sys
+
+sys.path.insert(0, '/haproxy')
+import haproxy_manager  # noqa: E402  (sys.path manipulation must come first)
+
+haproxy_manager.do_initial_setup()
@@ -20,12 +20,21 @@ log_error() {

 log_info "Starting certificate renewal process"

-# Run certbot renewal
-if certbot renew --quiet --no-random-sleep-on-renew; then
-    log_info "Certbot renewal completed"
+# Run certbot renewal — don't exit on failure, some certs may have
+# renewed successfully even if others failed (e.g., domain no longer
+# pointed here). Continue to copy/combine whatever succeeded.
+CERTBOT_OUTPUT=$(certbot renew --no-random-sleep-on-renew 2>&1)
+CERTBOT_EXIT=$?
+
+if [ $CERTBOT_EXIT -eq 0 ]; then
+    log_info "Certbot renewal completed successfully"
 else
-    log_error "Certbot renewal failed with exit code $?"
-    exit 1
+    log_error "Certbot renewal had failures (exit code $CERTBOT_EXIT):"
+    # Log the specific failures
+    echo "$CERTBOT_OUTPUT" | grep -E "Failed to renew|failure" | while read -r line; do
+        log_error "  $line"
+    done
+    log_info "Continuing to process successfully renewed certificates..."
 fi

 # Copy all certificates to HAProxy format
@@ -1,6 +1,61 @@
 #!/usr/bin/env bash
+# Container entrypoint. Two-phase startup:
+#   1. One-shot init (init.py): DB schema, certbot register, config gen, start HAProxy.
+#      Runs synchronously and to completion so haproxy is up before the API binds.
+#   2. WSGI serving via gunicorn (replacing the Flask dev server). Two gunicorn
+#      instances:
+#        - port 8080 -> default_app  (default page + blocked-ip page; HAProxy
+#          proxies unmatched / blocked traffic here)
+#        - port 8000 -> app          (management API)
+#
+# Why gunicorn:
+#   Flask's built-in werkzeug "development server" is single-threaded and leaks
+#   workers under sustained load. It carried haproxy-manager for a long time but
+#   stalled out around 24-48h uptime ("healthy" health-check, but every request
+#   queued behind a stuck worker). Gunicorn with --max-requests cycles workers
+#   periodically, which prevents the slow-leak failure mode entirely.

-# Exit on error
 set -eo pipefail
+
+# Ensure trusted IP whitelist files exist (volume-mounted /etc/haproxy may shadow image defaults)
+mkdir -p /etc/haproxy
+[ -f /etc/haproxy/trusted_ips.list ] || : > /etc/haproxy/trusted_ips.list
+[ -f /etc/haproxy/trusted_ips.map ]  || : > /etc/haproxy/trusted_ips.map
+
 cron &
-python /haproxy/haproxy_manager.py
+
+# Phase 1: container init
+python /haproxy/scripts/init.py
+
+# Phase 2: WSGI servers
+# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
+# (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
+# 1000 — worker recycle frequency).
+API_WORKERS="${HAPROXY_MGR_API_WORKERS:-1}"
+API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
+MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
+MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
+
+# Default page server on :8080. Stays in the background.
+# --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
+# without forking. --max-requests recycles the worker to bound memory drift.
+gunicorn \
+    --bind 0.0.0.0:8080 \
+    --workers 1 --threads 4 --worker-class gthread \
+    --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
+    --timeout 30 \
+    --access-logfile - --error-logfile - --log-level info \
+    --pythonpath /haproxy \
+    'haproxy_manager:default_app' &
+
+# Main API server on :8000 in the foreground. exec so signals propagate
+# correctly and the container exits if the API dies (docker --restart picks it
+# up). Longer --timeout because cert issuance hits ACME and can take a while.
+exec gunicorn \
+    --bind 0.0.0.0:8000 \
+    --workers "${API_WORKERS}" --threads 4 --worker-class gthread \
+    --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
+    --timeout "${API_TIMEOUT}" \
+    --access-logfile - --error-logfile - --log-level info \
+    --pythonpath /haproxy \
+    'haproxy_manager:app'
@@ -11,7 +11,7 @@ backend {{ name }}-backend
    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }

    {% for server in servers %}
-    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
    {% endfor %}

 # SSE-specific backend - optimized for Server-Sent Events long-lived connections
@@ -36,5 +36,5 @@ backend {{ name }}-sse-backend
    http-request set-header X-Forwarded-Proto http if !{ ssl_fc }

    {% for server in servers %}
-    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }}
+    server {{ server.server_name }} {{ server.server_address }}:{{ server.server_port }} {{ server.server_options }} resolvers docker_dns init-addr last,libc,none
    {% endfor %}
@@ -0,0 +1,16 @@
+# Coraza-SPOA backend.
+# Only rendered into haproxy.cfg when HAPROXY_CORAZA_SPOE_BACKEND env var is
+# set on the haproxy-manager container. SPOE traffic to this backend is TCP,
+# not HTTP. The agent target comes from the env var so a single image can be
+# deployed against different sidecar host:port pairs (typically the sidecar
+# container's name + 9000 inside the shared docker network).
+backend coraza-spoa-backend
+    mode tcp
+    # spop-check actually speaks the SPOE protocol against the agent —
+    # confirms the agent can negotiate a session, not just that the TCP
+    # port is open. Required to detect a half-broken SPOA that's listening
+    # but not actually processing.
+    option spop-check
+    timeout connect 5s
+    timeout server  30s
+    server coraza-spoa {{ agent_target }} check
@@ -0,0 +1,60 @@
+# Coraza SPOE engine configuration.
+#
+# Written to /etc/haproxy/coraza-spoe.cfg by haproxy_manager.generate_config()
+# when HAPROXY_CORAZA_SPOE_BACKEND env var is set. Referenced from haproxy.cfg
+# via `filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg`.
+#
+# Engine name "coraza" must match the engine name in the filter line in the
+# main config; group name "coraza-req" must match the send-spoe-group action.
+# Application name "haproxy" must match the application block in coraza-spoa's
+# config.yaml.
+#
+# Reference: this config follows the shape from coraza-spoa's upstream
+# example/haproxy/coraza.cfg (v0.7.1). Arg names + ordering are required by
+# Coraza-SPOA exactly as specified — DO NOT reorder or rename without
+# coordinating with the agent.
+
+[coraza]
+
+spoe-agent coraza
+    # `groups` (not `messages`) lists the spoe-group names this engine offers
+    # via `send-spoe-group` actions. The same group name appears below in a
+    # spoe-group block, which in turn references the actual message.
+    groups      coraza-req
+
+    # Prefix for variables the agent sets back on the request transaction —
+    # e.g. var(txn.coraza.error) when set-on-error triggers.
+    option      var-prefix coraza
+
+    # On agent error/timeout, set var(txn.coraza.error). We DON'T add a
+    # corresponding `http-request deny if { var(txn.coraza.error) -m bool }`
+    # in the frontend, so the request continues uninspected. This is the
+    # fail-open posture: WAF outage shouldn't 503 customer traffic.
+    option      set-on-error error
+
+    timeout     hello       2s
+    timeout     idle        2m
+    timeout     processing  100ms
+
+    use-backend coraza-spoa-backend
+    log         global
+
+# Per-request inspection message. No `event` directive — fires only when
+# explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.
+# Arg order/names are mandatory: Coraza-SPOA parses positionally and renames
+# break the agent. `app=str(haproxy)` is the literal application name from
+# coraza-spoa's config.yaml `applications:` block.
+#
+# src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
+# top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
+# only honoring those headers from trusted proxies). Falls back to `src`
+# when no proxy headers are present. This is what shows up as `client_ip`
+# in /var/log/coraza/audit.log and the rule manager's view-matches panel.
+spoe-message coraza-req
+    args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
+
+# Group binding for send-spoe-group invocation in the frontend. One group,
+# one message; could add more in the future (e.g. coraza-res for response
+# inspection — currently disabled in coraza-spoa's config.yaml).
+spoe-group coraza-req
+    messages coraza-req
@@ -33,6 +33,24 @@ global

    # Stats persistence for zero-downtime reloads
    stats-file /var/lib/haproxy/stats.dat
+
+#---------------------------------------------------------------------
+# DNS resolver for Docker container name resolution
+# Re-resolves backend server addresses so container IP changes
+# (from restarts, recreations, scaling) are picked up automatically
+#---------------------------------------------------------------------
+resolvers docker_dns
+    nameserver dns1 127.0.0.11:53
+    resolve_retries 3
+    timeout resolve 1s
+    timeout retry 1s
+    hold valid 10s
+    hold other 10s
+    hold refused 10s
+    hold nx 10s
+    hold timeout 10s
+    hold obsolete 10s
+
 #---------------------------------------------------------------------
 # common defaults that all the 'listen' and 'backend' sections will
 # use if not designated in their block
@@ -56,3 +74,14 @@ defaults
    timeout tarpit          10s  # Tarpit delay for low-level scanners (before silent-drop)
    maxconn                 3000

+    # Per-request unique reference, used:
+    #   - in the log line (httplog includes %ID)
+    #   - echoed to clients in the X-Request-Reference response header on
+    #     WAF blocks so a customer can quote it when opening a support ticket
+    #   - embedded in /etc/haproxy/errors/403-waf.html so a blocked visitor
+    #     sees it on the rendered 403 page
+    # Support correlates ref → /var/log/haproxy.log line → timestamp+client+host
+    # → /var/log/coraza/audit.log entry → rule_id.
+    unique-id-format        %[uuid()]
+    unique-id-header        X-Request-Reference
+    
@@ -13,31 +13,41 @@ frontend web
    acl has_x_real_ip req.hdr(X-Real-IP) -m found
    acl has_x_forwarded_for req.hdr(X-Forwarded-For) -m found

-    # Set the real IP based on available headers
-    http-request set-var(txn.real_ip) req.hdr(CF-Connecting-IP) if has_cf_connecting_ip
-    http-request set-var(txn.real_ip) req.hdr(X-Real-IP) if !has_cf_connecting_ip has_x_real_ip
-    http-request set-var(txn.real_ip) req.hdr(X-Forwarded-For) if !has_cf_connecting_ip !has_x_real_ip has_x_forwarded_for
+    # Set the real IP based on available headers. Use hdr_ip (not hdr) so the
+    # variable is typed as IP — required by the Coraza SPOE arg `src-ip` which
+    # decodes binary IP bytes (passing a string IP panics the SPOA goroutine).
+    # `hdr_ip(X-Forwarded-For,1)` extracts the FIRST address from a possibly
+    # comma-separated chain (original client, not intermediate proxies).
+    http-request set-var(txn.real_ip) req.hdr_ip(CF-Connecting-IP) if has_cf_connecting_ip
+    http-request set-var(txn.real_ip) req.hdr_ip(X-Real-IP) if !has_cf_connecting_ip has_x_real_ip
+    http-request set-var(txn.real_ip) req.hdr_ip(X-Forwarded-For,1) if !has_cf_connecting_ip !has_x_real_ip has_x_forwarded_for
    http-request set-var(txn.real_ip) src if !has_cf_connecting_ip !has_x_real_ip !has_x_forwarded_for

    # --- Connection & rate tracking ---
    stick-table type ip size 200k expire 10m store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(30s)
    http-request track-sc0 var(txn.real_ip)

-    # Whitelist: let health checks and local traffic bypass rate limits
+    # Whitelist: let health checks, local, and trusted traffic bypass rate limits
    acl is_local src 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
+    acl is_trusted_ip src -f /etc/haproxy/trusted_ips.list
    acl is_health_check path_beg /.well-known/acme-challenge
+    acl is_whitelisted var(txn.real_ip),map_ip(/etc/haproxy/trusted_ips.map,0) -m int gt 0

    # --- Rate limit rules (applied in order, first match wins) ---
-    # Hard block: >500 req/10s per IP (sustained flood)
-    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 500 } !is_local !is_health_check
-    # Tarpit: >200 req/10s per IP (aggressive scraping / light flood)
-    http-request tarpit deny_status 429 if { sc_http_req_rate(0) gt 200 } !is_local !is_health_check
-    # Connection rate limit: >150 new connections per 10s per IP
-    http-request deny deny_status 429 if { sc_conn_rate(0) gt 150 } !is_local !is_health_check
-    # Concurrent connection limit: >100 simultaneous connections per IP
-    http-request deny deny_status 429 if { sc_conn_cur(0) gt 100 } !is_local !is_health_check
-    # High error rate: >20 errors in 30s (scanner/fuzzer behavior)
-    http-request tarpit deny_status 403 if { sc_http_err_rate(0) gt 20 } !is_local !is_health_check
+    # Thresholds are generous to accommodate media-heavy sites where a
+    # single page can load 100+ images/assets. These only trigger on
+    # obvious automated abuse, not real users.
+    #
+    # Hard block: >5000 req/10s per IP (500 req/s — sustained flood)
+    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 5000 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # Tarpit: >3000 req/10s per IP (300 req/s — aggressive bot/scraper)
+    http-request tarpit deny_status 429 if { sc_http_req_rate(0) gt 3000 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # Connection rate limit: >500 new connections per 10s per IP
+    http-request deny deny_status 429 if { sc_conn_rate(0) gt 500 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # Concurrent connection limit: >500 simultaneous connections per IP
+    http-request deny deny_status 429 if { sc_conn_cur(0) gt 500 } !is_local !is_trusted_ip !is_whitelisted !is_health_check
+    # High error rate: >100 errors in 30s (scanner/fuzzer behavior)
+    http-request tarpit deny_status 403 if { sc_http_err_rate(0) gt 100 } !is_local !is_trusted_ip !is_whitelisted !is_health_check

    # IP blocking using map file (manual blocks only)
    # Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
@@ -47,3 +57,52 @@ frontend web
    acl is_blocked_ip var(txn.real_ip),map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0
    http-request set-path /blocked-ip if is_blocked_ip
    use_backend default-backend if is_blocked_ip
+{%- if suspension_enabled %}
+
+    # Site suspension routing. Any Host header listed in
+    # /etc/haproxy/suspended_domains.list is rewritten to /suspended and
+    # routed through default-backend, which is the same Flask app that
+    # serves the default page and blocked-ip page (port 8080 inside this
+    # container). The `/suspended` route returns HTTP 503 with a static
+    # suspension page. External tooling (e.g. WHP's site_disable.php)
+    # maintains the list file via `docker cp`. An empty list is safe —
+    # the ACL simply doesn't match. Sits after IP-blocking so 429/403
+    # still trigger first.
+    acl is_suspended_domain hdr(host),lower -f /etc/haproxy/suspended_domains.list
+    http-request set-path /suspended if is_suspended_domain
+    use_backend default-backend if is_suspended_domain
+{%- endif %}
+{%- if coraza_spoe_backend %}
+
+    # Coraza WAF inspection via SPOE. Runs AFTER rate-limit and IP-block
+    # guards (no point asking the WAF about requests we're already dropping)
+    # and AFTER the real-client-IP resolution (so Coraza sees the right src).
+    filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
+    http-request send-spoe-group coraza coraza-req
+
+    # Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to
+    # "deny" / "drop" / "redirect" when a rule with the corresponding
+    # disruptive action fires (depends on SecRuleEngine mode + per-rule
+    # ctl:ruleEngine overrides). Without these rules, Coraza would inspect
+    # but never block.
+    #
+    # On request-phase deny we return a rendered HTML page that surfaces the
+    # request reference (the unique-id) so a customer who's been blocked
+    # incorrectly can open a support ticket and quote it. lf-file expands
+    # log-format expressions inside the file at response time, so
+    # %[unique-id] / %[req.hdr(host)] / etc. get substituted live.
+    # Response-phase deny stays as a bare 403 — outbound blocks are rare in
+    # our config (Coraza response inspection is disabled by default) and
+    # an HTML body on a 403 generated mid-response could land mid-stream.
+    http-request return status 403 content-type "text/html; charset=utf-8" hdr waf-block "request" hdr x-request-reference "%[unique-id]" lf-file /haproxy/errors/403-waf.html if { var(txn.coraza.action) -m str deny }
+    http-response deny deny_status 403 hdr waf-block "response" hdr x-request-reference "%[unique-id]" if { var(txn.coraza.action) -m str deny }
+    http-request silent-drop if { var(txn.coraza.action) -m str drop }
+    http-response silent-drop if { var(txn.coraza.action) -m str drop }
+    http-request redirect code 302 location %[var(txn.coraza.data)] if { var(txn.coraza.action) -m str redirect }
+    http-response redirect code 302 location %[var(txn.coraza.data)] if { var(txn.coraza.action) -m str redirect }
+
+    # FAIL-OPEN on SPOA error. Upstream's example does the opposite — denies
+    # 500 if var(txn.coraza.error) is set — but for a hosting platform we'd
+    # rather lose WAF coverage briefly than 503 customer sites. The error
+    # variable still gets set, so monitoring can observe it.
+{%- endif %}
@@ -0,0 +1,58 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <meta name="robots" content="noindex,nofollow">
+    <title>Site temporarily unavailable</title>
+    <style>
+        body {
+            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
+            text-align: center;
+            padding: 50px 20px;
+            background: linear-gradient(135deg, #1e293b 0%, #0f172a 100%);
+            margin: 0;
+            min-height: 100vh;
+            display: flex;
+            align-items: center;
+            justify-content: center;
+            color: #e2e8f0;
+        }
+        .container {
+            background: #1e293b;
+            border: 1px solid #334155;
+            padding: 40px;
+            border-radius: 12px;
+            box-shadow: 0 10px 30px rgba(0,0,0,0.4);
+            max-width: 560px;
+            width: 100%;
+        }
+        h1 {
+            color: #f1f5f9;
+            margin: 0 0 20px;
+            font-size: 1.75em;
+            font-weight: 600;
+        }
+        p {
+            color: #cbd5e1;
+            line-height: 1.7;
+            margin: 0 0 12px;
+            font-size: 1.05em;
+        }
+        .note {
+            color: #94a3b8;
+            font-size: 0.9em;
+            margin-top: 24px;
+            padding-top: 24px;
+            border-top: 1px solid #334155;
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <h1>This site is temporarily unavailable.</h1>
+        <p>The site you are trying to reach is currently offline.</p>
+        <p class="note">Site owners: please contact support to restore service.</p>
+    </div>
+</body>
+</html>
@@ -0,0 +1,9 @@
+# Source-IP whitelist — exempt from HAProxy rate limits (one IP or CIDR per line).
+# Referenced by templates/hap_listener.tpl:
+#   acl is_trusted_ip src -f /etc/haproxy/trusted_ips.list
+#
+# Add trusted source IPs below. Do NOT commit real/personal IPs to this repo —
+# it is mirrored publicly. Keep real entries in an untracked local copy, or add
+# them directly on the server (the file lives in the /etc/haproxy named volume
+# and persists across container recreates).
+127.0.0.1
@@ -0,0 +1,9 @@
+# Real-IP whitelist for proxy-header matching — exempt from HAProxy rate limits.
+# Format: "<IP> 1" (one per line). Referenced by templates/hap_listener.tpl:
+#   acl is_whitelisted var(txn.real_ip),map_ip(/etc/haproxy/trusted_ips.map,0) -m int gt 0
+#
+# Add trusted real IPs below. Do NOT commit real/personal IPs to this repo —
+# it is mirrored publicly. Keep real entries in an untracked local copy, or add
+# them directly on the server (the file lives in the /etc/haproxy named volume
+# and persists across container recreates).
+127.0.0.1 1