feat(edge): unrouted hosts 404+noindex everywhere; access log moves into the container (2026.11.2)

Default splash (Josh 2026-11-08): unrouted hosts are 404 on EVERY path incl.
"/" (same static page), with cache-control: no-store, x-robots-tag: noindex,
nofollow, noarchive and nosniff; /robots.txt answers 200 "Disallow: /". So the
server IP/hostname is never indexed as a site. Routed sites, ACME, blocked
IPs, suspended hosts, rate limits, wp-* gates and WAF are unchanged.

Access log: the old target 172.18.0.1:514 (host rsyslog) had no listener on
the fleet. HAProxy now logs to a UNIX datagram socket in its chroot
(`log /log.sock`, i.e. /var/lib/haproxy/log.sock); scripts/haproxy-log-receiver.py
(started before init.py, supervised by a restart loop in start-up.sh) writes
/var/log/haproxy/access.log rotated at 40 MB + 4 gzip generations (<= 200 MB).
No network listener; sends are non-blocking, so a dead/stalled receiver or a
full disk only drops lines (DroppedLogs). Nothing is printed to docker logs
(WHP's AI monitor reads them). Same log-format (cip/host/status/Coraza fields).
HAPROXY_SYSLOG_TARGET still overrides. show-tarpit-ips.sh / monitor-attacks.sh
point at the new file.

Tests: test-default-splash.py (19, incl. log target), new
test-access-log-receiver.py (10), both in the image build; validate-rendered-
config.py scenarios all accepted by haproxy -c. Lab (pre-populated volume,
recreate): 404+headers on hostname/IP/unknown, robots 200, routed site 200,
attack on site 403, ACME 200, suspended 503; 2000/2000 burst lines logged;
receiver kill -9 -> requests 200, receiver back <3s; SIGSTOP + 2500 requests
-> all 200, 1475 DroppedLogs; rotation at 64 KB cap kept 1+2 gz; logs survive
recreate on the haproxy-logs volume.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb

.gitea/scripts/resolve-base-image.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+## resolve-base-image.sh IMAGE:TAG — CI only (not copied into images).
+## Prints IMAGE@sha256:<digest> for a Docker Hub tag, looked up DIRECTLY at
+## registry-1.docker.io over HTTPS (never via the pkg-cache mirror), so the
+## build can say `FROM IMAGE@sha256:...` and the mirror can only serve
+## content-addressed blobs that BuildKit verifies. Exits non-zero (prints
+## nothing on stdout) if the lookup fails; the workflow then skips the mirror
+## and pulls the tag directly from Docker Hub. HEAD requests do not count
+## against Docker Hub pull rate limits.
+## Canonical copy: cloud-apache-container/.gitea/scripts/resolve-base-image.sh
+## (docs/pkg-cache.md there); other repos carry identical copies.
+set -euo pipefail
+ref="${1:?usage: resolve-base-image.sh IMAGE:TAG}"
+name_re='^[a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)?$'
+tag_re='^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$'
+name="${ref%:*}"; tag="${ref##*:}"
+[ "$name" != "$ref" ] || { echo "resolve-base-image: '$ref' has no :tag" >&2; exit 2; }
+[[ "$name" =~ $name_re ]] || { echo "resolve-base-image: bad image name '$name'" >&2; exit 2; }
+[[ "$tag" =~ $tag_re ]] || { echo "resolve-base-image: bad tag '$tag'" >&2; exit 2; }
+repo="$name"; [[ "$repo" == */* ]] || repo="library/$repo"   # official images
+
+curl_opts=(-fsS --proto '=https' --retry 3 --retry-all-errors --max-time 20)
+token=$(curl "${curl_opts[@]}" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" |
+        python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')
+[ -n "$token" ] || { echo "resolve-base-image: no Docker Hub token for $repo" >&2; exit 1; }
+hdrs=$(curl "${curl_opts[@]}" -I \
+    -H "Authorization: Bearer ${token}" \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    "https://registry-1.docker.io/v2/${repo}/manifests/${tag}")
+digest=$(printf '%s\n' "$hdrs" | tr -d '\r' | awk 'tolower($1)=="docker-content-digest:" {print $2}' | tail -n1)
+[[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "resolve-base-image: no valid digest for $ref (got '${digest}')" >&2; exit 1; }
+printf '%s@%s\n' "$name" "$digest"

.gitea/workflows/build-push-coraza.yaml

@@ -21,6 +21,24 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
+      # Internal package cache (CT130 pkg-cache; canonical docs: docs/pkg-cache.md
+      # in cloud-apache-container). Used ONLY if it answers over TLS with our
+      # pinned CA; otherwise (down, unreachable, bad/expired cert) this job builds
+      # from the upstream repositories / default GOPROXY exactly as before. The
+      # build stages' base (golang:1.25) comes from the in-house registry
+      # (repo.anhonesthost.net, weekly digest-pinned mirror of Docker Hub -
+      # mirror-base-image.yaml) and the runtime base from gcr.io, not docker.io,
+      # so the pkg-cache registry mirror is never used here: repos + Go modules.
+      # The CA lives under coraza-spoa/ because that is this build's context.
+      - name: Probe package cache
+        id: pkgcache
+        run: |
+          url=""
+          if curl -fsS --max-time 5 --cacert coraza-spoa/keys/pkg-cache/ca.crt -o /dev/null https://192.168.1.40/; then url="https://192.168.1.40"; fi
+          echo "pkg-cache: repos=${url:-<unreachable: upstream>} registry-mirror=<not used: no docker.io base (base from repo.anhonesthost.net)>"
+          [ -n "$url" ] || echo "::warning::pkg-cache unreachable - building from upstream repositories"
+          echo "url=${url}" >> "$GITHUB_OUTPUT"
+
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
 
@@ -49,6 +67,8 @@ jobs:
           context: ./coraza-spoa
           platforms: linux/amd64
           push: true
+          build-args: |
+            PKG_CACHE=${{ steps.pkgcache.outputs.url }}
           tags: |
             repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest
             ghcr.io/shadowdao/coraza-spoa:latest

.gitea/workflows/build-push.yaml

@@ -12,6 +12,22 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
+      # Internal package cache (CT130 pkg-cache; canonical docs: docs/pkg-cache.md
+      # in cloud-apache-container). Used ONLY if it answers over TLS with our
+      # pinned CA; otherwise (down, unreachable, bad/expired cert) this job builds
+      # from the upstream repositories exactly as before. The base image comes
+      # from the in-house registry (repo.anhonesthost.net, weekly digest-pinned
+      # mirror of Docker Hub - mirror-base-image.yaml), not docker.io, so the
+      # pkg-cache registry mirror is never used here: repos only.
+      - name: Probe package cache
+        id: pkgcache
+        run: |
+          url=""
+          if curl -fsS --max-time 5 --cacert keys/pkg-cache/ca.crt -o /dev/null https://192.168.1.40/; then url="https://192.168.1.40"; fi
+          echo "pkg-cache: repos=${url:-<unreachable: upstream>} registry-mirror=<not used: no docker.io base (base from repo.anhonesthost.net)>"
+          [ -n "$url" ] || echo "::warning::pkg-cache unreachable - building from upstream repositories"
+          echo "url=${url}" >> "$GITHUB_OUTPUT"
+
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
 
@@ -50,6 +66,7 @@ jobs:
           push: true
           build-args: |
             VERSION=${{ steps.ver.outputs.version }}
+            PKG_CACHE=${{ steps.pkgcache.outputs.url }}
           # Three tags per registry: :latest (moving), :<version> (human-readable
           # release), :<sha> (immutable, guaranteed-unique rollback target).
           tags: |

.gitea/workflows/mirror-base-image.yaml

@@ -26,11 +26,35 @@ jobs:
       # others from refreshing.
       fail-fast: false
       matrix:
+        # hub: the Docker Hub IMAGE:TAG whose digest is resolved DIRECTLY at
+        # registry-1.docker.io (resolve-base-image.sh expects no docker.io/
+        # prefix; official images resolve as library/<image>).
         image:
-          - { src: 'docker.io/library/python:3.12-slim', dst_path: 'cloud-hosting-platform/python', tag: '3.12-slim' }
+          - { hub: 'python:3.12-slim', dst_path: 'cloud-hosting-platform/python', tag: '3.12-slim' }
-          - { src: 'docker.io/library/golang:1.25',      dst_path: 'cloud-hosting-platform/golang', tag: '1.25' }
+          - { hub: 'golang:1.25',      dst_path: 'cloud-hosting-platform/golang', tag: '1.25' }
 
     steps:
+      # Only for .gitea/scripts/resolve-base-image.sh.
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      # Supply chain: ask Docker Hub itself (over HTTPS, never via a mirror) which
+      # digest the tag points at right now, then pull exactly that digest. What
+      # lands in the in-house registry is therefore content-addressed to the
+      # digest Docker Hub reported - not whatever a `docker pull TAG` on the
+      # runner happened to resolve through any configured registry mirror.
+      # If the lookup fails the job FAILS and last week's mirror stays in place.
+      - name: Resolve ${{ matrix.image.hub }} digest (Docker Hub, direct)
+        id: base
+        env:
+          HUB_TAG: ${{ matrix.image.hub }}
+        run: |
+          set -euo pipefail
+          ref=$(.gitea/scripts/resolve-base-image.sh "$HUB_TAG")
+          [[ "$ref" =~ ^[a-z0-9][a-z0-9._/-]*@sha256:[0-9a-f]{64}$ ]] || { echo "::error::bad digest reference for ${HUB_TAG}"; exit 1; }
+          echo "base-image: ${HUB_TAG} -> ${ref} (digest from registry-1.docker.io)"
+          echo "ref=${ref}" >> "$GITHUB_OUTPUT"
+
       - name: Login to in-house registry
         uses: docker/login-action@v3
         with:
@@ -38,21 +62,31 @@ jobs:
           username: ${{ secrets.CI_USER }}
           password: ${{ secrets.CI_TOKEN }}
 
-      - name: Pull, retag, push ${{ matrix.image.src }}
+      - name: Pull by digest, retag, push ${{ matrix.image.hub }}
+        # Values reach the script via env, never via ${{ }} inside run:.
+        env:
+          REF: ${{ steps.base.outputs.ref }}
+          DST_PATH: ${{ matrix.image.dst_path }}
+          DST_TAG: ${{ matrix.image.tag }}
         run: |
           set -euo pipefail
-          SRC="${{ matrix.image.src }}"
+          [[ "$REF" =~ ^[a-z0-9][a-z0-9._/-]*@sha256:[0-9a-f]{64}$ ]] || { echo "::error::bad digest reference '${REF}'"; exit 1; }
-          DST="repo.anhonesthost.net/${{ matrix.image.dst_path }}:${{ matrix.image.tag }}"
+          [[ "$DST_PATH" =~ ^[a-z0-9][a-z0-9._/-]*$ && "$DST_TAG" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || { echo "::error::bad destination"; exit 1; }
+          name="${REF%@*}"; digest="${REF#*@}"
+          [[ "$name" == */* ]] || name="library/${name}"   # official images
+          SRC="docker.io/${name}@${digest}"
+          DST="repo.anhonesthost.net/${DST_PATH}:${DST_TAG}"
 
           echo "::group::Pulling ${SRC}"
           docker pull "${SRC}"
           echo "::endgroup::"
 
-          # Capture the upstream digest so the workflow log shows what we
+          # The pulled image must be recorded under exactly the digest Docker
-          # actually pushed. Helps diagnose "did the mirror really update"
+          # Hub reported (docker refuses content that does not match it anyway).
-          # questions later.
+          repo_digests=$(docker image inspect "${SRC}" -f '{{range .RepoDigests}}{{println .}}{{end}}')
-          SRC_DIGEST=$(docker image inspect "${SRC}" -f '{{index .RepoDigests 0}}')
+          grep -qxF -e "docker.io/${name}@${digest}" -e "${name#library/}@${digest}" -e "${name}@${digest}" <<<"$repo_digests" \
-          echo "upstream digest: ${SRC_DIGEST}"
+            || { echo "::error::pulled image is not recorded under ${digest}"; exit 1; }
+          echo "upstream digest: ${SRC}"
 
           docker tag "${SRC}" "${DST}"
 
@@ -60,6 +94,8 @@ jobs:
           docker push "${DST}"
           echo "::endgroup::"
 
-          # Sanity: the in-house tag should now resolve to the same content.
+          # The mirror tag's manifest digest (single-platform image as pushed);
-          DST_DIGEST=$(docker image inspect "${DST}" -f '{{index .RepoDigests 0}}')
+          # differs from the multi-arch index digest above by design.
-          echo "mirror digest:   ${DST_DIGEST}"
+          dst_digests=$(docker image inspect "${DST}" -f '{{range .RepoDigests}}{{println .}}{{end}}')
+          DST_DIGEST=$(grep -m1 -F "repo.anhonesthost.net/${DST_PATH}@" <<<"$dst_digests" || true)
+          echo "mirror digest:   ${DST_DIGEST:-<unknown>}"

CLAUDE.md

@@ -10,6 +10,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
 - **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
 - **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
+- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
+- **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine.
+- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
+- **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash".
+- **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`.
 - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
 
 ### Reading stick tables (and why it is easy to get silently wrong)
@@ -25,8 +30,10 @@ columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
 by NAME, and treat a missing field as an ERROR — never default it to `0`. The
 `web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
 `http_err_rate`; it holds **no history and no counter of past blocks**. What was
-actually denied/tarpitted is in the edge access log on the **host** at
+actually denied/tarpitted is in the edge access log, since 2026.11.2 INSIDE the
-`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table.
+container at `/var/log/haproxy/access.log` (in-container receiver, volume
+`haproxy-logs`; README "Access Log"), not in any stick table. (2026.08.8-2026.11.1
+sent it to host rsyslog on 172.18.0.1:514, which had no listener on the fleet.)
 
 This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
 reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
@@ -166,6 +173,8 @@ re-reads it on reload, so a failed runtime command must degrade to
 - Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
 - Stats page on port 8404 (localhost only inside container)
 - Management interface on port 8000 should be firewall-protected in production
-- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s`
+- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
+- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
+- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
 - Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
 - HAProxy is version 3.0.11

Dockerfile

@@ -39,11 +39,27 @@ LABEL org.opencontainers.image.title="haproxy-manager-base" \
 # updated — that failure is the point, not a bug. Check availability with:
 #   apt-cache policy haproxy
 ARG HAPROXY_VERSION=3.0.11-1+deb13u3
-RUN apt update -y && apt dist-upgrade -y && apt install socat "haproxy=${HAPROXY_VERSION}" cron certbot curl jq net-tools -y && apt-mark hold haproxy && apt clean && rm -rf /var/lib/apt/lists/*
+# OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40
+# (CT130 pkg-cache, https only; CI sets it only after probing the cache with the
+# pinned CA). Unset = the upstream Debian repositories, exactly as before. The
+# apt sources are restored before this RUN ends (restore FAILS the build if any
+# reference to the cache is left), so no shipped layer carries the cache address.
+# Canonical script + docs: cloud-apache-container (scripts/pkg-cache-repos.sh,
+# docs/pkg-cache.md there); this repo carries byte-identical copies.
+ARG PKG_CACHE=
+COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
+COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/
+RUN /usr/local/sbin/pkg-cache-repos.sh apt && \
+    apt update -y && apt dist-upgrade -y && apt install socat "haproxy=${HAPROXY_VERSION}" cron certbot curl jq net-tools -y && apt-mark hold haproxy && \
+    /usr/local/sbin/pkg-cache-repos.sh restore && \
+    apt clean && rm -rf /var/lib/apt/lists/*
 WORKDIR /haproxy
 COPY ./templates /haproxy/templates
 COPY requirements.txt /haproxy/
 COPY haproxy_manager.py /haproxy/
+COPY haproxy_gunicorn_worker.py /haproxy/
+# Read by GET /api/capabilities (WHP gates whole-site re-bundling on it).
+COPY VERSION /haproxy/VERSION
 COPY scripts /haproxy/scripts
 COPY trusted_ips.list /etc/haproxy/trusted_ips.list
 COPY trusted_ips.map /etc/haproxy/trusted_ips.map
@@ -89,12 +105,16 @@ RUN pip install -r requirements.txt
 RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
  && python3 /haproxy/scripts/test-trusted-proxy-gate.py \
  && python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \
+ && python3 /haproxy/scripts/test-default-splash.py \
+ && python3 /haproxy/scripts/test-access-log-receiver.py \
  && python3 /haproxy/scripts/test-config-rollback.py \
  && python3 /haproxy/scripts/test-cert-write-safety.py \
  && python3 /haproxy/scripts/test-cert-scripts.py \
+ && python3 /haproxy/scripts/test-waf-body-inspection.py \
+ && python3 /haproxy/scripts/test-gunicorn-socket-timeouts.py \
  && python3 /haproxy/scripts/validate-rendered-config.py
 # Create log directories
-RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
+RUN mkdir -p /var/log/haproxy && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
 RUN chmod 755 /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
 # Set up cron for certificate renewal with proper permissions and environment
 RUN mkdir -p /var/spool/cron/crontabs && \
@@ -105,7 +125,10 @@ RUN mkdir -p /var/spool/cron/crontabs && \
 # 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
 # must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
 EXPOSE 80 443 443/udp 8000
-# Add health check
+# Health check: management API (:8000, short timeout) AND HAProxy (:80).
+# See scripts/healthcheck.sh. NOTE: WHP's lib-edge-container.sh passes its own
+# --health-cmd, which REPLACES this one on deployed hosts; it must be pointed
+# at /haproxy/scripts/healthcheck.sh for the fleet to get the :8000 probe.
 HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
-    CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1
+    CMD /haproxy/scripts/healthcheck.sh
 CMD ["/haproxy/scripts/start-up.sh"]

README.md

@@ -379,9 +379,123 @@ tail -f /var/log/haproxy-manager-errors.log
 | `HAPROXY_DEFAULT_PAGE_TITLE` | Title for the default page | Site Not Configured |
 | `HAPROXY_DEFAULT_MAIN_MESSAGE` | Main message on the default page | This domain has not been configured yet. Please contact your system administrator to set up this website. |
 | `HAPROXY_DEFAULT_SECONDARY_MESSAGE` | Secondary message on the default page | If you believe this is an error, please check the domain name and try again. |
+| `HAPROXY_DEFAULT_SPLASH` | `off` (or `0`/`false`/`no`) disables the default splash for unrouted hosts and restores the default-backend page below | on |
+
+## Default Splash (unrouted hosts)
+
+Since 2026.11.1 (404-everywhere + robots since 2026.11.2), any request whose Host is **not routed to a site** -- the
+server's own hostname (registered cert-only, no backend), a bare IP, a stale
+or unknown name, a request with no Host at all -- is answered by HAProxy
+itself with the static page `errors/default-splash.html` (`http-request
+return ... file`): `404` on EVERY path (including `/`) so search engines never
+index the server's IP or hostname as a site, with `cache-control: no-store`,
+`x-robots-tag: noindex, nofollow, noarchive` and `x-content-type-options:
+nosniff`; `/robots.txt` is `200 text/plain` `User-agent: *` / `Disallow: /`
+(`errors/robots-unrouted.txt`). Monitors pointed at the bare server hostname
+on 80/443 must accept 404 (WHP's own probes -- healthcheck.sh, the watchdog --
+accept any status). The page
+names no version, echoes nothing from the request (it is loaded with `file`,
+not `lf-file`), and links nowhere -- in particular not to the control panel.
+
+**Why.** `use_backend default-backend` does not short-circuit HAProxy's
+`http-request` rules, so the old catch-all was inspected by Coraza with the
+full server-wide rule set like customer traffic. A browser on
+`https://<server-hostname>/` also sends the control panel's cookies (cookies
+are not port-scoped, so `:8443` cookies go to `:443`), and anything Coraza
+matched there showed the WAF block page for a host with no site behind it.
+The splash rules sit before the SPOE call, so unrouted requests never reach
+the WAF; there is nothing behind them to protect.
+
+**What still applies / is kept out:** rate limits, the IP block (blocked IPs
+still get the blocked-ip page), the wp-* gates, and the multipart guard run
+first; `/.well-known/acme-challenge/` still reaches certbot (the server
+hostname's own certificate is issued through it); hosts in
+`suspended_domains.list` still get the suspended page. Configured sites,
+wildcards and their WAF inspection are unchanged. Plain HTTP on `:80` gets
+the splash too, not a redirect: an unknown name usually has no valid
+certificate, so a redirect would trade a clean page for a TLS warning.
+Over TLS the presented certificate is whatever SNI selects -- the
+hostname's own cert if one is issued, otherwise the default bundle.
+
+**How "routed" is decided.** `generate_config()` writes
+`/etc/haproxy/routed_hosts.list` (exact names) and
+`/etc/haproxy/routed_wildcards.list` (`.base` suffixes) from the same domain
+rows, in the same call, as the routing ACLs, and both are in the rollback set
+next to `haproxy.cfg`. The splash ACLs compare the Host port-stripped and
+lowercased, a superset of the routing ACLs (which compare the raw Host), so a
+host that routes is never splashed. Never hand-edit the lists; they are
+rewritten on every regeneration (also when the splash is off, so turning it
+on never meets a missing `-f` file).
+
+**Deploying to an existing host.** The page ships at
+`/haproxy/errors/default-splash.html` and the template under
+`/haproxy/templates/` -- both in the image, OUTSIDE the `/etc/haproxy` named
+volume, so the volume cannot shadow them. The lists live in the volume but are
+generated, not copied. So an image bump plus a recreate of the container is
+the whole deployment: on start `generate_config()` renders the new listener
+and writes both lists, then HAProxy loads them. Nothing to copy into the
+volume, no WHP-side change. Verify after the recreate:
+
+```bash
+docker exec haproxy-manager grep -c default-splash.html /etc/haproxy/haproxy.cfg   # 2
+docker exec haproxy-manager cat /etc/haproxy/routed_hosts.list                     # every site
+curl -skI https://<server-hostname>/                       # 404 + x-robots-tag noindex
+curl -sk https://<server-hostname>/robots.txt              # User-agent: * / Disallow: /
+curl -sk -o /dev/null -w '%{http_code}\n' https://<a-customer-site>/                 # unchanged
+```
+
+Rollback: recreate on the previous image tag, or set
+`HAPROXY_DEFAULT_SPLASH=off` and regenerate.
+
+Covered by `scripts/test-default-splash.py` (template order/conditions, the
+page, list generation) and the `splash-off` scenario of
+`scripts/validate-rendered-config.py`.
+
+## Access Log (in-container, since 2026.11.2)
+
+HAProxy logs (`log /log.sock ... format rfc5424`, `hap_header.tpl`) to a UNIX
+datagram socket inside its chroot (`/var/lib/haproxy/log.sock`).
+`scripts/haproxy-log-receiver.py`, started and supervised by `start-up.sh` (a
+restart loop: back within ~2 s), writes one line per request to
+`/var/log/haproxy/access.log`, rotated at 40 MB with 4 gzip generations
+(worst case 200 MB; `HAPROXY_ACCESS_LOG_MAX_BYTES`, `HAPROXY_ACCESS_LOG_BACKUPS`).
+No network listener anywhere. Fields: client/real IP (`cip=`), Host, status,
+termination state, request id, UA, SNI, HTTP version, WAF body stamp and the
+Coraza `wrule=/wrules=/wscore=` fields (unchanged `log-format`).
+
+* **Never blocks HAProxy.** Sends are non-blocking: a dead or stalled receiver,
+  or a full disk, only drops log lines (counted as `DroppedLogs` in
+  `echo "@1 show info" | socat stdio /tmp/haproxy-cli`). Lab: with the
+  receiver SIGSTOPped, 2500/2500 requests still answered 200 in ~1 s;
+  1025 lines queued, 1475 dropped and counted.
+* **Not in `docker logs`.** The receiver prints no log lines (WHP's AI log
+  monitor reads `docker logs haproxy-manager`; a per-request firehose there
+  would storm it). Only its start line and rate-limited errors go to stderr.
+* **Persistence.** Mount a named volume at `/var/log/haproxy` (WHP:
+  `-v haproxy-logs:/var/log/haproxy`) or the log resets on every recreate.
+  Give the container `--sysctl net.unix.max_dgram_qlen=1024` (a netns starts
+  at 10) for burst headroom.
+* `HAPROXY_SYSLOG_TARGET=<ip>:<port>` still overrides the target for a remote
+  collector. The old default `172.18.0.1:514` (host rsyslog) is gone.
+
+Reading it:
+
+```bash
+docker exec haproxy-manager tail -f /var/log/haproxy/access.log
+docker exec haproxy-manager grep ' 403 ' /var/log/haproxy/access.log | tail
+docker exec haproxy-manager sh -c 'zcat /var/log/haproxy/access.log.*.gz' | grep <request-id>
+docker cp haproxy-manager:/var/log/haproxy/ ./edge-logs/    # or the volume's _data dir on the host
+```
+
+Rollout: image bump + recreate with the volume and sysctl (WHP's
+`lib-edge-container.sh` adds both). The receiver, its script and the template
+are in the image, outside the `/etc/haproxy` named volume, so nothing can be
+shadowed; `haproxy.cfg` is regenerated on start with the new `log` line.
 
 ## Default Backend Configuration
 
+With the default splash on (above), unrouted hosts no longer reach this page; it still serves the blocked-IP and suspended pages, and serves unrouted hosts again when `HAPROXY_DEFAULT_SPLASH=off`.
+
 When a domain is accessed that hasn't been configured in HAProxy, the system will serve a default page instead of showing an error. This default page:
 
 - Informs visitors that the site is not configured

VERSION

@@ -1 +1 @@
-2026.08.11
+2026.11.2

coraza-spoa/Dockerfile

@@ -18,23 +18,47 @@ ARG CORAZA_SPOA_VERSION=v0.7.1
 # weekly job.
 FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS build
 ARG CORAZA_SPOA_VERSION
+# OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40
+# (CT130 pkg-cache, https only; CI sets it only after probing the cache with the
+# pinned CA). Unset = upstream Debian repositories and the default GOPROXY,
+# exactly as before. Go modules stay verified against go.sum / sum.golang.org
+# (GOSUMDB unchanged). Every RUN that rewrites ends with `restore`, which FAILS
+# the build if any reference to the cache is left. Only build stages use it;
+# nothing from here reaches the distroless runtime image except the binaries.
+# scripts/ + keys/pkg-cache/ in this directory are byte-identical copies of the
+# repo-root ones (this image's build context is ./coraza-spoa); canonical
+# source: cloud-apache-container (scripts/pkg-cache-repos.sh, docs/pkg-cache.md).
+ARG PKG_CACHE=
+COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
+COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/
 WORKDIR /src
-RUN apt-get update \
+RUN /usr/local/sbin/pkg-cache-repos.sh apt \
+    && apt-get update \
     && apt-get install -y --no-install-recommends git \
+    && /usr/local/sbin/pkg-cache-repos.sh restore \
     && rm -rf /var/lib/apt/lists/*
+# The source itself is cloned from GitHub over HTTPS (not proxied).
 RUN git clone --depth 1 --branch "${CORAZA_SPOA_VERSION}" \
         https://github.com/corazawaf/coraza-spoa.git . \
+    && eval "$(/usr/local/sbin/pkg-cache-repos.sh go-env)" \
     && go mod download \
-    && CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa .
+    && CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa . \
+    && /usr/local/sbin/pkg-cache-repos.sh restore
 
 # Catalog extractor: walks the bundled CRS at build time and emits
 # rules-catalog.json so WHP's UI can render rule metadata without parsing
 # .conf files at runtime. Uses the SAME coraza-coreruleset version pin as
 # the coraza-spoa binary above (drift between the two would mislabel rules).
 FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog
+# Same optional package cache as the build stage (module downloads only).
+ARG PKG_CACHE=
+COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
+COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/
 WORKDIR /src
 COPY catalog-extractor/ .
-RUN go build -trimpath -o /out/catalog-extractor . \
+RUN eval "$(/usr/local/sbin/pkg-cache-repos.sh go-env)" \
+    && go build -trimpath -o /out/catalog-extractor . \
+    && /usr/local/sbin/pkg-cache-repos.sh restore \
     && /out/catalog-extractor > /out/rules-catalog.json
 
 # Distroless runtime: no shell, no package manager, no /tmp by default —

coraza-spoa/config.yaml

@@ -26,6 +26,37 @@ applications:
       # CRS-bundled defaults: recommended Coraza settings + CRS setup +
       # the rule pack itself (~16 MB of rules embedded in the binary).
       Include @coraza.conf-recommended
+
+      # Request-body limits, set explicitly instead of inheriting the
+      # recommended file's 12.5 MB / Reject. HAProxy only ever hands us one
+      # buffer of body (tune.bufsize 16384 in haproxy-manager's
+      # hap_header.tpl), and the SPOP library refuses frames over 65535
+      # bytes, so no body we receive can exceed 64 KB. The invariant is:
+      #     tune.bufsize (16 KB) < SPOP frame cap (65535) < SecRequestBodyLimit
+      # Breaking it is NOT harmless: SecRequestBodyLimitAction Reject
+      # interrupts with 413 even under DetectionOnly (Coraza
+      # setAndReturnBodyLimitInterruption ignores the engine mode), which the
+      # edge turns into a 403 on every large upload. ProcessPartial is the
+      # belt to that brace. InMemoryLimit == BodyLimit keeps the body buffer
+      # off disk (this image is distroless). Pinned by
+      # haproxy-manager-base/scripts/test-waf-body-inspection.py.
+      SecRequestBodyLimit 131072
+      SecRequestBodyInMemoryLimit 131072
+      SecRequestBodyLimitAction ProcessPartial
+
+      # JSON nesting limit (the recommended file sets 1024). Coraza's JSON
+      # flattener is quadratic in nesting: a 601-byte `[[[...]]]` of depth 300
+      # cost 297 ms and depth 1000 (2 KB) 3.1 s of CPU -- past the 100 ms SPOE
+      # timeout, so the request passed UNINSPECTED (fail-open) while the WAF
+      # burned CPU. Over the limit the body fails to parse (REQBODY_ERROR,
+      # 0.1 ms), which WHP's enforce-mode body-integrity rule denies on a
+      # complete body. Legitimate depth, measured on the JSON shipped in
+      # WordPress 6.x core, Gutenberg, WooCommerce and Elementor (theme.json,
+      # styles/*.json, block.json, checkout schemas): max 8, so a global-styles
+      # save is ~9; 64 leaves ~7x headroom for Divi/Elementor-style module
+      # trees that could not be sampled. Depth 64 itself costs ~21 ms.
+      SecRequestBodyJsonDepthLimit 64
+
       Include @crs-setup.conf.example
 
       # Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.

coraza-spoa/keys/pkg-cache/README

@@ -0,0 +1,32 @@
+Trust material for the OPT-IN internal package cache (scripts/pkg-cache-repos.sh,
+docs/pkg-cache.md). Copied into images at /usr/local/share/pkg-cache/; nothing
+here is added to the system trust store.
+
+ca.crt                 "pkg-cache internal CA (CT130)", EC P-256, valid to 2036-10-18.
+                       NAME-CONSTRAINED to IP 192.168.1.40 and DNS
+                       pkg-cache.internal.cloud-hosting.io / pkg-cache.geekhome.local.
+                       SHA-256 fingerprint BB:5B:9A:08:05:ED:2F:58:83:92:D0:99:9D:11:0A:30:14:88:68:F0:F0:88:65:5E:94:0B:75:EB:C2:A2:BD:F3
+                       (private key lives only on CT130: /etc/pkgcache-ca/ca.key)
+RPM-GPG-KEY-EPEL-10    Fedora (epel10) 7D8D 15CB FC4E 6268 8591 FB26 33D9 8517 E37E D158
+                       fetched 2026-10-21 over HTTPS from dl.fedoraproject.org and
+                       mirror.math.princeton.edu (identical); fingerprint matches
+                       https://fedoraproject.org/security/
+RPM-GPG-KEY-remi2024   Remi's RPM repository 2024 CF1D F005 7CE8 5DFF 5B2F 2A37 C2FD 3B2C 2A09 48E4
+                       fetched 2026-10-21 over HTTPS from rpms.remirepo.net; byte-identical
+                       to the key shipped in remi-release-10.2-1.el10.remi. Not found on
+                       keys.openpgp.org / keyserver.ubuntu.com - confirm out of band.
+                       Remi rotates keys (remi2025, ...): when remi-release starts being
+                       signed by a new key, `release-install remi` fails loudly - add the
+                       new key here (and to pkg-cache-repos.sh).
+RPM-GPG-KEY-EPEL-9     Fedora (epel9) FF8A D134 4597 106E CE81 3B91 8A38 72BF 3228 467C
+                       fetched 2026-10-22 over HTTPS from dl.fedoraproject.org and
+                       mirror.math.princeton.edu (identical); used by `release-install epel`
+                       on EL9 images (cloud-memcached-container, cloud-haproxy-container).
+RPM-GPG-KEY-nodesource Nodesource Operations <operations@nodesource.com>
+                       242B 8138 31AF 0956 2B6C 46F7 6B88 DA4E 3AF2 8A14 (rsa4096, 2023-12-22)
+                       fetched 2026-10-22 over HTTPS from
+                       https://rpm.nodesource.com/gpgkey/ns-operations-public.key (the key
+                       NodeSource's own setup_N.x scripts reference); verified by dnf
+                       gpgcheck against the nodejs RPMs. Not on keys.openpgp.org /
+                       keyserver.ubuntu.com - confirm out of band. Used by
+                       `pkg-cache-repos.sh nodesource MAJOR` (cloud-node-container).

coraza-spoa/keys/pkg-cache/RPM-GPG-KEY-EPEL-10

@@ -0,0 +1,29 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGV4X6kBEAC3eQxgiWuo08uc3mHo4ELux++uqTnYz/tJzEf9Ou3h36WnhumA
+Nvs+Ts5h8PBx879Y9/aIX1Z20p1kf6tBCinZnEJu59n+TAAsph0+XQlV1l5YkleK
+Z2ff/Fg65k8QcLXWaIGykA/FaKznRiSurGuD6tRGhJw7DawEwBJr8QZSkRUpnH1L
+URW97Q/iKrRPiE5VEayE0y8eAL28jIIiFvR+4oJMzvCsRRB/2wYZ2MlJOW91hcYf
+mbUoXKOBD5UzsJylu7kj25K/ge8rEJ7KicOOwcdYddxsU3DxGSSfwF8AMagENcm2
+XROeXknjm84A8sNlUkFZBJwfuc7eRTiZGJrnQQVYLrkKj8Mxpq9Ts7hU51TqAWNI
+uvGDlJdYNE3D2RMqjMEsZ8ej08Thrib6xslu4NzTBkt+6QNnXL4E3hEgYtoyio60
+GswSz2ulogKg7X4JrNdJYE8/qNowyF3hoVgj5TG1/wQRq+5HlMMOLjgGu9wzLUix
+fnVfEUnzaofbrUf4/GabCaeY8xRe4tFQrvzigQ4g+kgwKKnfAeqBmPov0yljkw9z
+BYJWR5zvaw0ffg9Ing00KUSaXBXA5jSlgk1603Y+LefY1SlXsTyqohiRvGH6FI77
+HNMo72DwoJfFcYjncZUzKgXWJECR4nhVsdj6pKoOjcQ4aSuyVxtsR86ASQARAQAB
+tChGZWRvcmEgKGVwZWwxMCkgPGVwZWxAZmVkb3JhcHJvamVjdC5vcmc+iQJOBBMB
+CAA4FiEEfY0Vy/xOYmiFkfsmM9mFF+N+0VgFAmV4X6kCGw8FCwkIBwIGFQoJCAsC
+BBYCAwECHgECF4AACgkQM9mFF+N+0Vhv/A/+PlhPLSctGRCUEahE+cN4764Acc3p
+l40ZYzXRhqR0/Tc1/cSDjlA3qVTc8SPohi5OJXwCyr9EiMqKoyoDN097euqbYpyp
+yN/Pj0lBjsXwcpdDtZ21WGeQU0Khb04N68bMtJbDaxeBciTvDDQravZuPPh0m4Rg
+Z6myEoa6Aa6EK0hI1Qwi1qIWeRiuEkVT671IaKVETBW5XiUpNBXDAB/L+6DzUF9u
+scBzfsUDiPO6NrpYDtV3jwq22y6gWluIct/Ka8brwPbqK2sBfFzrHboRhfqlTGjs
+7F9qUGwIQZn/A8iozXZYQ0+JG1bhQyvjA8eN1GOcRpT+O7H7JXN49o6IG2As4+iK
+F04+qjqAu2sVfpD8mzM2VubFNllcKKiyCzRYHhSbObRCPzsudDL9GPiXeGGaCuWg
+sDkiA1MESvf2tLETAGBs/TziO4GwmXUtlKbRiq1FYm90mVq9mBxPZ/Idn+yZusNB
+0O5SXIbI8lYZw5n4XTK4b+byHRBYsOTHiTsGvjTF2Y7oSwW2CVUmL6RZ23mI4qoY
+1p5kzRS+GjT1acnTei/FTsOlIKCsjfeHx7uxCkX6xpAD8P3UtLQqfsgH0CL4vSZt
+TGO6L1InQlp4ZG3OYIomTKbD3/R0wod3U3dTqdulQMXL895u6OLTY3spY2m2MO2k
+p9Dfd2pKuxK9Mys=
+=mhQZ
+-----END PGP PUBLIC KEY BLOCK-----

coraza-spoa/keys/pkg-cache/RPM-GPG-KEY-EPEL-9

@@ -0,0 +1,29 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGE3mOsBEACsU+XwJWDJVkItBaugXhXIIkb9oe+7aadELuVo0kBmc3HXt/Yp
+CJW9hHEiGZ6z2jwgPqyJjZhCvcAWvgzKcvqE+9i0NItV1rzfxrBe2BtUtZmVcuE6
+2b+SPfxQ2Hr8llaawRjt8BCFX/ZzM4/1Qk+EzlfTcEcpkMf6wdO7kD6ulBk/tbsW
+DHX2lNcxszTf+XP9HXHWJlA2xBfP+Dk4gl4DnO2Y1xR0OSywE/QtvEbN5cY94ieu
+n7CBy29AleMhmbnx9pw3NyxcFIAsEZHJoU4ZW9ulAJ/ogttSyAWeacW7eJGW31/Z
+39cS+I4KXJgeGRI20RmpqfH0tuT+X5Da59YpjYxkbhSK3HYBVnNPhoJFUc2j5iKy
+XLgkapu1xRnEJhw05kr4LCbud0NTvfecqSqa+59kuVc+zWmfTnGTYc0PXZ6Oa3rK
+44UOmE6eAT5zd/ToleDO0VesN+EO7CXfRsm7HWGpABF5wNK3vIEF2uRr2VJMvgqS
+9eNwhJyOzoca4xFSwCkc6dACGGkV+CqhufdFBhmcAsUotSxe3zmrBjqA0B/nxIvH
+DVgOAMnVCe+Lmv8T0mFgqZSJdIUdKjnOLu/GRFhjDKIak4jeMBMTYpVnU+HhMHLq
+uDiZkNEvEEGhBQmZuI8J55F/a6UURnxUwT3piyi3Pmr2IFD7ahBxPzOBCQARAQAB
+tCdGZWRvcmEgKGVwZWw5KSA8ZXBlbEBmZWRvcmFwcm9qZWN0Lm9yZz6JAk4EEwEI
+ADgWIQT/itE0RZcQbs6BO5GKOHK/MihGfAUCYTeY6wIbDwULCQgHAgYVCgkICwIE
+FgIDAQIeAQIXgAAKCRCKOHK/MihGfFX/EACBPWv20+ttYu1A5WvtHJPzwbj0U4yF
+3zTQpBglQ2UfkRpYdipTlT3Ih6j5h2VmgRPtINCc/ZE28adrWpBoeFIS2YAKOCLC
+nZYtHl2nCoLq1U7FSttUGsZ/t8uGCBgnugTfnIYcmlP1jKKA6RJAclK89evDQX5n
+R9ZD+Cq3CBMlttvSTCht0qQVlwycedH8iWyYgP/mF0W35BIn7NuuZwWhgR00n/VG
+4nbKPOzTWbsP45awcmivdrS74P6mL84WfkghipdmcoyVb1B8ZP4Y/Ke0RXOnLhNe
+CfrXXvuW+Pvg2RTfwRDtehGQPAgXbmLmz2ZkV69RGIr54HJv84NDbqZovRTMr7gL
+9k3ciCzXCiYQgM8yAyGHV0KEhFSQ1HV7gMnt9UmxbxBE2pGU7vu3CwjYga5DpwU7
+w5wu1TmM5KgZtZvuWOTDnqDLf0cKoIbW8FeeCOn24elcj32bnQDuF9DPey1mqcvT
+/yEo/Ushyz6CVYxN8DGgcy2M9JOsnmjDx02h6qgWGWDuKgb9jZrvRedpAQCeemEd
+fhEs6ihqVxRFl16HxC4EVijybhAL76SsM2nbtIqW1apBQJQpXWtQwwdvgTVpdEtE
+r4ArVJYX5LrswnWEQMOelugUG6S3ZjMfcyOa/O0364iY73vyVgaYK+2XtT2usMux
+VL469Kj5m13T6w==
+=Mjs/
+-----END PGP PUBLIC KEY BLOCK-----

coraza-spoa/keys/pkg-cache/RPM-GPG-KEY-nodesource

@@ -0,0 +1,52 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGWFsqMBEAC4YlSuNpHjOiQVTC8StolVx3J8E9SC6G4wmVWbytMD1Zivfanj
+3LKDqdy7vpbViXA9whpyGQgQYdOlPKaNHWw4GSeGTnP1FbvaK1NJ9MnAzUysh49R
+cqoktYeIQbqgOgZHw5/gVH5bYObf1KJQ2LY/je9WTlW3QMRe1TaW0F0AZnJMBIOG
+JllE33QIpxDnOQHnvdUGq+HscUOTrW4Q3AdzwsQxU4GX82d9qFFr8HjAmUjlUKyh
+PgvMpb8VDvR7NE6et+wWh6BMYILo3VJFP7/gTXGiy3WWs5GW/PNXIXw5/f2w1B+e
+Dh5tlfGG4q/4P8mbv7C6RPHoF2Lq+W69HLqSq/l42WUL41tHlFTsluCIKpeeMxAu
+olxx3Z5momHMMyh7rbZtQDqFMTAqxNjT1NinKkWyj7KP5WE9hVosyd3UMR4UYDMm
+Zb6LI4N7Hht887+CY7grxTJ9cGQhMFfvLyvlf/FWQY4S6G7sEaUV71N9StvXBNG/
+iNoIMzhwjBmrmvT/UGPkLSi0SIBOxeGw2rwngnYikIJxuTQ4V40lm8o6eMFca7rW
+BQvAEncV8pvi/3FqrUBZ/U53s7NO24hBDoaQPzFbVCw0GHHf9GD7KoQb5c9ONoCZ
+Qu2YWo7ajPLzl/h2pJ2Kn29Ywev9MpuuPTGvifiKilcBVLYP3ZJg4ZfRBwARAQAB
+tDFOb2Rlc291cmNlIE9wZXJhdGlvbnMgPG9wZXJhdGlvbnNAbm9kZXNvdXJjZS5j
+b20+iQJOBBMBCgA4FiEEJCuBODGvCVYrbEb3a4jaTjryihQFAmWFsqMCGwMFCwkI
+BwIGFQoJCAsCBBYCAwECHgECF4AACgkQa4jaTjryihTD+hAAtTU155ke7xmnT0ph
+odbGjm3mOGdkdaDH+xmWtPtufiPncgHMCYh4aIDRloog8rFE+pP3qfwJmUKxthwD
+drrLAzfUwOrGxPnYH3MWwg66RVVoiiU+Hh42gJTez+P/K0VMQZXH0XTZD96TiwT/
+S0+WvQPuJ9pda2L4cjV+YP2+m0ERUG9QPj0Shgw+/PWCGWKD6J9DRToiwfwCTOvc
+DNolF1EcRlfPMTsqI2Bdc+TQywqU1Py1MUDSAPXxdK6nW2oM7eC5l3XXefcQ6JOS
+AuNdEDIsaf+H76j6/wSGsuIcJjUo9wmKjEV1iJYwT6jLbjaVQxXgEPHBSb+kzanx
+FWyrYP6TZpeY2gT4bBMsOKd7ay1I3/nIHwe6x8Yzkqzz+7YVhNpMbybH72NTxDWQ
+UmiboJs9r7j1lkHlBjxU9RLF5laYSeCIXD62KQVoP5ao1LSPKsCXFqcdpUgLYtRv
+CXxdriiHp4m9eGzbS7RW1TLm6tMOKbI4LzOtMm4DoZzGD2FTfsUtn6ALx9FSgvl/
+VMlzqZes+rclcukACtutvtB8e0yRhuzPUN56ctLEZfI1XzQFoN8Vdj6HWtiXWRiJ
+EohuufE2cEebmX90NRsbh4KAFXNEQROlMeMf7ufuSsk7QXLTutxmWSQvWwyS+P5/
+ZqCapIDlRCJFZX4N9a1IXo0OgCC5Ag0EZYWyowEQAKz7sA6WBKFVcmzcyr6y9TI5
+RemnzotRUdqXj8tD70xvV1nBEf+BfTw3hsiV019ZboPBae+cRaD4plQbNb0du/+Q
+UFq4ja055ZEld01RqE49L562pWvA5cwSmjs9+ZMI1ja0HbTb2K4razpzX3be+Zul
+TbH20X5FHTz+Jyfmw+G3Y9okw3OIzuYo16Lv/Qs8VoBmj01PItzq8dX5wJiIlVxU
+8wKHKy1kiKGCgyPnvqcXCGen0EoUS9wsWK1QHfryscYGVF9y1TBQvJa9HIpWPrIZ
+slSiVS4tQhGHfLDVlaDsYFrH9SIUkvW7peSEiPE1TKQjvbMS3CcUZRaMcGfgyDPe
+gXwnNwZbJ2/cisXbdAJaGT49V1Y+Sc1oKSGSG4FaGUNhS3p3cB7qlX2zS/sR2ael
+90fXR6nhjvhMAS98i7663pCUv9q0Z/NXNZwdXjbSF4W8gq3PVt7hCnJrngsGSkGu
+Qz0dc227/sAjh9CxYm8kQqUuQtiPVfKwkcUUT+O7jr2/6eAXZbTYVZbvaLEGMrKA
+bdWKpZMMOKAI45f+ZAcEmJG4D68N0jzVrLCXMuuLq/yqqyEmxDjSOH0qVutl8jxg
+vfxPxKYX8nnAzJZJHGHmV/ihjfClSdYePKExHuFVVhvc4BxE1XfPUeCZG4JpB9/B
+gNCiDPsg4JIw6GO5LS2xABEBAAGJAjYEGAEKACAWIQQkK4E4Ma8JVitsRvdriNpO
+OvKKFAUCZYWyowIbDAAKCRBriNpOOvKKFOTKD/40WN6ZRfP6hVLAX26DdyN9Bqzg
+D+0hJilmHL1YFeLt9U/YPhOZJxPaDGrMfldC0FSXKKr7H5C7FQErl936108wGohz
+53KGbNid0crNeaWrQuWk3QjaMF62Yc+NVB8b5LoEk0TEt9KPO6kveYDtW6Ygkw9i
+5aaS/AC+8GChOH6lWhLHcO91T7+Hi5qRez7qyme/AlHUTV9/jE6JgFJ7thCCAqBQ
+AfX3mGFlTwSTMHDeYq81MMl9UhYgoFJtx8PuqmSV7Eb29ccJofz3J+b5OJvpmLSZ
+Hu22Csa8k2V/1ojeL0DjQHD06V75F7BerxLR/rbJ64GkIiOdm4ntXC4a335gjQfM
+5tI6LtGsIpr+Le4C068eDj7YpzS0ql3g6o4A1TbtapLccgEj4iw95H4KKh0Ro5r0
+0s6LoSLu9uva3LiLVn6xXvpw3iv8FnhTr5PEiPnorXvxcCnUrB3Rkk6Wh1SKjpxa
+u6KiFzH0CXR1AgUdWSC0rAmvEdFjXgaoQd+Z2eJFHEAsg91URd9NtdeDsyMwdbHh
+iBZPGX5ddPjJA7nwKbkibCv7xiodd9ph8NmGLqD/2SDU1rqvPuCDMqLuxc89Gvhs
+3CeLTSJK9Uzb3oNdZLxL8CECip1ZkowVfTL+jY23phS9+Z0Db/yZqot3ZZJAAG8g
+oSVkzT/Mwp9zFVQu5w==
+=drMP
+-----END PGP PUBLIC KEY BLOCK-----

coraza-spoa/keys/pkg-cache/RPM-GPG-KEY-remi2024

@@ -0,0 +1,52 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGWdEG0BEADV9c8aSIITQmoWKN0ac4c8AEhZcdzpWSRPwHIS8syVRw/gj30W
+Mn1OoxqFn2UECh2jjqYaQFECXNrM2/lzO4GhvcuoPn6p0rXFMvYvwtiOJ+rmDvTe
+UOwRxkZhYehpvsRPFoVhlYJh3U1BM0UHm2iE/TIqVusY9FHQuCMZVl0BHk81YGDa
+qE3cyu+9zOs8z1bYlsYBgLIbtb4+loQTRexFTfZkUcCWgAaNyur8Wy7Msdm5LOqN
+ZZu6bNKBOv0Cr+BOwGNoFRbi4IM+5H+0eI1n7V5if4CuXfdYHp+Pm+PQvftm/nqB
+jRLKqioRKzHjk7xNMsi5lM+SU4WyfU77pr3RXuJR6fkymLVAeu5sy2U6ySj0SXIO
+/I3oPp4/zJ8/e94jG5iDqh92b5IWyXSE+EGw4kkpUvZLrK8vyyEfk8h9Gf8qA5uL
+yGfw0vnSVjYb+RdnfZFVdPaIEmeY8tnobS87duj8JESZiSWo5sU4filkciW5e+MJ
+w9wohgl+Cqm2J5Ljqus6DRm4ptKbZD8QZRnmGcSk6O2VopBKt2XLat4eIJLr/EiN
+0CJ1EH/oJmEeZbYtUjxjsGsbPXicwDH6ZIeIMSaiDgqRxPm1LTGVLjSHAk+k0e5l
+ulEmaoA7t/p3U2peGc7ONvozXiyNHZdGiRkqFC1mA9/JpeWw4KFH6JSgmQARAQAB
+tEtSZW1pJ3MgUlBNIHJlcG9zaXRvcnkgMjAyNCAoaHR0cHM6Ly9ycG1zLnJlbWly
+ZXBvLm5ldC8pIDxyZW1pQHJlbWlyZXBvLm5ldD6JAlEEEwEIADsWIQTPHfAFfOhd
+/1svKjfC/TssKglI5AUCZZ0QbQIbAwULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIX
+gAAKCRDC/TssKglI5Na7EACDTGjTcuTyudfuu/JC41BSurclCcwZyPsGaClyc0qx
+Tk8Rs4rbyVw1u494fiFG/yEVl7LGGja84UJOlMdw82KgVUmr4fGLaBBFWm+KnZ4W
+4vvFcCIxUshENOaDrkhby9F5P379gVOK9I3n9l4Q2xt+sWUNwE74CNfCTUeYMIm/
+/AHbb1DTM9FuIT6FssgvILMTlksiyrw8U8bj+QUzPZrFCpvv+PnFA7ZDMEyCf2Tz
+/zawxoP3a3b+3/g61U9SVIkfToonnorqh9lkQYhsdKTFHW6eESqe7zcqMBZv/sSc
+3kFzl22KczSdU8eYxdujv9BmlW2SOrQsnuMYxvoD1LzhwSB/nOapNrWU+djhJCof
+ueZ+rBW4QWysmG0dEp5Wpz85TstAnQ+j5aCOMSAkiAVsrJ0Ff3JXakIiL7rEtccS
+dhgBT06LJtJy1NGIC0Sotn45n6dtfi6IuI8LnELCKeHxhdFSJcAPB8y95u0uHymy
+sqMfzjq1VcYtzC5II6Yd5Gtb2+QzyQwORslwXhUdXwZKUc5i6X5xFMSxU523fH9n
+i+kQlUTPkOUr66v4fmYDj+ZlJQSFoPylcHCke/vOo/rH5nj+YMPa6LdohsdzHKpd
+DkKrk+Udj5aDXtXGIFYwoNKO4BnCsz/ImZfB4CQ+ekJsY1pFill3+MRTM6IvSA+P
+7LkCDQRlnRBtARAAq8k4FM9euMMsXz0buxxyr0ew0ecWNhZGy24a6DXORVTEJbgI
+T3HpV/C2PWkwgEBswYzREm+en61jhWr5unCiytO2klT+K2WNcIUNW+z2ciL/JNaJ
+OcKwjkiY/A+KtDStzllCQ/XnpgAXGvFZjMjhHrNGQoAwNnPE/cMtYwcuVdW3Ipm3
+cOOtzxRmfnhP8YUEyHxdos/7AZowaiLIUkaL5RUaygGrfPGyUXPl1aXS1oiOm37W
+RiKCGUxCHSlpBqeh1WQPOaps8y+ME9jMzF9SLvqiXlKPllJIx9i8tHUOnX6Y2U2x
+5sXsCEiA+zNVMU3YQ04++Yx2Rp6sDhNEnAbTCqqequqIzaXHZWwszAeG6pj1G90m
+8PwDeHy641ySgDMwVPpnH5rHWmmT96XvF9QNg07OBtW9/phryzwAuHogrBY4DyFD
+pUxDkTv9TxfBaS/s1uJxEGknIgnKzQo79mRy206waGKRA2AV36ORSfU12sK8pl3r
+N95iIDh/OIuMl2STaXilhwb1hRpGFvwRlq3Hvvan4SY1aO219Q+VxMGXeqC5F/rm
+Anh7mRVMP2qu0rj7L8xJ/OD/bPfuWL72/PvviBr/jdRM9I+Vj8dPGjhUwhK1dpEG
+Q6yvoVXLHXucVHfIVR+ALLBmK8oHXNIFeIXhiUbgmRAilPHRz6Rf5pgbrC0AEQEA
+AYkCNgQYAQgAIBYhBM8d8AV86F3/Wy8qN8L9OywqCUjkBQJlnRBtAhsMAAoJEML9
+OywqCUjkQ2kP/iwqvt1sX85KM4OSE9ftqemwnvh19NizMKHMYGgikMAvy2CndEOI
+wzKVeE1iK9/ivZQQIbD5l0G9hzVZjIxbgu9dAcxM+jdmaPyUj7yn6AX2yeGAeHyA
+qzNKZaTLJPieRzdXk/uKPVyWUuzbR0vwR7n8N5N0pe/FEfO9O0uX5iz2kf5+b10c
+WqvzLmO+czuyCjTHnjhKfE73iIktYE3MnM8qFHPiKwm+jeTkUnmInoR3qIKR/SCV
+u2vk/q5uvpNaUSPSDKji6wh3/hbNbPocLSGIjugfWNm/4LhlSHPNnRJ2dIvSxeiI
+qtSWxhULEyHGBPMLDriVumCV7Rb/byJwiMbmCIpxB5XLtbgt7nIqWQTko+6ci71S
+4rDXolM3bytYbpqq4b/Xc1bhBuTX3omwrd0kxmGqad8eia6O6iBsP7GAl4QkySm4
+MfKQfohjXTnT60wqEzQwLGI4NKN9bHRSsGINHX5p+Kz9HLf87fTY4uglbFxg0/II
+Y3nko5KBw/gfe0WpwZY0sWjK20cXJn3xLQjoG8OjUNUCUYTRsiAvn4AiDRt++qtl
+bvJjJqOU1k4GrGoCVpEUiOc9Cw1YZREcFe8H2at3pGnxJ1DFlYA916kNAxYANp2s
+r3N3agwevgIzP78hynxjm6a5H2w0vkg189pZn2RR9zX9gmrvlqg1o0P1
+=meaU
+-----END PGP PUBLIC KEY BLOCK-----

coraza-spoa/keys/pkg-cache/ca.crt

@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE-----
+MIICPzCCAeagAwIBAgIUJI0QJZ+PCrSQ/nKZFENwwcp8Zr0wCgYIKoZIzj0EAwIw
+TDEmMCQGA1UEAwwdcGtnLWNhY2hlIGludGVybmFsIENBIChDVDEzMCkxIjAgBgNV
+BAoMGWNsb3VkLWhvc3RpbmcuaW8gaW50ZXJuYWwwHhcNMjYxMDIxMTkzNjIyWhcN
+MzYxMDE4MTkzNjIyWjBMMSYwJAYDVQQDDB1wa2ctY2FjaGUgaW50ZXJuYWwgQ0Eg
+KENUMTMwKTEiMCAGA1UECgwZY2xvdWQtaG9zdGluZy5pbyBpbnRlcm5hbDBZMBMG
+ByqGSM49AgEGCCqGSM49AwEHA0IABBJh75sC5vSzTnkYl0CYYEA/TKUHEXhbtanV
+OVNW4NlD3za5+bu5YuKjpbFAva4TCZqWsaS9IMd/JfsaFbf5XVejgaUwgaIwEgYD
+VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFP7UCTf/
+v7KU1KBrs1vqa71/TZl0MF0GA1UdHgEB/wRTMFGgTzAKhwjAqAEo/////zAlgiNw
+a2ctY2FjaGUuaW50ZXJuYWwuY2xvdWQtaG9zdGluZy5pbzAaghhwa2ctY2FjaGUu
+Z2Vla2hvbWUubG9jYWwwCgYIKoZIzj0EAwIDRwAwRAIgOod4wPHu0kAzYAw8KeGg
+cIrFMwQ4ATKVTkDioiSFOqICIEhY+sGj/YZ4oBMst3b/AAz9zMEJAXpnCkSTI6tM
+YAIl
+-----END CERTIFICATE-----

coraza-spoa/overrides.conf

@@ -47,7 +47,17 @@ SecRuleUpdateActionById 933170-933200 "ctl:ruleEngine=On"
 # vulnerable, but blocking at the edge keeps logs clean and protects
 # any future Java workloads.
 # ---------------------------------------------------------------------------
-SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
+# 944110 and 944120 are NOT promoted (changed 2026-10-09, 2026.10.2). Both are
+# CHAINS whose first link matches bare words anywhere in the request --
+# `runtime|processbuilder` and `getproperty|forclosure|filewriter|...` -- and
+# Coraza runs an appended ctl:ruleEngine=On on that first link even when the
+# chain then fails. Measured (Coraza v3.7.0 + CRS 4.25.0): in detect_only,
+# `?q=runtime&id=1' or '1'='1` was 403 while the SQLi alone was 200. Same
+# chain-starter defect as 920440. They still add to the anomaly score.
+# Log4Shell (944150/944151/944152, ...) stays promoted. WHP mirrors this split
+# in coraza_baked_promotion_ranges() (web-files/libs/coraza_blocking_set.php).
+SecRuleUpdateActionById 944100 "ctl:ruleEngine=On"
+SecRuleUpdateActionById 944130-944300 "ctl:ruleEngine=On"
 
 # ---------------------------------------------------------------------------
 # 920440 — URL file extension restricted by policy
@@ -59,7 +69,26 @@ SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
 # single GCP-hosted scanner. Zero false positives observed; standard
 # WP/WooCommerce/Divi/HPR URLs do not end in these extensions.
 # ---------------------------------------------------------------------------
-SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
+# HOW IT IS PROMOTED (changed 2026-10-09, haproxy-manager-base 2026.10.2).
+# This used to be `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"`, which
+# was WRONG: 920440 is a CHAIN whose first link is
+# `REQUEST_BASENAME "@rx \.([^.]+)$"` -- it matches EVERY URL with a file
+# extension -- and Coraza runs actions appended to a chain starter when that
+# first link matches, even though the chain then fails. Measured (Coraza
+# v3.7.0 + CRS 4.25.0, in-process and live): every *.php / *.css / ...
+# request ran with the engine On, so the whole CRS anomaly score (949110) and
+# 200002 enforced there -- in detect_only mode too, since SecRuleEngine
+# DetectionOnly cannot override a runtime ctl.
+#
+# Now: 920440 itself is untouched (it still sets tx.extension in its first
+# link and adds the anomaly score in its second), and this separate phase-1
+# rule, which runs after it, flips the engine only when the extension really
+# is in tx.restricted_extensions -- the original intent. Removing 920440
+# (break-glass, a per-host exception) also removes tx.extension, so this rule
+# goes quiet with it. Id 992920440 is outside WHP's generated range
+# (990000000-991999999).
+SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \
+    "id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"
 
 # ---------------------------------------------------------------------------
 # 930130 — Restricted File Access Attempt
@@ -76,6 +105,100 @@ SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
 # ---------------------------------------------------------------------------
 SecRuleUpdateActionById 930130 "ctl:ruleEngine=On"
 
+# ---------------------------------------------------------------------------
+# 209610-209616 -- PHP multipart "RCE via OOM corruption" mitigations
+# CVE-2026-109618 (GHSA-w8hw-w9fx-j79w) and CVE-2026-109619
+# (GHSA-6wmf-4rv5-p3rp). Interim edge mitigation until patched PHP is on every
+# customer image; remove once it is. haproxy-manager-base 2026.10.6.
+#
+# The advisory's request-body vectors and what covers each:
+#   (a) multipart with no usable Content-Length (chunked / H2-H3 without
+#       content-length) streams past post_max_size -> denied 400 by HAProxy
+#       itself (templates/hap_listener.tpl), before any of this runs.
+#   (b) unlimited headers per multipart part      -> 209611-209613 (budget)
+#   (c) empty file controls (do not count toward max_file_uploads)
+#       + deeply nested field names (a[][][]...)  -> 209614-209615, 209610
+#   -619: session upload progress during multipart -> 209616
+#
+# WHY THESE IDS. 200000-209999 is the Coraza engine range, and WHP's site
+# break-glass rule (991000001) removes that WHOLE range for a site in
+# emergency mode (coraza_blocking_set_ranges(), WHP_WAF_ENGINE_RANGE_LO..HI in
+# web-files/libs/coraza_blocking_set.php). Rules that deny on their own MUST be
+# inside the derived blocking set or break-glass leaves them armed; inside the
+# engine range they are covered without a WHP release. The engine itself uses
+# only 200000-200006 (coraza.conf-recommended); CRS uses none of 2xxxxx. WHP
+# also treats the range as staff-only for self-service exceptions.
+#
+# They switch the engine On for themselves (ctl:ruleEngine=On), exactly like
+# the day-one promotions above, so they deny in every mode the sidecar runs in.
+#
+# LIMITATION, read before trusting them: HAProxy hands Coraza only the first
+# buffer of the body (tune.bufsize 16 KB). A request that puts its abusive
+# parts after ~16 KB of padding, or whose single part has a >16 KB header
+# block, is not seen here. Vector (a) is the one enforced on the full request.
+# Coraza's multipart parser (Go mime/multipart) also canonicalises part
+# header names and stops at the first part it cannot read.
+# ---------------------------------------------------------------------------
+
+# (c) Field-name nesting. Counts '[' in every ARGS / FILES name (query,
+# urlencoded and multipart names are all raw, PHP-shaped here; Coraza's JSON
+# processor names args json.a.b and never produces '['). WordPress-ecosystem
+# form names nest a few levels (menu-item-settings[12][title],
+# variable_regular_price[0], input_3[]); PHP's own max_input_nesting_level
+# is 64, so 16 is far above real forms and far below PHP's ceiling.
+SecRule ARGS_NAMES|FILES_NAMES "@rx (?:[^\[]*\[){17}" \
+    "id:209610,phase:2,deny,status:403,log,t:none,\
+    msg:'Parameter name nested more than 16 levels deep (CVE-2026-109618 mitigation)',\
+    logdata:'%{MATCHED_VAR_NAME}',tag:'whp/cve-2026-109618',ctl:ruleEngine=On"
+
+# (b) Part-header budget. A legitimate part carries 1-2 headers
+# (Content-Disposition, usually Content-Type; a few clients add
+# Content-Transfer-Encoding). Budget = 4 per part - 1 per part header, and
+# the request is denied once it is 32 below zero: a single part needs 37+
+# headers, and parts averaging 4 or fewer headers never trip it however many
+# there are. Parts are counted by what they BECAME (each part is exactly one
+# ARGS_POST or FILES entry), not by Content-Disposition lines, because a part
+# can repeat Content-Disposition as easily as any other header.
+# @unconditionalMatch runs setvar once PER VALUE (measured), and runs on
+# empty values too, so empty fields and empty files are counted.
+# No REQBODY_PROCESSOR chain on 209611: a chained link's setvar would run once
+# per request, not once per part. On a non-multipart body it only RAISES the
+# budget, and MULTIPART_PART_HEADERS (the only thing that lowers it) is empty.
+SecRule ARGS_POST|FILES "@unconditionalMatch" \
+    "id:209611,phase:2,pass,nolog,t:none,setvar:tx.whp_cve109618_hdr_budget=+4"
+SecRule MULTIPART_PART_HEADERS "@unconditionalMatch" \
+    "id:209612,phase:2,pass,nolog,t:none,setvar:tx.whp_cve109618_hdr_budget=-1"
+SecRule TX:whp_cve109618_hdr_budget "@lt -32" \
+    "id:209613,phase:2,deny,status:403,log,t:none,\
+    msg:'Multipart part(s) with an excessive number of headers (CVE-2026-109618 mitigation)',\
+    logdata:'budget %{tx.whp_cve109618_hdr_budget}',tag:'whp/cve-2026-109618',ctl:ruleEngine=On"
+
+# (c) Empty file controls: parts with an EMPTY filename (`filename=""` or
+# `filename=`), which PHP does not count toward max_file_uploads. Coraza turns
+# such a part into an ARGS_POST field (no FILES entry), so they are counted
+# from the raw part headers. A browser sends one per untouched <input
+# type=file>; legitimate forms have a handful. Denied above 64.
+SecRule MULTIPART_PART_HEADERS "@rx (?i)^content-disposition:.*;\s*filename\s*=\s*(?:\"\"|;|$)" \
+    "id:209614,phase:2,pass,nolog,t:none,setvar:tx.whp_cve109618_empty_files=+1"
+SecRule TX:whp_cve109618_empty_files "@gt 64" \
+    "id:209615,phase:2,deny,status:403,log,t:none,\
+    msg:'Multipart body with more than 64 empty file parts (CVE-2026-109618 mitigation)',\
+    logdata:'%{tx.whp_cve109618_empty_files} empty file parts',tag:'whp/cve-2026-109618',ctl:ruleEngine=On"
+
+# -619: session upload progress. The engine flip sits on the LAST chain link
+# on purpose: Coraza runs a chain starter's ctl as soon as the first link
+# matches, even if the chain then fails (the 920440/944110 defect above).
+# PHP only tracks upload progress for a
+# multipart field named session.upload_progress.name (default
+# PHP_SESSION_UPLOAD_PROGRESS, compared case-sensitively) that precedes the
+# file parts, so that is exactly what is denied. WordPress core, WooCommerce,
+# Divi, Gravity Forms and CF7 do not use the feature.
+SecRule ARGS_POST_NAMES "@streq PHP_SESSION_UPLOAD_PROGRESS" \
+    "id:209616,phase:2,deny,status:403,log,t:none,chain,\
+    msg:'Multipart session upload-progress field (CVE-2026-109619 mitigation)',\
+    tag:'whp/cve-2026-109619'"
+SecRule REQBODY_PROCESSOR "@streq MULTIPART" "t:none,ctl:ruleEngine=On"
+
 # ---------------------------------------------------------------------------
 # Rule families intentionally kept at DETECT-ONLY for v1 — high FP rate
 # on customer mix. Promote individually after observation:

coraza-spoa/scripts/pkg-cache-repos.sh

@@ -0,0 +1,296 @@
+#!/bin/bash
+## pkg-cache-repos.sh — BUILD-TIME ONLY, OPT-IN. Points dnf/apt at the internal
+## package cache (nginx proxy_cache, CT130 "pkg-cache" on the Proxmox cluster)
+## when PKG_CACHE is set; without it the repo files are left alone and builds
+## use the upstream repositories as before.
+##
+##   PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh rpm      # AlmaLinux/EPEL/Remi/NodeSource
+##   PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh apt      # Ubuntu, Debian, LiteSpeed
+##   pkg-cache-repos.sh release-install epel [remi]             # GPG-verified release RPMs (EL9/EL10)
+##   pkg-cache-repos.sh nodesource MAJOR                        # NodeSource repo file + pinned key
+##   eval "$(pkg-cache-repos.sh go-env)"                        # GOPROXY via the cache (go.sum/sumdb verified)
+##   pkg-cache-repos.sh restore                                 # undo everything above
+##
+## CANONICAL copy: cloud-apache-container/scripts/pkg-cache-repos.sh (docs/pkg-cache.md
+## there). Other repos carry byte-identical copies - change it there first.
+##
+## Every RUN that rewrites must `restore` before it ends: it puts rewritten files
+## back, deletes files this script created, deletes any NEW repo/apt file that
+## still references the cache, then FAILS if any reference is left — so no shipped
+## layer carries the cache's LAN address (images run on hosts that can't reach it).
+##
+## Why: on 2026-10-21 (PHP security release) builds failed on Remi mirrors lagging
+## hours behind origin, Remi's origin timing out over HTTPS, and an EPEL mirror
+## with a broken certificate. The cache fetches from the ORIGINS over HTTPS (with
+## origin/mirror fallbacks and stale-on-error), keeps metadata ~3 minutes and
+## immutable .rpm/.deb files long-term.
+##
+## Integrity (security review 2026-10-21):
+##  * PKG_CACHE must be https://. The cache's certificate is issued by the
+##    name-constrained "pkg-cache internal CA" (keys/pkg-cache/ca.crt, valid only
+##    for 192.168.1.40 / pkg-cache.*). That CA is trusted ONLY by the rewritten
+##    repos (dnf sslcacert= bundle, apt per-host CaInfo) and only during the RUN —
+##    never added to the system trust store.
+##  * gpgcheck=1 is forced on every rewritten repo; repo_gpgcheck=1 on Remi and
+##    AlmaLinux, which sign repomd.xml (measured: repomd.xml.asc 200). EPEL
+##    publishes no repomd signature (404), so its metadata relies on HTTPS end
+##    to end (client->cache->dl.fedoraproject.org/mirrors, all verified).
+##  * release-install downloads epel-release / remi-release (from the cache, or
+##    from the HTTPS origin without PKG_CACHE), checks the package name, imports
+##    the signing keys committed in keys/pkg-cache/ and requires
+##    `rpm -K` "signatures OK" before installing with localpkg_gpgcheck=1.
+##  * The php.net version gates (EXPECT_PHP_MIN / EXPECT_PHP_VERSION) still fail
+##    a build that gets old PHP (replay of old-but-signed metadata).
+##
+## rpm: each rewritten baseurl keeps the upstream origin as a SECOND url, so if
+##      the cache dies mid-build dnf fails over to the origin by itself.
+## apt: apt has no per-source failover; CI only sets PKG_CACHE after probing the
+##      cache (with the CA), and a build without PKG_CACHE uses upstream sources.
+set -euo pipefail
+
+MODE="${1:?usage: pkg-cache-repos.sh rpm|apt|release-install NAME...|nodesource MAJOR|go-env|restore}"
+shift || true
+STATE=/var/lib/pkg-cache-repos          # saved originals + manifest + created files
+SHARE="${PKG_CACHE_SHARE:-/usr/local/share/pkg-cache}"   # ca.crt + RPM-GPG-KEY-*
+C="${PKG_CACHE:-}"
+C="${C%/}"
+
+die() { echo "FATAL(pkg-cache-repos): $*" >&2; exit 1; }
+
+if [ -n "$C" ]; then
+    case "$C" in
+        https://*) ;;
+        *) die "PKG_CACHE must be an https:// URL (plain http is refused), got '$C'" ;;
+    esac
+    [[ "$C" =~ ^https://[A-Za-z0-9.-]+(:[0-9]+)?(/[A-Za-z0-9._/-]*)?$ ]] || die "PKG_CACHE has unexpected characters: '$C'"
+    HOSTPORT="${C#https://}"; HOSTPORT="${HOSTPORT%%/*}"
+    [ -s "$SHARE/ca.crt" ] || die "PKG_CACHE set but $SHARE/ca.crt (pkg-cache CA) is missing"
+fi
+
+# System CA bundle (EL vs Debian/Ubuntu path); the pkg-cache CA is appended to a
+# copy in the state dir, never to the system store.
+sys_bundle() {
+    local b
+    for b in /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt; do
+        [ -s "$b" ] && { echo "$b"; return 0; }
+    done
+    die "no system CA bundle found"
+}
+make_bundle() { cat "$(sys_bundle)" "$SHARE/ca.crt" > "$STATE/ca-bundle.pem"; }
+
+# Files that may hold repo configuration; restore re-checks all of them.
+repo_files() {
+    find /etc/yum.repos.d /etc/dnf /etc/apt -type f 2>/dev/null | sort || true
+}
+
+start_session() {
+    [ -d "$STATE" ] && return 0
+    mkdir -p "$STATE/orig"
+    repo_files > "$STATE/manifest"     # what existed before this RUN touched anything
+    : > "$STATE/created"
+    printf '%s\n' "$HOSTPORT" > "$STATE/host"
+}
+save() {                                # keep the first-seen (pre-rewrite) copy
+    [ -e "$STATE/orig$1" ] && return 0
+    mkdir -p "$STATE/orig$(dirname "$1")"
+    cp -p "$1" "$STATE/orig$1"
+}
+created() { printf '%s\n' "$1" >> "$STATE/created"; }
+
+if [ "$MODE" = restore ]; then
+    [ -d "$STATE" ] || exit 0
+    host=$(cat "$STATE/host")
+    # mapfile from a process substitution, not `cmd | while read`: no pipeline,
+    # so nothing for pipefail to misreport (scripts/tests/lsphp-info-probe.test.sh).
+    mapfile -t saved < <(cd "$STATE/orig" && find . -type f)
+    for rel in "${saved[@]}"; do
+        cat "$STATE/orig/$rel" > "/${rel#./}"
+    done
+    while read -r f; do [ -n "$f" ] && rm -f -- "$f"; done < "$STATE/created"
+    # A repo/apt file that appeared during this RUN and still names the cache
+    # (e.g. written by a package from the rewritten repos) is removed too.
+    mapfile -t current < <(repo_files)
+    for f in "${current[@]}"; do
+        if ! grep -qxF -- "$f" "$STATE/manifest" && grep -qF -- "$host" "$f"; then
+            echo "pkg-cache-repos: removing new file that references the cache: $f"
+            rm -f -- "$f"
+        fi
+    done
+    left=$(repo_files | xargs -r grep -lF -- "$host" 2>/dev/null || true)
+    rm -rf "$STATE"
+    [ -z "$left" ] || die "cache address $host still present after restore in: $left"
+    echo "pkg-cache-repos: original repository files restored (no reference to $host left)"
+    exit 0
+fi
+
+if [ "$MODE" = release-install ]; then
+    [ $# -gt 0 ] || die "release-install needs: epel and/or remi"
+    tmp=$(mktemp -d)
+    curl_opts=(-fsS --proto '=https' --retry 5 --retry-all-errors --max-time 120)
+    if [ -n "$C" ]; then
+        start_session
+        make_bundle
+        curl_opts+=(--cacert "$STATE/ca-bundle.pem")
+    fi
+    files=()
+    el=$(rpm -E '%{?rhel}' 2>/dev/null || true)
+    case "$el" in 9|10) ;; *) die "release-install supports EL9/EL10 only (rpm -E %rhel = '$el')" ;; esac
+    for name in "$@"; do
+        case "$name" in
+            epel) pkg=epel-release key=RPM-GPG-KEY-EPEL-$el
+                  path=/epel/epel-release-latest-$el.noarch.rpm
+                  origin=https://dl.fedoraproject.org/pub/epel/epel-release-latest-$el.noarch.rpm ;;
+            remi) pkg=remi-release key=RPM-GPG-KEY-remi2024
+                  path=/remi/enterprise/remi-release-$el.rpm
+                  origin=https://rpms.remirepo.net/enterprise/remi-release-$el.rpm ;;
+            *) die "unknown release package '$name'" ;;
+        esac
+        url="$origin"; [ -n "$C" ] && url="$C$path"
+        f="$tmp/$pkg.rpm"
+        curl "${curl_opts[@]}" -o "$f" "$url" || die "download failed: $url"
+        rpm --import "$SHARE/$key" || die "cannot import $SHARE/$key"
+        got=$(rpm -qp --nosignature --qf '%{NAME}' "$f" 2>/dev/null || true)
+        [ "$got" = "$pkg" ] || die "$url is '$got', expected $pkg"
+        chk=$(rpm -K "$f" 2>&1 || true)
+        [[ "$chk" == *": digests signatures OK" ]] || die "signature check failed for $url: $chk"
+        echo "pkg-cache-repos: $pkg verified ($chk) from $url"
+        files+=("$f")
+    done
+    dnf install -y --setopt=localpkg_gpgcheck=1 "${files[@]}"
+    rm -rf "$tmp"
+    exit 0
+fi
+
+if [ -z "$C" ] && [ "$MODE" != nodesource ] && [ "$MODE" != go-env ]; then
+    echo "pkg-cache-repos($MODE): PKG_CACHE not set - using upstream repositories"
+    exit 0
+fi
+
+rewrite_rpm() {
+    local f="$1"
+    grep -q -E '^#? ?baseurl=https?://(repo\.almalinux\.org|download\.example|rpms\.remirepo\.net|rpm\.nodesource\.com)/' "$f" || return 0
+    save "$f"
+    # Per [section]: a (commented or live) baseurl on a known origin becomes
+    # "baseurl=<cache>/<prefix>/<path> <origin>/<path>"; that section's
+    # mirrorlist/metalink is commented out (dnf prefers them over baseurl) and it
+    # gets sslcacert=<system bundle + pkg-cache CA>, gpgcheck=1 and, for repos
+    # that sign repomd.xml, repo_gpgcheck=1. Other sections are untouched.
+    awk -v C="$C" -v BUNDLE="$STATE/ca-bundle.pem" '
+        function flush(   i) {
+            for (i = 1; i <= n; i++) {
+                if (hit && buf[i] ~ /^(mirrorlist|metalink)=/) buf[i] = "#" buf[i]
+                if (hit && buf[i] ~ /^(sslcacert|gpgcheck|repo_gpgcheck)[ \t]*=/) continue
+                print buf[i]
+            }
+            if (hit) {
+                print "sslcacert=" BUNDLE
+                print "gpgcheck=1"
+                if (signed) print "repo_gpgcheck=1"
+            }
+            n = 0; hit = 0; signed = 0
+        }
+        function rw(line, re, prefix, origin, sig,   p) {
+            if (match(line, re)) {
+                p = substr(line, RSTART + RLENGTH); sub(/[ \t].*$/, "", p)
+                hit = 1; if (sig) signed = 1
+                return "baseurl=" C "/" prefix "/" p " " origin "/" p
+            }
+            return line
+        }
+        /^\[/ { flush() }
+        {
+            l = $0
+            if (l ~ /^#? ?baseurl=/) {
+                l = rw(l, "^#? ?baseurl=https?://repo\\.almalinux\\.org/almalinux/", "almalinux", "https://repo.almalinux.org/almalinux", 1)
+                l = rw(l, "^#? ?baseurl=https?://download\\.example/pub/epel/", "epel", "https://dl.fedoraproject.org/pub/epel", 0)
+                l = rw(l, "^#? ?baseurl=https?://rpms\\.remirepo\\.net/", "remi", "https://rpms.remirepo.net", 1)
+                # NodeSource signs its RPMs but not repomd.xml (404): gpgcheck only.
+                l = rw(l, "^#? ?baseurl=https?://rpm\\.nodesource\\.com/", "nodesource", "https://rpm.nodesource.com", 0)
+            }
+            buf[++n] = l
+        }
+        END { flush() }
+    ' "$f" > "$f.pkgcache" && cat "$f.pkgcache" > "$f" && rm -f "$f.pkgcache"
+}
+
+rewrite_apt() {
+    local f="$1"
+    grep -q -E 'https?://([a-z]{2}\.)?(archive|security)\.ubuntu\.com/|https?://rpms\.litespeedtech\.com/|https?://(deb|security)\.debian\.org/' "$f" || return 0
+    save "$f"
+    # Debian: debian-security first, and "debian" only as a whole path segment,
+    # so .../debian-security is never rewritten as /debian/.
+    sed -i -E \
+        -e "s%https?://(archive|[a-z]{2}\.archive)\.ubuntu\.com/ubuntu/?%$C/ubuntu/%g" \
+        -e "s%https?://security\.ubuntu\.com/ubuntu/?%$C/ubuntu-security/%g" \
+        -e "s%https?://rpms\.litespeedtech\.com/%$C/litespeed/%g" \
+        -e "s%https?://(deb|security)\.debian\.org/debian-security/?([[:space:]]|\$)%$C/debian-security/\\2%g" \
+        -e "s%https?://security\.debian\.org/?([[:space:]]|\$)%$C/debian-security/\\1%g" \
+        -e "s%https?://deb\.debian\.org/debian/?([[:space:]]|\$)%$C/debian/\\1%g" \
+        "$f"
+}
+
+if [ "$MODE" = go-env ]; then
+    # Prints shell exports for `eval`. Without PKG_CACHE: nothing (Go defaults).
+    # GOPROXY "a|b" falls through on ANY error, so a dead cache costs one
+    # failed request, not the build. GOSUMDB stays the default sum.golang.org,
+    # reached directly: modules are still verified against go.sum + the sumdb.
+    [ -n "$C" ] || { echo "pkg-cache-repos(go-env): PKG_CACHE not set - default GOPROXY" >&2; exit 0; }
+    start_session
+    make_bundle
+    printf "export GOPROXY='%s/goproxy|https://proxy.golang.org|direct' SSL_CERT_FILE='%s'\n" "$C" "$STATE/ca-bundle.pem"
+    echo "pkg-cache-repos(go-env): GOPROXY via $C/goproxy (fallback proxy.golang.org)" >&2
+    exit 0
+fi
+
+if [ "$MODE" = nodesource ]; then
+    # Replaces `curl https://rpm.nodesource.com/setup_N.x | bash` (unverified
+    # remote script): writes the same nodesource-nodejs repo (origin URL, so the
+    # shipped image keeps a normal repo file) with gpgcheck=1 and the signing key
+    # committed in keys/pkg-cache/; with PKG_CACHE it is then rewritten like any
+    # other repo and `restore` puts the origin version back.
+    major="${1:?usage: pkg-cache-repos.sh nodesource MAJOR}"
+    [[ "$major" =~ ^[0-9]{1,3}$ ]] || die "bad Node major '$major'"
+    key="$SHARE/RPM-GPG-KEY-nodesource"
+    [ -s "$key" ] || die "missing $key"
+    rpm --import "$key" || die "cannot import $key"
+    rm -f /etc/yum.repos.d/nodesource*.repo
+    f=/etc/yum.repos.d/nodesource-nodejs.repo
+    printf '%s\n' "[nodesource-nodejs]" \
+        "name=Node.js Packages for Linux RPM based distros - \$basearch" \
+        "baseurl=https://rpm.nodesource.com/pub_${major}.x/nodistro/nodejs/\$basearch" \
+        "priority=9" "enabled=1" "gpgcheck=1" \
+        "gpgkey=file://$key" "module_hotfixes=1" > "$f"
+    echo "pkg-cache-repos: wrote $f (Node ${major}.x, key $(basename "$key"))"
+    if [ -n "$C" ]; then
+        start_session
+        make_bundle
+        rewrite_rpm "$f"
+        grep -h '^baseurl=' "$f" | sed 's/^/  /'
+    fi
+    exit 0
+fi
+
+case "$MODE" in
+    rpm)
+        start_session
+        make_bundle
+        for f in /etc/yum.repos.d/*.repo; do [ -f "$f" ] && rewrite_rpm "$f"; done
+        echo "pkg-cache-repos(rpm): repos now via $C:"
+        grep -h '^baseurl=' /etc/yum.repos.d/*.repo | grep -F "$C/" | sort -u | sed 's/^/  /'
+        ;;
+    apt)
+        start_session
+        conf=/etc/apt/apt.conf.d/99pkg-cache
+        # Trust the pkg-cache CA for the cache host ONLY (apt per-host option).
+        printf 'Acquire::https::%s::CaInfo "%s";\n' "${HOSTPORT%%:*}" "$SHARE/ca.crt" > "$conf"
+        created "$conf"
+        for f in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
+            [ -f "$f" ] && rewrite_apt "$f"
+        done
+        echo "pkg-cache-repos(apt): sources now via $C:"
+        # (|| true: Debian 13 images have no /etc/apt/sources.list; grep's exit 2
+        # for the missing file must not fail the build under pipefail)
+        { grep -h -E '^(deb|URIs:)' /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null || true; } | grep -F "$C/" | sort -u | sed 's/^/  /' || true
+        ;;
+    *) die "unknown mode '$MODE'" ;;
+esac

errors/default-splash.html

@@ -0,0 +1,34 @@
+<!DOCTYPE html>
+<!--
+  Default splash: served by HAProxy itself (`http-request return ... file`,
+  hap_listener.tpl) for any request whose Host is not routed to a site --
+  the server's own hostname, a bare IP, an unconfigured name. Static on
+  purpose: no version, no hostname echo, no customer data, no links (in
+  particular none into the control panel). Loaded with `file`, not
+  `lf-file`, so `%` is literal here and nothing from the request is
+  expanded into it. Must stay well under tune.bufsize (16 KB) or HAProxy
+  rejects the config.
+-->
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="robots" content="noindex, nofollow">
+<title>Web Hosting Platform</title>
+<style>
+:root{color-scheme:light dark;--bg:#f4f5f7;--card:#fff;--fg:#1d2330;--muted:#5b6475;--line:#dfe3ea}
+@media (prefers-color-scheme:dark){:root{--bg:#0f1217;--card:#181c23;--fg:#e6e9ef;--muted:#9aa3b2;--line:#2a303a}}
+html,body{margin:0;height:100%}
+body{display:flex;align-items:center;justify-content:center;background:var(--bg);color:var(--fg);font:16px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;padding:16px;box-sizing:border-box}
+main{background:var(--card);border:1px solid var(--line);border-radius:10px;padding:32px 28px;max-width:480px;width:100%;text-align:center}
+h1{font-size:1.25rem;margin:0 0 .5rem}
+p{margin:0;color:var(--muted)}
+</style>
+</head>
+<body>
+<main>
+<h1>Web Hosting Platform</h1>
+<p>This server is operated by a web hosting platform. There is no website configured at this address.</p>
+</main>
+</body>
+</html>

errors/robots-unrouted.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /

haproxy_gunicorn_worker.py

@@ -0,0 +1,502 @@
+"""gunicorn gthread worker that cannot be wedged by silent or slow clients.
+
+Incident 2026-10-09 (sdbees), and very likely the 2026-07-07 whp01 one too
+-------------------------------------------------------------------------
+Both :8000 API workers had 7 of 8 pool threads parked in the kernel in
+``tcp_recvmsg -> sk_wait_data``: a blocking socket read with no timeout.
+/health and /api/blocked-ips hung for 40s+, the container stayed "healthy".
+
+haproxy_manager.py makes NO in-process outbound network reads (no requests,
+urllib, http.client, raw socket or ssl client anywhere; every ACME/HAProxy
+interaction is a subprocess, already bounded since 2026.07.3). The reads that
+blocked were gunicorn's own reads of the CLIENT socket:
+
+    gunicorn/workers/gthread.py  TConn.init():  self.sock.setblocking(True)
+
+Once the main loop sees a connection become readable it hands it to a pool
+thread and turns the socket fully blocking -- timeout None -- before parsing
+the request. Any peer that sends a few bytes and then waits (a TLS ClientHello
+sent to this plain-HTTP port, a port scanner, a half-written request) pins
+that pool thread forever. :8000 was published on 0.0.0.0 by the deploy
+(`-p 8000:8000`, which bypasses firewalld) and internet scanners were measured
+connecting to it. 4 such connections exhaust a worker's pool.
+
+gunicorn's --timeout does not catch this: for gthread it only checks that the
+worker's MAIN thread keeps heart-beating, and the main thread keeps polling
+happily while every pool thread is stuck. --max-requests does not help either:
+a worker whose threads never finish a request never reaches the limit.
+socket.setdefaulttimeout() does not help: setblocking(True) explicitly resets
+the timeout to None, overriding any default.
+
+Stock gthread has two more exhaustion paths on the same port (review F1/F2):
+
+* A connection that NEVER sends a byte never becomes readable, so it is never
+  handed to a thread and never gets any deadline -- but it holds one of the
+  worker's ``worker_connections`` (1000) slots. At 1000 the run loop stops
+  polling altogether and only waits on futures; none exist, so even the
+  attacker's FINs are never seen and the worker is wedged PERMANENTLY, still
+  heart-beating, with no request in flight for a watchdog to notice.
+* Slowloris: one header byte every <timeout seconds never trips a per-recv
+  timeout, so a socket timeout alone does not bound header-read time.
+
+What this worker changes
+------------------------
+1. CLIENT_TIMEOUT (HAPROXY_MGR_CLIENT_TIMEOUT, default 20s): every client
+   socket gets an explicit socket timeout at the exact point gthread makes it
+   blocking (re-applied on every keep-alive reuse). Python applies it to each
+   recv() call and to each WHOLE sendall() call -- gunicorn writes a response
+   chunk with one sendall, so any single chunk that takes longer than this to
+   transmit is cut off. Responses here are KB-sized JSON/PEMs to localhost, so
+   that never happens in practice. It does NOT bound how long the app takes to
+   handle a request (a long certbot run is a subprocess with its own timeout
+   and touches the client socket only once it is done).
+
+2. FIRST_BYTE_TIMEOUT (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, default 10s): an
+   accepted connection that has not become readable within this is closed by
+   the main loop, freeing its slot (review F1).
+
+3. HEADER_DEADLINE (HAPROXY_MGR_HEADER_DEADLINE, default 10s): total time a
+   client may take to deliver the request line + headers, counted from when
+   the connection was ENQUEUED for a thread, with a floor of
+   HAPROXY_MGR_HEADER_PICKUP_GRACE (1s) after a thread picks it up:
+   deadline = max(enqueue + 10s, pickup + 1s). Disarmed in handle_request()
+   (which gunicorn calls only once the parser has returned a complete
+   request). A 1s monitor loop shuts down (SHUT_RDWR, not close -- no fd-reuse
+   race with the blocked recv) the socket of any connection past it; the
+   blocked recv returns EOF and the thread is freed (review F2). A legit
+   request queued behind busy threads (certbot runs) already has its headers
+   in the kernel buffer and parses in milliseconds inside the 1s grace; a
+   slow client that sat in the queue gets only the grace, so reconnecting
+   drippers cannot starve the pool by queueing (re-review). Bodies are only
+   read by routes behind require_api_key, so the unauthenticated surface is
+   exactly this phase.
+
+4. Wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS, default 3600s; 0 disables). If
+   EVERY pool thread has been inside the same request for longer than that the
+   worker logs every thread's Python stack to the error log and exits; the
+   gunicorn master respawns it in about a second, without touching HAProxy or
+   the edge. 3600s because a multi-domain /api/certificates/request runs one
+   certbot (up to 180s) per domain; with (1)-(3) closing the known client-side
+   sources, this is only a backstop for blocking sources not yet found.
+
+5. Peer filter (re-review R1). haproxy-manager shares the client-net bridge
+   with every tenant container, so tenant PHP could reach haproxy-manager:8000
+   directly; the 127.0.0.1 publish only removes the internet path. On the
+   ports in HAPROXY_MGR_PEER_FILTER_PORTS (default "8000" -- NOT the :8080
+   default-page server, which HAProxy feeds over loopback) accept() closes
+   any connection whose peer is not in HAPROXY_MGR_API_ALLOWED_PEERS (default
+   "127.0.0.1,::1,gateway"; IPs/CIDRs; "gateway" = this container's default
+   gateway, which is where host callers arrive from through docker-proxy).
+   Refused before a thread or connection slot is used; logged at most once a
+   minute as a per-peer summary. Effective only once the edge publishes
+   127.0.0.1:8000 -- with a 0.0.0.0 publish a tenant can hairpin through the
+   host and arrive from the gateway too.
+
+6. pool_status() exposes busy/total threads and the oldest in-flight age so
+   /health can report saturation.
+
+This overrides gthread internals (enqueue_req, accept,
+on_client_socket_readable, murder_keepalived, handle, handle_request) as they
+are in gunicorn 23.0.0, which requirements.txt pins. A warning is logged if a
+different version is loaded; scripts/test-gunicorn-socket-timeouts.py runs
+real gunicorn in the Docker build gate and fails on behavioural drift.
+"""
+
+import errno
+import ipaddress
+import logging
+import os
+import selectors
+import socket
+import sys
+import threading
+import time
+import traceback
+from collections import deque
+from functools import partial
+
+import gunicorn
+from gunicorn.workers.gthread import TConn, ThreadWorker
+
+
+def _env_float(name, default):
+    try:
+        return float(os.environ.get(name, default))
+    except (TypeError, ValueError):
+        return float(default)
+
+
+CLIENT_TIMEOUT = _env_float('HAPROXY_MGR_CLIENT_TIMEOUT', 20)
+FIRST_BYTE_TIMEOUT = _env_float('HAPROXY_MGR_FIRST_BYTE_TIMEOUT', 10)
+HEADER_DEADLINE = _env_float('HAPROXY_MGR_HEADER_DEADLINE', 10)
+WEDGE_SECONDS = _env_float('HAPROXY_MGR_WEDGE_SECONDS', 3600)
+WATCHDOG_INTERVAL = _env_float('HAPROXY_MGR_WATCHDOG_INTERVAL', 15)
+MONITOR_TICK = 1.0
+# Once a thread picks a queued connection up it gets at least this long to
+# finish its headers, even if its enqueue-based deadline has already passed.
+HEADER_PICKUP_GRACE = _env_float('HAPROXY_MGR_HEADER_PICKUP_GRACE', 1)
+
+# Peer filter (review R1). Listener ports it applies to (comma-separated;
+# empty disables) and the peers allowed through. "gateway" means this
+# container's default gateway on its bridge network -- the address host-side
+# callers arrive from via the 127.0.0.1:8000 publish (docker-proxy) and via a
+# direct host->container-IP connect. Tenant containers on the same bridge
+# (client-net) arrive from their OWN addresses and are refused.
+PEER_FILTER_PORTS = os.environ.get('HAPROXY_MGR_PEER_FILTER_PORTS', '8000')
+ALLOWED_PEERS_SPEC = os.environ.get('HAPROXY_MGR_API_ALLOWED_PEERS', '127.0.0.1,::1,gateway')
+PEER_REJECT_LOG_INTERVAL = 60.0
+
+
+def _default_gateways(route_file='/proc/net/route'):
+    """IPv4 default gateway(s) from the kernel routing table."""
+    gws = []
+    try:
+        with open(route_file) as fh:
+            next(fh, None)
+            for line in fh:
+                f = line.split()
+                if len(f) >= 4 and f[1] == '00000000' and int(f[3], 16) & 0x2:
+                    gws.append(str(ipaddress.IPv4Address(int(f[2], 16).to_bytes(4, 'little'))))
+    except (OSError, ValueError):
+        pass
+    return gws
+
+
+def parse_allowed_peers(spec, gateways=None):
+    """Return (networks, problems). Tokens: IPs, CIDRs, or the word
+    'gateway'. Unparseable tokens are reported, never silently widened."""
+    nets, problems = [], []
+    for tok in (t.strip() for t in (spec or '').split(',')):
+        if not tok:
+            continue
+        if tok.lower() == 'gateway':
+            gws = _default_gateways() if gateways is None else gateways
+            if not gws:
+                problems.append('gateway: no IPv4 default route found')
+            for g in gws:
+                nets.append(ipaddress.ip_network(g + '/32'))
+            continue
+        try:
+            nets.append(ipaddress.ip_network(tok, strict=False))
+        except ValueError:
+            problems.append('unparseable entry %r' % tok)
+    return nets, problems
+
+
+def peer_allowed(addr, nets):
+    try:
+        ip = ipaddress.ip_address(addr.split('%', 1)[0])
+    except (ValueError, AttributeError):
+        return False
+    if ip.version == 6 and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    return any(ip.version == n.version and ip in n for n in nets)
+
+TESTED_GUNICORN_VERSION = '23.0.0'
+
+# Exit status for a watchdog-killed worker. Must NOT be 3 (WORKER_BOOT_ERROR)
+# or 4 (APP_LOAD_ERROR): on those the gunicorn master halts the whole server
+# instead of respawning the worker.
+WEDGED_EXIT_CODE = 1
+
+_current_worker = None
+
+
+def pool_status():
+    """Busy/total threads of the worker serving the caller, or None when the
+    app is not running under this worker class (e.g. tests, dev server)."""
+    w = _current_worker
+    if w is None:
+        return None
+    return w.pool_status()
+
+
+class _QuietClientTimeout(logging.Filter):
+    """gthread logs any OSError without an errno it recognises as
+    log.exception("Socket error processing request.") -- a full traceback.
+    A client timeout is now an expected, bounded event (scanners hit this port
+    constantly), so reduce it to one line instead of flooding the error log."""
+
+    def filter(self, record):
+        exc = record.exc_info[1] if record.exc_info else None
+        if isinstance(exc, TimeoutError) and \
+                str(record.msg).startswith('Socket error processing request'):
+            record.msg = ('Closed client connection idle for more than %ss '
+                          '(HAPROXY_MGR_CLIENT_TIMEOUT)' % CLIENT_TIMEOUT)
+            record.args = ()
+            record.exc_info = None
+            record.exc_text = None
+            record.levelno = logging.INFO
+            record.levelname = 'INFO'
+        return True
+
+
+class _InFlight:
+    __slots__ = ('conn', 'start', 'header_deadline')
+
+    def __init__(self, conn, start, header_deadline):
+        self.conn = conn
+        self.start = start
+        self.header_deadline = header_deadline
+
+
+class TimeoutThreadWorker(ThreadWorker):
+    """gthread worker with client-socket timeouts, a first-byte deadline, a
+    total header deadline and a wedge watchdog. See module docstring."""
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._inflight = {}          # thread ident -> _InFlight
+        self._inflight_lock = threading.Lock()
+        self._pending = deque()      # accepted, not yet readable (FIFO by deadline)
+        self._filter_ports = set()
+        for tok in PEER_FILTER_PORTS.split(','):
+            tok = tok.strip()
+            if tok.isdigit():
+                self._filter_ports.add(int(tok))
+        self._allowed_nets = []
+        self._rejected = {}          # peer ip -> count since last log line
+        self._rejected_lock = threading.Lock()
+        self._last_reject_log = 0.0
+
+    # -- (1) client socket timeout ------------------------------------------
+    def enqueue_req(self, conn):
+        # Same three steps as ThreadWorker.enqueue_req, with the timeout set
+        # right after conn.init() -- init() is what calls setblocking(True)
+        # (== settimeout(None)), and it runs again on every keep-alive reuse,
+        # so the timeout must be re-applied here each time, not at accept().
+        conn.init()
+        if CLIENT_TIMEOUT > 0:
+            conn.sock.settimeout(CLIENT_TIMEOUT)
+        conn.whp_enqueued = time.monotonic()   # header deadline counts from here
+        fs = self.tpool.submit(self.handle, conn)
+        self._wrap_future(fs, conn)
+
+    # -- (2) first-byte deadline for never-readable connections ---------------
+    def accept(self, server, listener):
+        # ThreadWorker.accept (23.0.0) plus: a deadline, and the conn is
+        # tracked in self._pending so murder_keepalived() can reap it.
+        try:
+            sock, client = listener.accept()
+            if self._filter_ports and self._server_port(server) in self._filter_ports \
+                    and not peer_allowed(client[0] if isinstance(client, tuple) else '',
+                                         self._allowed_nets):
+                # Refused before it costs a TConn, a connection slot, a poller
+                # registration or a thread.
+                sock.close()
+                self._note_rejected(client[0] if isinstance(client, tuple) else str(client))
+                return
+            conn = TConn(self.cfg, sock, client, server)
+            conn.timeout = time.time() + FIRST_BYTE_TIMEOUT
+
+            self.nr_conns += 1
+            with self._lock:
+                self._pending.append(conn)
+                self.poller.register(conn.sock, selectors.EVENT_READ,
+                                     partial(self.on_client_socket_readable, conn))
+        except OSError as e:
+            if e.errno not in (errno.EAGAIN, errno.ECONNABORTED,
+                               errno.EWOULDBLOCK):
+                raise
+
+    @staticmethod
+    def _server_port(server):
+        try:
+            return int(server[1])
+        except (TypeError, IndexError, ValueError):
+            return None
+
+    def _note_rejected(self, peer):
+        with self._rejected_lock:
+            self._rejected[peer] = self._rejected.get(peer, 0) + 1
+        if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
+            self._flush_rejected()
+
+    def _flush_rejected(self):
+        with self._rejected_lock:
+            if not self._rejected:
+                return
+            items, self._rejected = self._rejected, {}
+            self._last_reject_log = time.monotonic()
+        total = sum(items.values())
+        top = sorted(items.items(), key=lambda kv: -kv[1])[:5]
+        self.log.warning(
+            'Refused %d connection(s) to the management API from %d non-allowed peer(s) '
+            '(HAPROXY_MGR_API_ALLOWED_PEERS=%s): %s%s', total, len(items), ALLOWED_PEERS_SPEC,
+            ', '.join('%s x%d' % kv for kv in top), ' ...' if len(items) > 5 else '')
+
+    def on_client_socket_readable(self, conn, client):
+        # ThreadWorker version only knows about self._keep; a first request
+        # comes off self._pending instead.
+        with self._lock:
+            self.poller.unregister(client)
+            queue = self._keep if conn.initialized else self._pending
+            try:
+                queue.remove(conn)
+            except ValueError:
+                # already reaped (race with murder_keepalived)
+                return
+        self.enqueue_req(conn)
+
+    def murder_keepalived(self):
+        # Runs on every pass of ThreadWorker.run() -- including the branch
+        # taken when nr_conns >= worker_connections, where the poller is not
+        # serviced. Reaping here is what breaks that permanent deadlock.
+        super().murder_keepalived()
+        if FIRST_BYTE_TIMEOUT <= 0:
+            return
+        now = time.time()
+        while True:
+            with self._lock:
+                if not self._pending or self._pending[0].timeout > now:
+                    break
+                conn = self._pending.popleft()
+                try:
+                    self.poller.unregister(conn.sock)
+                except (OSError, KeyError, ValueError):
+                    pass
+            self.nr_conns -= 1
+            conn.close()
+
+    # -- (3) header deadline + busy tracking ----------------------------------
+    def handle(self, conn):
+        ident = threading.get_ident()
+        now = time.monotonic()
+        deadline = None
+        if HEADER_DEADLINE > 0:
+            # max(enqueue + D, pickup + grace): a legit request that waited in
+            # the queue behind busy threads already has its headers in the
+            # kernel buffer and parses in milliseconds; a slow client that
+            # waited gets only the grace, not a fresh full deadline (review:
+            # 40 reconnecting drippers otherwise starved legit requests).
+            enq = getattr(conn, 'whp_enqueued', now)
+            deadline = max(enq + HEADER_DEADLINE, now + HEADER_PICKUP_GRACE)
+        entry = _InFlight(conn, now, deadline)
+        with self._inflight_lock:
+            self._inflight[ident] = entry
+        try:
+            return super().handle(conn)
+        finally:
+            with self._inflight_lock:
+                self._inflight.pop(ident, None)
+
+    def handle_request(self, req, conn):
+        # The parser has produced a complete request line + headers.
+        entry = self._inflight.get(threading.get_ident())
+        if entry is not None:
+            entry.header_deadline = None
+        return super().handle_request(req, conn)
+
+    def _sweep_header_deadlines(self):
+        now = time.monotonic()
+        with self._inflight_lock:
+            late = [e for e in self._inflight.values()
+                    if e.header_deadline is not None and e.header_deadline < now]
+        for e in late:
+            e.header_deadline = None   # one shutdown per connection
+            try:
+                # shutdown, not close: wakes the blocked recv() with EOF
+                # without releasing the fd while another thread is in it.
+                e.conn.sock.shutdown(socket.SHUT_RDWR)
+            except OSError:
+                pass
+        if late:
+            self.log.info('Cut %d client connection(s) still sending request headers '
+                          'after %ss (HAPROXY_MGR_HEADER_DEADLINE)', len(late), HEADER_DEADLINE)
+
+    def pool_status(self):
+        now = time.monotonic()
+        with self._inflight_lock:
+            starts = [e.start for e in self._inflight.values()]
+        return {
+            'pid': os.getpid(),
+            'threads': self.cfg.threads,
+            'busy': len(starts),
+            'oldest_busy_seconds': round(now - min(starts), 1) if starts else 0.0,
+            'client_timeout_seconds': CLIENT_TIMEOUT,
+        }
+
+    # -- (4) watchdog ---------------------------------------------------------
+    def _wedged_for(self):
+        """Seconds the pool has been completely wedged (every thread busy in
+        its current request), measured from the YOUNGEST in-flight request.
+        0 when at least one thread is free."""
+        with self._inflight_lock:
+            starts = [e.start for e in self._inflight.values()]
+        if len(starts) < self.cfg.threads:
+            return 0.0
+        return time.monotonic() - max(starts)
+
+    def _dump_stacks(self):
+        frames = sys._current_frames()
+        names = {t.ident: t.name for t in threading.enumerate()}
+        out = []
+        for ident, frame in frames.items():
+            out.append('--- thread %s (%s)\n%s' % (
+                ident, names.get(ident, '?'), ''.join(traceback.format_stack(frame))))
+        return '\n'.join(out)
+
+    def _check_wedged(self):
+        wedged = self._wedged_for()
+        if WEDGE_SECONDS > 0 and wedged > WEDGE_SECONDS:
+            try:
+                self.log.critical(
+                    'WEDGED WORKER (pid %s): all %s threads busy in the same request '
+                    'for %.0fs > HAPROXY_MGR_WEDGE_SECONDS=%s. Exiting so the master '
+                    'respawns it. Thread stacks:\n%s',
+                    os.getpid(), self.cfg.threads, wedged, WEDGE_SECONDS,
+                    self._dump_stacks())
+                for h in logging.getLogger('gunicorn.error').handlers:
+                    try:
+                        h.flush()
+                    except Exception:
+                        pass
+            finally:
+                # os._exit, not SIGTERM: a graceful exit joins the (stuck,
+                # non-daemon) pool threads at interpreter shutdown and would
+                # hang until the master's --timeout escalates to SIGKILL.
+                os._exit(WEDGED_EXIT_CODE)
+
+    def _monitor(self):
+        """One daemon thread: header-deadline sweep every second, wedge check
+        every WATCHDOG_INTERVAL. Never allowed to die silently."""
+        last_wedge_check = time.monotonic()
+        while self.alive:
+            time.sleep(MONITOR_TICK)
+            try:
+                if HEADER_DEADLINE > 0:
+                    self._sweep_header_deadlines()
+                if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
+                    self._flush_rejected()
+                if time.monotonic() - last_wedge_check >= WATCHDOG_INTERVAL:
+                    last_wedge_check = time.monotonic()
+                    self._check_wedged()
+            except Exception:
+                try:
+                    self.log.exception('haproxy_gunicorn_worker monitor error (continuing)')
+                except Exception:
+                    pass
+
+    def init_process(self):
+        global _current_worker
+        _current_worker = self
+        try:
+            self.log.error_log.addFilter(_QuietClientTimeout())
+        except AttributeError:
+            pass
+        if gunicorn.__version__ != TESTED_GUNICORN_VERSION:
+            self.log.warning(
+                'haproxy_gunicorn_worker overrides gthread internals as of gunicorn %s; '
+                'running %s -- re-run scripts/test-gunicorn-socket-timeouts.py',
+                TESTED_GUNICORN_VERSION, gunicorn.__version__)
+        if self._filter_ports:
+            self._allowed_nets, problems = parse_allowed_peers(ALLOWED_PEERS_SPEC)
+            for prob in problems:
+                self.log.error('HAPROXY_MGR_API_ALLOWED_PEERS: %s -- that entry allows nothing '
+                               '(fail closed)', prob)
+            self.log.info('Management API peer filter on port(s) %s: allowing %s',
+                          ','.join(str(p) for p in sorted(self._filter_ports)),
+                          ', '.join(str(n) for n in self._allowed_nets) or 'NOTHING')
+        threading.Thread(target=self._monitor, name='worker-monitor',
+                         daemon=True).start()
+        super().init_process()   # runs the event loop; does not return

haproxy_manager.py

@@ -7,8 +7,9 @@ import jinja2
 import socket
 import psutil
 import functools
+import hmac
 import logging
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
 import json
 import ipaddress
 import shutil
@@ -51,6 +52,28 @@ def _bounded_subprocess_run(*args, **kwargs):
 
 subprocess.run = _bounded_subprocess_run
 
+# ---------------------------------------------------------------------------
+# Default socket timeout (incident 2026-10-09) -- defence in depth only
+# ---------------------------------------------------------------------------
+# This module makes no outbound network calls today (ACME, HAProxy's admin
+# socket etc. all go through bounded subprocesses above). This default makes
+# sure any socket created in-process later -- a urllib/http.client call added
+# to some handler -- cannot block a worker thread forever by omission.
+#
+# It does NOT protect the API's CLIENT connections: gunicorn's gthread worker
+# calls sock.setblocking(True) on each one, which resets the timeout to None
+# and overrides this default. That is what actually wedged the pool on
+# 2026-10-09; it is fixed by haproxy_gunicorn_worker.TimeoutThreadWorker.
+DEFAULT_SOCKET_TIMEOUT = float(os.environ.get('HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT', '60'))
+if DEFAULT_SOCKET_TIMEOUT > 0:
+    socket.setdefaulttimeout(DEFAULT_SOCKET_TIMEOUT)
+
+try:
+    from haproxy_gunicorn_worker import pool_status as _gunicorn_pool_status
+except Exception:  # dev server / tests without gunicorn
+    def _gunicorn_pool_status():
+        return None
+
 app = Flask(__name__)
 
 # Default page server (port 8080) — served to HAProxy clients whose request hit
@@ -112,6 +135,16 @@ BLOCKED_IPS_MAP_BACKUP_PATH = '/etc/haproxy/blocked_ips.map.backup'
 # while leaving a broken coraza-spoe.cfg behind still fails validation.
 CORAZA_SPOE_CONFIG_PATH = '/etc/haproxy/coraza-spoe.cfg'
 CORAZA_SPOE_BACKUP_PATH = '/etc/haproxy/coraza-spoe.cfg.backup'
+# Hosts the frontend routes to a site (default splash, hap_listener.tpl). The
+# paths are hardcoded in the template's `-f` ACLs, like blocked_ips.map, and
+# written by generate_config() from the same rows as the routing ACLs.
+# HAProxy's access-log target: the receiver's socket, CHROOT-RELATIVE
+# (hap_header.tpl `chroot /var/lib/haproxy`), i.e. /var/lib/haproxy/log.sock.
+DEFAULT_LOG_TARGET = '/log.sock'
+ROUTED_HOSTS_PATH = '/etc/haproxy/routed_hosts.list'
+ROUTED_HOSTS_BACKUP_PATH = '/etc/haproxy/routed_hosts.list.backup'
+ROUTED_WILDCARDS_PATH = '/etc/haproxy/routed_wildcards.list'
+ROUTED_WILDCARDS_BACKUP_PATH = '/etc/haproxy/routed_wildcards.list.backup'
 HAPROXY_SOCKET_PATH = '/var/run/haproxy.sock'
 # HAProxy loads this path as a DIRECTORY (`bind ... ssl crt /etc/haproxy/certs`),
 # which means it tries to load EVERY file it finds in here. Nothing but final,
@@ -143,7 +176,12 @@ def require_api_key(f):
     def decorated_function(*args, **kwargs):
         if API_KEY:
             auth_header = request.headers.get('Authorization')
-            if not auth_header or auth_header != f'Bearer {API_KEY}':
+            # Constant-time comparison so response timing doesn't leak how
+            # much of a guessed key matched. Compared as bytes because
+            # compare_digest rejects non-ASCII str.
+            if not auth_header or not hmac.compare_digest(
+                    auth_header.encode('utf-8', 'surrogateescape'),
+                    f'Bearer {API_KEY}'.encode('utf-8', 'surrogateescape')):
                 return jsonify({'error': 'Unauthorized - Invalid or missing API key'}), 401
         return f(*args, **kwargs)
     return decorated_function
@@ -570,6 +608,19 @@ def backup_existing_pem(dest_path):
         return None
 
 
+def leaf_is_expired(path):
+    """True only when openssl positively says the first certificate in `path`
+    has expired. Unreadable/missing openssl returns False: those cases are
+    rejected, with a better message, by validate_pem_bundle()."""
+    try:
+        proc = subprocess.run(
+            ['openssl', 'x509', '-checkend', '0', '-noout', '-in', path],
+            capture_output=True, text=True, stdin=subprocess.DEVNULL)
+    except Exception:
+        return False
+    return proc.returncode == 1 and 'will expire' in (proc.stdout or '')
+
+
 def publish_pem_bundle(dest_path, source_paths):
     """Publish cert+key as a combined PEM at dest_path, atomically.
 
@@ -611,6 +662,15 @@ def publish_pem_bundle(dest_path, source_paths):
         raise CertificatePublishError(
             f'assembled bundle for {dest_path} is not usable: {msg}')
 
+    # Never publish an expired certificate. HAProxy loads the crt directory
+    # wholesale and serves the first file that claims an SNI name, so an
+    # expired file in there can only ever shadow a valid one (brain-jar.com.pem
+    # did exactly that to brainjar.net on 2026-10-07).
+    if leaf_is_expired(source_paths[0]):
+        raise CertificatePublishError(
+            f'refusing to publish {dest_path}: the certificate in '
+            f'{source_paths[0]} has expired')
+
     dest_dir = os.path.dirname(dest_path) or '.'
     try:
         os.makedirs(dest_dir, exist_ok=True)
@@ -741,17 +801,49 @@ def health_check():
             cursor.execute('SELECT 1')
             cursor.fetchone()
 
-        return jsonify({
+        body = {
             'status': 'healthy',
             'haproxy_status': 'running' if haproxy_running else 'stopped',
             'database': 'connected'
-        }), 200
+        }
+        # Pool saturation of the worker that answered ('busy' includes this
+        # probe). Leaking threads show as busy/oldest_busy_seconds climbing;
+        # a fully wedged worker cannot answer at all, which is what the
+        # healthcheck's short --max-time catches.
+        pool = _gunicorn_pool_status()
+        if pool is not None:
+            body['worker'] = pool
+        return jsonify(body), 200
     except Exception as e:
         return jsonify({
             'status': 'unhealthy',
             'error': str(e)
         }), 500
 
+# What this edge supports. WHP's orchestrator only switches to whole-site
+# re-bundling with unreachable_names when it sees these (HARD RULE: the WHP side
+# must not rely on 2026.10.3 behaviour against an older image, whose CN-only
+# quarantine + `certbot delete` would act on the larger bundles).
+EDGE_CAPABILITIES = ['bundle_renewed_flag', 'san_overlap_retirement',
+                     'unreachable_names', 'retired_lineage_tree']
+
+
+def _image_version():
+    for path in ('/haproxy/VERSION', os.path.join(os.path.dirname(os.path.abspath(__file__)), 'VERSION')):
+        try:
+            with open(path) as fh:
+                return fh.read().strip()
+        except OSError:
+            continue
+    return 'unknown'
+
+
+@app.route('/api/capabilities', methods=['GET'])
+@require_api_key
+def get_capabilities():
+    return jsonify({'version': _image_version(), 'capabilities': EDGE_CAPABILITIES})
+
+
 @app.route('/api/regenerate', methods=['GET'])
 @require_api_key
 def regenerate_conf():
@@ -972,131 +1064,371 @@ def request_ssl():
         log_operation('request_ssl', False, str(e))
         return jsonify({'status': 'error', 'message': str(e)}), 500
 
-def _quarantine_superseded_certs(keep_path, keep_lineage, bundle_names):
+# ---------------------------------------------------------------------------
-    """Move aside cert files that the just-issued bundle supersedes.
+# Superseded-certificate retirement (incident 2026-10-07, brainjar.net)
+# ---------------------------------------------------------------------------
+# HAProxy loads /etc/haproxy/certs as a DIRECTORY and, when two files claim the
+# same SNI name, serves whichever it loaded first. So a new bundle has no effect
+# while an older file that also lists one of its names is still in there.
+#
+# brainjar.net served an EXPIRED certificate for two days for exactly that
+# reason. The orchestrator re-bundled brainjar.net and its seven siblings
+# without the dead brain-jar.com, but the cleanup that was meant to move the old
+# file aside matched on the old certificate's CN only. Its CN was brain-jar.com,
+# which was not in the new bundle, so brain-jar.com.pem (which also listed all
+# eight brainjar names) stayed put and kept winning the SNI match. On top of
+# that, renew-certificates.sh re-published the old lineage from live/ every 12
+# hours, so moving the file alone would not have stuck.
+#
+# What retirement does now:
+#   * any OTHER pem/lineage that shares ANY name with the new bundle (CN or
+#     SAN) is a candidate, not only one whose CN is in the bundle;
+#   * a candidate is moved aside only if no name that this edge still serves
+#     would lose its last valid certificate (the shared-PEM hazard: one bundle
+#     routinely serves several sites' names, see domains_referencing_cert_path);
+#   * nothing is deleted. The pem goes to the cert-retired sibling directory and
+#     the lineage's live/<name>, archive/<name> and renewal/<name>.conf go
+#     together to /etc/letsencrypt/retired/<stamp>/ (retire_lineage_tree), so
+#     certbot stops renewing it, renew-certificates.sh stops re-publishing it,
+#     and the name can still be issued again later. Every move is logged and
+#     can be reversed with mv.
+#     (This used to `certbot delete` the lineage, which is irreversible.)
 
-    A `.pem` in /etc/haproxy/certs/ is "superseded" iff its certificate's CN
+# Where certbot keeps its lineages. A module global so the tests can repoint
-    is one of the bundle's names AND the file isn't the bundle's own combined
+# it; find_certbot_live_dir() predates it and still hardcodes the live path.
-    file. We don't look at SANs of the OLD certs — being the CN is enough,
+LETSENCRYPT_DIR = '/etc/letsencrypt'
-    since that's what HAProxy SNI-matches against and what the file
-    convention names it after.
 
-    This used to os.remove() the file and immediately `certbot delete` the
-    lineage, both BEFORE anything had checked that the replacement bundle was
-    usable — destroying the only two copies of a working certificate on the
-    strength of a `cat` whose exit status nobody read. Now:
 
-      * the superseded file is MOVED to the cert backup directory instead of
+def cert_retired_dir():
-        unlinked, so an operator can put it back by hand;
+    """Where retired .pem files are moved. Sibling of the crt dir, never inside."""
-      * the certbot lineage is left alone here. Deleting it is irreversible
+    return _cert_sibling_dir('cert-retired')
-        (archive, live and renewal config all go) and recovering means fresh,
-        rate-limited ACME orders, so it happens only in
-        _delete_superseded_lineages(), after HAProxy has actually loaded the
-        replacement.
 
-    Moving still solves the problem the removal existed for: the old file no
-    longer sits in the crt directory shadowing the new bundle's SNI match.
 
-    Returns a summary dict for logging / API response. Entries in 'removed'
+def parse_cert_text(text):
-    carry the lineage name for the deletion phase.
+    """Parse `openssl x509 -noout -subject -nameopt multiline -enddate
+    -fingerprint -sha256 -ext subjectAltName` output.
+
+    Returns {'cn', 'names', 'not_after', 'fingerprint'}; names is the lower-cased
+    set of the CN plus every DNS SAN. not_after is a UTC epoch or None.
     """
-    summary = {'removed': [], 'errors': [], 'skipped': []}
+    cn = None
+    names = set()
+    not_after = None
+    fingerprint = None
+    in_san = False
+    for raw in text.splitlines():
+        line = raw.strip()
+        if line.startswith('commonName') and '=' in line and cn is None:
+            cn = line.split('=', 1)[1].strip().lower()
+            continue
+        if line.startswith('notAfter='):
+            try:
+                not_after = int(datetime.strptime(
+                    ' '.join(line.split('=', 1)[1].split()),
+                    '%b %d %H:%M:%S %Y %Z').replace(
+                        tzinfo=timezone.utc).timestamp())
+            except ValueError:
+                not_after = None
+            continue
+        if '=' in line and line.lower().split('=', 1)[0].endswith('fingerprint'):
+            fingerprint = line.split('=', 1)[1].strip().upper()
+            continue
+        if line.startswith('X509v3 Subject Alternative Name'):
+            in_san = True
+            continue
+        if in_san:
+            for part in line.split(','):
+                part = part.strip()
+                if part.startswith('DNS:'):
+                    names.add(part[4:].strip().lower())
+            in_san = False
+    if cn:
+        names.add(cn)
+    return {'cn': cn, 'names': names, 'not_after': not_after,
+            'fingerprint': fingerprint}
 
+
+def read_cert_info(path):
+    """parse_cert_text() for the first certificate in `path`, or None."""
+    try:
+        proc = subprocess.run(
+            ['openssl', 'x509', '-in', path, '-noout', '-subject',
+             '-nameopt', 'multiline', '-enddate', '-fingerprint', '-sha256',
+             '-ext', 'subjectAltName'],
+            capture_output=True, text=True, stdin=subprocess.DEVNULL)
+    except Exception:
+        return None
+    if proc.returncode != 0:
+        return None
+    info = parse_cert_text(proc.stdout)
+    return info if info['names'] else None
+
+
+def cert_name_covers(cert_name, host):
+    """Does certificate name `cert_name` (maybe `*.x`) cover hostname `host`?"""
+    cert_name = (cert_name or '').lower()
+    host = (host or '').lower()
+    if cert_name == host:
+        return True
+    if cert_name.startswith('*.') and not host.startswith('*.'):
+        suffix = cert_name[1:]  # ".example.com"
+        head = host[:-len(suffix)] if host.endswith(suffix) else ''
+        return bool(head) and '.' not in head
+    return False
+
+
+def _cert_covers(names, host):
+    return any(cert_name_covers(n, host) for n in names)
+
+
+def _served_names():
+    """Every hostname this edge has a route for (the SQLite domains table)."""
+    try:
+        with sqlite3.connect(DB_FILE) as conn:
+            return {row[0].lower() for row in
+                    conn.execute('SELECT domain FROM domains').fetchall()
+                    if row[0]}
+    except Exception as e:
+        # Unknown served set -> treat every name as served, i.e. never retire a
+        # still-valid certificate on the strength of a failed query.
+        logger.error("Could not read the domains table for retirement checks: %s", e)
+        return None
+
+
+def retirement_blockers(candidate, others, bundle_names, served_names,
+                        unreachable_names, now=None):
+    """Names that would lose their LAST valid certificate if `candidate` went.
+
+    candidate/others are read_cert_info() dicts. `others` are the certificates
+    that would remain in the crt directory. Pure, so it is unit-testable.
+
+    * An expired candidate blocks nothing: it is not a valid certificate for
+      anyone, and while it sits in the crt dir it can only shadow a good one.
+    * A name the caller asserts is not served here (unreachable_names - DNS
+      points elsewhere or nowhere) blocks nothing.
+    * A name with no route on this edge blocks nothing.
+    * served_names=None (could not read the DB) means "assume everything is
+      served", so a valid candidate is kept unless the bundle covers it all.
+    """
+    now = time.time() if now is None else now
+    if candidate.get('not_after') is None:
+        return ['<unreadable expiry>']
+    if candidate['not_after'] <= now:
+        return []
+    unreachable = {n.lower() for n in (unreachable_names or ())}
+    bundle = {n.lower() for n in bundle_names}
+    if served_names is None:
+        candidates_to_check = set(candidate['names'])
+    else:
+        candidates_to_check = {s for s in served_names
+                               if _cert_covers(candidate['names'], s)}
+    blockers = []
+    for name in sorted(candidates_to_check):
+        if name in unreachable or _cert_covers(bundle, name):
+            continue
+        covered_elsewhere = any(
+            o is not candidate
+            and o.get('fingerprint') != candidate.get('fingerprint')
+            and o.get('not_after') is not None and o['not_after'] > now
+            and _cert_covers(o['names'], name)
+            for o in others)
+        if not covered_elsewhere:
+            blockers.append(name)
+    return blockers
+
+
+def _overlaps_bundle(info, bundle_names):
+    return any(_cert_covers(info['names'], b) for b in bundle_names) or \
+        any(n in {b.lower() for b in bundle_names} for n in info['names'])
+
+
+def _unique_dest(directory, name):
+    dest = os.path.join(directory, name)
+    if not os.path.exists(dest):
+        return dest
+    stamp = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')
+    dest = os.path.join(directory, f'{name}.{stamp}')
+    n = 1
+    while os.path.exists(dest):
+        dest = os.path.join(directory, f'{name}.{stamp}.{n}')
+        n += 1
+    return dest
+
+
+def _quarantine_superseded_certs(keep_path, keep_lineage, bundle_names,
+                                 unreachable_names=()):
+    """Move aside .pem files in the crt dir that overlap the new bundle.
+
+    Runs after the new bundle is published and before HAProxy reloads, so the
+    replacement is never the only thing we have. See the block comment above
+    for the rules. Returns {'removed', 'kept', 'skipped', 'errors'}.
+    """
+    summary = {'removed': [], 'kept': [], 'skipped': [], 'errors': []}
     if not os.path.isdir(SSL_CERTS_DIR):
         return summary
 
     keep_basename = os.path.basename(keep_path)
+    served = _served_names()
 
+    # Inventory once; `remaining` shrinks as files are retired so the coverage
+    # check for the next candidate sees what will actually be left.
+    inventory = {}
     for fname in sorted(os.listdir(SSL_CERTS_DIR)):
         if not fname.endswith('.pem'):
             continue
+        info = read_cert_info(os.path.join(SSL_CERTS_DIR, fname))
+        if info is None:
+            summary['skipped'].append({'file': fname, 'reason': 'openssl could not read it'})
+            continue
+        inventory[fname] = info
+
+    remaining = dict(inventory)
+    for fname, info in inventory.items():
         if fname == keep_basename:
             continue
-        fpath = os.path.join(SSL_CERTS_DIR, fname)
+        if not _overlaps_bundle(info, bundle_names):
+            continue
+        others = [i for f, i in remaining.items() if f != fname]
+        blockers = retirement_blockers(info, others, bundle_names, served,
+                                       unreachable_names)
+        if blockers:
+            logger.warning(
+                "Not retiring %s although it overlaps the new %s bundle: it is "
+                "still the only valid certificate for %s", fname, keep_lineage,
+                ', '.join(blockers))
+            summary['kept'].append({'file': fname, 'blockers': blockers})
+            continue
         try:
-            cn_proc = subprocess.run(
+            os.makedirs(cert_retired_dir(), exist_ok=True)
-                ['openssl', 'x509', '-in', fpath, '-noout', '-subject', '-nameopt', 'multiline'],
+            dest = _unique_dest(cert_retired_dir(), fname)
-                capture_output=True, text=True
+            shutil.move(os.path.join(SSL_CERTS_DIR, fname), dest)
-            )
+            remaining.pop(fname, None)
-            if cn_proc.returncode != 0:
+            logger.warning(
-                summary['skipped'].append({'file': fname, 'reason': 'openssl read failed'})
+                "Retired superseded certificate %s (names: %s; expires %s) -> %s "
-                continue
+                "because it overlaps the new %s bundle",
-            # `-nameopt multiline` lays out the subject one RDN per line; CN is
+                fname, ', '.join(sorted(info['names'])),
-            # the row matching `commonName`. Robust against unusual subject orderings.
+                datetime.fromtimestamp(info['not_after'], timezone.utc).isoformat()
-            cn = None
+                if info['not_after'] else 'unknown', dest, keep_lineage)
-            for line in cn_proc.stdout.splitlines():
-                line = line.strip()
-                if line.startswith('commonName'):
-                    # format: "commonName                = example.com"
-                    parts = line.split('=', 1)
-                    if len(parts) == 2:
-                        cn = parts[1].strip()
-                    break
-            if not cn:
-                summary['skipped'].append({'file': fname, 'reason': 'no CN found'})
-                continue
-        except Exception as e:
-            summary['skipped'].append({'file': fname, 'reason': f'inspect failed: {e}'})
-            continue
-
-        if cn not in bundle_names:
-            continue  # not superseded — different domain group
-
-        # This file's CN is now part of our new bundle — supersede it.
-        lineage_name = fname[:-len('.pem')]
-        if lineage_name == keep_lineage:
-            # Defensive: shouldn't happen because of keep_basename check, but
-            # don't accidentally drop the lineage we just wrote.
-            continue
-
-        try:
-            os.makedirs(cert_backup_dir(), exist_ok=True)
-            quarantine_path = os.path.join(cert_backup_dir(), fname)
-            # Move, not unlink: out of the crt directory (so it stops shadowing
-            # the new bundle) but still on disk for manual recovery.
-            shutil.move(fpath, quarantine_path)
             summary['removed'].append({
                 'file': fname,
-                'cn': cn,
+                'cn': info['cn'],
-                'lineage': lineage_name,
+                'names': sorted(info['names']),
-                'moved_to': quarantine_path,
+                'expired': bool(info['not_after'] and info['not_after'] <= time.time()),
-                'lineage_deleted': False,
+                'moved_to': dest,
             })
         except Exception as e:
             summary['errors'].append({'file': fname, 'error': str(e)})
-
     return summary
 
 
-def _delete_superseded_lineages(summary):
+def retire_lineage_tree(lineage, stamp=None):
-    """`certbot delete` the lineages quarantined by _quarantine_superseded_certs().
+    """Move a certbot lineage COMPLETELY out of certbot's view, without deleting.
 
-    IRREVERSIBLE: certbot removes the lineage's archive, live symlinks and
+    live/<x>, archive/<x> and renewal/<x>.conf go together to
-    renewal config. If it turns out we needed that certificate, the only way
+    LETSENCRYPT_DIR/retired/<UTCstamp>/{live,archive,renewal}/. All three must
-    back is a fresh ACME order, which Let's Encrypt rate-limits — an outage
+    move: certbot's storage.RenewableCert.new_lineage() refuses to create a
-    measured in hours. So this is gated hardest of anything in this module: it
+    lineage while archive/<x> exists ("archive directory exists for <x>") - and
-    runs only after the replacement bundle has been assembled, validated,
+    it has already written an empty renewal/<x>.conf by then, which breaks every
-    published, AND loaded by a HAProxy that reloaded successfully.
+    later --cert-name <x> and every `certbot renew` parse (review H2, verified
+    against certbot 2.11). Moving only live/ and renewal/ (as the first version
+    of this did) therefore made the name permanently un-issuable.
 
-    Mutates `summary` in place. Best-effort: some files have no corresponding
+    The retired tree keeps the same depth as /etc/letsencrypt, so the relative
-    lineage (e.g. self-signed dev certs) and a failure here is harmless — a
+    live/<x>/*.pem -> ../../archive/<x>/*N.pem symlinks still resolve, and a
-    dead lineage merely wastes a renewal attempt on the next 12h cron tick.
+    restore is three mv's back. Returns {'retired_to': dir, 'moved': [...]}.
+    Raises on failure after moving back anything already moved.
     """
-    for entry in summary.get('removed', []):
+    stamp = stamp or datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')
-        lineage_name = entry.get('lineage')
+    root = os.path.join(LETSENCRYPT_DIR, 'retired', stamp)
-        if not lineage_name:
+    n = 1
-            continue
+    while any(os.path.exists(os.path.join(root, part, name)) for part, name in
+              (('live', lineage), ('archive', lineage), ('renewal', f'{lineage}.conf'))):
+        root = os.path.join(LETSENCRYPT_DIR, 'retired', f'{stamp}.{n}')
+        n += 1
+    plan = [(os.path.join(LETSENCRYPT_DIR, part, name), os.path.join(root, part, name))
+            for part, name in (('renewal', f'{lineage}.conf'), ('live', lineage),
+                               ('archive', lineage))]
+    moved = []
     try:
-            cb_proc = subprocess.run(
+        for src, dst in plan:
-                ['certbot', 'delete', '--cert-name', lineage_name, '-n'],
+            if os.path.lexists(src):
-                capture_output=True, text=True
+                os.makedirs(os.path.dirname(dst), exist_ok=True)
-            )
+                shutil.move(src, dst)
-            entry['lineage_deleted'] = (cb_proc.returncode == 0)
+                moved.append((src, dst))
-            if cb_proc.returncode != 0:
+    except Exception:
-                entry['certbot_stderr'] = (cb_proc.stderr or '').strip()[:200]
+        for src, dst in reversed(moved):
-        except Exception as e:
+            try:
-            entry['certbot_error'] = str(e)
+                shutil.move(dst, src)
+            except Exception as e:  # pragma: no cover - logged, nothing else to do
+                logger.critical("Could not restore %s from %s: %s", src, dst, e)
+        raise
+    return {'retired_to': root, 'moved': [d for _, d in moved]}
+
+
+def _retire_superseded_lineages(keep_lineage, bundle_names, unreachable_names=(),
+                                keep_fingerprint=None):
+    """Move overlapping certbot lineages out of certbot's tree (retire_lineage_tree).
+
+    Same overlap and safety rules as _quarantine_superseded_certs(), evaluated
+    against the pem files that are left in the crt dir. Runs only after HAProxy
+    has loaded the new bundle. keep_lineage is the bundle's --cert-name and is
+    never retired.
+    """
+    summary = {'retired': [], 'kept': [], 'errors': []}
+    live_root = os.path.join(LETSENCRYPT_DIR, 'live')
+    if not os.path.isdir(live_root):
         return summary
+    served = _served_names()
+
+    remaining_pems = []
+    if os.path.isdir(SSL_CERTS_DIR):
+        for fname in sorted(os.listdir(SSL_CERTS_DIR)):
+            if fname.endswith('.pem'):
+                info = read_cert_info(os.path.join(SSL_CERTS_DIR, fname))
+                if info is not None:
+                    info['_file'] = fname
+                    remaining_pems.append(info)
+
+    for lineage in sorted(os.listdir(live_root)):
+        lineage_dir = os.path.join(live_root, lineage)
+        # Hard guard: the bundle's own --cert-name lineage is never a candidate,
+        # whatever any other check says (review H1).
+        if lineage == keep_lineage or not os.path.isdir(lineage_dir):
+            continue
+        cert_file = os.path.join(lineage_dir, 'cert.pem')
+        if not os.path.exists(cert_file):
+            cert_file = os.path.join(lineage_dir, 'fullchain.pem')
+        info = read_cert_info(cert_file)
+        if info is None or not _overlaps_bundle(info, bundle_names):
+            continue
+        if keep_fingerprint and info.get('fingerprint') == keep_fingerprint:
+            continue
+        # The lineage's own published pem is not "another" certificate, even if
+        # its fingerprint drifted from the lineage (renewed, not republished).
+        others = [p for p in remaining_pems if p['_file'] != f'{lineage}.pem']
+        blockers = retirement_blockers(info, others, bundle_names, served,
+                                       unreachable_names)
+        if blockers:
+            logger.warning(
+                "Not retiring certbot lineage %s although it overlaps the new %s "
+                "bundle: it is still the only valid certificate for %s",
+                lineage, keep_lineage, ', '.join(blockers))
+            summary['kept'].append({'lineage': lineage, 'blockers': blockers})
+            continue
+        entry = {'lineage': lineage, 'names': sorted(info['names'])}
+        try:
+            entry.update(retire_lineage_tree(lineage))
+            logger.warning(
+                "Retired certbot lineage %s (names: %s) because it overlaps the "
+                "new %s bundle: live, archive and renewal conf moved to %s. "
+                "Restore with three mv's back.", lineage,
+                ', '.join(entry['names']), keep_lineage, entry['retired_to'])
+            summary['retired'].append(entry)
+        except Exception as e:
+            entry['error'] = str(e)
+            summary['errors'].append(entry)
+    return summary
+
 
 @app.route('/api/ssl/bundle', methods=['POST'])
 @require_api_key
@@ -1108,7 +1440,18 @@ def request_ssl_bundle():
     /api/ssl calls when a site has multiple domains.
 
     Body:
-      {"primary": "example.com", "sans": ["www.example.com", ...]}
+      {"primary": "example.com", "sans": ["www.example.com", ...],
+       "unreachable_names": ["dead.example.com", ...]}   # optional
+
+    unreachable_names: names the caller (WHP's orchestrator) has established do
+    not resolve to this host. They are left out of the bundle by the caller;
+    here they only tell superseded-certificate retirement that an older cert's
+    coverage of them is not worth keeping (see retirement_blockers()).
+
+    The response says whether certbot actually issued ("renewed") or found the
+    existing lineage not yet due and did nothing, and carries the published
+    certificate's not_after, so the caller can schedule from the real expiry
+    instead of recording a no-op as a renewal.
 
     The cert lineage uses --cert-name <primary>, so renewal under the same
     name doesn't proliferate -0001/-0002 dirs (the issue we hit with the
@@ -1146,6 +1489,13 @@ def request_ssl_bundle():
             return jsonify({'status': 'error', 'message': f'invalid SAN: {s!r}'}), 400
         cleaned_sans.append(s)
 
+    unreachable_names = data.get('unreachable_names') or []
+    if not isinstance(unreachable_names, list) or not all(
+            isinstance(n, str) for n in unreachable_names):
+        return jsonify({'status': 'error',
+                        'message': '"unreachable_names" must be a list of strings'}), 400
+    unreachable_names = [n.strip().lower() for n in unreachable_names if n.strip()]
+
     seen = {primary}
     names = [primary]
     for s in cleaned_sans:
@@ -1160,8 +1510,11 @@ def request_ssl_bundle():
             'message': f'Too many SANs ({len(names)}); Let\'s Encrypt limit is 100',
         }), 400
 
+    # --keep-until-expiring is what -n does anyway when the lineage exists with
+    # the same names; it is spelled out so the no-op case is deliberate, and it
+    # is detected below rather than reported as an issuance.
     cmd = [
-        'certbot', 'certonly', '-n', '--standalone',
+        'certbot', 'certonly', '-n', '--standalone', '--keep-until-expiring',
         '--preferred-challenges', 'http', '--http-01-port=8688',
         '--cert-name', primary,
     ]
@@ -1172,6 +1525,17 @@ def request_ssl_bundle():
         # Defensive: clear any stale lock left by a SIGKILLed prior run.
         clear_stale_certbot_locks()
 
+        # The lineage is EXACTLY live/<primary>: certbot was given
+        # --cert-name <primary>, which is authoritative. find_certbot_live_dir()
+        # must NOT be used here - it returns the newest of <primary> and
+        # <primary>-NNNN, so after a "not yet due" no-op a stale, more recently
+        # renewed <primary>-0001 would be published as <primary>.pem and the
+        # real <primary> lineage would then be retired as "superseded" (review
+        # H1, reproduced with brainjar.net). fullchain.pem starts with the leaf.
+        lineage_dir = os.path.join(LETSENCRYPT_DIR, 'live', primary)
+        before_info = read_cert_info(os.path.join(lineage_dir, 'fullchain.pem')) \
+            if os.path.isdir(lineage_dir) else None
+
         result = subprocess.run(cmd, capture_output=True, text=True)
         if result.returncode != 0:
             stderr_excerpt = (result.stderr or '').strip()[:800]
@@ -1184,9 +1548,8 @@ def request_ssl_bundle():
                 'attempted_names': names,
             }), 500
 
-        # Locate the lineage. With --cert-name primary, this should be a
+        # --cert-name <primary> means the lineage is live/<primary>, exactly.
-        # stable directory name (no -NNNN suffix on the first issuance).
+        live_dir = lineage_dir if os.path.isdir(lineage_dir) else None
-        live_dir = find_certbot_live_dir(primary)
         if not live_dir:
             error_msg = f'Bundle issued but live dir not found for {primary}'
             log_operation('request_ssl_bundle', False, error_msg)
@@ -1196,6 +1559,20 @@ def request_ssl_bundle():
         key_path = os.path.join(live_dir, 'privkey.pem')
         combined_path = f'{SSL_CERTS_DIR}/{primary}.pem'
 
+        # Did certbot issue, or did it find the lineage not yet due? Measured by
+        # the certificate itself (fingerprint before vs after), with certbot's
+        # own wording as a second signal.
+        new_info = read_cert_info(cert_path)
+        output = (result.stdout or '') + (result.stderr or '')
+        certbot_not_due = ('not yet due for renewal' in output.lower()
+                           or 'no action taken' in output.lower())
+        if before_info is None:
+            renewed = new_info is not None
+        elif new_info is None:
+            renewed = not certbot_not_due
+        else:
+            renewed = new_info.get('fingerprint') != before_info.get('fingerprint')
+
         os.makedirs(SSL_CERTS_DIR, exist_ok=True)
         try:
             publish_pem_bundle(combined_path, [cert_path, key_path])
@@ -1237,6 +1614,7 @@ def request_ssl_bundle():
             keep_path=combined_path,
             keep_lineage=primary,
             bundle_names=set(names),
+            unreachable_names=unreachable_names,
         )
 
         # Raises if the config does not validate or HAProxy does not reload;
@@ -1249,26 +1627,47 @@ def request_ssl_bundle():
                 logger.critical(
                     "HAProxy did not reload after publishing the bundle for "
                     "%s. The superseded certificate files were moved to %s and "
-                    "their certbot lineages were NOT deleted, so they can be "
+                    "their certbot lineages were NOT retired, so they can be "
                     "restored by hand: %s",
-                    primary, cert_backup_dir(),
+                    primary, cert_retired_dir(),
                     [e['file'] for e in cleanup_summary['removed']])
             raise
 
-        _delete_superseded_lineages(cleanup_summary)
+        # Only now that HAProxy has loaded the new bundle: stop certbot renewing
+        # (and renew-certificates.sh re-publishing) overlapping old lineages.
+        lineage_summary = _retire_superseded_lineages(
+            keep_lineage=primary,
+            bundle_names=set(names),
+            unreachable_names=unreachable_names,
+            keep_fingerprint=new_info.get('fingerprint') if new_info else None,
+        )
+        cleanup_summary['lineages'] = lineage_summary
 
+        not_after = new_info.get('not_after') if new_info else None
         log_operation(
             'request_ssl_bundle', True,
-            f'SSL bundle issued for {primary} covering {len(names)} names; '
+            f'SSL bundle {"issued" if renewed else "not yet due (existing certificate kept)"} '
-            f'cleaned up {len(cleanup_summary["removed"])} superseded lineage(s)'
+            f'for {primary} covering {len(names)} names; retired '
+            f'{len(cleanup_summary["removed"])} superseded pem(s) and '
+            f'{len(lineage_summary["retired"])} lineage(s)'
         )
         return jsonify({
             'status': 'success',
             'primary': primary,
             'names': names,
             'cert_path': combined_path,
+            'renewed': bool(renewed),
+            'capabilities': EDGE_CAPABILITIES,
+            'certbot_not_due': bool(certbot_not_due),
+            'not_after': not_after,
+            'not_after_iso': datetime.fromtimestamp(not_after, timezone.utc).isoformat()
+            if not_after else None,
+            'fingerprint': new_info.get('fingerprint') if new_info else None,
             'cleanup': cleanup_summary,
-            'message': f'Bundled certificate obtained for {len(names)} names',
+            'message': (f'Bundled certificate obtained for {len(names)} names'
+                        if renewed else
+                        f'Existing certificate for {len(names)} names is not yet '
+                        'due for renewal; nothing was issued'),
         })
     except Exception as e:
         log_operation('request_ssl_bundle', False, str(e))
@@ -2749,20 +3148,28 @@ def generate_config():
                 except Exception as e:
                     logger.error(f"Failed to create {suspended_list_path}: {e}")
 
-        # Access-log destination for the `log` line in the global section.
+        # Access-log destination for the `log` line in the global section:
-        # Default 172.18.0.1:514 is the docker bridge gateway for WHP's
+        # the in-container receiver's UNIX socket (scripts/haproxy-log-receiver.py),
-        # `client-net`, i.e. the host, where rsyslog's imudp listener is bound
+        # given relative to haproxy's chroot (/var/lib/haproxy) -- see
-        # by setup-haproxy-logrotate.sh. Overridable so this image stays usable
+        # hap_header.tpl. HAPROXY_SYSLOG_TARGET overrides it (`<ip>:<port>` for
-        # on standalone/home deployments with a different bridge subnet or a
+        # a remote collector). The old default, 172.18.0.1:514 (host rsyslog on
-        # remote log collector -- set HAPROXY_SYSLOG_TARGET to `<ip>:<port>`.
+        # the docker bridge), had no listener on the fleet.
-        # UDP, so an absent listener drops log lines and never affects request
+        syslog_target = (os.environ.get('HAPROXY_SYSLOG_TARGET', '').strip()
-        # handling.
+                         or DEFAULT_LOG_TARGET)
-        syslog_target = os.environ.get('HAPROXY_SYSLOG_TARGET', '172.18.0.1:514').strip()
+
+        # Default splash for unrouted hosts (server hostname, bare IP, unknown
+        # names) answered by HAProxy before the WAF -- see hap_listener.tpl.
+        # ON by default: it depends on no sidecar and only changes what an
+        # unrouted request sees. HAPROXY_DEFAULT_SPLASH=off restores the old
+        # default-backend page (WAF-inspected).
+        splash_raw = os.environ.get('HAPROXY_DEFAULT_SPLASH', '').strip().lower()
+        default_splash = splash_raw not in ('0', 'false', 'no', 'off')
 
         # Add Haproxy Default Headers
         default_headers = template_env.get_template('hap_header.tpl').render(
             cluster_secret = get_or_create_cluster_secret(),
             syslog_target = syslog_target,
+            coraza_spoe_backend = coraza_spoe_backend,
         )
         config_parts.append(default_headers)
 
@@ -2778,6 +3185,7 @@ def generate_config():
             crt_path = SSL_CERTS_DIR,
             coraza_spoe_backend = coraza_spoe_backend,
             suspension_enabled = suspension_enabled,
+            default_splash = default_splash,
         )
         config_parts.append(listener_block)
 
@@ -2786,6 +3194,9 @@ def generate_config():
         config_parts.append(letsencrypt_acl)
         config_acls = []
         config_backends = []
+        # Every name a routing ACL is rendered for (default splash lists).
+        routed_hosts = []
+        routed_wildcards = []
 
         # Add default backend rule (will be used when no domain matches)
         default_rule = "    # Default backend for unmatched domains\n    default_backend default-backend\n"
@@ -2841,6 +3252,7 @@ def generate_config():
                     name=domain['backend_name']
                 )
                 config_acls.append(domain_acl)
+                routed_hosts.append(domain['domain'])
                 logger.info(f"Added ACL for domain: {domain['domain']}")
             except Exception as e:
                 logger.error(f"Error generating domain ACL for {domain['domain']}: {e}")
@@ -2867,6 +3279,7 @@ def generate_config():
                     base_domain=base_domain
                 )
                 config_acls.append(domain_acl)
+                routed_wildcards.append('.' + base_domain)
                 logger.info(f"Added wildcard ACL for domain: {domain['domain']}")
             except Exception as e:
                 logger.error(f"Error generating wildcard ACL for {domain['domain']}: {e}")
@@ -2928,6 +3341,14 @@ backend default-backend
                         f"{CORAZA_SPOE_CONFIG_PATH} "
                         f"(SPOA target: {coraza_spoe_backend})")
 
+        # Routed-host lists for the default splash. Written on every render
+        # (also with the splash off, so turning it on later never meets a
+        # missing `-f` file, which is FATAL). Same render, same rows as the
+        # routing ACLs above, and in the rollback set, so the loaded
+        # haproxy.cfg and these lists always come from one generation.
+        write_config_atomically(ROUTED_HOSTS_PATH, render_routed_list(routed_hosts))
+        write_config_atomically(ROUTED_WILDCARDS_PATH, render_routed_list(routed_wildcards))
+
         config_content = '\n'.join(config_parts)
         logger.debug("Generated HAProxy configuration")
 
@@ -3011,6 +3432,28 @@ def _config_set_matches_backup():
     return True
 
 
+def render_routed_list(names):
+    """Pattern-file body for the default-splash routed-host ACLs.
+
+    One lowercased name per line, sorted and de-duplicated so an unchanged
+    fleet renders byte-identical files (create_backup() skips re-validation
+    when the live set equals the backup). A name containing whitespace cannot
+    be one pattern line (a newline would split it in two) and could not be a
+    working `hdr(host) -i` routing ACL either; it is dropped and logged at
+    ERROR, because a missing entry means the splash answers that host.
+    """
+    out = set()
+    for name in names:
+        clean = (name or '').strip().lower()
+        if not clean or any(c.isspace() for c in clean):
+            logger.error(f"default splash: cannot list routed host {name!r}; "
+                         "it will be answered by the splash page")
+            continue
+        out.add(clean)
+    body = ''.join(f'{n}\n' for n in sorted(out))
+    return '# Generated by haproxy_manager.generate_config() - do not edit\n' + body
+
+
 def _config_backup_pairs():
     """(live, backup) pairs forming one restorable config set.
 
@@ -3021,6 +3464,8 @@ def _config_backup_pairs():
         (HAPROXY_CONFIG_PATH, HAPROXY_BACKUP_PATH),
         (BLOCKED_IPS_MAP_PATH, BLOCKED_IPS_MAP_BACKUP_PATH),
         (CORAZA_SPOE_CONFIG_PATH, CORAZA_SPOE_BACKUP_PATH),
+        (ROUTED_HOSTS_PATH, ROUTED_HOSTS_BACKUP_PATH),
+        (ROUTED_WILDCARDS_PATH, ROUTED_WILDCARDS_BACKUP_PATH),
     )
 
 

keys/pkg-cache/README

@@ -0,0 +1,32 @@
+Trust material for the OPT-IN internal package cache (scripts/pkg-cache-repos.sh,
+docs/pkg-cache.md). Copied into images at /usr/local/share/pkg-cache/; nothing
+here is added to the system trust store.
+
+ca.crt                 "pkg-cache internal CA (CT130)", EC P-256, valid to 2036-10-18.
+                       NAME-CONSTRAINED to IP 192.168.1.40 and DNS
+                       pkg-cache.internal.cloud-hosting.io / pkg-cache.geekhome.local.
+                       SHA-256 fingerprint BB:5B:9A:08:05:ED:2F:58:83:92:D0:99:9D:11:0A:30:14:88:68:F0:F0:88:65:5E:94:0B:75:EB:C2:A2:BD:F3
+                       (private key lives only on CT130: /etc/pkgcache-ca/ca.key)
+RPM-GPG-KEY-EPEL-10    Fedora (epel10) 7D8D 15CB FC4E 6268 8591 FB26 33D9 8517 E37E D158
+                       fetched 2026-10-21 over HTTPS from dl.fedoraproject.org and
+                       mirror.math.princeton.edu (identical); fingerprint matches
+                       https://fedoraproject.org/security/
+RPM-GPG-KEY-remi2024   Remi's RPM repository 2024 CF1D F005 7CE8 5DFF 5B2F 2A37 C2FD 3B2C 2A09 48E4
+                       fetched 2026-10-21 over HTTPS from rpms.remirepo.net; byte-identical
+                       to the key shipped in remi-release-10.2-1.el10.remi. Not found on
+                       keys.openpgp.org / keyserver.ubuntu.com - confirm out of band.
+                       Remi rotates keys (remi2025, ...): when remi-release starts being
+                       signed by a new key, `release-install remi` fails loudly - add the
+                       new key here (and to pkg-cache-repos.sh).
+RPM-GPG-KEY-EPEL-9     Fedora (epel9) FF8A D134 4597 106E CE81 3B91 8A38 72BF 3228 467C
+                       fetched 2026-10-22 over HTTPS from dl.fedoraproject.org and
+                       mirror.math.princeton.edu (identical); used by `release-install epel`
+                       on EL9 images (cloud-memcached-container, cloud-haproxy-container).
+RPM-GPG-KEY-nodesource Nodesource Operations <operations@nodesource.com>
+                       242B 8138 31AF 0956 2B6C 46F7 6B88 DA4E 3AF2 8A14 (rsa4096, 2023-12-22)
+                       fetched 2026-10-22 over HTTPS from
+                       https://rpm.nodesource.com/gpgkey/ns-operations-public.key (the key
+                       NodeSource's own setup_N.x scripts reference); verified by dnf
+                       gpgcheck against the nodejs RPMs. Not on keys.openpgp.org /
+                       keyserver.ubuntu.com - confirm out of band. Used by
+                       `pkg-cache-repos.sh nodesource MAJOR` (cloud-node-container).

keys/pkg-cache/RPM-GPG-KEY-EPEL-10

@@ -0,0 +1,29 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGV4X6kBEAC3eQxgiWuo08uc3mHo4ELux++uqTnYz/tJzEf9Ou3h36WnhumA
+Nvs+Ts5h8PBx879Y9/aIX1Z20p1kf6tBCinZnEJu59n+TAAsph0+XQlV1l5YkleK
+Z2ff/Fg65k8QcLXWaIGykA/FaKznRiSurGuD6tRGhJw7DawEwBJr8QZSkRUpnH1L
+URW97Q/iKrRPiE5VEayE0y8eAL28jIIiFvR+4oJMzvCsRRB/2wYZ2MlJOW91hcYf
+mbUoXKOBD5UzsJylu7kj25K/ge8rEJ7KicOOwcdYddxsU3DxGSSfwF8AMagENcm2
+XROeXknjm84A8sNlUkFZBJwfuc7eRTiZGJrnQQVYLrkKj8Mxpq9Ts7hU51TqAWNI
+uvGDlJdYNE3D2RMqjMEsZ8ej08Thrib6xslu4NzTBkt+6QNnXL4E3hEgYtoyio60
+GswSz2ulogKg7X4JrNdJYE8/qNowyF3hoVgj5TG1/wQRq+5HlMMOLjgGu9wzLUix
+fnVfEUnzaofbrUf4/GabCaeY8xRe4tFQrvzigQ4g+kgwKKnfAeqBmPov0yljkw9z
+BYJWR5zvaw0ffg9Ing00KUSaXBXA5jSlgk1603Y+LefY1SlXsTyqohiRvGH6FI77
+HNMo72DwoJfFcYjncZUzKgXWJECR4nhVsdj6pKoOjcQ4aSuyVxtsR86ASQARAQAB
+tChGZWRvcmEgKGVwZWwxMCkgPGVwZWxAZmVkb3JhcHJvamVjdC5vcmc+iQJOBBMB
+CAA4FiEEfY0Vy/xOYmiFkfsmM9mFF+N+0VgFAmV4X6kCGw8FCwkIBwIGFQoJCAsC
+BBYCAwECHgECF4AACgkQM9mFF+N+0Vhv/A/+PlhPLSctGRCUEahE+cN4764Acc3p
+l40ZYzXRhqR0/Tc1/cSDjlA3qVTc8SPohi5OJXwCyr9EiMqKoyoDN097euqbYpyp
+yN/Pj0lBjsXwcpdDtZ21WGeQU0Khb04N68bMtJbDaxeBciTvDDQravZuPPh0m4Rg
+Z6myEoa6Aa6EK0hI1Qwi1qIWeRiuEkVT671IaKVETBW5XiUpNBXDAB/L+6DzUF9u
+scBzfsUDiPO6NrpYDtV3jwq22y6gWluIct/Ka8brwPbqK2sBfFzrHboRhfqlTGjs
+7F9qUGwIQZn/A8iozXZYQ0+JG1bhQyvjA8eN1GOcRpT+O7H7JXN49o6IG2As4+iK
+F04+qjqAu2sVfpD8mzM2VubFNllcKKiyCzRYHhSbObRCPzsudDL9GPiXeGGaCuWg
+sDkiA1MESvf2tLETAGBs/TziO4GwmXUtlKbRiq1FYm90mVq9mBxPZ/Idn+yZusNB
+0O5SXIbI8lYZw5n4XTK4b+byHRBYsOTHiTsGvjTF2Y7oSwW2CVUmL6RZ23mI4qoY
+1p5kzRS+GjT1acnTei/FTsOlIKCsjfeHx7uxCkX6xpAD8P3UtLQqfsgH0CL4vSZt
+TGO6L1InQlp4ZG3OYIomTKbD3/R0wod3U3dTqdulQMXL895u6OLTY3spY2m2MO2k
+p9Dfd2pKuxK9Mys=
+=mhQZ
+-----END PGP PUBLIC KEY BLOCK-----

keys/pkg-cache/RPM-GPG-KEY-EPEL-9

@@ -0,0 +1,29 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGE3mOsBEACsU+XwJWDJVkItBaugXhXIIkb9oe+7aadELuVo0kBmc3HXt/Yp
+CJW9hHEiGZ6z2jwgPqyJjZhCvcAWvgzKcvqE+9i0NItV1rzfxrBe2BtUtZmVcuE6
+2b+SPfxQ2Hr8llaawRjt8BCFX/ZzM4/1Qk+EzlfTcEcpkMf6wdO7kD6ulBk/tbsW
+DHX2lNcxszTf+XP9HXHWJlA2xBfP+Dk4gl4DnO2Y1xR0OSywE/QtvEbN5cY94ieu
+n7CBy29AleMhmbnx9pw3NyxcFIAsEZHJoU4ZW9ulAJ/ogttSyAWeacW7eJGW31/Z
+39cS+I4KXJgeGRI20RmpqfH0tuT+X5Da59YpjYxkbhSK3HYBVnNPhoJFUc2j5iKy
+XLgkapu1xRnEJhw05kr4LCbud0NTvfecqSqa+59kuVc+zWmfTnGTYc0PXZ6Oa3rK
+44UOmE6eAT5zd/ToleDO0VesN+EO7CXfRsm7HWGpABF5wNK3vIEF2uRr2VJMvgqS
+9eNwhJyOzoca4xFSwCkc6dACGGkV+CqhufdFBhmcAsUotSxe3zmrBjqA0B/nxIvH
+DVgOAMnVCe+Lmv8T0mFgqZSJdIUdKjnOLu/GRFhjDKIak4jeMBMTYpVnU+HhMHLq
+uDiZkNEvEEGhBQmZuI8J55F/a6UURnxUwT3piyi3Pmr2IFD7ahBxPzOBCQARAQAB
+tCdGZWRvcmEgKGVwZWw5KSA8ZXBlbEBmZWRvcmFwcm9qZWN0Lm9yZz6JAk4EEwEI
+ADgWIQT/itE0RZcQbs6BO5GKOHK/MihGfAUCYTeY6wIbDwULCQgHAgYVCgkICwIE
+FgIDAQIeAQIXgAAKCRCKOHK/MihGfFX/EACBPWv20+ttYu1A5WvtHJPzwbj0U4yF
+3zTQpBglQ2UfkRpYdipTlT3Ih6j5h2VmgRPtINCc/ZE28adrWpBoeFIS2YAKOCLC
+nZYtHl2nCoLq1U7FSttUGsZ/t8uGCBgnugTfnIYcmlP1jKKA6RJAclK89evDQX5n
+R9ZD+Cq3CBMlttvSTCht0qQVlwycedH8iWyYgP/mF0W35BIn7NuuZwWhgR00n/VG
+4nbKPOzTWbsP45awcmivdrS74P6mL84WfkghipdmcoyVb1B8ZP4Y/Ke0RXOnLhNe
+CfrXXvuW+Pvg2RTfwRDtehGQPAgXbmLmz2ZkV69RGIr54HJv84NDbqZovRTMr7gL
+9k3ciCzXCiYQgM8yAyGHV0KEhFSQ1HV7gMnt9UmxbxBE2pGU7vu3CwjYga5DpwU7
+w5wu1TmM5KgZtZvuWOTDnqDLf0cKoIbW8FeeCOn24elcj32bnQDuF9DPey1mqcvT
+/yEo/Ushyz6CVYxN8DGgcy2M9JOsnmjDx02h6qgWGWDuKgb9jZrvRedpAQCeemEd
+fhEs6ihqVxRFl16HxC4EVijybhAL76SsM2nbtIqW1apBQJQpXWtQwwdvgTVpdEtE
+r4ArVJYX5LrswnWEQMOelugUG6S3ZjMfcyOa/O0364iY73vyVgaYK+2XtT2usMux
+VL469Kj5m13T6w==
+=Mjs/
+-----END PGP PUBLIC KEY BLOCK-----

keys/pkg-cache/RPM-GPG-KEY-nodesource

@@ -0,0 +1,52 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGWFsqMBEAC4YlSuNpHjOiQVTC8StolVx3J8E9SC6G4wmVWbytMD1Zivfanj
+3LKDqdy7vpbViXA9whpyGQgQYdOlPKaNHWw4GSeGTnP1FbvaK1NJ9MnAzUysh49R
+cqoktYeIQbqgOgZHw5/gVH5bYObf1KJQ2LY/je9WTlW3QMRe1TaW0F0AZnJMBIOG
+JllE33QIpxDnOQHnvdUGq+HscUOTrW4Q3AdzwsQxU4GX82d9qFFr8HjAmUjlUKyh
+PgvMpb8VDvR7NE6et+wWh6BMYILo3VJFP7/gTXGiy3WWs5GW/PNXIXw5/f2w1B+e
+Dh5tlfGG4q/4P8mbv7C6RPHoF2Lq+W69HLqSq/l42WUL41tHlFTsluCIKpeeMxAu
+olxx3Z5momHMMyh7rbZtQDqFMTAqxNjT1NinKkWyj7KP5WE9hVosyd3UMR4UYDMm
+Zb6LI4N7Hht887+CY7grxTJ9cGQhMFfvLyvlf/FWQY4S6G7sEaUV71N9StvXBNG/
+iNoIMzhwjBmrmvT/UGPkLSi0SIBOxeGw2rwngnYikIJxuTQ4V40lm8o6eMFca7rW
+BQvAEncV8pvi/3FqrUBZ/U53s7NO24hBDoaQPzFbVCw0GHHf9GD7KoQb5c9ONoCZ
+Qu2YWo7ajPLzl/h2pJ2Kn29Ywev9MpuuPTGvifiKilcBVLYP3ZJg4ZfRBwARAQAB
+tDFOb2Rlc291cmNlIE9wZXJhdGlvbnMgPG9wZXJhdGlvbnNAbm9kZXNvdXJjZS5j
+b20+iQJOBBMBCgA4FiEEJCuBODGvCVYrbEb3a4jaTjryihQFAmWFsqMCGwMFCwkI
+BwIGFQoJCAsCBBYCAwECHgECF4AACgkQa4jaTjryihTD+hAAtTU155ke7xmnT0ph
+odbGjm3mOGdkdaDH+xmWtPtufiPncgHMCYh4aIDRloog8rFE+pP3qfwJmUKxthwD
+drrLAzfUwOrGxPnYH3MWwg66RVVoiiU+Hh42gJTez+P/K0VMQZXH0XTZD96TiwT/
+S0+WvQPuJ9pda2L4cjV+YP2+m0ERUG9QPj0Shgw+/PWCGWKD6J9DRToiwfwCTOvc
+DNolF1EcRlfPMTsqI2Bdc+TQywqU1Py1MUDSAPXxdK6nW2oM7eC5l3XXefcQ6JOS
+AuNdEDIsaf+H76j6/wSGsuIcJjUo9wmKjEV1iJYwT6jLbjaVQxXgEPHBSb+kzanx
+FWyrYP6TZpeY2gT4bBMsOKd7ay1I3/nIHwe6x8Yzkqzz+7YVhNpMbybH72NTxDWQ
+UmiboJs9r7j1lkHlBjxU9RLF5laYSeCIXD62KQVoP5ao1LSPKsCXFqcdpUgLYtRv
+CXxdriiHp4m9eGzbS7RW1TLm6tMOKbI4LzOtMm4DoZzGD2FTfsUtn6ALx9FSgvl/
+VMlzqZes+rclcukACtutvtB8e0yRhuzPUN56ctLEZfI1XzQFoN8Vdj6HWtiXWRiJ
+EohuufE2cEebmX90NRsbh4KAFXNEQROlMeMf7ufuSsk7QXLTutxmWSQvWwyS+P5/
+ZqCapIDlRCJFZX4N9a1IXo0OgCC5Ag0EZYWyowEQAKz7sA6WBKFVcmzcyr6y9TI5
+RemnzotRUdqXj8tD70xvV1nBEf+BfTw3hsiV019ZboPBae+cRaD4plQbNb0du/+Q
+UFq4ja055ZEld01RqE49L562pWvA5cwSmjs9+ZMI1ja0HbTb2K4razpzX3be+Zul
+TbH20X5FHTz+Jyfmw+G3Y9okw3OIzuYo16Lv/Qs8VoBmj01PItzq8dX5wJiIlVxU
+8wKHKy1kiKGCgyPnvqcXCGen0EoUS9wsWK1QHfryscYGVF9y1TBQvJa9HIpWPrIZ
+slSiVS4tQhGHfLDVlaDsYFrH9SIUkvW7peSEiPE1TKQjvbMS3CcUZRaMcGfgyDPe
+gXwnNwZbJ2/cisXbdAJaGT49V1Y+Sc1oKSGSG4FaGUNhS3p3cB7qlX2zS/sR2ael
+90fXR6nhjvhMAS98i7663pCUv9q0Z/NXNZwdXjbSF4W8gq3PVt7hCnJrngsGSkGu
+Qz0dc227/sAjh9CxYm8kQqUuQtiPVfKwkcUUT+O7jr2/6eAXZbTYVZbvaLEGMrKA
+bdWKpZMMOKAI45f+ZAcEmJG4D68N0jzVrLCXMuuLq/yqqyEmxDjSOH0qVutl8jxg
+vfxPxKYX8nnAzJZJHGHmV/ihjfClSdYePKExHuFVVhvc4BxE1XfPUeCZG4JpB9/B
+gNCiDPsg4JIw6GO5LS2xABEBAAGJAjYEGAEKACAWIQQkK4E4Ma8JVitsRvdriNpO
+OvKKFAUCZYWyowIbDAAKCRBriNpOOvKKFOTKD/40WN6ZRfP6hVLAX26DdyN9Bqzg
+D+0hJilmHL1YFeLt9U/YPhOZJxPaDGrMfldC0FSXKKr7H5C7FQErl936108wGohz
+53KGbNid0crNeaWrQuWk3QjaMF62Yc+NVB8b5LoEk0TEt9KPO6kveYDtW6Ygkw9i
+5aaS/AC+8GChOH6lWhLHcO91T7+Hi5qRez7qyme/AlHUTV9/jE6JgFJ7thCCAqBQ
+AfX3mGFlTwSTMHDeYq81MMl9UhYgoFJtx8PuqmSV7Eb29ccJofz3J+b5OJvpmLSZ
+Hu22Csa8k2V/1ojeL0DjQHD06V75F7BerxLR/rbJ64GkIiOdm4ntXC4a335gjQfM
+5tI6LtGsIpr+Le4C068eDj7YpzS0ql3g6o4A1TbtapLccgEj4iw95H4KKh0Ro5r0
+0s6LoSLu9uva3LiLVn6xXvpw3iv8FnhTr5PEiPnorXvxcCnUrB3Rkk6Wh1SKjpxa
+u6KiFzH0CXR1AgUdWSC0rAmvEdFjXgaoQd+Z2eJFHEAsg91URd9NtdeDsyMwdbHh
+iBZPGX5ddPjJA7nwKbkibCv7xiodd9ph8NmGLqD/2SDU1rqvPuCDMqLuxc89Gvhs
+3CeLTSJK9Uzb3oNdZLxL8CECip1ZkowVfTL+jY23phS9+Z0Db/yZqot3ZZJAAG8g
+oSVkzT/Mwp9zFVQu5w==
+=drMP
+-----END PGP PUBLIC KEY BLOCK-----

keys/pkg-cache/RPM-GPG-KEY-remi2024

@@ -0,0 +1,52 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGWdEG0BEADV9c8aSIITQmoWKN0ac4c8AEhZcdzpWSRPwHIS8syVRw/gj30W
+Mn1OoxqFn2UECh2jjqYaQFECXNrM2/lzO4GhvcuoPn6p0rXFMvYvwtiOJ+rmDvTe
+UOwRxkZhYehpvsRPFoVhlYJh3U1BM0UHm2iE/TIqVusY9FHQuCMZVl0BHk81YGDa
+qE3cyu+9zOs8z1bYlsYBgLIbtb4+loQTRexFTfZkUcCWgAaNyur8Wy7Msdm5LOqN
+ZZu6bNKBOv0Cr+BOwGNoFRbi4IM+5H+0eI1n7V5if4CuXfdYHp+Pm+PQvftm/nqB
+jRLKqioRKzHjk7xNMsi5lM+SU4WyfU77pr3RXuJR6fkymLVAeu5sy2U6ySj0SXIO
+/I3oPp4/zJ8/e94jG5iDqh92b5IWyXSE+EGw4kkpUvZLrK8vyyEfk8h9Gf8qA5uL
+yGfw0vnSVjYb+RdnfZFVdPaIEmeY8tnobS87duj8JESZiSWo5sU4filkciW5e+MJ
+w9wohgl+Cqm2J5Ljqus6DRm4ptKbZD8QZRnmGcSk6O2VopBKt2XLat4eIJLr/EiN
+0CJ1EH/oJmEeZbYtUjxjsGsbPXicwDH6ZIeIMSaiDgqRxPm1LTGVLjSHAk+k0e5l
+ulEmaoA7t/p3U2peGc7ONvozXiyNHZdGiRkqFC1mA9/JpeWw4KFH6JSgmQARAQAB
+tEtSZW1pJ3MgUlBNIHJlcG9zaXRvcnkgMjAyNCAoaHR0cHM6Ly9ycG1zLnJlbWly
+ZXBvLm5ldC8pIDxyZW1pQHJlbWlyZXBvLm5ldD6JAlEEEwEIADsWIQTPHfAFfOhd
+/1svKjfC/TssKglI5AUCZZ0QbQIbAwULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIX
+gAAKCRDC/TssKglI5Na7EACDTGjTcuTyudfuu/JC41BSurclCcwZyPsGaClyc0qx
+Tk8Rs4rbyVw1u494fiFG/yEVl7LGGja84UJOlMdw82KgVUmr4fGLaBBFWm+KnZ4W
+4vvFcCIxUshENOaDrkhby9F5P379gVOK9I3n9l4Q2xt+sWUNwE74CNfCTUeYMIm/
+/AHbb1DTM9FuIT6FssgvILMTlksiyrw8U8bj+QUzPZrFCpvv+PnFA7ZDMEyCf2Tz
+/zawxoP3a3b+3/g61U9SVIkfToonnorqh9lkQYhsdKTFHW6eESqe7zcqMBZv/sSc
+3kFzl22KczSdU8eYxdujv9BmlW2SOrQsnuMYxvoD1LzhwSB/nOapNrWU+djhJCof
+ueZ+rBW4QWysmG0dEp5Wpz85TstAnQ+j5aCOMSAkiAVsrJ0Ff3JXakIiL7rEtccS
+dhgBT06LJtJy1NGIC0Sotn45n6dtfi6IuI8LnELCKeHxhdFSJcAPB8y95u0uHymy
+sqMfzjq1VcYtzC5II6Yd5Gtb2+QzyQwORslwXhUdXwZKUc5i6X5xFMSxU523fH9n
+i+kQlUTPkOUr66v4fmYDj+ZlJQSFoPylcHCke/vOo/rH5nj+YMPa6LdohsdzHKpd
+DkKrk+Udj5aDXtXGIFYwoNKO4BnCsz/ImZfB4CQ+ekJsY1pFill3+MRTM6IvSA+P
+7LkCDQRlnRBtARAAq8k4FM9euMMsXz0buxxyr0ew0ecWNhZGy24a6DXORVTEJbgI
+T3HpV/C2PWkwgEBswYzREm+en61jhWr5unCiytO2klT+K2WNcIUNW+z2ciL/JNaJ
+OcKwjkiY/A+KtDStzllCQ/XnpgAXGvFZjMjhHrNGQoAwNnPE/cMtYwcuVdW3Ipm3
+cOOtzxRmfnhP8YUEyHxdos/7AZowaiLIUkaL5RUaygGrfPGyUXPl1aXS1oiOm37W
+RiKCGUxCHSlpBqeh1WQPOaps8y+ME9jMzF9SLvqiXlKPllJIx9i8tHUOnX6Y2U2x
+5sXsCEiA+zNVMU3YQ04++Yx2Rp6sDhNEnAbTCqqequqIzaXHZWwszAeG6pj1G90m
+8PwDeHy641ySgDMwVPpnH5rHWmmT96XvF9QNg07OBtW9/phryzwAuHogrBY4DyFD
+pUxDkTv9TxfBaS/s1uJxEGknIgnKzQo79mRy206waGKRA2AV36ORSfU12sK8pl3r
+N95iIDh/OIuMl2STaXilhwb1hRpGFvwRlq3Hvvan4SY1aO219Q+VxMGXeqC5F/rm
+Anh7mRVMP2qu0rj7L8xJ/OD/bPfuWL72/PvviBr/jdRM9I+Vj8dPGjhUwhK1dpEG
+Q6yvoVXLHXucVHfIVR+ALLBmK8oHXNIFeIXhiUbgmRAilPHRz6Rf5pgbrC0AEQEA
+AYkCNgQYAQgAIBYhBM8d8AV86F3/Wy8qN8L9OywqCUjkBQJlnRBtAhsMAAoJEML9
+OywqCUjkQ2kP/iwqvt1sX85KM4OSE9ftqemwnvh19NizMKHMYGgikMAvy2CndEOI
+wzKVeE1iK9/ivZQQIbD5l0G9hzVZjIxbgu9dAcxM+jdmaPyUj7yn6AX2yeGAeHyA
+qzNKZaTLJPieRzdXk/uKPVyWUuzbR0vwR7n8N5N0pe/FEfO9O0uX5iz2kf5+b10c
+WqvzLmO+czuyCjTHnjhKfE73iIktYE3MnM8qFHPiKwm+jeTkUnmInoR3qIKR/SCV
+u2vk/q5uvpNaUSPSDKji6wh3/hbNbPocLSGIjugfWNm/4LhlSHPNnRJ2dIvSxeiI
+qtSWxhULEyHGBPMLDriVumCV7Rb/byJwiMbmCIpxB5XLtbgt7nIqWQTko+6ci71S
+4rDXolM3bytYbpqq4b/Xc1bhBuTX3omwrd0kxmGqad8eia6O6iBsP7GAl4QkySm4
+MfKQfohjXTnT60wqEzQwLGI4NKN9bHRSsGINHX5p+Kz9HLf87fTY4uglbFxg0/II
+Y3nko5KBw/gfe0WpwZY0sWjK20cXJn3xLQjoG8OjUNUCUYTRsiAvn4AiDRt++qtl
+bvJjJqOU1k4GrGoCVpEUiOc9Cw1YZREcFe8H2at3pGnxJ1DFlYA916kNAxYANp2s
+r3N3agwevgIzP78hynxjm6a5H2w0vkg189pZn2RR9zX9gmrvlqg1o0P1
+=meaU
+-----END PGP PUBLIC KEY BLOCK-----

keys/pkg-cache/ca.crt

@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE-----
+MIICPzCCAeagAwIBAgIUJI0QJZ+PCrSQ/nKZFENwwcp8Zr0wCgYIKoZIzj0EAwIw
+TDEmMCQGA1UEAwwdcGtnLWNhY2hlIGludGVybmFsIENBIChDVDEzMCkxIjAgBgNV
+BAoMGWNsb3VkLWhvc3RpbmcuaW8gaW50ZXJuYWwwHhcNMjYxMDIxMTkzNjIyWhcN
+MzYxMDE4MTkzNjIyWjBMMSYwJAYDVQQDDB1wa2ctY2FjaGUgaW50ZXJuYWwgQ0Eg
+KENUMTMwKTEiMCAGA1UECgwZY2xvdWQtaG9zdGluZy5pbyBpbnRlcm5hbDBZMBMG
+ByqGSM49AgEGCCqGSM49AwEHA0IABBJh75sC5vSzTnkYl0CYYEA/TKUHEXhbtanV
+OVNW4NlD3za5+bu5YuKjpbFAva4TCZqWsaS9IMd/JfsaFbf5XVejgaUwgaIwEgYD
+VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFP7UCTf/
+v7KU1KBrs1vqa71/TZl0MF0GA1UdHgEB/wRTMFGgTzAKhwjAqAEo/////zAlgiNw
+a2ctY2FjaGUuaW50ZXJuYWwuY2xvdWQtaG9zdGluZy5pbzAaghhwa2ctY2FjaGUu
+Z2Vla2hvbWUubG9jYWwwCgYIKoZIzj0EAwIDRwAwRAIgOod4wPHu0kAzYAw8KeGg
+cIrFMwQ4ATKVTkDioiSFOqICIEhY+sGj/YZ4oBMst3b/AAz9zMEJAXpnCkSTI6tM
+YAIl
+-----END CERTIFICATE-----

scripts/cert-publish-lib.sh

@@ -203,6 +203,82 @@ cert_bundle_valid() {
     return 0
 }
 
+# cert_is_expired CERT_FILE
+#
+# Returns 0 only when openssl positively reports that the first certificate in
+# CERT_FILE has expired. Anything else (not expired, unreadable, no openssl)
+# returns 1; unreadable material is rejected with a better message by
+# cert_bundle_valid().
+#
+# Why: HAProxy loads the crt directory wholesale and serves the FIRST file that
+# claims an SNI name. An expired bundle in there is never useful and can only
+# shadow a valid one - on 2026-10-07 renew-certificates.sh kept re-publishing
+# the expired brain-jar.com lineage every 12 hours, and HAProxy served it for
+# brainjar.net instead of the valid brainjar.net bundle next to it.
+cert_is_expired() {
+    local file="$1" out
+    [ -f "$file" ] || return 1
+    command -v openssl >/dev/null 2>&1 || return 1
+    out="$(openssl x509 -checkend 0 -noout -in "$file" 2>/dev/null </dev/null)"
+    [ $? -eq 1 ] && [[ "$out" == *"will expire"* ]]
+}
+
+# cert_not_after CERT_FILE - the notAfter date string, or empty.
+cert_not_after() {
+    openssl x509 -enddate -noout -in "$1" 2>/dev/null </dev/null | sed 's/^notAfter=//'
+}
+
+cert_retired_dir() {
+    if [ -n "${CERT_RETIRED_DIR:-}" ]; then
+        echo "$CERT_RETIRED_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-retired"
+    fi
+}
+
+# cert_retire_expired_pems
+#
+# Move every EXPIRED *.pem out of $SSL_CERTS_DIR into cert-retired/ (a sibling,
+# never inside the crt dir). Sets CERT_RETIRED_EXPIRED to the number moved.
+#
+# Not publishing expired lineages stops the re-publish loop, but a pem that was
+# published while valid and has expired since stays in the crt dir and keeps
+# shadowing - that was exactly brain-jar.com.pem (review M2). An expired
+# certificate is never a valid certificate for anyone, so it can always go: the
+# only-valid-cert safety rule cannot be violated by moving it. Two exceptions:
+#   * default_self_signed_cert.pem - HAProxy's fallback, regenerated separately;
+#   * the LAST .pem in the directory - a crt directory with no certificate at
+#     all fails the whole :443 bind, which is worse than serving an expired one.
+# Nothing is deleted; reverse with mv.
+cert_retire_expired_pems() {
+    local dir="${SSL_CERTS_DIR:-/etc/haproxy/certs}" retired f base dest remaining
+    CERT_RETIRED_EXPIRED=0
+    retired="$(cert_retired_dir)"
+    for f in "$dir"/*.pem; do
+        [ -f "$f" ] || continue
+        base="$(basename "$f")"
+        [ "$base" = "default_self_signed_cert.pem" ] && continue
+        cert_is_expired "$f" || continue
+        remaining="$(find "$dir" -maxdepth 1 -type f -name '*.pem' | wc -l)"
+        if [ "$remaining" -le 1 ]; then
+            log_error "Expired certificate $f is the only certificate left in $dir - NOT retiring it (an empty crt directory fails the :443 bind)"
+            continue
+        fi
+        if ! mkdir -p "$retired"; then
+            log_error "cannot create $retired - expired $f left in place"
+            continue
+        fi
+        dest="$retired/$base"
+        [ -e "$dest" ] && dest="$dest.$(date -u +%Y%m%dT%H%M%SZ)"
+        if mv -- "$f" "$dest"; then
+            log_error "Retired EXPIRED certificate $base (notAfter $(cert_not_after "$dest")) from $dir to $dest - it could only shadow valid certificates"
+            CERT_RETIRED_EXPIRED=$((CERT_RETIRED_EXPIRED + 1))
+        else
+            log_error "failed to move expired $f to $dest"
+        fi
+    done
+}
+
 # cert_publish CERT_FILE KEY_FILE DEST_FILE
 #
 # Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
@@ -227,6 +303,11 @@ cert_publish() {
         log_error "cert_publish: private key $key_file is missing or empty"
         return 1
     fi
+    # (a2) never publish an expired certificate (see cert_is_expired).
+    if cert_is_expired "$cert_file"; then
+        log_error "cert_publish: certificate $cert_file expired ($(cert_not_after "$cert_file")) - refusing to publish $dest_file"
+        return 1
+    fi
 
     # (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
     #     scans wholesale.

scripts/haproxy-log-receiver.py

@@ -0,0 +1,264 @@
+#!/usr/bin/env python3
+"""In-container receiver for the HAProxy access log.
+
+HAProxy sends its access log (`log <socket> ... format rfc5424`, hap_header.tpl)
+as datagrams to a UNIX socket inside this container. This process is the only
+reader: it writes one line per datagram to a size-capped, rotating file in the
+haproxy-logs volume (default /var/log/haproxy/access.log).
+
+Why it looks the way it does
+----------------------------
+* UNIX DATAGRAM socket, no network listener. Nothing is exposed on the host
+  or on client-net (the old target was the host's rsyslog on the docker
+  bridge, 172.18.0.1:514, which had no listener on the fleet).
+* HAProxy never waits for us. Its log sends are non-blocking: a dead receiver
+  (ECONNREFUSED/ENOENT) or a full socket buffer (EAGAIN) drops that line and
+  HAProxy carries on. So a crash, a stall or a full disk here can only lose
+  log lines, never slow or fail a request. That is the intended direction.
+* Bounded on disk: the live file is rotated at MAX_BYTES and BACKUPS gzip
+  generations are kept, so the worst case (incompressible data) is
+  (1 + BACKUPS) * MAX_BYTES -- 5 * 40 MB = 200 MB by default. Compression runs
+  in a thread so a rotation does not stall reading.
+* A full disk (ENOSPC) or any write error drops lines and is reported at most
+  once a minute; the file is reopened on the next attempt.
+* Lines go to the FILE ONLY, never stdout: `docker logs haproxy-manager` is a
+  source of WHP's AI log monitor, and a per-request firehose there would storm
+  it (see the WHP note on audit/firehose sources). Our own status messages are
+  rare and go to stderr.
+* Control characters inside a datagram are replaced so one datagram is always
+  exactly one line (a crafted header cannot forge extra log lines).
+
+Supervised by start-up.sh (restart loop), started before HAProxy.
+
+Environment
+-----------
+  HAPROXY_LOG_SOCKET       socket path   (default /var/lib/haproxy/log.sock)
+  HAPROXY_ACCESS_LOG       file path     (default /var/log/haproxy/access.log)
+  HAPROXY_ACCESS_LOG_MAX_BYTES  rotate size (default 41943040 = 40 MiB)
+  HAPROXY_ACCESS_LOG_BACKUPS    gzip generations kept (default 4)
+"""
+
+import gzip
+import os
+import shutil
+import socket
+import sys
+import threading
+import time
+
+# HAProxy sends at most `len` (2048) bytes per line; leave headroom.
+MAX_DATAGRAM = 65535
+FLUSH_INTERVAL = 1.0
+ERROR_REPORT_INTERVAL = 60.0
+# Kernel receive buffer to absorb bursts while a write is slow.
+RCVBUF = 4 * 1024 * 1024
+
+_CTRL = {c: ' ' for c in range(0x20) if c != 0x09}
+_CTRL[0x7f] = ' '
+
+
+def _env_int(name, default, minimum):
+    try:
+        value = int(os.environ.get(name, default))
+    except ValueError:
+        value = default
+    return max(minimum, value)
+
+
+def clean_line(datagram):
+    """One datagram -> one log line (str, no trailing newline).
+
+    Drops the RFC 5424 `<PRI>` prefix (constant for our single facility) and
+    replaces any control character so the datagram cannot span lines.
+    """
+    text = datagram.decode('utf-8', 'replace').rstrip('\r\n\x00')
+    if text.startswith('<'):
+        end = text.find('>', 1, 6)
+        if end > 0 and text[1:end].isdigit():
+            text = text[end + 1:]
+    return text.translate(_CTRL)
+
+
+class RotatingWriter:
+    """Size-capped writer: live file + `backups` gzip generations."""
+
+    def __init__(self, path, max_bytes, backups, report=None):
+        self.path = path
+        self.max_bytes = max_bytes
+        self.backups = backups
+        self.report = report or (lambda msg: None)
+        self.fh = None
+        self.size = 0
+        self.dirty = False
+        self.dropped = 0
+        self._compressor = None
+        self._last_error = 0.0
+
+    # -- file handling --------------------------------------------------
+    def _open(self):
+        os.makedirs(os.path.dirname(self.path) or '.', exist_ok=True)
+        self.fh = open(self.path, 'a', encoding='utf-8', buffering=65536)
+        os.chmod(self.path, 0o640)
+        self.size = self.fh.tell()
+
+    def _error(self, what, exc):
+        self.dropped += 1
+        try:
+            if self.fh:
+                self.fh.close()
+        except OSError:
+            pass
+        self.fh = None
+        now = time.monotonic()
+        if now - self._last_error >= ERROR_REPORT_INTERVAL:
+            self._last_error = now
+            self.report(f'{what} failed ({exc}); dropping access-log lines '
+                        f'(dropped so far: {self.dropped})')
+
+    def _gen(self, n):
+        return f'{self.path}.{n}.gz'
+
+    def _compress(self, src, dst):
+        tmp = dst + '.tmp'
+        try:
+            with open(src, 'rb') as fin, gzip.open(tmp, 'wb', compresslevel=6) as fout:
+                shutil.copyfileobj(fin, fout, 1024 * 1024)
+            os.replace(tmp, dst)
+            os.chmod(dst, 0o640)
+        except OSError as exc:
+            self.report(f'compressing {src} failed ({exc}); removing it to keep the size cap')
+            for leftover in (tmp,):
+                try:
+                    os.unlink(leftover)
+                except OSError:
+                    pass
+        finally:
+            try:
+                os.unlink(src)
+            except OSError:
+                pass
+
+    def rotate(self):
+        if self._compressor is not None:
+            self._compressor.join()
+            self._compressor = None
+        if self.fh:
+            try:
+                self.fh.close()
+            except OSError:
+                pass
+            self.fh = None
+        if self.backups <= 0:
+            try:
+                os.unlink(self.path)
+            except OSError:
+                pass
+        else:
+            try:
+                os.unlink(self._gen(self.backups))
+            except OSError:
+                pass
+            for n in range(self.backups - 1, 0, -1):
+                if os.path.exists(self._gen(n)):
+                    os.replace(self._gen(n), self._gen(n + 1))
+            staged = f'{self.path}.1'
+            if os.path.exists(self.path):
+                os.replace(self.path, staged)
+                self._compressor = threading.Thread(
+                    target=self._compress, args=(staged, self._gen(1)), daemon=True)
+                self._compressor.start()
+        self.size = 0
+
+    # -- public ---------------------------------------------------------
+    def write(self, line):
+        data = line + '\n'
+        try:
+            if self.fh is None:
+                self._open()
+            if self.size and self.size + len(data) > self.max_bytes:
+                self.rotate()
+                self._open()
+            self.fh.write(data)
+            self.size += len(data.encode('utf-8'))
+            self.dirty = True
+        except OSError as exc:
+            self._error('writing ' + self.path, exc)
+
+    def flush(self):
+        if self.fh and self.dirty:
+            try:
+                self.fh.flush()
+                self.dirty = False
+            except OSError as exc:
+                self._error('flushing ' + self.path, exc)
+
+    def close(self):
+        self.flush()
+        if self._compressor is not None:
+            self._compressor.join()
+        if self.fh:
+            try:
+                self.fh.close()
+            except OSError:
+                pass
+
+
+def bind_socket(path):
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    try:
+        sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, RCVBUF)
+    except OSError:
+        pass
+    os.makedirs(os.path.dirname(path) or '.', exist_ok=True)
+    try:
+        os.unlink(path)  # stale socket from a previous (crashed) receiver
+    except FileNotFoundError:
+        pass
+    sock.bind(path)
+    # HAProxy's workers run as user haproxy (hap_header.tpl `user haproxy`).
+    os.chmod(path, 0o666)
+    sock.settimeout(FLUSH_INTERVAL)
+    return sock
+
+
+def serve(sock, writer, stop=None):
+    last_flush = time.monotonic()
+    while stop is None or not stop.is_set():
+        try:
+            datagram = sock.recv(MAX_DATAGRAM)
+        except socket.timeout:
+            datagram = None
+        except OSError:
+            if stop is not None and stop.is_set():
+                break
+            raise
+        if datagram:
+            writer.write(clean_line(datagram))
+        now = time.monotonic()
+        if now - last_flush >= FLUSH_INTERVAL:
+            writer.flush()
+            last_flush = now
+
+
+def main():
+    sock_path = os.environ.get('HAPROXY_LOG_SOCKET', '/var/lib/haproxy/log.sock')
+    log_path = os.environ.get('HAPROXY_ACCESS_LOG', '/var/log/haproxy/access.log')
+    max_bytes = _env_int('HAPROXY_ACCESS_LOG_MAX_BYTES', 40 * 1024 * 1024, 64 * 1024)
+    backups = _env_int('HAPROXY_ACCESS_LOG_BACKUPS', 4, 0)
+
+    def report(msg):
+        sys.stderr.write(f'haproxy-log-receiver: {msg}\n')
+        sys.stderr.flush()
+
+    writer = RotatingWriter(log_path, max_bytes, backups, report)
+    sock = bind_socket(sock_path)
+    report(f'listening on {sock_path} -> {log_path} '
+           f'(rotate at {max_bytes} bytes, keep {backups} gz)')
+    try:
+        serve(sock, writer)
+    finally:
+        writer.close()
+
+
+if __name__ == '__main__':
+    main()

scripts/healthcheck.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Container liveness probe: BOTH planes must answer quickly.
+#
+#   1. Management API (:8000) -- GET /health with a short --max-time. On
+#      2026-10-09 (sdbees) the API workers' thread pools were wedged for long
+#      enough that /health hung 40s+, while the container reported "healthy"
+#      because the deployed --health-cmd only probed :80. A hung API is a dead
+#      control plane (every WHP site update times out), so it must turn the
+#      container unhealthy.
+#   2. HAProxy data plane (:80) -- any HTTP answer at all (the default page,
+#      a redirect, a 4xx/5xx) proves HAProxy accepted and answered. Only a
+#      connect/timeout failure fails it, which is why there is no -f here.
+#
+# 127.0.0.1, not localhost: the API binds IPv4 only, and resolving localhost
+# to ::1 first costs a failed connect on every probe.
+#
+# WHP's lib-edge-container.sh overrides the image HEALTHCHECK with its own
+# --health-cmd; point that at this script so the fleet gets the same probe.
+set -u
+API_TIMEOUT="${HAPROXY_MGR_HEALTH_API_TIMEOUT:-4}"
+EDGE_TIMEOUT="${HAPROXY_MGR_HEALTH_EDGE_TIMEOUT:-5}"
+
+if ! curl -sf -o /dev/null --max-time "$API_TIMEOUT" http://127.0.0.1:8000/health; then
+    echo "management API :8000 /health failed or took longer than ${API_TIMEOUT}s"
+    exit 1
+fi
+if ! curl -s -o /dev/null --max-time "$EDGE_TIMEOUT" http://127.0.0.1/; then
+    echo "HAProxy :80 did not answer within ${EDGE_TIMEOUT}s"
+    exit 1
+fi
+exit 0

scripts/monitor-attacks.sh

@@ -20,13 +20,13 @@
 #   against.
 #
 # Usage: monitor-attacks.sh [live]
-# Env:   LOG_FILE=<path>  access log to read (default /var/log/haproxy.log)
+# Env:   LOG_FILE=<path>  access log to read (default /var/log/haproxy/access.log)
 #        LOG_LINES=<n>    how many trailing log lines to scan (default 500)
 
 set -uo pipefail
 
 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-LOG_FILE="${LOG_FILE:-/var/log/haproxy.log}"
+LOG_FILE="${LOG_FILE:-/var/log/haproxy/access.log}"
 LOG_LINES="${LOG_LINES:-500}"
 
 # --- Section 1: current rates (real stick-table data) -----------------------
@@ -47,12 +47,12 @@ show_recent_blocks() {
   syslog on the DOCKER HOST, and the file lives on the host, not in here.
   Read it from the host instead:
 
-    grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+    grep -aE ' (PT|PR)--' /var/log/haproxy/access.log | tail -20   # tarpit / deny
-    grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limited / blocked
+    grep -aE ' (429|403) ' /var/log/haproxy/access.log | tail -20  # rate-limited / blocked
-    grep -a 'cip=<IP>'  /var/log/haproxy.log | tail -50     # one client IP
+    grep -a 'cip=<IP>'  /var/log/haproxy/access.log | tail -50     # one client IP
-    grep -a 'id=<uuid>' /var/log/haproxy.log                # one request reference
+    grep -a 'id=<uuid>' /var/log/haproxy/access.log                # one request reference
                                                             # (the UUID on the block page)
-    tail -f /var/log/haproxy.log | grep -aE ' (PT|PR)--'    # live
+    tail -f /var/log/haproxy/access.log | grep -aE ' (PT|PR)--'    # live
 
   Or point this script at a copy:  LOG_FILE=/path/to/haproxy.log $0
 MSG

scripts/pkg-cache-repos.sh

@@ -0,0 +1,296 @@
+#!/bin/bash
+## pkg-cache-repos.sh — BUILD-TIME ONLY, OPT-IN. Points dnf/apt at the internal
+## package cache (nginx proxy_cache, CT130 "pkg-cache" on the Proxmox cluster)
+## when PKG_CACHE is set; without it the repo files are left alone and builds
+## use the upstream repositories as before.
+##
+##   PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh rpm      # AlmaLinux/EPEL/Remi/NodeSource
+##   PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh apt      # Ubuntu, Debian, LiteSpeed
+##   pkg-cache-repos.sh release-install epel [remi]             # GPG-verified release RPMs (EL9/EL10)
+##   pkg-cache-repos.sh nodesource MAJOR                        # NodeSource repo file + pinned key
+##   eval "$(pkg-cache-repos.sh go-env)"                        # GOPROXY via the cache (go.sum/sumdb verified)
+##   pkg-cache-repos.sh restore                                 # undo everything above
+##
+## CANONICAL copy: cloud-apache-container/scripts/pkg-cache-repos.sh (docs/pkg-cache.md
+## there). Other repos carry byte-identical copies - change it there first.
+##
+## Every RUN that rewrites must `restore` before it ends: it puts rewritten files
+## back, deletes files this script created, deletes any NEW repo/apt file that
+## still references the cache, then FAILS if any reference is left — so no shipped
+## layer carries the cache's LAN address (images run on hosts that can't reach it).
+##
+## Why: on 2026-10-21 (PHP security release) builds failed on Remi mirrors lagging
+## hours behind origin, Remi's origin timing out over HTTPS, and an EPEL mirror
+## with a broken certificate. The cache fetches from the ORIGINS over HTTPS (with
+## origin/mirror fallbacks and stale-on-error), keeps metadata ~3 minutes and
+## immutable .rpm/.deb files long-term.
+##
+## Integrity (security review 2026-10-21):
+##  * PKG_CACHE must be https://. The cache's certificate is issued by the
+##    name-constrained "pkg-cache internal CA" (keys/pkg-cache/ca.crt, valid only
+##    for 192.168.1.40 / pkg-cache.*). That CA is trusted ONLY by the rewritten
+##    repos (dnf sslcacert= bundle, apt per-host CaInfo) and only during the RUN —
+##    never added to the system trust store.
+##  * gpgcheck=1 is forced on every rewritten repo; repo_gpgcheck=1 on Remi and
+##    AlmaLinux, which sign repomd.xml (measured: repomd.xml.asc 200). EPEL
+##    publishes no repomd signature (404), so its metadata relies on HTTPS end
+##    to end (client->cache->dl.fedoraproject.org/mirrors, all verified).
+##  * release-install downloads epel-release / remi-release (from the cache, or
+##    from the HTTPS origin without PKG_CACHE), checks the package name, imports
+##    the signing keys committed in keys/pkg-cache/ and requires
+##    `rpm -K` "signatures OK" before installing with localpkg_gpgcheck=1.
+##  * The php.net version gates (EXPECT_PHP_MIN / EXPECT_PHP_VERSION) still fail
+##    a build that gets old PHP (replay of old-but-signed metadata).
+##
+## rpm: each rewritten baseurl keeps the upstream origin as a SECOND url, so if
+##      the cache dies mid-build dnf fails over to the origin by itself.
+## apt: apt has no per-source failover; CI only sets PKG_CACHE after probing the
+##      cache (with the CA), and a build without PKG_CACHE uses upstream sources.
+set -euo pipefail
+
+MODE="${1:?usage: pkg-cache-repos.sh rpm|apt|release-install NAME...|nodesource MAJOR|go-env|restore}"
+shift || true
+STATE=/var/lib/pkg-cache-repos          # saved originals + manifest + created files
+SHARE="${PKG_CACHE_SHARE:-/usr/local/share/pkg-cache}"   # ca.crt + RPM-GPG-KEY-*
+C="${PKG_CACHE:-}"
+C="${C%/}"
+
+die() { echo "FATAL(pkg-cache-repos): $*" >&2; exit 1; }
+
+if [ -n "$C" ]; then
+    case "$C" in
+        https://*) ;;
+        *) die "PKG_CACHE must be an https:// URL (plain http is refused), got '$C'" ;;
+    esac
+    [[ "$C" =~ ^https://[A-Za-z0-9.-]+(:[0-9]+)?(/[A-Za-z0-9._/-]*)?$ ]] || die "PKG_CACHE has unexpected characters: '$C'"
+    HOSTPORT="${C#https://}"; HOSTPORT="${HOSTPORT%%/*}"
+    [ -s "$SHARE/ca.crt" ] || die "PKG_CACHE set but $SHARE/ca.crt (pkg-cache CA) is missing"
+fi
+
+# System CA bundle (EL vs Debian/Ubuntu path); the pkg-cache CA is appended to a
+# copy in the state dir, never to the system store.
+sys_bundle() {
+    local b
+    for b in /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt; do
+        [ -s "$b" ] && { echo "$b"; return 0; }
+    done
+    die "no system CA bundle found"
+}
+make_bundle() { cat "$(sys_bundle)" "$SHARE/ca.crt" > "$STATE/ca-bundle.pem"; }
+
+# Files that may hold repo configuration; restore re-checks all of them.
+repo_files() {
+    find /etc/yum.repos.d /etc/dnf /etc/apt -type f 2>/dev/null | sort || true
+}
+
+start_session() {
+    [ -d "$STATE" ] && return 0
+    mkdir -p "$STATE/orig"
+    repo_files > "$STATE/manifest"     # what existed before this RUN touched anything
+    : > "$STATE/created"
+    printf '%s\n' "$HOSTPORT" > "$STATE/host"
+}
+save() {                                # keep the first-seen (pre-rewrite) copy
+    [ -e "$STATE/orig$1" ] && return 0
+    mkdir -p "$STATE/orig$(dirname "$1")"
+    cp -p "$1" "$STATE/orig$1"
+}
+created() { printf '%s\n' "$1" >> "$STATE/created"; }
+
+if [ "$MODE" = restore ]; then
+    [ -d "$STATE" ] || exit 0
+    host=$(cat "$STATE/host")
+    # mapfile from a process substitution, not `cmd | while read`: no pipeline,
+    # so nothing for pipefail to misreport (scripts/tests/lsphp-info-probe.test.sh).
+    mapfile -t saved < <(cd "$STATE/orig" && find . -type f)
+    for rel in "${saved[@]}"; do
+        cat "$STATE/orig/$rel" > "/${rel#./}"
+    done
+    while read -r f; do [ -n "$f" ] && rm -f -- "$f"; done < "$STATE/created"
+    # A repo/apt file that appeared during this RUN and still names the cache
+    # (e.g. written by a package from the rewritten repos) is removed too.
+    mapfile -t current < <(repo_files)
+    for f in "${current[@]}"; do
+        if ! grep -qxF -- "$f" "$STATE/manifest" && grep -qF -- "$host" "$f"; then
+            echo "pkg-cache-repos: removing new file that references the cache: $f"
+            rm -f -- "$f"
+        fi
+    done
+    left=$(repo_files | xargs -r grep -lF -- "$host" 2>/dev/null || true)
+    rm -rf "$STATE"
+    [ -z "$left" ] || die "cache address $host still present after restore in: $left"
+    echo "pkg-cache-repos: original repository files restored (no reference to $host left)"
+    exit 0
+fi
+
+if [ "$MODE" = release-install ]; then
+    [ $# -gt 0 ] || die "release-install needs: epel and/or remi"
+    tmp=$(mktemp -d)
+    curl_opts=(-fsS --proto '=https' --retry 5 --retry-all-errors --max-time 120)
+    if [ -n "$C" ]; then
+        start_session
+        make_bundle
+        curl_opts+=(--cacert "$STATE/ca-bundle.pem")
+    fi
+    files=()
+    el=$(rpm -E '%{?rhel}' 2>/dev/null || true)
+    case "$el" in 9|10) ;; *) die "release-install supports EL9/EL10 only (rpm -E %rhel = '$el')" ;; esac
+    for name in "$@"; do
+        case "$name" in
+            epel) pkg=epel-release key=RPM-GPG-KEY-EPEL-$el
+                  path=/epel/epel-release-latest-$el.noarch.rpm
+                  origin=https://dl.fedoraproject.org/pub/epel/epel-release-latest-$el.noarch.rpm ;;
+            remi) pkg=remi-release key=RPM-GPG-KEY-remi2024
+                  path=/remi/enterprise/remi-release-$el.rpm
+                  origin=https://rpms.remirepo.net/enterprise/remi-release-$el.rpm ;;
+            *) die "unknown release package '$name'" ;;
+        esac
+        url="$origin"; [ -n "$C" ] && url="$C$path"
+        f="$tmp/$pkg.rpm"
+        curl "${curl_opts[@]}" -o "$f" "$url" || die "download failed: $url"
+        rpm --import "$SHARE/$key" || die "cannot import $SHARE/$key"
+        got=$(rpm -qp --nosignature --qf '%{NAME}' "$f" 2>/dev/null || true)
+        [ "$got" = "$pkg" ] || die "$url is '$got', expected $pkg"
+        chk=$(rpm -K "$f" 2>&1 || true)
+        [[ "$chk" == *": digests signatures OK" ]] || die "signature check failed for $url: $chk"
+        echo "pkg-cache-repos: $pkg verified ($chk) from $url"
+        files+=("$f")
+    done
+    dnf install -y --setopt=localpkg_gpgcheck=1 "${files[@]}"
+    rm -rf "$tmp"
+    exit 0
+fi
+
+if [ -z "$C" ] && [ "$MODE" != nodesource ] && [ "$MODE" != go-env ]; then
+    echo "pkg-cache-repos($MODE): PKG_CACHE not set - using upstream repositories"
+    exit 0
+fi
+
+rewrite_rpm() {
+    local f="$1"
+    grep -q -E '^#? ?baseurl=https?://(repo\.almalinux\.org|download\.example|rpms\.remirepo\.net|rpm\.nodesource\.com)/' "$f" || return 0
+    save "$f"
+    # Per [section]: a (commented or live) baseurl on a known origin becomes
+    # "baseurl=<cache>/<prefix>/<path> <origin>/<path>"; that section's
+    # mirrorlist/metalink is commented out (dnf prefers them over baseurl) and it
+    # gets sslcacert=<system bundle + pkg-cache CA>, gpgcheck=1 and, for repos
+    # that sign repomd.xml, repo_gpgcheck=1. Other sections are untouched.
+    awk -v C="$C" -v BUNDLE="$STATE/ca-bundle.pem" '
+        function flush(   i) {
+            for (i = 1; i <= n; i++) {
+                if (hit && buf[i] ~ /^(mirrorlist|metalink)=/) buf[i] = "#" buf[i]
+                if (hit && buf[i] ~ /^(sslcacert|gpgcheck|repo_gpgcheck)[ \t]*=/) continue
+                print buf[i]
+            }
+            if (hit) {
+                print "sslcacert=" BUNDLE
+                print "gpgcheck=1"
+                if (signed) print "repo_gpgcheck=1"
+            }
+            n = 0; hit = 0; signed = 0
+        }
+        function rw(line, re, prefix, origin, sig,   p) {
+            if (match(line, re)) {
+                p = substr(line, RSTART + RLENGTH); sub(/[ \t].*$/, "", p)
+                hit = 1; if (sig) signed = 1
+                return "baseurl=" C "/" prefix "/" p " " origin "/" p
+            }
+            return line
+        }
+        /^\[/ { flush() }
+        {
+            l = $0
+            if (l ~ /^#? ?baseurl=/) {
+                l = rw(l, "^#? ?baseurl=https?://repo\\.almalinux\\.org/almalinux/", "almalinux", "https://repo.almalinux.org/almalinux", 1)
+                l = rw(l, "^#? ?baseurl=https?://download\\.example/pub/epel/", "epel", "https://dl.fedoraproject.org/pub/epel", 0)
+                l = rw(l, "^#? ?baseurl=https?://rpms\\.remirepo\\.net/", "remi", "https://rpms.remirepo.net", 1)
+                # NodeSource signs its RPMs but not repomd.xml (404): gpgcheck only.
+                l = rw(l, "^#? ?baseurl=https?://rpm\\.nodesource\\.com/", "nodesource", "https://rpm.nodesource.com", 0)
+            }
+            buf[++n] = l
+        }
+        END { flush() }
+    ' "$f" > "$f.pkgcache" && cat "$f.pkgcache" > "$f" && rm -f "$f.pkgcache"
+}
+
+rewrite_apt() {
+    local f="$1"
+    grep -q -E 'https?://([a-z]{2}\.)?(archive|security)\.ubuntu\.com/|https?://rpms\.litespeedtech\.com/|https?://(deb|security)\.debian\.org/' "$f" || return 0
+    save "$f"
+    # Debian: debian-security first, and "debian" only as a whole path segment,
+    # so .../debian-security is never rewritten as /debian/.
+    sed -i -E \
+        -e "s%https?://(archive|[a-z]{2}\.archive)\.ubuntu\.com/ubuntu/?%$C/ubuntu/%g" \
+        -e "s%https?://security\.ubuntu\.com/ubuntu/?%$C/ubuntu-security/%g" \
+        -e "s%https?://rpms\.litespeedtech\.com/%$C/litespeed/%g" \
+        -e "s%https?://(deb|security)\.debian\.org/debian-security/?([[:space:]]|\$)%$C/debian-security/\\2%g" \
+        -e "s%https?://security\.debian\.org/?([[:space:]]|\$)%$C/debian-security/\\1%g" \
+        -e "s%https?://deb\.debian\.org/debian/?([[:space:]]|\$)%$C/debian/\\1%g" \
+        "$f"
+}
+
+if [ "$MODE" = go-env ]; then
+    # Prints shell exports for `eval`. Without PKG_CACHE: nothing (Go defaults).
+    # GOPROXY "a|b" falls through on ANY error, so a dead cache costs one
+    # failed request, not the build. GOSUMDB stays the default sum.golang.org,
+    # reached directly: modules are still verified against go.sum + the sumdb.
+    [ -n "$C" ] || { echo "pkg-cache-repos(go-env): PKG_CACHE not set - default GOPROXY" >&2; exit 0; }
+    start_session
+    make_bundle
+    printf "export GOPROXY='%s/goproxy|https://proxy.golang.org|direct' SSL_CERT_FILE='%s'\n" "$C" "$STATE/ca-bundle.pem"
+    echo "pkg-cache-repos(go-env): GOPROXY via $C/goproxy (fallback proxy.golang.org)" >&2
+    exit 0
+fi
+
+if [ "$MODE" = nodesource ]; then
+    # Replaces `curl https://rpm.nodesource.com/setup_N.x | bash` (unverified
+    # remote script): writes the same nodesource-nodejs repo (origin URL, so the
+    # shipped image keeps a normal repo file) with gpgcheck=1 and the signing key
+    # committed in keys/pkg-cache/; with PKG_CACHE it is then rewritten like any
+    # other repo and `restore` puts the origin version back.
+    major="${1:?usage: pkg-cache-repos.sh nodesource MAJOR}"
+    [[ "$major" =~ ^[0-9]{1,3}$ ]] || die "bad Node major '$major'"
+    key="$SHARE/RPM-GPG-KEY-nodesource"
+    [ -s "$key" ] || die "missing $key"
+    rpm --import "$key" || die "cannot import $key"
+    rm -f /etc/yum.repos.d/nodesource*.repo
+    f=/etc/yum.repos.d/nodesource-nodejs.repo
+    printf '%s\n' "[nodesource-nodejs]" \
+        "name=Node.js Packages for Linux RPM based distros - \$basearch" \
+        "baseurl=https://rpm.nodesource.com/pub_${major}.x/nodistro/nodejs/\$basearch" \
+        "priority=9" "enabled=1" "gpgcheck=1" \
+        "gpgkey=file://$key" "module_hotfixes=1" > "$f"
+    echo "pkg-cache-repos: wrote $f (Node ${major}.x, key $(basename "$key"))"
+    if [ -n "$C" ]; then
+        start_session
+        make_bundle
+        rewrite_rpm "$f"
+        grep -h '^baseurl=' "$f" | sed 's/^/  /'
+    fi
+    exit 0
+fi
+
+case "$MODE" in
+    rpm)
+        start_session
+        make_bundle
+        for f in /etc/yum.repos.d/*.repo; do [ -f "$f" ] && rewrite_rpm "$f"; done
+        echo "pkg-cache-repos(rpm): repos now via $C:"
+        grep -h '^baseurl=' /etc/yum.repos.d/*.repo | grep -F "$C/" | sort -u | sed 's/^/  /'
+        ;;
+    apt)
+        start_session
+        conf=/etc/apt/apt.conf.d/99pkg-cache
+        # Trust the pkg-cache CA for the cache host ONLY (apt per-host option).
+        printf 'Acquire::https::%s::CaInfo "%s";\n' "${HOSTPORT%%:*}" "$SHARE/ca.crt" > "$conf"
+        created "$conf"
+        for f in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
+            [ -f "$f" ] && rewrite_apt "$f"
+        done
+        echo "pkg-cache-repos(apt): sources now via $C:"
+        # (|| true: Debian 13 images have no /etc/apt/sources.list; grep's exit 2
+        # for the missing file must not fail the build under pipefail)
+        { grep -h -E '^(deb|URIs:)' /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null || true; } | grep -F "$C/" | sort -u | sed 's/^/  /' || true
+        ;;
+    *) die "unknown mode '$MODE'" ;;
+esac

scripts/renew-certificates.sh

@@ -65,13 +65,22 @@ fi
 # Copy certificates for each domain
 UPDATED=0
 FAILED=0
+SKIPPED_EXPIRED=0
 
 while read -r domain; do
     CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
     KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
-    if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
+    if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
+        # Never publish an expired lineage. It cannot help any client and,
+        # because HAProxy serves the first file claiming an SNI name, it can
+        # shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
+        # Counted separately from FAILED: a dead lineage that keeps expiring is
+        # reported on every run (ERROR log) without making every run "fail".
+        log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
+        SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
+    elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
         # Assemble in a staging dir and rename into place. NEVER redirect into
         # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
         # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -89,10 +98,15 @@ while read -r domain; do
     fi
 done <<< "$DOMAINS"
 
-log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
+# Already-published pems that have expired since are moved out too (they
+# can only shadow; see cert_retire_expired_pems).
+cert_retire_expired_pems
+RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
+
+log_info "Certificate update completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
 
 # Reload HAProxy if any certificates were updated
-if [ $UPDATED -gt 0 ]; then
+if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
     # Never reload onto unvalidated material: a reload that fails to load the
     # certs directory drops HTTPS for every site on this host.
     if ! haproxy_config_ok; then

scripts/show-tarpit-ips.sh

@@ -12,9 +12,9 @@
 #
 # The stick table also keeps NO history, so nothing in it can identify who was
 # tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST
-# at /var/log/haproxy.log (it does not exist inside this container):
+# at /var/log/haproxy/access.log (inside this container since 2026.11.2):
-#   grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+#   grep -aE ' (PT|PR)--' /var/log/haproxy/access.log | tail -20   # tarpit / deny
-#   grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limit / block
+#   grep -aE ' (429|403) ' /var/log/haproxy/access.log | tail -20  # rate-limit / block
 #
 # What IS knowable from the stick table — the current per-IP rates — is printed
 # by show-edge-ip-rates.sh, which this shim now runs.
@@ -25,8 +25,8 @@ cat >&2 <<'NOTE'
 NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits.
       The HAProxy stick table stores no history and no gpc0/gpc1 counters, so
       the old "Scan Count"/"BLOCKED" columns were fabricated numbers.
-      Actual tarpit/deny events are in /var/log/haproxy.log ON THE HOST:
+      Actual tarpit/deny events are in /var/log/haproxy/access.log (in this container since 2026.11.2):
-        grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+        grep -aE ' (PT|PR)--' /var/log/haproxy/access.log | tail -20
       Running show-edge-ip-rates.sh instead (current rates, real values):
 
 NOTE

scripts/start-up.sh

@@ -55,6 +55,24 @@ if [ ! -f /etc/haproxy/wpadmin_gate_exempt.list ]; then
     fi
 fi
 
+# HAProxy access log receiver (see scripts/haproxy-log-receiver.py and the
+# `log` line in hap_header.tpl). HAProxy sends datagrams to a UNIX socket
+# inside its chroot; this process writes them to a size-capped, rotating
+# /var/log/haproxy/access.log (WHP mounts the haproxy-logs volume there).
+# Started BEFORE init.py so haproxy's first lines land. Supervised by a
+# restart loop: if it dies it is back within 2s, and while it is down HAProxy
+# just drops log lines (non-blocking sends), it never blocks on them.
+# stdout is discarded on purpose -- it never prints log lines, and
+# `docker logs` must not carry a per-request firehose (WHP's AI monitor reads
+# it); its rare status/error lines go to stderr.
+mkdir -p /var/log/haproxy /var/lib/haproxy
+(
+    while true; do
+        python /haproxy/scripts/haproxy-log-receiver.py >/dev/null || true
+        sleep 2
+    done
+) &
+
 cron &
 
 # Phase 1: container init
@@ -77,7 +95,7 @@ HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
 ) &
 
 # Phase 2: WSGI servers
-# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
+# Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
 # (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
 # 1000 — worker recycle frequency).
 #
@@ -91,12 +109,27 @@ API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
 MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
 MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
 
+# Worker class (incident 2026-10-09): stock gthread makes every client socket
+# fully blocking with NO timeout, so any peer that sends a few bytes and goes
+# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
+# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. Stock gthread
+# also never reaps connections that send nothing (2x1000 of them wedge it
+# permanently) and has no total header-read bound (slowloris). This subclass
+# adds: a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, 20s), a
+# first-byte deadline (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s), a total
+# request-line+headers deadline (HAPROXY_MGR_HEADER_DEADLINE, 10s), and a
+# watchdog that exits a worker whose threads have ALL been stuck for
+# HAPROXY_MGR_WEDGE_SECONDS (3600) so the master respawns it.
+# See haproxy_gunicorn_worker.py. --timeout below does NOT cover any of these:
+# for gthread it only watches the main thread's heartbeat.
+WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
+
 # Default page server on :8080. Stays in the background.
 # --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
 # without forking. --max-requests recycles the worker to bound memory drift.
 gunicorn \
     --bind 0.0.0.0:8080 \
-    --workers 1 --threads 4 --worker-class gthread \
+    --workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout 30 \
     --access-logfile - --error-logfile - --log-level info \
@@ -108,7 +141,7 @@ gunicorn \
 # up). Longer --timeout because cert issuance hits ACME and can take a while.
 exec gunicorn \
     --bind 0.0.0.0:8000 \
-    --workers "${API_WORKERS}" --threads 4 --worker-class gthread \
+    --workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout "${API_TIMEOUT}" \
     --access-logfile - --error-logfile - --log-level info \

scripts/sync-certificates.sh

@@ -47,13 +47,22 @@ fi
 # Copy certificates for each domain
 UPDATED=0
 FAILED=0
+SKIPPED_EXPIRED=0
 
 while read -r domain; do
     CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
     KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
-    if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
+    if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
+        # Never publish an expired lineage. It cannot help any client and,
+        # because HAProxy serves the first file claiming an SNI name, it can
+        # shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
+        # Counted separately from FAILED: a dead lineage that keeps expiring is
+        # reported on every run (ERROR log) without making every run "fail".
+        log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
+        SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
+    elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
         # Assemble in a staging dir and rename into place. NEVER redirect into
         # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
         # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -71,10 +80,15 @@ while read -r domain; do
     fi
 done <<< "$DOMAINS"
 
-log_info "Certificate sync completed: $UPDATED updated, $FAILED failed"
+# Already-published pems that have expired since are moved out too (they
+# can only shadow; see cert_retire_expired_pems).
+cert_retire_expired_pems
+RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
+
+log_info "Certificate sync completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
 
 # Reload HAProxy if any certificates were updated
-if [ $UPDATED -gt 0 ]; then
+if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
     # Never reload onto unvalidated material: a reload that fails to load the
     # certs directory drops HTTPS for every site on this host.
     if ! haproxy_config_ok; then

scripts/test-access-log-receiver.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""Tests for scripts/haproxy-log-receiver.py (in-container access log).
+
+Pins: one datagram == one line (PRI stripped, control chars neutralised),
+the size cap (live file + N gzip generations, oldest dropped), a write error
+(full disk / unwritable dir) drops lines without raising, and an end-to-end
+datagram path through a real UNIX socket.
+
+Running
+-------
+    python3 scripts/test-access-log-receiver.py
+"""
+
+import gzip
+import importlib.util
+import os
+import shutil
+import socket
+import tempfile
+import threading
+import time
+import unittest
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+_spec = importlib.util.spec_from_file_location(
+    'haproxy_log_receiver', os.path.join(HERE, 'haproxy-log-receiver.py'))
+rx = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(rx)
+
+
+class CleanLine(unittest.TestCase):
+
+    def test_strips_pri_and_trailing_newline(self):
+        self.assertEqual(rx.clean_line(b'<150>1 2026-11-08T00:00:00Z h haproxy 7 - - GET /\n'),
+                         '1 2026-11-08T00:00:00Z h haproxy 7 - - GET /')
+
+    def test_embedded_newlines_cannot_forge_lines(self):
+        out = rx.clean_line(b'<150>1 x ua=evil\r\nFAKE 200 line\x1b[2J')
+        self.assertNotIn('\n', out)
+        self.assertNotIn('\r', out)
+        self.assertNotIn('\x1b', out)
+        self.assertIn('FAKE 200 line', out)
+
+    def test_invalid_utf8_does_not_raise(self):
+        self.assertIn('�', rx.clean_line(b'<150>1 \xff\xfe'))
+
+    def test_non_pri_angle_bracket_kept(self):
+        self.assertEqual(rx.clean_line(b'<html> x'), '<html> x')
+
+
+class Rotation(unittest.TestCase):
+
+    def setUp(self):
+        self.dir = tempfile.mkdtemp(prefix='hlr-test-')
+        self.path = os.path.join(self.dir, 'access.log')
+
+    def tearDown(self):
+        shutil.rmtree(self.dir, ignore_errors=True)
+
+    def _files(self):
+        return sorted(os.listdir(self.dir))
+
+    def test_size_cap_and_generations(self):
+        max_bytes, backups = 4096, 3
+        w = rx.RotatingWriter(self.path, max_bytes, backups)
+        line = 'x' * 99  # 100 bytes with newline
+        for i in range(1000):  # ~100 KB -> ~24 rotations
+            w.write(f'{i:05d}' + line[5:])
+        w.close()
+        files = self._files()
+        self.assertEqual(files, ['access.log', 'access.log.1.gz',
+                                 'access.log.2.gz', 'access.log.3.gz'], files)
+        self.assertLessEqual(os.path.getsize(self.path), max_bytes)
+        total = sum(os.path.getsize(os.path.join(self.dir, f)) for f in files)
+        self.assertLessEqual(total, (1 + backups) * max_bytes)
+        # newest generation holds the lines written just before the live file
+        with gzip.open(os.path.join(self.dir, 'access.log.1.gz'), 'rt') as fh:
+            gen1 = fh.read().splitlines()
+        with open(self.path) as fh:
+            live = fh.read().splitlines()
+        self.assertEqual(int(gen1[-1][:5]) + 1, int(live[0][:5]))
+        self.assertEqual(live[-1][:5], '00999')
+        for l in gen1:
+            self.assertEqual(len(l), 99)
+
+    def test_zero_backups_keeps_only_live_file(self):
+        w = rx.RotatingWriter(self.path, 1024, 0)
+        for _ in range(100):
+            w.write('y' * 63)
+        w.close()
+        self.assertEqual(self._files(), ['access.log'])
+        self.assertLessEqual(os.path.getsize(self.path), 1024)
+
+    def test_reopens_existing_file_and_counts_its_size(self):
+        with open(self.path, 'w') as fh:
+            fh.write('z' * 1000 + '\n')
+        w = rx.RotatingWriter(self.path, 1500, 2)
+        w.write('a' * 600)  # 1001 + 601 > 1500 -> rotate first
+        w.close()
+        self.assertIn('access.log.1.gz', self._files())
+        with open(self.path) as fh:
+            self.assertEqual(fh.read(), 'a' * 600 + '\n')
+
+    def test_write_error_drops_and_never_raises(self):
+        blocker = os.path.join(self.dir, 'not-a-dir')
+        with open(blocker, 'w') as fh:
+            fh.write('')
+        msgs = []
+        w = rx.RotatingWriter(os.path.join(blocker, 'access.log'), 4096, 2, msgs.append)
+        for _ in range(50):
+            w.write('line')
+        w.flush()
+        w.close()
+        self.assertEqual(w.dropped, 50)
+        self.assertEqual(len(msgs), 1, msgs)  # rate-limited report
+
+
+class EndToEnd(unittest.TestCase):
+
+    def test_datagrams_become_lines(self):
+        d = tempfile.mkdtemp(prefix='hlr-e2e-')
+        try:
+            sock_path = os.path.join(d, 'log.sock')
+            log_path = os.path.join(d, 'logs', 'access.log')
+            sock = rx.bind_socket(sock_path)
+            self.assertEqual(os.stat(sock_path).st_mode & 0o777, 0o666)
+            w = rx.RotatingWriter(log_path, 1 << 20, 2)
+            stop = threading.Event()
+            t = threading.Thread(target=rx.serve, args=(sock, w, stop), daemon=True)
+            t.start()
+            cl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+            cl.setblocking(False)
+            for i in range(20):
+                msg = f'<150>1 t h haproxy 1 - - req {i} host=a.example cip=192.0.2.1'.encode()
+                # The default net.unix.max_dgram_qlen is 10 per netns: a full
+                # queue is EAGAIN (HAProxy drops the line); here, retry.
+                for _ in range(100):
+                    try:
+                        cl.sendto(msg, sock_path)
+                        break
+                    except BlockingIOError:
+                        time.sleep(0.01)
+            deadline = time.time() + 5
+            lines = []
+            while time.time() < deadline:
+                time.sleep(0.2)
+                w.flush()
+                if os.path.exists(log_path):
+                    with open(log_path) as fh:
+                        lines = fh.read().splitlines()
+                if len(lines) == 20:
+                    break
+            stop.set()
+            t.join(3)
+            sock.close()
+            w.close()
+            self.assertEqual(len(lines), 20, lines)
+            self.assertTrue(lines[0].startswith('1 t h haproxy 1 - - req 0 host=a.example'))
+        finally:
+            shutil.rmtree(d, ignore_errors=True)
+
+    def test_sender_to_dead_receiver_fails_fast_nonblocking(self):
+        """What HAProxy sees when the receiver is down: an immediate error,
+        never a block (HAProxy then just counts a dropped log line)."""
+        d = tempfile.mkdtemp(prefix='hlr-dead-')
+        try:
+            cl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+            cl.setblocking(False)
+            start = time.monotonic()
+            with self.assertRaises(OSError):
+                cl.sendto(b'x', os.path.join(d, 'log.sock'))
+            self.assertLess(time.monotonic() - start, 0.5)
+        finally:
+            shutil.rmtree(d, ignore_errors=True)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-cert-scripts.py

@@ -73,6 +73,15 @@ LIB = os.path.join(SCRIPTS_DIR, 'cert-publish-lib.sh')
 BROKEN_TOKEN = '__BROKEN__'
 DOMAIN = 'test.example.com'
 
+# Real SAN certificates (see scripts/test-fixtures/san-certs/). expired_old is
+# the brain-jar.com shape: a multi-SAN certificate that expired in 2020.
+SAN_FIXTURES = os.path.join(SCRIPTS_DIR, 'test-fixtures', 'san-certs')
+
+
+def san_fixture(name):
+    with open(os.path.join(SAN_FIXTURES, name)) as fh:
+        return fh.read()
+
 # --- test key material -------------------------------------------------------
 # openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \
 #         -subj /CN=test.example.com
@@ -618,6 +627,72 @@ class CertScriptBehaviour:
         self.assertFalse(self.reload_attempted())
 
 
+    # -- never publish an expired certificate (brainjar.net, 2026-10-07) ----
+    def test_expired_lineage_is_skipped_and_reported(self):
+        """renew-certificates.sh used to re-publish EVERY live/ lineage every 12h,
+        expired or not; the expired brain-jar.com bundle then shadowed the valid
+        brainjar.net bundle for the names they shared."""
+        dead = os.path.join(self.le_live, 'brain-jar.com')
+        os.makedirs(dead)
+        write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
+        write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
+
+        result = self.run_it()
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertFalse(
+            os.path.exists(os.path.join(self.certs_dir, 'brain-jar.com.pem')),
+            'an expired lineage was published into the crt directory')
+        # The healthy lineage next to it is still published.
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+        self.assertIn('Skipping EXPIRED certificate lineage brain-jar.com', self.logs())
+        self.assertIn('1 updated, 0 failed, 1 expired lineage(s) skipped', self.logs())
+        self.assertIn('brain-jar.com', read(self.error_log),
+                      'the skip must land in the monitored error log')
+
+    def test_expired_lineage_does_not_replace_an_existing_pem(self):
+        dead = os.path.join(self.le_live, 'brain-jar.com')
+        os.makedirs(dead)
+        write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
+        write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
+        existing = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'), OTHER_BUNDLE)
+
+        self.run_it()
+
+        self.assertEqual(read(existing), OTHER_BUNDLE,
+                         'an expired lineage overwrote the file in the crt dir')
+
+    def test_already_published_expired_pem_is_retired(self):
+        """Review M2: skipping publication is not enough - an expired pem that
+        is ALREADY in the crt dir (brain-jar.com.pem) keeps shadowing."""
+        expired = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'),
+                        san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertFalse(os.path.exists(expired), 'expired pem still in the crt dir')
+        retired = os.path.join(self.haproxy_dir, 'cert-retired', 'brain-jar.com.pem')
+        self.assertTrue(os.path.exists(retired), 'expired pem was not moved to cert-retired/ (never delete)')
+        self.assertIn('Retired EXPIRED certificate brain-jar.com.pem', self.logs())
+        self.assertIn('1 expired pem(s) retired', self.logs())
+        self.assertTrue(self.reload_attempted())
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE, 'valid neighbours untouched')
+        self.assert_certs_dir_is_clean()
+
+    def test_retiring_an_expired_pem_alone_triggers_a_reload(self):
+        self.seed_previous_bundle(NEW_BUNDLE)
+        shutil.rmtree(self.domain_dir)
+        other = os.path.join(self.le_live, 'other.example')
+        os.makedirs(other)
+        write(os.path.join(other, 'fullchain.pem'), san_fixture('expired_old.crt'))
+        write(os.path.join(other, 'privkey.pem'), san_fixture('expired_old.key'))
+        write(os.path.join(self.certs_dir, 'other.example.pem'),
+              san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertIn('0 updated', self.logs())
+        self.assertTrue(self.reload_attempted(), 'HAProxy must reload to drop the retired pem')
+
+
 class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture):
     SCRIPT = 'renew-certificates.sh'
 
@@ -638,6 +713,52 @@ class TestCertPublishLibrary(CertScriptFixture):
         path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
         self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0)
 
+    def test_cert_is_expired_reports_both_directions(self):
+        expired = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
+        valid = write(os.path.join(self.tmp, 'valid.crt'), san_fixture('new_bundle.crt'))
+        self.assertEqual(0, self.call('cert_is_expired "$1"', expired).returncode,
+                         'an expired certificate was not recognised as expired')
+        self.assertEqual(1, self.call('cert_is_expired "$1"', valid).returncode,
+                         'a valid certificate was reported as expired')
+        self.assertEqual(1, self.call('cert_is_expired "$1"',
+                                      os.path.join(self.tmp, 'missing')).returncode)
+
+    def test_expired_sweep_keeps_the_last_pem_and_the_default_cert(self):
+        exp = san_fixture('expired_old.crt') + san_fixture('expired_old.key')
+        only = write(os.path.join(self.certs_dir, 'only.pem'), exp)
+        r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
+        self.assertIn('n=0', r.stdout)
+        self.assertTrue(os.path.exists(only), 'emptied the crt dir: that fails the whole :443 bind')
+        dflt = write(os.path.join(self.certs_dir, 'default_self_signed_cert.pem'), exp)
+        r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
+        self.assertIn('n=1', r.stdout)
+        self.assertTrue(os.path.exists(dflt), 'the default self-signed cert is not ours to retire')
+        self.assertFalse(os.path.exists(only))
+
+    def test_expired_sweep_never_touches_a_valid_pem(self):
+        valid = write(os.path.join(self.certs_dir, 'a.pem'), NEW_BUNDLE)
+        write(os.path.join(self.certs_dir, 'b.pem'), OTHER_BUNDLE)
+        r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
+        self.assertIn('n=0', r.stdout)
+        self.assertEqual(read(valid), NEW_BUNDLE)
+
+    def test_cert_publish_refuses_an_expired_certificate(self):
+        cert = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
+        key = write(os.path.join(self.tmp, 'expired.key'), san_fixture('expired_old.key'))
+        dest = os.path.join(self.certs_dir, 'brain-jar.com.pem')
+
+        result = self.call('cert_publish "$1" "$2" "$3"', cert, key, dest)
+
+        self.assert_rejected(result, because='expired')
+        self.assertFalse(os.path.exists(dest), 'expired certificate was published')
+        # And the same material with a valid certificate IS publishable, so the
+        # refusal above is about expiry, not about the fixture.
+        good_cert = write(os.path.join(self.tmp, 'good.crt'), san_fixture('new_bundle.crt'))
+        good_key = write(os.path.join(self.tmp, 'good.key'), san_fixture('new_bundle.key'))
+        ok = self.call('cert_publish "$1" "$2" "$3"', good_cert, good_key, dest)
+        self.assertEqual(0, ok.returncode, ok.stdout + ok.stderr)
+        self.assertTrue(os.path.exists(dest))
+
     def test_empty_and_missing_bundles_rejected(self):
         empty = write(os.path.join(self.tmp, 'empty.pem'), '')
         self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),

scripts/test-cert-write-safety.py

@@ -57,6 +57,7 @@ import subprocess
 import sys
 import tempfile
 import textwrap
+import time
 import unittest
 
 BROKEN_TOKEN = '__BROKEN__'
@@ -254,6 +255,9 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
 
 # certbot stub: succeeds, announces a renewal, and records every invocation so
 # tests can assert that `certbot delete` did or did not run.
+#   TEST_CERTBOT_INSTALL_SRC/_DIR: on certonly, copy fullchain.pem + privkey.pem
+#       from SRC into DIR (a real issuance changes the lineage's certificate).
+#   TEST_CERTBOT_NOT_DUE: on certonly, print certbot's "not yet due" wording.
 FAKE_CERTBOT = textwrap.dedent("""\
     #!/bin/sh
     if [ -n "$TEST_CERTBOT_LOG" ]; then
@@ -262,6 +266,14 @@ FAKE_CERTBOT = textwrap.dedent("""\
     case "$1" in
       renew) echo "Congratulations, all renewals succeeded" ;;
       delete) [ -n "$TEST_CERTBOT_DELETE_FAILS" ] && exit 1 ;;
+      certonly)
+        if [ -n "$TEST_CERTBOT_INSTALL_SRC" ]; then
+          mkdir -p "$TEST_CERTBOT_INSTALL_DIR"
+          cp "$TEST_CERTBOT_INSTALL_SRC/fullchain.pem" "$TEST_CERTBOT_INSTALL_SRC/privkey.pem" "$TEST_CERTBOT_INSTALL_DIR/"
+        fi
+        if [ -n "$TEST_CERTBOT_NOT_DUE" ]; then
+          echo "Certificate not yet due for renewal; no action taken."
+        fi ;;
     esac
     exit 0
 """)
@@ -331,6 +343,7 @@ class CertPublishTestCase(unittest.TestCase):
             'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'),
             'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'),
             'SSL_CERTS_DIR': self.certs,
+            'LETSENCRYPT_DIR': os.path.join(self.tmp, 'letsencrypt'),
             'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'),
             'API_KEY': None,
         }
@@ -574,7 +587,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
 
     @FIX_ONLY
     @NEEDS_OPENSSL
-    def test_lineage_is_deleted_only_after_haproxy_loads_the_bundle(self):
+    def test_lineage_is_untouched_when_haproxy_does_not_load_the_bundle(self):
         """A reload failure must leave the lineage intact and the file recoverable."""
         old_path = self._setup_supersede(broken=False)
         # Make generate_config() produce a config the validator rejects, so the
@@ -591,7 +604,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
             [], self.certbot_deletes(),
             'the lineage must not be deleted when HAProxy did not reload')
         self.assertFalse(os.path.exists(old_path))
-        quarantined = os.path.join(hm.cert_backup_dir(),
+        quarantined = os.path.join(hm.cert_retired_dir(),
                                    os.path.basename(old_path))
         self.assertTrue(os.path.exists(quarantined),
                         'the superseded file must be recoverable by hand')
@@ -599,8 +612,9 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
 
     @FIX_ONLY
     @NEEDS_OPENSSL
-    def test_successful_bundle_still_supersedes_and_deletes(self):
+    def test_successful_bundle_still_supersedes_without_deleting(self):
-        """The cleanup must still do its job on the happy path."""
+        """The cleanup must still do its job on the happy path - by moving, never
+        by `certbot delete` (irreversible; recovery is a rate-limited ACME order)."""
         old_path = self._setup_supersede(broken=False)
 
         resp = self.client.post('/api/ssl/bundle', json={
@@ -612,12 +626,381 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
         self.assertFalse(os.path.exists(old_path),
                          'the superseded file must leave the crt directory or '
                          'it keeps shadowing the new bundle')
-        self.assertEqual(['delete --cert-name www.test.example.com -n'],
+        self.assertTrue(os.path.exists(os.path.join(
-                         self.certbot_deletes())
+            hm.cert_retired_dir(), os.path.basename(old_path))),
+            'the superseded file must be moved aside, not destroyed')
+        self.assertEqual([], self.certbot_deletes(),
+                         'retirement must never `certbot delete` anything')
         self.assertTrue(self.edge_would_start())
         self.assert_only_final_pems_in_certs_dir()
 
 
+
+# --- SAN-overlap retirement (brainjar.net, 2026-10-07) -----------------------
+SAN_FIXTURES = os.path.join(MODULE_DIR, 'scripts', 'test-fixtures', 'san-certs')
+
+
+def san(name):
+    with open(os.path.join(SAN_FIXTURES, name)) as fh:
+        return fh.read()
+
+
+def make_certbot_lineage(le_dir, lineage, cert, key):
+    """archive/<x>/{cert,chain,fullchain,privkey}1.pem + live/<x>/*.pem
+    symlinks (../../archive/<x>/...1.pem) + renewal/<x>.conf, like certbot."""
+    archive = os.path.join(le_dir, 'archive', lineage)
+    live = os.path.join(le_dir, 'live', lineage)
+    os.makedirs(archive, exist_ok=True)
+    os.makedirs(live, exist_ok=True)
+    for kind, body in (('cert', cert), ('chain', cert), ('fullchain', cert), ('privkey', key)):
+        with open(os.path.join(archive, f'{kind}1.pem'), 'w') as fh:
+            fh.write(body)
+        link = os.path.join(live, f'{kind}.pem')
+        if os.path.lexists(link):
+            os.remove(link)
+        os.symlink(f'../../archive/{lineage}/{kind}1.pem', link)
+    os.makedirs(os.path.join(le_dir, 'renewal'), exist_ok=True)
+    with open(os.path.join(le_dir, 'renewal', f'{lineage}.conf'), 'w') as fh:
+        fh.write(f'archive_dir = {archive}\ncert = {live}/cert.pem\n')
+    return live
+
+
+def retired_path(le_dir, part, name):
+    hits = sorted(__import__('glob').glob(os.path.join(le_dir, 'retired', '*', part, name)))
+    return hits[-1] if hits else os.path.join(le_dir, 'retired', '<none>', part, name)
+
+
+FIX_ONLY_RETIRE = unittest.skipIf(
+    TESTING_FOREIGN_TREE and not hasattr(hm, 'retirement_blockers'),
+    'HAPROXY_MANAGER_DIR points at a tree without SAN-overlap retirement')
+
+
+class TestRetirementRules(unittest.TestCase):
+    """retirement_blockers() is the shared-PEM safety rule, tested directly."""
+
+    NOW = 2_000_000_000
+
+    def cert(self, names, days_left, fp):
+        return {'names': set(names), 'fingerprint': fp,
+                'not_after': self.NOW + days_left * 86400, 'cn': sorted(names)[0]}
+
+    @FIX_ONLY_RETIRE
+    def test_expired_candidate_never_blocks(self):
+        old = self.cert(['brain-jar.com', 'brainjar.net'], -2, 'A')
+        self.assertEqual([], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'brain-jar.com', 'brainjar.net'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_valid_candidate_that_is_last_cert_for_a_served_name_is_kept(self):
+        old = self.cert(['shared.example', 'brainjar.net', 'other.example'], 40, 'A')
+        self.assertEqual(['other.example'], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'brainjar.net', 'other.example'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_unreachable_names_do_not_block(self):
+        old = self.cert(['dead.example', 'brainjar.net'], 40, 'A')
+        self.assertEqual([], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'brainjar.net', 'dead.example'},
+            ['dead.example'], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_name_covered_by_another_valid_cert_does_not_block(self):
+        old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
+        other = self.cert(['other.example'], 60, 'B')
+        self.assertEqual([], hm.retirement_blockers(
+            old, [other], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_cover_by_an_expired_or_identical_cert_does_not_count(self):
+        old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
+        expired = self.cert(['other.example'], -1, 'B')
+        same = self.cert(['other.example'], 60, 'A')
+        self.assertEqual(['other.example'], hm.retirement_blockers(
+            old, [expired, same], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_unknown_served_set_keeps_every_valid_name(self):
+        old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
+        self.assertEqual(['other.example'], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, None, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_wildcard_coverage(self):
+        self.assertTrue(hm.cert_name_covers('*.example.com', 'a.example.com'))
+        self.assertFalse(hm.cert_name_covers('*.example.com', 'a.b.example.com'))
+        self.assertFalse(hm.cert_name_covers('*.example.com', 'example.com'))
+        old = self.cert(['*.example.com', 'brainjar.net'], 40, 'A')
+        self.assertEqual(['shop.example.com'], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'shop.example.com'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_read_cert_info_reads_sans_and_expiry(self):
+        tmp = tempfile.mkdtemp(prefix='cert-info-')
+        self.addCleanup(shutil.rmtree, tmp, True)
+        path = os.path.join(tmp, 'c.pem')
+        with open(path, 'w') as fh:
+            fh.write(san('expired_old.crt'))
+        info = hm.read_cert_info(path)
+        self.assertEqual({'brain-jar.com', 'www.brain-jar.com', 'brainjar.net',
+                          'www.brainjar.net'}, info['names'])
+        self.assertEqual('brain-jar.com', info['cn'])
+        self.assertEqual(1585699200, info['not_after'])  # 2020-04-01T00:00:00Z
+        self.assertTrue(info['fingerprint'])
+
+
+class TestSanOverlapRetirement(CertPublishTestCase):
+    """The new bundle must not stay shadowed by an older cert sharing a SAN."""
+
+    def setUp(self):
+        super().setUp()
+        self.le = os.path.join(self.tmp, 'letsencrypt')
+        self.new_src = os.path.join(self.tmp, 'issued', 'brainjar.net')
+        self.write(os.path.join(self.new_src, 'fullchain.pem'), san('new_bundle.crt'))
+        self.write(os.path.join(self.new_src, 'privkey.pem'), san('new_bundle.key'))
+        # certbot "issues" brainjar.net into live/ when called.
+        os.environ['TEST_CERTBOT_INSTALL_SRC'] = self.new_src
+        os.environ['TEST_CERTBOT_INSTALL_DIR'] = os.path.join(self.le_live, 'brainjar.net')
+        for name in ('brainjar.net', 'www.brainjar.net'):
+            self.add_domain(name, name.replace('.', '_'))
+
+    def old_lineage(self, lineage, fixture, publish=True):
+        """A lineage laid out the way certbot does it: real files in
+        archive/<x>/*1.pem, relative symlinks in live/<x>/, a renewal conf."""
+        live = make_certbot_lineage(self.le, lineage, san(f'{fixture}.crt'),
+                                    san(f'{fixture}.key'))
+        conf = os.path.join(self.le, 'renewal', f'{lineage}.conf')
+        if not publish:
+            return live, conf, None
+        pem = self.write(os.path.join(self.certs, f'{lineage}.pem'),
+                         san(f'{fixture}.crt') + san(f'{fixture}.key'))
+        return live, conf, pem
+
+    def bundle(self, **extra):
+        body = {'primary': 'brainjar.net', 'sans': ['www.brainjar.net']}
+        body.update(extra)
+        resp = self.client.post('/api/ssl/bundle', json=body)
+        self.assertEqual(200, resp.status_code, resp.get_data(as_text=True))
+        return resp.get_json()
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_expired_overlapping_cert_with_a_different_cn_is_retired(self):
+        """The incident: CN brain-jar.com, SANs include brainjar.net."""
+        self.add_domain('brain-jar.com', 'bjc')
+        self.add_domain('www.brain-jar.com', 'wbjc')
+        live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
+
+        data = self.bundle(unreachable_names=['brain-jar.com', 'www.brain-jar.com'])
+
+        self.assertFalse(os.path.exists(pem),
+                         'the expired SAN-overlapping pem is still shadowing the bundle')
+        retired = [e['file'] for e in data['cleanup']['removed']]
+        self.assertEqual(['brain-jar.com.pem'], retired)
+        self.assertTrue(os.path.exists(os.path.join(hm.cert_retired_dir(), 'brain-jar.com.pem')))
+        # Lineage retired, not deleted: renew-certificates.sh can no longer
+        # re-publish it and certbot can no longer try to renew it.
+        self.assertFalse(os.path.exists(live))
+        self.assertFalse(os.path.exists(conf))
+        # live, archive AND renewal leave together (review H2), into one tree.
+        self.assertFalse(os.path.exists(os.path.join(self.le, 'archive', 'brain-jar.com')))
+        self.assertTrue(os.path.isdir(retired_path(self.le, 'live', 'brain-jar.com')))
+        self.assertTrue(os.path.isdir(retired_path(self.le, 'archive', 'brain-jar.com')))
+        self.assertTrue(os.path.exists(retired_path(self.le, 'renewal', 'brain-jar.com.conf')))
+        # Relative symlinks still resolve in the retired tree (restore = mv back).
+        self.assertIn('BEGIN CERTIFICATE', self.read(
+            os.path.join(retired_path(self.le, 'live', 'brain-jar.com'), 'cert.pem')))
+        self.assertEqual([], self.certbot_deletes())
+        self.assertTrue(os.path.exists(os.path.join(self.certs, 'brainjar.net.pem')))
+        self.assertTrue(self.edge_would_start())
+        self.assert_only_final_pems_in_certs_dir()
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_expired_overlap_is_retired_even_without_the_unreachable_hint(self):
+        self.add_domain('brain-jar.com', 'bjc')
+        live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
+        self.bundle()
+        self.assertFalse(os.path.exists(pem))
+        self.assertFalse(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_shared_pem_still_serving_another_site_is_kept(self):
+        """The shared-PEM hazard: never retire the only valid cert for a served name."""
+        self.add_domain('other-site.example', 'other_site')
+        live, conf, pem = self.old_lineage('shared-old.example', 'shared_old')
+
+        data = self.bundle()
+
+        self.assertTrue(os.path.exists(pem),
+                        'retired the only valid certificate for other-site.example')
+        self.assertTrue(os.path.exists(live))
+        self.assertTrue(os.path.exists(conf))
+        kept = {e['file']: e['blockers'] for e in data['cleanup']['kept']}
+        self.assertEqual({'shared-old.example.pem': ['other-site.example']}, kept)
+        self.assertEqual(['other-site.example'],
+                         data['cleanup']['lineages']['kept'][0]['blockers'])
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_valid_overlap_whose_other_names_are_unreachable_is_retired(self):
+        self.add_domain('dead.example', 'dead')
+        live, conf, pem = self.old_lineage('dead.example', 'dead_old')
+
+        self.bundle(unreachable_names=['dead.example'])
+
+        self.assertFalse(os.path.exists(pem))
+        self.assertFalse(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_valid_overlap_still_served_without_the_hint_is_kept(self):
+        """Same files as above, but nobody said dead.example is gone: keep it."""
+        self.add_domain('dead.example', 'dead')
+        live, conf, pem = self.old_lineage('dead.example', 'dead_old')
+
+        self.bundle()
+
+        self.assertTrue(os.path.exists(pem))
+        self.assertTrue(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_non_overlapping_certificates_are_untouched(self):
+        live, conf, pem = self.old_lineage('unrelated.example', 'unrelated')
+        self.bundle()
+        self.assertTrue(os.path.exists(pem), 'retired a cert sharing no name with the bundle')
+        self.assertTrue(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_response_distinguishes_issuance_from_not_yet_due(self):
+        issued = self.bundle()
+        self.assertTrue(issued['renewed'], 'a fresh issuance was not reported as one')
+        self.assertEqual(4922899200, issued['not_after'])  # 2126-01-01T00:00:00Z
+
+        # Lineage now exists with the same names; certbot keeps it.
+        del os.environ['TEST_CERTBOT_INSTALL_SRC']
+        os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
+        noop = self.bundle()
+        self.assertFalse(noop['renewed'], 'a certbot no-op was reported as a renewal')
+        self.assertTrue(noop['certbot_not_due'])
+        self.assertEqual(4922899200, noop['not_after'])
+        self.assertIn('--keep-until-expiring', self.certbot_invocations()[-1])
+
+
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_not_due_noop_never_retires_its_own_cert_name_lineage(self):
+        """Review H1: find_certbot_live_dir() returns the NEWEST of brainjar.net
+        and brainjar.net-0001. After a certbot no-op that was the stale -0001:
+        it got published as brainjar.net.pem and the real brainjar.net lineage
+        was retired as "superseded"."""
+        del os.environ['TEST_CERTBOT_INSTALL_SRC']
+        os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
+        real = make_certbot_lineage(self.le, 'brainjar.net', san('new_bundle.crt'), san('new_bundle.key'))
+        stale = make_certbot_lineage(self.le, 'brainjar.net-0001', san('dead_old.crt'), san('dead_old.key'))
+        later = time.time() + 60
+        os.utime(os.path.join(self.le, 'archive', 'brainjar.net-0001', 'fullchain1.pem'), (later, later))
+        # Production behaviour of find_certbot_live_dir(): newest wins.
+        hm.find_certbot_live_dir = lambda base: stale if base == 'brainjar.net' else None
+
+        data = self.bundle()
+
+        self.assertTrue(os.path.isdir(real), 'the --cert-name lineage itself was retired')
+        self.assertTrue(os.path.exists(os.path.join(self.le, 'renewal', 'brainjar.net.conf')))
+        self.assertNotIn('brainjar.net', [e['lineage'] for e in data['cleanup']['lineages']['retired']])
+        published = self.read(os.path.join(self.certs, 'brainjar.net.pem'))
+        self.assertIn(san('new_bundle.crt').strip().splitlines()[1], published,
+                      'published the stale -0001 certificate instead of live/brainjar.net')
+        self.assertFalse(data['renewed'])
+
+    @FIX_ONLY_RETIRE
+    def test_bundle_reports_capabilities(self):
+        data = self.bundle()
+        self.assertIn('san_overlap_retirement', data.get('capabilities', []))
+        caps = self.client.get('/api/capabilities').get_json()
+        self.assertIn('unreachable_names', caps['capabilities'])
+
+
+class TestRetiredLineageCanBeReissued(CertPublishTestCase):
+    """Review H2: retirement must leave certbot able to issue the name again."""
+
+    def setUp(self):
+        super().setUp()
+        self.le = hm.LETSENCRYPT_DIR
+
+    @FIX_ONLY_RETIRE
+    def test_retirement_removes_every_part_certbot_checks(self):
+        make_certbot_lineage(self.le, 'brain-jar.com', san('expired_old.crt'), san('expired_old.key'))
+        out = hm.retire_lineage_tree('brain-jar.com')
+        # certbot new_lineage() refuses if any of these exist for the name.
+        for part in (('live', 'brain-jar.com'), ('archive', 'brain-jar.com'),
+                     ('renewal', 'brain-jar.com.conf')):
+            self.assertFalse(os.path.lexists(os.path.join(self.le, *part)), part)
+            self.assertTrue(os.path.lexists(os.path.join(out['retired_to'], *part)), part)
+        self.assertIn('BEGIN CERTIFICATE',
+                      self.read(os.path.join(out['retired_to'], 'live', 'brain-jar.com', 'fullchain.pem')))
+
+    @FIX_ONLY_RETIRE
+    def test_second_retirement_of_the_same_name_does_not_collide(self):
+        make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
+        a = hm.retire_lineage_tree('x.example', stamp='S')
+        make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
+        b = hm.retire_lineage_tree('x.example', stamp='S')
+        self.assertNotEqual(a['retired_to'], b['retired_to'])
+        self.assertTrue(os.path.isdir(os.path.join(a['retired_to'], 'archive', 'x.example')))
+
+    @FIX_ONLY_RETIRE
+    def test_certbot_can_issue_the_name_again_after_retirement(self):
+        """Against certbot's own storage code (skipped where certbot is not
+        importable, e.g. the image's app python)."""
+        # Import AND initialisation are both guarded: a half-installed or
+        # version-skewed certbot can fail with far more than ImportError (a
+        # missing transitive dependency, a pkg_resources/entry-point error, a
+        # moved internal). Any of those means "cannot run this check here",
+        # not "retirement is broken". new_lineage() itself is NOT inside the
+        # guard - that is the assertion.
+        try:
+            import types
+            from certbot._internal import storage
+            real_rv = storage.relevant_values
+            new_lineage = storage.RenewableCert.new_lineage
+            cfg = types.SimpleNamespace(
+                config_dir=self.le, renewal_configs_dir=os.path.join(self.le, 'renewal'),
+                default_archive_dir=os.path.join(self.le, 'archive'),
+                live_dir=os.path.join(self.le, 'live'))
+        except Exception as e:  # noqa: BLE001 - deliberately broad, see above
+            self.skipTest(f'certbot unusable from this python ({type(e).__name__}: {e})')
+        storage.relevant_values = lambda c: {'server': 'https://acme.invalid/directory'}
+        self.addCleanup(setattr, storage, 'relevant_values', real_rv)
+        cert, key = san('new_bundle.crt').encode(), san('new_bundle.key').encode()
+
+        first = new_lineage('brain-jar.com', cert, key, cert, cfg)
+        self.assertEqual('brain-jar.com', first.lineagename)
+        hm.retire_lineage_tree('brain-jar.com')
+        again = new_lineage('brain-jar.com', cert, key, cert, cfg)
+        self.assertEqual('brain-jar.com', again.lineagename,
+                         'certbot could not re-create the lineage under the same name')
+        self.assertTrue(os.path.getsize(os.path.join(self.le, 'renewal', 'brain-jar.com.conf')) > 0)
+
+class TestExpiredCertificateIsNeverPublished(CertPublishTestCase):
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_publisher_refuses_an_expired_leaf(self):
+        src = os.path.join(self.tmp, 'expired')
+        cert = self.write(os.path.join(src, 'fullchain.pem'), san('expired_old.crt'))
+        key = self.write(os.path.join(src, 'privkey.pem'), san('expired_old.key'))
+        live = self.publish_live_bundle('brain-jar.com')
+        with self.assertRaises(hm.CertificatePublishError) as ctx:
+            hm.publish_pem_bundle(live, [cert, key])
+        self.assertIn('expired', str(ctx.exception))
+        self.assertEqual(PREVIOUS_BUNDLE, self.read(live))
+
+
 class TestBundleValidation(CertPublishTestCase):
     """What may and may not be published."""
 

scripts/test-default-splash.py

@@ -0,0 +1,304 @@
+#!/usr/bin/env python3
+"""Regression tests for the default splash (unrouted hosts answered pre-WAF).
+
+Why this file exists
+--------------------
+Visiting a WHP server's own hostname (https://whpNN.<domain>/ on 80/443, not
+the panel port), its bare IP, or any name with no site behind it used to fall
+through to `default_backend default-backend` -- but only AFTER every
+http-request rule had run, Coraza's send-spoe-group included. `use_backend`
+does not short-circuit http-request processing. So the catch-all was WAF
+inspected with the server-wide rule set, and a browser on the server hostname
+also carries the control panel's cookies (cookies are not port-scoped), which
+produced the WAF block page for a host that has nothing behind it to protect.
+
+The splash answers those requests with `http-request return` BEFORE the SPOE
+call. Its safety property is that it must NEVER answer a host that routes to
+a site. That is held by:
+  - routed_hosts.list / routed_wildcards.list being written from the same
+    rows, in the same generate_config() call, as the routing ACLs
+  - the lists being port-stripped + lowercased (a superset of the raw-Host
+    routing ACLs)
+  - the lists being part of the rollback set
+and it must keep ACME http-01, the blocked-IP page and the suspended page.
+
+Running
+-------
+    python3 scripts/test-default-splash.py
+"""
+
+import logging
+import os
+import re
+import sqlite3
+import sys
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+try:
+    import haproxy_manager as hm  # noqa: E402
+finally:
+    logging.FileHandler = _real_file_handler
+logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
+
+SPLASH_FILE = os.path.join(MODULE_DIR, 'errors', 'default-splash.html')
+
+
+def render_listener(**kw):
+    args = dict(crt_path='/etc/haproxy/certs', suspension_enabled=False,
+                coraza_spoe_backend='127.0.0.1:9000', default_splash=True)
+    args.update(kw)
+    return hm.template_env.get_template('hap_listener.tpl').render(**args)
+
+
+def return_rules(cfg):
+    return [l.strip() for l in cfg.splitlines()
+            if l.strip().startswith('http-request return')
+            and ('default-splash.html' in l or 'robots-unrouted.txt' in l)]
+
+
+class Template(unittest.TestCase):
+
+    def test_off_renders_nothing(self):
+        cfg = render_listener(default_splash=False)
+        self.assertNotIn('default-splash', cfg)
+        self.assertNotIn('splash_routed_host', cfg)
+
+    def test_404_everywhere_robots_txt_200(self):
+        """Josh 2026-11-08: unrouted hosts are 404 on EVERY path (incl. "/")
+        so search engines never index the server IP/hostname; /robots.txt is
+        the one 200 and disallows everything."""
+        rules = return_rules(render_listener())
+        self.assertEqual(len(rules), 2, rules)
+        robots, splash = rules
+        self.assertTrue(robots.startswith('http-request return status 200 content-type "text/plain'))
+        self.assertIn(' file /haproxy/errors/robots-unrouted.txt ', robots)
+        self.assertTrue(robots.endswith(' splash_robots'))
+        self.assertTrue(splash.startswith('http-request return status 404 content-type "text/html'))
+        self.assertTrue(splash.endswith(' !splash_robots'))
+        self.assertNotIn('splash_root', render_listener())
+        self.assertRegex(render_listener(), r'acl splash_robots path /robots\.txt\n')
+
+    def test_noindex_and_no_store_headers_on_both(self):
+        for rule in return_rules(render_listener()):
+            self.assertIn('hdr x-robots-tag "noindex, nofollow, noarchive"', rule)
+            self.assertIn('hdr cache-control "no-store"', rule)
+            self.assertIn('hdr x-content-type-options "nosniff"', rule)
+
+    def test_static_file_not_lf_file(self):
+        """`file` (verbatim), never `lf-file`: nothing from the request may be
+        expanded into the page (Host echo, unique-id, ...)."""
+        for rule in return_rules(render_listener()):
+            self.assertRegex(rule, r' file /haproxy/errors/(default-splash\.html|robots-unrouted\.txt) ')
+            self.assertNotIn('lf-file', rule)
+
+    def test_conditions_keep_acme_blocked_and_suspended_out(self):
+        for susp in (False, True):
+            for rule in return_rules(render_listener(suspension_enabled=susp)):
+                cond = rule.split(' if ', 1)[1].split()
+                self.assertIn('!splash_routed_host', cond)
+                self.assertIn('!splash_acme', cond)
+                self.assertIn('!is_blocked_ip', cond)
+                if susp:
+                    self.assertIn('!is_suspended_domain', cond)
+                else:
+                    self.assertNotIn('!is_suspended_domain', cond)
+
+    def test_acme_acl_is_the_challenge_prefix(self):
+        self.assertRegex(render_listener(),
+                         r'acl splash_acme path_beg /\.well-known/acme-challenge/\n')
+
+    def test_routed_acls_strip_port_and_lowercase(self):
+        cfg = render_listener()
+        self.assertIn('acl splash_routed_host req.hdr(host),field(1,:),lower '
+                      '-f /etc/haproxy/routed_hosts.list', cfg)
+        self.assertIn('acl splash_routed_host req.hdr(host),field(1,:),lower '
+                      '-m end -f /etc/haproxy/routed_wildcards.list', cfg)
+
+    def test_splash_runs_before_waf_and_after_ip_block_and_suspension(self):
+        for susp in (False, True):
+            cfg = render_listener(suspension_enabled=susp)
+            first_return = cfg.index('file /haproxy/errors/robots-unrouted.txt')
+            self.assertLess(first_return, cfg.index('http-request send-spoe-group coraza'))
+            self.assertLess(first_return, cfg.index('filter spoe engine coraza'))
+            self.assertGreater(first_return, cfg.index('acl is_blocked_ip'))
+            self.assertGreater(first_return, cfg.index('http-request track-sc0'))
+            if susp:
+                self.assertGreater(first_return, cfg.index('acl is_suspended_domain'))
+
+
+class Page(unittest.TestCase):
+
+    def setUp(self):
+        with open(SPLASH_FILE, 'rb') as fh:
+            self.raw = fh.read()
+        self.text = self.raw.decode('utf-8')
+
+    def test_small_enough_for_one_buffer(self):
+        # http-request return payload must fit tune.bufsize (16384) with headers.
+        self.assertLess(len(self.raw), 8192)
+
+    def test_no_links_scripts_or_external_resources(self):
+        low = self.text.lower()
+        for needle in ('href', 'src=', '<script', '<form', '<a ', 'http://', 'https://',
+                       '8443', '@import', 'url('):
+            self.assertNotIn(needle, low, needle)
+
+    def test_no_version_or_template_expansions(self):
+        self.assertNotRegex(self.text, r'20\d\d\.\d+\.\d+')
+        self.assertNotIn('{{', self.text)
+        self.assertNotIn('%[', self.text)
+
+
+class Robots(unittest.TestCase):
+
+    def test_robots_disallows_everything(self):
+        with open(os.path.join(MODULE_DIR, 'errors', 'robots-unrouted.txt')) as fh:
+            self.assertEqual(fh.read(), 'User-agent: *\nDisallow: /\n')
+
+
+class RoutedLists(unittest.TestCase):
+
+    def test_render_sorted_dedup_lower(self):
+        body = hm.render_routed_list(['B.example.com', 'a.example.com', 'b.example.com ', ''])
+        lines = [l for l in body.splitlines() if not l.startswith('#')]
+        self.assertEqual(lines, ['a.example.com', 'b.example.com'])
+
+    def test_name_with_whitespace_is_dropped_not_split(self):
+        body = hm.render_routed_list(['ok.example.com', 'bad\nevil.example.com', 'a b'])
+        lines = [l for l in body.splitlines() if not l.startswith('#')]
+        self.assertEqual(lines, ['ok.example.com'])
+
+    def test_lists_are_in_the_rollback_set(self):
+        pairs = dict(hm._config_backup_pairs())
+        self.assertIn(hm.ROUTED_HOSTS_PATH, pairs)
+        self.assertIn(hm.ROUTED_WILDCARDS_PATH, pairs)
+
+
+class GenerateConfigWritesLists(unittest.TestCase):
+    """Run the real generate_config() with paths redirected and reload stubbed."""
+
+    def setUp(self):
+        self.tmp = tempfile.mkdtemp(prefix='splash-test-')
+        self.saved = {k: getattr(hm, k) for k in (
+            'DB_FILE', 'HAPROXY_CONFIG_PATH', 'HAPROXY_BACKUP_PATH',
+            'CLUSTER_SECRET_PATH', 'HAPROXY_SOCKET_PATH', 'SSL_CERTS_DIR',
+            'BLOCKED_IPS_MAP_PATH', 'BLOCKED_IPS_MAP_BACKUP_PATH',
+            'CORAZA_SPOE_CONFIG_PATH', 'CORAZA_SPOE_BACKUP_PATH',
+            'ROUTED_HOSTS_PATH', 'ROUTED_HOSTS_BACKUP_PATH',
+            'ROUTED_WILDCARDS_PATH', 'ROUTED_WILDCARDS_BACKUP_PATH',
+            'reload_haproxy_safely', 'create_backup')}
+        j = lambda n: os.path.join(self.tmp, n)  # noqa: E731
+        hm.DB_FILE = j('db.sqlite')
+        hm.HAPROXY_CONFIG_PATH = j('haproxy.cfg')
+        hm.HAPROXY_BACKUP_PATH = j('haproxy.cfg.backup')
+        hm.CLUSTER_SECRET_PATH = j('cluster-secret')
+        hm.HAPROXY_SOCKET_PATH = j('haproxy.sock')
+        hm.SSL_CERTS_DIR = j('certs')
+        hm.BLOCKED_IPS_MAP_PATH = j('blocked_ips.map')
+        hm.BLOCKED_IPS_MAP_BACKUP_PATH = j('blocked_ips.map.backup')
+        hm.CORAZA_SPOE_CONFIG_PATH = j('coraza-spoe.cfg')
+        hm.CORAZA_SPOE_BACKUP_PATH = j('coraza-spoe.cfg.backup')
+        hm.ROUTED_HOSTS_PATH = j('routed_hosts.list')
+        hm.ROUTED_HOSTS_BACKUP_PATH = j('routed_hosts.list.backup')
+        hm.ROUTED_WILDCARDS_PATH = j('routed_wildcards.list')
+        hm.ROUTED_WILDCARDS_BACKUP_PATH = j('routed_wildcards.list.backup')
+        hm.reload_haproxy_safely = lambda *a, **kw: (True, 'stubbed')
+        hm.create_backup = lambda *a, **kw: (True, 'unavailable')
+        self.env = {k: os.environ.pop(k, None) for k in (
+            'HAPROXY_DEFAULT_SPLASH', 'HAPROXY_SUSPENSION_ENABLED',
+            'HAPROXY_CORAZA_SPOE_BACKEND')}
+        hm.init_db()
+        with sqlite3.connect(hm.DB_FILE) as conn:
+            cur = conn.cursor()
+
+            def add(domain, backend, wildcard=0):
+                cur.execute('INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
+                            'template_override, is_wildcard) VALUES (?, 1, NULL, NULL, ?)',
+                            (domain, wildcard))
+                did = cur.lastrowid
+                if backend:
+                    cur.execute('INSERT INTO backends (name, domain_id, settings) VALUES (?, ?, NULL)',
+                                (backend, did))
+                    cur.execute('INSERT INTO backend_servers (backend_id, server_name, '
+                                'server_address, server_port, server_options) VALUES (?, ?, ?, ?, ?)',
+                                (cur.lastrowid, 'web1', '10.0.0.10', 80, 'check'))
+            add('Site.Example.com', 'site')
+            add('*.wild.example.com', 'wild', wildcard=1)
+            add('whp01.example.net', None)   # server hostname: cert-only
+            conn.commit()
+
+    def tearDown(self):
+        for k, v in self.saved.items():
+            setattr(hm, k, v)
+        for k, v in self.env.items():
+            os.environ.pop(k, None)
+            if v is not None:
+                os.environ[k] = v
+        import shutil
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def _lines(self, path):
+        with open(path) as fh:
+            return [l for l in fh.read().splitlines() if l and not l.startswith('#')]
+
+    def test_lists_written_from_routed_rows_only(self):
+        hm.generate_config()
+        self.assertEqual(self._lines(hm.ROUTED_HOSTS_PATH), ['site.example.com'])
+        self.assertEqual(self._lines(hm.ROUTED_WILDCARDS_PATH), ['.wild.example.com'])
+        with open(hm.HAPROXY_CONFIG_PATH) as fh:
+            cfg = fh.read()
+        # every exact routing ACL name is in the list (superset property)
+        for name in re.findall(r'acl \S+-acl hdr\(host\) -i (\S+)', cfg):
+            self.assertIn(name.lower(), self._lines(hm.ROUTED_HOSTS_PATH))
+        for base in re.findall(r'acl \S+-acl hdr_end\(host\) -i (\S+)', cfg):
+            self.assertIn(base.lower(), self._lines(hm.ROUTED_WILDCARDS_PATH))
+        self.assertIn('default-splash.html', cfg)
+
+    def test_default_log_target_is_the_in_container_socket(self):
+        """Access log goes to the receiver's UNIX socket (chroot-relative),
+        never the old host bridge target 172.18.0.1:514."""
+        os.environ.pop('HAPROXY_SYSLOG_TARGET', None)
+        hm.generate_config()
+        with open(hm.HAPROXY_CONFIG_PATH) as fh:
+            cfg = fh.read()
+        self.assertRegex(cfg, r'\n    log         /log\.sock len 2048 format rfc5424 local2 info\n')
+        self.assertIn('chroot      /var/lib/haproxy', cfg)
+        log_lines = [l for l in cfg.splitlines() if l.strip().startswith('log ')
+                     and 'global' not in l]
+        self.assertEqual(len(log_lines), 1, log_lines)
+        self.assertNotIn('172.18.0.1', log_lines[0])
+
+    def test_log_target_env_override(self):
+        os.environ['HAPROXY_SYSLOG_TARGET'] = '192.0.2.5:514'
+        try:
+            hm.generate_config()
+        finally:
+            os.environ.pop('HAPROXY_SYSLOG_TARGET', None)
+        with open(hm.HAPROXY_CONFIG_PATH) as fh:
+            self.assertIn('log         192.0.2.5:514 len 2048', fh.read())
+
+    def test_lists_written_even_with_splash_off(self):
+        os.environ['HAPROXY_DEFAULT_SPLASH'] = 'off'
+        hm.generate_config()
+        self.assertTrue(os.path.exists(hm.ROUTED_HOSTS_PATH))
+        self.assertTrue(os.path.exists(hm.ROUTED_WILDCARDS_PATH))
+        with open(hm.HAPROXY_CONFIG_PATH) as fh:
+            self.assertNotIn('default-splash.html', fh.read())
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-fixtures/san-certs/dead_old.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC9DCCAdygAwIBAgICEAMwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMZGVh
+ZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
+EwYDVQQDDAxkZWFkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQCumL3T27a3D6fZbIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnO
+abvnQNxEVqyYtTJOn9taitVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BU
+SpW96rK40KVftzzDEUnH+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0
+seOIdyP/n3T2cX1wsCa/F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9b
+PBd2cBaBwTe/3TICu0RnvvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2
+wzG4zlKU31fdLvQOzMGH1W5H91sBAgMBAAGjSDBGMCUGA1UdEQQeMByCDGRlYWQu
+ZXhhbXBsZYIMYnJhaW5qYXIubmV0MB0GA1UdDgQWBBSPDJiv0Z2YpuloexOfvpIQ
+UrWufTANBgkqhkiG9w0BAQsFAAOCAQEAJwBtN3Qw41XvYuSa8Mjn869QOTKEgkaR
+EuNQ8WXVBOTRJdxaLHDunGO4ExjHHswCWzXkefYOoIClAU9qLnk9M667BSNxOcz0
+VcG/DUMd0tO8X4Zm0zOMjt4VC7whgj7MdYtYVg2+Jhb4sMSx6OqmCUAD4X6ULSTn
+lZK2holEC/H3bKt8Q6cecgeubgWojP1bWmWK7jQL5NRIzxQ0KzFxgAf3t2Z05nX8
+loABkBiR7vDthkQseUW6RQMayDyzhVqORLt+gZ9D5T3hT1M7VaDc4Twr7UbWPFcR
+GuQ7RwKplPIIbPe7+c91pm+agdSDjnV5HmKcAMCwf7bWGQ9/Xc0BFg==
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/dead_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCumL3T27a3D6fZ
+bIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnOabvnQNxEVqyYtTJOn9ta
+itVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BUSpW96rK40KVftzzDEUnH
++3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0seOIdyP/n3T2cX1wsCa/
+F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9bPBd2cBaBwTe/3TICu0Rn
+vvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2wzG4zlKU31fdLvQOzMGH
+1W5H91sBAgMBAAECggEADUvYBFLn6gPWSZj55nxOqR+2Lmtwh04omgbu2fpWhPTE
+ujgKd/23srnnMZXnR6qZQQIPGwJiOPnJuRjiu/PMZSj8j54bbwfA6XmbH9SCIzm0
+la50UGBSkw0T50jBI76vICMm6PBkf0rT1oKN9VOd/qlkJoCUiFcGCh177Fbt07oY
+R3iAKu7/+yEANX0iEcxbO330TPW3oI0ah5OryfKVoNvxpF8NUfV9pWY66APkOYMJ
+zsYvIMvzV/KIgfrtyxJjSjgd7Ud8uictYG1BrZBXpIuGxuxyKUJ7OXOvMMVNnEME
+2uzk/aL75UjpurJJU0LCywdUOGwRqS2q4PYQiTeDLQKBgQDjdywH7RODoYeH05Rd
+sa2MkNx0Day1i3A4IBds55mzn8jxpT55HlqvHWhNA30kEw01H8+6sB7iMX04xQtS
+u3N2opR1VlCvMJhGyRUYUtfu4NnAi/HSu7mCOFGAwaLclI7BAI2or2UrclrlUhGb
+CYIBKfEr6Tn6b6AffWaOruCbtQKBgQDEf70R922fnJK+//QuiYoaESwJd3/k10yJ
+nH0ZAgbnFcNLT9YVFxHvlt4RHGIotaBjfN+FKMn045jeekgeVdX+uF8+7FwEazAH
+zOfxswOeKd8EOooqCD7n7cW3Ma6ev8oWmXwJsz2coRW7Z5zCGQWSzCEDB4yFU9Mt
+0z6aN9SJnQKBgQCvBZSIqL+6KgD53zArA7sRDKnEVn1M13TKis7bij02/DyGyuoO
+EfAfDBakI4oXxt0Dwm15LWkmDIRYuS0kRH3si96FhpvgoRiaZsDnjV7/PF3wjP3O
+c4mjDCiSVOw2o8xP71gQspy62GveRc6sJ52zvY0Hh5aH/3oKoNfX8/F5fQKBgALQ
+vbnDLMvrp2KbMjl0eQkUk+fy0V9LAmq6VkSEderbsfejRjL5KKWhKL9gA7wHliu2
+v1EeG36pwCf34o6ktjM3Gw5AvrTFgGOsOOqKoROkD/D0kcQ2aAhUrM+KI7FcmB2o
+x1EwQ0K6QCB2RJW3XBAVYqezdPz4DX+LlksJX781AoGBAISGOer3s4+oRjO/2VuT
+hnMQrET3A0GTol8cgYL2N9wfx7jy5bhSfiJIX/dkh5IrPGYoBdWM87gcUgNQ8AZ7
+il5dVeG80oc7KFU/QltZJu9VS32vwXnNsnrXupQ5Ee0ztTLciTMWU5jxvB0ZJHtK
+StgYepr4IPDUaObqrPx4Jvfm
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/expired_old.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDGjCCAgKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwGDEWMBQGA1UEAwwNYnJh
+aW4tamFyLmNvbTAeFw0yMDAxMDEwMDAwMDBaFw0yMDA0MDEwMDAwMDBaMBgxFjAU
+BgNVBAMMDWJyYWluLWphci5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQC9SJUMsK5HhVEATu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9s
+i5KzgYp0K2z899wQAKuLzZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ
+7jax2QWFMQVhyy6M5QTCHUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyN
+N8eolukhvFxQNU3E7oo1jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSy
+UEc+Gje6dJLZ9CdJtSQSNGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8
+FHC7atJOe5EoaekaAXw5Tq+yB+ODAgMBAAGjbjBsMEsGA1UdEQREMEKCDWJyYWlu
+LWphci5jb22CEXd3dy5icmFpbi1qYXIuY29tggxicmFpbmphci5uZXSCEHd3dy5i
+cmFpbmphci5uZXQwHQYDVR0OBBYEFAWqbBh2PDUvqQzZoyh960cpuhcOMA0GCSqG
+SIb3DQEBCwUAA4IBAQCaovtrXl56YcFlXWMrW/XZBe2/fXI14860qyiR5rb07lS6
+CbKOsnRyQdxZ3cdfRwTN1UMmz5IPOZ6fhdo3IPaS2Wae1OdeupRaRWr1SAJtpVZ3
+WWugDaumsWV5eWVxVTf93yDEkMERWtTGMviCKBeeHjzHIFiD/EjZznE87V5DYwuf
+guOd1F4wXCMRFu8q7sBdPXjQ8rZMlQT+dQXzMST9Ee5PXiInmCAA5nT8vMInAwd2
+rP6FxuHJB75ekYzs57CJ82BKKboOCESjUbm54wmVV5pBL305/EjxGf+VovNlRktg
+t1znolntIIofE25UCbVNPdrtuDkZgKT3Rv8tCdTP
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/expired_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC9SJUMsK5HhVEA
+Tu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9si5KzgYp0K2z899wQAKuL
+zZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ7jax2QWFMQVhyy6M5QTC
+HUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyNN8eolukhvFxQNU3E7oo1
+jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSyUEc+Gje6dJLZ9CdJtSQS
+NGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8FHC7atJOe5EoaekaAXw5
+Tq+yB+ODAgMBAAECggEABAeOCL71eYH4jKRMOBbXu98483u1CxnTEa8klnw9ZVES
+JctJjY96B/EMwv+g/V+gxhgAkiK9VWfg6bEplOVDWgR9fRBoP57mVkOv3QPn5fcR
+xlt9LQQZ+pULQRZoUAFEkVEkV1sfIBJmDLMufOnbAHCKa9+hVlrlCqu0lJgM8put
+2bkpSAqwroHVfmD0SmKFAZggBhFwXFJzwmUf3WL+N25f4RZpJsX17FOJ216mlhQ5
+edje6dcDpmko2VQtGJXYr8Ge02ZYeRyKGcTsgUZlarB4xUZq1pcnUUSy0Ez7LVbk
+tKjPcUpZLlnAA/a2pDgHmwN1+fv/X+Wwv/vZaQMBTQKBgQDuZFyfDRSHCssHtQ4+
+Ezxg4WEjwXkq58WxL0zTtLTcquZfQZ1VXgLwAylVwver0I4/UgCfW/AYY+2eLrba
+FhuYOosd/3rsUQqf9y+TkSkhe1uVS2/1zjzC1G/5coleLddjzd+YBpEtVxfxDRbA
+rVePY3GxkJiTrjAg5OETk7GYpwKBgQDLQ6bPhaocJPQmDi3hKyW8zPJ8wAJNyI0c
+RWt9gjeidjmMvwCKtJbBDcV5V0oQ5SB2M5E49mT58B62totAD7FS18c4a4133haV
+a1pLBX0U524eQgDzMJ6YA6N9bHdXOH4FZd+Pw90bHetbT1HcKpOdXv4SF/XITju5
+Ro5xiymdxQKBgBB4fq2+iZyK8lwl9kXzrkhDSiLeSzDbspz7V6OI6LYWWLRcJDKx
+vCI5FbIHNnV4g1t8nu5ju8LzpIc5UwjeXim2YcZA0I1jTAihW3gmn2Dg1QMLo0Co
+bypG4zWo5DTNlgQfRUOLsJwhWFYfQWMuls4hYMmT4/bQIiMjrSldGvghAoGALpaB
+7DyZobvCjdpgH6TREHX0+izUGlnmJd2vZmPrNnxccmrwx9neV789TLN6EWdCynTF
+S0YGHNcxYvVx8yvlm7RDQCY+lYO8r2CcoGdQpPLqZVsBdrvlDbh/x2aN79rN35Ta
+6cudqPJxmEsU2x6TGNXF7AW72r47qBqI2XNpBXUCgYAFQdHRRxa2E5rgV0WkL7PF
+vXok3isAzpnKAsb5oxPZbhIdLo5EPUGCslsTE1Na8NBdUSoyjhgkMELEDx/ceGL7
+IIlNrc/4j71+IahljzaX/D5b5qaW+RnoTVAlbMRR7L/ecbkoovhQR9d37QdoiD/t
+B9si92+XB4HYDAjXFxTK3w==
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/new_bundle.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC+DCCAeCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMYnJh
+aW5qYXIubmV0MCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
+EwYDVQQDDAxicmFpbmphci5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQCyhL5YcFhPEekc/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdb
+l4twhlkcyvyaEbcHIgLEgz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgU
+QcAAvdAyP77jDF6lX11oCjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zz
+XRVp4l2WyqEUJmKxcpppAnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU
+0v78QfI+ueCw261UWZ+iXwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQ
+dFOfKjH0QGInyByji9eAaXlmvCKtAgMBAAGjTDBKMCkGA1UdEQQiMCCCDGJyYWlu
+amFyLm5ldIIQd3d3LmJyYWluamFyLm5ldDAdBgNVHQ4EFgQUdDxfZKX96j61mW7O
+NiQpN4T+sZ4wDQYJKoZIhvcNAQELBQADggEBAJE1lP62caByZT7GkKtJLfOj84Ih
+MeP751w2gdU8ZfiO8GxXXiR3ZjYzbJgsEi9LiNT9jqHeBf3pv/mJLjPVZ+U557ws
+Gq3V5A3uKlRKGcejXlHaWnExhWqLyIaLTeLKstbJyiB/ja4yeTL2TYcOvvlu6weh
+ERqwx0p6r7LSQ3nRfTdjEhNq3FVUmqQ8neBP/vXLp2hKr2ACZndGyQOiv0oZWzud
+part6RMd8gRNq1Tctrl0YqRYoLRIddBrnTtUtac4FLqxLSBNc03/I9ZgF56Yf+Sl
+M1ikmyDJ2ZYJFU6v42Og4/8o3fpq0ttP6pKPi8fInGaFQ+KXO1NWFf1ogD0=
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/new_bundle.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCyhL5YcFhPEekc
+/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdbl4twhlkcyvyaEbcHIgLE
+gz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgUQcAAvdAyP77jDF6lX11o
+CjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zzXRVp4l2WyqEUJmKxcppp
+AnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU0v78QfI+ueCw261UWZ+i
+XwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQdFOfKjH0QGInyByji9eA
+aXlmvCKtAgMBAAECggEAEQi8lvjFXGojmco3H7PVD7SZftLOFidVh9Fo8/JAe+2Z
+LQM/hIjZ1i7sNverbQXMzAQMfXESS2Jx7Clv0mZ0aRgE0hq7wXrabyHfFUPOtqDa
+6cPHsz8YvwAOmcQ6W3fBPYI6pzSfdV03rv1GGeCC7ETI3mIupwgUbANDuiDcVY1q
++VuK/hDzPJ5KslDHUNR42MgWacIftq99fV9jN0zQQw1EcyO8QwpXzTSEbdyiAkT+
+n/tR9He2MYhgRhjyWPMaxmY6XQpv1h5ASq1yT1kI2oNTaRT3xvd40sLkbfwpZnxi
+HnqzCkPptEpWeCLrme+r71+Hxu/+Bph7tPtcuK7jgQKBgQD8XCoPRiWbgCl6kM2G
+E3GeM6fftTieTIs7sFVte7wutIKlsec4vizc94ipGaoT7yGEG0T8g9+5hAMbebc2
+zzhxvJMB2s18Z/He+Ujv1L1AhBBNVY3zCAxfdq8/n5MGXppEdVRo9QOQndoBfuAV
+Qw2Vp4bcy728n4T7sBugW0Ad7QKBgQC1F+utuiF8XI2yx0mX0+Fbty6o+odyM0eb
+G/QQDlcGpY0K7V3WAFMESNfytig8OhidzRfWn8lNoqhd80SJfCuiYjiGJ8tjBzcp
++Vqrd/KAxJySJ3NJ2MwyAp7W3l3HAEZLGVYVCankoa4O3qR8YqaIZwc2nk5+8KnG
+nBYFlSF/wQKBgQCqaRYh0E2O7wGTeXq/ohukJ9+hqf88rhWLTeyyDM8rfsuGVlb8
+MzeaUzEed20pHt60FbIFDowh+yoaAJwez9mevCdNJlKCQrLKze1Zvn+5tDRJ9I7x
+YSrxv/Rhc0FWGMX1moFQirtIib9wKRvqX6gpucHREWfjmpizWN2e8F+glQKBgAOp
+/jy/nVnUzZEmcxi9Urv2Ut1cZE0tOG+s+x0wDRa3Vsk30EQ6MXPOvX/Nq7a8Pi9D
+LQ5MoVgOyHVOLL7TYlagwiAEwhq41KbmwqzhemickcZrQevmlQsV/ny+dwtFuCZg
+1tsKHQPhOZJXcuCJZnEwFHdIcLIk0GqSUee/ArSBAoGALOD+Opk+0j4ObNcqtrpP
+UcEYqhOuAscbxLoUv53wnOvc7XzMG8JCK1y5LthB2Xanr1JSlheMRrvGxC+WlDmc
+5v0Nz71cq5QQ1p8AXnosLRjuy/+edMvGIxxwFMEJBO2ra2Jizpdwd34bjpQ42+CQ
+6pVwFQX0oet8SI5xWUWx7NU=
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/shared_old.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDGjCCAgKgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwHTEbMBkGA1UEAwwSc2hh
+cmVkLW9sZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAw
+WjAdMRswGQYDVQQDDBJzaGFyZWQtb2xkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEB
+AQUAA4IBDwAwggEKAoIBAQCBg7iYuvjhIG7nlFeja5Lyv2t2fdtNvxzUib7ygejC
++WQvNdH4k12q8vCXNYuadbGAceSejG4bazpjo5/ym2KFZPwiSZoqKUugi9lCoKuL
+2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMHYFwURjARRAku3a/GJUHQR0C572mW
+mt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64fGeYSOaYOHEOp/VqIRlzsy7eGQh3
+xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2JbqFxvijCLnBz5WsrS/S/VWOELNEfGr
+84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLfWuSgp5hRAgMBAAGjYjBgMD8GA1Ud
+EQQ4MDaCEnNoYXJlZC1vbGQuZXhhbXBsZYIMYnJhaW5qYXIubmV0ghJvdGhlci1z
+aXRlLmV4YW1wbGUwHQYDVR0OBBYEFP+AdZ9MkBmFjAQTgLnWZxbJtnvkMA0GCSqG
+SIb3DQEBCwUAA4IBAQA32Xl/1IrOyESd2UQcWp/fuHvkivLgamJrheKvKP0Y0vMV
+atRu0D0GhqMOr79OT60H/H1FFW7bWQ+UPcZpnuXec40/PcrHxwaSoiLoIUVWhB4t
+BOhtTDReX29gEjnB7OqBCJu4ub3HnpFTflUZd7n5A4EcetBsowc5EZ9UAtfvkgGD
+nK9ONvNVGurk/V8xgtMJ877ZK1fn+UtgvIWp/F1Gc/taLhe+oCZOPkOqvvqUnS8q
+QyIhqmMWRUkBP+jwRA4TP4pj7OH38VY/Ae408Xe4Y97GF7NbOjAL0bkX/bHkLnSG
+WzoGO0TK7pCemeFQ+Ghic8Yl0eDynoP7rZIhCKS0
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/shared_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCBg7iYuvjhIG7n
+lFeja5Lyv2t2fdtNvxzUib7ygejC+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpj
+o5/ym2KFZPwiSZoqKUugi9lCoKuL2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMH
+YFwURjARRAku3a/GJUHQR0C572mWmt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64
+fGeYSOaYOHEOp/VqIRlzsy7eGQh3xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2Jbq
+FxvijCLnBz5WsrS/S/VWOELNEfGr84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLf
+WuSgp5hRAgMBAAECggEACB1tvTwciKvp5wliGF45feDv5c5q+vXE462h0OYXjyfl
+1sNuMuoLjhYyFvdcQO2rlTBjB8Ald1hXoeidsVYlF9XvqmJRjjS6MAazMpEzTnba
+66V6U1j5gOQ6pi3SjfBJGFYo8L+cy/NqUIBWz0+6hp/BsjW3uW2aMGer2Y/zs2Oi
+s0kUNqqUWQpGpEDxbDnUAzto4OOcTOO4A5NB5Oe2ifzFiXCWtTNqFIC0ae9tiqGy
+m0sggF2hzmlH1WC54rbv7xujYT6tOAEi1OKp/Cpj8XVLbjaqju5avRl1t2nDRQri
+9vnOI9P4bnJKbIYzwJVyy8B1M5iK+W2K7wUQ4ZMetQKBgQC2zwtywQ4jUduLWf4f
+6HLUIvmcHdJI0YBHS1+4fKc7tsakRo/0FHxNdUlBhqkYTyp57h6/R1T2gG70Kla1
+MarGIgyPIlUXsRojlUcNyKXqSVs+kejftIClb+VDXWrpKtGtyxuESsMjaJBsBBAN
+hg3yUx2NL2ct4nnlpl1iq3pH9wKBgQC1XkrjntuzjVYlFeg2ih5va3PUD2qmH9Ej
+leNXTGhzZL1eQWz4GaRCgzJH8q41pXBxcXQ8hMXZCWdW5GHBYfVqjTRKMUPdI6FI
+ypwMbbTBfQfw1kz5VHfLH3romxikNQXeorZkIZqv7/aLPsxLY78XdW8nSDIsT9ZP
+mh7It+Tf9wKBgHT7VSYf6D4t2cH42qfshd6+auC+LJywg9gboatbCWEq+z6QcPoO
+SeT/Utt9oqONtW7FNCPPL812LopMtaPJnrCfhU3q+GzqxgFTO17UsWeVfbAyisTs
+qw4y5pEPLKT846TFPlyPwwPbm9+jUep1iGMzUMvSM8KnFL4ikcvcU6nNAoGBAJQI
+MVHV0HadzokYd19EnpDMgE6+goqy/21vIo6XY82DlPjAp+NccnCDSt/kmQJaVQuS
+0Y+vwxhhj4IhiBRiEwBz3hEMYtq1ASg7juCqDlW66mdNUqhSpDAbMqN2K8ejK3ZL
+EllqACrHzlBa4j5KXKcWB7JlP+ZhB4gTQDsNtXaDAoGBAJCktIdyM3b6UtM5OofB
+lzn6/2uT5u46ECBRybk/J+Y2gfcS2cFzkH179SUB4WgfR0h21FlLmjns5ZvHVk0p
+/U4aY3/J42JU6lCYrsyV5BBH+O0stSmrUG/4JdKRe6Mq/pB321RsWD06LVJrBDpi
+6NUPW03tNLc+Qe+cnQu0c7y+
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/unrelated.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC8zCCAdugAwIBAgICEAQwDQYJKoZIhvcNAQELBQAwHDEaMBgGA1UEAwwRdW5y
+ZWxhdGVkLmV4YW1wbGUwHhcNMjAwMTAxMDAwMDAwWhcNMjAwNDAxMDAwMDAwWjAc
+MRowGAYDVQQDDBF1bnJlbGF0ZWQuZXhhbXBsZTCCASIwDQYJKoZIhvcNAQEBBQAD
+ggEPADCCAQoCggEBALEpfqdRPBSrbzixAxD2WhsO1AOTUd/Gguf3zjDZ3TUmWJKb
+OhlFthtU6PJVhMdAqOk+hiz9Coo/UngWjykyNMOj1YIpq3M4/P8wBTNxkJd/z2x0
+iAQBKvWlB6hxf9c1kenu/ea1iMIThA/uudtX3g8k39Asj/37+o070cdBJExWGnbG
+lz103FCO6kBzlJPGIVQwzKH6OKthTzi7Uxy5Fxc/ymS9gc43F0tyY6892Wyl8Y62
+a3r1PpZoWR0GDFxPIyCS7NP44T15DSL/2RBi4YnWXOC8QMvPnHbSKjoQ6GLQtGkO
+1NO4Uy3XbwDFanYXv6Wxv0oUmZMuY7CRTOMdxCMCAwEAAaM/MD0wHAYDVR0RBBUw
+E4IRdW5yZWxhdGVkLmV4YW1wbGUwHQYDVR0OBBYEFBh0L/5U5f94TylibBCgS9pD
+/VXkMA0GCSqGSIb3DQEBCwUAA4IBAQB5MLqNjTqIdyztjJsKPdYUkA27NpfdqqY6
+Yprv5N7U0slsGUqFIDrSuJ1mMMwTpt02IAFyCOWMUtze+/0PqM9k9TqxJH2cTODp
+0ET34Qh8NTo0B534EQUWhny+EzgNAzWK02wbpgm88+Mp/t//sHzhWzOm7uk1dlkq
++5ty/8OveWwAtqAWlDtcsRhkEjzynZKUCAF1extwUSuapFSm20mJ4ivt7Ff5MkTI
+fq8P+3XAxWTaBW8DzBIzYd1mwFHK2pcklLMRRTW3hukPxEcTivBgJdwsHMskM3UO
+/ejy5EHem5ohCMbwMybvUqLS4xdoqXV36Sj2I1zv0IjgO0wgdK1r
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/unrelated.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCxKX6nUTwUq284
+sQMQ9lobDtQDk1HfxoLn984w2d01JliSmzoZRbYbVOjyVYTHQKjpPoYs/QqKP1J4
+Fo8pMjTDo9WCKatzOPz/MAUzcZCXf89sdIgEASr1pQeocX/XNZHp7v3mtYjCE4QP
+7rnbV94PJN/QLI/9+/qNO9HHQSRMVhp2xpc9dNxQjupAc5STxiFUMMyh+jirYU84
+u1McuRcXP8pkvYHONxdLcmOvPdlspfGOtmt69T6WaFkdBgxcTyMgkuzT+OE9eQ0i
+/9kQYuGJ1lzgvEDLz5x20io6EOhi0LRpDtTTuFMt128AxWp2F7+lsb9KFJmTLmOw
+kUzjHcQjAgMBAAECggEAJjBKo+iH6fC2beMCRTLFY5XDr+2etX0GFNi+8prT6zQT
+3a5mTLJROf0Ipcw3bfgLMRQQOjxlfD2AGnDKdVfkNdPoH0G6OoErUBra1CcDwxSC
+8t7/vtU9nGvxhabJputeQ64HvyacjzFDw2dzWsvLCzqooCgq9dXOQYmGg8wP9gAx
+pxfKnd84h41C3pXERdufi6nJHPjZPRW+dKsQ1NMLmtgO5YQnTt3LfVOkZaRa7Izq
+ZFWZCBfdsNHHuXCYj3FtqYrqSxELuZAxJ62hxpizrfQdKixguGTObHhC4muIo9fR
+eIMCvrAzrs4Fs8nM/K/IPyDU9wPwINK6Z+sxJkhM+QKBgQDaL3Yt3T/LE6bTYyb0
+EjVcXoybnuAjFQAenCTIsBRJv21Yv2ddqjq6rOTD64hFuFWPMEOi1TJxIPsFFDMw
+NeAwF41fiSrtU8awXs0hTKRQeHjElSBckumOl1DrHu2oKuwWXLniViD62vMsWmpZ
+89KT2rgFQuMPvsD3pps8l6W2HwKBgQDP3eTwdVJTdRYlZonqd0+LHG7h715TT1OE
+QJOdN9vK5g/9GOCOGHoUxB3IxJVmvz57PrfWaD2rSZ6JMs4TcAY5a0PbJEpgqoJY
+kIm/EDnSrFxu2L8DNsAzuThfzXrwWc7PzajYJctYtEfvxCyfo/GdZahg2xaEU0Le
+dp+7fxtJfQKBgGVdLWtLYarH+3Yzl1JG71F1f5+A5SKHTCHsas5tt083xuY3Lh9+
+yYAWeMVotEunwuAeDR4mLCQQrW4Dn3BjZz8QghaDVFUj2afwsx4oTCIyhfDWU02O
+10DunaJOIR1Cq6IL/DPhzbqdY+IUM1qNAyS7tUiEdZbICUBEv9Qbo9oDAoGAORnY
+cSGZaumoGsAPNMRvqmQ2JYSO4c3RwGvI4RBeJV+LlthFn2Mj7h5tnCA8dkwkytBQ
+yeo6vpnrnf7B018bppxLuKBAekt8oP2biUvYxoGopp6N4kEhDVRdIxwcIHhbMoRp
+VG8VbBmiBUBbjcMIdVdeqLSme9UvclMdz/C7uzUCgYEAzD2dvoJSu9zfo3fACw1X
+te53Rvr6Kq6DTw0SS4UrAnxlhX9TwD9gymIrR9rxnC+UPwfMwWMdpYqY+Da4sF1G
+uwd+nyZ4iFNq+lbLii9rmJ58h03+uo4lUXWTrAik6qOF1ztIfYKWFrOyBQmxjWwo
+HnUinudalENFJYZRwSUAtXo=
+-----END PRIVATE KEY-----

scripts/test-gunicorn-socket-timeouts.py

@@ -0,0 +1,631 @@
+#!/usr/bin/env python3
+"""Regression tests: a silent peer must not be able to wedge the API's threads.
+
+Why this file exists
+--------------------
+On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool threads
+blocked in the kernel in tcp_recvmsg -> sk_wait_data, /health hung 40s+, and the
+container stayed "healthy". The app makes no in-process outbound socket reads;
+the blocked reads were gunicorn gthread reading the CLIENT socket, which it
+makes fully blocking with no timeout (TConn.init -> setblocking(True)).
+
+These tests run REAL gunicorn processes (no mocks) against tiny WSGI apps and
+pin:
+
+  * the failure mode itself, on the stock gthread worker (control);
+  * that haproxy_gunicorn_worker.TimeoutThreadWorker frees the thread when a
+    client stalls before / during the request line, mid-body, or on a reused
+    keep-alive connection;
+  * that a legitimately SLOW request (handling time > client timeout) is not
+    cut off -- the timeout bounds socket I/O, not request duration;
+  * the watchdog: a worker whose every thread is stuck is replaced;
+  * review round 1, at production shape (2 workers x 4 threads): 2050
+    connections that never send a byte are reaped (F1, stock gthread wedges
+    permanently), and slowloris drip connections are cut by the total header
+    deadline while /health stays responsive (F2);
+  * the timeout is logged as one line, not a traceback per scanner;
+  * haproxy_manager's socket.setdefaulttimeout: an in-process client call to a
+    stalled upstream (accepts, never replies) times out instead of hanging;
+  * why that default alone is not enough (setblocking(True) clears it).
+
+Running
+-------
+    python3 scripts/test-gunicorn-socket-timeouts.py
+
+stdlib unittest, like the other scripts/test-*.py; needs only the app's own
+requirements (gunicorn, Flask...). Runs in the Docker build gate. ~70s.
+"""
+
+import os
+import socket
+import subprocess
+import sys
+import tempfile
+import textwrap
+import threading
+import time
+import unittest
+import urllib.request
+
+REPO_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')))
+
+try:
+    import gunicorn  # noqa: F401
+except ImportError:
+    sys.stderr.write('SKIP: gunicorn not installed (pip install -r requirements.txt)\n')
+    raise SystemExit(77)
+
+WSGI_APP = textwrap.dedent('''\
+    import os, time
+
+    def app(environ, start_response):
+        path = environ.get('PATH_INFO', '/')
+        if path == '/sleep':
+            time.sleep(float(environ.get('QUERY_STRING') or 3))
+        elif path == '/hang':
+            time.sleep(3600)
+        elif path == '/body':
+            n = int(environ.get('CONTENT_LENGTH') or 0)
+            environ['wsgi.input'].read(n)
+        body = ('pid=%d' % os.getpid()).encode()
+        start_response('200 OK', [('Content-Type', 'text/plain'),
+                                  ('Content-Length', str(len(body)))])
+        return [body]
+''')
+
+
+def free_port():
+    s = socket.socket()
+    s.bind(('127.0.0.1', 0))
+    port = s.getsockname()[1]
+    s.close()
+    return port
+
+
+def http_get(port, path='/', timeout=5):
+    with urllib.request.urlopen('http://127.0.0.1:%d%s' % (port, path),
+                                timeout=timeout) as r:
+        return r.read().decode()
+
+
+class Gunicorn:
+    """A real gunicorn master on a free port, with its output captured."""
+
+    def __init__(self, worker_class, threads=1, env=None, workers=1, extra_ports=()):
+        self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
+        with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
+            fh.write(WSGI_APP)
+        self.port = free_port()
+        self.log_path = os.path.join(self.tmp, 'gunicorn.log')
+        e = dict(os.environ)
+        e.update(env or {})
+        self.log = open(self.log_path, 'w')
+        self.proc = subprocess.Popen(
+            [sys.executable, '-m', 'gunicorn',
+             '--bind', '127.0.0.1:%d' % self.port,
+             *sum((['--bind', '127.0.0.1:%d' % p] for p in extra_ports), []),
+             '--workers', str(workers), '--threads', str(threads),
+             '--worker-class', worker_class,
+             '--timeout', '120', '--graceful-timeout', '2',
+             '--log-level', 'info', '--error-logfile', '-',
+             '--pythonpath', '%s,%s' % (REPO_DIR, self.tmp),
+             'tapp:app'],
+            stdout=self.log, stderr=subprocess.STDOUT, env=e)
+        deadline = time.time() + 15
+        while time.time() < deadline:
+            try:
+                http_get(self.port, '/', timeout=1)
+                return
+            except Exception:
+                time.sleep(0.2)
+        self.stop()
+        raise RuntimeError('gunicorn did not come up:\n' + self.output())
+
+    def output(self):
+        self.log.flush()
+        with open(self.log_path) as fh:
+            return fh.read()
+
+    def stop(self):
+        if self.proc.poll() is None:
+            self.proc.terminate()
+            try:
+                self.proc.wait(5)
+            except subprocess.TimeoutExpired:
+                self.proc.kill()
+                self.proc.wait()
+        self.log.close()
+
+
+def stall_client(port, payload=b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03'):
+    """Connect and send a few bytes that are not a complete HTTP request (by
+    default the start of a TLS ClientHello -- what an HTTPS scanner sends to
+    this plain-HTTP port), then go silent and keep the connection open."""
+    s = socket.create_connection(('127.0.0.1', port))
+    if payload:
+        s.sendall(payload)
+    return s
+
+
+def wait_closed_by_server(sock, within):
+    """True if the server closed (EOF/RST) the connection within `within` s."""
+    sock.settimeout(within)
+    try:
+        while True:
+            data = sock.recv(4096)
+            if not data:
+                return True
+    except (ConnectionResetError, BrokenPipeError):
+        return True
+    except socket.timeout:
+        return False
+
+
+STOCK = 'gthread'
+FIXED = 'haproxy_gunicorn_worker.TimeoutThreadWorker'
+FAST = {'HAPROXY_MGR_CLIENT_TIMEOUT': '1'}
+
+
+class StockGthreadControl(unittest.TestCase):
+    """Proves the failure mode is real, so the fixed-worker tests mean something."""
+
+    def test_stock_gthread_is_wedged_by_one_silent_client(self):
+        g = Gunicorn(STOCK, threads=1)
+        self.addCleanup(g.stop)
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        time.sleep(0.3)
+        with self.assertRaises(Exception, msg='stock gthread should be wedged'):
+            http_get(g.port, '/', timeout=3)
+
+
+class TimeoutWorkerTests(unittest.TestCase):
+
+    def start(self, threads=1, env=None):
+        g = Gunicorn(FIXED, threads=threads, env=env or FAST)
+        self.addCleanup(g.stop)
+        return g
+
+    def test_silent_client_is_dropped_and_thread_freed(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        t0 = time.time()
+        # Only one thread: this request can only be served once the stalled
+        # connection has released it.
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertLess(time.time() - t0, 4)
+        self.assertTrue(wait_closed_by_server(s, 3),
+                        'server should have closed the stalled connection')
+
+    def test_connect_and_send_nothing_does_not_hold_a_thread(self):
+        # Never readable -> never handed to a pool thread (main loop only).
+        g = self.start()
+        s = stall_client(g.port, payload=b'')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
+
+    def test_half_written_headers(self):
+        g = self.start()
+        s = stall_client(g.port, payload=b'GET /health HTTP/1.1\r\nHost: x\r\n')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_body_shorter_than_content_length(self):
+        g = self.start()
+        s = stall_client(g.port, payload=(
+            b'POST /body HTTP/1.1\r\nHost: x\r\nContent-Length: 100\r\n\r\n0123456789'))
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+
+    def test_timeout_reapplied_on_keepalive_reuse(self):
+        g = self.start()
+        s = socket.create_connection(('127.0.0.1', g.port))
+        self.addCleanup(s.close)
+        s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
+        s.settimeout(3)
+        self.assertIn(b'200 OK', s.recv(4096))
+        # Same connection, now stall mid-request. gthread calls conn.init()
+        # (setblocking(True)) again for the reused connection.
+        s.sendall(b'GET / HTT')
+        time.sleep(0.3)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_slow_legit_request_is_not_cut_off(self):
+        # Handling takes 2.5s, client timeout is 1s: must still succeed.
+        g = self.start()
+        self.assertTrue(http_get(g.port, '/sleep?2.5', timeout=10).startswith('pid='))
+
+    def test_timeout_logged_as_one_line_not_traceback(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        http_get(g.port, '/', timeout=5)
+        time.sleep(0.3)
+        out = g.output()
+        self.assertIn('Closed client connection idle', out)
+        self.assertNotIn('Traceback', out)
+
+    def test_watchdog_replaces_fully_wedged_worker(self):
+        g = self.start(env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                            'HAPROXY_MGR_WEDGE_SECONDS': '2',
+                            'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.5'})
+        pid_before = http_get(g.port, '/')
+        # Wedge the only thread inside the app (not a socket read).
+        hung = threading.Thread(target=lambda: self._swallow(g.port), daemon=True)
+        hung.start()
+        time.sleep(0.5)
+        deadline = time.time() + 12
+        pid_after = None
+        while time.time() < deadline:
+            try:
+                pid_after = http_get(g.port, '/', timeout=2)
+                if pid_after != pid_before:
+                    break
+            except Exception:
+                pass
+            time.sleep(0.3)
+        self.assertIsNotNone(pid_after)
+        self.assertNotEqual(pid_before, pid_after, 'worker was not replaced')
+        out = g.output()
+        self.assertIn('WEDGED WORKER', out)
+        self.assertIn('tapp.py', out)  # the stuck thread's stack was dumped
+        self.assertIsNone(g.proc.poll(), 'master must keep running')
+
+    def test_watchdog_leaves_partially_busy_pool_alone(self):
+        # 2 threads, one hung forever: the pool still serves, so no restart.
+        g = self.start(threads=2, env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                                       'HAPROXY_MGR_WEDGE_SECONDS': '1',
+                                       'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.3'})
+        pid_before = http_get(g.port, '/')
+        threading.Thread(target=lambda: self._swallow(g.port), daemon=True).start()
+        time.sleep(3)
+        self.assertEqual(pid_before, http_get(g.port, '/', timeout=3))
+        self.assertNotIn('WEDGED WORKER', g.output())
+
+    @staticmethod
+    def _swallow(port):
+        try:
+            http_get(port, '/hang', timeout=60)
+        except Exception:
+            pass
+
+
+def raise_nofile(n):
+    try:
+        import resource
+        soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
+        if soft < n:
+            resource.setrlimit(resource.RLIMIT_NOFILE, (min(n, hard), hard))
+        return resource.getrlimit(resource.RLIMIT_NOFILE)[0]
+    except (ImportError, ValueError, OSError):
+        return None
+
+
+class ProductionShapeExhaustion(unittest.TestCase):
+    """Review round 1 (F1/F2), at the production shape: 2 workers x 4 threads,
+    default worker_connections=1000."""
+
+    def start(self, env):
+        g = Gunicorn(FIXED, threads=4, workers=2, env=env)
+        self.addCleanup(g.stop)
+        return g
+
+    def test_f1_2050_idle_conns_reaped_and_health_recovers(self):
+        # Stock gthread: 2x1000 never-readable conns fill every slot, the run
+        # loop stops polling, and the worker stays dead even after the
+        # attacker closes everything. Here they must be reaped after
+        # FIRST_BYTE_TIMEOUT -- even while the attacker still holds them open.
+        if (raise_nofile(4500) or 0) < 4300:
+            self.skipTest('RLIMIT_NOFILE too low for 2050 client sockets')
+        g = self.start({'HAPROXY_MGR_FIRST_BYTE_TIMEOUT': '3'})
+        socks = []
+        self.addCleanup(lambda: [x.close() for x in socks])
+        for _ in range(2050):
+            s = socket.socket()
+            s.settimeout(2)
+            s.connect(('127.0.0.1', g.port))
+            socks.append(s)
+        # Attacker keeps every socket open. Within FIRST_BYTE_TIMEOUT plus the
+        # time to drain the backlog, the API must answer again.
+        deadline = time.time() + 20
+        ok = None
+        while time.time() < deadline:
+            try:
+                ok = http_get(g.port, '/', timeout=3)
+                break
+            except Exception:
+                time.sleep(0.5)
+        self.assertIsNotNone(ok, '/health never recovered while idle conns were held')
+        held = time.time()
+        # The reaped sockets were closed by the server.
+        closed = sum(1 for x in socks[:1500] if wait_closed_by_server(x, 0.01))
+        self.assertGreater(closed, 1000, 'server did not close idle conns (%d)' % closed)
+        for x in socks:
+            x.close()
+        socks.clear()
+        time.sleep(1)
+        for _ in range(5):
+            self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
+        self.assertLess(held, time.time())
+
+    def test_f2_drip_connections_cut_and_health_responsive_throughout(self):
+        # 8 connections (= every thread at production shape) each drip one
+        # header byte per second -- far under the 20s per-recv timeout, so only
+        # the total HEADER_DEADLINE can cut them.
+        g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '3'})   # CLIENT_TIMEOUT stays 20
+        stop = threading.Event()
+        cut_after = []
+
+        def drip():
+            s = socket.create_connection(('127.0.0.1', g.port))
+            t0 = time.time()
+            req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 1000
+            try:
+                for i in range(len(req)):
+                    if stop.is_set():
+                        return
+                    s.send(req[i:i + 1])
+                    time.sleep(1.0)
+                    s.settimeout(0.01)
+                    try:
+                        if s.recv(1) == b'':
+                            raise ConnectionResetError
+                    except socket.timeout:
+                        pass
+            except OSError:
+                cut_after.append(time.time() - t0)
+            finally:
+                s.close()
+
+        drippers = [threading.Thread(target=drip, daemon=True) for _ in range(8)]
+        for t in drippers:
+            t.start()
+        self.addCleanup(stop.set)
+        time.sleep(0.5)
+        worst = 0.0
+        t_end = time.time() + 12
+        while time.time() < t_end:
+            t0 = time.time()
+            self.assertTrue(http_get(g.port, '/', timeout=6).startswith('pid='))
+            worst = max(worst, time.time() - t0)
+            time.sleep(0.5)
+        stop.set()
+        self.assertEqual(len(cut_after), 8, 'not every drip connection was cut: %r' % cut_after)
+        # The deadline is armed when a THREAD picks the connection up, not at
+        # accept: a request queued behind busy threads (e.g. legit certbot
+        # runs) must not be cut for waiting. So a drip conn that was queued
+        # (the 2 workers rarely split 8 conns 4/4) is cut at up to about
+        # 2 x deadline; one that got a thread at once within ~deadline + 1s.
+        self.assertLessEqual(min(cut_after), 3 + 2, cut_after)
+        self.assertTrue(all(c < 2 * 3 + 2 for c in cut_after), cut_after)
+        self.assertLess(worst, 5.0)
+        self.assertIn('HAPROXY_MGR_HEADER_DEADLINE', g.output())
+
+    def test_f2_slow_but_complete_headers_within_deadline_are_served(self):
+        g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '4'})
+        s = socket.create_connection(('127.0.0.1', g.port))
+        self.addCleanup(s.close)
+        for part in (b'GET / HTTP/1.1\r\n', b'Host: x\r\n', b'\r\n'):
+            s.sendall(part)
+            time.sleep(1)
+        s.settimeout(5)
+        self.assertIn(b'200 OK', s.recv(4096))
+
+
+def refused(port, wait=2.0):
+    """True if the server closes a fresh connection without answering a
+    complete request (peer filter)."""
+    s = socket.create_connection(('127.0.0.1', port), timeout=wait)
+    try:
+        try:
+            s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
+        except OSError:
+            return True
+        s.settimeout(wait)
+        try:
+            return s.recv(4096) == b''
+        except (ConnectionResetError, BrokenPipeError):
+            return True
+        except socket.timeout:
+            return False
+    finally:
+        s.close()
+
+
+class PeerFilter(unittest.TestCase):
+    """Re-review R1: tenant containers share client-net with haproxy-manager,
+    so the API port must refuse peers that are not loopback / the host path."""
+
+    @classmethod
+    def setUpClass(cls):
+        sys.path.insert(0, REPO_DIR)
+        import haproxy_gunicorn_worker as w
+        cls.w = w
+
+    def test_allowlist_parsing_and_matching(self):
+        nets, problems = self.w.parse_allowed_peers('127.0.0.1,::1,gateway', gateways=['172.18.0.1'])
+        self.assertEqual(problems, [])
+        ok = self.w.peer_allowed
+        self.assertTrue(ok('127.0.0.1', nets))
+        self.assertTrue(ok('::1', nets))
+        self.assertTrue(ok('::ffff:127.0.0.1', nets))
+        self.assertTrue(ok('172.18.0.1', nets))           # docker-proxy / host
+        self.assertFalse(ok('172.18.0.7', nets))          # a tenant container
+        self.assertFalse(ok('127.0.0.2', nets))
+        self.assertFalse(ok('', nets))
+        self.assertFalse(ok('garbage', nets))
+
+    def test_bad_entries_fail_closed_and_are_reported(self):
+        nets, problems = self.w.parse_allowed_peers('nonsense,gateway', gateways=[])
+        self.assertEqual(nets, [])
+        self.assertEqual(len(problems), 2, problems)
+        nets, _ = self.w.parse_allowed_peers('10.0.0.0/8')
+        self.assertTrue(self.w.peer_allowed('10.1.2.3', nets))
+
+    def test_default_gateway_parsed_from_route_table(self):
+        fn = os.path.join(tempfile.mkdtemp(), 'route')
+        with open(fn, 'w') as fh:
+            fh.write('Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\n'
+                     'eth0\t00000000\t010012AC\t0003\t0\t0\t0\t00000000\n'
+                     'eth0\t000012AC\t00000000\t0001\t0\t0\t0\t0000FFFF\n')
+        self.assertEqual(self.w._default_gateways(fn), ['172.18.0.1'])
+
+    def test_filtered_port_refuses_unlisted_peer_other_port_unaffected(self):
+        other = free_port()
+        g = Gunicorn(FIXED, threads=1, extra_ports=(other,), env={
+            'HAPROXY_MGR_PEER_FILTER_PORTS': str(other),
+            'HAPROXY_MGR_API_ALLOWED_PEERS': '10.255.255.1'})
+        self.addCleanup(g.stop)
+        # g.port (unfiltered) works; `other` refuses 127.0.0.1 (not listed).
+        self.assertTrue(http_get(g.port, '/').startswith('pid='))
+        for _ in range(5):
+            self.assertTrue(refused(other), 'filtered port answered an unlisted peer')
+        self.assertTrue(http_get(g.port, '/').startswith('pid='))
+        out = g.output()
+        self.assertIn('Refused 1 connection(s) to the management API', out)  # first one, immediately
+        self.assertEqual(out.count('Refused '), 1, 'reject logging is not rate-limited')
+
+    def test_filtered_port_serves_listed_peer(self):
+        port = free_port()
+        g = Gunicorn(FIXED, threads=1, extra_ports=(port,), env={
+            'HAPROXY_MGR_PEER_FILTER_PORTS': str(port),
+            'HAPROXY_MGR_API_ALLOWED_PEERS': '127.0.0.1,::1'})
+        self.addCleanup(g.stop)
+        self.assertTrue(http_get(port, '/').startswith('pid='))
+        self.assertNotIn('Refused', g.output())
+
+
+class QueuedDripStarvation(unittest.TestCase):
+    """Re-review: with the deadline armed at thread pickup, 40 reconnecting
+    drippers queued behind each other starved legit requests (every probe
+    >30s). Armed as max(enqueue + D, pickup + 1s) they must not."""
+
+    def test_40_reconnecting_drippers_do_not_starve_legit_requests(self):
+        g = Gunicorn(FIXED, threads=4, workers=2, env={})   # production defaults
+        self.addCleanup(g.stop)
+        stop = threading.Event()
+        self.addCleanup(stop.set)
+
+        def drip():
+            while not stop.is_set():
+                try:
+                    s = socket.create_connection(('127.0.0.1', g.port))
+                except OSError:
+                    time.sleep(0.2)
+                    continue
+                try:
+                    req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 5000
+                    for i in range(len(req)):
+                        if stop.is_set():
+                            break
+                        s.send(req[i:i + 1])
+                        time.sleep(1.0)
+                except OSError:
+                    pass
+                finally:
+                    s.close()
+
+        for _ in range(40):
+            threading.Thread(target=drip, daemon=True).start()
+        time.sleep(2)
+        lat = []
+        t_end = time.time() + 25
+        while time.time() < t_end:
+            t0 = time.time()
+            try:
+                http_get(g.port, '/', timeout=30)
+                lat.append(time.time() - t0)
+            except Exception:
+                lat.append(99.0)
+            time.sleep(1)
+        stop.set()
+        lat.sort()
+        median = lat[len(lat) // 2]
+        sys.stderr.write('\n  40 drippers: probes=%d median=%.2fs worst=%.2fs\n'
+                         % (len(lat), median, lat[-1]))
+        # Armed at pickup, every probe here exceeded 30s (total starvation).
+        # With max(enqueue + D, pickup + 1s) queued drippers get ~1s of thread
+        # time, so legit requests keep getting through. They still queue: the
+        # equilibrium wait is ~(drippers/threads) x hold time, about 9s at 40
+        # drippers on 8 threads -- measured, see the round-2 report. The
+        # remaining fix for that is not admitting partial requests to a thread
+        # at all; the peer filter makes it a local-only scenario.
+        # The property under test is "no starvation" (every probe answered
+        # inside 30s). The latency bounds depend on host load: a shared CI
+        # runner measured median 14.97s on 2 probes, so they are only enforced
+        # when HAPROXY_MGR_STRICT_TIMING=1 (quiet local box).
+        self.assertNotIn(99.0, lat, 'a legit request was starved for 30s+')
+        if os.environ.get('HAPROXY_MGR_STRICT_TIMING') == '1':
+            self.assertLess(median, 12.0, lat)
+            self.assertLess(lat[-1], 20.0, lat)
+
+
+class StalledUpstreamDefaultTimeout(unittest.TestCase):
+    """haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
+    client call to an upstream that accepts and never replies must time out."""
+
+    def test_in_process_client_to_stalled_upstream_times_out(self):
+        srv = socket.socket()
+        srv.bind(('127.0.0.1', 0))
+        srv.listen(5)
+        self.addCleanup(srv.close)
+        port = srv.getsockname()[1]
+        accepted = []
+        threading.Thread(target=lambda: accepted.append(srv.accept()),
+                         daemon=True).start()
+        # Import the real module in a child (it changes process-global socket
+        # state and opens its log files at import), then make a client call
+        # with NO explicit timeout -- what a careless future handler would do.
+        probe = textwrap.dedent('''\
+            import logging, os, sys, tempfile, time, urllib.request
+            d = tempfile.mkdtemp()
+            real = logging.FileHandler
+            logging.FileHandler = lambda fn, *a, **k: real(os.path.join(d, os.path.basename(fn)), *a, **k)
+            os.chdir(sys.argv[1]); sys.path.insert(0, sys.argv[1])
+            import haproxy_manager
+            logging.FileHandler = real
+            import socket
+            print('default', socket.getdefaulttimeout())
+            t0 = time.time()
+            try:
+                urllib.request.urlopen('http://127.0.0.1:%s/' % sys.argv[2]).read()
+                print('result returned')
+            except Exception as e:
+                print('result', type(e).__name__, round(time.time() - t0, 1))
+        ''')
+        env = dict(os.environ, HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT='1.5')
+        r = subprocess.run([sys.executable, '-c', probe, REPO_DIR, str(port)],
+                           capture_output=True, text=True, timeout=30, env=env)
+        if 'No module named' in r.stderr and 'flask' in r.stderr.lower():
+            self.skipTest('Flask not installed')
+        self.assertIn('default 1.5', r.stdout, r.stderr)
+        line = [l for l in r.stdout.splitlines() if l.startswith('result')][0]
+        self.assertTrue(line.startswith('result TimeoutError '), line)
+        elapsed = float(line.split()[-1])
+        self.assertLess(elapsed, 5)
+        self.assertTrue(accepted, 'upstream never saw the connection')
+        accepted[0][0].close()
+
+    def test_default_timeout_alone_does_not_cover_gthread_client_sockets(self):
+        # Documents why the custom worker exists: gthread's setblocking(True)
+        # resets any timeout -- including the process default -- to None.
+        old = socket.getdefaulttimeout()
+        socket.setdefaulttimeout(60)
+        try:
+            a, b = socket.socketpair()
+        finally:
+            socket.setdefaulttimeout(old)
+        self.addCleanup(a.close)
+        self.addCleanup(b.close)
+        self.assertEqual(a.gettimeout(), 60)
+        a.setblocking(True)
+        self.assertIsNone(a.gettimeout())
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-multipart-cve-mitigation.py

@@ -0,0 +1,156 @@
+#!/usr/bin/env python3
+"""Contract tests for the CVE-2026-109618 / -109619 edge mitigations.
+
+  * templates/hap_listener.tpl denies multipart/form-data without a
+    Content-Length (400), with AND without the Coraza SPOE block rendered.
+  * coraza-spoa/overrides.conf carries the body rules 209610-209616.
+
+What these rules DO is measured elsewhere: the Coraza verdicts by WHP's
+scripts/tests/coraza-body-parse-harness (cve109618.go, run with
+HMB_OVERRIDES pointing at this checkout's overrides.conf), the HAProxy verdict
+by live curl against a host. This file pins the invariants that make those
+measurements stay true:
+
+  * the deny is outside the `coraza_spoe_backend` conditional;
+  * it keys on the Content-Length HEADER count, never req.body_size (see the
+    H3 note in hap_listener.tpl);
+  * every 2096xx id is inside 200000-209999 -- WHP's break-glass rule
+    (991000001) removes exactly that range, so an id outside it would stay
+    armed for a site in emergency mode;
+  * no ctl:ruleEngine=On sits on a chain STARTER (Coraza runs it as soon as
+    the first link matches -- the 920440 defect).
+
+Running
+-------
+    python3 scripts/test-multipart-cve-mitigation.py
+"""
+
+import logging
+import os
+import re
+import sys
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+CORAZA_OVERRIDES = os.path.join(MODULE_DIR, 'coraza-spoa', 'overrides.conf')
+ENGINE_LO, ENGINE_HI = 200000, 209999   # WHP_WAF_ENGINE_RANGE_LO..HI
+CVE_IDS = range(209610, 209617)
+
+
+def render(coraza):
+    kw = dict(crt_path='/etc/haproxy/certs', suspension_enabled=True,
+              coraza_spoe_backend='coraza-spoa:9000' if coraza else None,
+              cluster_secret=None, syslog_target='172.18.0.1:514')
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(**kw)
+
+
+def active_lines(text):
+    return [l.strip() for l in text.splitlines()
+            if l.strip() and not l.strip().startswith('#')]
+
+
+DENY_RE = re.compile(
+    r'^http-request deny deny_status 400 .*\bif is_multipart_form '
+    r'!\{ req\.hdr_cnt\(content-length\) gt 0 \}$')
+
+
+class ListenerDeny(unittest.TestCase):
+
+    def check(self, coraza):
+        lines = active_lines(render(coraza))
+        acl = [i for i, l in enumerate(lines) if l.startswith('acl is_multipart_form ')]
+        deny = [i for i, l in enumerate(lines) if DENY_RE.match(l)]
+        self.assertEqual(len(acl), 1, 'is_multipart_form acl missing or duplicated')
+        self.assertEqual(len(deny), 1, 'multipart no-Content-Length deny missing or duplicated')
+        self.assertEqual(lines[acl[0]],
+                         'acl is_multipart_form req.fhdr(content-type) -m sub -i multipart/form-data')
+        self.assertLess(acl[0], deny[0])
+        self.assertNotIn('body_size', lines[deny[0]])
+        # after the real-IP resolution, before any backend selection rule
+        real_ip = max(i for i, l in enumerate(lines) if 'set-var(txn.real_ip)' in l)
+        self.assertLess(real_ip, deny[0])
+        if coraza:
+            spoe = [i for i, l in enumerate(lines) if l.startswith('http-request send-spoe-group')]
+            self.assertTrue(spoe, 'coraza render has no send-spoe-group -- test is blind')
+            self.assertLess(deny[0], spoe[0], 'deny must run before the WAF is consulted')
+
+    def test_with_coraza(self):
+        self.check(True)
+
+    def test_without_coraza(self):
+        self.check(False)
+
+
+@unittest.skipUnless(os.path.exists(CORAZA_OVERRIDES),
+                     'coraza-spoa/overrides.conf is not shipped in this image')
+class OverrideRules(unittest.TestCase):
+
+    def setUp(self):
+        with open(CORAZA_OVERRIDES) as fh:
+            text = fh.read()
+        # Join backslash continuations so one directive is one string.
+        joined = re.sub(r'\\\n\s*', '', text)
+        self.rules = [l.strip() for l in joined.splitlines()
+                      if l.strip().startswith('SecRule') or l.strip().startswith('SecAction')]
+
+    def ids(self):
+        out = []
+        for r in self.rules:
+            m = re.search(r'\bid:(\d+)', r)
+            if m:
+                out.append(int(m.group(1)))
+        return out
+
+    def test_every_cve_id_present_once(self):
+        ids = self.ids()
+        for rid in CVE_IDS:
+            self.assertEqual(ids.count(rid), 1, f'{rid} missing or duplicated')
+
+    def test_every_id_inside_the_break_glass_engine_range(self):
+        # every rule that belongs to the mitigation, found by what it DOES
+        # (its tag or its tx variables), not by its id
+        cve = [int(re.search(r'\bid:(\d+)', r).group(1)) for r in self.rules
+               if re.search(r'\bid:\d+', r)
+               and ('whp/cve-2026-1096' in r or 'whp_cve109618' in r)]
+        self.assertGreaterEqual(len(cve), 7, 'CVE rules not found -- the test is blind')
+        self.assertTrue(cve, 'no CVE ids parsed -- the test is blind')
+        for rid in cve:
+            self.assertTrue(ENGINE_LO <= rid <= ENGINE_HI, rid)
+
+    def test_deny_rules_flip_the_engine_for_themselves(self):
+        denies = [r for r in self.rules if re.search(r'\bid:2096\d\d\b', r) and ',deny,' in r]
+        self.assertEqual(sorted(int(re.search(r'id:(\d+)', r).group(1)) for r in denies),
+                         [209610, 209613, 209615, 209616])
+        for i, r in enumerate(self.rules):
+            if r not in denies:
+                continue
+            if ',chain' in r:
+                self.assertNotIn('ctl:ruleEngine', r, 'engine flip on a chain starter')
+                self.assertIn('ctl:ruleEngine=On', self.rules[i + 1])
+            else:
+                self.assertIn('ctl:ruleEngine=On', r)
+
+    def test_no_chain_starter_anywhere_in_the_block_flips_the_engine(self):
+        for r in self.rules:
+            if re.search(r'\bid:2096\d\d\b', r) and ',chain' in r:
+                self.assertNotIn('ctl:ruleEngine', r)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-waf-body-inspection.py

@@ -0,0 +1,395 @@
+#!/usr/bin/env python3
+"""Regression tests for WAF request-body inspection (Coraza via SPOE).
+
+Why this file exists
+--------------------
+Measured 2026-10-09 on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0
+(report: waf-body-hardening-report.md):
+
+  * Without `option http-buffer-request`, send-spoe-group fires at header
+    time and `req.body` holds only what happened to arrive with the headers.
+    A 14-byte `content=$a($b)` POST (CRS 933180, an ENFORCED rule) passed
+    uninspected when its body was written in a second send(); a body sent 2s
+    late, or after `Expect: 100-continue`, was never inspected at all.
+  * Coraza+CRS costs ~3.3 ms per KB of body. The SPOE `timeout processing`
+    is 100 ms and FAILS OPEN, so raising tune.bufsize to 32 KB turned a
+    payload at byte 0 of a 30 KB base64 body into a bypass. bufsize is a
+    security limit, not a performance knob.
+  * coraza-spoa's SPOP library refuses frames > 65535 bytes, and Coraza's
+    SecRequestBodyLimitAction Reject 413s even under DetectionOnly, so the
+    three limits must stay ordered.
+  * The X-WHP-WAF-Body stamp drives enforcement of body-parse errors
+    (200002/200003) in WHP's generated Coraza rules. A client must never be
+    able to set it, and a backend must never see it.
+
+These tests pin the rendered config. That HAProxy ACCEPTS it is
+validate-rendered-config.py's job; that it BEHAVES is the live measurement.
+
+Running
+-------
+    python3 scripts/test-waf-body-inspection.py
+"""
+
+import logging
+import os
+import re
+import sys
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+SPOA_FRAME_CAP = 65535          # dropmorepackets/haproxy-go v0.0.8 maxFrameSize
+CORAZA_CONFIG = os.path.join(MODULE_DIR, 'coraza-spoa', 'config.yaml')
+
+
+def render(template, coraza):
+    kw = dict(crt_path='/etc/haproxy/certs', suspension_enabled=False,
+              coraza_spoe_backend='coraza-spoa:9000' if coraza else None,
+              cluster_secret=None, syslog_target='172.18.0.1:514')
+    return haproxy_manager.template_env.get_template(template).render(**kw)
+
+
+def active_lines(text):
+    """Config lines with comments and blanks stripped (comments mention the
+    directives by name, so a raw substring check would pass on prose)."""
+    out = []
+    for line in text.splitlines():
+        s = line.strip()
+        if s and not s.startswith('#'):
+            out.append(s)
+    return out
+
+
+def index_of(lines, pattern):
+    for i, line in enumerate(lines):
+        if re.search(pattern, line):
+            return i
+    return -1
+
+
+class ListenerWithCoraza(unittest.TestCase):
+
+    def setUp(self):
+        self.lines = active_lines(render('hap_listener.tpl', coraza=True))
+
+    def test_body_is_buffered_before_inspection(self):
+        self.assertIn('option http-buffer-request', self.lines)
+
+    def test_stamp_order_strip_set_inspect_strip(self):
+        strip1 = index_of(self.lines, r'^http-request del-header X-WHP-WAF-Body$')
+        size = index_of(self.lines, r'^http-request set-var\(txn\.waf_body_size\) req\.hdr_val\(content-length\) if \{ req\.hdr_cnt\(content-length\) eq 1 \}$')
+        complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
+        partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if \{ var\(txn\.waf_body_size\) -m found \} !\{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
+        hdrs_len = index_of(self.lines, r'^http-request set-var\(txn\.waf_hdrs_len\) req\.hdrs,length if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \}$')
+        nocl_complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_hdrs_len\) -m found \} \{ req\.body_len,add\(txn\.waf_hdrs_len\) lt (\d+) \}$')
+        nocl_partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \} !\{ var\(txn\.waf_body\) -m found \}$')
+        stamp = index_of(self.lines, r'^http-request set-header X-WHP-WAF-Body %\[var\(txn\.waf_body\)\] if \{ var\(txn\.waf_body\) -m found \}$')
+        spoe = index_of(self.lines, r'^http-request send-spoe-group coraza coraza-req$')
+        deny = index_of(self.lines, r'^http-request return status 403 .*var\(txn\.coraza\.action\) -m str deny')
+        dup_ct = index_of(self.lines, r'^http-request deny deny_status 400 if \{ req\.fhdr_cnt\(content-type\) gt 1 \}$')
+        self.assertGreaterEqual(dup_ct, 0, 'duplicate Content-Type must be rejected at the edge')
+        self.assertLess(dup_ct, spoe, 'and before Coraza is asked')
+        for name, idx in (('client-copy strip', strip1), ('size var', size), ('complete', complete),
+                          ('partial', partial), ('no-CL header length', hdrs_len),
+                          ('no-CL complete', nocl_complete),
+                          ('no-CL partial', nocl_partial), ('stamp', stamp),
+                          ('send-spoe-group', spoe), ('deny', deny)):
+            self.assertGreaterEqual(idx, 0, f'{name} rule missing from the rendered listener')
+        # Client copies stripped BEFORE we stamp, stamp BEFORE the SPOE call.
+        self.assertLess(strip1, size)
+        self.assertLess(size, complete)
+        # The no-CL `partial` fallback relies on "-m found" of an earlier
+        # `complete`: order is load-bearing.
+        self.assertLess(size, hdrs_len)
+        self.assertLess(hdrs_len, nocl_complete)
+        self.assertLess(nocl_complete, nocl_partial)
+        self.assertLess(max(complete, partial, nocl_partial), stamp)
+        self.assertLess(stamp, spoe)
+        # Stripped again AFTER the SPOE call so no backend ever sees it.
+        strip2 = [i for i, l in enumerate(self.lines)
+                  if l == 'http-request del-header X-WHP-WAF-Body' and i > spoe]
+        self.assertTrue(strip2, 'X-WHP-WAF-Body must be deleted again after send-spoe-group')
+        self.assertLess(strip2[0], deny)
+
+    def test_no_content_length_complete_cannot_have_filled_the_buffer(self):
+        """Without a Content-Length, `complete` is inferred from "the buffer
+        cannot have been full". body + headers thresholds must sit well below
+        bufsize - maxrewrite (1024), or a truncated chunked body would be
+        stamped complete and a parse error on it would 403."""
+        line = [l for l in self.lines if 'str(complete) if { var(txn.waf_hdrs_len)' in l][0]
+        total_lt = int(re.search(r'add\(txn\.waf_hdrs_len\) lt (\d+)', line).group(1))
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
+        self.assertLessEqual(total_lt, (bufsize - 1024) * 0.85)
+
+    def test_stamp_never_derived_from_req_body_size(self):
+        """req.body_size is "buffered + htx->extra" and HAProxy 3.0's H3 mux
+        never sets htx->extra, so over HTTP/3 it equals req.body_len and a
+        body truncated at one buffer reads as `complete` (measured on whp01
+        2026-10-17: six Divi sync-to-server saves of 252-417 KB 403'd by
+        WHP 991900020; lab-reproduced on haproxy 3.0.11 +QUIC). The
+        advertised length must come from the Content-Length header."""
+        uses = [l for l in self.lines if 'req.body_size' in l]
+        self.assertEqual(uses, [], 'req.body_size must not feed the X-WHP-WAF-Body stamp')
+        size = [l for l in self.lines if l.startswith('http-request set-var(txn.waf_body_size) ')]
+        self.assertEqual(len(size), 1)
+        self.assertIn('req.hdr_val(content-length)', size[0])
+
+    def test_complete_and_partial_are_mutually_exclusive_conditions(self):
+        """With a Content-Length, `complete` and `partial` must be the same
+        condition, negated -- an inverted or loosened `complete` would enforce
+        parse errors on truncated bodies (403 on large legitimate posts)."""
+        c = [l for l in self.lines if 'str(complete) if { var(txn.waf_body_size)' in l]
+        p = [l for l in self.lines if 'str(partial) if { var(txn.waf_body_size)' in l]
+        self.assertEqual((len(c), len(p)), (1, 1))
+        c, p = c[0], p[0]
+        self.assertEqual(c.split(' if ', 1)[1].replace('!{', '{'),
+                         p.split(' if ', 1)[1].replace('!{', '{'))
+        self.assertIn('!{ req.body_len', p)
+        self.assertNotIn('!{', c)
+
+    def test_stamp_never_set_from_client_input(self):
+        """The only set-header for the stamp is from our own var."""
+        sets = [l for l in self.lines if re.search(r'set-header\s+X-WHP-WAF-Body', l, re.I)]
+        self.assertEqual(len(sets), 1)
+        self.assertIn('%[var(txn.waf_body)]', sets[0])
+        self.assertFalse([l for l in self.lines if re.search(r'add-header\s+X-WHP-WAF-Body', l, re.I)])
+
+    def test_access_log_carries_coverage_fields(self):
+        lf = [l for l in self.lines if l.startswith('log-format ')]
+        self.assertEqual(len(lf), 1)
+        self.assertIn('wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)]', lf[0])
+
+    def test_access_log_carries_waf_verdict_fields(self):
+        """wrule/wrules/wscore are appended LAST so any consumer anchored on
+        the earlier fields keeps working."""
+        lf = [l for l in self.lines if l.startswith('log-format ')]
+        self.assertEqual(len(lf), 1)
+        self.assertTrue(lf[0].endswith(
+            'wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] '
+            'wscore=%[var(txn.coraza.anomaly_score)]"'))
+
+    def test_fail_open_preserved(self):
+        """Nothing may deny on var(txn.coraza.error): an SPOA outage must not
+        503 customer sites."""
+        self.assertFalse([l for l in self.lines if 'txn.coraza.error' in l and
+                          re.search(r'\b(deny|reject|return|silent-drop|tarpit)\b', l)])
+
+
+class ListenerWithoutCoraza(unittest.TestCase):
+
+    def test_waf_off_config_has_no_body_buffering(self):
+        lines = active_lines(render('hap_listener.tpl', coraza=False))
+        self.assertNotIn('option http-buffer-request', lines)
+        self.assertFalse([l for l in lines if 'X-WHP-WAF-Body' in l])
+        self.assertFalse([l for l in lines if 'send-spoe-group' in l])
+
+    def test_waf_off_header_keeps_default_bufsize(self):
+        lines = active_lines(render('hap_header.tpl', coraza=False))
+        self.assertFalse([l for l in lines if l.startswith('tune.bufsize')])
+
+
+class BufsizeCeilings(unittest.TestCase):
+
+    def setUp(self):
+        lines = active_lines(render('hap_header.tpl', coraza=True))
+        found = [l for l in lines if l.startswith('tune.bufsize')]
+        self.assertEqual(len(found), 1, 'tune.bufsize must be pinned when Coraza is on')
+        self.bufsize = int(found[0].split()[1])
+        spoe = active_lines(haproxy_manager.template_env.get_template(
+            'hap_coraza_spoe_engine.tpl').render())
+        m = [re.match(r'timeout\s+processing\s+(\d+)ms$', l) for l in spoe]
+        m = [x for x in m if x]
+        self.assertEqual(len(m), 1)
+        self.processing_ms = int(m[0].group(1))
+
+    def test_bufsize_fits_the_spoe_time_budget(self):
+        """~3.3 ms/KB measured; 31 KB took 104 ms against a 100 ms timeout and
+        FAILED OPEN. Keep the worst case at about half the budget."""
+        worst_ms = (self.bufsize / 1024.0) * 3.3
+        self.assertLessEqual(worst_ms, self.processing_ms * 0.6,
+                             f'tune.bufsize {self.bufsize} => ~{worst_ms:.0f} ms of Coraza time '
+                             f'against a {self.processing_ms} ms fail-open timeout')
+
+    def test_bufsize_below_spop_frame_cap(self):
+        self.assertLess(self.bufsize, SPOA_FRAME_CAP)
+
+    def test_body_arg_is_the_full_buffered_body(self):
+        """The message must keep sending req.body unsliced: the `complete`
+        stamp compares req.body_len to Content-Length, so a slice would make
+        a truncated body look complete to Coraza."""
+        spoe = haproxy_manager.template_env.get_template('hap_coraza_spoe_engine.tpl').render()
+        args = [l.strip() for l in spoe.splitlines() if l.strip().startswith('args ')]
+        self.assertEqual(len(args), 1)
+        self.assertTrue(args[0].endswith(' body=req.body'))
+
+
+class CorazaTransactionIdIsTheRequestReference(unittest.TestCase):
+    """Coraza's transaction ID must be HAProxy's unique-id, so the reference a
+    blocked customer quotes (403-waf.html / X-Request-Reference / `id=` in
+    haproxy.log) is the `unique_id` in /var/log/coraza/audit.log and
+    spoa.log. Without `id=` coraza-spoa v0.7.1 invents a random 16-letter ID
+    (internal/application.go HandleRequest) that joins to nothing."""
+
+    def setUp(self):
+        spoe = haproxy_manager.template_env.get_template(
+            'hap_coraza_spoe_engine.tpl').render()
+        lines = active_lines(spoe)
+        self.messages = {}
+        current = None
+        for line in lines:
+            m = re.match(r'^spoe-message\s+(\S+)$', line)
+            if m:
+                current = m.group(1)
+                continue
+            if line.startswith('spoe-') or line.startswith('['):
+                current = None
+            elif current and line.startswith('args '):
+                self.messages[current] = line.split()[1:]
+
+    def args_of(self, message):
+        return dict(a.split('=', 1) for a in self.messages[message])
+
+    def test_request_message_sends_unique_id(self):
+        self.assertIn('coraza-req', self.messages)
+        self.assertEqual(self.args_of('coraza-req').get('id'), 'unique-id')
+
+    def test_request_message_exports_rule_ids(self):
+        """Without exportRuleIDs coraza-spoa never sets txn.coraza.rule_ids
+        and the access log's wrules= is always '-'."""
+        self.assertEqual(self.args_of('coraza-req').get('exportRuleIDs'), 'bool(true)')
+
+    def test_app_is_still_the_first_arg(self):
+        """agent.go HandleSPOE Panics unless the FIRST kv entry is `app`."""
+        for name, args in self.messages.items():
+            self.assertTrue(args[0].startswith('app='),
+                            f'{name}: first arg must be app=, got {args[0]}')
+
+    def test_every_response_message_carries_the_same_id(self):
+        """HandleResponse finds the request transaction by `id`; a coraza-res
+        without it errors ("response id is empty") on every response."""
+        for name in self.messages:
+            if name != 'coraza-req':
+                self.assertEqual(self.args_of(name).get('id'), 'unique-id', name)
+
+    def test_unique_id_is_the_header_reference(self):
+        """The fetch is only meaningful while unique-id-format is the UUID that
+        is also emitted as X-Request-Reference and on the 403 page."""
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        self.assertIn('unique-id-format %[uuid()]', [re.sub(r'\s+', ' ', l) for l in header])
+        self.assertIn('unique-id-header X-Request-Reference', [re.sub(r'\s+', ' ', l) for l in header])
+        listener = render('hap_listener.tpl', coraza=True)
+        self.assertIn('x-request-reference "%[unique-id]"', listener)
+
+
+CORAZA_OVERRIDES = os.path.join(MODULE_DIR, 'coraza-spoa', 'overrides.conf')
+
+
+@unittest.skipUnless(os.path.exists(CORAZA_OVERRIDES),
+                     'coraza-spoa/overrides.conf is not shipped in this image; '
+                     'this class runs from a repo checkout')
+class BakedPromotions(unittest.TestCase):
+    """920440 is a CHAIN whose first link matches every URL with a file
+    extension. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on that
+    first link, so every *.php / *.css request was evaluated with the engine
+    On (full CRS anomaly enforcement, in detect_only mode too). Measured
+    in-process and live, 2026-10-09; fixed by keying the flip on the chain's
+    real condition."""
+
+    def setUp(self):
+        with open(CORAZA_OVERRIDES) as fh:
+            self.lines = active_lines(fh.read())
+
+    def test_920440_is_not_promoted_on_its_chain_starter(self):
+        self.assertFalse([l for l in self.lines
+                          if re.match(r'SecRuleUpdateActionById\s+(\S*-)?920440\b', l)],
+                         '920440 promoted via SecRuleUpdateActionById again')
+
+    def test_no_chain_starter_is_promoted(self):
+        """Chain-starter rules in the promoted ranges (CRS 4.25.0): 920440,
+        944110, 944120. None may be covered by a SecRuleUpdateActionById."""
+        chain_starters = (920440, 944110, 944120)
+        promoted = []
+        for l in self.lines:
+            m = re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l)
+            if m:
+                lo = int(m.group(1))
+                promoted.append((lo, int(m.group(2) or lo)))
+        self.assertTrue(promoted, 'no promotions parsed -- the test is blind')
+        for rid in chain_starters:
+            self.assertFalse([r for r in promoted if r[0] <= rid <= r[1]],
+                             f'{rid} is a CRS chain starter and must not be promoted')
+
+    def test_log4shell_range_still_promoted(self):
+        for rid in (944100, 944130, 944150, 944151, 944152, 944300):
+            self.assertTrue([1 for l in self.lines
+                             if (m := re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l))
+                             and int(m.group(1)) <= rid <= int(m.group(2) or m.group(1))],
+                            f'{rid} lost its promotion')
+
+    def test_920440_flip_keys_on_the_restricted_extension(self):
+        i = self.lines.index('SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \\')
+        self.assertEqual(self.lines[i + 1],
+                         '"id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"')
+
+
+@unittest.skipUnless(os.path.exists(CORAZA_CONFIG),
+                     'coraza-spoa/config.yaml is not shipped in this image; '
+                     'this class runs from a repo checkout')
+class CorazaBodyLimits(unittest.TestCase):
+
+    def setUp(self):
+        with open(CORAZA_CONFIG) as fh:
+            self.cfg = active_lines(fh.read())
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        self.bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
+
+    def directive(self, name):
+        vals = [l.split(None, 1)[1] for l in self.cfg if l.split()[0] == name]
+        self.assertEqual(len(vals), 1, f'{name} must be set exactly once in config.yaml')
+        return vals[0]
+
+    def test_limit_strictly_above_anything_haproxy_can_send(self):
+        limit = int(self.directive('SecRequestBodyLimit'))
+        self.assertGreater(limit, SPOA_FRAME_CAP)
+        self.assertGreater(limit, self.bufsize)
+
+    def test_over_limit_never_rejects(self):
+        """Reject interrupts even under DetectionOnly -> 403 on uploads."""
+        self.assertEqual(self.directive('SecRequestBodyLimitAction'), 'ProcessPartial')
+
+    def test_body_buffer_stays_in_memory(self):
+        self.assertGreaterEqual(int(self.directive('SecRequestBodyInMemoryLimit')),
+                                int(self.directive('SecRequestBodyLimit')))
+
+    def test_json_depth_limit_pinned(self):
+        """Quadratic JSON flattening: depth 300 = 297 ms, past the fail-open
+        timeout. Legitimate WordPress JSON measured at depth <= 9."""
+        depth = int(self.directive('SecRequestBodyJsonDepthLimit'))
+        self.assertGreaterEqual(depth, 32, 'too tight for real block/theme payloads')
+        self.assertLessEqual(depth, 64, 'depth 100 already costs ~40 ms')
+
+    def test_limits_are_set_after_the_recommended_file(self):
+        inc = self.cfg.index('Include @coraza.conf-recommended')
+        for name in ('SecRequestBodyLimit ', 'SecRequestBodyJsonDepthLimit '):
+            lim = [i for i, l in enumerate(self.cfg) if l.startswith(name)][0]
+            self.assertLess(inc, lim, f'the recommended file would override {name.strip()}')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/validate-rendered-config.py

@@ -128,6 +128,9 @@ STUB_FILES = {
     # `lf-file` on the Coraza deny rule; loaded at parse time. Present in the
     # image (COPY errors /haproxy/errors), stubbed for anything else.
     '/haproxy/errors/403-waf.html': '<html><body>blocked %[unique-id]</body></html>\n',
+    # `file` on the default-splash return rules; loaded at parse time too.
+    '/haproxy/errors/default-splash.html': '<html><body>splash</body></html>\n',
+    '/haproxy/errors/robots-unrouted.txt': 'User-agent: *\nDisallow: /\n',
 }
 
 # (suspension_enabled, coraza_spoe_backend) combinations to render + validate.
@@ -137,6 +140,12 @@ SCENARIOS = (
         'HAPROXY_SUSPENSION_ENABLED': 'true',
         'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
     }),
+    # The default splash is ON unless switched off; render the off shape too.
+    ('splash-off', {
+        'HAPROXY_SUSPENSION_ENABLED': 'true',
+        'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
+        'HAPROXY_DEFAULT_SPLASH': 'off',
+    }),
 )
 
 
@@ -298,7 +307,8 @@ def render(scenario_name, env_overrides, workdir):
     seed_database(os.path.join(scenario_dir, 'haproxy_config.db'))
 
     saved_env = {}
-    for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND'):
+    for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND',
+                'HAPROXY_DEFAULT_SPLASH'):
         saved_env[key] = os.environ.pop(key, None)
     os.environ.update(env_overrides)
 
@@ -330,6 +340,8 @@ def assert_scenario_branches(scenario_name, config_path, env_overrides):
                        'HAPROXY_SUSPENSION_ENABLED' in env_overrides),
         'coraza': ('filter spoe engine coraza',
                    'HAPROXY_CORAZA_SPOE_BACKEND' in env_overrides),
+        'splash': ('file /haproxy/errors/default-splash.html',
+                   env_overrides.get('HAPROXY_DEFAULT_SPLASH') != 'off'),
     }
     for label, (needle, should_be_present) in expected.items():
         present = needle in text

templates/hap_backend.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -29,7 +29,7 @@ backend {{ name }}-sse-backend
 
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_http_check.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option httpchk
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     {% if ssl_enabled %}http-request set-header X-Forwarded-Proto https if { ssl_fc }{% endif %}

templates/hap_backend_longlived.tpl

@@ -14,7 +14,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -31,7 +31,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_websocket.tpl

@@ -7,7 +7,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -24,7 +24,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_coraza_spoe_engine.tpl

@@ -10,9 +10,9 @@
 # config.yaml.
 #
 # Reference: this config follows the shape from coraza-spoa's upstream
-# example/haproxy/coraza.cfg (v0.7.1). Arg names + ordering are required by
+# example/haproxy/coraza.cfg (v0.7.1). Arg NAMES are a contract with
-# Coraza-SPOA exactly as specified — DO NOT reorder or rename without
+# coraza-spoa — DO NOT rename without coordinating with the agent. Ordering:
-# coordinating with the agent.
+# only `app` is positional (it must be FIRST); the rest are matched by name.
 
 [coraza]
 
@@ -56,9 +56,41 @@ spoe-agent coraza
 
 # Per-request inspection message. No `event` directive — fires only when
 # explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.
-# Arg order/names are mandatory: Coraza-SPOA parses positionally and renames
+#
-# break the agent. `app=str(haproxy)` is the literal application name from
+# How coraza-spoa v0.7.1 reads these args (measured from source, not inferred):
-# coraza-spoa's config.yaml `applications:` block.
+#   * `app` MUST be the first arg. internal/agent.go HandleSPOE reads exactly
+#     one KV entry and Panics if its name is not "app".
+#   * Every other arg is matched BY NAME in a switch (internal/application.go
+#     HandleRequest); order is irrelevant and unknown names are ignored with a
+#     debug log. So a typo'd name silently drops that input — renames still
+#     break inspection, they just don't error.
+# `app=str(haproxy)` is the literal application name from coraza-spoa's
+# config.yaml `applications:` block.
+#
+# id=unique-id makes Coraza's transaction ID the SAME UUID HAProxy generates
+# via `unique-id-format %[uuid()]` (hap_header.tpl) — the value shown on the
+# 403-waf.html page, sent as X-Request-Reference, and logged as `id=` in
+# /var/log/haproxy.log. coraza-spoa passes it to waf.NewTransactionWithID(),
+# so it becomes `transaction.id` in /var/log/coraza/audit.log and
+# `match.unique_id` in /var/log/coraza/spoa.log: a customer-quoted reference
+# greps straight to the rule that fired. Without it coraza-spoa invents a
+# random 16-letter A-Z ID that appears nowhere on the HAProxy side. If the
+# fetch ever came back empty the agent falls back to that random ID, so this
+# cannot break inspection.
+#
+# exportRuleIDs=bool(true) makes coraza-spoa (application.go exportWAFMetrics)
+# set var(txn.coraza.rule_ids) = comma list of every matched CRS attack rule
+# (910000-959999 + 190000-199999; control rules such as our 99xxxxxxx host
+# exceptions and 2000xx body rules are NOT listed), alongside rules_hit and
+# anomaly_score which it sets regardless. var(txn.coraza.ruleid) is set on
+# every interruption (agent.go) -- for anomaly blocks that is 949110, so
+# rule_ids is what actually says WHY. All three are logged in the access
+# log (hap_listener.tpl log-format: wrule= wrules= wscore=).
+#
+# Response inspection (a `coraza-res` message) is NOT configured:
+# config.yaml has response_check: false. If it is ever enabled, coraza-res
+# MUST also send `id=unique-id` -- HandleResponse
+# looks the request transaction up by that ID and errors without it.
 #
 # src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
 # top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
@@ -66,7 +98,7 @@ spoe-agent coraza
 # when no proxy headers are present. This is what shows up as `client_ip`
 # in /var/log/coraza/audit.log and the rule manager's view-matches panel.
 spoe-message coraza-req
-    args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
+    args app=str(haproxy) id=unique-id exportRuleIDs=bool(true) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
 
 # Group binding for send-spoe-group invocation in the frontend. One group,
 # one message; could add more in the future (e.g. coraza-res for response

templates/hap_header.tpl

@@ -2,42 +2,33 @@
 # Global settings
 #---------------------------------------------------------------------
 global
-    # ACCESS LOG DESTINATION.
+    # ACCESS LOG DESTINATION: a UNIX datagram socket INSIDE this container.
     #
-    # This used to be `log 127.0.0.1 local2`, which was a silent black hole:
+    # scripts/haproxy-log-receiver.py (supervised by start-up.sh) reads it and
-    # 127.0.0.1 is the CONTAINER's own loopback, nothing has ever listened on
+    # writes a size-capped, rotating /var/log/haproxy/access.log in the
-    # udp/514 in the container netns, and there is no /dev/log in the image.
+    # haproxy-logs volume. No network listener anywhere.
-    # Every access log line -- ~1.5M/day across the whole edge -- was written
-    # to a socket with no receiver and dropped. Nothing errored, nothing
-    # warned, and `haproxy -c` was perfectly happy. The cost only shows up
-    # during an incident: per-IP 429s, tarpits, wp-admin gate redirects,
-    # WAF 403s and `silent-drop`s left no record anywhere, so the edge could
-    # not be asked what it had rejected. Only aggregate stick-table counters
-    # survived.
     #
-    # Now points at the DOCKER BRIDGE GATEWAY, where the host's rsyslog has an
+    # History: `log 127.0.0.1 local2` was a black hole (nothing listened in
-    # imudp listener bound (installed idempotently by WHP's
+    # the container netns); its successor, the docker bridge gateway
-    # setup-haproxy-logrotate.sh, which also writes the logrotate stanza).
+    # 172.18.0.1:514 with a host rsyslog imudp listener, had no listener on
-    # The host writes local2 to /var/log/haproxy.log and stops it there, so it
+    # the fleet either (probable cause, not measured: rsyslog binds that
-    # does not also flood /var/log/messages or the Graylog forwarder.
+    # address once at boot, before docker has created the bridge, and does
+    # not retry). Either way per-IP
+    # 429s, tarpits, gate redirects and WAF 403s left no record.
     #
-    # WHY NOT `log stdout format raw local0`: it is INCOMPATIBLE with the
+    # WHY NOT `log stdout`: silently emits NOTHING with `daemon` (below) on
-    # `daemon` keyword below, and incompatible SILENTLY. Verified on the
+    # the pinned 3.0.11 (`haproxy -c` still passes), and dropping `daemon`
-    # pinned 3.0.11 binary: with `daemon` set, a `log stdout` config serves
+    # breaks the synchronous `haproxy -W` launch sites in haproxy_manager.py.
-    # traffic normally and emits ZERO log lines, and `haproxy -c` returns 0
+    # Also, `docker logs haproxy-manager` feeds WHP's AI log monitor, which
-    # with no error and no warning -- so the CI config gate
+    # must not get a per-request firehose.
-    # (scripts/validate-rendered-config.py) cannot catch it either. Making it
-    # work means dropping `daemon` / adding -db so haproxy stays in the
-    # foreground, which in turn breaks the three synchronous
-    # `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites in
-    # haproxy_manager.py (they would block until the 180s timeout and then be
-    # killed). That is a change to the exact code path whose failure mode is
-    # "container Up, ports 80/443 never bound, every site down, /health still
-    # 200". Not worth it for a logging change.
     #
-    # UDP means a dead listener degrades to dropped log lines, never to a
+    # The path is relative to the `chroot` below: HAProxy resolves it after
-    # stalled or failing request path -- the correct failure direction for an
+    # chrooting, so `/log.sock` here is /var/lib/haproxy/log.sock on disk,
-    # edge fronting ~60 customer sites.
+    # which is where the receiver binds (HAPROXY_LOG_SOCKET).
+    #
+    # Sends are non-blocking: a dead receiver or a full socket buffer drops
+    # log lines, never stalls a request. HAPROXY_SYSLOG_TARGET still
+    # overrides the target (`<ip>:<port>` for a remote collector).
     #
     # len 2048 accommodates the enriched log-format in hap_listener.tpl
     # (URL + User-Agent + UUID); the default 1024 would truncate long ones.
@@ -106,6 +97,41 @@ global
 
     # Stats persistence for zero-downtime reloads
     stats-file /var/lib/haproxy/stats.dat
+{%- if coraza_spoe_backend %}
+
+    # WAF BODY INSPECTION LIMIT. Only emitted when Coraza is wired in, so a
+    # WAF-off / standalone config is unchanged byte-for-byte. 16384 is also
+    # HAProxy's default: this line changes no behaviour or memory, it pins
+    # the value so nobody raises it without reading this.
+    #
+    # The request body Coraza sees is whatever `req.body` holds when
+    # send-spoe-group fires, and with `option http-buffer-request`
+    # (hap_listener.tpl) that is "the whole body, or as much as fits in ONE
+    # buffer". So bufsize IS the inspection limit: a body up to roughly
+    # bufsize - maxrewrite (1024) - the request headers (~15 KB) is inspected
+    # whole and stamped `complete`; anything larger is forwarded intact, its
+    # first ~15 KB inspected, and stamped `partial`. Large uploads are never
+    # rejected or held for it.
+    #
+    # DO NOT RAISE THIS WITHOUT RE-MEASURING -- three ceilings, all measured:
+    #   * TIME. Coraza+CRS 4.25 costs ~3.3 ms per KB of body (linear; 31 KB
+    #     base64 = 104 ms, 15 KB = ~50 ms on a 32-core box). The SPOE
+    #     `timeout processing` is 100 ms (hap_coraza_spoe_engine.tpl) and a
+    #     timeout FAILS OPEN: with tune.bufsize 32768 a payload at byte 0 of a
+    #     30 KB base64 multipart body passed 933180 uninspected. A larger
+    #     buffer is a bypass, not more coverage, unless the timeout (and CPU
+    #     budget) grows with it.
+    #   * FRAME. coraza-spoa's SPOP library (dropmorepackets/haproxy-go
+    #     v0.0.8) refuses a HELLO whose max-frame-size exceeds 65535; HAProxy
+    #     derives its offer from bufsize. bufsize >= 64 KB = every SPOE
+    #     handshake fails = WAF silently off for all traffic.
+    #   * MEMORY. maxconn 4000 x 2 buffers x bufsize, inside the 1 GB edge
+    #     container.
+    # Coraza's SecRequestBodyLimit (coraza-spoa/config.yaml, 131072) must stay
+    # above anything this can deliver. Pinned by
+    # scripts/test-waf-body-inspection.py.
+    tune.bufsize 16384
+{%- endif %}
 
 #---------------------------------------------------------------------
 # DNS resolver for Docker container name resolution

templates/hap_listener.tpl

@@ -63,7 +63,18 @@ frontend web
     #   host=/ua= identify the vhost and client; %ST/%B/%tsc give status,
     #        bytes and the termination state that distinguishes a rate-limit
     #        deny (PR--) from a tarpit (PT--) from a normal close.
-    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major]"
+    #   wafb=/wafe= WAF body coverage: wafb is complete|partial|- (did Coraza
+    #        get the whole request body, see the Coraza block below);
+    #        wafe is var(txn.coraza.error), set when the SPOE call failed or
+    #        timed out and the request went through UNINSPECTED (fail-open).
+    #        Both are "-" when the WAF is off or the request had no body.
+    #   wrule=/wrules=/wscore= WHY Coraza acted, without opening audit.log:
+    #        wrule is the interrupting rule (949110 for anomaly blocks),
+    #        wrules every matched CRS attack rule (e.g. 920440,949110),
+    #        wscore the inbound anomaly score. Set by coraza-spoa via the
+    #        exportRuleIDs SPOE arg (hap_coraza_spoe_engine.tpl). All "-"
+    #        when the WAF is off; wrule is "-" unless Coraza interrupted.
+    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major] wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)] wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] wscore=%[var(txn.coraza.anomaly_score)]"
 
     # --- URI normalisation (MUST be the first path-touching block here) ---
     # Every path-based control in this frontend (the ACME health-check bypass,
@@ -605,13 +616,158 @@ frontend web
     http-request set-path /suspended if is_suspended_domain
     use_backend default-backend if is_suspended_domain
 {%- endif %}
+{%- if default_splash %}
+
+    # Default splash for traffic that is NOT routed to any site: the server's
+    # own hostname (registered cert-only, no backend), a bare IP, a stale or
+    # mistyped name. HAProxy answers it ITSELF with a static page and the
+    # request goes no further -- in particular it never reaches the Coraza
+    # SPOE call below.
+    #
+    # Why: `use_backend default-backend` does not short-circuit anything --
+    # HAProxy runs EVERY http-request rule (Coraza included) before it picks
+    # a backend. So the catch-all used to be WAF-inspected with the full
+    # server-wide rule set like customer traffic, and a browser visiting the
+    # server hostname also carries the control panel's cookies (cookies are
+    # not port-scoped: whp01:8443 cookies are sent to whp01:443). Anything
+    # Coraza matched there showed the WAF block page for a host that has no
+    # site behind it to protect. A fixed static reply has nothing to protect.
+    #
+    # "Routed" is decided from two lists generate_config() writes from the
+    # SAME domain rows it renders the routing ACLs from (exact names, and
+    # ".base" suffixes for wildcards), port stripped and lowercased. They
+    # are a SUPERSET of what routes -- the routing ACLs below compare the raw
+    # Host (port included) -- so a host that routes is never splashed; the
+    # worst a mismatch can do is send an unrouted request through the WAF to
+    # default-backend, which is the pre-splash behaviour. Both lists are part
+    # of the rollback set (_config_backup_pairs) so they cannot drift from
+    # the haproxy.cfg that is actually loaded.
+    #
+    # Kept OUT of the splash: ACME http-01 (the server hostname's own cert is
+    # issued through it), blocked IPs (blocked-ip page) and suspended hosts
+    # (suspended page). Rate limits, the IP block and the wp-* gates above
+    # still apply. EVERY path is 404 with the same page, plus
+    # X-Robots-Tag noindex and /robots.txt "Disallow: /", so search engines
+    # never index the server's IP or hostname as a site.
+    # Disable with HAPROXY_DEFAULT_SPLASH=off (back to default-backend).
+    acl splash_routed_host req.hdr(host),field(1,:),lower -f /etc/haproxy/routed_hosts.list
+    acl splash_routed_host req.hdr(host),field(1,:),lower -m end -f /etc/haproxy/routed_wildcards.list
+    acl splash_acme path_beg /.well-known/acme-challenge/
+    acl splash_robots path /robots.txt
+    http-request return status 200 content-type "text/plain; charset=utf-8" hdr cache-control "no-store" hdr x-robots-tag "noindex, nofollow, noarchive" hdr x-content-type-options "nosniff" file /haproxy/errors/robots-unrouted.txt if !splash_routed_host !splash_acme !is_blocked_ip{% if suspension_enabled %} !is_suspended_domain{% endif %} splash_robots
+    http-request return status 404 content-type "text/html; charset=utf-8" hdr cache-control "no-store" hdr x-robots-tag "noindex, nofollow, noarchive" hdr x-content-type-options "nosniff" file /haproxy/errors/default-splash.html if !splash_routed_host !splash_acme !is_blocked_ip{% if suspension_enabled %} !is_suspended_domain{% endif %} !splash_robots
+{%- endif %}
+
+    # CVE-2026-109618 (GHSA-w8hw-w9fx-j79w, PHP "RCE via OOM corruption"),
+    # vector (a): a multipart/form-data body with no declared length is not
+    # bounded by post_max_size -- PHP's multipart parser keeps reading it. A
+    # body WITH a Content-Length is checked against post_max_size before it is
+    # parsed. So a multipart request must carry exactly what every browser and
+    # HTTP library already sends for a form: a Content-Length.
+    #
+    # req.hdr_cnt(content-length) is the header as the client sent it, on
+    # every protocol: HTTP/1.1 chunked has none (and the H1 mux DROPS a
+    # Content-Length sent alongside Transfer-Encoding, so smuggling one in
+    # beside chunked does not pass either); HTTP/2 and HTTP/3 have none when
+    # the client streams the body. H1/H2/H3 all check a declared
+    # content-length against the DATA received, so it cannot lie about the
+    # length (same reasoning as the WAF body stamp below).
+    # Browsers send content-length for form posts and FormData uploads on
+    # h1/h2/h3; the only body a browser sends without one is a fetch() with a
+    # ReadableStream body (Chrome, h2/h3 only, `duplex: 'half'`), which is not
+    # what form uploads use.
+    #
+    # Not WAF-gated and not exempt for trusted IPs: this is a protocol
+    # guarantee the vulnerable parser needs, independent of Coraza and its
+    # mode. req.fhdr (not req.hdr) so a comma inside the value cannot split it.
+    # The rest of the advisory (part-header floods, empty file controls, deep
+    # field names) needs the body and lives in coraza-spoa/overrides.conf
+    # (209610-209616). Remove once patched PHP is on every customer image.
+    acl is_multipart_form req.fhdr(content-type) -m sub -i multipart/form-data
+    http-request deny deny_status 400 hdr x-whp-edge-block "multipart-without-content-length" if is_multipart_form !{ req.hdr_cnt(content-length) gt 0 }
 {%- if coraza_spoe_backend %}
 
     # Coraza WAF inspection via SPOE. Runs AFTER rate-limit and IP-block
     # guards (no point asking the WAF about requests we're already dropping)
     # and AFTER the real-client-IP resolution (so Coraza sees the right src).
     filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
+
+    # --- Make Coraza actually see the request body -------------------------
+    # send-spoe-group sends `req.body` = whatever body bytes are in the buffer
+    # AT THAT MOMENT. Without buffering that is usually NOTHING: measured on
+    # haproxy 3.0.11 + coraza-spoa v0.7.1, a 14-byte `content=$a($b)` POST
+    # whose body was written in a second send() right after the headers
+    # passed 933180 (an enforced rule) uninspected, while the identical bytes
+    # in one send() were blocked. A body sent 2s late, or after an
+    # `Expect: 100-continue`, was never inspected at all.
+    #
+    # `option http-buffer-request` makes HAProxy wait, before evaluating the
+    # http-request rules, until the WHOLE body is in or the buffer is full,
+    # bounded by `timeout http-request` (30s, hap_header.tpl). It answers
+    # `Expect: 100-continue` itself so the client actually sends the body.
+    # It does NOT hold uploads: once one buffer (tune.bufsize, 16 KB) is full
+    # the request is forwarded and the rest streams through as before -- a
+    # 50 MB upload is not stored at the edge.
+    option http-buffer-request
+
+    # Exactly one Content-Type, or 400. With two Content-Type lines Coraza and
+    # the backend disagree about what the body IS: Coraza evaluates every value
+    # (`text/xml` on either line selects its XML processor, which accepts a
+    # form body without error), while Apache merges the lines with ", " and
+    # PHP cuts the media type at the first ',' / ';'. Measured (Coraza v3.7.0
+    # + PHP 8.3): `application/x-www-form-urlencoded` + `text/xml` puts the
+    # payload in $_POST with nothing inspected. req.fhdr_cnt counts header
+    # LINES, so a single comma-joined value (which browsers produce when a
+    # script calls setRequestHeader twice) is not affected here; that form is
+    # parsed consistently enough by both sides (see the report's matrix).
+    # Browsers, curl, PHP-curl and the common HTTP libraries send one line.
+    http-request deny deny_status 400 if { req.fhdr_cnt(content-type) gt 1 }
+
+    # Tell Coraza whether it got the whole body. Coraza cannot tell on its
+    # own (REQUEST_BODY_LENGTH is only filled by the urlencoded/raw
+    # processors), and the difference matters: a TRUNCATED JSON body, or a
+    # multipart body cut inside a boundary/part header, fails to parse by
+    # construction, so the body-parse-error rules (200002/200003) may only
+    # deny a COMPLETE body. WHP's generated Coraza rules
+    # (regenerate-coraza-overrides.sh) key on this header:
+    #   complete -> 200002/200003 deny on multipart/JSON (enforce mode)
+    #   partial  -> 200002/200003 removed for this request (every mode)
+    #
+    # With a Content-Length: complete iff req.body_len (buffered) equals the
+    # Content-Length HEADER value (req.hdr_val). NOT req.body_size: that is
+    # "buffered + htx->extra", and in HAProxy 3.0 only the H1 and H2 muxes
+    # set htx->extra from Content-Length -- the H3 mux never does (h3.c), so
+    # over HTTP/3 req.body_size == req.body_len and every body cut at one
+    # buffer was stamped `complete`. Measured: whp01 logged 73,795 H3
+    # `complete` and 0 `partial` (Oct 6-19 2026); a 265,778-byte Divi save
+    # over H3 reached Coraza as its first ~15 KB, failed to parse and was
+    # 403'd by WHP's 991900020. H1/H2/H3 all validate Content-Length against
+    # the DATA actually received (h3.c rejects a mismatch), so the header is
+    # a protocol-independent "how long the body really is".
+    # Without one (HTTP/1.1 chunked, HTTP/2-3 without content-length) HAProxy
+    # 3.0 has no "end of message" fetch and req.body_size is just what is
+    # buffered. But http-buffer-request only releases the request early when
+    # the body ENDED or the buffer FILLED (a stalled body gets a 408 instead),
+    # and body + headers well below one buffer (12 KB of the ~15 KB usable)
+    # cannot have filled it. So that is complete, anything else partial.
+    # Without this branch an attacker only had to add
+    # `Transfer-Encoding: chunked` to escape enforcement (measured).
+    #
+    # Client-supplied copies are deleted FIRST so it cannot be forged, and it
+    # is deleted again after the SPOE call so backends never see it.
+    # txn.waf_body is also logged (`wafb=` in the log-format above) so the
+    # complete/partial mix is measurable per host.
+    http-request del-header X-WHP-WAF-Body
+    http-request set-var(txn.waf_body_size) req.hdr_val(content-length) if { req.hdr_cnt(content-length) eq 1 }
+    http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_body_size) -m found } { req.body_len,sub(txn.waf_body_size) eq 0 }
+    http-request set-var(txn.waf_body) str(partial) if { var(txn.waf_body_size) -m found } !{ req.body_len,sub(txn.waf_body_size) eq 0 }
+    http-request set-var(txn.waf_hdrs_len) req.hdrs,length if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 }
+    http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_hdrs_len) -m found } { req.body_len,add(txn.waf_hdrs_len) lt 12288 }
+    http-request set-var(txn.waf_body) str(partial) if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 } !{ var(txn.waf_body) -m found }
+    http-request set-header X-WHP-WAF-Body %[var(txn.waf_body)] if { var(txn.waf_body) -m found }
+
     http-request send-spoe-group coraza coraza-req
+    http-request del-header X-WHP-WAF-Body
 
     # Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to
     # "deny" / "drop" / "redirect" when a rule with the corresponding