feat(waf): log why Coraza acted (wrule/wrules/wscore); VERSION 2026.10.4

coraza-spoa v0.7.1 already returns the interrupting rule id
(txn.coraza.ruleid) and, when the request message sends
exportRuleIDs=bool(true), the comma list of every matched CRS attack
rule (txn.coraza.rule_ids) plus the inbound anomaly score. Send the arg
and append all three to the access log so a block's cause (e.g.
920440,949110 score 5) is visible in haproxy.log without opening the
Coraza audit log. Fields are appended last; existing ones unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAUDE.md

@@ -7,8 +7,58 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ### Testing
 - **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
 - **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
+- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
+- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
+- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
+- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
+- **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine.
+- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
 - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
 
+### Reading stick tables (and why it is easy to get silently wrong)
+
+`/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Worker commands
+(`show table`, `show map`, `add map`, ...) need an `@1` prefix. Without it
+HAProxy answers `Unknown command: 'show', ...` **and socat still exits 0** — so
+an exit-status check passes and the help text gets parsed as data. Always use
+`haproxy_cli(cmd, worker=True)` in Python, which inspects the response body.
+
+Stick-table entries are `name=value` / `name(window_ms)=value` pairs, not fixed
+columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
+by NAME, and treat a missing field as an ERROR — never default it to `0`. The
+`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
+`http_err_rate`; it holds **no history and no counter of past blocks**. What was
+actually denied/tarpitted is in the edge access log on the **host** at
+`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table.
+
+This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
+reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
+stick table has ever stored — for their entire existence. See the header of
+`haproxy_tarpit_config.txt` and the contract test.
+
+### Changing a runtime map (`add map` / `del map`)
+
+Same socket, two more ways to fail silently — and both were live in
+`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
+existence:
+
+* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
+  config-parse time and move on every config regeneration (on whp01
+  `blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
+  `add map /etc/haproxy/blocked_ips.map <ip> 1`.
+* **A mutation answers NOTHING on success**, so an empty body is the only
+  success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
+  *also* answers nothing and adds nothing, so the body cannot prove an add
+  worked. **Read it back** with `@1 get map <path> <key>`.
+* Entries must carry the value `1`; haproxy.cfg matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
+
+In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
+and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
+only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
+re-reads it on reload, so a failed runtime command must degrade to
+"enforced on reload" and be reported, never swallowed.
+
 ### Running the Application
 - **Docker Build**: `docker build -t haproxy-manager .`
 - **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)
@@ -119,6 +169,8 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
 - Stats page on port 8404 (localhost only inside container)
 - Management interface on port 8000 should be firewall-protected in production
-- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s`
+- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
+- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
+- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
 - Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
 - HAProxy is version 3.0.11

Dockerfile

@@ -23,21 +23,81 @@ LABEL org.opencontainers.image.title="haproxy-manager-base" \
       org.opencontainers.image.version="${VERSION}" \
       org.opencontainers.image.licenses="MIT"
 
-RUN apt update -y && apt dist-upgrade -y && apt install socat haproxy cron certbot curl jq net-tools -y && apt clean && rm -rf /var/lib/apt/lists/*
+# haproxy is PINNED. It was previously unpinned, so the binary could move under
+# us at Debian's timing — an upstream release that rejected our config would have
+# broken an unrelated commit's build, or worse, shipped an edge that refuses to
+# start (see the `haproxy -c` gate below for why that matters: a config HAProxy
+# rejects leaves the container Up with 80/443 unbound and /health still 200).
+#
+# Pinning does NOT make the gate redundant, and the gate does NOT make pinning
+# unnecessary — they compose. Pinned means the version moves deliberately; the
+# gate then answers immediately whether the new binary still accepts fleet config.
+# It also makes the image reproducible, which it previously was not.
+#
+# To move it: bump the version here, rebuild, and let the gate verify. If Debian
+# security-updates the package (e.g. -1+deb13u4) the build FAILS until this pin is
+# updated — that failure is the point, not a bug. Check availability with:
+#   apt-cache policy haproxy
+ARG HAPROXY_VERSION=3.0.11-1+deb13u3
+RUN apt update -y && apt dist-upgrade -y && apt install socat "haproxy=${HAPROXY_VERSION}" cron certbot curl jq net-tools -y && apt-mark hold haproxy && apt clean && rm -rf /var/lib/apt/lists/*
 WORKDIR /haproxy
 COPY ./templates /haproxy/templates
 COPY requirements.txt /haproxy/
 COPY haproxy_manager.py /haproxy/
+COPY haproxy_gunicorn_worker.py /haproxy/
+# Read by GET /api/capabilities (WHP gates whole-site re-bundling on it).
+COPY VERSION /haproxy/VERSION
 COPY scripts /haproxy/scripts
 COPY trusted_ips.list /etc/haproxy/trusted_ips.list
 COPY trusted_ips.map /etc/haproxy/trusted_ips.map
 # /etc/haproxy is a named volume in deployed containers, so baked-in files
-# under that path get shadowed by the volume on existing deployments.
+# under that path get shadowed by the volume on existing deployments. The
+# trusted_ips.* pair above predates that discovery and is handled by the
+# older start-up.sh guard (out of scope here). cloudflare_ips.list and
+# trusted_proxies.list are staged under /haproxy/defaults instead, so
+# start-up.sh can always read the image's baked copy regardless of what the
+# volume shadows /etc/haproxy with.
+COPY cloudflare_ips.list /haproxy/defaults/cloudflare_ips.list
+COPY trusted_proxies.list /haproxy/defaults/trusted_proxies.list
+COPY wpadmin_gate_exempt.list /haproxy/defaults/wpadmin_gate_exempt.list
 # Place errorfiles outside the volumed path; the HAProxy config references
 # them by absolute path.
 COPY errors /haproxy/errors
 RUN chmod +x /haproxy/scripts/*
 RUN pip install -r requirements.txt
+# ---------------------------------------------------------------------------
+# Build gate: no image ships unless the real haproxy binary accepts the config
+# this image's templates actually produce.
+#
+# On 2026-08-14 a template change rendered fine, passed all 13 unit tests, and
+# was rejected by HAProxy ("invalid arg 2 in converter 'regsub'"). It was only
+# caught because someone built an image by hand and ran `haproxy -c`. Nothing
+# in the build or in CI would have stopped it: .gitea/workflows/build-push.yaml
+# is checkout -> build -> push, and test-config-rollback.py's "haproxy" is a
+# shell stub that only rejects a sentinel token. In production an invalid
+# haproxy.cfg means init.py refuses to start HAProxy while the container stays
+# Up - ports 80/443 unbound, every site on the host down, /health still 200.
+#
+# This lives in the Dockerfile rather than in the workflow deliberately:
+#   * it cannot be skipped, and it protects local `docker build` too;
+#   * no workflow restructuring (build-push-action builds and pushes in one
+#     step, so gating in CI would mean splitting build from push);
+#   * it validates against the EXACT haproxy binary in this image. Line 26
+#     installs haproxy unpinned, so that binary moves between builds - this
+#     turns "the new haproxy rejects our config" from a silent production
+#     risk into a build failure.
+#
+# The unit suites run here too. They had never run anywhere automated either,
+# and they cost a few seconds.
+RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
+ && python3 /haproxy/scripts/test-trusted-proxy-gate.py \
+ && python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \
+ && python3 /haproxy/scripts/test-config-rollback.py \
+ && python3 /haproxy/scripts/test-cert-write-safety.py \
+ && python3 /haproxy/scripts/test-cert-scripts.py \
+ && python3 /haproxy/scripts/test-waf-body-inspection.py \
+ && python3 /haproxy/scripts/test-gunicorn-socket-timeouts.py \
+ && python3 /haproxy/scripts/validate-rendered-config.py
 # Create log directories
 RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
 RUN chmod 755 /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
@@ -50,7 +110,10 @@ RUN mkdir -p /var/spool/cron/crontabs && \
 # 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
 # must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
 EXPOSE 80 443 443/udp 8000
-# Add health check
+# Health check: management API (:8000, short timeout) AND HAProxy (:80).
+# See scripts/healthcheck.sh. NOTE: WHP's lib-edge-container.sh passes its own
+# --health-cmd, which REPLACES this one on deployed hosts; it must be pointed
+# at /haproxy/scripts/healthcheck.sh for the fleet to get the :8000 probe.
 HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
-    CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1
+    CMD /haproxy/scripts/healthcheck.sh
 CMD ["/haproxy/scripts/start-up.sh"]

IP_BLOCKING_API.md

@@ -508,20 +508,45 @@ curl -X POST http://localhost:8000/api/blocked-ips/sync \
 
 For advanced users, you can interact directly with HAProxy's runtime API:
 
+Three things about these commands are easy to get wrong, and each one fails
+**silently** (socat exits 0 either way — the rejection, if any, is only in the
+response body):
+
+* `/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Map commands are
+  worker commands and need the `@1` prefix. Without it the reply is
+  `Unknown command: 'add', ...`.
+* Reference the map by its **file path**, never by `#<id>`. Ids are assigned at
+  config-parse time and move on every config regeneration (on a live edge,
+  `blocked_ips.map` is id 37, `trusted_ips.map` is 10 — there is no id 0).
+  Worse, `@1 add map #0 <ip> 1` returns an **empty** reply and adds nothing.
+* Entries must carry the value `1`. `haproxy.cfg` matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block. (`add map`
+  with no value is rejected: `'add map' expects three parameters ...`.)
+
 ```bash
+MAP=/etc/haproxy/blocked_ips.map
+
 # Add IP to runtime (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map $MAP 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP from runtime
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
+
+# Confirm what actually happened (do not trust the exit status)
+echo "@1 get map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
 
 # Clear all blocked IPs from runtime
-echo "clear map #0" | socat stdio /var/run/haproxy.sock
+echo "@1 clear map $MAP" | socat stdio /tmp/haproxy-cli
 
-# Show all runtime map entries
-echo "show map #0" | socat stdio /var/run/haproxy.sock
+# Show all runtime map entries, and the map ids currently in use
+echo "@1 show map $MAP" | socat stdio /tmp/haproxy-cli
+echo "@1 show map" | socat stdio /tmp/haproxy-cli
 ```
 
+The runtime map is a **fast path only**. `/etc/haproxy/blocked_ips.map` is
+authoritative: HAProxy re-reads it on reload, so a failed runtime command
+delays a block until the next reload rather than losing it.
+
 ## Migration from ACL Method
 
 If you're upgrading from the old ACL-based method:

MIGRATION_GUIDE.md

@@ -50,14 +50,22 @@ http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }
 - **Graceful error handling**
 
 ### 2. Runtime IP Management
+Map commands go to a **worker** (`@1`), reference the map by **file path**
+(ids move between config regenerations, and `#0` silently adds nothing), and
+carry the value `1` that `map_ip(...,0) -m int gt 0` matches on:
+
 ```bash
 # Add IP without reload (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map /etc/haproxy/blocked_ips.map 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP without reload
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map /etc/haproxy/blocked_ips.map 192.168.1.100" | socat stdio /tmp/haproxy-cli
 ```
 
+socat exits 0 even when HAProxy rejects the command, so read the response body
+(or read the entry back with `@1 get map ...`) rather than the exit status.
+See IP_BLOCKING_API.md for the full set.
+
 ### 3. New API Endpoints
 
 #### Safe Config Reload

VERSION

@@ -1 +1 @@
-2026.08.1
+2026.10.4

cloudflare_ips.list

@@ -0,0 +1,34 @@
+# Cloudflare edge ranges — peers allowed to set CF-Connecting-IP / X-Real-IP /
+# X-Forwarded-For. Referenced by templates/hap_listener.tpl:
+#   acl from_trusted_proxy src -f /etc/haproxy/cloudflare_ips.list -f /etc/haproxy/trusted_proxies.list
+#
+# PUBLIC DATA — safe to commit. Source: https://www.cloudflare.com/ips-v4
+# and https://www.cloudflare.com/ips-v6. Keep in sync with the IPv4 snapshot in
+# WHP's ssl_renewal_orchestrator.php::isCloudflareIP() (that one is IPv4-only).
+# Refresh at release time; Cloudflare has not added a range since 2021.
+#
+# IPv4
+173.245.48.0/20
+103.21.244.0/22
+103.22.200.0/22
+103.31.4.0/22
+141.101.64.0/18
+108.162.192.0/18
+190.93.240.0/20
+188.114.96.0/20
+197.234.240.0/22
+198.41.128.0/17
+162.158.0.0/15
+104.16.0.0/13
+104.24.0.0/14
+172.64.0.0/13
+131.0.72.0/22
+#
+# IPv6
+2400:cb00::/32
+2606:4700::/32
+2803:f800::/32
+2405:b500::/32
+2405:8100::/32
+2a06:98c0::/29
+2c0f:f248::/32

coraza-spoa/config.yaml

@@ -26,6 +26,37 @@ applications:
       # CRS-bundled defaults: recommended Coraza settings + CRS setup +
       # the rule pack itself (~16 MB of rules embedded in the binary).
       Include @coraza.conf-recommended
+
+      # Request-body limits, set explicitly instead of inheriting the
+      # recommended file's 12.5 MB / Reject. HAProxy only ever hands us one
+      # buffer of body (tune.bufsize 16384 in haproxy-manager's
+      # hap_header.tpl), and the SPOP library refuses frames over 65535
+      # bytes, so no body we receive can exceed 64 KB. The invariant is:
+      #     tune.bufsize (16 KB) < SPOP frame cap (65535) < SecRequestBodyLimit
+      # Breaking it is NOT harmless: SecRequestBodyLimitAction Reject
+      # interrupts with 413 even under DetectionOnly (Coraza
+      # setAndReturnBodyLimitInterruption ignores the engine mode), which the
+      # edge turns into a 403 on every large upload. ProcessPartial is the
+      # belt to that brace. InMemoryLimit == BodyLimit keeps the body buffer
+      # off disk (this image is distroless). Pinned by
+      # haproxy-manager-base/scripts/test-waf-body-inspection.py.
+      SecRequestBodyLimit 131072
+      SecRequestBodyInMemoryLimit 131072
+      SecRequestBodyLimitAction ProcessPartial
+
+      # JSON nesting limit (the recommended file sets 1024). Coraza's JSON
+      # flattener is quadratic in nesting: a 601-byte `[[[...]]]` of depth 300
+      # cost 297 ms and depth 1000 (2 KB) 3.1 s of CPU -- past the 100 ms SPOE
+      # timeout, so the request passed UNINSPECTED (fail-open) while the WAF
+      # burned CPU. Over the limit the body fails to parse (REQBODY_ERROR,
+      # 0.1 ms), which WHP's enforce-mode body-integrity rule denies on a
+      # complete body. Legitimate depth, measured on the JSON shipped in
+      # WordPress 6.x core, Gutenberg, WooCommerce and Elementor (theme.json,
+      # styles/*.json, block.json, checkout schemas): max 8, so a global-styles
+      # save is ~9; 64 leaves ~7x headroom for Divi/Elementor-style module
+      # trees that could not be sampled. Depth 64 itself costs ~21 ms.
+      SecRequestBodyJsonDepthLimit 64
+
       Include @crs-setup.conf.example
 
       # Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.

coraza-spoa/overrides.conf

@@ -47,7 +47,17 @@ SecRuleUpdateActionById 933170-933200 "ctl:ruleEngine=On"
 # vulnerable, but blocking at the edge keeps logs clean and protects
 # any future Java workloads.
 # ---------------------------------------------------------------------------
-SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
+# 944110 and 944120 are NOT promoted (changed 2026-10-09, 2026.10.2). Both are
+# CHAINS whose first link matches bare words anywhere in the request --
+# `runtime|processbuilder` and `getproperty|forclosure|filewriter|...` -- and
+# Coraza runs an appended ctl:ruleEngine=On on that first link even when the
+# chain then fails. Measured (Coraza v3.7.0 + CRS 4.25.0): in detect_only,
+# `?q=runtime&id=1' or '1'='1` was 403 while the SQLi alone was 200. Same
+# chain-starter defect as 920440. They still add to the anomaly score.
+# Log4Shell (944150/944151/944152, ...) stays promoted. WHP mirrors this split
+# in coraza_baked_promotion_ranges() (web-files/libs/coraza_blocking_set.php).
+SecRuleUpdateActionById 944100 "ctl:ruleEngine=On"
+SecRuleUpdateActionById 944130-944300 "ctl:ruleEngine=On"
 
 # ---------------------------------------------------------------------------
 # 920440 — URL file extension restricted by policy
@@ -59,7 +69,26 @@ SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
 # single GCP-hosted scanner. Zero false positives observed; standard
 # WP/WooCommerce/Divi/HPR URLs do not end in these extensions.
 # ---------------------------------------------------------------------------
-SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
+# HOW IT IS PROMOTED (changed 2026-10-09, haproxy-manager-base 2026.10.2).
+# This used to be `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"`, which
+# was WRONG: 920440 is a CHAIN whose first link is
+# `REQUEST_BASENAME "@rx \.([^.]+)$"` -- it matches EVERY URL with a file
+# extension -- and Coraza runs actions appended to a chain starter when that
+# first link matches, even though the chain then fails. Measured (Coraza
+# v3.7.0 + CRS 4.25.0, in-process and live): every *.php / *.css / ...
+# request ran with the engine On, so the whole CRS anomaly score (949110) and
+# 200002 enforced there -- in detect_only mode too, since SecRuleEngine
+# DetectionOnly cannot override a runtime ctl.
+#
+# Now: 920440 itself is untouched (it still sets tx.extension in its first
+# link and adds the anomaly score in its second), and this separate phase-1
+# rule, which runs after it, flips the engine only when the extension really
+# is in tx.restricted_extensions -- the original intent. Removing 920440
+# (break-glass, a per-host exception) also removes tx.extension, so this rule
+# goes quiet with it. Id 992920440 is outside WHP's generated range
+# (990000000-991999999).
+SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \
+    "id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"
 
 # ---------------------------------------------------------------------------
 # 930130 — Restricted File Access Attempt

haproxy_gunicorn_worker.py

@@ -0,0 +1,502 @@
+"""gunicorn gthread worker that cannot be wedged by silent or slow clients.
+
+Incident 2026-10-09 (sdbees), and very likely the 2026-07-07 whp01 one too
+-------------------------------------------------------------------------
+Both :8000 API workers had 7 of 8 pool threads parked in the kernel in
+``tcp_recvmsg -> sk_wait_data``: a blocking socket read with no timeout.
+/health and /api/blocked-ips hung for 40s+, the container stayed "healthy".
+
+haproxy_manager.py makes NO in-process outbound network reads (no requests,
+urllib, http.client, raw socket or ssl client anywhere; every ACME/HAProxy
+interaction is a subprocess, already bounded since 2026.07.3). The reads that
+blocked were gunicorn's own reads of the CLIENT socket:
+
+    gunicorn/workers/gthread.py  TConn.init():  self.sock.setblocking(True)
+
+Once the main loop sees a connection become readable it hands it to a pool
+thread and turns the socket fully blocking -- timeout None -- before parsing
+the request. Any peer that sends a few bytes and then waits (a TLS ClientHello
+sent to this plain-HTTP port, a port scanner, a half-written request) pins
+that pool thread forever. :8000 was published on 0.0.0.0 by the deploy
+(`-p 8000:8000`, which bypasses firewalld) and internet scanners were measured
+connecting to it. 4 such connections exhaust a worker's pool.
+
+gunicorn's --timeout does not catch this: for gthread it only checks that the
+worker's MAIN thread keeps heart-beating, and the main thread keeps polling
+happily while every pool thread is stuck. --max-requests does not help either:
+a worker whose threads never finish a request never reaches the limit.
+socket.setdefaulttimeout() does not help: setblocking(True) explicitly resets
+the timeout to None, overriding any default.
+
+Stock gthread has two more exhaustion paths on the same port (review F1/F2):
+
+* A connection that NEVER sends a byte never becomes readable, so it is never
+  handed to a thread and never gets any deadline -- but it holds one of the
+  worker's ``worker_connections`` (1000) slots. At 1000 the run loop stops
+  polling altogether and only waits on futures; none exist, so even the
+  attacker's FINs are never seen and the worker is wedged PERMANENTLY, still
+  heart-beating, with no request in flight for a watchdog to notice.
+* Slowloris: one header byte every <timeout seconds never trips a per-recv
+  timeout, so a socket timeout alone does not bound header-read time.
+
+What this worker changes
+------------------------
+1. CLIENT_TIMEOUT (HAPROXY_MGR_CLIENT_TIMEOUT, default 20s): every client
+   socket gets an explicit socket timeout at the exact point gthread makes it
+   blocking (re-applied on every keep-alive reuse). Python applies it to each
+   recv() call and to each WHOLE sendall() call -- gunicorn writes a response
+   chunk with one sendall, so any single chunk that takes longer than this to
+   transmit is cut off. Responses here are KB-sized JSON/PEMs to localhost, so
+   that never happens in practice. It does NOT bound how long the app takes to
+   handle a request (a long certbot run is a subprocess with its own timeout
+   and touches the client socket only once it is done).
+
+2. FIRST_BYTE_TIMEOUT (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, default 10s): an
+   accepted connection that has not become readable within this is closed by
+   the main loop, freeing its slot (review F1).
+
+3. HEADER_DEADLINE (HAPROXY_MGR_HEADER_DEADLINE, default 10s): total time a
+   client may take to deliver the request line + headers, counted from when
+   the connection was ENQUEUED for a thread, with a floor of
+   HAPROXY_MGR_HEADER_PICKUP_GRACE (1s) after a thread picks it up:
+   deadline = max(enqueue + 10s, pickup + 1s). Disarmed in handle_request()
+   (which gunicorn calls only once the parser has returned a complete
+   request). A 1s monitor loop shuts down (SHUT_RDWR, not close -- no fd-reuse
+   race with the blocked recv) the socket of any connection past it; the
+   blocked recv returns EOF and the thread is freed (review F2). A legit
+   request queued behind busy threads (certbot runs) already has its headers
+   in the kernel buffer and parses in milliseconds inside the 1s grace; a
+   slow client that sat in the queue gets only the grace, so reconnecting
+   drippers cannot starve the pool by queueing (re-review). Bodies are only
+   read by routes behind require_api_key, so the unauthenticated surface is
+   exactly this phase.
+
+4. Wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS, default 3600s; 0 disables). If
+   EVERY pool thread has been inside the same request for longer than that the
+   worker logs every thread's Python stack to the error log and exits; the
+   gunicorn master respawns it in about a second, without touching HAProxy or
+   the edge. 3600s because a multi-domain /api/certificates/request runs one
+   certbot (up to 180s) per domain; with (1)-(3) closing the known client-side
+   sources, this is only a backstop for blocking sources not yet found.
+
+5. Peer filter (re-review R1). haproxy-manager shares the client-net bridge
+   with every tenant container, so tenant PHP could reach haproxy-manager:8000
+   directly; the 127.0.0.1 publish only removes the internet path. On the
+   ports in HAPROXY_MGR_PEER_FILTER_PORTS (default "8000" -- NOT the :8080
+   default-page server, which HAProxy feeds over loopback) accept() closes
+   any connection whose peer is not in HAPROXY_MGR_API_ALLOWED_PEERS (default
+   "127.0.0.1,::1,gateway"; IPs/CIDRs; "gateway" = this container's default
+   gateway, which is where host callers arrive from through docker-proxy).
+   Refused before a thread or connection slot is used; logged at most once a
+   minute as a per-peer summary. Effective only once the edge publishes
+   127.0.0.1:8000 -- with a 0.0.0.0 publish a tenant can hairpin through the
+   host and arrive from the gateway too.
+
+6. pool_status() exposes busy/total threads and the oldest in-flight age so
+   /health can report saturation.
+
+This overrides gthread internals (enqueue_req, accept,
+on_client_socket_readable, murder_keepalived, handle, handle_request) as they
+are in gunicorn 23.0.0, which requirements.txt pins. A warning is logged if a
+different version is loaded; scripts/test-gunicorn-socket-timeouts.py runs
+real gunicorn in the Docker build gate and fails on behavioural drift.
+"""
+
+import errno
+import ipaddress
+import logging
+import os
+import selectors
+import socket
+import sys
+import threading
+import time
+import traceback
+from collections import deque
+from functools import partial
+
+import gunicorn
+from gunicorn.workers.gthread import TConn, ThreadWorker
+
+
+def _env_float(name, default):
+    try:
+        return float(os.environ.get(name, default))
+    except (TypeError, ValueError):
+        return float(default)
+
+
+CLIENT_TIMEOUT = _env_float('HAPROXY_MGR_CLIENT_TIMEOUT', 20)
+FIRST_BYTE_TIMEOUT = _env_float('HAPROXY_MGR_FIRST_BYTE_TIMEOUT', 10)
+HEADER_DEADLINE = _env_float('HAPROXY_MGR_HEADER_DEADLINE', 10)
+WEDGE_SECONDS = _env_float('HAPROXY_MGR_WEDGE_SECONDS', 3600)
+WATCHDOG_INTERVAL = _env_float('HAPROXY_MGR_WATCHDOG_INTERVAL', 15)
+MONITOR_TICK = 1.0
+# Once a thread picks a queued connection up it gets at least this long to
+# finish its headers, even if its enqueue-based deadline has already passed.
+HEADER_PICKUP_GRACE = _env_float('HAPROXY_MGR_HEADER_PICKUP_GRACE', 1)
+
+# Peer filter (review R1). Listener ports it applies to (comma-separated;
+# empty disables) and the peers allowed through. "gateway" means this
+# container's default gateway on its bridge network -- the address host-side
+# callers arrive from via the 127.0.0.1:8000 publish (docker-proxy) and via a
+# direct host->container-IP connect. Tenant containers on the same bridge
+# (client-net) arrive from their OWN addresses and are refused.
+PEER_FILTER_PORTS = os.environ.get('HAPROXY_MGR_PEER_FILTER_PORTS', '8000')
+ALLOWED_PEERS_SPEC = os.environ.get('HAPROXY_MGR_API_ALLOWED_PEERS', '127.0.0.1,::1,gateway')
+PEER_REJECT_LOG_INTERVAL = 60.0
+
+
+def _default_gateways(route_file='/proc/net/route'):
+    """IPv4 default gateway(s) from the kernel routing table."""
+    gws = []
+    try:
+        with open(route_file) as fh:
+            next(fh, None)
+            for line in fh:
+                f = line.split()
+                if len(f) >= 4 and f[1] == '00000000' and int(f[3], 16) & 0x2:
+                    gws.append(str(ipaddress.IPv4Address(int(f[2], 16).to_bytes(4, 'little'))))
+    except (OSError, ValueError):
+        pass
+    return gws
+
+
+def parse_allowed_peers(spec, gateways=None):
+    """Return (networks, problems). Tokens: IPs, CIDRs, or the word
+    'gateway'. Unparseable tokens are reported, never silently widened."""
+    nets, problems = [], []
+    for tok in (t.strip() for t in (spec or '').split(',')):
+        if not tok:
+            continue
+        if tok.lower() == 'gateway':
+            gws = _default_gateways() if gateways is None else gateways
+            if not gws:
+                problems.append('gateway: no IPv4 default route found')
+            for g in gws:
+                nets.append(ipaddress.ip_network(g + '/32'))
+            continue
+        try:
+            nets.append(ipaddress.ip_network(tok, strict=False))
+        except ValueError:
+            problems.append('unparseable entry %r' % tok)
+    return nets, problems
+
+
+def peer_allowed(addr, nets):
+    try:
+        ip = ipaddress.ip_address(addr.split('%', 1)[0])
+    except (ValueError, AttributeError):
+        return False
+    if ip.version == 6 and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    return any(ip.version == n.version and ip in n for n in nets)
+
+TESTED_GUNICORN_VERSION = '23.0.0'
+
+# Exit status for a watchdog-killed worker. Must NOT be 3 (WORKER_BOOT_ERROR)
+# or 4 (APP_LOAD_ERROR): on those the gunicorn master halts the whole server
+# instead of respawning the worker.
+WEDGED_EXIT_CODE = 1
+
+_current_worker = None
+
+
+def pool_status():
+    """Busy/total threads of the worker serving the caller, or None when the
+    app is not running under this worker class (e.g. tests, dev server)."""
+    w = _current_worker
+    if w is None:
+        return None
+    return w.pool_status()
+
+
+class _QuietClientTimeout(logging.Filter):
+    """gthread logs any OSError without an errno it recognises as
+    log.exception("Socket error processing request.") -- a full traceback.
+    A client timeout is now an expected, bounded event (scanners hit this port
+    constantly), so reduce it to one line instead of flooding the error log."""
+
+    def filter(self, record):
+        exc = record.exc_info[1] if record.exc_info else None
+        if isinstance(exc, TimeoutError) and \
+                str(record.msg).startswith('Socket error processing request'):
+            record.msg = ('Closed client connection idle for more than %ss '
+                          '(HAPROXY_MGR_CLIENT_TIMEOUT)' % CLIENT_TIMEOUT)
+            record.args = ()
+            record.exc_info = None
+            record.exc_text = None
+            record.levelno = logging.INFO
+            record.levelname = 'INFO'
+        return True
+
+
+class _InFlight:
+    __slots__ = ('conn', 'start', 'header_deadline')
+
+    def __init__(self, conn, start, header_deadline):
+        self.conn = conn
+        self.start = start
+        self.header_deadline = header_deadline
+
+
+class TimeoutThreadWorker(ThreadWorker):
+    """gthread worker with client-socket timeouts, a first-byte deadline, a
+    total header deadline and a wedge watchdog. See module docstring."""
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._inflight = {}          # thread ident -> _InFlight
+        self._inflight_lock = threading.Lock()
+        self._pending = deque()      # accepted, not yet readable (FIFO by deadline)
+        self._filter_ports = set()
+        for tok in PEER_FILTER_PORTS.split(','):
+            tok = tok.strip()
+            if tok.isdigit():
+                self._filter_ports.add(int(tok))
+        self._allowed_nets = []
+        self._rejected = {}          # peer ip -> count since last log line
+        self._rejected_lock = threading.Lock()
+        self._last_reject_log = 0.0
+
+    # -- (1) client socket timeout ------------------------------------------
+    def enqueue_req(self, conn):
+        # Same three steps as ThreadWorker.enqueue_req, with the timeout set
+        # right after conn.init() -- init() is what calls setblocking(True)
+        # (== settimeout(None)), and it runs again on every keep-alive reuse,
+        # so the timeout must be re-applied here each time, not at accept().
+        conn.init()
+        if CLIENT_TIMEOUT > 0:
+            conn.sock.settimeout(CLIENT_TIMEOUT)
+        conn.whp_enqueued = time.monotonic()   # header deadline counts from here
+        fs = self.tpool.submit(self.handle, conn)
+        self._wrap_future(fs, conn)
+
+    # -- (2) first-byte deadline for never-readable connections ---------------
+    def accept(self, server, listener):
+        # ThreadWorker.accept (23.0.0) plus: a deadline, and the conn is
+        # tracked in self._pending so murder_keepalived() can reap it.
+        try:
+            sock, client = listener.accept()
+            if self._filter_ports and self._server_port(server) in self._filter_ports \
+                    and not peer_allowed(client[0] if isinstance(client, tuple) else '',
+                                         self._allowed_nets):
+                # Refused before it costs a TConn, a connection slot, a poller
+                # registration or a thread.
+                sock.close()
+                self._note_rejected(client[0] if isinstance(client, tuple) else str(client))
+                return
+            conn = TConn(self.cfg, sock, client, server)
+            conn.timeout = time.time() + FIRST_BYTE_TIMEOUT
+
+            self.nr_conns += 1
+            with self._lock:
+                self._pending.append(conn)
+                self.poller.register(conn.sock, selectors.EVENT_READ,
+                                     partial(self.on_client_socket_readable, conn))
+        except OSError as e:
+            if e.errno not in (errno.EAGAIN, errno.ECONNABORTED,
+                               errno.EWOULDBLOCK):
+                raise
+
+    @staticmethod
+    def _server_port(server):
+        try:
+            return int(server[1])
+        except (TypeError, IndexError, ValueError):
+            return None
+
+    def _note_rejected(self, peer):
+        with self._rejected_lock:
+            self._rejected[peer] = self._rejected.get(peer, 0) + 1
+        if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
+            self._flush_rejected()
+
+    def _flush_rejected(self):
+        with self._rejected_lock:
+            if not self._rejected:
+                return
+            items, self._rejected = self._rejected, {}
+            self._last_reject_log = time.monotonic()
+        total = sum(items.values())
+        top = sorted(items.items(), key=lambda kv: -kv[1])[:5]
+        self.log.warning(
+            'Refused %d connection(s) to the management API from %d non-allowed peer(s) '
+            '(HAPROXY_MGR_API_ALLOWED_PEERS=%s): %s%s', total, len(items), ALLOWED_PEERS_SPEC,
+            ', '.join('%s x%d' % kv for kv in top), ' ...' if len(items) > 5 else '')
+
+    def on_client_socket_readable(self, conn, client):
+        # ThreadWorker version only knows about self._keep; a first request
+        # comes off self._pending instead.
+        with self._lock:
+            self.poller.unregister(client)
+            queue = self._keep if conn.initialized else self._pending
+            try:
+                queue.remove(conn)
+            except ValueError:
+                # already reaped (race with murder_keepalived)
+                return
+        self.enqueue_req(conn)
+
+    def murder_keepalived(self):
+        # Runs on every pass of ThreadWorker.run() -- including the branch
+        # taken when nr_conns >= worker_connections, where the poller is not
+        # serviced. Reaping here is what breaks that permanent deadlock.
+        super().murder_keepalived()
+        if FIRST_BYTE_TIMEOUT <= 0:
+            return
+        now = time.time()
+        while True:
+            with self._lock:
+                if not self._pending or self._pending[0].timeout > now:
+                    break
+                conn = self._pending.popleft()
+                try:
+                    self.poller.unregister(conn.sock)
+                except (OSError, KeyError, ValueError):
+                    pass
+            self.nr_conns -= 1
+            conn.close()
+
+    # -- (3) header deadline + busy tracking ----------------------------------
+    def handle(self, conn):
+        ident = threading.get_ident()
+        now = time.monotonic()
+        deadline = None
+        if HEADER_DEADLINE > 0:
+            # max(enqueue + D, pickup + grace): a legit request that waited in
+            # the queue behind busy threads already has its headers in the
+            # kernel buffer and parses in milliseconds; a slow client that
+            # waited gets only the grace, not a fresh full deadline (review:
+            # 40 reconnecting drippers otherwise starved legit requests).
+            enq = getattr(conn, 'whp_enqueued', now)
+            deadline = max(enq + HEADER_DEADLINE, now + HEADER_PICKUP_GRACE)
+        entry = _InFlight(conn, now, deadline)
+        with self._inflight_lock:
+            self._inflight[ident] = entry
+        try:
+            return super().handle(conn)
+        finally:
+            with self._inflight_lock:
+                self._inflight.pop(ident, None)
+
+    def handle_request(self, req, conn):
+        # The parser has produced a complete request line + headers.
+        entry = self._inflight.get(threading.get_ident())
+        if entry is not None:
+            entry.header_deadline = None
+        return super().handle_request(req, conn)
+
+    def _sweep_header_deadlines(self):
+        now = time.monotonic()
+        with self._inflight_lock:
+            late = [e for e in self._inflight.values()
+                    if e.header_deadline is not None and e.header_deadline < now]
+        for e in late:
+            e.header_deadline = None   # one shutdown per connection
+            try:
+                # shutdown, not close: wakes the blocked recv() with EOF
+                # without releasing the fd while another thread is in it.
+                e.conn.sock.shutdown(socket.SHUT_RDWR)
+            except OSError:
+                pass
+        if late:
+            self.log.info('Cut %d client connection(s) still sending request headers '
+                          'after %ss (HAPROXY_MGR_HEADER_DEADLINE)', len(late), HEADER_DEADLINE)
+
+    def pool_status(self):
+        now = time.monotonic()
+        with self._inflight_lock:
+            starts = [e.start for e in self._inflight.values()]
+        return {
+            'pid': os.getpid(),
+            'threads': self.cfg.threads,
+            'busy': len(starts),
+            'oldest_busy_seconds': round(now - min(starts), 1) if starts else 0.0,
+            'client_timeout_seconds': CLIENT_TIMEOUT,
+        }
+
+    # -- (4) watchdog ---------------------------------------------------------
+    def _wedged_for(self):
+        """Seconds the pool has been completely wedged (every thread busy in
+        its current request), measured from the YOUNGEST in-flight request.
+        0 when at least one thread is free."""
+        with self._inflight_lock:
+            starts = [e.start for e in self._inflight.values()]
+        if len(starts) < self.cfg.threads:
+            return 0.0
+        return time.monotonic() - max(starts)
+
+    def _dump_stacks(self):
+        frames = sys._current_frames()
+        names = {t.ident: t.name for t in threading.enumerate()}
+        out = []
+        for ident, frame in frames.items():
+            out.append('--- thread %s (%s)\n%s' % (
+                ident, names.get(ident, '?'), ''.join(traceback.format_stack(frame))))
+        return '\n'.join(out)
+
+    def _check_wedged(self):
+        wedged = self._wedged_for()
+        if WEDGE_SECONDS > 0 and wedged > WEDGE_SECONDS:
+            try:
+                self.log.critical(
+                    'WEDGED WORKER (pid %s): all %s threads busy in the same request '
+                    'for %.0fs > HAPROXY_MGR_WEDGE_SECONDS=%s. Exiting so the master '
+                    'respawns it. Thread stacks:\n%s',
+                    os.getpid(), self.cfg.threads, wedged, WEDGE_SECONDS,
+                    self._dump_stacks())
+                for h in logging.getLogger('gunicorn.error').handlers:
+                    try:
+                        h.flush()
+                    except Exception:
+                        pass
+            finally:
+                # os._exit, not SIGTERM: a graceful exit joins the (stuck,
+                # non-daemon) pool threads at interpreter shutdown and would
+                # hang until the master's --timeout escalates to SIGKILL.
+                os._exit(WEDGED_EXIT_CODE)
+
+    def _monitor(self):
+        """One daemon thread: header-deadline sweep every second, wedge check
+        every WATCHDOG_INTERVAL. Never allowed to die silently."""
+        last_wedge_check = time.monotonic()
+        while self.alive:
+            time.sleep(MONITOR_TICK)
+            try:
+                if HEADER_DEADLINE > 0:
+                    self._sweep_header_deadlines()
+                if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
+                    self._flush_rejected()
+                if time.monotonic() - last_wedge_check >= WATCHDOG_INTERVAL:
+                    last_wedge_check = time.monotonic()
+                    self._check_wedged()
+            except Exception:
+                try:
+                    self.log.exception('haproxy_gunicorn_worker monitor error (continuing)')
+                except Exception:
+                    pass
+
+    def init_process(self):
+        global _current_worker
+        _current_worker = self
+        try:
+            self.log.error_log.addFilter(_QuietClientTimeout())
+        except AttributeError:
+            pass
+        if gunicorn.__version__ != TESTED_GUNICORN_VERSION:
+            self.log.warning(
+                'haproxy_gunicorn_worker overrides gthread internals as of gunicorn %s; '
+                'running %s -- re-run scripts/test-gunicorn-socket-timeouts.py',
+                TESTED_GUNICORN_VERSION, gunicorn.__version__)
+        if self._filter_ports:
+            self._allowed_nets, problems = parse_allowed_peers(ALLOWED_PEERS_SPEC)
+            for prob in problems:
+                self.log.error('HAPROXY_MGR_API_ALLOWED_PEERS: %s -- that entry allows nothing '
+                               '(fail closed)', prob)
+            self.log.info('Management API peer filter on port(s) %s: allowing %s',
+                          ','.join(str(p) for p in sorted(self._filter_ports)),
+                          ', '.join(str(n) for n in self._allowed_nets) or 'NOTHING')
+        threading.Thread(target=self._monitor, name='worker-monitor',
+                         daemon=True).start()
+        super().init_process()   # runs the event loop; does not return

haproxy_tarpit_config.txt

@@ -1,3 +1,37 @@
+# =============================================================================
+# NOT IMPLEMENTED. THIS FILE IS A DESIGN SKETCH THAT WAS NEVER SHIPPED.
+# =============================================================================
+#
+# Nothing here is deployed, has ever been deployed, or is rendered into
+# haproxy.cfg. The real edge config is generated from templates/*.tpl. Compare:
+#
+#   THIS FILE proposes:  store gpc0,gpc1,gpc2,http_err_rate(30s),...
+#                        plus sc-inc-gpc0/1/2 scan-escalation rules
+#   templates/hap_listener.tpl ACTUALLY has:
+#                        store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(30s)
+#
+# There is no gpc0, no gpc1, no gpc2, and no scan-escalation state anywhere on
+# the edge, and never has been.
+#
+# WHY THE BANNER. This file was mistaken for the shipped config. Two consumers
+# -- /api/security/stats in haproxy_manager.py and scripts/show-tarpit-ips.sh --
+# were written to parse gpc0/gpc1 out of `show table web`, defaulting missing
+# fields to 0. The result was a "Scan Count" column and BLOCKED/TARPITTED
+# statuses that were pure fabrication, presented to an operator as fact, for
+# the entire life of both tools. Fixed 2026-08-22; scripts/test-stick-table-
+# contract.py now fails if any consumer's field expectations and the templates'
+# `store` clauses ever drift apart again.
+#
+# IF YOU WANT TO REVIVE ANY OF THIS: the counters must be added to a template
+# `store` clause and to STICK_TABLE_FIELD_CONTRACT in haproxy_manager.py first.
+# Adding them to a consumer alone produces confident zeros, not data. Note also
+# that sc0/sc1/sc2 are all in use and HAProxy's tune.stick-counters defaults to
+# 3, and that since 2026.08.8 the edge has real per-request access logging on
+# the host at /var/log/haproxy.log -- which records what was actually denied,
+# tarpitted and rate-limited, with request references. That log is a better
+# source for most of what this sketch was reaching for.
+# =============================================================================
+
 global
     daemon
     log stdout local0 info

scripts/cert-publish-lib.sh

@@ -0,0 +1,452 @@
+# shellcheck shell=bash
+# cert-publish-lib.sh - safe publication of HAProxy certificate bundles.
+#
+# This file is SOURCED, never executed (hence no shebang / no exec bit).
+#
+# Why this exists
+# ---------------
+# renew-certificates.sh and sync-certificates.sh used to publish a bundle with
+#
+#     cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"
+#
+# where $COMBINED_FILE is the LIVE pem HAProxy is serving right now. The shell
+# truncates the destination to zero bytes when it sets up the redirect, BEFORE
+# cat ever runs, so any failure after that point (unreadable source, ENOSPC,
+# container killed mid-write) leaves a zero-length or key-less pem in place.
+# Checking cat's exit status does not help: the damage is already done.
+#
+# That matters more here than for an ordinary file because HAProxy loads
+# $SSL_CERTS_DIR as a DIRECTORY:
+#
+#     bind 0.0.0.0:443 ssl crt /etc/haproxy/certs
+#
+# It tries to load *every* file in that directory, and one unloadable file
+# fails the whole bind - i.e. HTTPS goes down for every customer on the host.
+#
+# Two consequences drive the design below:
+#   1. Assemble somewhere else and rename into place, so the live pem is either
+#      the old bundle or the new one and never a half-written one.
+#   2. NEVER create a temp file, .tmp, .backup or any other non-final file
+#      inside $SSL_CERTS_DIR. Staging and backups live in SIBLING directories.
+#
+# Directory layout (kept identical to the Python half in haproxy_manager.py):
+#   staging: $(dirname $SSL_CERTS_DIR)/cert-staging   [$CERT_STAGING_DIR]
+#   backups: $(dirname $SSL_CERTS_DIR)/cert-backups   [$CERT_BACKUP_DIR]
+# Both siblings of the certs dir, so they are on the same filesystem and the
+# final mv is a rename(2) - atomic. There is deliberately no "just write it
+# directly into the certs dir" fallback path.
+#
+# The same-filesystem property is CHECKED, not assumed (see cert_publish step
+# (c)). An earlier revision of this comment claimed a cross-device mv would
+# "fail loudly and leave the live pem alone". It does not: GNU mv falls back to
+# copy-then-unlink across filesystems, so it OPENS THE DESTINATION FOR WRITING
+# and only then discovers it cannot finish - e.g. with a full destination
+# filesystem the live pem is already overwritten when mv reports failure. That
+# is precisely the truncation this library exists to prevent, so the device
+# numbers of the staging dir and the certs dir are compared with stat(1) before
+# anything is written, and a mismatch aborts the publish. Both directories are
+# env-overridable ($CERT_STAGING_DIR / $SSL_CERTS_DIR), so "they are siblings"
+# is not something the code can take on faith. This mirrors the explicit
+# st_dev check in publish_pem_bundle() on the Python side.
+
+# Logging: the callers define their own log_info/log_error. Only provide
+# fallbacks so this library is usable standalone (e.g. from a test or a shell).
+declare -F log_info >/dev/null || log_info() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [INFO] $*"
+}
+declare -F log_error >/dev/null || log_error() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" >&2
+}
+# log_warn is not part of the callers' vocabulary; route it through log_info
+# with a loud prefix so it lands in the main log without tripping the
+# error-log monitors (scripts/monitor-errors.sh) for non-fatal conditions.
+declare -F log_warn >/dev/null || log_warn() {
+    log_info "WARNING: $*"
+}
+
+cert_staging_dir() {
+    if [ -n "${CERT_STAGING_DIR:-}" ]; then
+        echo "$CERT_STAGING_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-staging"
+    fi
+}
+
+cert_backup_dir() {
+    if [ -n "${CERT_BACKUP_DIR:-}" ]; then
+        echo "$CERT_BACKUP_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-backups"
+    fi
+}
+
+# cert_bundle_valid FILE
+#
+# Returns 0 if FILE is publishable as an HAProxy pem bundle.
+#
+# Layer 1 (structure, pure shell/grep): covers the truncation / partial-write /
+#         key-less failure modes this library exists to prevent.
+# Layer 2 (cryptographic pairing via the openssl CLI): covers what layer 1
+#         cannot see. Structural checks are weak on their own - a bundle of
+#         EMPTY pem blocks ("-----BEGIN CERTIFICATE-----" immediately followed
+#         by "-----END CERTIFICATE-----") satisfies every grep below and is
+#         caught only by openssl.
+#
+# BOTH LAYERS ARE MANDATORY. An absent openssl binary is a hard failure, not a
+# skip.
+#
+# The previous revision made layer 2 best-effort and justified it with "the
+# container image installs haproxy, certbot and socat, but not necessarily the
+# openssl CLI". That premise is false. openssl 3.x is present in the image: it
+# is a dependency of ca-certificates, which certbot needs, and
+# generate_self_signed_cert() in haproxy_manager.py shells out to `openssl req`
+# with check=True during first-run setup, so a container that reached the point
+# of publishing a bundle has always had it. The "unavailable" branch therefore
+# never fired in production, which means the fail-open was safe only by
+# accident - and a comment that justifies a decision on a false premise is
+# worse than no comment, because the next person extends the reasoning.
+#
+# Making it mandatory does mean a hypothetical image without openssl stops
+# publishing renewals. That is the right trade: it fails immediately and
+# loudly, into the monitored error log, on the first renewal run, whereas
+# publishing an unpaired or empty-block bundle takes the whole :443 bind (i.e.
+# every site on the host) down at the next reload. There is deliberately no
+# python `cryptography` fallback: the app runs on /usr/local/bin/python3 (the
+# base image's 3.12), where cryptography is NOT importable - it is installed
+# for Debian's /usr/bin/python3 as a certbot dependency. Coding to a hardcoded
+# /usr/bin/python3 would just be a second unverified premise.
+cert_bundle_valid() {
+    local file="$1"
+
+    if [ -z "$file" ]; then
+        log_error "cert_bundle_valid: no file given"
+        return 1
+    fi
+    if [ ! -f "$file" ]; then
+        log_error "Certificate bundle $file does not exist (or is not a regular file)"
+        return 1
+    fi
+
+    # openssl is checked BEFORE any content check so a broken image is reported
+    # as a broken image rather than as a bad certificate.
+    if ! command -v openssl >/dev/null 2>&1; then
+        log_error "openssl binary not found - REFUSING to publish $file." \
+                  "The cert/key pairing check (openssl x509 -pubkey vs openssl pkey -pubout)" \
+                  "is mandatory; structural checks alone cannot tell a real bundle from" \
+                  "empty pem blocks. Install openssl in this image."
+        return 1
+    fi
+
+    # Read the file ONCE and run every check against that snapshot.
+    #
+    # This used to open $file six times (two [ ] tests, three greps, two
+    # openssl invocations). cert_bundle_valid() is called on the LIVE pem in
+    # cert_publish() step (d), where a concurrent publisher can replace it
+    # between two of those opens - each open then sees a different file. The
+    # observable symptom was a spurious "private key does not match the
+    # certificate" ERROR in the monitored error log for a pair that was fine:
+    # openssl x509 read the old bundle and openssl pkey the new one.
+    local content
+    if ! content="$(cat -- "$file" 2>/dev/null)"; then
+        log_error "Certificate bundle $file could not be read"
+        return 1
+    fi
+    if [ -z "$content" ]; then
+        log_error "Certificate bundle $file is empty"
+        return 1
+    fi
+
+    # --- layer 1: structure -------------------------------------------------
+    if ! grep -qF -- '-----BEGIN CERTIFICATE-----' <<< "$content"; then
+        log_error "Certificate bundle $file contains no certificate block"
+        return 1
+    fi
+    if ! grep -qF -- '-----END CERTIFICATE-----' <<< "$content"; then
+        log_error "Certificate bundle $file has an unterminated certificate block (truncated?)"
+        return 1
+    fi
+
+    local key_begin key_end
+    key_begin="$(grep -m1 -oE -- '-----BEGIN (RSA |EC )?PRIVATE KEY-----' <<< "$content")"
+    if [ -z "$key_begin" ]; then
+        log_error "Certificate bundle $file contains no private key block"
+        return 1
+    fi
+    key_end="${key_begin/BEGIN/END}"
+    if ! grep -qF -- "$key_end" <<< "$content"; then
+        log_error "Certificate bundle $file has an unterminated private key block (truncated?)"
+        return 1
+    fi
+
+    # --- layer 2: cert/key pairing ------------------------------------------
+    # Fed from the same snapshot on stdin (openssl reads stdin when -in is
+    # omitted) rather than re-opening $file, so layer 2 judges exactly the
+    # bytes layer 1 judged. -passin pass: means an encrypted key fails fast
+    # instead of prompting - a passphrase prompt in a cron job is a hang, not
+    # an error.
+    local cert_pub key_pub
+    if ! cert_pub="$(openssl x509 -noout -pubkey 2>/dev/null <<< "$content")" \
+       || [ -z "$cert_pub" ]; then
+        log_error "Certificate bundle $file: openssl could not read the certificate"
+        return 1
+    fi
+    if ! key_pub="$(openssl pkey -pubout -passin pass: 2>/dev/null <<< "$content")" \
+       || [ -z "$key_pub" ]; then
+        log_error "Certificate bundle $file: openssl could not read the private key"
+        return 1
+    fi
+    if [ "$cert_pub" != "$key_pub" ]; then
+        log_error "Certificate bundle $file: private key does not match the certificate"
+        return 1
+    fi
+
+    return 0
+}
+
+# cert_is_expired CERT_FILE
+#
+# Returns 0 only when openssl positively reports that the first certificate in
+# CERT_FILE has expired. Anything else (not expired, unreadable, no openssl)
+# returns 1; unreadable material is rejected with a better message by
+# cert_bundle_valid().
+#
+# Why: HAProxy loads the crt directory wholesale and serves the FIRST file that
+# claims an SNI name. An expired bundle in there is never useful and can only
+# shadow a valid one - on 2026-10-07 renew-certificates.sh kept re-publishing
+# the expired brain-jar.com lineage every 12 hours, and HAProxy served it for
+# brainjar.net instead of the valid brainjar.net bundle next to it.
+cert_is_expired() {
+    local file="$1" out
+    [ -f "$file" ] || return 1
+    command -v openssl >/dev/null 2>&1 || return 1
+    out="$(openssl x509 -checkend 0 -noout -in "$file" 2>/dev/null </dev/null)"
+    [ $? -eq 1 ] && [[ "$out" == *"will expire"* ]]
+}
+
+# cert_not_after CERT_FILE - the notAfter date string, or empty.
+cert_not_after() {
+    openssl x509 -enddate -noout -in "$1" 2>/dev/null </dev/null | sed 's/^notAfter=//'
+}
+
+cert_retired_dir() {
+    if [ -n "${CERT_RETIRED_DIR:-}" ]; then
+        echo "$CERT_RETIRED_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-retired"
+    fi
+}
+
+# cert_retire_expired_pems
+#
+# Move every EXPIRED *.pem out of $SSL_CERTS_DIR into cert-retired/ (a sibling,
+# never inside the crt dir). Sets CERT_RETIRED_EXPIRED to the number moved.
+#
+# Not publishing expired lineages stops the re-publish loop, but a pem that was
+# published while valid and has expired since stays in the crt dir and keeps
+# shadowing - that was exactly brain-jar.com.pem (review M2). An expired
+# certificate is never a valid certificate for anyone, so it can always go: the
+# only-valid-cert safety rule cannot be violated by moving it. Two exceptions:
+#   * default_self_signed_cert.pem - HAProxy's fallback, regenerated separately;
+#   * the LAST .pem in the directory - a crt directory with no certificate at
+#     all fails the whole :443 bind, which is worse than serving an expired one.
+# Nothing is deleted; reverse with mv.
+cert_retire_expired_pems() {
+    local dir="${SSL_CERTS_DIR:-/etc/haproxy/certs}" retired f base dest remaining
+    CERT_RETIRED_EXPIRED=0
+    retired="$(cert_retired_dir)"
+    for f in "$dir"/*.pem; do
+        [ -f "$f" ] || continue
+        base="$(basename "$f")"
+        [ "$base" = "default_self_signed_cert.pem" ] && continue
+        cert_is_expired "$f" || continue
+        remaining="$(find "$dir" -maxdepth 1 -type f -name '*.pem' | wc -l)"
+        if [ "$remaining" -le 1 ]; then
+            log_error "Expired certificate $f is the only certificate left in $dir - NOT retiring it (an empty crt directory fails the :443 bind)"
+            continue
+        fi
+        if ! mkdir -p "$retired"; then
+            log_error "cannot create $retired - expired $f left in place"
+            continue
+        fi
+        dest="$retired/$base"
+        [ -e "$dest" ] && dest="$dest.$(date -u +%Y%m%dT%H%M%SZ)"
+        if mv -- "$f" "$dest"; then
+            log_error "Retired EXPIRED certificate $base (notAfter $(cert_not_after "$dest")) from $dir to $dest - it could only shadow valid certificates"
+            CERT_RETIRED_EXPIRED=$((CERT_RETIRED_EXPIRED + 1))
+        else
+            log_error "failed to move expired $f to $dest"
+        fi
+    done
+}
+
+# cert_publish CERT_FILE KEY_FILE DEST_FILE
+#
+# Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
+# partially written DEST_FILE to HAProxy. Returns 0 on success.
+#
+# On ANY failure DEST_FILE is left exactly as it was.
+cert_publish() {
+    if [ $# -ne 3 ]; then
+        log_error "cert_publish: expected 3 arguments (cert key dest), got $#"
+        return 1
+    fi
+
+    local cert_file="$1" key_file="$2" dest_file="$3"
+    local staging_dir backup_dir dest_dir tmp base
+
+    # (a) sources must exist and be non-empty before we touch anything.
+    if [ ! -s "$cert_file" ]; then
+        log_error "cert_publish: certificate $cert_file is missing or empty"
+        return 1
+    fi
+    if [ ! -s "$key_file" ]; then
+        log_error "cert_publish: private key $key_file is missing or empty"
+        return 1
+    fi
+    # (a2) never publish an expired certificate (see cert_is_expired).
+    if cert_is_expired "$cert_file"; then
+        log_error "cert_publish: certificate $cert_file expired ($(cert_not_after "$cert_file")) - refusing to publish $dest_file"
+        return 1
+    fi
+
+    # (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
+    #     scans wholesale.
+    staging_dir="$(cert_staging_dir)"
+    dest_dir="$(dirname "$dest_file")"
+    if ! mkdir -p "$staging_dir" || ! mkdir -p "$dest_dir"; then
+        log_error "cert_publish: cannot create staging directory $staging_dir or destination directory $dest_dir"
+        return 1
+    fi
+
+    # (c) the final swap is `mv`, which is only a rename(2) - and therefore only
+    #     atomic - within one filesystem. Across filesystems GNU mv copies:
+    #     it truncates and writes the DESTINATION, then unlinks the source, so a
+    #     failure part-way through (ENOSPC is the realistic one) leaves exactly
+    #     the half-written live pem this library exists to prevent. Both paths
+    #     are env-overridable, so check instead of assuming. Same check as the
+    #     st_dev comparison in publish_pem_bundle() on the Python side.
+    local staging_dev dest_dev
+    staging_dev="$(stat -Lc '%d' "$staging_dir" 2>/dev/null)"
+    dest_dev="$(stat -Lc '%d' "$dest_dir" 2>/dev/null)"
+    if [ -z "$staging_dev" ] || [ -z "$dest_dev" ]; then
+        log_error "cert_publish: cannot stat $staging_dir and/or $dest_dir - refusing to publish $dest_file"
+        return 1
+    fi
+    if [ "$staging_dev" != "$dest_dev" ]; then
+        log_error "cert_publish: staging dir $staging_dir and destination dir $dest_dir" \
+                  "are on different filesystems, so the bundle cannot be swapped in atomically." \
+                  "Refusing to publish $dest_file (live file left untouched);" \
+                  "point CERT_STAGING_DIR at a directory on the same filesystem as $dest_dir"
+        return 1
+    fi
+
+    # Sweep temps orphaned by a kill -9 / OOM in an earlier run. Two shapes,
+    # because two publishers share this directory: mktemp's six-X suffix from
+    # this library, and `<name>.<random>.tmp` from write_config_atomically() on
+    # the Python side (tempfile.mkstemp(prefix=name + '.', suffix='.tmp')).
+    # Matching only the mktemp shape - as this did - left every Python-side
+    # temp behind forever.
+    find "$staging_dir" -maxdepth 1 -type f \
+        \( -name '*.??????' -o -name '*.tmp' \) -mmin +1440 -delete 2>/dev/null
+
+    base="$(basename "$dest_file")"
+    tmp="$(mktemp "${staging_dir}/${base}.XXXXXX" 2>/dev/null)"
+    if [ -z "$tmp" ] || [ ! -f "$tmp" ]; then
+        log_error "cert_publish: cannot create a staging file in $staging_dir"
+        return 1
+    fi
+    # 0600 while the temp file holds a private key; the final mode is matched
+    # to the file being replaced just before the swap (see below).
+    chmod 600 "$tmp" 2>/dev/null
+
+    if ! cat "$cert_file" "$key_file" > "$tmp"; then
+        log_error "cert_publish: failed to assemble $cert_file + $key_file (live $dest_file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+    if [ ! -s "$tmp" ]; then
+        log_error "cert_publish: assembled bundle for $dest_file is empty (live file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    # (d) never promote something HAProxy would choke on.
+    if ! cert_bundle_valid "$tmp"; then
+        log_error "cert_publish: assembled bundle for $dest_file failed validation (live file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    # (e) back up the bundle we are about to replace - but only if it is itself
+    #     valid. Overwriting a good backup with garbage would turn "restore the
+    #     backup" into "restore a different broken file". Same semantics as
+    #     create_backup(require_valid=True) in haproxy_manager.py.
+    if [ -e "$dest_file" ]; then
+        backup_dir="$(cert_backup_dir)"
+        if cert_bundle_valid "$dest_file"; then
+            if mkdir -p "$backup_dir"; then
+                if ! cp -p "$dest_file" "${backup_dir}/${base}"; then
+                    log_warn "could not back up $dest_file to ${backup_dir}/${base}; publishing anyway"
+                fi
+            else
+                log_warn "could not create backup directory $backup_dir; publishing without a backup"
+            fi
+        else
+            log_warn "existing $dest_file is not a valid bundle - KEEPING the previous backup" \
+                     "in $backup_dir rather than overwriting it with an unusable one"
+        fi
+    fi
+
+    # Preserve the mode of the bundle being replaced (0644 by default, which is
+    # what `cat > file` produced under the standard umask). mktemp gives 0600,
+    # and mv carries the temp file's mode onto the destination, so without this
+    # every publish would silently tighten the live pem's permissions. Changing
+    # who can read these files is not something a write-safety fix should do as
+    # a side effect - and it must match write_config_atomically() on the Python
+    # side, which preserves the mode the same way.
+    #
+    # -L (follow symlinks) matters: stat without it reports the mode of the
+    # SYMLINK, which is 0777 on Linux and is not a permission at all. A live
+    # pem that is a symlink therefore produced a world-WRITABLE 0777 private
+    # key sitting in the crt directory. With -L we copy the mode of the file
+    # the link points at, which is the mode an operator actually chose.
+    local mode
+    mode="$(stat -Lc '%a' "$dest_file" 2>/dev/null)"
+    [ -n "$mode" ] || mode=644
+    chmod "$mode" "$tmp" 2>/dev/null
+
+    # (f) atomic swap. Same filesystem, verified in (c); if it still fails,
+    #     stop - do not fall back to writing into the certs dir.
+    if ! mv -f "$tmp" "$dest_file"; then
+        log_error "cert_publish: failed to move $tmp into place as $dest_file" \
+                  "(live file left untouched; NOT falling back to a direct write)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    return 0
+}
+
+# haproxy_config_ok
+#
+# Gate a reload on `haproxy -c`. Returns 0 if the config validates, or if we
+# cannot check (no haproxy binary) - a missing checker must not block a reload
+# that is otherwise needed, but a checker that says "no" always wins.
+haproxy_config_ok() {
+    local cfg="${HAPROXY_CONFIG:-/etc/haproxy/haproxy.cfg}"
+    local out rc
+
+    if ! command -v haproxy >/dev/null 2>&1; then
+        log_warn "haproxy binary not found - skipping 'haproxy -c' validation before reload"
+        return 0
+    fi
+
+    out="$(haproxy -c -f "$cfg" 2>&1 </dev/null)"
+    rc=$?
+    if [ $rc -eq 0 ]; then
+        return 0
+    fi
+
+    log_error "haproxy -c -f $cfg failed (exit $rc): $out"
+    return 1
+}

scripts/healthcheck.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Container liveness probe: BOTH planes must answer quickly.
+#
+#   1. Management API (:8000) -- GET /health with a short --max-time. On
+#      2026-10-09 (sdbees) the API workers' thread pools were wedged for long
+#      enough that /health hung 40s+, while the container reported "healthy"
+#      because the deployed --health-cmd only probed :80. A hung API is a dead
+#      control plane (every WHP site update times out), so it must turn the
+#      container unhealthy.
+#   2. HAProxy data plane (:80) -- any HTTP answer at all (the default page,
+#      a redirect, a 4xx/5xx) proves HAProxy accepted and answered. Only a
+#      connect/timeout failure fails it, which is why there is no -f here.
+#
+# 127.0.0.1, not localhost: the API binds IPv4 only, and resolving localhost
+# to ::1 first costs a failed connect on every probe.
+#
+# WHP's lib-edge-container.sh overrides the image HEALTHCHECK with its own
+# --health-cmd; point that at this script so the fleet gets the same probe.
+set -u
+API_TIMEOUT="${HAPROXY_MGR_HEALTH_API_TIMEOUT:-4}"
+EDGE_TIMEOUT="${HAPROXY_MGR_HEALTH_EDGE_TIMEOUT:-5}"
+
+if ! curl -sf -o /dev/null --max-time "$API_TIMEOUT" http://127.0.0.1:8000/health; then
+    echo "management API :8000 /health failed or took longer than ${API_TIMEOUT}s"
+    exit 1
+fi
+if ! curl -s -o /dev/null --max-time "$EDGE_TIMEOUT" http://127.0.0.1/; then
+    echo "HAProxy :80 did not answer within ${EDGE_TIMEOUT}s"
+    exit 1
+fi
+exit 0

scripts/monitor-attacks.sh

@@ -1,136 +1,148 @@
-#!/bin/bash
+#!/usr/bin/env bash
+#
+# monitor-attacks.sh — HAProxy edge activity monitor.
+#
+# Two sections, both fed from real data:
+#   1. Current per-IP rates, from the `web` stick table (delegated to
+#      show-edge-ip-rates.sh — there is exactly one stick-table parser).
+#   2. Recent enforcement events, from the HAProxy access log.
+#
+# HISTORY / WHY THIS IS SHORTER THAN IT USED TO BE
+#   The previous version printed a "Threat Intelligence Dashboard" with
+#   fourteen categories (auth_fail, authz_fail, scanner, sql_inj, traversal,
+#   wp_brute, admin_scan, shell_att, repeat_off, manual_bl, auto_bl,
+#   glitch_rate, ...) and a composite "threat score", all parsed out of
+#   gpc(0), gpc(1), gpc(3), gpc(12), gpc(13) and glitch_rate(300s). NONE of
+#   those fields exist: the `web` table stores only conn_cur, conn_rate,
+#   http_req_rate and http_err_rate. Every category was permanently 0 and the
+#   whole dashboard printed nothing while implying it was watching. All of it
+#   has been deleted rather than "fixed" — there was no data source to fix it
+#   against.
+#
+# Usage: monitor-attacks.sh [live]
+# Env:   LOG_FILE=<path>  access log to read (default /var/log/haproxy.log)
+#        LOG_LINES=<n>    how many trailing log lines to scan (default 500)
 
-# Real-time attack monitoring for HAProxy
-# Shows blocked requests and suspicious activity
+set -uo pipefail
 
-LOG_FILE="/var/log/haproxy.log"
-SOCKET="/tmp/haproxy-cli"
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+LOG_FILE="${LOG_FILE:-/var/log/haproxy.log}"
+LOG_LINES="${LOG_LINES:-500}"
 
-echo "==================================================="
-echo "HAProxy Security Monitor - Real-time Attack Detection"
-echo "==================================================="
-echo ""
-
-# Function to show current threats with HAProxy 3.0.11 metrics
-show_threats() {
-    echo "HAProxy 3.0.11 Threat Intelligence Dashboard:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 {
-            # Parse the stick table output for array-based GPC values
-            ip = $1
-            # Look for GPC array values in the data
-            auth_fail = 0
-            authz_fail = 0
-            rate_viol = 0
-            scanner = 0
-            sql_inj = 0
-            traversal = 0
-            wp_brute = 0
-            admin_scan = 0
-            shell_att = 0
-            repeat_off = 0
-            manual_bl = 0
-            auto_bl = 0
-            glitch_rate = 0
-            threat_score = 0
-
-            # Extract relevant metrics (simplified parsing)
-            if ($0 ~ /gpc\(0\)=([0-9]+)/) {
-                match($0, /gpc\(0\)=([0-9]+)/, arr); auth_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(1\)=([0-9]+)/) {
-                match($0, /gpc\(1\)=([0-9]+)/, arr); authz_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(3\)=([0-9]+)/) {
-                match($0, /gpc\(3\)=([0-9]+)/, arr); scanner = arr[1]
-            }
-            if ($0 ~ /gpc\(12\)=([0-9]+)/) {
-                match($0, /gpc\(12\)=([0-9]+)/, arr); repeat_off = arr[1]
-            }
-            if ($0 ~ /gpc\(13\)=([0-9]+)/) {
-                match($0, /gpc\(13\)=([0-9]+)/, arr); manual_bl = arr[1]
-            }
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr); glitch_rate = arr[1]
-            }
-
-            # Calculate composite threat score (simplified)
-            threat_score = auth_fail*10 + authz_fail*8 + scanner*12 + repeat_off*25 + manual_bl*100
-
-            # Only show IPs with significant threat indicators
-            if (auth_fail > 0 || authz_fail > 0 || scanner > 0 || repeat_off > 0 || manual_bl > 0 || glitch_rate > 0) {
-                threat_level = "LOW"
-                if (threat_score >= 100) threat_level = "CRITICAL"
-                else if (threat_score >= 50) threat_level = "HIGH"
-                else if (threat_score >= 20) threat_level = "MEDIUM"
-
-                printf "%-15s [%8s] Score:%-3d Auth:%-2d Authz:%-2d Scanner:%-1d Repeat:%-1d Glitch:%-2d\n",
-                       ip, threat_level, threat_score, auth_fail, authz_fail, scanner, repeat_off, glitch_rate
-            }
-        }' | head -15
-
-    echo ""
-    echo "Top HTTP/2 Protocol Violators:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 && $0 ~ /glitch/ {
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr)
-                if (arr[1] > 2) {
-                    printf "%-15s glitch_rate:%-3s\n", $1, arr[1]
-                }
-            }
-        }' | head -5
-    echo "---------------------------------------------------"
+# --- Section 1: current rates (real stick-table data) -----------------------
+show_rates() {
+    "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
 }
 
-# Function to show recent blocks
+# --- Section 2: recent enforcement events (real access-log data) ------------
 show_recent_blocks() {
-    echo "Recent Blocked Requests:"
-    tail -100 "$LOG_FILE" 2>/dev/null | \
-        grep -E "(bot_scanner|scan_admin|scan_shells|sql_injection|directory_traversal|rate_abuse|tarpit|denied|403)" | \
-        tail -10 | \
-        awk '{
-            if (match($0, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+/)) {
-                ip = substr($0, RSTART, RLENGTH)
-                gsub(/:.*/, "", ip)
-                reason = ""
-                if ($0 ~ /bot_scanner/) reason = "BOT_SCANNER"
-                else if ($0 ~ /scan_admin/) reason = "ADMIN_SCAN"
-                else if ($0 ~ /scan_shells/) reason = "SHELL_SCAN"
-                else if ($0 ~ /sql_injection/) reason = "SQL_INJECTION"
-                else if ($0 ~ /directory_traversal/) reason = "DIR_TRAVERSAL"
-                else if ($0 ~ /rate_abuse/) reason = "RATE_ABUSE"
-                else if ($0 ~ /tarpit/) reason = "TARPIT"
-                else if ($0 ~ /denied/) reason = "DENIED"
-                else if ($0 ~ /403/) reason = "BLOCKED"
-                printf "[%s] %-15s %s\n", strftime("%H:%M:%S"), ip, reason
+    echo "Recent enforcement events (last $LOG_LINES log lines):"
+    echo
+
+    if [ ! -f "$LOG_FILE" ] || [ ! -r "$LOG_FILE" ]; then
+        cat <<MSG
+  Access log not readable at: $LOG_FILE
+
+  This is expected INSIDE the haproxy-manager container: HAProxy logs to
+  syslog on the DOCKER HOST, and the file lives on the host, not in here.
+  Read it from the host instead:
+
+    grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+    grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limited / blocked
+    grep -a 'cip=<IP>'  /var/log/haproxy.log | tail -50     # one client IP
+    grep -a 'id=<uuid>' /var/log/haproxy.log                # one request reference
+                                                            # (the UUID on the block page)
+    tail -f /var/log/haproxy.log | grep -aE ' (PT|PR)--'    # live
+
+  Or point this script at a copy:  LOG_FILE=/path/to/haproxy.log $0
+MSG
+        echo
+        return 0
+    fi
+
+    printf "%-15s %-16s %-4s %-5s %-28s %s\n" "TIME" "CLIENT IP" "CODE" "TERM" "HOST" "REQUEST / REQUEST-ID"
+    printf "%s\n" "-----------------------------------------------------------------------------------------------------"
+
+    local found
+    found=$(tail -n "$LOG_LINES" "$LOG_FILE" 2>/dev/null | awk '
+    {
+        status = ""; term = ""; cip = ""; host = ""; id = ""; ts = ""; req = ""
+
+        # %tr is bracketed: [22/Aug/2026:10:11:12.345] -> keep HH:MM:SS
+        # No {n} interval expressions here: not every awk in a slim Debian
+        # image supports them. Spelled out instead.
+        if (match($0, /\[[0-9][0-9]\/[A-Za-z][A-Za-z][A-Za-z]\/[0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/)) {
+            ts = substr($0, RSTART + 13, 8)
+        }
+
+        # Anchor on the %TR/%Tw/%Tc/%Tr/%Ta timers block: %ST follows it, and
+        # the termination state (%tsc) is 4 fields further on (%B %CC %CS %tsc).
+        for (i = 1; i <= NF; i++) {
+            if ($i ~ /^[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+$/) {
+                status = $(i + 1)
+                term = $(i + 5)
+                break
             }
-        }'
-    echo ""
+        }
+
+        # Only enforcement outcomes: tarpit (PT--), deny (PR--), 403, 429.
+        if (!(term ~ /^PT/ || term ~ /^PR/ || status == "403" || status == "429")) next
+
+        for (i = 1; i <= NF; i++) {
+            if (substr($i, 1, 4) == "cip=")  cip  = substr($i, 5)
+            if (substr($i, 1, 5) == "host=") host = substr($i, 6)
+            if (substr($i, 1, 3) == "id=")   id   = substr($i, 4)
+        }
+
+        if (match($0, /"[A-Z]+ [^"]*"/)) {
+            req = substr($0, RSTART + 1, RLENGTH - 2)
+            if (length(req) > 42) req = substr(req, 1, 41) "..."
+        }
+
+        if (cip == "") cip = "-"
+        if (host == "") host = "-"
+        if (term == "") term = "-"
+        if (status == "") status = "-"
+        printf "%-15s %-16s %-4s %-5s %-28s %s\n", ts, cip, status, term, host, req
+        if (id != "" && id != "-") printf "%-15s %s\n", "", "  id=" id
+        n++
+    }
+    END { if (n == 0) print "(no tarpit/deny/403/429 events in the scanned window)" }
+    ')
+    printf '%s\n' "$found"
+    echo
+    echo "TERM = HAProxy termination state: PT-- tarpit, PR-- deny (incl. WAF/rate limit)."
+    echo "id=  = request reference; it is printed on the block page and is how a"
+    echo "       customer support ticket correlates to an exact request here."
 }
 
-# Monitor mode selection
-if [ "$1" == "live" ]; then
-    echo "Live monitoring mode - Press Ctrl+C to exit"
-    echo ""
+banner() {
+    echo "==================================================="
+    echo "HAProxy Edge Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
+    echo "==================================================="
+    echo
+}
 
+if [ "${1:-}" = "live" ]; then
+    echo "Live monitoring mode - Press Ctrl+C to exit"
     while true; do
         clear
-        echo "==================================================="
-        echo "HAProxy Security Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
-        echo "==================================================="
-        echo ""
-        show_threats
-        echo ""
+        banner
+        show_rates || true
+        echo
         show_recent_blocks
         sleep 5
     done
 else
-    # Single run mode
-    show_threats
-    echo ""
+    banner
+    rc=0
+    show_rates || rc=$?
+    echo
     show_recent_blocks
-    echo ""
-    echo "Tip: Run with 'live' parameter for continuous monitoring"
+    echo
+    echo "Tip: run with 'live' for a refreshing view."
     echo "Usage: $0 [live]"
-fi
+    # Propagate a stick-table read failure: if the rates section could not be
+    # produced, this run did NOT report what it claims to report.
+    exit "$rc"
+fi

scripts/renew-certificates.sh

@@ -8,6 +8,7 @@ LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
 ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
 DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
 SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
+LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
 
 # Logging functions
 log_info() {
@@ -18,6 +19,18 @@ log_error() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
 }
 
+# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
+# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
+# uses this script's logging rather than its own fallbacks.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=cert-publish-lib.sh
+if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
+    . "${SCRIPT_DIR}/cert-publish-lib.sh"
+else
+    log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
+    exit 1
+fi
+
 log_info "Starting certificate renewal process"
 
 # Run certbot renewal — don't exit on failure, some certs may have
@@ -42,7 +55,7 @@ fi
 mkdir -p "$SSL_CERTS_DIR"
 
 # Get all SSL-enabled domains from database
-DOMAINS=$(find /etc/letsencrypt/live/ -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
+DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
 
 if [ -z "$DOMAINS" ]; then
     log_info "No SSL-enabled domains found"
@@ -52,15 +65,27 @@ fi
 # Copy certificates for each domain
 UPDATED=0
 FAILED=0
+SKIPPED_EXPIRED=0
 
 while read -r domain; do
-    CERT_FILE="/etc/letsencrypt/live/${domain}/fullchain.pem"
-    KEY_FILE="/etc/letsencrypt/live/${domain}/privkey.pem"
+    CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
+    KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
-    if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
-        # Combine cert and key into single file for HAProxy
-        if cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"; then
+    if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
+        # Never publish an expired lineage. It cannot help any client and,
+        # because HAProxy serves the first file claiming an SNI name, it can
+        # shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
+        # Counted separately from FAILED: a dead lineage that keeps expiring is
+        # reported on every run (ERROR log) without making every run "fail".
+        log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
+        SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
+    elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
+        # Assemble in a staging dir and rename into place. NEVER redirect into
+        # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
+        # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
+        # takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
+        if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
             log_info "Updated certificate for $domain"
             UPDATED=$((UPDATED + 1))
         else
@@ -73,10 +98,22 @@ while read -r domain; do
     fi
 done <<< "$DOMAINS"
 
-log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
+# Already-published pems that have expired since are moved out too (they
+# can only shadow; see cert_retire_expired_pems).
+cert_retire_expired_pems
+RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
+
+log_info "Certificate update completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
 
 # Reload HAProxy if any certificates were updated
-if [ $UPDATED -gt 0 ]; then
+if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
+    # Never reload onto unvalidated material: a reload that fails to load the
+    # certs directory drops HTTPS for every site on this host.
+    if ! haproxy_config_ok; then
+        log_error "HAProxy configuration does not validate - refusing to reload after certificate renewal"
+        exit 1
+    fi
+
     if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
         log_info "HAProxy reloaded successfully"
     else
@@ -85,5 +122,18 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# A per-domain publication failure is a real failure and must be reported as
+# one. This script used to `exit 0` no matter how many domains failed, so
+# "0 updated, 12 failed" - a host that has completely stopped publishing
+# renewals - looked identical to a clean run to cron, to
+# host-renew-certificates.sh (which branches on this exit code) and to any
+# external monitoring. The first visible symptom would have been certificates
+# expiring. The loop above deliberately continues past a failed domain so the
+# others still get published; the status is reported here instead.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate renewal process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate renewal process completed"
 exit 0

scripts/show-edge-ip-rates.sh

@@ -0,0 +1,314 @@
+#!/usr/bin/env bash
+#
+# show-edge-ip-rates.sh — real, current per-IP rate counters from the HAProxy
+# `web` stick table.
+#
+# WHAT THIS CAN TELL YOU
+#   The `web` stick table (templates/hap_listener.tpl) stores exactly four
+#   counters per client IP:
+#       conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
+#   Those are INSTANTANEOUS values — the current concurrency and the current
+#   sliding-window rates. This script prints them, and nothing else.
+#
+# WHAT THIS CANNOT TELL YOU
+#   * Who has been tarpitted, denied, or rate-limited. The stick table stores
+#     NO history and NO counter of past enforcement actions. It has no gpc0 /
+#     gpc1 / gpc(N) / gpc_rate / glitch_rate columns at all — any tool that
+#     claims to read them from this table is fabricating numbers.
+#   * Anything about an IP that has gone quiet: entries expire after 10m.
+#
+#   Real enforcement events live in the HAProxy ACCESS LOG, which is on the
+#   DOCKER HOST at /var/log/haproxy.log (it does NOT exist inside this
+#   container). The log-format carries the HAProxy termination state plus
+#   cip= (real client IP), host=, ua= and id= (the request UUID shown on the
+#   block page, which correlates with customer support tickets).
+#
+#   Ready to run ON THE HOST:
+#       # last 20 tarpitted (PT--) or denied (PR--) requests
+#       grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+#       # everything HAProxy answered 429/403 to, newest last
+#       grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20
+#       # everything for one client IP
+#       grep -a 'cip=203.0.113.7' /var/log/haproxy.log | tail -50
+#       # look up one request reference from a support ticket
+#       grep -a 'id=<uuid-from-the-block-page>' /var/log/haproxy.log
+#
+# USAGE
+#   show-edge-ip-rates.sh [-a|--all]
+#     -a, --all   also show rows whose counters are all zero (off by default:
+#                 a table with hundreds of idle entries is pure noise)
+#
+# ENVIRONMENT
+#   SHOW_ALL=1               same as --all
+#   HAPROXY_SOCKET=<path>    override the CLI socket (default /tmp/haproxy-cli)
+#   HAPROXY_TABLE_DUMP=<file>
+#                            parse a previously captured `show table web` dump
+#                            from a file instead of talking to the socket.
+#                            Supported seam for offline analysis of a captured
+#                            support bundle, and for testing this parser.
+#
+# NOTE ON THE SOCKET
+#   /tmp/haproxy-cli is HAProxy's MASTER CLI socket, so worker commands need an
+#   `@1` prefix. Without it HAProxy answers "Unknown command: 'show' ..." AND
+#   socat still exits 0 — so exit status is worthless here and this script
+#   inspects the RESPONSE BODY instead.
+
+set -euo pipefail
+
+SOCKET="${HAPROXY_SOCKET:-/tmp/haproxy-cli}"
+TABLE="web"
+
+# Fields this script expects the `web` stick table to store. Keep on ONE line
+# in this exact NAME=(a b c d) shape — the contract test greps for it, and the
+# parser below is driven entirely by it.
+EXPECTED_FIELDS=(conn_cur conn_rate http_req_rate http_err_rate)
+
+# Which of EXPECTED_FIELDS to sort on (descending). Falls back to the first
+# field if this name is not in the list.
+SORT_FIELD="http_req_rate"
+
+SHOW_ALL="${SHOW_ALL:-0}"
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -a|--all) SHOW_ALL=1 ;;
+        -h|--help) sed -n '2,60p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
+        *) echo "Unknown argument: $1" >&2; echo "Usage: $0 [-a|--all]" >&2; exit 2 ;;
+    esac
+    shift
+done
+
+die() { echo "ERROR: $*" >&2; exit 1; }
+
+# First non-blank line of a blob.
+#
+# Deliberately NOT `printf ... | sed -n '/./{p;q;}'`. sed quits after the first
+# match and closes the pipe; on a real 550-entry table dump printf is still
+# writing and takes SIGPIPE, so under `set -o pipefail` the whole command
+# substitution returns 141 and `set -e` kills the script -- silently, with no
+# output at all. That is the same class of failure this script exists to stop
+# hiding, so it does not get to happen here. A plain read loop has no pipeline
+# and no early close.
+first_nonblank() {
+    local line
+    while IFS= read -r line || [ -n "$line" ]; do
+        case "$line" in
+            *[![:space:]]*) printf '%s\n' "$line"; return 0 ;;
+        esac
+    done <<EOF
+$1
+EOF
+    return 0
+}
+
+# Return 0 if the CLI response body is a rejection rather than table data.
+# Checked on the body because socat's exit status is 0 either way.
+body_is_rejected() {
+    local first
+    first=$(first_nonblank "$1")
+    case "$first" in
+        "Unknown command"*|"No such table"*|"Permission denied"*) return 0 ;;
+        *) return 1 ;;
+    esac
+}
+
+send_cmd() {
+    printf '%s\n' "$1" | socat stdio "$SOCKET" 2>/dev/null
+}
+
+# ---------------------------------------------------------------- fetch data
+BODY=""
+SOURCE=""
+if [ -n "${HAPROXY_TABLE_DUMP:-}" ]; then
+    [ -r "$HAPROXY_TABLE_DUMP" ] || die "HAPROXY_TABLE_DUMP is set but '$HAPROXY_TABLE_DUMP' is not readable."
+    BODY=$(cat "$HAPROXY_TABLE_DUMP")
+    SOURCE="file $HAPROXY_TABLE_DUMP"
+else
+    [ -S "$SOCKET" ] || die "HAProxy CLI socket not found at $SOCKET (is HAProxy running, and are you inside the haproxy-manager container?)"
+    command -v socat >/dev/null 2>&1 || die "socat is not installed; cannot talk to $SOCKET"
+
+    # Master socket form first, then the plain stats-socket form.
+    BODY=$(send_cmd "@1 show table $TABLE" || true)
+    SOURCE="socket $SOCKET (@1 show table $TABLE)"
+    if [ -z "${BODY//[[:space:]]/}" ] || body_is_rejected "$BODY"; then
+        FALLBACK=$(send_cmd "show table $TABLE" || true)
+        if [ -n "${FALLBACK//[[:space:]]/}" ] && ! body_is_rejected "$FALLBACK"; then
+            BODY="$FALLBACK"
+            SOURCE="socket $SOCKET (show table $TABLE)"
+        else
+            echo "ERROR: HAProxy rejected BOTH '@1 show table $TABLE' and 'show table $TABLE'." >&2
+            echo "  @1 response : $(first_nonblank "$BODY")" >&2
+            echo "  bare response: $(first_nonblank "$FALLBACK")" >&2
+            echo "  Check the socket is HAProxy's CLI and that the table '$TABLE' exists" >&2
+            echo "  (a config reload without the frontend would drop it)." >&2
+            exit 1
+        fi
+    fi
+fi
+
+# ------------------------------------------------------------- header checks
+HEADER=$(first_nonblank "$BODY")
+case "$HEADER" in
+    "# table: $TABLE,"*) : ;;
+    *)
+        echo "ERROR: unexpected first line from '$SOURCE'." >&2
+        echo "  expected it to start with: # table: $TABLE," >&2
+        echo "  got                      : $HEADER" >&2
+        exit 1
+        ;;
+esac
+
+TBL_SIZE=$(printf '%s\n' "$HEADER" | sed -n 's/.*size:\([0-9]*\).*/\1/p')
+TBL_USED=$(printf '%s\n' "$HEADER" | sed -n 's/.*used:\([0-9]*\).*/\1/p')
+[ -n "$TBL_SIZE" ] || TBL_SIZE="?"
+[ -n "$TBL_USED" ] || TBL_USED="?"
+
+# ------------------------------------------------------------------- parsing
+# awk emits:
+#   W \t <field>:<window-seconds-or-dash> ...       (one line, from first row)
+#   R \t <sortkey> \t <ip> \t <value per EXPECTED_FIELDS in order>
+# and exits 1 after reporting any row missing an expected field.
+PARSED=""
+if ! PARSED=$(printf '%s\n' "$BODY" | awk -v fieldlist="${EXPECTED_FIELDS[*]}" -v sortfield="$SORT_FIELD" '
+BEGIN {
+    nf = split(fieldlist, F, " ")
+    sortidx = 1
+    for (i = 1; i <= nf; i++) if (F[i] == sortfield) sortidx = i
+    wprinted = 0
+}
+/^#/ { next }
+!/key=/ { next }
+{
+    split("", val, " "); split("", win, " ")
+    for (i = 1; i <= NF; i++) {
+        tok = $i
+        p = index(tok, "=")
+        if (p == 0) continue
+        lhs = substr(tok, 1, p - 1)
+        rhs = substr(tok, p + 1)
+        b = index(lhs, "(")
+        if (b > 0) {
+            nm = substr(lhs, 1, b - 1)
+            win[nm] = substr(lhs, b + 1, length(lhs) - b - 1)
+        } else {
+            nm = lhs
+            win[nm] = ""
+        }
+        val[nm] = rhs
+    }
+
+    missing = ""
+    for (i = 1; i <= nf; i++) if (!(F[i] in val)) missing = missing (missing == "" ? "" : ", ") F[i]
+    if (missing != "") {
+        printf "ERROR: stick table row is missing expected field(s): %s\n", missing > "/dev/stderr"
+        printf "  offending row: %s\n", $0 > "/dev/stderr"
+        printf "  this script expects the web table to store: %s\n", fieldlist > "/dev/stderr"
+        print  "  Those expectations and the templates/hap_listener.tpl `store` clause have DRIFTED." > "/dev/stderr"
+        print  "  Fix one or the other; refusing to print 0 for a counter HAProxy never reported." > "/dev/stderr"
+        exit 1
+    }
+    if (!("key" in val)) {
+        printf "ERROR: stick table row has no key= field: %s\n", $0 > "/dev/stderr"
+        exit 1
+    }
+
+    if (!wprinted) {
+        line = "W"
+        for (i = 1; i <= nf; i++) {
+            w = win[F[i]]
+            if (w ~ /^[0-9]+$/) w = sprintf("%g", w / 1000); else w = "-"
+            line = line "\t" F[i] ":" w
+        }
+        print line
+        wprinted = 1
+    }
+
+    nonzero = 0
+    row = ""
+    for (i = 1; i <= nf; i++) {
+        v = val[F[i]]
+        if (v + 0 != 0) nonzero = 1
+        row = row "\t" v
+    }
+    printf "R\t%s\t%s\t%d%s\n", val[F[sortidx]] + 0, val["key"], nonzero, row
+}
+'); then
+    exit 1
+fi
+
+# ------------------------------------------------------------------ printing
+WINSPEC=$(printf '%s\n' "$PARSED" | sed -n 's/^W\t//p' || true)
+
+echo "==================================================================="
+echo "  HAProxy edge IP rates — table '$TABLE' (current values only)"
+echo "==================================================================="
+echo "Source     : $SOURCE"
+echo "Tracked    : ${TBL_USED} of ${TBL_SIZE} slots in use"
+if [ "$SHOW_ALL" = "1" ]; then
+    echo "Filter     : showing ALL tracked IPs"
+else
+    echo "Filter     : showing only IPs with a non-zero counter (use --all for every row)"
+fi
+echo
+
+# Column headers, with each counter's window rendered in SECONDS (HAProxy
+# reports the window in milliseconds, e.g. conn_rate(10000) = 10s).
+HDR=$(printf "%-18s" "IP Address")
+i=0
+for f in "${EXPECTED_FIELDS[@]}"; do
+    w=$(printf '%s\n' "$WINSPEC" | tr '\t' '\n' | sed -n "s/^${f}://p")
+    if [ -n "$w" ] && [ "$w" != "-" ]; then
+        label="${f}/${w}s"
+    else
+        label="$f"
+    fi
+    HDR="$HDR $(printf '%18s' "$label")"
+    i=$((i + 1))
+done
+echo "$HDR"
+printf '%s\n' "$HDR" | sed 's/./-/g'
+
+ROWS=$(printf '%s\n' "$PARSED" | sed -n 's/^R\t//p' || true)
+shown=0
+if [ -n "$ROWS" ]; then
+    while IFS=$'\t' read -r sortkey ip nonzero rest; do
+        [ -n "${ip:-}" ] || continue
+        if [ "$SHOW_ALL" != "1" ] && [ "$nonzero" = "0" ]; then
+            continue
+        fi
+        line=$(printf "%-18s" "$ip")
+        oldifs="$IFS"; IFS=$'\t'
+        # shellcheck disable=SC2086
+        set -- $rest
+        IFS="$oldifs"
+        for v in "$@"; do
+            line="$line $(printf '%18s' "$v")"
+        done
+        echo "$line"
+        shown=$((shown + 1))
+    done < <(printf '%s\n' "$ROWS" | sort -t"$(printf '\t')" -k1,1nr)
+fi
+
+if [ "$shown" -eq 0 ]; then
+    if [ "$SHOW_ALL" = "1" ]; then
+        echo "(no IPs currently tracked)"
+    else
+        echo "(no IP currently has a non-zero counter — re-run with --all to list idle entries)"
+    fi
+fi
+
+echo
+echo "==================================================================="
+echo "These are CURRENT values. The table keeps no history and no record of"
+echo "past tarpits/denials. For actual enforcement events, read the access"
+echo "log ON THE DOCKER HOST (it does not exist in this container):"
+echo "  grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20     # tarpit / deny"
+echo "  grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20    # rate-limit / block"
+echo "  grep -a 'cip=<IP>' /var/log/haproxy.log | tail -50        # one client"
+echo "  grep -a 'id=<uuid>' /var/log/haproxy.log                  # one request reference"
+echo
+echo "Operator actions (via the MASTER CLI socket — the @1 prefix is required):"
+echo "  printf '@1 show table $TABLE key <IP>\\n'  | socat stdio $SOCKET"
+echo "  printf '@1 set table $TABLE key <IP> data.http_req_rate 0\\n' | socat stdio $SOCKET"
+echo "  printf '@1 clear table $TABLE key <IP>\\n' | socat stdio $SOCKET   # drop one entry"
+echo "  printf '@1 clear table $TABLE\\n'          | socat stdio $SOCKET   # drop ALL entries"
+echo "==================================================================="

scripts/show-tarpit-ips.sh

@@ -1,123 +1,35 @@
-#!/bin/bash
-
-# Script to display IPs that have been tarpitted by HAProxy 3.0
-# Uses HAProxy stats socket to query stick-table data
+#!/usr/bin/env bash
 #
-# Usage in Docker container:
-#   docker exec -it haproxy-manager /haproxy/scripts/show-tarpit-ips.sh
+# DEPRECATED SHIM — kept so existing docs/runbooks/muscle memory keep working.
+#
+# This script used to print a "Tarpitted IPs Report" with a "Scan Count" and a
+# BLOCKED / SILENT-DROP / TARPIT status per IP, all derived from gpc0 and gpc1
+# stick-table columns. Those columns DO NOT EXIST: the `web` table
+# (templates/hap_listener.tpl) stores only conn_cur, conn_rate, http_req_rate
+# and http_err_rate. The old parser defaulted every missing field to 0, so the
+# whole report was fabricated — every IP showed "Scan Count 0 / Normal"
+# regardless of what it was actually doing.
+#
+# The stick table also keeps NO history, so nothing in it can identify who was
+# tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST
+# at /var/log/haproxy.log (it does not exist inside this container):
+#   grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+#   grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limit / block
+#
+# What IS knowable from the stick table — the current per-IP rates — is printed
+# by show-edge-ip-rates.sh, which this shim now runs.
 
-SOCKET="/tmp/haproxy-cli"
+set -euo pipefail
 
-# Check if socket exists
-if [ ! -S "$SOCKET" ]; then
-    echo "Error: HAProxy socket not found at $SOCKET"
-    echo "Make sure HAProxy is running with stats socket enabled"
-    exit 1
-fi
+cat >&2 <<'NOTE'
+NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits.
+      The HAProxy stick table stores no history and no gpc0/gpc1 counters, so
+      the old "Scan Count"/"BLOCKED" columns were fabricated numbers.
+      Actual tarpit/deny events are in /var/log/haproxy.log ON THE HOST:
+        grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+      Running show-edge-ip-rates.sh instead (current rates, real values):
 
-echo "==================================================================="
-echo "                    HAProxy Tarpitted IPs Report                   "
-echo "==================================================================="
-echo
-echo "Showing IPs tracked in the stick-table with scan detection counters:"
-echo "(gpc0 = total scan attempts, gpc1 = escalation level)"
-echo
+NOTE
 
-# In HAProxy 3.0, we need to use the proper process prefix
-# The web frontend table is in the worker process, not master
-# First check which process has the table
-# Note: grep for actual worker line, not the header
-PROCESS_ID=$(echo "show proc" | socat stdio "$SOCKET" 2>/dev/null | grep -E '^[0-9]+.*worker' | awk '{print $1}' | head -1)
-
-if [ -z "$PROCESS_ID" ]; then
-    echo "Error: Could not find HAProxy worker process"
-    echo "Try: echo 'show proc' | socat stdio $SOCKET"
-    exit 1
-fi
-
-# Show stick-table entries from the web frontend using the worker process
-# Use printf to avoid bash history expansion issues with !
-printf "@!%s show table web\n" "${PROCESS_ID}" | socat stdio "$SOCKET" 2>/dev/null | {
-    # Skip the header line
-    read header
-    
-    # Check if we got an error or empty response
-    if echo "$header" | grep -q "No such table"; then
-        echo "Error: Table 'web' not found. HAProxy may need to be reloaded."
-        exit 1
-    fi
-    
-    has_data=false
-    echo "IP Address           | Scan Count | Level | HTTP Err Rate | Status"
-    echo "---------------------|------------|-------|---------------|------------------"
-    
-    # Process each line
-    while IFS= read -r line; do
-        # Skip empty lines and comments
-        if [ -z "$line" ] || echo "$line" | grep -q "^#"; then
-            continue
-        fi
-        
-        # HAProxy 3.0 format: 0x... key=<ip> use=... exp=... gpc0=... gpc1=... http_err_rate(10s)=...
-        if echo "$line" | grep -q "key="; then
-            has_data=true
-            
-            # Extract IP and counters
-            ip=$(echo "$line" | grep -o 'key=[^ ]*' | cut -d'=' -f2)
-            gpc0=$(echo "$line" | grep -o 'gpc0=[0-9]*' | cut -d'=' -f2)
-            gpc1=$(echo "$line" | grep -o 'gpc1=[0-9]*' | cut -d'=' -f2)
-            err_rate=$(echo "$line" | grep -o 'http_err_rate([^)]*=[0-9]*' | grep -o '[0-9]*$')
-            
-            # Set defaults if values are empty
-            gpc0=${gpc0:-0}
-            gpc1=${gpc1:-0}
-            err_rate=${err_rate:-0}
-            
-            # Determine status based on scan count and escalation
-            status=""
-            if [ "$gpc0" -ge 100 ]; then
-                status="BLOCKED (429)"
-            elif [ "$gpc0" -ge 60 ]; then
-                status="SILENT-DROP"
-            elif [ "$gpc0" -ge 40 ]; then
-                if [ "$gpc1" -ge 2 ]; then
-                    status="SILENT-DROP (repeat)"
-                else
-                    status="TARPIT 10s"
-                fi
-            elif [ "$gpc0" -ge 25 ]; then
-                status="TARPIT 10s"
-            else
-                status="Normal"
-            fi
-            
-            # Format output
-            printf "%-20s | %10s | %5s | %13s | %s\n" "$ip" "$gpc0" "$gpc1" "$err_rate/10s" "$status"
-        fi
-    done
-    
-    if [ "$has_data" = false ]; then
-        echo "(No IPs currently tracked - table is empty)"
-    fi
-}
-
-echo
-echo "==================================================================="
-echo "Legend:"
-echo "  - Scan Count 25-39: Low scanner → TARPIT 10s delay"
-echo "  - Scan Count 40-59: Medium scanner → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "  - Scan Count 60-99: High scanner → SILENT-DROP (immediate disconnect)"
-echo "  - Scan Count 100+:  Critical scanner → BLOCKED (429 response)"
-echo "  - Burst (5+ in 10s): → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "==================================================================="
-echo "Note: Only counts suspicious scripts/configs, NOT missing images/fonts/CSS"
-echo "Note: IPs are tracked for 1 hour since last activity"
-echo
-echo "To clear a specific IP from the table:"
-echo "  printf '@!${PROCESS_ID} del table web key <IP>\\n' | socat stdio $SOCKET"
-echo
-echo "To clear all entries:"
-echo "  printf '@!${PROCESS_ID} clear table web\\n' | socat stdio $SOCKET"
-echo
-echo "Debug: Worker PID is ${PROCESS_ID}"
-echo
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+exec "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"

scripts/start-up.sh

@@ -22,6 +22,39 @@ mkdir -p /etc/haproxy
 [ -f /etc/haproxy/trusted_ips.list ] || : > /etc/haproxy/trusted_ips.list
 [ -f /etc/haproxy/trusted_ips.map ]  || : > /etc/haproxy/trusted_ips.map
 
+# cloudflare_ips.list is SHIPPED DATA: it must always match what this image
+# bakes in (/haproxy/defaults), so a Cloudflare range refresh actually reaches
+# existing hosts instead of being permanently shadowed by the volume.
+# Overwrite it from the baked copy on every start.
+#
+# trusted_proxies.list and wpadmin_gate_exempt.list are OPERATOR DATA:
+# operators add entries directly on the server and those must survive
+# restarts/recreates. Seed each from the baked copy only when it's missing;
+# never overwrite an existing one.
+#
+# All branches fall back to an empty file if the baked default is somehow
+# absent, because "acl ... -f <missing file>" is a fatal HAProxy config
+# error -- the list files must exist unconditionally by the time HAProxy starts.
+if [ -f /haproxy/defaults/cloudflare_ips.list ]; then
+    cp /haproxy/defaults/cloudflare_ips.list /etc/haproxy/cloudflare_ips.list
+else
+    [ -f /etc/haproxy/cloudflare_ips.list ] || : > /etc/haproxy/cloudflare_ips.list
+fi
+if [ ! -f /etc/haproxy/trusted_proxies.list ]; then
+    if [ -f /haproxy/defaults/trusted_proxies.list ]; then
+        cp /haproxy/defaults/trusted_proxies.list /etc/haproxy/trusted_proxies.list
+    else
+        : > /etc/haproxy/trusted_proxies.list
+    fi
+fi
+if [ ! -f /etc/haproxy/wpadmin_gate_exempt.list ]; then
+    if [ -f /haproxy/defaults/wpadmin_gate_exempt.list ]; then
+        cp /haproxy/defaults/wpadmin_gate_exempt.list /etc/haproxy/wpadmin_gate_exempt.list
+    else
+        : > /etc/haproxy/wpadmin_gate_exempt.list
+    fi
+fi
+
 cron &
 
 # Phase 1: container init
@@ -44,7 +77,7 @@ HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
 ) &
 
 # Phase 2: WSGI servers
-# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
+# Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
 # (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
 # 1000 — worker recycle frequency).
 #
@@ -58,12 +91,27 @@ API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
 MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
 MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
 
+# Worker class (incident 2026-10-09): stock gthread makes every client socket
+# fully blocking with NO timeout, so any peer that sends a few bytes and goes
+# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
+# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. Stock gthread
+# also never reaps connections that send nothing (2x1000 of them wedge it
+# permanently) and has no total header-read bound (slowloris). This subclass
+# adds: a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, 20s), a
+# first-byte deadline (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s), a total
+# request-line+headers deadline (HAPROXY_MGR_HEADER_DEADLINE, 10s), and a
+# watchdog that exits a worker whose threads have ALL been stuck for
+# HAPROXY_MGR_WEDGE_SECONDS (3600) so the master respawns it.
+# See haproxy_gunicorn_worker.py. --timeout below does NOT cover any of these:
+# for gthread it only watches the main thread's heartbeat.
+WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
+
 # Default page server on :8080. Stays in the background.
 # --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
 # without forking. --max-requests recycles the worker to bound memory drift.
 gunicorn \
     --bind 0.0.0.0:8080 \
-    --workers 1 --threads 4 --worker-class gthread \
+    --workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout 30 \
     --access-logfile - --error-logfile - --log-level info \
@@ -75,7 +123,7 @@ gunicorn \
 # up). Longer --timeout because cert issuance hits ACME and can take a while.
 exec gunicorn \
     --bind 0.0.0.0:8000 \
-    --workers "${API_WORKERS}" --threads 4 --worker-class gthread \
+    --workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout "${API_TIMEOUT}" \
     --access-logfile - --error-logfile - --log-level info \

scripts/sync-certificates.sh

@@ -8,6 +8,7 @@ LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
 ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
 DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
 SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
+LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
 
 # Logging functions
 log_info() {
@@ -18,13 +19,25 @@ log_error() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
 }
 
+# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
+# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
+# uses this script's logging rather than its own fallbacks.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=cert-publish-lib.sh
+if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
+    . "${SCRIPT_DIR}/cert-publish-lib.sh"
+else
+    log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
+    exit 1
+fi
+
 log_info "Starting certificate sync process"
 
 # Ensure SSL certs directory exists
 mkdir -p "$SSL_CERTS_DIR"
 
 # Get all SSL-enabled domains from database
-DOMAINS=$(find /etc/letsencrypt/live/ -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
+DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
 
 if [ -z "$DOMAINS" ]; then
     log_info "No SSL-enabled domains found"
@@ -34,15 +47,27 @@ fi
 # Copy certificates for each domain
 UPDATED=0
 FAILED=0
+SKIPPED_EXPIRED=0
 
 while read -r domain; do
-    CERT_FILE="/etc/letsencrypt/live/${domain}/fullchain.pem"
-    KEY_FILE="/etc/letsencrypt/live/${domain}/privkey.pem"
+    CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
+    KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
-    if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
-        # Combine cert and key into single file for HAProxy
-        if cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"; then
+    if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
+        # Never publish an expired lineage. It cannot help any client and,
+        # because HAProxy serves the first file claiming an SNI name, it can
+        # shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
+        # Counted separately from FAILED: a dead lineage that keeps expiring is
+        # reported on every run (ERROR log) without making every run "fail".
+        log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
+        SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
+    elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
+        # Assemble in a staging dir and rename into place. NEVER redirect into
+        # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
+        # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
+        # takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
+        if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
             log_info "Updated certificate for $domain"
             UPDATED=$((UPDATED + 1))
         else
@@ -55,10 +80,22 @@ while read -r domain; do
     fi
 done <<< "$DOMAINS"
 
-log_info "Certificate sync completed: $UPDATED updated, $FAILED failed"
+# Already-published pems that have expired since are moved out too (they
+# can only shadow; see cert_retire_expired_pems).
+cert_retire_expired_pems
+RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
+
+log_info "Certificate sync completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
 
 # Reload HAProxy if any certificates were updated
-if [ $UPDATED -gt 0 ]; then
+if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
+    # Never reload onto unvalidated material: a reload that fails to load the
+    # certs directory drops HTTPS for every site on this host.
+    if ! haproxy_config_ok; then
+        log_error "HAProxy configuration does not validate - refusing to reload after certificate sync"
+        exit 1
+    fi
+
     if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
         log_info "HAProxy reloaded successfully"
     else
@@ -67,5 +104,12 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# See the matching block in renew-certificates.sh: a run in which every domain
+# failed to publish must not look like a clean run to its caller.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate sync process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate sync process completed"
 exit 0

scripts/test-config-rollback.py

@@ -31,8 +31,10 @@ __BROKEN__, which is how the tests inject an invalid configuration.
 """
 
 import os
+import re
 import sys
 import shutil
+import inspect
 import sqlite3
 import logging
 import tempfile
@@ -412,12 +414,79 @@ class TestBackupPrimitives(RollbackTestCase):
         self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
 
     def test_backup_set_covers_every_file_generate_config_writes(self):
-        pairs = dict(hm._config_backup_pairs())
-        for path in (hm.HAPROXY_CONFIG_PATH, hm.BLOCKED_IPS_MAP_PATH,
-                     hm.CORAZA_SPOE_CONFIG_PATH):
-            self.assertIn(path, pairs,
-                          f'{path} is written by generate_config() but is not '
-                          'part of the backed-up config set')
+        """Derived, not restated.
+
+        An earlier version of this test listed the three files it expected and
+        checked they were in the backup set, so it could never have noticed a
+        FOURTH file being added. Here the set of files generate_config() writes
+        is observed (and, for env-gated branches this fixture cannot safely
+        execute, read out of the source), and anything not backed up has to be
+        on the documented exclusion list below.
+        """
+        # Written by generate_config() but deliberately NOT restorable, with
+        # the reason. Everything else must be in the backup set: `haproxy -c`
+        # validates the config as a set, so a file it loads that is not
+        # restored alongside haproxy.cfg breaks rollback.
+        excluded = {
+            # Only ever created empty-when-missing (haproxy refuses to start
+            # with an ACL -f pointing at a missing file); its contents are
+            # owned by the /suspended API, not by generate_config(), so there
+            # is nothing here for a config rollback to undo.
+            'suspended_domains.list',
+            # Generated once and then read, never rewritten with new content.
+            # Its value is rendered INTO haproxy.cfg, so restoring an older
+            # haproxy.cfg alongside the current secret file is consistent.
+            'cluster-secret',
+        }
+        backed_up = {os.path.basename(p) for p, _ in hm._config_backup_pairs()}
+
+        # 1. Observed: run a generation with every patchable optional branch on
+        #    and see what actually changed on disk.
+        self.add_domain('derive.example.com', 'derive_backend')
+        os.environ['HAPROXY_CORAZA_SPOE_BACKEND'] = '127.0.0.1:9000'
+        self.addCleanup(os.environ.pop, 'HAPROXY_CORAZA_SPOE_BACKEND', None)
+        before = self._snapshot_etc()
+        hm.generate_config()
+        after = self._snapshot_etc()
+        touched = {name for name, blob in after.items()
+                   if before.get(name) != blob}
+        self.assertIn('coraza-spoe.cfg', touched,
+                      'fixture precondition: the Coraza branch did not run')
+
+        # 2. Read out of the source: branches this fixture must not execute
+        #    (suspension writes a hardcoded /etc/haproxy path that no test
+        #    global can redirect) still have to be accounted for.
+        touched |= {
+            os.path.basename(m)
+            for m in re.findall(r"'(/etc/haproxy/[\w.+-]+)'",
+                                inspect.getsource(hm.generate_config))
+        }
+
+        unaccounted = touched - backed_up - excluded
+        self.assertEqual(
+            unaccounted, set(),
+            f'generate_config() writes {sorted(unaccounted)}, which is neither '
+            'in the backup set nor on the documented exclusion list - a '
+            'rollback would restore a mixed-vintage config set')
+
+    def _snapshot_etc(self):
+        """Contents of every plain file in the fake /etc/haproxy.
+
+        Skips the backup halves (they are the thing being maintained), the
+        SQLite database and its journals, and the stats socket.
+        """
+        skip_prefixes = (os.path.basename(hm.DB_FILE),
+                         os.path.basename(hm.HAPROXY_SOCKET_PATH))
+        state = {}
+        for name in os.listdir(self.etc):
+            path = os.path.join(self.etc, name)
+            if not os.path.isfile(path) or name.endswith('.backup'):
+                continue
+            if name.startswith(skip_prefixes):
+                continue
+            with open(path, 'rb') as fh:
+                state[name] = fh.read()
+        return state
 
     def test_coraza_spoe_config_round_trips(self):
         self.generate_good_config()
@@ -452,8 +521,12 @@ class TestAtomicWrite(RollbackTestCase):
             fh.write('old content\n')
 
         # Anything that makes f.write() blow up mid-flight stands in for a full
-        # disk / killed container.
-        with self.assertRaises(Exception):
+        # disk / killed container. TypeError specifically, not Exception: a
+        # bare assertRaises(Exception) also swallows the AttributeError raised
+        # when write_config_atomically does not exist at all, so this test
+        # passed against the pre-fix tree and would keep passing if the
+        # function were deleted.
+        with self.assertRaises(TypeError):
             hm.write_config_atomically(path, object())
 
         self.assertEqual(self.read(path), 'old content\n',
@@ -462,6 +535,241 @@ class TestAtomicWrite(RollbackTestCase):
         self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
 
 
+class TestBackupFailureGuard(RollbackTestCase):
+    """generate_config() refuses to write when no rollback target could be taken."""
+
+    def test_a_failed_backup_stops_the_config_from_being_written(self):
+        good = self.generate_good_config()
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        real_create_backup = hm.create_backup
+        hm.create_backup = lambda *a, **kw: (False, 'error')
+        self.addCleanup(setattr, hm, 'create_backup', real_create_backup)
+
+        self.add_domain('second.example.com', 'second_backend', '10.0.0.3')
+        with self.assertRaises(Exception) as ctx:
+            hm.generate_config()
+
+        self.assertIn('Refusing to regenerate', str(ctx.exception),
+                      'a backup failure was not reported as a refusal')
+        self.assertEqual(
+            self.read(hm.HAPROXY_CONFIG_PATH), good,
+            'a new config was written even though the snapshot failed - a bad '
+            'change could not have been rolled back')
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+            'the blocked IPs map was rewritten even though the snapshot failed')
+
+
+class TestValidatorAvailability(RollbackTestCase):
+    """'the validator could not run' is not the same as 'the config is bad'."""
+
+    def _hide_the_haproxy_binary(self):
+        empty = os.path.join(self.tmp, 'empty-bin')
+        os.makedirs(empty, exist_ok=True)
+        os.environ['PATH'] = empty
+        # setUp's cleanup restores the original PATH.
+
+    def test_a_missing_validator_is_unavailable_not_invalid(self):
+        good = self.generate_good_config()
+        self._hide_the_haproxy_binary()
+
+        status, message = hm.validate_config_file(hm.HAPROXY_CONFIG_PATH)
+        self.assertEqual(status, 'unavailable',
+                         'a validator that could not run was reported as a '
+                         f'verdict on the config ({status}: {message})')
+
+        # And the consequence create_backup() draws from it: a config it could
+        # not check is still snapshotted, because refusing would leave the box
+        # with no rollback target at all. Contrast
+        # test_a_broken_current_config_does_not_replace_a_good_backup, where a
+        # real 'invalid' verdict yields 'kept_previous'.
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write('hand written, unverifiable\n')
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'created',
+                         'an unverifiable config was treated as a rejected one')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH),
+                         'hand written, unverifiable\n')
+        self.assertNotEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
+
+
+class TestFileComparison(RollbackTestCase):
+    """The fast path compares bytes, not sizes."""
+
+    def test_same_size_different_content_is_not_identical(self):
+        a = os.path.join(self.etc, 'a')
+        b = os.path.join(self.etc, 'b')
+        with open(a, 'w') as fh:
+            fh.write('aaaa\n')
+        with open(b, 'w') as fh:
+            fh.write('aaba\n')
+        self.assertEqual(os.path.getsize(a), os.path.getsize(b),
+                         'fixture: the two files must be the same size')
+        self.assertFalse(hm._files_identical(a, b),
+                         'two same-size files with different bytes compared equal')
+
+    def test_a_same_size_drifted_config_still_hits_the_validation_gate(self):
+        """The case the byte-compare exists for.
+
+        A config edited in place without changing its length (one character
+        swapped, a hostname replaced by another of the same width) must not be
+        mistaken for the known-good backup and waved through.
+        """
+        good = self.generate_good_config()
+        # Sized in BYTES, not characters: the rendered config contains
+        # non-ASCII (em dashes in template comments), so len(str) would be
+        # smaller than the file and this test would pass for the wrong reason.
+        size = os.path.getsize(hm.HAPROXY_CONFIG_PATH)
+        broken = f'# {BROKEN_TOKEN}\n'.encode()
+        broken += b'#' * (size - len(broken) - 1) + b'\n'
+        with open(hm.HAPROXY_CONFIG_PATH, 'wb') as fh:
+            fh.write(broken)
+        self.assertEqual(os.path.getsize(hm.HAPROXY_CONFIG_PATH), size,
+                         'fixture: the drifted config must be the same size')
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(
+            status, 'kept_previous',
+            'a same-size broken config was accepted as unchanged and skipped '
+            'the validation gate')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'the known-good backup was overwritten')
+
+
+class TestBlockedIpsMapWrites(RollbackTestCase):
+    """blocked_ips.map is loaded by `haproxy -c`, so it gets the same care.
+
+    Verified against HAProxy 2.8: with the map referenced by
+    map_ip(/etc/haproxy/blocked_ips.map,0), a half-written final line makes the
+    WHOLE configuration invalid ("'198.51.10' is not a valid IPv4 or IPv6
+    address at line 2 of file ..."), not merely a dropped entry.
+    """
+
+    def test_the_map_goes_through_the_atomic_writer(self):
+        seen = []
+        real_write = hm.write_config_atomically
+
+        def spy(path, content, *args, **kwargs):
+            seen.append(path)
+            return real_write(path, content, *args, **kwargs)
+
+        hm.write_config_atomically = spy
+        self.addCleanup(setattr, hm, 'write_config_atomically', real_write)
+
+        self.block_ip('192.0.2.10')
+        self.assertTrue(hm.update_blocked_ips_map())
+        self.assertIn(hm.BLOCKED_IPS_MAP_PATH, seen,
+                      'the blocked IPs map was written without the atomic '
+                      'writer - a truncated map is a fatal config')
+
+    def test_a_crash_before_the_rename_leaves_the_old_map_intact(self):
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        real_replace = os.replace
+
+        def boom(src, dst, *args, **kwargs):
+            if dst == hm.BLOCKED_IPS_MAP_PATH:
+                raise OSError('simulated crash between write and rename')
+            return real_replace(src, dst, *args, **kwargs)
+
+        os.replace = boom
+        self.addCleanup(setattr, os, 'replace', real_replace)
+
+        self.block_ip('198.51.100.20')
+        self.assertFalse(hm.update_blocked_ips_map(),
+                         'a failed map write was reported as success')
+        os.replace = real_replace
+
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+            'an interrupted map write clobbered the map HAProxy is running')
+        leftovers = [n for n in os.listdir(self.etc) if n.endswith('.tmp')]
+        self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
+
+    def test_a_malformed_map_is_not_recorded_as_known_good(self):
+        """The map backup must stay something HAProxy would actually load."""
+        self.generate_good_config()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_BACKUP_PATH)
+
+        # Straight into the table, the way a bad row gets there in the first
+        # place - the API route is not the only writer.
+        self.block_ip('not-an-ip')
+        hm.update_blocked_ips_map()
+
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_BACKUP_PATH), good_map,
+            'a map HAProxy cannot parse was promoted to the rollback target')
+
+    def test_no_map_backup_is_fabricated_before_a_config_exists(self):
+        """Nothing to stay in step with means nothing to write."""
+        self.block_ip('192.0.2.10')
+        self.assertTrue(hm.update_blocked_ips_map())
+        self.assertFalse(
+            os.path.exists(hm.BLOCKED_IPS_MAP_BACKUP_PATH),
+            'a rollback target was invented out of a map write alone')
+
+
+class TestValidationCost(RollbackTestCase):
+    """`haproxy -c` runs are the customer-facing cost of a config change.
+
+    generate_config() runs synchronously inside the API call that adds a
+    domain, and on an edge with hundreds of certificates `haproxy -c` is the
+    expensive part. These counts are the contract; changing them should be a
+    deliberate decision, not a side effect.
+    """
+
+    def _count_validations(self, action):
+        calls = []
+        real_validate = hm.validate_config_file
+
+        def spy(path):
+            calls.append(path)
+            return real_validate(path)
+
+        hm.validate_config_file = spy
+        try:
+            action()
+        finally:
+            hm.validate_config_file = real_validate
+        return len(calls)
+
+    def test_blocking_an_ip_does_not_add_a_validation_to_the_next_change(self):
+        self.generate_good_config()
+
+        def add(domain, backend, address):
+            self.add_domain(domain, backend, address)
+            hm.generate_config()
+
+        steady = self._count_validations(
+            lambda: add('a.example.com', 'a_backend', '10.0.0.4'))
+        self.assertEqual(
+            steady, 1,
+            'a steady-state config change should cost exactly one `haproxy -c` '
+            '(the pre-reload gate); the known-good fast path should skip the '
+            f'other one, but {steady} ran')
+
+        # What POST /api/blocked-ips does: rewrite the map outside
+        # generate_config(). This fleet blocks IPs automatically, so it happens
+        # between most config changes.
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+
+        after_block = self._count_validations(
+            lambda: add('b.example.com', 'b_backend', '10.0.0.5'))
+        self.assertEqual(
+            after_block, steady,
+            'an IP block left the map out of step with its backup, so the next '
+            f'domain add paid {after_block} `haproxy -c` runs instead of '
+            f'{steady} - on the customer-facing call')
+
+
 if __name__ == '__main__':
     print(f"testing haproxy_manager from: {MODULE_DIR}")
     unittest.main(verbosity=2)

scripts/test-fixtures/san-certs/dead_old.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC9DCCAdygAwIBAgICEAMwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMZGVh
+ZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
+EwYDVQQDDAxkZWFkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQCumL3T27a3D6fZbIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnO
+abvnQNxEVqyYtTJOn9taitVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BU
+SpW96rK40KVftzzDEUnH+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0
+seOIdyP/n3T2cX1wsCa/F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9b
+PBd2cBaBwTe/3TICu0RnvvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2
+wzG4zlKU31fdLvQOzMGH1W5H91sBAgMBAAGjSDBGMCUGA1UdEQQeMByCDGRlYWQu
+ZXhhbXBsZYIMYnJhaW5qYXIubmV0MB0GA1UdDgQWBBSPDJiv0Z2YpuloexOfvpIQ
+UrWufTANBgkqhkiG9w0BAQsFAAOCAQEAJwBtN3Qw41XvYuSa8Mjn869QOTKEgkaR
+EuNQ8WXVBOTRJdxaLHDunGO4ExjHHswCWzXkefYOoIClAU9qLnk9M667BSNxOcz0
+VcG/DUMd0tO8X4Zm0zOMjt4VC7whgj7MdYtYVg2+Jhb4sMSx6OqmCUAD4X6ULSTn
+lZK2holEC/H3bKt8Q6cecgeubgWojP1bWmWK7jQL5NRIzxQ0KzFxgAf3t2Z05nX8
+loABkBiR7vDthkQseUW6RQMayDyzhVqORLt+gZ9D5T3hT1M7VaDc4Twr7UbWPFcR
+GuQ7RwKplPIIbPe7+c91pm+agdSDjnV5HmKcAMCwf7bWGQ9/Xc0BFg==
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/dead_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCumL3T27a3D6fZ
+bIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnOabvnQNxEVqyYtTJOn9ta
+itVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BUSpW96rK40KVftzzDEUnH
++3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0seOIdyP/n3T2cX1wsCa/
+F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9bPBd2cBaBwTe/3TICu0Rn
+vvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2wzG4zlKU31fdLvQOzMGH
+1W5H91sBAgMBAAECggEADUvYBFLn6gPWSZj55nxOqR+2Lmtwh04omgbu2fpWhPTE
+ujgKd/23srnnMZXnR6qZQQIPGwJiOPnJuRjiu/PMZSj8j54bbwfA6XmbH9SCIzm0
+la50UGBSkw0T50jBI76vICMm6PBkf0rT1oKN9VOd/qlkJoCUiFcGCh177Fbt07oY
+R3iAKu7/+yEANX0iEcxbO330TPW3oI0ah5OryfKVoNvxpF8NUfV9pWY66APkOYMJ
+zsYvIMvzV/KIgfrtyxJjSjgd7Ud8uictYG1BrZBXpIuGxuxyKUJ7OXOvMMVNnEME
+2uzk/aL75UjpurJJU0LCywdUOGwRqS2q4PYQiTeDLQKBgQDjdywH7RODoYeH05Rd
+sa2MkNx0Day1i3A4IBds55mzn8jxpT55HlqvHWhNA30kEw01H8+6sB7iMX04xQtS
+u3N2opR1VlCvMJhGyRUYUtfu4NnAi/HSu7mCOFGAwaLclI7BAI2or2UrclrlUhGb
+CYIBKfEr6Tn6b6AffWaOruCbtQKBgQDEf70R922fnJK+//QuiYoaESwJd3/k10yJ
+nH0ZAgbnFcNLT9YVFxHvlt4RHGIotaBjfN+FKMn045jeekgeVdX+uF8+7FwEazAH
+zOfxswOeKd8EOooqCD7n7cW3Ma6ev8oWmXwJsz2coRW7Z5zCGQWSzCEDB4yFU9Mt
+0z6aN9SJnQKBgQCvBZSIqL+6KgD53zArA7sRDKnEVn1M13TKis7bij02/DyGyuoO
+EfAfDBakI4oXxt0Dwm15LWkmDIRYuS0kRH3si96FhpvgoRiaZsDnjV7/PF3wjP3O
+c4mjDCiSVOw2o8xP71gQspy62GveRc6sJ52zvY0Hh5aH/3oKoNfX8/F5fQKBgALQ
+vbnDLMvrp2KbMjl0eQkUk+fy0V9LAmq6VkSEderbsfejRjL5KKWhKL9gA7wHliu2
+v1EeG36pwCf34o6ktjM3Gw5AvrTFgGOsOOqKoROkD/D0kcQ2aAhUrM+KI7FcmB2o
+x1EwQ0K6QCB2RJW3XBAVYqezdPz4DX+LlksJX781AoGBAISGOer3s4+oRjO/2VuT
+hnMQrET3A0GTol8cgYL2N9wfx7jy5bhSfiJIX/dkh5IrPGYoBdWM87gcUgNQ8AZ7
+il5dVeG80oc7KFU/QltZJu9VS32vwXnNsnrXupQ5Ee0ztTLciTMWU5jxvB0ZJHtK
+StgYepr4IPDUaObqrPx4Jvfm
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/expired_old.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDGjCCAgKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwGDEWMBQGA1UEAwwNYnJh
+aW4tamFyLmNvbTAeFw0yMDAxMDEwMDAwMDBaFw0yMDA0MDEwMDAwMDBaMBgxFjAU
+BgNVBAMMDWJyYWluLWphci5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQC9SJUMsK5HhVEATu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9s
+i5KzgYp0K2z899wQAKuLzZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ
+7jax2QWFMQVhyy6M5QTCHUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyN
+N8eolukhvFxQNU3E7oo1jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSy
+UEc+Gje6dJLZ9CdJtSQSNGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8
+FHC7atJOe5EoaekaAXw5Tq+yB+ODAgMBAAGjbjBsMEsGA1UdEQREMEKCDWJyYWlu
+LWphci5jb22CEXd3dy5icmFpbi1qYXIuY29tggxicmFpbmphci5uZXSCEHd3dy5i
+cmFpbmphci5uZXQwHQYDVR0OBBYEFAWqbBh2PDUvqQzZoyh960cpuhcOMA0GCSqG
+SIb3DQEBCwUAA4IBAQCaovtrXl56YcFlXWMrW/XZBe2/fXI14860qyiR5rb07lS6
+CbKOsnRyQdxZ3cdfRwTN1UMmz5IPOZ6fhdo3IPaS2Wae1OdeupRaRWr1SAJtpVZ3
+WWugDaumsWV5eWVxVTf93yDEkMERWtTGMviCKBeeHjzHIFiD/EjZznE87V5DYwuf
+guOd1F4wXCMRFu8q7sBdPXjQ8rZMlQT+dQXzMST9Ee5PXiInmCAA5nT8vMInAwd2
+rP6FxuHJB75ekYzs57CJ82BKKboOCESjUbm54wmVV5pBL305/EjxGf+VovNlRktg
+t1znolntIIofE25UCbVNPdrtuDkZgKT3Rv8tCdTP
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/expired_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC9SJUMsK5HhVEA
+Tu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9si5KzgYp0K2z899wQAKuL
+zZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ7jax2QWFMQVhyy6M5QTC
+HUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyNN8eolukhvFxQNU3E7oo1
+jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSyUEc+Gje6dJLZ9CdJtSQS
+NGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8FHC7atJOe5EoaekaAXw5
+Tq+yB+ODAgMBAAECggEABAeOCL71eYH4jKRMOBbXu98483u1CxnTEa8klnw9ZVES
+JctJjY96B/EMwv+g/V+gxhgAkiK9VWfg6bEplOVDWgR9fRBoP57mVkOv3QPn5fcR
+xlt9LQQZ+pULQRZoUAFEkVEkV1sfIBJmDLMufOnbAHCKa9+hVlrlCqu0lJgM8put
+2bkpSAqwroHVfmD0SmKFAZggBhFwXFJzwmUf3WL+N25f4RZpJsX17FOJ216mlhQ5
+edje6dcDpmko2VQtGJXYr8Ge02ZYeRyKGcTsgUZlarB4xUZq1pcnUUSy0Ez7LVbk
+tKjPcUpZLlnAA/a2pDgHmwN1+fv/X+Wwv/vZaQMBTQKBgQDuZFyfDRSHCssHtQ4+
+Ezxg4WEjwXkq58WxL0zTtLTcquZfQZ1VXgLwAylVwver0I4/UgCfW/AYY+2eLrba
+FhuYOosd/3rsUQqf9y+TkSkhe1uVS2/1zjzC1G/5coleLddjzd+YBpEtVxfxDRbA
+rVePY3GxkJiTrjAg5OETk7GYpwKBgQDLQ6bPhaocJPQmDi3hKyW8zPJ8wAJNyI0c
+RWt9gjeidjmMvwCKtJbBDcV5V0oQ5SB2M5E49mT58B62totAD7FS18c4a4133haV
+a1pLBX0U524eQgDzMJ6YA6N9bHdXOH4FZd+Pw90bHetbT1HcKpOdXv4SF/XITju5
+Ro5xiymdxQKBgBB4fq2+iZyK8lwl9kXzrkhDSiLeSzDbspz7V6OI6LYWWLRcJDKx
+vCI5FbIHNnV4g1t8nu5ju8LzpIc5UwjeXim2YcZA0I1jTAihW3gmn2Dg1QMLo0Co
+bypG4zWo5DTNlgQfRUOLsJwhWFYfQWMuls4hYMmT4/bQIiMjrSldGvghAoGALpaB
+7DyZobvCjdpgH6TREHX0+izUGlnmJd2vZmPrNnxccmrwx9neV789TLN6EWdCynTF
+S0YGHNcxYvVx8yvlm7RDQCY+lYO8r2CcoGdQpPLqZVsBdrvlDbh/x2aN79rN35Ta
+6cudqPJxmEsU2x6TGNXF7AW72r47qBqI2XNpBXUCgYAFQdHRRxa2E5rgV0WkL7PF
+vXok3isAzpnKAsb5oxPZbhIdLo5EPUGCslsTE1Na8NBdUSoyjhgkMELEDx/ceGL7
+IIlNrc/4j71+IahljzaX/D5b5qaW+RnoTVAlbMRR7L/ecbkoovhQR9d37QdoiD/t
+B9si92+XB4HYDAjXFxTK3w==
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/new_bundle.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC+DCCAeCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMYnJh
+aW5qYXIubmV0MCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
+EwYDVQQDDAxicmFpbmphci5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQCyhL5YcFhPEekc/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdb
+l4twhlkcyvyaEbcHIgLEgz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgU
+QcAAvdAyP77jDF6lX11oCjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zz
+XRVp4l2WyqEUJmKxcpppAnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU
+0v78QfI+ueCw261UWZ+iXwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQ
+dFOfKjH0QGInyByji9eAaXlmvCKtAgMBAAGjTDBKMCkGA1UdEQQiMCCCDGJyYWlu
+amFyLm5ldIIQd3d3LmJyYWluamFyLm5ldDAdBgNVHQ4EFgQUdDxfZKX96j61mW7O
+NiQpN4T+sZ4wDQYJKoZIhvcNAQELBQADggEBAJE1lP62caByZT7GkKtJLfOj84Ih
+MeP751w2gdU8ZfiO8GxXXiR3ZjYzbJgsEi9LiNT9jqHeBf3pv/mJLjPVZ+U557ws
+Gq3V5A3uKlRKGcejXlHaWnExhWqLyIaLTeLKstbJyiB/ja4yeTL2TYcOvvlu6weh
+ERqwx0p6r7LSQ3nRfTdjEhNq3FVUmqQ8neBP/vXLp2hKr2ACZndGyQOiv0oZWzud
+part6RMd8gRNq1Tctrl0YqRYoLRIddBrnTtUtac4FLqxLSBNc03/I9ZgF56Yf+Sl
+M1ikmyDJ2ZYJFU6v42Og4/8o3fpq0ttP6pKPi8fInGaFQ+KXO1NWFf1ogD0=
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/new_bundle.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCyhL5YcFhPEekc
+/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdbl4twhlkcyvyaEbcHIgLE
+gz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgUQcAAvdAyP77jDF6lX11o
+CjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zzXRVp4l2WyqEUJmKxcppp
+AnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU0v78QfI+ueCw261UWZ+i
+XwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQdFOfKjH0QGInyByji9eA
+aXlmvCKtAgMBAAECggEAEQi8lvjFXGojmco3H7PVD7SZftLOFidVh9Fo8/JAe+2Z
+LQM/hIjZ1i7sNverbQXMzAQMfXESS2Jx7Clv0mZ0aRgE0hq7wXrabyHfFUPOtqDa
+6cPHsz8YvwAOmcQ6W3fBPYI6pzSfdV03rv1GGeCC7ETI3mIupwgUbANDuiDcVY1q
++VuK/hDzPJ5KslDHUNR42MgWacIftq99fV9jN0zQQw1EcyO8QwpXzTSEbdyiAkT+
+n/tR9He2MYhgRhjyWPMaxmY6XQpv1h5ASq1yT1kI2oNTaRT3xvd40sLkbfwpZnxi
+HnqzCkPptEpWeCLrme+r71+Hxu/+Bph7tPtcuK7jgQKBgQD8XCoPRiWbgCl6kM2G
+E3GeM6fftTieTIs7sFVte7wutIKlsec4vizc94ipGaoT7yGEG0T8g9+5hAMbebc2
+zzhxvJMB2s18Z/He+Ujv1L1AhBBNVY3zCAxfdq8/n5MGXppEdVRo9QOQndoBfuAV
+Qw2Vp4bcy728n4T7sBugW0Ad7QKBgQC1F+utuiF8XI2yx0mX0+Fbty6o+odyM0eb
+G/QQDlcGpY0K7V3WAFMESNfytig8OhidzRfWn8lNoqhd80SJfCuiYjiGJ8tjBzcp
++Vqrd/KAxJySJ3NJ2MwyAp7W3l3HAEZLGVYVCankoa4O3qR8YqaIZwc2nk5+8KnG
+nBYFlSF/wQKBgQCqaRYh0E2O7wGTeXq/ohukJ9+hqf88rhWLTeyyDM8rfsuGVlb8
+MzeaUzEed20pHt60FbIFDowh+yoaAJwez9mevCdNJlKCQrLKze1Zvn+5tDRJ9I7x
+YSrxv/Rhc0FWGMX1moFQirtIib9wKRvqX6gpucHREWfjmpizWN2e8F+glQKBgAOp
+/jy/nVnUzZEmcxi9Urv2Ut1cZE0tOG+s+x0wDRa3Vsk30EQ6MXPOvX/Nq7a8Pi9D
+LQ5MoVgOyHVOLL7TYlagwiAEwhq41KbmwqzhemickcZrQevmlQsV/ny+dwtFuCZg
+1tsKHQPhOZJXcuCJZnEwFHdIcLIk0GqSUee/ArSBAoGALOD+Opk+0j4ObNcqtrpP
+UcEYqhOuAscbxLoUv53wnOvc7XzMG8JCK1y5LthB2Xanr1JSlheMRrvGxC+WlDmc
+5v0Nz71cq5QQ1p8AXnosLRjuy/+edMvGIxxwFMEJBO2ra2Jizpdwd34bjpQ42+CQ
+6pVwFQX0oet8SI5xWUWx7NU=
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/shared_old.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDGjCCAgKgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwHTEbMBkGA1UEAwwSc2hh
+cmVkLW9sZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAw
+WjAdMRswGQYDVQQDDBJzaGFyZWQtb2xkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEB
+AQUAA4IBDwAwggEKAoIBAQCBg7iYuvjhIG7nlFeja5Lyv2t2fdtNvxzUib7ygejC
++WQvNdH4k12q8vCXNYuadbGAceSejG4bazpjo5/ym2KFZPwiSZoqKUugi9lCoKuL
+2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMHYFwURjARRAku3a/GJUHQR0C572mW
+mt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64fGeYSOaYOHEOp/VqIRlzsy7eGQh3
+xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2JbqFxvijCLnBz5WsrS/S/VWOELNEfGr
+84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLfWuSgp5hRAgMBAAGjYjBgMD8GA1Ud
+EQQ4MDaCEnNoYXJlZC1vbGQuZXhhbXBsZYIMYnJhaW5qYXIubmV0ghJvdGhlci1z
+aXRlLmV4YW1wbGUwHQYDVR0OBBYEFP+AdZ9MkBmFjAQTgLnWZxbJtnvkMA0GCSqG
+SIb3DQEBCwUAA4IBAQA32Xl/1IrOyESd2UQcWp/fuHvkivLgamJrheKvKP0Y0vMV
+atRu0D0GhqMOr79OT60H/H1FFW7bWQ+UPcZpnuXec40/PcrHxwaSoiLoIUVWhB4t
+BOhtTDReX29gEjnB7OqBCJu4ub3HnpFTflUZd7n5A4EcetBsowc5EZ9UAtfvkgGD
+nK9ONvNVGurk/V8xgtMJ877ZK1fn+UtgvIWp/F1Gc/taLhe+oCZOPkOqvvqUnS8q
+QyIhqmMWRUkBP+jwRA4TP4pj7OH38VY/Ae408Xe4Y97GF7NbOjAL0bkX/bHkLnSG
+WzoGO0TK7pCemeFQ+Ghic8Yl0eDynoP7rZIhCKS0
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/shared_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCBg7iYuvjhIG7n
+lFeja5Lyv2t2fdtNvxzUib7ygejC+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpj
+o5/ym2KFZPwiSZoqKUugi9lCoKuL2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMH
+YFwURjARRAku3a/GJUHQR0C572mWmt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64
+fGeYSOaYOHEOp/VqIRlzsy7eGQh3xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2Jbq
+FxvijCLnBz5WsrS/S/VWOELNEfGr84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLf
+WuSgp5hRAgMBAAECggEACB1tvTwciKvp5wliGF45feDv5c5q+vXE462h0OYXjyfl
+1sNuMuoLjhYyFvdcQO2rlTBjB8Ald1hXoeidsVYlF9XvqmJRjjS6MAazMpEzTnba
+66V6U1j5gOQ6pi3SjfBJGFYo8L+cy/NqUIBWz0+6hp/BsjW3uW2aMGer2Y/zs2Oi
+s0kUNqqUWQpGpEDxbDnUAzto4OOcTOO4A5NB5Oe2ifzFiXCWtTNqFIC0ae9tiqGy
+m0sggF2hzmlH1WC54rbv7xujYT6tOAEi1OKp/Cpj8XVLbjaqju5avRl1t2nDRQri
+9vnOI9P4bnJKbIYzwJVyy8B1M5iK+W2K7wUQ4ZMetQKBgQC2zwtywQ4jUduLWf4f
+6HLUIvmcHdJI0YBHS1+4fKc7tsakRo/0FHxNdUlBhqkYTyp57h6/R1T2gG70Kla1
+MarGIgyPIlUXsRojlUcNyKXqSVs+kejftIClb+VDXWrpKtGtyxuESsMjaJBsBBAN
+hg3yUx2NL2ct4nnlpl1iq3pH9wKBgQC1XkrjntuzjVYlFeg2ih5va3PUD2qmH9Ej
+leNXTGhzZL1eQWz4GaRCgzJH8q41pXBxcXQ8hMXZCWdW5GHBYfVqjTRKMUPdI6FI
+ypwMbbTBfQfw1kz5VHfLH3romxikNQXeorZkIZqv7/aLPsxLY78XdW8nSDIsT9ZP
+mh7It+Tf9wKBgHT7VSYf6D4t2cH42qfshd6+auC+LJywg9gboatbCWEq+z6QcPoO
+SeT/Utt9oqONtW7FNCPPL812LopMtaPJnrCfhU3q+GzqxgFTO17UsWeVfbAyisTs
+qw4y5pEPLKT846TFPlyPwwPbm9+jUep1iGMzUMvSM8KnFL4ikcvcU6nNAoGBAJQI
+MVHV0HadzokYd19EnpDMgE6+goqy/21vIo6XY82DlPjAp+NccnCDSt/kmQJaVQuS
+0Y+vwxhhj4IhiBRiEwBz3hEMYtq1ASg7juCqDlW66mdNUqhSpDAbMqN2K8ejK3ZL
+EllqACrHzlBa4j5KXKcWB7JlP+ZhB4gTQDsNtXaDAoGBAJCktIdyM3b6UtM5OofB
+lzn6/2uT5u46ECBRybk/J+Y2gfcS2cFzkH179SUB4WgfR0h21FlLmjns5ZvHVk0p
+/U4aY3/J42JU6lCYrsyV5BBH+O0stSmrUG/4JdKRe6Mq/pB321RsWD06LVJrBDpi
+6NUPW03tNLc+Qe+cnQu0c7y+
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/unrelated.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC8zCCAdugAwIBAgICEAQwDQYJKoZIhvcNAQELBQAwHDEaMBgGA1UEAwwRdW5y
+ZWxhdGVkLmV4YW1wbGUwHhcNMjAwMTAxMDAwMDAwWhcNMjAwNDAxMDAwMDAwWjAc
+MRowGAYDVQQDDBF1bnJlbGF0ZWQuZXhhbXBsZTCCASIwDQYJKoZIhvcNAQEBBQAD
+ggEPADCCAQoCggEBALEpfqdRPBSrbzixAxD2WhsO1AOTUd/Gguf3zjDZ3TUmWJKb
+OhlFthtU6PJVhMdAqOk+hiz9Coo/UngWjykyNMOj1YIpq3M4/P8wBTNxkJd/z2x0
+iAQBKvWlB6hxf9c1kenu/ea1iMIThA/uudtX3g8k39Asj/37+o070cdBJExWGnbG
+lz103FCO6kBzlJPGIVQwzKH6OKthTzi7Uxy5Fxc/ymS9gc43F0tyY6892Wyl8Y62
+a3r1PpZoWR0GDFxPIyCS7NP44T15DSL/2RBi4YnWXOC8QMvPnHbSKjoQ6GLQtGkO
+1NO4Uy3XbwDFanYXv6Wxv0oUmZMuY7CRTOMdxCMCAwEAAaM/MD0wHAYDVR0RBBUw
+E4IRdW5yZWxhdGVkLmV4YW1wbGUwHQYDVR0OBBYEFBh0L/5U5f94TylibBCgS9pD
+/VXkMA0GCSqGSIb3DQEBCwUAA4IBAQB5MLqNjTqIdyztjJsKPdYUkA27NpfdqqY6
+Yprv5N7U0slsGUqFIDrSuJ1mMMwTpt02IAFyCOWMUtze+/0PqM9k9TqxJH2cTODp
+0ET34Qh8NTo0B534EQUWhny+EzgNAzWK02wbpgm88+Mp/t//sHzhWzOm7uk1dlkq
++5ty/8OveWwAtqAWlDtcsRhkEjzynZKUCAF1extwUSuapFSm20mJ4ivt7Ff5MkTI
+fq8P+3XAxWTaBW8DzBIzYd1mwFHK2pcklLMRRTW3hukPxEcTivBgJdwsHMskM3UO
+/ejy5EHem5ohCMbwMybvUqLS4xdoqXV36Sj2I1zv0IjgO0wgdK1r
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/unrelated.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCxKX6nUTwUq284
+sQMQ9lobDtQDk1HfxoLn984w2d01JliSmzoZRbYbVOjyVYTHQKjpPoYs/QqKP1J4
+Fo8pMjTDo9WCKatzOPz/MAUzcZCXf89sdIgEASr1pQeocX/XNZHp7v3mtYjCE4QP
+7rnbV94PJN/QLI/9+/qNO9HHQSRMVhp2xpc9dNxQjupAc5STxiFUMMyh+jirYU84
+u1McuRcXP8pkvYHONxdLcmOvPdlspfGOtmt69T6WaFkdBgxcTyMgkuzT+OE9eQ0i
+/9kQYuGJ1lzgvEDLz5x20io6EOhi0LRpDtTTuFMt128AxWp2F7+lsb9KFJmTLmOw
+kUzjHcQjAgMBAAECggEAJjBKo+iH6fC2beMCRTLFY5XDr+2etX0GFNi+8prT6zQT
+3a5mTLJROf0Ipcw3bfgLMRQQOjxlfD2AGnDKdVfkNdPoH0G6OoErUBra1CcDwxSC
+8t7/vtU9nGvxhabJputeQ64HvyacjzFDw2dzWsvLCzqooCgq9dXOQYmGg8wP9gAx
+pxfKnd84h41C3pXERdufi6nJHPjZPRW+dKsQ1NMLmtgO5YQnTt3LfVOkZaRa7Izq
+ZFWZCBfdsNHHuXCYj3FtqYrqSxELuZAxJ62hxpizrfQdKixguGTObHhC4muIo9fR
+eIMCvrAzrs4Fs8nM/K/IPyDU9wPwINK6Z+sxJkhM+QKBgQDaL3Yt3T/LE6bTYyb0
+EjVcXoybnuAjFQAenCTIsBRJv21Yv2ddqjq6rOTD64hFuFWPMEOi1TJxIPsFFDMw
+NeAwF41fiSrtU8awXs0hTKRQeHjElSBckumOl1DrHu2oKuwWXLniViD62vMsWmpZ
+89KT2rgFQuMPvsD3pps8l6W2HwKBgQDP3eTwdVJTdRYlZonqd0+LHG7h715TT1OE
+QJOdN9vK5g/9GOCOGHoUxB3IxJVmvz57PrfWaD2rSZ6JMs4TcAY5a0PbJEpgqoJY
+kIm/EDnSrFxu2L8DNsAzuThfzXrwWc7PzajYJctYtEfvxCyfo/GdZahg2xaEU0Le
+dp+7fxtJfQKBgGVdLWtLYarH+3Yzl1JG71F1f5+A5SKHTCHsas5tt083xuY3Lh9+
+yYAWeMVotEunwuAeDR4mLCQQrW4Dn3BjZz8QghaDVFUj2afwsx4oTCIyhfDWU02O
+10DunaJOIR1Cq6IL/DPhzbqdY+IUM1qNAyS7tUiEdZbICUBEv9Qbo9oDAoGAORnY
+cSGZaumoGsAPNMRvqmQ2JYSO4c3RwGvI4RBeJV+LlthFn2Mj7h5tnCA8dkwkytBQ
+yeo6vpnrnf7B018bppxLuKBAekt8oP2biUvYxoGopp6N4kEhDVRdIxwcIHhbMoRp
+VG8VbBmiBUBbjcMIdVdeqLSme9UvclMdz/C7uzUCgYEAzD2dvoJSu9zfo3fACw1X
+te53Rvr6Kq6DTw0SS4UrAnxlhX9TwD9gymIrR9rxnC+UPwfMwWMdpYqY+Da4sF1G
+uwd+nyZ4iFNq+lbLii9rmJ58h03+uo4lUXWTrAik6qOF1ztIfYKWFrOyBQmxjWwo
+HnUinudalENFJYZRwSUAtXo=
+-----END PRIVATE KEY-----

scripts/test-gunicorn-socket-timeouts.py

@@ -0,0 +1,631 @@
+#!/usr/bin/env python3
+"""Regression tests: a silent peer must not be able to wedge the API's threads.
+
+Why this file exists
+--------------------
+On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool threads
+blocked in the kernel in tcp_recvmsg -> sk_wait_data, /health hung 40s+, and the
+container stayed "healthy". The app makes no in-process outbound socket reads;
+the blocked reads were gunicorn gthread reading the CLIENT socket, which it
+makes fully blocking with no timeout (TConn.init -> setblocking(True)).
+
+These tests run REAL gunicorn processes (no mocks) against tiny WSGI apps and
+pin:
+
+  * the failure mode itself, on the stock gthread worker (control);
+  * that haproxy_gunicorn_worker.TimeoutThreadWorker frees the thread when a
+    client stalls before / during the request line, mid-body, or on a reused
+    keep-alive connection;
+  * that a legitimately SLOW request (handling time > client timeout) is not
+    cut off -- the timeout bounds socket I/O, not request duration;
+  * the watchdog: a worker whose every thread is stuck is replaced;
+  * review round 1, at production shape (2 workers x 4 threads): 2050
+    connections that never send a byte are reaped (F1, stock gthread wedges
+    permanently), and slowloris drip connections are cut by the total header
+    deadline while /health stays responsive (F2);
+  * the timeout is logged as one line, not a traceback per scanner;
+  * haproxy_manager's socket.setdefaulttimeout: an in-process client call to a
+    stalled upstream (accepts, never replies) times out instead of hanging;
+  * why that default alone is not enough (setblocking(True) clears it).
+
+Running
+-------
+    python3 scripts/test-gunicorn-socket-timeouts.py
+
+stdlib unittest, like the other scripts/test-*.py; needs only the app's own
+requirements (gunicorn, Flask...). Runs in the Docker build gate. ~70s.
+"""
+
+import os
+import socket
+import subprocess
+import sys
+import tempfile
+import textwrap
+import threading
+import time
+import unittest
+import urllib.request
+
+REPO_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')))
+
+try:
+    import gunicorn  # noqa: F401
+except ImportError:
+    sys.stderr.write('SKIP: gunicorn not installed (pip install -r requirements.txt)\n')
+    raise SystemExit(77)
+
+WSGI_APP = textwrap.dedent('''\
+    import os, time
+
+    def app(environ, start_response):
+        path = environ.get('PATH_INFO', '/')
+        if path == '/sleep':
+            time.sleep(float(environ.get('QUERY_STRING') or 3))
+        elif path == '/hang':
+            time.sleep(3600)
+        elif path == '/body':
+            n = int(environ.get('CONTENT_LENGTH') or 0)
+            environ['wsgi.input'].read(n)
+        body = ('pid=%d' % os.getpid()).encode()
+        start_response('200 OK', [('Content-Type', 'text/plain'),
+                                  ('Content-Length', str(len(body)))])
+        return [body]
+''')
+
+
+def free_port():
+    s = socket.socket()
+    s.bind(('127.0.0.1', 0))
+    port = s.getsockname()[1]
+    s.close()
+    return port
+
+
+def http_get(port, path='/', timeout=5):
+    with urllib.request.urlopen('http://127.0.0.1:%d%s' % (port, path),
+                                timeout=timeout) as r:
+        return r.read().decode()
+
+
+class Gunicorn:
+    """A real gunicorn master on a free port, with its output captured."""
+
+    def __init__(self, worker_class, threads=1, env=None, workers=1, extra_ports=()):
+        self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
+        with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
+            fh.write(WSGI_APP)
+        self.port = free_port()
+        self.log_path = os.path.join(self.tmp, 'gunicorn.log')
+        e = dict(os.environ)
+        e.update(env or {})
+        self.log = open(self.log_path, 'w')
+        self.proc = subprocess.Popen(
+            [sys.executable, '-m', 'gunicorn',
+             '--bind', '127.0.0.1:%d' % self.port,
+             *sum((['--bind', '127.0.0.1:%d' % p] for p in extra_ports), []),
+             '--workers', str(workers), '--threads', str(threads),
+             '--worker-class', worker_class,
+             '--timeout', '120', '--graceful-timeout', '2',
+             '--log-level', 'info', '--error-logfile', '-',
+             '--pythonpath', '%s,%s' % (REPO_DIR, self.tmp),
+             'tapp:app'],
+            stdout=self.log, stderr=subprocess.STDOUT, env=e)
+        deadline = time.time() + 15
+        while time.time() < deadline:
+            try:
+                http_get(self.port, '/', timeout=1)
+                return
+            except Exception:
+                time.sleep(0.2)
+        self.stop()
+        raise RuntimeError('gunicorn did not come up:\n' + self.output())
+
+    def output(self):
+        self.log.flush()
+        with open(self.log_path) as fh:
+            return fh.read()
+
+    def stop(self):
+        if self.proc.poll() is None:
+            self.proc.terminate()
+            try:
+                self.proc.wait(5)
+            except subprocess.TimeoutExpired:
+                self.proc.kill()
+                self.proc.wait()
+        self.log.close()
+
+
+def stall_client(port, payload=b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03'):
+    """Connect and send a few bytes that are not a complete HTTP request (by
+    default the start of a TLS ClientHello -- what an HTTPS scanner sends to
+    this plain-HTTP port), then go silent and keep the connection open."""
+    s = socket.create_connection(('127.0.0.1', port))
+    if payload:
+        s.sendall(payload)
+    return s
+
+
+def wait_closed_by_server(sock, within):
+    """True if the server closed (EOF/RST) the connection within `within` s."""
+    sock.settimeout(within)
+    try:
+        while True:
+            data = sock.recv(4096)
+            if not data:
+                return True
+    except (ConnectionResetError, BrokenPipeError):
+        return True
+    except socket.timeout:
+        return False
+
+
+STOCK = 'gthread'
+FIXED = 'haproxy_gunicorn_worker.TimeoutThreadWorker'
+FAST = {'HAPROXY_MGR_CLIENT_TIMEOUT': '1'}
+
+
+class StockGthreadControl(unittest.TestCase):
+    """Proves the failure mode is real, so the fixed-worker tests mean something."""
+
+    def test_stock_gthread_is_wedged_by_one_silent_client(self):
+        g = Gunicorn(STOCK, threads=1)
+        self.addCleanup(g.stop)
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        time.sleep(0.3)
+        with self.assertRaises(Exception, msg='stock gthread should be wedged'):
+            http_get(g.port, '/', timeout=3)
+
+
+class TimeoutWorkerTests(unittest.TestCase):
+
+    def start(self, threads=1, env=None):
+        g = Gunicorn(FIXED, threads=threads, env=env or FAST)
+        self.addCleanup(g.stop)
+        return g
+
+    def test_silent_client_is_dropped_and_thread_freed(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        t0 = time.time()
+        # Only one thread: this request can only be served once the stalled
+        # connection has released it.
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertLess(time.time() - t0, 4)
+        self.assertTrue(wait_closed_by_server(s, 3),
+                        'server should have closed the stalled connection')
+
+    def test_connect_and_send_nothing_does_not_hold_a_thread(self):
+        # Never readable -> never handed to a pool thread (main loop only).
+        g = self.start()
+        s = stall_client(g.port, payload=b'')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
+
+    def test_half_written_headers(self):
+        g = self.start()
+        s = stall_client(g.port, payload=b'GET /health HTTP/1.1\r\nHost: x\r\n')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_body_shorter_than_content_length(self):
+        g = self.start()
+        s = stall_client(g.port, payload=(
+            b'POST /body HTTP/1.1\r\nHost: x\r\nContent-Length: 100\r\n\r\n0123456789'))
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+
+    def test_timeout_reapplied_on_keepalive_reuse(self):
+        g = self.start()
+        s = socket.create_connection(('127.0.0.1', g.port))
+        self.addCleanup(s.close)
+        s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
+        s.settimeout(3)
+        self.assertIn(b'200 OK', s.recv(4096))
+        # Same connection, now stall mid-request. gthread calls conn.init()
+        # (setblocking(True)) again for the reused connection.
+        s.sendall(b'GET / HTT')
+        time.sleep(0.3)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_slow_legit_request_is_not_cut_off(self):
+        # Handling takes 2.5s, client timeout is 1s: must still succeed.
+        g = self.start()
+        self.assertTrue(http_get(g.port, '/sleep?2.5', timeout=10).startswith('pid='))
+
+    def test_timeout_logged_as_one_line_not_traceback(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        http_get(g.port, '/', timeout=5)
+        time.sleep(0.3)
+        out = g.output()
+        self.assertIn('Closed client connection idle', out)
+        self.assertNotIn('Traceback', out)
+
+    def test_watchdog_replaces_fully_wedged_worker(self):
+        g = self.start(env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                            'HAPROXY_MGR_WEDGE_SECONDS': '2',
+                            'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.5'})
+        pid_before = http_get(g.port, '/')
+        # Wedge the only thread inside the app (not a socket read).
+        hung = threading.Thread(target=lambda: self._swallow(g.port), daemon=True)
+        hung.start()
+        time.sleep(0.5)
+        deadline = time.time() + 12
+        pid_after = None
+        while time.time() < deadline:
+            try:
+                pid_after = http_get(g.port, '/', timeout=2)
+                if pid_after != pid_before:
+                    break
+            except Exception:
+                pass
+            time.sleep(0.3)
+        self.assertIsNotNone(pid_after)
+        self.assertNotEqual(pid_before, pid_after, 'worker was not replaced')
+        out = g.output()
+        self.assertIn('WEDGED WORKER', out)
+        self.assertIn('tapp.py', out)  # the stuck thread's stack was dumped
+        self.assertIsNone(g.proc.poll(), 'master must keep running')
+
+    def test_watchdog_leaves_partially_busy_pool_alone(self):
+        # 2 threads, one hung forever: the pool still serves, so no restart.
+        g = self.start(threads=2, env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                                       'HAPROXY_MGR_WEDGE_SECONDS': '1',
+                                       'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.3'})
+        pid_before = http_get(g.port, '/')
+        threading.Thread(target=lambda: self._swallow(g.port), daemon=True).start()
+        time.sleep(3)
+        self.assertEqual(pid_before, http_get(g.port, '/', timeout=3))
+        self.assertNotIn('WEDGED WORKER', g.output())
+
+    @staticmethod
+    def _swallow(port):
+        try:
+            http_get(port, '/hang', timeout=60)
+        except Exception:
+            pass
+
+
+def raise_nofile(n):
+    try:
+        import resource
+        soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
+        if soft < n:
+            resource.setrlimit(resource.RLIMIT_NOFILE, (min(n, hard), hard))
+        return resource.getrlimit(resource.RLIMIT_NOFILE)[0]
+    except (ImportError, ValueError, OSError):
+        return None
+
+
+class ProductionShapeExhaustion(unittest.TestCase):
+    """Review round 1 (F1/F2), at the production shape: 2 workers x 4 threads,
+    default worker_connections=1000."""
+
+    def start(self, env):
+        g = Gunicorn(FIXED, threads=4, workers=2, env=env)
+        self.addCleanup(g.stop)
+        return g
+
+    def test_f1_2050_idle_conns_reaped_and_health_recovers(self):
+        # Stock gthread: 2x1000 never-readable conns fill every slot, the run
+        # loop stops polling, and the worker stays dead even after the
+        # attacker closes everything. Here they must be reaped after
+        # FIRST_BYTE_TIMEOUT -- even while the attacker still holds them open.
+        if (raise_nofile(4500) or 0) < 4300:
+            self.skipTest('RLIMIT_NOFILE too low for 2050 client sockets')
+        g = self.start({'HAPROXY_MGR_FIRST_BYTE_TIMEOUT': '3'})
+        socks = []
+        self.addCleanup(lambda: [x.close() for x in socks])
+        for _ in range(2050):
+            s = socket.socket()
+            s.settimeout(2)
+            s.connect(('127.0.0.1', g.port))
+            socks.append(s)
+        # Attacker keeps every socket open. Within FIRST_BYTE_TIMEOUT plus the
+        # time to drain the backlog, the API must answer again.
+        deadline = time.time() + 20
+        ok = None
+        while time.time() < deadline:
+            try:
+                ok = http_get(g.port, '/', timeout=3)
+                break
+            except Exception:
+                time.sleep(0.5)
+        self.assertIsNotNone(ok, '/health never recovered while idle conns were held')
+        held = time.time()
+        # The reaped sockets were closed by the server.
+        closed = sum(1 for x in socks[:1500] if wait_closed_by_server(x, 0.01))
+        self.assertGreater(closed, 1000, 'server did not close idle conns (%d)' % closed)
+        for x in socks:
+            x.close()
+        socks.clear()
+        time.sleep(1)
+        for _ in range(5):
+            self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
+        self.assertLess(held, time.time())
+
+    def test_f2_drip_connections_cut_and_health_responsive_throughout(self):
+        # 8 connections (= every thread at production shape) each drip one
+        # header byte per second -- far under the 20s per-recv timeout, so only
+        # the total HEADER_DEADLINE can cut them.
+        g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '3'})   # CLIENT_TIMEOUT stays 20
+        stop = threading.Event()
+        cut_after = []
+
+        def drip():
+            s = socket.create_connection(('127.0.0.1', g.port))
+            t0 = time.time()
+            req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 1000
+            try:
+                for i in range(len(req)):
+                    if stop.is_set():
+                        return
+                    s.send(req[i:i + 1])
+                    time.sleep(1.0)
+                    s.settimeout(0.01)
+                    try:
+                        if s.recv(1) == b'':
+                            raise ConnectionResetError
+                    except socket.timeout:
+                        pass
+            except OSError:
+                cut_after.append(time.time() - t0)
+            finally:
+                s.close()
+
+        drippers = [threading.Thread(target=drip, daemon=True) for _ in range(8)]
+        for t in drippers:
+            t.start()
+        self.addCleanup(stop.set)
+        time.sleep(0.5)
+        worst = 0.0
+        t_end = time.time() + 12
+        while time.time() < t_end:
+            t0 = time.time()
+            self.assertTrue(http_get(g.port, '/', timeout=6).startswith('pid='))
+            worst = max(worst, time.time() - t0)
+            time.sleep(0.5)
+        stop.set()
+        self.assertEqual(len(cut_after), 8, 'not every drip connection was cut: %r' % cut_after)
+        # The deadline is armed when a THREAD picks the connection up, not at
+        # accept: a request queued behind busy threads (e.g. legit certbot
+        # runs) must not be cut for waiting. So a drip conn that was queued
+        # (the 2 workers rarely split 8 conns 4/4) is cut at up to about
+        # 2 x deadline; one that got a thread at once within ~deadline + 1s.
+        self.assertLessEqual(min(cut_after), 3 + 2, cut_after)
+        self.assertTrue(all(c < 2 * 3 + 2 for c in cut_after), cut_after)
+        self.assertLess(worst, 5.0)
+        self.assertIn('HAPROXY_MGR_HEADER_DEADLINE', g.output())
+
+    def test_f2_slow_but_complete_headers_within_deadline_are_served(self):
+        g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '4'})
+        s = socket.create_connection(('127.0.0.1', g.port))
+        self.addCleanup(s.close)
+        for part in (b'GET / HTTP/1.1\r\n', b'Host: x\r\n', b'\r\n'):
+            s.sendall(part)
+            time.sleep(1)
+        s.settimeout(5)
+        self.assertIn(b'200 OK', s.recv(4096))
+
+
+def refused(port, wait=2.0):
+    """True if the server closes a fresh connection without answering a
+    complete request (peer filter)."""
+    s = socket.create_connection(('127.0.0.1', port), timeout=wait)
+    try:
+        try:
+            s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
+        except OSError:
+            return True
+        s.settimeout(wait)
+        try:
+            return s.recv(4096) == b''
+        except (ConnectionResetError, BrokenPipeError):
+            return True
+        except socket.timeout:
+            return False
+    finally:
+        s.close()
+
+
+class PeerFilter(unittest.TestCase):
+    """Re-review R1: tenant containers share client-net with haproxy-manager,
+    so the API port must refuse peers that are not loopback / the host path."""
+
+    @classmethod
+    def setUpClass(cls):
+        sys.path.insert(0, REPO_DIR)
+        import haproxy_gunicorn_worker as w
+        cls.w = w
+
+    def test_allowlist_parsing_and_matching(self):
+        nets, problems = self.w.parse_allowed_peers('127.0.0.1,::1,gateway', gateways=['172.18.0.1'])
+        self.assertEqual(problems, [])
+        ok = self.w.peer_allowed
+        self.assertTrue(ok('127.0.0.1', nets))
+        self.assertTrue(ok('::1', nets))
+        self.assertTrue(ok('::ffff:127.0.0.1', nets))
+        self.assertTrue(ok('172.18.0.1', nets))           # docker-proxy / host
+        self.assertFalse(ok('172.18.0.7', nets))          # a tenant container
+        self.assertFalse(ok('127.0.0.2', nets))
+        self.assertFalse(ok('', nets))
+        self.assertFalse(ok('garbage', nets))
+
+    def test_bad_entries_fail_closed_and_are_reported(self):
+        nets, problems = self.w.parse_allowed_peers('nonsense,gateway', gateways=[])
+        self.assertEqual(nets, [])
+        self.assertEqual(len(problems), 2, problems)
+        nets, _ = self.w.parse_allowed_peers('10.0.0.0/8')
+        self.assertTrue(self.w.peer_allowed('10.1.2.3', nets))
+
+    def test_default_gateway_parsed_from_route_table(self):
+        fn = os.path.join(tempfile.mkdtemp(), 'route')
+        with open(fn, 'w') as fh:
+            fh.write('Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\n'
+                     'eth0\t00000000\t010012AC\t0003\t0\t0\t0\t00000000\n'
+                     'eth0\t000012AC\t00000000\t0001\t0\t0\t0\t0000FFFF\n')
+        self.assertEqual(self.w._default_gateways(fn), ['172.18.0.1'])
+
+    def test_filtered_port_refuses_unlisted_peer_other_port_unaffected(self):
+        other = free_port()
+        g = Gunicorn(FIXED, threads=1, extra_ports=(other,), env={
+            'HAPROXY_MGR_PEER_FILTER_PORTS': str(other),
+            'HAPROXY_MGR_API_ALLOWED_PEERS': '10.255.255.1'})
+        self.addCleanup(g.stop)
+        # g.port (unfiltered) works; `other` refuses 127.0.0.1 (not listed).
+        self.assertTrue(http_get(g.port, '/').startswith('pid='))
+        for _ in range(5):
+            self.assertTrue(refused(other), 'filtered port answered an unlisted peer')
+        self.assertTrue(http_get(g.port, '/').startswith('pid='))
+        out = g.output()
+        self.assertIn('Refused 1 connection(s) to the management API', out)  # first one, immediately
+        self.assertEqual(out.count('Refused '), 1, 'reject logging is not rate-limited')
+
+    def test_filtered_port_serves_listed_peer(self):
+        port = free_port()
+        g = Gunicorn(FIXED, threads=1, extra_ports=(port,), env={
+            'HAPROXY_MGR_PEER_FILTER_PORTS': str(port),
+            'HAPROXY_MGR_API_ALLOWED_PEERS': '127.0.0.1,::1'})
+        self.addCleanup(g.stop)
+        self.assertTrue(http_get(port, '/').startswith('pid='))
+        self.assertNotIn('Refused', g.output())
+
+
+class QueuedDripStarvation(unittest.TestCase):
+    """Re-review: with the deadline armed at thread pickup, 40 reconnecting
+    drippers queued behind each other starved legit requests (every probe
+    >30s). Armed as max(enqueue + D, pickup + 1s) they must not."""
+
+    def test_40_reconnecting_drippers_do_not_starve_legit_requests(self):
+        g = Gunicorn(FIXED, threads=4, workers=2, env={})   # production defaults
+        self.addCleanup(g.stop)
+        stop = threading.Event()
+        self.addCleanup(stop.set)
+
+        def drip():
+            while not stop.is_set():
+                try:
+                    s = socket.create_connection(('127.0.0.1', g.port))
+                except OSError:
+                    time.sleep(0.2)
+                    continue
+                try:
+                    req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 5000
+                    for i in range(len(req)):
+                        if stop.is_set():
+                            break
+                        s.send(req[i:i + 1])
+                        time.sleep(1.0)
+                except OSError:
+                    pass
+                finally:
+                    s.close()
+
+        for _ in range(40):
+            threading.Thread(target=drip, daemon=True).start()
+        time.sleep(2)
+        lat = []
+        t_end = time.time() + 25
+        while time.time() < t_end:
+            t0 = time.time()
+            try:
+                http_get(g.port, '/', timeout=30)
+                lat.append(time.time() - t0)
+            except Exception:
+                lat.append(99.0)
+            time.sleep(1)
+        stop.set()
+        lat.sort()
+        median = lat[len(lat) // 2]
+        sys.stderr.write('\n  40 drippers: probes=%d median=%.2fs worst=%.2fs\n'
+                         % (len(lat), median, lat[-1]))
+        # Armed at pickup, every probe here exceeded 30s (total starvation).
+        # With max(enqueue + D, pickup + 1s) queued drippers get ~1s of thread
+        # time, so legit requests keep getting through. They still queue: the
+        # equilibrium wait is ~(drippers/threads) x hold time, about 9s at 40
+        # drippers on 8 threads -- measured, see the round-2 report. The
+        # remaining fix for that is not admitting partial requests to a thread
+        # at all; the peer filter makes it a local-only scenario.
+        # The property under test is "no starvation" (every probe answered
+        # inside 30s). The latency bounds depend on host load: a shared CI
+        # runner measured median 14.97s on 2 probes, so they are only enforced
+        # when HAPROXY_MGR_STRICT_TIMING=1 (quiet local box).
+        self.assertNotIn(99.0, lat, 'a legit request was starved for 30s+')
+        if os.environ.get('HAPROXY_MGR_STRICT_TIMING') == '1':
+            self.assertLess(median, 12.0, lat)
+            self.assertLess(lat[-1], 20.0, lat)
+
+
+class StalledUpstreamDefaultTimeout(unittest.TestCase):
+    """haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
+    client call to an upstream that accepts and never replies must time out."""
+
+    def test_in_process_client_to_stalled_upstream_times_out(self):
+        srv = socket.socket()
+        srv.bind(('127.0.0.1', 0))
+        srv.listen(5)
+        self.addCleanup(srv.close)
+        port = srv.getsockname()[1]
+        accepted = []
+        threading.Thread(target=lambda: accepted.append(srv.accept()),
+                         daemon=True).start()
+        # Import the real module in a child (it changes process-global socket
+        # state and opens its log files at import), then make a client call
+        # with NO explicit timeout -- what a careless future handler would do.
+        probe = textwrap.dedent('''\
+            import logging, os, sys, tempfile, time, urllib.request
+            d = tempfile.mkdtemp()
+            real = logging.FileHandler
+            logging.FileHandler = lambda fn, *a, **k: real(os.path.join(d, os.path.basename(fn)), *a, **k)
+            os.chdir(sys.argv[1]); sys.path.insert(0, sys.argv[1])
+            import haproxy_manager
+            logging.FileHandler = real
+            import socket
+            print('default', socket.getdefaulttimeout())
+            t0 = time.time()
+            try:
+                urllib.request.urlopen('http://127.0.0.1:%s/' % sys.argv[2]).read()
+                print('result returned')
+            except Exception as e:
+                print('result', type(e).__name__, round(time.time() - t0, 1))
+        ''')
+        env = dict(os.environ, HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT='1.5')
+        r = subprocess.run([sys.executable, '-c', probe, REPO_DIR, str(port)],
+                           capture_output=True, text=True, timeout=30, env=env)
+        if 'No module named' in r.stderr and 'flask' in r.stderr.lower():
+            self.skipTest('Flask not installed')
+        self.assertIn('default 1.5', r.stdout, r.stderr)
+        line = [l for l in r.stdout.splitlines() if l.startswith('result')][0]
+        self.assertTrue(line.startswith('result TimeoutError '), line)
+        elapsed = float(line.split()[-1])
+        self.assertLess(elapsed, 5)
+        self.assertTrue(accepted, 'upstream never saw the connection')
+        accepted[0][0].close()
+
+    def test_default_timeout_alone_does_not_cover_gthread_client_sockets(self):
+        # Documents why the custom worker exists: gthread's setblocking(True)
+        # resets any timeout -- including the process default -- to None.
+        old = socket.getdefaulttimeout()
+        socket.setdefaulttimeout(60)
+        try:
+            a, b = socket.socketpair()
+        finally:
+            socket.setdefaulttimeout(old)
+        self.addCleanup(a.close)
+        self.addCleanup(b.close)
+        self.assertEqual(a.gettimeout(), 60)
+        a.setblocking(True)
+        self.assertIsNone(a.gettimeout())
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-runtime-map-contract.py

@@ -0,0 +1,397 @@
+#!/usr/bin/env python3
+"""Contract test: the runtime-map fast path (blocked IPs).
+
+Why this file exists
+--------------------
+`add_ip_to_runtime_map()` and `remove_ip_from_runtime_map()` spent their whole
+existence sending
+
+    add map #0 <ip> 1
+    del map #0 <ip>
+
+to `/tmp/haproxy-cli` and returning True whenever socat exited 0. Neither
+command has ever worked. Two independent defects:
+
+  * **No `@1` prefix.** `/tmp/haproxy-cli` is HAProxy's MASTER CLI socket; map
+    commands are worker commands. The master answers `Unknown command: 'add',
+    but maybe one of the following ones is a better match: ...` -- and **socat
+    still exits 0**, so `result.returncode == 0` was true and the function
+    logged "Added IP x to runtime map".
+  * **`#0` is not a valid map id.** Ids are assigned at config-parse time and
+    move on every config regeneration; on the live edge `blocked_ips.map` is
+    id 37 and `trusted_ips.map` is 10. There is no id 0. Any hardcoded number
+    is wrong -- the map must be referenced by its FILE PATH, which is what
+    haproxy.cfg itself names in `map_ip(/etc/haproxy/blocked_ips.map,0)`.
+
+And a third silence that makes a response-body check alone insufficient:
+`@1 add map #0 <ip> 1` returns an **empty body**, exit 0, and adds nothing
+anywhere -- while `@1 del map #0 <ip>` answers `Unknown map identifier.`. The
+add path can therefore only be trusted after reading the entry back.
+
+IP blocking still worked, because `update_blocked_ips_map()` rewrites
+`/etc/haproxy/blocked_ips.map` and the callers reload HAProxy, which re-reads
+it. The FILE is authoritative; this is the no-reload fast path, and it has
+never once run while reporting that it did.
+
+What it enforces
+----------------
+  1. The command strings actually sent: `@1` prefix first, map referenced by
+     PATH and never by `#<id>`, and the value `1` that
+     `map_ip(...,0) -m int gt 0` requires.
+  2. Every captured rejection is classified as FAILURE (returns False), not
+     success -- including the two that carry no error text at all.
+  3. Success is only reported when the entry reads back in the state asked
+     for. Not the exit status, not an empty reply.
+  4. No source in this repo builds a map command with a `#<id>` reference.
+     Comments may describe the old form; code may not use it.
+
+Runs fully offline: `_cli_send()` is replaced, so no socket, no socat, no
+HAProxy, no network.
+
+Running
+-------
+    python3 scripts/test-runtime-map-contract.py
+"""
+
+import ast
+import io
+import os
+import re
+import sys
+import glob
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+logging.getLogger().setLevel(logging.CRITICAL)
+
+MAP = haproxy_manager.BLOCKED_IPS_MAP_PATH
+IP = '192.0.2.77'
+
+# ---------------------------------------------------------------------------
+# Responses captured VERBATIM from the haproxy-manager container on the live
+# edge (HAProxy 3.0.11, 2026-08-22). socat exited 0 for every single one of
+# them, which is the entire reason none of this is decided on exit status.
+# ---------------------------------------------------------------------------
+
+# What the MASTER socket answers to an unprefixed `add map ...` -- i.e. the
+# reply the old code read as success.
+MASTER_REJECTS_ADD = """\
+Unknown command: 'add', but maybe one of the following ones is a better match:
+  @!<pid>                                 : send a command to the <pid> process
+  @master                                 : send a command to the master process
+  hard-reload                             : achieve a hard-reload (-st) of haproxy
+  reload                                  : achieve a soft-reload (-sf) of haproxy
+  user                                    : lower the level of the current CLI session to user
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+MASTER_REJECTS_DEL = MASTER_REJECTS_ADD.replace("'add'", "'del'")
+
+# `@1 del map #0 <ip>` / `@1 get map /etc/haproxy/nope.map <ip>`.
+UNKNOWN_MAP_IDENTIFIER = 'Unknown map identifier. Please use #<id> or <file>.\n'
+
+# `@1 add map <map> <ip>` with the value omitted (as the old docs showed).
+ADD_MAP_MISSING_VALUE = (
+    "'add map' expects three parameters (map identifier, key and value) or one "
+    "parameter (map identifier) and a payload\n")
+
+# `@1 del map <map> <ip>` for a key the runtime map does not hold.
+KEY_NOT_FOUND = 'Key not found.\n'
+
+# A successful mutation. THIS IS THE WHOLE PROBLEM: it is byte-for-byte what
+# `@1 add map #0 <ip> 1` also returns while adding nothing at all.
+MUTATION_OK = ''
+
+GET_FOUND = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+             'value="1", type="str"\n' % IP)
+GET_NOT_FOUND = 'type=ip, case=sensitive, found=no\n'
+# `@1 get map #0 <ip>` -- "found", but with no value. haproxy.cfg matches with
+# `-m int gt 0`, so an entry like this does NOT block.
+GET_FOUND_NO_VALUE = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+                      'value=none\n' % IP)
+
+SHOW_MAP_SAMPLE = (
+    '0x7f6e5a788700 101.36.109.130 1\n'
+    '0x7f6e5a788780 101.47.140.218 1\n'
+    '0x7f6e5a1fd500 %s 1\n' % IP)
+
+
+class FakeSocket(object):
+    """Replaces _cli_send(). Records every command, answers from a script.
+
+    `script` is a list of (substring, response) pairs, consulted in order; the
+    first whose substring appears in the command wins. Anything unmatched is
+    an explicit test bug, not a silent default.
+    """
+
+    def __init__(self, script):
+        self.script = script
+        self.sent = []
+
+    def __call__(self, command, socket_path, timeout):
+        self.sent.append(command)
+        for needle, response in self.script:
+            if needle in command:
+                return response
+        raise AssertionError('test script has no response for %r' % command)
+
+
+def with_socket(script):
+    """Install a FakeSocket for the duration of a `with` block."""
+    class _Ctx(object):
+        def __enter__(self):
+            self.fake = FakeSocket(script)
+            self._real = haproxy_manager._cli_send
+            haproxy_manager._cli_send = self.fake
+            return self.fake
+
+        def __exit__(self, *exc):
+            haproxy_manager._cli_send = self._real
+            return False
+    return _Ctx()
+
+
+# Every command the happy path needs, in the order the code issues them.
+HAPPY_ADD = [('get map', GET_FOUND), ('add map', MUTATION_OK)]
+HAPPY_DEL = [('get map', GET_NOT_FOUND), ('del map', MUTATION_OK)]
+
+
+class CommandsAreWellFormed(unittest.TestCase):
+    """Guard 1: the exact bytes on the wire.
+
+    Both original defects are visible here and nowhere else -- a `#0` map
+    reference and a missing `@1` are perfectly ordinary-looking Python.
+    """
+
+    def test_add_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_ADD) as fake:
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 add map %s %s 1' % (MAP, IP))
+
+    def test_del_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_DEL) as fake:
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 del map %s %s' % (MAP, IP))
+
+    def test_every_command_is_tried_on_the_worker_first(self):
+        """`/tmp/haproxy-cli` is the MASTER socket; bare map commands 404."""
+        for run in (lambda: haproxy_manager.add_ip_to_runtime_map(IP),
+                    lambda: haproxy_manager.remove_ip_from_runtime_map(IP)):
+            with with_socket(HAPPY_ADD + HAPPY_DEL) as fake:
+                run()
+            for command in fake.sent:
+                self.assertTrue(command.startswith('@1 '),
+                                '%r is missing the @1 worker prefix' % command)
+
+    def test_no_command_references_a_map_by_id(self):
+        """Map ids move on every config regeneration. Path, always."""
+        script = HAPPY_ADD + HAPPY_DEL + [('show map', SHOW_MAP_SAMPLE)]
+        with with_socket(script) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+            haproxy_manager.remove_ip_from_runtime_map(IP)
+            haproxy_manager.runtime_map_keys(MAP)
+        for command in fake.sent:
+            self.assertNotRegex(
+                command, r'\bmap\s+#',
+                '%r references a map by id; ids are not stable' % command)
+            self.assertIn(MAP, command,
+                          '%r does not name the map file' % command)
+
+    def test_add_carries_the_value_the_config_matches_on(self):
+        """`map_ip(...,0) -m int gt 0`: a valueless entry does not block."""
+        self.assertEqual(haproxy_manager.BLOCKED_IPS_MAP_VALUE, '1')
+        with with_socket(HAPPY_ADD) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+        self.assertTrue(fake.sent[0].endswith(' %s 1' % IP),
+                        '%r has no value; HAProxy rejects it' % fake.sent[0])
+
+
+class RejectionIsFailure(unittest.TestCase):
+    """Guard 2+3: nothing may report success unless the map really changed.
+
+    Every response below was returned by the live socket with **exit code 0**.
+    The old code returned True for all of them.
+    """
+
+    def test_master_socket_rejection_of_add_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_master_socket_rejection_of_del_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MASTER_REJECTS_DEL)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_unknown_map_identifier_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket([('get map', GET_FOUND), ('del map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_missing_value_rejection_is_failure(self):
+        """Carries no marker word at all -- caught by 'a mutation says nothing'."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', ADD_MAP_MISSING_VALUE)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_silent_noop_add_is_failure(self):
+        """The `#0` failure mode: accepted, empty reply, nothing added.
+
+        Nothing in the response distinguishes this from success. Only the
+        read-back does -- which is why the read-back is not optional.
+        """
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_add_that_lands_without_a_value_is_failure(self):
+        with with_socket([('get map', GET_FOUND_NO_VALUE), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_del_that_leaves_the_key_behind_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_key_not_found_on_del_is_the_requested_end_state(self):
+        """Not a failure: the runtime map already lacks the key."""
+        with with_socket([('get map', GET_NOT_FOUND), ('del map', KEY_NOT_FOUND)]):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_verified_success_is_reported_as_success(self):
+        with with_socket(HAPPY_ADD):
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket(HAPPY_DEL):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_a_runtime_failure_never_raises_into_the_request_handler(self):
+        """The map file + reload still enforces the block; degrade, don't 500."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertIs(haproxy_manager.add_ip_to_runtime_map(IP), False)
+
+    def test_mutations_are_checked_for_an_empty_body_not_a_marker_list(self):
+        """_HAPROXY_CLI_ERROR_MARKERS can only know rejections already seen."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            with with_socket([('add map', 'something nobody has ever seen\n')]):
+                haproxy_manager.haproxy_cli('add map %s x 1' % MAP,
+                                            worker=True, expect_empty=True)
+
+    def test_the_new_error_markers_are_recognised(self):
+        for response in (UNKNOWN_MAP_IDENTIFIER, KEY_NOT_FOUND,
+                         MASTER_REJECTS_ADD):
+            self.assertTrue(haproxy_manager._cli_response_is_error(response),
+                            '%r must be classified as an error' % response[:40])
+        for response in (GET_FOUND, GET_NOT_FOUND, SHOW_MAP_SAMPLE):
+            self.assertFalse(haproxy_manager._cli_response_is_error(response),
+                             '%r is data, not an error' % response[:40])
+
+    def test_socat_exit_zero_carries_no_information(self):
+        """FakeSocket never signals failure any other way, and neither did socat."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]) as fake:
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertTrue(fake.sent, 'the command was sent and "succeeded" at the '
+                                   'process level; only the body says otherwise')
+
+
+class ReadBack(unittest.TestCase):
+    """The read-back primitives the guarantees above rest on."""
+
+    def test_lookup_reports_found_with_value(self):
+        with with_socket([('get map', GET_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (True, '1'))
+
+    def test_lookup_reports_not_found(self):
+        with with_socket([('get map', GET_NOT_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (False, None))
+
+    def test_lookup_raises_on_a_rejected_reference(self):
+        with with_socket([('get map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_lookup(MAP, IP)
+
+    def test_keys_parses_show_map_output(self):
+        with with_socket([('show map', SHOW_MAP_SAMPLE)]):
+            self.assertEqual(
+                haproxy_manager.runtime_map_keys(MAP),
+                {'101.36.109.130', '101.47.140.218', IP})
+
+    def test_empty_map_is_not_a_rejection(self):
+        with with_socket([('show map', '')]):
+            self.assertEqual(haproxy_manager.runtime_map_keys(MAP), set())
+
+    def test_rejected_show_map_still_raises(self):
+        with with_socket([('show map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_keys(MAP)
+
+
+MAP_BY_ID_RE = re.compile(r'\b(?:add|del|clear|show|get)\s+map\s+#')
+
+
+class NoSourceBuildsAMapIdCommand(unittest.TestCase):
+    """Guard 4: `map #<id>` may be described in comments, never executed.
+
+    Scanning string literals rather than raw text is deliberate -- the whole
+    reason this bug is documented at length in the source is so the next reader
+    does not reintroduce it, and a plain grep would fail on those comments.
+    """
+
+    def test_no_python_string_literal_builds_a_map_id_command(self):
+        tree = ast.parse(io.open('haproxy_manager.py', encoding='utf-8').read())
+        offenders = [
+            node.value for node in ast.walk(tree)
+            if isinstance(node, ast.Constant) and isinstance(node.value, str)
+            and MAP_BY_ID_RE.search(node.value)
+            and not (ast.get_docstring(tree) == node.value)
+        ]
+        # Docstrings are string literals too; exclude any literal that is a
+        # docstring of a module/class/function.
+        docstrings = set()
+        for node in ast.walk(tree):
+            if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef,
+                                 ast.AsyncFunctionDef)):
+                doc = ast.get_docstring(node, clean=False)
+                if doc:
+                    docstrings.add(doc)
+        offenders = [o for o in offenders if o not in docstrings]
+        self.assertEqual(offenders, [],
+                         'these string literals build a map command with an '
+                         'unstable #<id> reference')
+
+    def test_no_shell_or_template_code_line_uses_a_map_id(self):
+        targets = (glob.glob('scripts/*.sh') + glob.glob('templates/*.tpl')
+                   + ['Dockerfile'])
+        offenders = []
+        for path in targets:
+            if not os.path.exists(path):
+                continue
+            for lineno, line in enumerate(
+                    io.open(path, encoding='utf-8').read().splitlines(), 1):
+                if line.lstrip().startswith('#'):
+                    continue  # a comment describing the old form is fine
+                if MAP_BY_ID_RE.search(line):
+                    offenders.append('%s:%d: %s' % (path, lineno, line.strip()))
+        self.assertEqual(offenders, [],
+                         'map ids are assigned at config-parse time and move; '
+                         'reference the map file by path')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-stick-table-contract.py

@@ -0,0 +1,475 @@
+#!/usr/bin/env python3
+"""Contract test: what the stick tables STORE vs what the consumers READ.
+
+Why this file exists
+--------------------
+`/api/security/stats` and `scripts/show-tarpit-ips.sh` spent their whole
+existence reporting "Scan Count", "offense count" and "BLOCKED" figures parsed
+out of `gpc0` and `gpc1`. **No stick table in this repo has ever stored a
+general-purpose counter.** Every one of those numbers was fabricated, and an
+operator was making decisions on them.
+
+Nothing caught it, because each layer failed silently in a different way:
+
+  * `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
+    and the loop `continue`d -- so the endpoint answered `active_threats: 0`
+    with an empty list. "No threats" and "the parser is broken" looked
+    identical.
+  * The command went to the MASTER CLI socket without the `@1` worker prefix.
+    HAProxy answered `Unknown command: 'show', but maybe one of the following
+    ones is a better match: ...` and **socat still exited 0**, so the
+    `returncode != 0` guard never fired. The reported `total_tracked_ips` was
+    the line count of that help text (8) while the real table held 388 entries.
+  * The shell consumers wrote `gpc0=${gpc0:-0}`, so a field that does not exist
+    rendered as a confident zero.
+
+The durable fix is not "parse better" -- it is making the template and its
+consumers unable to drift apart without something going red. That is this file.
+
+What it enforces
+----------------
+  1. `STICK_TABLE_FIELD_CONTRACT` in haproxy_manager.py equals, exactly and in
+     both directions, the `store` clauses in the rendered templates.
+  2. Every shell consumer's `EXPECTED_FIELDS=(...)` array equals the contract.
+  3. A captured sample of REAL `show table` output from the live edge parses to
+     exactly the contract's fields plus the entry metadata -- so the contract
+     describes reality, not just itself.
+  4. The loud-failure behaviour: `read_stick_table()` RAISES on a rejected
+     command, on a non-table response, and on a row missing a contract field.
+     It must never answer zeros. Guard 4 is the one that would have caught the
+     original bug on day one.
+  5. No `store` clause names a general-purpose counter, and no template tracks
+     one -- the state this repo is actually in, asserted rather than assumed.
+
+Assertions about the TEMPLATES go through `rule_lines()`, which strips comments
+before matching. These templates quote their own rules in prose at length; a
+bare `assertIn` over the rendered text passes just as happily against a rule
+that has been commented out. Same lesson, and same helper, as
+scripts/test-wpadmin-gate.py.
+
+Runs fully offline -- no HAProxy, no socket, no network.
+
+Running
+-------
+    python3 scripts/test-stick-table-contract.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# A captured sample of REAL output, so this test can assert against reality
+# without a live socket.
+#
+# Provenance: `echo "@1 show table web" | socat stdio /tmp/haproxy-cli` inside
+# the haproxy-manager container on whp01, 2026-08-22, HAProxy 3.0.11. Rows
+# trimmed for length; format byte-for-byte as emitted.
+#
+# Note what is and is NOT here: no gpc0, no gpc1, no gpc(N), no gpc_rate, no
+# glitch_rate. Note also that field windows come back in MILLISECONDS
+# (`conn_rate(10000)`), not the `10s` the template writes -- a consumer that
+# labels the raw number "10s" is off by a factor of 1000.
+# ---------------------------------------------------------------------------
+LIVE_TABLE_SAMPLE = """\
+# table: web, type: ip, size:204800, used:388
+0x7f6e5447e728: key=17.58.57.102 use=0 exp=368842 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e541cf7c8: key=43.173.182.9 use=0 exp=171404 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e5547f528: key=95.129.255.180 use=0 exp=590639 shard=0 conn_rate(10000)=1 conn_cur=0 http_req_rate(10000)=1 http_err_rate(30000)=0
+0x7f6e5447e488: key=5.9.105.254 use=0 exp=556277 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=1
+"""
+
+# The verbatim reply the MASTER CLI socket gives to an unprefixed worker
+# command. Captured the same way. socat exits 0 on this -- which is the entire
+# reason haproxy_cli() inspects the body.
+MASTER_SOCKET_REJECTION = """\
+Unknown command: 'show', but maybe one of the following ones is a better match:
+  show cli level                          : display the level of the current CLI session
+  show cli sockets                        : dump list of cli sockets
+  show proc                               : show processes status
+  show startup-logs                       : report logs emitted during HAProxy startup
+  show version                            : show version of the current process
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+# Anything matching these in a `store` clause is a general-purpose counter.
+GPC_PATTERN = re.compile(r'\bgpc|glitch')
+
+# Shell consumers that must declare their field expectations as a single
+# EXPECTED_FIELDS array, and the table each one reads.
+SHELL_CONSUMERS = {
+    'scripts/show-edge-ip-rates.sh': 'web',
+}
+
+EXPECTED_FIELDS_RE = re.compile(r'^\s*EXPECTED_FIELDS=\(([^)]*)\)\s*$', re.M)
+
+
+def rule_lines(cfg, needle):
+    """Comment-stripped config lines containing `needle`.
+
+    A line that is entirely a comment is dropped; a line mixing config with a
+    trailing comment is truncated at the first ' #' before matching. Without
+    this, every assertion below would pass against a rule that had been
+    commented out but whose text survived in the surrounding prose -- and these
+    templates quote their own rules in prose constantly. See
+    scripts/test-wpadmin-gate.py, where a mutation audit proved the point.
+    """
+    out = []
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if stripped and not stripped.startswith('#'):
+            code = stripped.split(' #', 1)[0].rstrip()
+            if code and needle in code:
+                out.append(code)
+    return out
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_security_tables():
+    return haproxy_manager.template_env.get_template('hap_security_tables.tpl').render()
+
+
+def stick_tables_from(cfg):
+    """{table name: (field, ...)} for every stick-table declared in `cfg`.
+
+    A stick-table takes the name of the frontend/backend/listen section that
+    declares it -- that name is what `show table <name>` wants, so the section
+    header is part of the contract, not incidental. Section headers and
+    stick-table lines are both read comment-stripped.
+    """
+    tables = {}
+    section = None
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if not stripped or stripped.startswith('#'):
+            continue
+        code = stripped.split(' #', 1)[0].rstrip()
+        header = re.match(r'^(frontend|backend|listen)\s+(\S+)', code)
+        if header:
+            section = header.group(2)
+            continue
+        if code.startswith('stick-table'):
+            store = re.search(r'\bstore\s+(\S+)', code)
+            if not store:
+                raise AssertionError(
+                    'stick-table in section %r has no `store` clause: %r'
+                    % (section, code))
+            if section is None:
+                raise AssertionError(
+                    'stick-table declared outside any section: %r' % code)
+            # store is a comma-separated list; each item is name or name(window)
+            fields = tuple(item.split('(')[0]
+                           for item in store.group(1).split(','))
+            tables[section] = fields
+    return tables
+
+
+def all_template_stick_tables():
+    tables = {}
+    for cfg in (render_listener(), render_security_tables()):
+        for name, fields in stick_tables_from(cfg).items():
+            if name in tables:
+                raise AssertionError('stick table %r declared twice' % name)
+            tables[name] = fields
+    return tables
+
+
+class StickTableContract(unittest.TestCase):
+    """Guard 1 + 5: the templates and the Python contract, held together."""
+
+    def setUp(self):
+        self.templates = all_template_stick_tables()
+        self.contract = haproxy_manager.STICK_TABLE_FIELD_CONTRACT
+
+    def test_templates_actually_declare_stick_tables(self):
+        """Guard the guard: an empty parse would make every other check vacuous."""
+        self.assertTrue(self.templates,
+                        'parsed no stick tables out of the templates at all -- '
+                        'stick_tables_from() is broken, not the templates')
+
+    def test_same_table_names(self):
+        self.assertEqual(
+            sorted(self.templates), sorted(self.contract),
+            'STICK_TABLE_FIELD_CONTRACT and the templates disagree on WHICH '
+            'stick tables exist. Add/remove the table in both places.')
+
+    def test_same_fields_per_table(self):
+        for table in sorted(self.templates):
+            with self.subTest(table=table):
+                self.assertEqual(
+                    sorted(self.templates[table]),
+                    sorted(self.contract.get(table, ())),
+                    "stick table %r stores %s but STICK_TABLE_FIELD_CONTRACT "
+                    "claims %s. Whichever is wrong, a consumer is about to read "
+                    "a field that is never populated -- which is the bug this "
+                    "test exists for." % (table,
+                                          list(self.templates[table]),
+                                          list(self.contract.get(table, ()))))
+
+    def test_web_table_is_the_one_the_api_reads(self):
+        self.assertIn('web', self.contract)
+        self.assertIn('web', self.templates)
+
+    def test_no_general_purpose_counters_are_stored(self):
+        """The state of the world today, asserted rather than assumed.
+
+        If a gpc/glitch counter is ever genuinely added to a template, this
+        test is the place to update -- and updating it forces whoever does so
+        to also add the field to STICK_TABLE_FIELD_CONTRACT (test_same_fields_
+        per_table) and to the shell consumers (test_shell_consumers_match_
+        contract). That chain is the point: a counter cannot appear in a
+        consumer without existing in the table, and cannot appear in the table
+        without the consumers being updated.
+        """
+        for table, fields in self.templates.items():
+            for field in fields:
+                self.assertIsNone(
+                    GPC_PATTERN.search(field),
+                    'stick table %r now stores %r. Update this test, '
+                    'STICK_TABLE_FIELD_CONTRACT, and every consumer.'
+                    % (table, field))
+
+    def test_track_sc_counters_have_a_table_each(self):
+        """Every `track-scN ... table X` names a table that really exists.
+
+        A typo here is invisible to `haproxy -c` only in the sense that it is
+        NOT -- but it is invisible to the consumers, which would query a table
+        that is never written.
+        """
+        cfg = render_listener()
+        for line in rule_lines(cfg, 'track-sc'):
+            named = re.search(r'\btable\s+(\S+)', line)
+            if named:
+                self.assertIn(
+                    named.group(1), self.templates,
+                    'track-sc rule references undeclared table %r: %r'
+                    % (named.group(1), line))
+
+    def test_sc_counter_indices_fit_haproxys_limit(self):
+        """sc0/sc1/sc2 are all HAProxy gives us by default.
+
+        `tune.stick-counters` defaults to 3. A `track-sc3` without raising it
+        is a config-time failure, and the templates' own comments assume the
+        limit -- so assert it rather than leaving it as folklore.
+        """
+        cfg = render_listener() + '\n' + render_security_tables()
+        raised = rule_lines(cfg, 'tune.stick-counters')
+        limit = 3
+        if raised:
+            limit = int(re.search(r'(\d+)', raised[-1]).group(1))
+        for line in rule_lines(cfg, 'track-sc'):
+            idx = int(re.search(r'track-sc(\d+)', line).group(1))
+            self.assertLess(
+                idx, limit,
+                'track-sc%d exceeds tune.stick-counters (%d): %r'
+                % (idx, limit, line))
+
+
+class ShellConsumerContract(unittest.TestCase):
+    """Guard 2: the shell consumers cannot drift from the contract."""
+
+    def test_shell_consumers_match_contract(self):
+        for path, table in sorted(SHELL_CONSUMERS.items()):
+            with self.subTest(script=path):
+                full = os.path.join(MODULE_DIR, path)
+                self.assertTrue(os.path.exists(full),
+                                '%s is missing; it is a declared consumer of '
+                                'stick table %r' % (path, table))
+                with open(full) as fh:
+                    src = fh.read()
+                m = EXPECTED_FIELDS_RE.search(src)
+                self.assertIsNotNone(
+                    m, '%s must declare its field expectations once as a '
+                       'single-line `EXPECTED_FIELDS=(a b c)` array so this '
+                       'test can hold it to the template' % path)
+                declared = sorted(m.group(1).split())
+                self.assertEqual(
+                    declared,
+                    sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table]),
+                    '%s reads %s but stick table %r stores %s'
+                    % (path, declared, table,
+                       sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table])))
+
+    def test_retired_script_no_longer_parses_phantom_counters(self):
+        """show-tarpit-ips.sh may explain gpc0/gpc1; it may not extract them.
+
+        The shim is allowed -- encouraged -- to name the fields in prose so an
+        operator who runs it learns why its numbers went away. What it must not
+        do is go back to pulling values out of them.
+        """
+        path = os.path.join(MODULE_DIR, 'scripts/show-tarpit-ips.sh')
+        if not os.path.exists(path):
+            self.skipTest('show-tarpit-ips.sh has been removed outright')
+        with open(path) as fh:
+            lines = fh.readlines()
+        for raw in lines:
+            stripped = raw.strip()
+            if not stripped or stripped.startswith('#'):
+                continue
+            code = stripped.split(' #', 1)[0]
+            self.assertIsNone(
+                re.search(r"grep -o ['\"]?gpc|gpc[0-9]*=\$|sc_get_gpc|sc-inc-gpc", code),
+                'show-tarpit-ips.sh is extracting a general-purpose counter '
+                'again: %r' % stripped)
+
+
+class LiveSampleParses(unittest.TestCase):
+    """Guard 3: the contract describes real HAProxy output, not just itself."""
+
+    def test_header_parses(self):
+        header, entries = self._read()
+        self.assertEqual(header['name'], 'web')
+        self.assertEqual(header['type'], 'ip')
+        self.assertEqual(header['size'], 204800)
+        self.assertEqual(header['used'], 388)
+        self.assertEqual(len(entries), 4)
+
+    def test_sample_fields_are_exactly_contract_plus_metadata(self):
+        expected = set(haproxy_manager.STICK_TABLE_FIELD_CONTRACT['web'])
+        expected |= set(haproxy_manager.STICK_TABLE_ENTRY_META)
+        _, entries = self._read()
+        for line, fields in entries:
+            self.assertEqual(
+                set(fields), expected,
+                'real `show table web` output carries %s, contract+metadata '
+                'expects %s. Row: %r'
+                % (sorted(fields), sorted(expected), line))
+
+    def test_key_is_the_ip_not_the_allocation_pointer(self):
+        """The original bug read parts[0] -- the `0x...:` pointer -- as the IP."""
+        _, entries = self._read()
+        ips = [f['key']['value'] for _, f in entries]
+        self.assertIn('95.129.255.180', ips)
+        for ip in ips:
+            self.assertFalse(ip.startswith('0x'),
+                             'parsed a memory address as an IP: %r' % ip)
+
+    def test_windows_are_milliseconds(self):
+        """HAProxy reports `conn_rate(10000)` for a `conn_rate(10s)` store.
+
+        Asserted because labelling that raw 10000 as "10s" (or as seconds) is
+        an easy and completely silent way to be wrong by 1000x in the panel.
+        """
+        _, entries = self._read()
+        _, fields = entries[0]
+        self.assertEqual(fields['conn_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_req_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_err_rate']['window_ms'], 30000)
+        self.assertIsNone(fields['conn_cur']['window_ms'],
+                          'conn_cur is a gauge, not a rate; it has no window')
+
+    def test_values_are_the_real_ones(self):
+        _, entries = self._read()
+        by_ip = {f['key']['value']: f for _, f in entries}
+        self.assertEqual(by_ip['95.129.255.180']['http_req_rate']['value'], '1')
+        self.assertEqual(by_ip['5.9.105.254']['http_err_rate']['value'], '1')
+        self.assertEqual(by_ip['17.58.57.102']['http_req_rate']['value'], '0')
+
+    def _read(self):
+        return _read_table_from(LIVE_TABLE_SAMPLE)
+
+
+class FailsLoudly(unittest.TestCase):
+    """Guard 4: every way this can go wrong must raise, never return zeros.
+
+    This is the guard that would have caught the original bug immediately. Each
+    case below is a real response the old code accepted silently.
+    """
+
+    def test_master_socket_rejection_is_not_data(self):
+        """The exact reply that used to be reported as `total_tracked_ips: 8`."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(MASTER_SOCKET_REJECTION)
+        self.assertIn('not a stick-table dump', str(ctx.exception))
+
+    def test_socat_exit_zero_does_not_mean_success(self):
+        """haproxy_cli() must reject on the BODY, not the exit status.
+
+        socat returns 0 for every response above -- the rejection is only ever
+        visible in the text.
+        """
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error(MASTER_SOCKET_REJECTION))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('No such table\n'))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('Permission denied\n'))
+        self.assertFalse(
+            haproxy_manager._cli_response_is_error(LIVE_TABLE_SAMPLE))
+
+    def test_missing_contract_field_raises_and_names_it(self):
+        """A field the table stopped storing must not silently become 0."""
+        degraded = LIVE_TABLE_SAMPLE.replace(' http_err_rate(30000)=0', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(degraded)
+        msg = str(ctx.exception)
+        self.assertIn('http_err_rate', msg,
+                      'the error must name the missing field')
+        self.assertIn('drifted', msg,
+                      'the error must say what actually went wrong')
+
+    def test_empty_response_raises(self):
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            _read_table_from('')
+
+    def test_row_without_key_raises(self):
+        broken = LIVE_TABLE_SAMPLE.replace('key=17.58.57.102 ', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(broken)
+        self.assertIn('no key=', str(ctx.exception))
+
+    def test_unknown_table_raises_before_touching_the_socket(self):
+        """Querying a table with no contract is a programming error, not a 0."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            haproxy_manager.read_stick_table('does_not_exist')
+        self.assertIn('no field contract', str(ctx.exception))
+
+    def test_empty_table_is_not_an_error(self):
+        """A table with zero entries is a legitimate, distinguishable result."""
+        header, entries = _read_table_from(
+            '# table: web, type: ip, size:204800, used:0\n')
+        self.assertEqual(header['used'], 0)
+        self.assertEqual(entries, [])
+
+
+def _read_table_from(response, table='web'):
+    """Run read_stick_table() against a canned response instead of a socket."""
+    real = haproxy_manager.haproxy_cli
+    haproxy_manager.haproxy_cli = lambda cmd, worker=False, timeout=None: response
+    try:
+        return haproxy_manager.read_stick_table(table)
+    finally:
+        haproxy_manager.haproxy_cli = real
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-trusted-proxy-gate.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Regression tests for the trusted-proxy header gate in hap_listener.tpl.
+
+Why this file exists
+--------------------
+txn.real_ip was derived from CF-Connecting-IP / X-Real-IP / X-Forwarded-For
+with no check on the peer, so any direct client could assert any client IP.
+That variable drives rate limiting, the trusted-IP whitelist, the wp-login
+brute-force table and cookie challenge, the wp-json/batch/v1 virtual patch,
+IP blocking, and Coraza's src-ip -- so a spoofed header bypassed all of them.
+
+These tests pin the invariant that the header strip is rendered BEFORE any
+real-IP resolution. Ordering is the whole fix: a del-header emitted after the
+set-var chain would parse fine, validate fine, and do nothing.
+
+Running
+-------
+    python3 scripts/test-trusted-proxy-gate.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+GATED_HEADERS = ('CF-Connecting-IP', 'X-Real-IP', 'X-Forwarded-For')
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+class TrustedProxyGate(unittest.TestCase):
+
+    def setUp(self):
+        self.cfg = render_listener()
+
+    def test_trusted_proxy_acl_is_defined(self):
+        self.assertIn('acl from_trusted_proxy src', self.cfg)
+        self.assertIn('/etc/haproxy/cloudflare_ips.list', self.cfg)
+        self.assertIn('/etc/haproxy/trusted_proxies.list', self.cfg)
+
+    def test_each_header_is_deleted_for_untrusted_peers(self):
+        for header in GATED_HEADERS:
+            with self.subTest(header=header):
+                pattern = (r'http-request\s+del-header\s+%s\s+if\s+!from_trusted_proxy'
+                           % re.escape(header))
+                self.assertRegex(self.cfg, pattern)
+
+    def test_strip_precedes_real_ip_resolution(self):
+        """The ordering invariant. A strip after the set-var chain is a no-op."""
+        last_strip = max(
+            self.cfg.index('del-header %s' % h) for h in GATED_HEADERS
+        )
+        first_setvar = self.cfg.index('set-var(txn.real_ip)')
+        self.assertLess(
+            last_strip, first_setvar,
+            'del-header rules must be rendered before set-var(txn.real_ip)')
+
+    def test_src_fallback_still_present(self):
+        """Direct clients must fall through to the real TCP peer."""
+        self.assertIn('set-var(txn.real_ip) src', self.cfg)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-waf-body-inspection.py

@@ -0,0 +1,382 @@
+#!/usr/bin/env python3
+"""Regression tests for WAF request-body inspection (Coraza via SPOE).
+
+Why this file exists
+--------------------
+Measured 2026-10-09 on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0
+(report: waf-body-hardening-report.md):
+
+  * Without `option http-buffer-request`, send-spoe-group fires at header
+    time and `req.body` holds only what happened to arrive with the headers.
+    A 14-byte `content=$a($b)` POST (CRS 933180, an ENFORCED rule) passed
+    uninspected when its body was written in a second send(); a body sent 2s
+    late, or after `Expect: 100-continue`, was never inspected at all.
+  * Coraza+CRS costs ~3.3 ms per KB of body. The SPOE `timeout processing`
+    is 100 ms and FAILS OPEN, so raising tune.bufsize to 32 KB turned a
+    payload at byte 0 of a 30 KB base64 body into a bypass. bufsize is a
+    security limit, not a performance knob.
+  * coraza-spoa's SPOP library refuses frames > 65535 bytes, and Coraza's
+    SecRequestBodyLimitAction Reject 413s even under DetectionOnly, so the
+    three limits must stay ordered.
+  * The X-WHP-WAF-Body stamp drives enforcement of body-parse errors
+    (200002/200003) in WHP's generated Coraza rules. A client must never be
+    able to set it, and a backend must never see it.
+
+These tests pin the rendered config. That HAProxy ACCEPTS it is
+validate-rendered-config.py's job; that it BEHAVES is the live measurement.
+
+Running
+-------
+    python3 scripts/test-waf-body-inspection.py
+"""
+
+import logging
+import os
+import re
+import sys
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+SPOA_FRAME_CAP = 65535          # dropmorepackets/haproxy-go v0.0.8 maxFrameSize
+CORAZA_CONFIG = os.path.join(MODULE_DIR, 'coraza-spoa', 'config.yaml')
+
+
+def render(template, coraza):
+    kw = dict(crt_path='/etc/haproxy/certs', suspension_enabled=False,
+              coraza_spoe_backend='coraza-spoa:9000' if coraza else None,
+              cluster_secret=None, syslog_target='172.18.0.1:514')
+    return haproxy_manager.template_env.get_template(template).render(**kw)
+
+
+def active_lines(text):
+    """Config lines with comments and blanks stripped (comments mention the
+    directives by name, so a raw substring check would pass on prose)."""
+    out = []
+    for line in text.splitlines():
+        s = line.strip()
+        if s and not s.startswith('#'):
+            out.append(s)
+    return out
+
+
+def index_of(lines, pattern):
+    for i, line in enumerate(lines):
+        if re.search(pattern, line):
+            return i
+    return -1
+
+
+class ListenerWithCoraza(unittest.TestCase):
+
+    def setUp(self):
+        self.lines = active_lines(render('hap_listener.tpl', coraza=True))
+
+    def test_body_is_buffered_before_inspection(self):
+        self.assertIn('option http-buffer-request', self.lines)
+
+    def test_stamp_order_strip_set_inspect_strip(self):
+        strip1 = index_of(self.lines, r'^http-request del-header X-WHP-WAF-Body$')
+        size = index_of(self.lines, r'^http-request set-var\(txn\.waf_body_size\) req\.body_size if \{ req\.hdr_cnt\(content-length\) eq 1 \}$')
+        complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
+        partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if \{ var\(txn\.waf_body_size\) -m found \} !\{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
+        hdrs_len = index_of(self.lines, r'^http-request set-var\(txn\.waf_hdrs_len\) req\.hdrs,length if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \}$')
+        nocl_complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_hdrs_len\) -m found \} \{ req\.body_len,add\(txn\.waf_hdrs_len\) lt (\d+) \}$')
+        nocl_partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \} !\{ var\(txn\.waf_body\) -m found \}$')
+        stamp = index_of(self.lines, r'^http-request set-header X-WHP-WAF-Body %\[var\(txn\.waf_body\)\] if \{ var\(txn\.waf_body\) -m found \}$')
+        spoe = index_of(self.lines, r'^http-request send-spoe-group coraza coraza-req$')
+        deny = index_of(self.lines, r'^http-request return status 403 .*var\(txn\.coraza\.action\) -m str deny')
+        dup_ct = index_of(self.lines, r'^http-request deny deny_status 400 if \{ req\.fhdr_cnt\(content-type\) gt 1 \}$')
+        self.assertGreaterEqual(dup_ct, 0, 'duplicate Content-Type must be rejected at the edge')
+        self.assertLess(dup_ct, spoe, 'and before Coraza is asked')
+        for name, idx in (('client-copy strip', strip1), ('size var', size), ('complete', complete),
+                          ('partial', partial), ('no-CL header length', hdrs_len),
+                          ('no-CL complete', nocl_complete),
+                          ('no-CL partial', nocl_partial), ('stamp', stamp),
+                          ('send-spoe-group', spoe), ('deny', deny)):
+            self.assertGreaterEqual(idx, 0, f'{name} rule missing from the rendered listener')
+        # Client copies stripped BEFORE we stamp, stamp BEFORE the SPOE call.
+        self.assertLess(strip1, size)
+        self.assertLess(size, complete)
+        # The no-CL `partial` fallback relies on "-m found" of an earlier
+        # `complete`: order is load-bearing.
+        self.assertLess(size, hdrs_len)
+        self.assertLess(hdrs_len, nocl_complete)
+        self.assertLess(nocl_complete, nocl_partial)
+        self.assertLess(max(complete, partial, nocl_partial), stamp)
+        self.assertLess(stamp, spoe)
+        # Stripped again AFTER the SPOE call so no backend ever sees it.
+        strip2 = [i for i, l in enumerate(self.lines)
+                  if l == 'http-request del-header X-WHP-WAF-Body' and i > spoe]
+        self.assertTrue(strip2, 'X-WHP-WAF-Body must be deleted again after send-spoe-group')
+        self.assertLess(strip2[0], deny)
+
+    def test_no_content_length_complete_cannot_have_filled_the_buffer(self):
+        """Without a Content-Length, `complete` is inferred from "the buffer
+        cannot have been full". body + headers thresholds must sit well below
+        bufsize - maxrewrite (1024), or a truncated chunked body would be
+        stamped complete and a parse error on it would 403."""
+        line = [l for l in self.lines if 'str(complete) if { var(txn.waf_hdrs_len)' in l][0]
+        total_lt = int(re.search(r'add\(txn\.waf_hdrs_len\) lt (\d+)', line).group(1))
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
+        self.assertLessEqual(total_lt, (bufsize - 1024) * 0.85)
+
+    def test_complete_and_partial_are_mutually_exclusive_conditions(self):
+        """With a Content-Length, `complete` and `partial` must be the same
+        condition, negated -- an inverted or loosened `complete` would enforce
+        parse errors on truncated bodies (403 on large legitimate posts)."""
+        c = [l for l in self.lines if 'str(complete) if { var(txn.waf_body_size)' in l]
+        p = [l for l in self.lines if 'str(partial) if { var(txn.waf_body_size)' in l]
+        self.assertEqual((len(c), len(p)), (1, 1))
+        c, p = c[0], p[0]
+        self.assertEqual(c.split(' if ', 1)[1].replace('!{', '{'),
+                         p.split(' if ', 1)[1].replace('!{', '{'))
+        self.assertIn('!{ req.body_len', p)
+        self.assertNotIn('!{', c)
+
+    def test_stamp_never_set_from_client_input(self):
+        """The only set-header for the stamp is from our own var."""
+        sets = [l for l in self.lines if re.search(r'set-header\s+X-WHP-WAF-Body', l, re.I)]
+        self.assertEqual(len(sets), 1)
+        self.assertIn('%[var(txn.waf_body)]', sets[0])
+        self.assertFalse([l for l in self.lines if re.search(r'add-header\s+X-WHP-WAF-Body', l, re.I)])
+
+    def test_access_log_carries_coverage_fields(self):
+        lf = [l for l in self.lines if l.startswith('log-format ')]
+        self.assertEqual(len(lf), 1)
+        self.assertIn('wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)]', lf[0])
+
+    def test_access_log_carries_waf_verdict_fields(self):
+        """wrule/wrules/wscore are appended LAST so any consumer anchored on
+        the earlier fields keeps working."""
+        lf = [l for l in self.lines if l.startswith('log-format ')]
+        self.assertEqual(len(lf), 1)
+        self.assertTrue(lf[0].endswith(
+            'wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] '
+            'wscore=%[var(txn.coraza.anomaly_score)]"'))
+
+    def test_fail_open_preserved(self):
+        """Nothing may deny on var(txn.coraza.error): an SPOA outage must not
+        503 customer sites."""
+        self.assertFalse([l for l in self.lines if 'txn.coraza.error' in l and
+                          re.search(r'\b(deny|reject|return|silent-drop|tarpit)\b', l)])
+
+
+class ListenerWithoutCoraza(unittest.TestCase):
+
+    def test_waf_off_config_has_no_body_buffering(self):
+        lines = active_lines(render('hap_listener.tpl', coraza=False))
+        self.assertNotIn('option http-buffer-request', lines)
+        self.assertFalse([l for l in lines if 'X-WHP-WAF-Body' in l])
+        self.assertFalse([l for l in lines if 'send-spoe-group' in l])
+
+    def test_waf_off_header_keeps_default_bufsize(self):
+        lines = active_lines(render('hap_header.tpl', coraza=False))
+        self.assertFalse([l for l in lines if l.startswith('tune.bufsize')])
+
+
+class BufsizeCeilings(unittest.TestCase):
+
+    def setUp(self):
+        lines = active_lines(render('hap_header.tpl', coraza=True))
+        found = [l for l in lines if l.startswith('tune.bufsize')]
+        self.assertEqual(len(found), 1, 'tune.bufsize must be pinned when Coraza is on')
+        self.bufsize = int(found[0].split()[1])
+        spoe = active_lines(haproxy_manager.template_env.get_template(
+            'hap_coraza_spoe_engine.tpl').render())
+        m = [re.match(r'timeout\s+processing\s+(\d+)ms$', l) for l in spoe]
+        m = [x for x in m if x]
+        self.assertEqual(len(m), 1)
+        self.processing_ms = int(m[0].group(1))
+
+    def test_bufsize_fits_the_spoe_time_budget(self):
+        """~3.3 ms/KB measured; 31 KB took 104 ms against a 100 ms timeout and
+        FAILED OPEN. Keep the worst case at about half the budget."""
+        worst_ms = (self.bufsize / 1024.0) * 3.3
+        self.assertLessEqual(worst_ms, self.processing_ms * 0.6,
+                             f'tune.bufsize {self.bufsize} => ~{worst_ms:.0f} ms of Coraza time '
+                             f'against a {self.processing_ms} ms fail-open timeout')
+
+    def test_bufsize_below_spop_frame_cap(self):
+        self.assertLess(self.bufsize, SPOA_FRAME_CAP)
+
+    def test_body_arg_is_the_full_buffered_body(self):
+        """The message must keep sending req.body unsliced: the `complete`
+        stamp compares req.body_len to Content-Length, so a slice would make
+        a truncated body look complete to Coraza."""
+        spoe = haproxy_manager.template_env.get_template('hap_coraza_spoe_engine.tpl').render()
+        args = [l.strip() for l in spoe.splitlines() if l.strip().startswith('args ')]
+        self.assertEqual(len(args), 1)
+        self.assertTrue(args[0].endswith(' body=req.body'))
+
+
+class CorazaTransactionIdIsTheRequestReference(unittest.TestCase):
+    """Coraza's transaction ID must be HAProxy's unique-id, so the reference a
+    blocked customer quotes (403-waf.html / X-Request-Reference / `id=` in
+    haproxy.log) is the `unique_id` in /var/log/coraza/audit.log and
+    spoa.log. Without `id=` coraza-spoa v0.7.1 invents a random 16-letter ID
+    (internal/application.go HandleRequest) that joins to nothing."""
+
+    def setUp(self):
+        spoe = haproxy_manager.template_env.get_template(
+            'hap_coraza_spoe_engine.tpl').render()
+        lines = active_lines(spoe)
+        self.messages = {}
+        current = None
+        for line in lines:
+            m = re.match(r'^spoe-message\s+(\S+)$', line)
+            if m:
+                current = m.group(1)
+                continue
+            if line.startswith('spoe-') or line.startswith('['):
+                current = None
+            elif current and line.startswith('args '):
+                self.messages[current] = line.split()[1:]
+
+    def args_of(self, message):
+        return dict(a.split('=', 1) for a in self.messages[message])
+
+    def test_request_message_sends_unique_id(self):
+        self.assertIn('coraza-req', self.messages)
+        self.assertEqual(self.args_of('coraza-req').get('id'), 'unique-id')
+
+    def test_request_message_exports_rule_ids(self):
+        """Without exportRuleIDs coraza-spoa never sets txn.coraza.rule_ids
+        and the access log's wrules= is always '-'."""
+        self.assertEqual(self.args_of('coraza-req').get('exportRuleIDs'), 'bool(true)')
+
+    def test_app_is_still_the_first_arg(self):
+        """agent.go HandleSPOE Panics unless the FIRST kv entry is `app`."""
+        for name, args in self.messages.items():
+            self.assertTrue(args[0].startswith('app='),
+                            f'{name}: first arg must be app=, got {args[0]}')
+
+    def test_every_response_message_carries_the_same_id(self):
+        """HandleResponse finds the request transaction by `id`; a coraza-res
+        without it errors ("response id is empty") on every response."""
+        for name in self.messages:
+            if name != 'coraza-req':
+                self.assertEqual(self.args_of(name).get('id'), 'unique-id', name)
+
+    def test_unique_id_is_the_header_reference(self):
+        """The fetch is only meaningful while unique-id-format is the UUID that
+        is also emitted as X-Request-Reference and on the 403 page."""
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        self.assertIn('unique-id-format %[uuid()]', [re.sub(r'\s+', ' ', l) for l in header])
+        self.assertIn('unique-id-header X-Request-Reference', [re.sub(r'\s+', ' ', l) for l in header])
+        listener = render('hap_listener.tpl', coraza=True)
+        self.assertIn('x-request-reference "%[unique-id]"', listener)
+
+
+CORAZA_OVERRIDES = os.path.join(MODULE_DIR, 'coraza-spoa', 'overrides.conf')
+
+
+@unittest.skipUnless(os.path.exists(CORAZA_OVERRIDES),
+                     'coraza-spoa/overrides.conf is not shipped in this image; '
+                     'this class runs from a repo checkout')
+class BakedPromotions(unittest.TestCase):
+    """920440 is a CHAIN whose first link matches every URL with a file
+    extension. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on that
+    first link, so every *.php / *.css request was evaluated with the engine
+    On (full CRS anomaly enforcement, in detect_only mode too). Measured
+    in-process and live, 2026-10-09; fixed by keying the flip on the chain's
+    real condition."""
+
+    def setUp(self):
+        with open(CORAZA_OVERRIDES) as fh:
+            self.lines = active_lines(fh.read())
+
+    def test_920440_is_not_promoted_on_its_chain_starter(self):
+        self.assertFalse([l for l in self.lines
+                          if re.match(r'SecRuleUpdateActionById\s+(\S*-)?920440\b', l)],
+                         '920440 promoted via SecRuleUpdateActionById again')
+
+    def test_no_chain_starter_is_promoted(self):
+        """Chain-starter rules in the promoted ranges (CRS 4.25.0): 920440,
+        944110, 944120. None may be covered by a SecRuleUpdateActionById."""
+        chain_starters = (920440, 944110, 944120)
+        promoted = []
+        for l in self.lines:
+            m = re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l)
+            if m:
+                lo = int(m.group(1))
+                promoted.append((lo, int(m.group(2) or lo)))
+        self.assertTrue(promoted, 'no promotions parsed -- the test is blind')
+        for rid in chain_starters:
+            self.assertFalse([r for r in promoted if r[0] <= rid <= r[1]],
+                             f'{rid} is a CRS chain starter and must not be promoted')
+
+    def test_log4shell_range_still_promoted(self):
+        for rid in (944100, 944130, 944150, 944151, 944152, 944300):
+            self.assertTrue([1 for l in self.lines
+                             if (m := re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l))
+                             and int(m.group(1)) <= rid <= int(m.group(2) or m.group(1))],
+                            f'{rid} lost its promotion')
+
+    def test_920440_flip_keys_on_the_restricted_extension(self):
+        i = self.lines.index('SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \\')
+        self.assertEqual(self.lines[i + 1],
+                         '"id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"')
+
+
+@unittest.skipUnless(os.path.exists(CORAZA_CONFIG),
+                     'coraza-spoa/config.yaml is not shipped in this image; '
+                     'this class runs from a repo checkout')
+class CorazaBodyLimits(unittest.TestCase):
+
+    def setUp(self):
+        with open(CORAZA_CONFIG) as fh:
+            self.cfg = active_lines(fh.read())
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        self.bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
+
+    def directive(self, name):
+        vals = [l.split(None, 1)[1] for l in self.cfg if l.split()[0] == name]
+        self.assertEqual(len(vals), 1, f'{name} must be set exactly once in config.yaml')
+        return vals[0]
+
+    def test_limit_strictly_above_anything_haproxy_can_send(self):
+        limit = int(self.directive('SecRequestBodyLimit'))
+        self.assertGreater(limit, SPOA_FRAME_CAP)
+        self.assertGreater(limit, self.bufsize)
+
+    def test_over_limit_never_rejects(self):
+        """Reject interrupts even under DetectionOnly -> 403 on uploads."""
+        self.assertEqual(self.directive('SecRequestBodyLimitAction'), 'ProcessPartial')
+
+    def test_body_buffer_stays_in_memory(self):
+        self.assertGreaterEqual(int(self.directive('SecRequestBodyInMemoryLimit')),
+                                int(self.directive('SecRequestBodyLimit')))
+
+    def test_json_depth_limit_pinned(self):
+        """Quadratic JSON flattening: depth 300 = 297 ms, past the fail-open
+        timeout. Legitimate WordPress JSON measured at depth <= 9."""
+        depth = int(self.directive('SecRequestBodyJsonDepthLimit'))
+        self.assertGreaterEqual(depth, 32, 'too tight for real block/theme payloads')
+        self.assertLessEqual(depth, 64, 'depth 100 already costs ~40 ms')
+
+    def test_limits_are_set_after_the_recommended_file(self):
+        inc = self.cfg.index('Include @coraza.conf-recommended')
+        for name in ('SecRequestBodyLimit ', 'SecRequestBodyJsonDepthLimit '):
+            lim = [i for i, l in enumerate(self.cfg) if l.startswith(name)][0]
+            self.assertLess(inc, lim, f'the recommended file would override {name.strip()}')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-wpadmin-gate.py

@@ -0,0 +1,530 @@
+#!/usr/bin/env python3
+"""Regression tests for the WordPress admin edge gate in hap_listener.tpl.
+
+Why this file exists
+--------------------
+Unauthenticated GETs to /wp-admin/* were reaching PHP, booting WordPress just to
+produce a login redirect and exhausting lsphp pools under a distributed
+low-and-slow attack. The gate redirects them at the edge instead.
+
+Two properties are easy to get wrong and invisible to `haproxy -c`:
+
+  * ORDERING. `is_whitelisted` reads var(txn.real_ip). If the rule renders before
+    the set-var chain, the whitelist evaluates against an unset variable.
+  * THE ALLOWLIST. wp-login.php loads its OWN css/js from /wp-admin/. Dropping
+    those entries leaves every login page on the fleet unstyled, while still
+    returning 200 -- a silent regression.
+
+A THIRD property, added after an adversarial mutation audit: every assertion
+here must be scoped to the CODE, not the surrounding prose. This file's own
+comment blocks quote ACL names, rule fragments and even whole rules to explain
+them -- which means a bare `assertIn` / `re.search` / `str.index` run over the
+raw rendered config passes just as happily when the real rule has been deleted
+(or merely commented out) and only its explanation survives. The audit proved
+this concretely: commenting out the entire redirect rule, or the
+`acl wp_admin_allowed` line, or all five normalizers, left the previous version
+of this file at 26/26 PASS. See `rule_lines()` below, and use it (or one of the
+guarded helpers built on it) for every assertion about whether a rule exists,
+what it says, or where it sits relative to another rule. Do not add a new
+`self.cfg.index(...)`, `self.assertIn(x, self.cfg)`, or `re.search(pattern,
+self.cfg)` to this file -- none of them can tell code from comment.
+
+Running
+-------
+    python3 scripts/test-wpadmin-gate.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+ALLOWLIST = ('/admin-ajax.php', '/admin-post.php',
+             '/load-styles.php', '/load-scripts.php')
+EXCLUSIONS = ('!wp_admin_allowed', '!wp_admin_asset', '!has_wp_logged_in',
+              '!wp_gate_exempt', '!is_local', '!is_trusted_ip', '!is_whitelisted')
+
+# The normalizer set, in the order it MUST render. Decoding has to precede the
+# path walkers or "%2e%2e" is decoded to ".." only after path-strip-dotdot has
+# already run, leaving the ".." unresolved -- measured against real HAProxy
+# 3.0.11, both orders side by side.
+NORMALIZERS = ('percent-to-uppercase',
+               'percent-decode-unreserved',
+               'path-merge-slashes',
+               'path-strip-dot',
+               'path-strip-dotdot full')
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_header():
+    return haproxy_manager.template_env.get_template('hap_header.tpl').render(
+        cluster_secret=None,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Comment-safe config inspection
+#
+# Every helper below operates on `rule_positions()`'s output, never on the raw
+# rendered string. That is the one rule this whole module exists to enforce on
+# itself: a mutation audit found that commenting out a real rule (prefixing it
+# with '#', or -- more slyly -- deleting it and appending its own text as a
+# TRAILING comment on the line above) left the previous version of these tests
+# fully green, because plain `str.index` / `assertIn` / `re.search` over
+# `self.cfg` cannot distinguish code from a comment that merely quotes it.
+# ---------------------------------------------------------------------------
+
+def rule_positions(cfg, needle):
+    """[(comment-stripped line, char offset in cfg)] for every non-comment
+    line containing `needle`, in document order.
+
+    Two things a bare substring/regex search over `self.cfg` gets wrong, both
+    fixed here:
+
+      1. A line that is ENTIRELY a comment (starts with '#' once stripped) is
+         dropped. This is necessary but not sufficient -- see (2).
+      2. A line that MIXES real config with a trailing comment
+         (`live-code  # note`, or the decoy `live-code  # was: <the other
+         rule's exact text>`) is truncated at the first ' #' before matching,
+         so text stuffed into a trailing comment cannot masquerade as the
+         rule itself. A real HAProxy comment always starts at a '#' preceded
+         by whitespace here -- none of these templates use bare '#' as a
+         value character -- so this truncation does not clip real rules.
+
+    Returning (line, position) pairs together -- rather than making callers
+    re-derive one from the other with a second `cfg.index(line)` -- also
+    avoids a subtler bug: if the same comment-stripped line occurs twice
+    (e.g. a duplicated rule), re-deriving the position with `str.index` always
+    finds the FIRST copy regardless of which one you meant. Walking the file
+    once and recording positions as we go keeps first/last unambiguous.
+    """
+    out = []
+    pos = 0
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if stripped and not stripped.startswith('#'):
+            code = stripped.split(' #', 1)[0].rstrip()
+            if code and needle in code:
+                out.append((code, pos))
+        pos += len(raw) + 1  # +1 for the '\n' split() consumed
+    return out
+
+
+def rule_lines(cfg, needle):
+    """Comment-stripped rule lines containing `needle` (text only, no
+    position). See `rule_positions()` for what this guards against. Every
+    assertion about a RULE's presence or content must go through this (or
+    `rule_positions`/the guarded helpers below) -- never a bare
+    `needle in cfg` or `re.search(pattern, cfg)`.
+    """
+    return [line for line, _ in rule_positions(cfg, needle)]
+
+
+def require_rule(cfg, needle, what=None):
+    """The single rule line containing `needle`.
+
+    Raises a plain AssertionError naming what was being looked for -- not
+    IndexError from an unguarded `rule_lines(...)[0]`, and not
+    `ValueError: substring not found` from a bare `cfg.index(...)` -- when
+    the rule is missing. A missing rule and a broken test harness must not
+    look identical in a failure report.
+
+    Raises the same way if `needle` is ambiguous (matches more than one rule
+    line): silently taking the first match in that case would hide the
+    ambiguity instead of surfacing it.
+    """
+    label = what or needle
+    lines = rule_lines(cfg, needle)
+    if not lines:
+        raise AssertionError('no rule found for %r (expected: %s)' % (needle, label))
+    if len(lines) > 1:
+        raise AssertionError(
+            '%r matched %d rule lines, expected exactly one (%s): %r'
+            % (needle, len(lines), label, lines))
+    return lines[0]
+
+
+def require_position(cfg, needle, what=None, last=False):
+    """(line, char offset) for an ordering assertion, guarded the same way as
+    `require_rule` -- but tolerant of the needle matching multiple lines
+    (e.g. a multi-line set-var "chain"), since ordering checks often want the
+    first or last of several. Pass last=True for the last occurrence.
+    """
+    label = what or needle
+    positions = rule_positions(cfg, needle)
+    if not positions:
+        raise AssertionError(
+            'no rule found for %r, cannot check ordering (expected: %s)' % (needle, label))
+    return positions[-1] if last else positions[0]
+
+
+class WpAdminGate(unittest.TestCase):
+
+    def setUp(self):
+        self.cfg = render_listener()
+
+    def test_wp_admin_path_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_path', 'wp_admin_path ACL')
+        self.assertIn('path_reg', line)
+
+    def test_wp_admin_asset_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertIn('path_reg', line)
+
+    def test_wp_admin_allowed_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        self.assertIn('path_end', line)
+
+    def test_wp_gate_exempt_acl_declared(self):
+        """Scoped to the ACL line itself, not `self.cfg` as a whole -- the
+        surrounding prose (see hap_listener.tpl's "per-site opt-out" comment)
+        also spells out /etc/haproxy/wpadmin_gate_exempt.list verbatim, so an
+        unscoped `assertIn` would still pass with the real ACL deleted.
+        """
+        line = require_rule(self.cfg, 'acl wp_gate_exempt', 'wp_gate_exempt ACL')
+        self.assertIn('/etc/haproxy/wpadmin_gate_exempt.list', line)
+
+    def test_allowlist_entries_present(self):
+        """wp-login.php loads its own css/js from /wp-admin/ -- see module docstring."""
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        for entry in ALLOWLIST:
+            with self.subTest(entry=entry):
+                self.assertIn(entry, line)
+
+    def test_static_asset_dirs_allowed(self):
+        line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertRegex(line, r'path_reg.*\(css\|js\|images\)')
+
+    def test_install_php_is_NOT_allowlisted(self):
+        """install.php is deliberately gated -- a takeover vector on abandoned installs."""
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        self.assertNotIn('install.php', line)
+
+    def test_redirect_rule_has_all_exclusions(self):
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        for excl in EXCLUSIONS:
+            with self.subTest(exclusion=excl):
+                self.assertIn(excl, rule)
+
+    def test_rule_renders_after_real_ip_resolution(self):
+        """is_whitelisted reads txn.real_ip; before the set-var chain it is unset."""
+        _, setvar_pos = require_position(self.cfg, 'set-var(txn.real_ip)',
+                                          'real_ip set-var chain')
+        _, rule_pos = require_position(self.cfg, 'wp_admin_path', 'wp_admin_path ACL/rule')
+        self.assertLess(setvar_pos, rule_pos,
+                         'wp_admin_path renders before txn.real_ip is resolved')
+
+    def test_rule_renders_after_has_wp_logged_in_declared(self):
+        """HAProxy resolves ACLs as it parses; use-before-declare fails."""
+        _, decl_pos = require_position(self.cfg, 'acl has_wp_logged_in',
+                                        'has_wp_logged_in ACL declaration')
+        _, rule_pos = require_position(self.cfg, 'wp_admin_path', 'wp_admin_path ACL/rule')
+        self.assertLess(decl_pos, rule_pos,
+                         'wp_admin_path renders before has_wp_logged_in is declared')
+
+    def test_only_one_has_wp_logged_in_declaration(self):
+        self.assertEqual(len(rule_lines(self.cfg, 'acl has_wp_logged_in')), 1)
+
+    def test_allowlist_entries_are_anchored_to_wp_admin(self):
+        """Bare `path_end /admin-ajax.php` also matches
+        /wp-admin/evil/admin-ajax.php, which ALSO matches wp_admin_path
+        (path_reg only requires /wp-admin/ to appear somewhere) -- an
+        attacker-inserted path segment would then sail through the
+        allowlist ungated. Entries must be anchored to sit directly under
+        wp-admin/. Scoped to the captured ACL line only, since the
+        surrounding comment block also mentions these bare filenames.
+        """
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        for entry in ALLOWLIST:
+            with self.subTest(entry=entry):
+                self.assertIn('/wp-admin' + entry, line)
+                self.assertNotRegex(
+                    line, r'(?<!wp-admin)' + re.escape(entry) + r'(?!\S)',
+                    'found a bare, unanchored allowlist entry: ' + entry)
+
+    def test_wp_login_url_setvar_renders_in_correct_order(self):
+        """The inline regsub-in-`location` form is rejected by real HAProxy
+        3.0.11 (invalid arg 2 in converter 'regsub'), so the regsub is
+        computed in its own set-var line instead. That set-var must render
+        after the set-var(txn.real_ip) chain (it must not disturb that
+        load-bearing chain) and before the redirect rule that consumes it.
+        """
+        _, last_real_ip_pos = require_position(self.cfg, 'set-var(txn.real_ip)',
+                                                'real_ip set-var chain', last=True)
+        _, wp_login_pos = require_position(self.cfg, 'set-var(txn.wp_login_url)',
+                                            'wp_login_url set-var')
+        _, redirect_pos = require_position(
+            self.cfg, 'http-request redirect code 302 location %[var(txn.wp_login_url)]',
+            'wp-admin redirect rule')
+        self.assertLess(last_real_ip_pos, wp_login_pos,
+                         'wp_login_url set-var must render after the real_ip set-var chain')
+        self.assertLess(wp_login_pos, redirect_pos,
+                         'wp_login_url set-var must render before the redirect rule that uses it')
+
+    def test_redirect_rule_uses_setvar_not_inline_regsub(self):
+        """Guards against reintroducing the rejected inline form."""
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        self.assertIn('%[var(txn.wp_login_url)]', rule)
+        self.assertNotIn('regsub', rule)
+
+    def test_redirect_rule_requires_safe_path(self):
+        """OPEN REDIRECT guard. The redirect target is built by rewriting
+        `path` with regsub, which only replaces the matched substring --
+        everything before "/wp-admin/" survives untouched in the output.
+        Three concrete requests turn that into an off-site `Location:`
+        header: "//evil.example.com/wp-admin/x.php" (protocol-relative,
+        browsers resolve "//host/path" to "https://host/path"),
+        "/\\evil.example.com/wp-admin/x.php" (browsers normalise a leading
+        "/\\" the same as "//"), and an RFC 7230 absolute-form request
+        target ("https://evil.example.com/wp-admin/x.php") which can make
+        HAProxy's `path` fetch return a full URI. wp_admin_safe_path
+        (requiring a well-formed absolute path) must be a POSITIVE
+        condition on the redirect rule -- scoped to the captured rule line
+        only, since the surrounding comment block also mentions this ACL
+        name and a bare substring match would pass even if the condition
+        were dropped from the rule itself.
+        """
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        self.assertIn('wp_admin_safe_path', rule)
+        self.assertNotIn('!wp_admin_safe_path', rule,
+                          'wp_admin_safe_path must be a positive condition, not negated')
+
+    def test_wp_admin_safe_path_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_safe_path', 'wp_admin_safe_path ACL')
+        self.assertIn('path_reg', line)
+
+    def test_unsafe_wp_admin_path_is_denied_not_passed_through(self):
+        """wp_admin_safe_path being a POSITIVE condition on the redirect means
+        a path that fails it is simply not redirected -- which used to mean it
+        fell through to the backend UNGATED, i.e. exactly the PHP-booting
+        request the gate exists to stop. Normalisation removes the "//"
+        spelling of that, but not "/\\", so the fall-through must be closed
+        with an explicit deny rather than left implicit.
+        """
+        denies = rule_lines(self.cfg, '!wp_admin_safe_path')
+        self.assertTrue(denies,
+                        'no rule denies a wp-admin path that fails wp_admin_safe_path')
+        self.assertTrue(any(d.startswith('http-request deny') for d in denies),
+                        'the !wp_admin_safe_path rule must be a deny: %r' % denies)
+
+
+def require_rule_by_predicate(cfg, needle, predicate, what):
+    """Like require_rule(), but for rules identified by needle + a predicate
+    over the comment-stripped line (e.g. "the `http-request redirect` line
+    that also mentions wp_admin_path", since the frontend has more than one
+    `http-request redirect`). Raises a clear AssertionError, not IndexError
+    or a silently-empty match, if no line satisfies both.
+    """
+    candidates = [ln for ln in rule_lines(cfg, needle) if predicate(ln)]
+    if not candidates:
+        raise AssertionError('no rule found matching %s' % what)
+    if len(candidates) > 1:
+        raise AssertionError('%s matched more than one rule line: %r' % (what, candidates))
+    return candidates[0]
+
+
+class UriNormalisation(unittest.TestCase):
+    """The gate matches the RAW path; the backend normalises and decodes it.
+    Every gap between those is a bypass -- five were found this way. These
+    tests pin the normalisation that closes the gap as a class.
+
+    NOTE: these are config-TEXT assertions. They are necessary but NOT
+    sufficient: the previous revision of this file passed while five live
+    bypasses shipped. The real evidence is the behavioural matrix run against
+    real haproxy 3.0.11 with raw sockets -- see
+    .superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-4-normalize-report.md.
+    """
+
+    def setUp(self):
+        self.cfg = render_listener()
+        self.header = render_header()
+
+    def test_experimental_directives_exposed_in_global(self):
+        """normalize-uri is experimental in 3.0; without this HAProxy refuses
+        to start (`haproxy -c` exits 1, ALERT). That does NOT crash-loop the
+        container, though -- see hap_header.tpl's comment and
+        haproxy_manager.py's start_haproxy()/do_initial_setup(): the failure
+        is swallowed, and the container comes up with haproxy simply never
+        running. This test exists so that silent-outage mode is never
+        reintroduced by dropping this line.
+        """
+        lines = rule_lines(self.header, 'expose-experimental-directives')
+        matches = [ln for ln in lines if ln == 'expose-experimental-directives']
+        self.assertTrue(
+            matches, 'expose-experimental-directives missing from the global section')
+
+    def test_all_normalizers_render(self):
+        for norm in NORMALIZERS:
+            with self.subTest(normalizer=norm):
+                self.assertTrue(
+                    rule_lines(self.cfg, 'normalize-uri ' + norm),
+                    'missing normalizer: ' + norm)
+
+    def test_normalizer_order_decode_before_path_walkers(self):
+        """Reverse this order and /wp-admin/js/%2e%2e/plugins.php reaches the
+        ACLs as /wp-admin/js/../plugins.php -- decoded but unresolved.
+        """
+        lines = [ln for ln in rule_lines(self.cfg, 'http-request normalize-uri')
+                 if ln.startswith('http-request normalize-uri')]
+        names = [ln.split('normalize-uri ', 1)[1] for ln in lines]
+        self.assertEqual(
+            names, list(NORMALIZERS),
+            'normalize-uri directives are missing, reordered, or duplicated: %r' % names)
+
+    def test_normalisation_precedes_every_path_based_rule(self):
+        """A normalizer placed after a path rule normalises nothing for it."""
+        norm_positions = []
+        for n in NORMALIZERS:
+            _, pos = require_position(self.cfg, 'http-request normalize-uri ' + n,
+                                       'normalizer: ' + n)
+            norm_positions.append(pos)
+        last_norm = max(norm_positions)
+        for marker in ('acl is_health_check', 'acl wp_login_path',
+                       'acl xmlrpc_path', 'acl wp_batch_path',
+                       'acl wp_admin_path', 'http-request set-path'):
+            with self.subTest(rule=marker):
+                _, marker_pos = require_position(self.cfg, marker, marker)
+                self.assertLess(last_norm, marker_pos,
+                                 marker + ' renders before URI normalisation')
+
+    def test_query_sort_by_name_is_not_enabled(self):
+        """query-sort-by-name reorders query parameters, which would break
+        anything that signs or caches on the exact query string. This is NOT
+        because the enabled normalizers already leave the query alone --
+        percent-to-uppercase and percent-decode-unreserved rewrite the WHOLE
+        request-target, query string included (see hap_listener.tpl's BLAST
+        RADIUS comment) -- it is a deliberate line between "case-fold /
+        decode" (no-ops under RFC 3986) and "reorder" (not a no-op for a
+        signed/cached query string).
+        """
+        self.assertFalse(rule_lines(self.cfg, 'normalize-uri query-sort-by-name'))
+
+    def test_dotdot_normalizer_uses_full(self):
+        """Without "full", ".." segments that climb above the root are left in
+        place and /../../wp-admin/plugins.php survives -- measured.
+        """
+        lines = rule_lines(self.cfg, 'normalize-uri path-strip-dotdot')
+        self.assertTrue(lines, 'missing normalizer: path-strip-dotdot')
+        for ln in lines:
+            self.assertTrue(ln.endswith('path-strip-dotdot full'), ln)
+
+    def test_encoded_separator_on_wp_admin_is_denied(self):
+        """percent-decode-unreserved deliberately leaves %2F encoded ("/" is
+        reserved), but OpenLiteSpeed decodes it and serves the file --
+        /wp-admin%2Fplugins.php was measured booting PHP on the OLS tier while
+        matching no wp-admin ACL. Normalisation cannot close this; it needs its
+        own rule.
+        """
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        acl_line = require_rule(self.cfg, 'acl path_has_encoded_sep', 'path_has_encoded_sep ACL')
+        self.assertIn('%2f', acl_line.lower())
+
+    def test_encoded_separator_deny_is_scoped_to_wp_admin(self):
+        """A blanket "deny any %2F in any path" would break non-WordPress
+        customer apps that legitimately pass an encoded slash in a path
+        parameter. The deny must be conditioned on the path mentioning
+        wp-admin.
+        """
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        for d in denies:
+            self.assertIn('wp_admin_word', d)
+
+    def test_encoded_separator_deny_honors_the_same_whitelist(self):
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        for d in denies:
+            for excl in ('!has_wp_logged_in', '!wp_gate_exempt', '!is_local',
+                         '!is_trusted_ip', '!is_whitelisted'):
+                with self.subTest(rule=d, exclusion=excl):
+                    self.assertIn(excl, d)
+
+    def test_encoded_separator_acl_matches_a_substring_not_a_prefix(self):
+        """/blog%2Fwp-admin/plugins.php hides the separator BEFORE "wp-admin",
+        where an anchored pattern never matches, and OLS still resolves it.
+        """
+        acl_line = require_rule(self.cfg, 'acl path_has_encoded_sep', 'path_has_encoded_sep ACL')
+        self.assertIn('-m sub', acl_line)
+
+    def test_wp_admin_asset_bypass_cannot_cover_a_php_entrypoint(self):
+        """The asset bypass anchored its prefix but not its suffix, so
+        /wp-admin/css/../plugins.php took it and the backend then resolved
+        ".." and booted plugins.php. path-strip-dotdot is the real fix; this
+        keeps the bypass structurally incapable of covering PHP.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertIn('.php', acl_line,
+                      'wp_admin_asset must exclude .php explicitly')
+
+    def test_wp_admin_asset_pattern_is_end_of_flags_guarded(self):
+        """A pattern starting with "(" makes HAProxy warn on EVERY load/reload:
+
+            parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
+            '(^|/)wp-admin/...' is likely a mistake ... Maybe you need to
+            remove the extraneous space before '('.
+
+        "--" is the end-of-flags marker HAProxy itself names as the fix. It is
+        cosmetic to matching but not to operations: an unsilenced warning on
+        every reload on every host trains people to skim past warnings, which
+        is how a real one gets missed. Assert the pattern is still the one we
+        think it is, so this can never pass by the pattern having been changed.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertRegex(
+            acl_line, r'path_reg\s+--\s+\(',
+            'wp_admin_asset pattern begins with "(" and MUST be preceded by "--"')
+        self.assertIn('(^|/)wp-admin/(css|js|images)/', acl_line)
+
+    def test_case_insensitive_acl_and_regsub_are_kept_in_sync(self):
+        """A case-insensitive wp_admin_path with a case-sensitive regsub is an
+        INFINITE REDIRECT LOOP: regsub finds no "/wp-admin/" in
+        "/WP-ADMIN/plugins.php", returns `path` unchanged, and the Location
+        then points at the request's own URL.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_path', 'wp_admin_path ACL')
+        setvar_line = require_rule(self.cfg, 'set-var(txn.wp_login_url)', 'wp_login_url set-var')
+        acl_ci = bool(re.search(r'path_reg\s+-i\s', acl_line))
+        regsub_ci = bool(re.search(r'regsub\([^)]*,\s*i\)', setvar_line))
+        self.assertEqual(
+            acl_ci, regsub_ci,
+            'wp_admin_path case-sensitivity (%s) and regsub flags (%s) disagree'
+            % (acl_line, setvar_line))
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-xmlrpc-rate-limit.py

@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""Regression tests for per-client-IP rate limiting on POST /xmlrpc.php.
+
+Why this file exists
+--------------------
+POST /xmlrpc.php floods were unthrottled fleet-wide. The generic frontend
+rate limits (hap_listener.tpl) trigger at 3000/5000 req/10s -- i.e. 300-500
+req/s -- but the observed floods run at a few req/s for hours, well under
+that ceiling. The existing wp_bruteforce mechanism (dedicated stick-table,
+60s window, per real client IP) solves exactly this shape of problem for
+POST /wp-login.php; this change adds an equivalent dedicated table/rule pair
+for POST /xmlrpc.php.
+
+This is only safe to key on var(txn.real_ip) because of the trusted-proxy
+gate added earlier (release 2026.08.3, see test-trusted-proxy-gate.py) --
+before that fix, a direct client could spoof any client IP via
+X-Forwarded-For and evade all per-IP tracking.
+
+These tests pin:
+  - a dedicated stick-table for xmlrpc tracking exists in
+    hap_security_tables.tpl (own sc slot / own counter -- not sharing the
+    wp_bruteforce counter, so a wp-login brute-force run and an xmlrpc flood
+    from the same IP don't inflate each other's rate)
+  - the tracking rule only fires on POST /xmlrpc.php (path_end, so
+    subdirectory WP installs are covered)
+  - the limiting rule tarpits over the chosen threshold
+  - the limiting rule honors the same whitelist as every other rule in the
+    file (!is_local !is_trusted_ip !is_whitelisted)
+  - xmlrpc is not blocked outright -- only the rate-limit ACL is present,
+    there's no blanket deny of the path
+
+Running
+-------
+    python3 scripts/test-xmlrpc-rate-limit.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_security_tables():
+    return haproxy_manager.template_env.get_template(
+        'hap_security_tables.tpl').render()
+
+
+class XmlrpcRateLimit(unittest.TestCase):
+
+    def setUp(self):
+        self.listener_cfg = render_listener()
+        self.tables_cfg = render_security_tables()
+
+    def test_dedicated_stick_table_defined(self):
+        """A dedicated table (not wp_bruteforce) tracks xmlrpc requests, so a
+        wp-login brute-force run and an xmlrpc flood from the same IP can't
+        inflate each other's rate counter."""
+        self.assertRegex(
+            self.tables_cfg,
+            r'backend\s+xmlrpc_bruteforce\s*\n\s*stick-table\s+type\s+ip\b.*store.*http_req_rate',
+        )
+        # Must not be the same table wp-login already uses.
+        self.assertNotIn('backend wp_bruteforce\n    stick-table type ip size 100k expire 30m store http_req_rate(60s)\nbackend xmlrpc_bruteforce', self.tables_cfg)
+
+    def test_xmlrpc_path_acl_uses_path_end(self):
+        """path_end (not path_beg) so subdirectory WP installs are covered,
+        matching the wp-login rule's reasoning."""
+        self.assertRegex(
+            self.listener_cfg,
+            r'acl\s+xmlrpc_path\s+path_end\s+/xmlrpc\.php',
+        )
+
+    def test_tracking_rule_only_fires_on_post_xmlrpc(self):
+        self.assertRegex(
+            self.listener_cfg,
+            r'http-request\s+track-sc2\s+var\(txn\.real_ip\)\s+table\s+xmlrpc_bruteforce\s+if\s+METH_POST\s+xmlrpc_path',
+        )
+
+    def test_limiting_rule_tarpits_over_threshold_with_whitelist(self):
+        pattern = (
+            r'http-request\s+tarpit\s+deny_status\s+429\s+if\s+METH_POST\s+xmlrpc_path\s+'
+            r'\{\s*sc_http_req_rate\(2\)\s+gt\s+(\d+)\s*\}\s+'
+            r'!is_local\s+!is_trusted_ip\s+!is_whitelisted'
+        )
+        match = re.search(pattern, self.listener_cfg)
+        self.assertIsNotNone(
+            match, 'expected a tarpit rule tracking sc2 with the full whitelist')
+        threshold = int(match.group(1))
+        self.assertGreater(threshold, 0)
+
+    def test_xmlrpc_not_blocked_outright(self):
+        """The endpoint must remain functional for clients under the
+        threshold -- only a rate-limit ACL, no blanket deny of the path."""
+        self.assertNotRegex(
+            self.listener_cfg,
+            r'http-request\s+deny\s+deny_status\s+\d+\s+if\s+(?:METH_POST\s+)?xmlrpc_path\s*(?:!is_local|\n)',
+        )
+
+    def test_rule_order_after_wp_login_block(self):
+        """Not load-bearing for correctness (mutually exclusive paths), but
+        keep the new block grouped with the other WordPress-specific rules
+        rather than scattered elsewhere in the file."""
+        wp_login_idx = self.listener_cfg.index('wp_login_path')
+        xmlrpc_idx = self.listener_cfg.index('xmlrpc_path')
+        self.assertLess(wp_login_idx, xmlrpc_idx)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/validate-rendered-config.py

@@ -0,0 +1,423 @@
+#!/usr/bin/env python3
+"""Build gate: render the real HAProxy config and hand it to the real `haproxy -c`.
+
+Why this file exists
+--------------------
+On 2026-08-14 a change to hap_listener.tpl rendered perfectly, passed every
+unit test in scripts/ (13 green), and was then rejected outright by HAProxy:
+
+    [ALERT] config : parsing [/etc/haproxy/haproxy.cfg:...] :
+            invalid arg 2 in converter 'regsub' : ... unexpected empty
+            replacement string
+
+Nothing between "commit" and "running in production" would have caught it.
+The unit tests assert on the *text* of the rendered config with regexes, which
+tells you what the template says, never whether HAProxy will accept it. And
+scripts/test-config-rollback.py stubs the `haproxy` binary with a shell script
+that only rejects a literal sentinel token, so its "validation" has never
+parsed a single line of real HAProxy syntax.
+
+The failure mode this guards is not cosmetic. When haproxy.cfg is invalid,
+scripts/init.py refuses to start HAProxy but the container still comes up:
+ports 80/443 are unbound, every site on the host is down, and /health keeps
+answering 200 because the Flask API is fine.
+
+So: render the config through the SAME code path production uses
+(haproxy_manager.generate_config(), templates and all), then run the actual
+`haproxy -c` against the result and gate on its exit code.
+
+This runs as a RUN step in the Dockerfile, which means it also validates
+against the exact haproxy binary that ships in the image being built - note
+that the Dockerfile installs haproxy UNPINNED, so that binary can move under
+us between builds. Any syntax the new binary rejects now fails the build
+instead of failing at 3am on an edge node.
+
+What it covers
+--------------
+  * every template generate_config() touches, assembled in the real order
+  * a domain with SSL + a backend, a wildcard domain, a cert-only domain with
+    no backend, and two template_override backends
+  * blocked-IP map entries (single IP and CIDR)
+  * BOTH sides of the two conditional blocks in hap_listener.tpl -
+    {%- if suspension_enabled %} and {%- if coraza_spoe_backend %} - because a
+    syntax error inside a conditional ships undetected otherwise. Scenario
+    "full" turns both on; scenario "default" leaves both off, which is the
+    byte-identical-to-standalone shape.
+
+Warnings vs failures
+--------------------
+`haproxy -c` emits warnings on a clean config here (at minimum "Can't load
+stats file" because /var/lib/haproxy/stats.dat doesn't exist at build time,
+plus assorted path_reg/ACL advisories). Those are NOT failures. This gate keys
+on the process EXIT CODE only, and dumps the full output when it is non-zero.
+
+Running
+-------
+    python3 scripts/validate-rendered-config.py
+
+Needs the real `haproxy` binary, the application's Python dependencies, and
+write access to /etc/haproxy (several templates reference files there by
+absolute path - see _REAL_PATH_NOTE below). Inside the image build all three
+hold. On a workstation, run it in the container instead.
+"""
+
+import logging
+import os
+import re
+import shutil
+import sqlite3
+import subprocess
+import sys
+import tempfile
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+# Same trick the other suites use: haproxy_manager configures logging at import
+# time against /var/log/haproxy-manager.log. Redirect the handlers so this runs
+# without root and without polluting the image's log files.
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-validate-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+try:
+    import haproxy_manager as hm  # noqa: E402
+finally:
+    logging.FileHandler = _real_file_handler
+
+# The application logs a lot at INFO during a render, and legitimately logs at
+# ERROR about things that are only true in this harness ("no existing HAProxy
+# config on disk ... ROLLBACK IS NOT AVAILABLE" - correct, there is no live
+# config during a build). Silence it so the build log carries the gate's own
+# verdict and haproxy's output, which is what matters.
+logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
+
+
+# _REAL_PATH_NOTE
+# ---------------
+# Most paths haproxy_manager writes to are module-level constants and are
+# redirected into a temp dir below. Two cannot be:
+#
+#   /etc/haproxy/blocked_ips.map   - hardcoded inside hap_listener.tpl's
+#                                    map_ip() converter
+#   /etc/haproxy/coraza-spoe.cfg   - hardcoded in the `filter spoe engine`
+#                                    line, and parsed by haproxy -c
+#
+# Redirecting the constants without editing the templates would just make
+# haproxy read a different (missing) file, so those two are left at their real
+# paths. Everything this script creates under /etc/haproxy is removed again on
+# exit; pre-existing files (the baked trusted_ips.*) are never touched.
+ETC_HAPROXY = '/etc/haproxy'
+
+# Files referenced with `-f` / map_ip() from the templates. A missing `-f` file
+# is a FATAL haproxy error, so a gate that didn't create these would fail for
+# reasons that have nothing to do with the config being tested.
+STUB_FILES = {
+    os.path.join(ETC_HAPROXY, 'trusted_ips.list'): '# validation stub\n203.0.113.10\n',
+    os.path.join(ETC_HAPROXY, 'trusted_ips.map'): '# validation stub\n203.0.113.11 1\n',
+    os.path.join(ETC_HAPROXY, 'cloudflare_ips.list'): '# validation stub\n198.51.100.0/24\n',
+    os.path.join(ETC_HAPROXY, 'trusted_proxies.list'): '# validation stub\n192.0.2.0/24\n',
+    os.path.join(ETC_HAPROXY, 'wpadmin_gate_exempt.list'): '# validation stub\nexempt.example.test\n',
+    os.path.join(ETC_HAPROXY, 'suspended_domains.list'): 'suspended.example.test\n',
+    # `lf-file` on the Coraza deny rule; loaded at parse time. Present in the
+    # image (COPY errors /haproxy/errors), stubbed for anything else.
+    '/haproxy/errors/403-waf.html': '<html><body>blocked %[unique-id]</body></html>\n',
+}
+
+# (suspension_enabled, coraza_spoe_backend) combinations to render + validate.
+SCENARIOS = (
+    ('default', {}),
+    ('full', {
+        'HAPROXY_SUSPENSION_ENABLED': 'true',
+        'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
+    }),
+)
+
+
+def log(msg):
+    sys.stdout.write(f'[validate-config] {msg}\n')
+    sys.stdout.flush()
+
+
+def fail(msg):
+    sys.stderr.write(f'[validate-config] FAIL: {msg}\n')
+    sys.stderr.flush()
+    raise SystemExit(1)
+
+
+class CreatedFiles:
+    """Tracks what we put on disk outside the temp dir so it can be removed.
+
+    Two sources: files we create explicitly, and files generate_config() itself
+    writes into /etc/haproxy (blocked_ips.map, coraza-spoe.cfg and their
+    .backup copies). The latter are caught by diffing the directory listing,
+    which also picks up anything a future change starts writing there.
+    """
+
+    def __init__(self):
+        self.explicit = []
+        self.etc_before = self._listdir(ETC_HAPROXY)
+
+    @staticmethod
+    def _listdir(path):
+        try:
+            return set(os.listdir(path))
+        except OSError:
+            return set()
+
+    def ensure(self, path, content):
+        """Create path with content if it does not already exist."""
+        if os.path.exists(path):
+            return
+        os.makedirs(os.path.dirname(path), exist_ok=True)
+        with open(path, 'w') as fh:
+            fh.write(content)
+        os.chmod(path, 0o644)
+        self.explicit.append(path)
+
+    def cleanup(self):
+        for path in self.explicit:
+            try:
+                os.unlink(path)
+            except OSError:
+                pass
+        for name in self._listdir(ETC_HAPROXY) - self.etc_before:
+            try:
+                os.unlink(os.path.join(ETC_HAPROXY, name))
+            except OSError:
+                pass
+
+
+def require_haproxy_binary():
+    """Fail closed. A gate that skips itself when the binary is missing is a
+    gate that would have let the 2026-08-14 change through."""
+    path = shutil.which('haproxy')
+    if not path:
+        fail('no `haproxy` binary on PATH - this gate cannot validate anything. '
+             'Run it inside the image (the Dockerfile installs haproxy).')
+    version = subprocess.run([path, '-v'], capture_output=True, text=True)
+    log(f'using {path}: {version.stdout.strip().splitlines()[0] if version.stdout else "unknown version"}')
+
+
+def make_self_signed_cert(certs_dir):
+    """HAProxy loads every file in the `bind ... ssl crt <dir>` directory at
+    parse time, so the directory has to hold a real, loadable bundle."""
+    os.makedirs(certs_dir, exist_ok=True)
+    cert = os.path.join(certs_dir, 'cert.tmp')
+    key = os.path.join(certs_dir, 'key.tmp')
+    subprocess.run(
+        ['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes',
+         '-keyout', key, '-out', cert, '-days', '1',
+         '-subj', '/CN=validate.example.test'],
+        check=True, capture_output=True,
+    )
+    bundle = os.path.join(certs_dir, 'validate.example.test.pem')
+    with open(bundle, 'w') as out:
+        for part in (cert, key):
+            with open(part) as fh:
+                out.write(fh.read())
+    os.unlink(cert)
+    os.unlink(key)
+    os.chmod(bundle, 0o600)
+
+
+def seed_database(db_path):
+    """A representative fleet: SSL + backend, wildcard, cert-only, overrides."""
+    hm.DB_FILE = db_path
+    hm.init_db()
+    with sqlite3.connect(db_path) as conn:
+        cur = conn.cursor()
+
+        def add_site(domain, backend, ssl_enabled=1, wildcard=0, override=None,
+                     servers=(('web1', '10.0.0.10', 8080, 'check'),)):
+            cur.execute(
+                'INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
+                'template_override, is_wildcard) VALUES (?, ?, ?, ?, ?)',
+                (domain, ssl_enabled, f'/etc/haproxy/certs/{domain}.pem',
+                 override, wildcard),
+            )
+            domain_id = cur.lastrowid
+            cur.execute('INSERT INTO backends (name, domain_id, settings) '
+                        'VALUES (?, ?, ?)', (backend, domain_id, None))
+            backend_id = cur.lastrowid
+            for name, addr, port, opts in servers:
+                cur.execute(
+                    'INSERT INTO backend_servers (backend_id, server_name, '
+                    'server_address, server_port, server_options) '
+                    'VALUES (?, ?, ?, ?, ?)',
+                    (backend_id, name, addr, port, opts),
+                )
+
+        add_site('site-one.example.test', 'site-one',
+                 servers=(('web1', '10.0.0.10', 8080, 'check'),
+                          ('web2', '10.0.0.11', 8080, 'check backup')))
+        add_site('site-two.example.test', 'site-two', ssl_enabled=0)
+        add_site('*.wildcard.example.test', 'wildcard-site', wildcard=1)
+        add_site('ws.example.test', 'ws-site', override='hap_backend_websocket')
+        add_site('sse.example.test', 'sse-site', override='hap_backend_longlived')
+
+        # Cert/management-only domain: registered for certificates, no backend.
+        # generate_config() has an explicit branch for this.
+        cur.execute(
+            'INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
+            'template_override, is_wildcard) VALUES (?, ?, ?, ?, ?)',
+            ('panel.example.test', 1, '/etc/haproxy/certs/panel.pem', None, 0),
+        )
+
+        # Both map_ip() shapes: a single address and a CIDR.
+        for ip in ('203.0.113.66', '198.51.100.0/24'):
+            cur.execute('INSERT INTO blocked_ips (ip_address, reason, blocked_by) '
+                        'VALUES (?, ?, ?)', (ip, 'validation fixture', 'gate'))
+        conn.commit()
+
+
+def render(scenario_name, env_overrides, workdir):
+    """Render via generate_config() and return the path to the assembled config.
+
+    generate_config() is the real production entry point: it takes the rollback
+    snapshot, writes the blocked-IP map, renders every template, and writes
+    haproxy.cfg. Only the reload is stubbed - there is no HAProxy process to
+    reload during a build, and validating the file is the whole point.
+    """
+    scenario_dir = os.path.join(workdir, scenario_name)
+    certs_dir = os.path.join(scenario_dir, 'certs')
+    os.makedirs(scenario_dir)
+    make_self_signed_cert(certs_dir)
+
+    hm.HAPROXY_CONFIG_PATH = os.path.join(scenario_dir, 'haproxy.cfg')
+    hm.HAPROXY_BACKUP_PATH = os.path.join(scenario_dir, 'haproxy.cfg.backup')
+    hm.CLUSTER_SECRET_PATH = os.path.join(scenario_dir, 'cluster-secret')
+    hm.HAPROXY_SOCKET_PATH = os.path.join(scenario_dir, 'haproxy.sock')
+    hm.SSL_CERTS_DIR = certs_dir
+    seed_database(os.path.join(scenario_dir, 'haproxy_config.db'))
+
+    saved_env = {}
+    for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND'):
+        saved_env[key] = os.environ.pop(key, None)
+    os.environ.update(env_overrides)
+
+    real_reload = hm.reload_haproxy_safely
+    hm.reload_haproxy_safely = lambda *a, **kw: (True, 'reload skipped: build-time validation')
+    try:
+        hm.generate_config()
+    finally:
+        hm.reload_haproxy_safely = real_reload
+        for key, value in saved_env.items():
+            os.environ.pop(key, None)
+            if value is not None:
+                os.environ[key] = value
+
+    return hm.HAPROXY_CONFIG_PATH
+
+
+def assert_scenario_branches(scenario_name, config_path, env_overrides):
+    """Cheap sanity check that the conditional blocks actually rendered.
+
+    Without this, a template refactor that silently stopped emitting the
+    suspension or Coraza block would leave the gate passing while covering
+    less than it claims to.
+    """
+    with open(config_path) as fh:
+        text = fh.read()
+    expected = {
+        'suspension': ('acl is_suspended_domain',
+                       'HAPROXY_SUSPENSION_ENABLED' in env_overrides),
+        'coraza': ('filter spoe engine coraza',
+                   'HAPROXY_CORAZA_SPOE_BACKEND' in env_overrides),
+    }
+    for label, (needle, should_be_present) in expected.items():
+        present = needle in text
+        if present != should_be_present:
+            fail(f'[{scenario_name}] {label} block {"missing" if should_be_present else "unexpectedly present"} '
+                 f'in the rendered config (looked for {needle!r}). The gate is '
+                 f'not covering what it thinks it is.')
+
+
+def _dump_context(config_path, output):
+    """Print the rendered lines HAProxy complained about.
+
+    The temp dir is deleted on the way out, so the build log has to carry the
+    evidence. HAProxy reports `parsing [<file>:<line>]`; show a window around
+    each reported line rather than dumping ~1500 lines of config.
+    """
+    line_numbers = sorted({
+        int(n) for n in re.findall(
+            r'parsing \[' + re.escape(config_path) + r':(\d+)\]', output)
+    })
+    if not line_numbers:
+        return
+    with open(config_path) as fh:
+        lines = fh.read().splitlines()
+    sys.stderr.write('[validate-config] --- rendered config around the error ---\n')
+    for number in line_numbers:
+        start = max(1, number - 6)
+        end = min(len(lines), number + 4)
+        for index in range(start, end + 1):
+            marker = '>>' if index == number else '  '
+            sys.stderr.write(f'{marker}{index:6d}| {lines[index - 1]}\n')
+        sys.stderr.write('[validate-config] ---\n')
+
+
+def validate(scenario_name, config_path):
+    result = subprocess.run(['haproxy', '-c', '-f', config_path],
+                            capture_output=True, text=True)
+    if result.returncode != 0:
+        output = result.stdout + result.stderr
+        sys.stderr.write(
+            f'\n[validate-config] ===== {scenario_name}: haproxy REJECTED the '
+            f'rendered configuration (exit {result.returncode}) =====\n')
+        sys.stderr.write(output if output.endswith('\n') else output + '\n')
+        _dump_context(config_path, output)
+        sys.stderr.write(
+            '[validate-config] This is a real HAProxy parse failure. Shipping it '
+            'would leave the container Up with ports 80/443 unbound and every '
+            'site on the host down, while /health still returns 200.\n')
+        sys.stderr.flush()
+        raise SystemExit(1)
+
+    # Exit code 0 is the verdict. Warnings are expected and are NOT failures:
+    # "Can't load stats file" always fires at build time, and HAProxy emits
+    # path_reg/ACL advisories on a perfectly valid config.
+    noise = (result.stdout + result.stderr).strip()
+    log(f'{scenario_name}: haproxy -c OK (exit 0)')
+    if noise:
+        for line in noise.splitlines():
+            log(f'  {scenario_name}: haproxy said: {line}')
+
+
+def main():
+    require_haproxy_binary()
+
+    if not os.path.isdir(ETC_HAPROXY) or not os.access(ETC_HAPROXY, os.W_OK):
+        fail(f'{ETC_HAPROXY} must exist and be writable - several templates '
+             'reference files there by absolute path. Run this inside the image.')
+
+    created = CreatedFiles()
+    workdir = tempfile.mkdtemp(prefix='haproxy-validate-')
+    try:
+        for path, content in STUB_FILES.items():
+            created.ensure(path, content)
+
+        for scenario_name, env_overrides in SCENARIOS:
+            log(f'rendering scenario "{scenario_name}" '
+                f'({env_overrides or "no optional features"})')
+            config_path = render(scenario_name, env_overrides, workdir)
+            assert_scenario_branches(scenario_name, config_path, env_overrides)
+            validate(scenario_name, config_path)
+    finally:
+        created.cleanup()
+        shutil.rmtree(workdir, ignore_errors=True)
+        shutil.rmtree(_LOG_DIR, ignore_errors=True)
+
+    log('all scenarios accepted by the real haproxy binary')
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

templates/hap_backend.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -29,7 +29,7 @@ backend {{ name }}-sse-backend
 
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_http_check.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option httpchk
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     {% if ssl_enabled %}http-request set-header X-Forwarded-Proto https if { ssl_fc }{% endif %}

templates/hap_backend_longlived.tpl

@@ -14,7 +14,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -31,7 +31,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_websocket.tpl

@@ -7,7 +7,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -24,7 +24,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_coraza_spoe_engine.tpl

@@ -10,9 +10,9 @@
 # config.yaml.
 #
 # Reference: this config follows the shape from coraza-spoa's upstream
-# example/haproxy/coraza.cfg (v0.7.1). Arg names + ordering are required by
-# Coraza-SPOA exactly as specified — DO NOT reorder or rename without
-# coordinating with the agent.
+# example/haproxy/coraza.cfg (v0.7.1). Arg NAMES are a contract with
+# coraza-spoa — DO NOT rename without coordinating with the agent. Ordering:
+# only `app` is positional (it must be FIRST); the rest are matched by name.
 
 [coraza]
 
@@ -37,13 +37,60 @@ spoe-agent coraza
     timeout     processing  100ms
 
     use-backend coraza-spoa-backend
-    log         global
+
+    # NO `log global` here, deliberately.
+    #
+    # `log global` in a spoe-agent emits one line PER INSPECTED REQUEST, e.g.
+    #   SPOE: [coraza] <GROUP:coraza-req> sid=537 st=0 0/0/0/0/0 32/32 0/0 0/467
+    # Measured on whp01 immediately after access logging started working:
+    # 618 SPOE lines vs 669 real access lines -- it was ~48% of the log volume,
+    # i.e. it would roughly DOUBLE the edge's log footprint (~400 MB/day extra)
+    # to record `st=0` over and over.
+    #
+    # It carries nothing incident response needs: the WAF's verdict is already
+    # visible in the access log line (status 403 + the `id=` UUID, which joins
+    # to /var/log/coraza/audit.log for the rule_id), and per-transaction WAF
+    # detail is written by the SPOA itself to /var/log/coraza/spoa.log.
+    # Agent-level failures still surface via `option set-on-error error` ->
+    # var(txn.coraza.error) and the fail-open path in hap_listener.tpl.
 
 # Per-request inspection message. No `event` directive — fires only when
 # explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.
-# Arg order/names are mandatory: Coraza-SPOA parses positionally and renames
-# break the agent. `app=str(haproxy)` is the literal application name from
-# coraza-spoa's config.yaml `applications:` block.
+#
+# How coraza-spoa v0.7.1 reads these args (measured from source, not inferred):
+#   * `app` MUST be the first arg. internal/agent.go HandleSPOE reads exactly
+#     one KV entry and Panics if its name is not "app".
+#   * Every other arg is matched BY NAME in a switch (internal/application.go
+#     HandleRequest); order is irrelevant and unknown names are ignored with a
+#     debug log. So a typo'd name silently drops that input — renames still
+#     break inspection, they just don't error.
+# `app=str(haproxy)` is the literal application name from coraza-spoa's
+# config.yaml `applications:` block.
+#
+# id=unique-id makes Coraza's transaction ID the SAME UUID HAProxy generates
+# via `unique-id-format %[uuid()]` (hap_header.tpl) — the value shown on the
+# 403-waf.html page, sent as X-Request-Reference, and logged as `id=` in
+# /var/log/haproxy.log. coraza-spoa passes it to waf.NewTransactionWithID(),
+# so it becomes `transaction.id` in /var/log/coraza/audit.log and
+# `match.unique_id` in /var/log/coraza/spoa.log: a customer-quoted reference
+# greps straight to the rule that fired. Without it coraza-spoa invents a
+# random 16-letter A-Z ID that appears nowhere on the HAProxy side. If the
+# fetch ever came back empty the agent falls back to that random ID, so this
+# cannot break inspection.
+#
+# exportRuleIDs=bool(true) makes coraza-spoa (application.go exportWAFMetrics)
+# set var(txn.coraza.rule_ids) = comma list of every matched CRS attack rule
+# (910000-959999 + 190000-199999; control rules such as our 99xxxxxxx host
+# exceptions and 2000xx body rules are NOT listed), alongside rules_hit and
+# anomaly_score which it sets regardless. var(txn.coraza.ruleid) is set on
+# every interruption (agent.go) -- for anomaly blocks that is 949110, so
+# rule_ids is what actually says WHY. All three are logged in the access
+# log (hap_listener.tpl log-format: wrule= wrules= wscore=).
+#
+# Response inspection (a `coraza-res` message) is NOT configured:
+# config.yaml has response_check: false. If it is ever enabled, coraza-res
+# MUST also send `id=unique-id` -- HandleResponse
+# looks the request transaction up by that ID and errors without it.
 #
 # src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
 # top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
@@ -51,7 +98,7 @@ spoe-agent coraza
 # when no proxy headers are present. This is what shows up as `client_ip`
 # in /var/log/coraza/audit.log and the rule manager's view-matches panel.
 spoe-message coraza-req
-    args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
+    args app=str(haproxy) id=unique-id exportRuleIDs=bool(true) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
 
 # Group binding for send-spoe-group invocation in the frontend. One group,
 # one message; could add more in the future (e.g. coraza-res for response

templates/hap_header.tpl

@@ -2,20 +2,46 @@
 # Global settings
 #---------------------------------------------------------------------
 global
-    # to have these messages end up in /var/log/haproxy.log you will
-    # need to:
+    # ACCESS LOG DESTINATION.
     #
-    # 1) configure syslog to accept network log events.  This is done
-    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
-    #    /etc/sysconfig/syslog
+    # This used to be `log 127.0.0.1 local2`, which was a silent black hole:
+    # 127.0.0.1 is the CONTAINER's own loopback, nothing has ever listened on
+    # udp/514 in the container netns, and there is no /dev/log in the image.
+    # Every access log line -- ~1.5M/day across the whole edge -- was written
+    # to a socket with no receiver and dropped. Nothing errored, nothing
+    # warned, and `haproxy -c` was perfectly happy. The cost only shows up
+    # during an incident: per-IP 429s, tarpits, wp-admin gate redirects,
+    # WAF 403s and `silent-drop`s left no record anywhere, so the edge could
+    # not be asked what it had rejected. Only aggregate stick-table counters
+    # survived.
     #
-    # 2) configure local2 events to go to the /var/log/haproxy.log
-    #   file. A line like the following can be added to
-    #   /etc/sysconfig/syslog
+    # Now points at the DOCKER BRIDGE GATEWAY, where the host's rsyslog has an
+    # imudp listener bound (installed idempotently by WHP's
+    # setup-haproxy-logrotate.sh, which also writes the logrotate stanza).
+    # The host writes local2 to /var/log/haproxy.log and stops it there, so it
+    # does not also flood /var/log/messages or the Graylog forwarder.
     #
-    #    local2.*                       /var/log/haproxy.log
+    # WHY NOT `log stdout format raw local0`: it is INCOMPATIBLE with the
+    # `daemon` keyword below, and incompatible SILENTLY. Verified on the
+    # pinned 3.0.11 binary: with `daemon` set, a `log stdout` config serves
+    # traffic normally and emits ZERO log lines, and `haproxy -c` returns 0
+    # with no error and no warning -- so the CI config gate
+    # (scripts/validate-rendered-config.py) cannot catch it either. Making it
+    # work means dropping `daemon` / adding -db so haproxy stays in the
+    # foreground, which in turn breaks the three synchronous
+    # `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites in
+    # haproxy_manager.py (they would block until the 180s timeout and then be
+    # killed). That is a change to the exact code path whose failure mode is
+    # "container Up, ports 80/443 never bound, every site down, /health still
+    # 200". Not worth it for a logging change.
     #
-    log         127.0.0.1 local2
+    # UDP means a dead listener degrades to dropped log lines, never to a
+    # stalled or failing request path -- the correct failure direction for an
+    # edge fronting ~60 customer sites.
+    #
+    # len 2048 accommodates the enriched log-format in hap_listener.tpl
+    # (URL + User-Agent + UUID); the default 1024 would truncate long ones.
+    log         {{ syslog_target }} len 2048 format rfc5424 local2 info
 
     chroot      /var/lib/haproxy
     pidfile     /var/run/haproxy.pid
@@ -27,6 +53,36 @@ global
     # SSL and Performance
     tune.ssl.default-dh-param 2048
 
+    # Required by the `http-request normalize-uri` chain at the top of the
+    # `web` frontend (hap_listener.tpl). normalize-uri is still flagged
+    # EXPERIMENTAL in HAProxy 3.0, and HAProxy REFUSES TO START without this
+    # opt-in -- not a warning, a fatal:
+    #   [ALERT] config : parsing [...] : 'normalize-uri' action is
+    #   experimental, must be allowed via a global
+    #   'expose-experimental-directives'
+    # (verified against real haproxy 3.0.11-9e587df: `haproxy -c` exits 1).
+    # So this line and the normalize-uri rules must be added/removed together.
+    #
+    # Dropping this one alone does NOT crash-loop the container -- the truth
+    # is worse: it is a SILENT TOTAL OUTAGE that nothing escalates. Container
+    # init (scripts/init.py -> haproxy_manager.do_initial_setup()) calls
+    # generate_config() (which still succeeds -- Jinja doesn't validate
+    # HAProxy semantics) and then start_haproxy(), which runs `haproxy -c`,
+    # sees it fail, logs an error, and RETURNS WITHOUT RAISING. init.py exits
+    # 0. scripts/start-up.sh then execs gunicorn as PID 1 regardless. Result:
+    # the container stays "Up", ports 80/443 are never bound, EVERY SITE ON
+    # THE HOST IS DOWN, and the in-container supervisor loop
+    # (ensure_haproxy.py, every HAPROXY_SUPERVISOR_INTERVAL seconds) retries
+    # the identical failing render forever without ever escalating. Worse
+    # still, GET /health keeps returning HTTP 200 -- health_check() only
+    # answers 500 on a database error; a dead haproxy just flips the JSON
+    # body's "haproxy_status" to "stopped" while the status code a naive
+    # monitor checks never changes. Do not trust /health alone to catch this.
+    #
+    # This exposes ONLY the experimental directives that are actually used --
+    # it does not change the behaviour of anything else in this file.
+    expose-experimental-directives
+
     # HTTP/3 over QUIC. The Debian haproxy package is built against system
     # OpenSSL via the compatibility shim (USE_QUIC_OPENSSL_COMPAT), which is
     # not a native QUIC TLS stack. HAProxy therefore rejects `quic*@` binds
@@ -50,6 +106,41 @@ global
 
     # Stats persistence for zero-downtime reloads
     stats-file /var/lib/haproxy/stats.dat
+{%- if coraza_spoe_backend %}
+
+    # WAF BODY INSPECTION LIMIT. Only emitted when Coraza is wired in, so a
+    # WAF-off / standalone config is unchanged byte-for-byte. 16384 is also
+    # HAProxy's default: this line changes no behaviour or memory, it pins
+    # the value so nobody raises it without reading this.
+    #
+    # The request body Coraza sees is whatever `req.body` holds when
+    # send-spoe-group fires, and with `option http-buffer-request`
+    # (hap_listener.tpl) that is "the whole body, or as much as fits in ONE
+    # buffer". So bufsize IS the inspection limit: a body up to roughly
+    # bufsize - maxrewrite (1024) - the request headers (~15 KB) is inspected
+    # whole and stamped `complete`; anything larger is forwarded intact, its
+    # first ~15 KB inspected, and stamped `partial`. Large uploads are never
+    # rejected or held for it.
+    #
+    # DO NOT RAISE THIS WITHOUT RE-MEASURING -- three ceilings, all measured:
+    #   * TIME. Coraza+CRS 4.25 costs ~3.3 ms per KB of body (linear; 31 KB
+    #     base64 = 104 ms, 15 KB = ~50 ms on a 32-core box). The SPOE
+    #     `timeout processing` is 100 ms (hap_coraza_spoe_engine.tpl) and a
+    #     timeout FAILS OPEN: with tune.bufsize 32768 a payload at byte 0 of a
+    #     30 KB base64 multipart body passed 933180 uninspected. A larger
+    #     buffer is a bypass, not more coverage, unless the timeout (and CPU
+    #     budget) grows with it.
+    #   * FRAME. coraza-spoa's SPOP library (dropmorepackets/haproxy-go
+    #     v0.0.8) refuses a HELLO whose max-frame-size exceeds 65535; HAProxy
+    #     derives its offer from bufsize. bufsize >= 64 KB = every SPOE
+    #     handshake fails = WAF silently off for all traffic.
+    #   * MEMORY. maxconn 4000 x 2 buffers x bufsize, inside the 1 GB edge
+    #     container.
+    # Coraza's SecRequestBodyLimit (coraza-spoa/config.yaml, 131072) must stay
+    # above anything this can deliver. Pinned by
+    # scripts/test-waf-body-inspection.py.
+    tune.bufsize 16384
+{%- endif %}
 
 #---------------------------------------------------------------------
 # DNS resolver for Docker container name resolution
@@ -92,7 +183,11 @@ defaults
     maxconn                 3000
 
     # Per-request unique reference, used:
-    #   - in the log line (httplog includes %ID)
+    #   - in the access log line, as the `id=` field of the custom log-format
+    #     in hap_listener.tpl. NOTE: `option httplog` does NOT include %ID
+    #     (verified against haproxy 3.0.11) -- this comment used to claim it
+    #     did, which made the support workflow below look supported when it
+    #     was not. The explicit log-format is what actually carries it.
     #   - echoed to clients in the X-Request-Reference response header on
     #     WAF blocks so a customer can quote it when opening a support ticket
     #   - embedded in /etc/haproxy/errors/403-waf.html so a blocked visitor

templates/hap_listener.tpl

@@ -19,9 +19,182 @@ frontend web
     # response, including haproxy-generated ones (blocks, default page).
     http-after-response set-header alt-svc "h3=\":443\"; ma=86400"
 
-    # Capture Host header so it appears in httplog output (in %hr field)
+    # Capture Host header so it appears in httplog output (in %hr field).
+    # ORDER IS LOAD-BEARING: this is capture slot 0, referenced by the
+    # access log-format below as %[capture.req.hdr(0)].
     http-request capture req.hdr(Host) len 64
 
+    # Capture slot 1 = User-Agent. Incident response needs it to tell a
+    # scanner from a browser, and it is not in `option httplog` output.
+    # Any new capture MUST be appended AFTER this line, never inserted
+    # above it, or the slot indices in the log-format silently shift and
+    # the access log starts attributing the wrong string to the wrong field.
+    # req.fhdr(), NOT req.hdr(): req.hdr() treats the header as a comma-
+    # separated list and returns only the LAST element. Real User-Agent strings
+    # contain commas -- "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
+    # AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
+    # captured with req.hdr() logs as just "like Gecko) Chrome/131.0.0.0
+    # Safari/537.36", silently losing the platform half -- which is exactly the
+    # half you need to tell a spoofed crawler from a real browser.
+    # Observed in production on whp01 before this was corrected.
+    http-request capture req.fhdr(User-Agent) len 200
+
+    # --- Access logging -----------------------------------------------------
+    # Scoped to THIS frontend on purpose: it references capture slots and
+    # var(txn.real_ip), which only exist here. Putting it in `defaults` would
+    # apply it to the stats frontend and every backend too, where those
+    # samples are undefined.
+    #
+    # `log-format` overrides `option httplog` for this proxy (haproxy emits a
+    # harmless warning saying so). The first 16 fields are byte-identical to
+    # the 3.0 httplog default, so anything that already parses httplog keeps
+    # working; the `key=value` tail is additive.
+    #
+    # Why the default httplog is not enough for incident response:
+    #   %ci  is the PROXY's address for Cloudflare-fronted sites, not the
+    #        visitor. The real client is var(txn.real_ip), resolved further
+    #        down from CF-Connecting-IP / X-Real-IP / X-Forwarded-For and only
+    #        honoured from trusted proxies. BOTH are logged: cip= is who to
+    #        rate-limit or block, %ci is which edge it arrived through.
+    #   %ID  is NOT included by `option httplog` (verified against
+    #        haproxy 3.0.11). Without it the documented support workflow
+    #        -- X-Request-Reference -> access log -> coraza audit.log -> rule_id
+    #        -- cannot be completed. id= is what makes that join possible.
+    #   host=/ua= identify the vhost and client; %ST/%B/%tsc give status,
+    #        bytes and the termination state that distinguishes a rate-limit
+    #        deny (PR--) from a tarpit (PT--) from a normal close.
+    #   wafb=/wafe= WAF body coverage: wafb is complete|partial|- (did Coraza
+    #        get the whole request body, see the Coraza block below);
+    #        wafe is var(txn.coraza.error), set when the SPOE call failed or
+    #        timed out and the request went through UNINSPECTED (fail-open).
+    #        Both are "-" when the WAF is off or the request had no body.
+    #   wrule=/wrules=/wscore= WHY Coraza acted, without opening audit.log:
+    #        wrule is the interrupting rule (949110 for anomaly blocks),
+    #        wrules every matched CRS attack rule (e.g. 920440,949110),
+    #        wscore the inbound anomaly score. Set by coraza-spoa via the
+    #        exportRuleIDs SPOE arg (hap_coraza_spoe_engine.tpl). All "-"
+    #        when the WAF is off; wrule is "-" unless Coraza interrupted.
+    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major] wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)] wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] wscore=%[var(txn.coraza.anomaly_score)]"
+
+    # --- URI normalisation (MUST be the first path-touching block here) ---
+    # Every path-based control in this frontend (the ACME health-check bypass,
+    # wp-login, xmlrpc, the wp-json/batch virtual patch, the wp-admin gate, the
+    # blocked-IP and suspension set-path rules, and everything Coraza inspects)
+    # matched the RAW request-target while the backend NORMALISED and DECODED
+    # it before resolving a file. Every gap between those two behaviours is a
+    # bypass, and each one had to be patched individually. Five were found in
+    # the wp-admin gate alone, all the same class:
+    #
+    #   //wp-admin/plugins.php            raw path starts "//" -> the safe-path
+    #                                     guard failed -> request fell through
+    #                                     ungated, backend served it
+    #   /wp-admin/css/../plugins.php      matched the css/js/images asset
+    #                                     bypass; backend resolved ".." and
+    #                                     booted plugins.php
+    #   /wp-admin/js/%2e%2e/plugins.php   same, with the ".." percent-encoded
+    #   /wp%2Dadmin/plugins.php           "wp-admin" spelled with %2D never
+    #                                     matched any wp-admin ACL at all
+    #   /wp-admin%2Fplugins.php           encoded separator; see the dedicated
+    #                                     rule in the wp-admin gate below
+    #
+    # Rather than keep bolting a counter-pattern onto each rule, normalise the
+    # URI once, here, so every rule below matches the SAME string the backend
+    # will resolve. HAProxy rewrites the request-target in place, so the
+    # backend receives the normalised form too.
+    #
+    # ORDER IS LOAD-BEARING and was determined empirically against real
+    # haproxy 3.0.11, not from the docs. The decoders must run BEFORE the path
+    # walkers: with the reverse order, /wp-admin/js/%2e%2e/plugins.php ends up
+    # as /wp-admin/js/../plugins.php -- decoded, but the ".." left unresolved,
+    # because path-strip-dotdot had already run by the time the "%2e%2e"
+    # became "..". Verified both directions side by side.
+    #
+    #   percent-to-uppercase       %2f -> %2F. Canonicalises the spelling of
+    #                              whatever stays encoded, so downstream rules
+    #                              need one case of each escape, not two.
+    #   percent-decode-unreserved  Decodes ONLY RFC 3986 unreserved chars
+    #                              (A-Za-z0-9-._~). This is what turns %2e%2e
+    #                              into .. and wp%2Dadmin into wp-admin.
+    #                              Reserved escapes are deliberately left
+    #                              alone -- %2F in particular, which is why
+    #                              the wp-admin gate needs its own encoded-
+    #                              slash rule (see below).
+    #   path-merge-slashes         //x -> /x. Also removes the entire class of
+    #                              "leading // defeats an anchored regex".
+    #   path-strip-dot             /a/./b -> /a/b.
+    #   path-strip-dotdot full     /a/b/../c -> /a/c. "full" additionally
+    #                              resolves ".." segments that would climb
+    #                              above the root (/../../wp-admin/x.php ->
+    #                              /wp-admin/x.php); without "full" HAProxy
+    #                              leaves those in place and the vector
+    #                              survives -- measured, both forms tested.
+    #
+    # DELIBERATELY NOT ENABLED: query-sort-by-name. Reordering query-string
+    # parameters would silently break anything that signs or caches on the
+    # exact query string (signed asset URLs, HMAC'd callbacks, CDN cache
+    # keys). This is NOT because the enabled normalizers already leave the
+    # query alone -- see BLAST RADIUS just below, they don't -- it is a
+    # deliberate line drawn between "case-fold / decode", which RFC 3986
+    # defines as no-ops, and "reorder", which is not a no-op for a caller
+    # treating the query as an opaque signed string.
+    #
+    # BLAST RADIUS: this block applies to EVERY request for EVERY site on
+    # EVERY tier, so the decoding was kept minimal on purpose -- and it is
+    # NOT scoped to the path. percent-to-uppercase and percent-decode-
+    # unreserved normalise the WHOLE request-target as HAProxy parses it --
+    # query string included, not just the path component the ACLs below
+    # match on -- and the BACKEND receives the rewritten query on the wire,
+    # not just an internal haproxy view of it. Measured against real HAProxy
+    # 3.0.11:
+    #   /a?sig=%2babc%2fdef  -> /a?sig=%2Babc%2Fdef   (percent-hex upper-cased)
+    #   /a?b=%41%42%43       -> /a?b=ABC               (unreserved chars decoded)
+    #   /a?tok=%7e%2d%5f%2e  -> /a?tok=~-_.            (unreserved chars decoded)
+    # Only parameter ORDER is preserved -- that guarantee is exactly why
+    # query-sort-by-name above is the one normalizer in this family left
+    # disabled. Per RFC 3986 both enabled rewrites are defined as the same
+    # URI (case in a percent-escape, and an unreserved character vs. its
+    # escape, carry no distinct meaning), but "the same URI" is not "the
+    # same bytes": an application that HMACs or otherwise signs the RAW
+    # query string, rather than parsing it first, could see a mutated value
+    # and fail to verify an otherwise-legitimate request. Checked against
+    # this fleet: no .NET backends (.NET's UrlEncode emits lowercase
+    # percent-hex, which percent-to-uppercase would rewrite) and no
+    # URL-in-path proxies, so there is no known victim today -- but do not
+    # assume "path only" from this block; that was the actual bug in an
+    # earlier draft of this comment.
+    #
+    # normalize-uri is EXPERIMENTAL in 3.0 and requires
+    # `expose-experimental-directives` in the global section
+    # (hap_header.tpl). Without it HAProxy does not start. Remove one and you
+    # must remove the other.
+    http-request normalize-uri percent-to-uppercase
+    http-request normalize-uri percent-decode-unreserved
+    http-request normalize-uri path-merge-slashes
+    http-request normalize-uri path-strip-dot
+    http-request normalize-uri path-strip-dotdot full
+
+    # --- Trusted-proxy gate (MUST precede real-IP resolution below) ---
+    # CF-Connecting-IP / X-Real-IP / X-Forwarded-For are client-supplied. Any
+    # peer that is not a known reverse proxy gets them stripped, so the
+    # set-var chain below falls through to `src` -- the real TCP peer.
+    #
+    # Without this, a direct client dictates txn.real_ip, and every control
+    # keyed on that variable trusts the attacker's own claim: rate limiting
+    # (track-sc0), the trusted-IP whitelist, the wp-login brute-force table and
+    # cookie challenge, the wp-json/batch/v1 virtual patch, IP blocking, and
+    # Coraza's src-ip. Spoofing a whitelisted IP bypassed all of them.
+    #
+    # ORDER MATTERS: these must come before the set-var lines. HAProxy applies
+    # http-request rules in file order, so a strip placed afterwards would
+    # validate cleanly and accomplish nothing.
+    #
+    # Cloudflare-fronted domains keep working: CF's edge matches
+    # from_trusted_proxy, so its CF-Connecting-IP survives.
+    acl from_trusted_proxy src -f /etc/haproxy/cloudflare_ips.list -f /etc/haproxy/trusted_proxies.list
+    http-request del-header CF-Connecting-IP if !from_trusted_proxy
+    http-request del-header X-Real-IP        if !from_trusted_proxy
+    http-request del-header X-Forwarded-For  if !from_trusted_proxy
+
     # Detect real client IP from proxy headers if they exist
     # Priority: CF-Connecting-IP (Cloudflare) > X-Real-IP > X-Forwarded-For > src
     acl has_cf_connecting_ip req.hdr(CF-Connecting-IP) -m found
@@ -96,6 +269,42 @@ frontend web
     acl has_login_cookie req.cook(whplc) -m found
     http-request deny deny_status 403 if METH_POST wp_login_path !has_login_cookie !is_local !is_trusted_ip !is_whitelisted
 
+    # --- WordPress xmlrpc.php flood protection ---
+    # xmlrpc.php floods are a common, sustained abuse pattern that the generic
+    # limits above don't catch: those trigger at 3000/5000 req/10s (300-500
+    # req/s, sized for media-heavy pageloads), while observed xmlrpc floods run
+    # at just a few req/s for hours -- comfortably under that ceiling but still
+    # enough to pin PHP-FPM workers and show up as 503s for the rest of the
+    # site. Same shape of problem as wp-login credential stuffing, so it gets
+    # the same fix: track POSTs to xmlrpc.php per real client IP in a DEDICATED
+    # 60s table (sc2 / backend xmlrpc_bruteforce, defined in
+    # hap_security_tables.tpl -- kept separate from wp_bruteforce so the two
+    # endpoints' traffic can't inflate each other's counter, see that file for
+    # the reasoning) and tarpit once an IP exceeds the threshold.
+    #
+    # Threshold is 60/min (double wp-login's 30/min), not because xmlrpc abuse
+    # is less severe but because legitimate traffic here is machine-to-machine
+    # rather than a human filling out a form: Jetpack sync, the WordPress
+    # mobile app, and remote-publishing clients (e.g. an offline blog editor)
+    # can legitimately burst several xmlrpc calls in quick succession. 60/min
+    # (1 req/s average over the window) comfortably absorbs that burst while
+    # still tripping well before an hours-long few-req/s flood does real
+    # damage -- at 2 req/s sustained the 60s counter clears the threshold in
+    # under a minute.
+    #
+    # Tarpit (not deny), matching the wp-login rule: this is per-IP tracking
+    # of a bounded set of offenders, not the wp-login cookie challenge's
+    # distributed hundreds-of-thousands-of-IPs scenario where holding
+    # connections would exhaust HAProxy itself, so tying up the flooding IP's
+    # connections is the cheaper and more effective response. path_end (not
+    # path_beg) covers subdirectory WP installs, same reasoning as wp-login.
+    # Honors the same whitelist (RFC1918 / trusted_ips.list / trusted_ips.map)
+    # so health checks and trusted infrastructure are unaffected, and legit
+    # clients under the threshold are never blocked outright.
+    acl xmlrpc_path path_end /xmlrpc.php
+    http-request track-sc2 var(txn.real_ip) table xmlrpc_bruteforce if METH_POST xmlrpc_path
+    http-request tarpit deny_status 429 if METH_POST xmlrpc_path { sc_http_req_rate(2) gt 60 } !is_local !is_trusted_ip !is_whitelisted
+
     # WordPress REST batch endpoint lockdown ("wp2shell": CVE-2026-63030 +
     # CVE-2026-60137). Chaining a core SQL injection with REST batch-route
     # confusion gives unauthenticated RCE on WP 6.9.0-6.9.4 and 7.0.0-7.0.1
@@ -125,9 +334,268 @@ frontend web
     http-request deny deny_status 403 if wp_batch_route !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
     http-request deny deny_status 403 if wp_batch_route_enc !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
 
+    # --- WordPress admin edge gate ---
+    # Measured on whp02, 2026-08-14: distributed unauthenticated GETs booting
+    # WordPress just to bounce back a login redirect --
+    #   2801 GET /wp-admin/profile.php   2781 GET /wp-admin/edit.php   2761 GET /wp-admin/plugins.php
+    # -- spread across many source IPs at roughly 4 req/min per IP, each hit
+    # burning a PHP-FPM/lsphp worker. Two sites absorbed 1,243 resulting 503s
+    # in nine hours as their pools saturated.
+    #
+    # IDENTITY, NOT RATE. This is the one rule in this file that tracks
+    # nothing and has no stick-table counter or threshold. Every rate-based
+    # control above (the generic limits, wp_bruteforce, xmlrpc_bruteforce) is
+    # per-IP, and per-IP rate is exactly what this attack is engineered to
+    # stay under: ~4 req/min from any single IP is indistinguishable from a
+    # slow human, and the source set is large enough that no threshold can be
+    # lowered to catch it without also catching real visitors. There is also
+    # no free stick-table slot left to try anyway -- sc0/sc1/sc2 are already
+    # used above and HAProxy's default tune.stick-counters is 3, so a fourth
+    # tracked counter is not an option here. DO NOT "simplify" this into a
+    # rate/threshold rule later: the whole point is that a threshold cannot
+    # see this traffic. Instead we gate on identity -- a real logged-in
+    # WordPress user always carries a wordpress_logged_in_* cookie (the same
+    # ACL the wp2shell block above already declares), and an unauthenticated
+    # request to a wp-admin page has no legitimate reason to boot PHP at all.
+    #
+    # 302, not 403. WordPress itself redirects an unauthenticated /wp-admin/
+    # request to wp-login.php, so replicating that at the edge means an admin
+    # whose session merely expired lands on the normal login screen instead
+    # of an error page -- we are not trading a bot problem for a support
+    # ticket. Bots get a cheap redirect they ignore.
+    #
+    # path_reg, not path_beg. A subdirectory install at /blog/wp-admin/ slips
+    # past a prefix match; path_reg with an optional leading "/" catches both
+    # root and subdirectory installs, same reasoning as the wp-login and
+    # xmlrpc path_end rules above.
+    #
+    # regsub rewrites the redirect target itself, so /blog/wp-admin/x.php
+    # redirects to /blog/wp-login.php rather than 404ing at the site root.
+    #
+    # DO NOT write regsub's regex argument with a capturing group / literal
+    # parentheses, e.g. regsub((^|/)wp-admin/.*,\1wp-login.php) -- neither
+    # inlined into the redirect's `location` nor in a standalone set-var.
+    # HAProxy 3.0.11's converter-argument parser counts parens to find the
+    # end of the regsub(...) call itself, so the *inner* "(^|/)" grouping
+    # parens are misread as closing the outer call early -- it does not
+    # matter whether the argument is quoted ("...": still fails) or the
+    # parens are backslash-escaped (\(...\): still fails). Every such form
+    # was verified against real HAProxy 3.0.11-1+deb13u3 and all produce the
+    # same ALERT: "invalid arg 2 in converter 'regsub' : missing arguments
+    # (got 1/2)". This is a converter-argument-parsing limitation, not a
+    # log-format/`%[...]` issue -- the identical failure reproduces in a
+    # plain set-var (outside any log-format string), which rules out the
+    # `location` value's log-format context as the cause.
+    #
+    # The fix sidesteps groups/backreferences entirely: HTTP paths always
+    # start with "/", so the leading "(^|/)" alternation is redundant --
+    # matching the literal substring "/wp-admin/" (both slashes, no group)
+    # is sufficient to anchor to a real path segment (a false match like
+    # "/somewp-admin/" doesn't contain "/wp-admin/" as a substring, since
+    # there's no "/" directly before "wp-admin"). No backreference is
+    # needed either: regsub only replaces the matched substring, so
+    # replacing "/wp-admin/.*" with a literal "/wp-login.php" leaves
+    # whatever precedes it (the subdirectory-install prefix, if any)
+    # untouched. Computed in its own set-var so it is a plain sample
+    # expression, not something baked into the redirect's log-format
+    # string. Behaviorally verified live against real HAProxy 3.0.11:
+    # /wp-admin/edit.php -> /wp-login.php and
+    # /blog/wp-admin/plugins.php -> /blog/wp-login.php, both with
+    # redirect_to preserved. See
+    # .superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-3b-report.md.
+    #
+    # redirect_to carries the path only (%[path,url_enc]), not the query
+    # string -- deliberate, see design spec. An admin bounced off
+    # post.php?post=123&action=edit lands back on a blank post.php rather
+    # than that exact post. Capturing the full URI needs capture.req.uri
+    # (extra config, and the captured value is length-capped) for a benefit
+    # that only matters on session expiry, so it was not worth it.
+    #
+    # THE ALLOWLIST IS MEASURED, NOT GUESSED -- taken from actual fleet
+    # traffic returning 200 on /wp-admin/*. admin-ajax.php and admin-post.php
+    # are the standard front-end AJAX/form-handler endpoints real themes and
+    # plugins call while logged out. Critically, wp-login.php loads its OWN
+    # css/js FROM /wp-admin/ (load-styles.php, load-scripts.php, and the
+    # static wp_admin_asset dirs below) -- miss those and every login page on
+    # the fleet renders unstyled with a broken password-strength meter, while
+    # wp-login.php itself still returns 200, making it a silent regression
+    # that "looks like" the gate is working.
+    #
+    # Each allowlist entry is anchored to /wp-admin/<file>, not a bare
+    # filename suffix. A bare `path_end /admin-ajax.php` also matches
+    # /wp-admin/evil/admin-ajax.php -- which ALSO matches wp_admin_path
+    # (path_reg only requires /wp-admin/ to appear somewhere), so an
+    # attacker-inserted path segment would sail through this allowlist
+    # ungated and boot full WordPress, exactly the resource exhaustion this
+    # gate exists to stop. Anchoring still covers subdirectory installs via
+    # suffix matching (/blog/wp-admin/admin-ajax.php ends with
+    # /wp-admin/admin-ajax.php) while rejecting an inserted directory.
+    #
+    # install.php is DELIBERATELY NOT allowlisted. It is legitimately
+    # reachable without a cookie during a fresh install, but it is also a
+    # standing scanner target and a real takeover vector on a site that was
+    # half-installed and then abandoned. Anyone genuinely installing uses the
+    # per-site exempt-list opt-out below instead.
+    #
+    # Honors the same whitelist as every other rule in this frontend
+    # (RFC1918 / trusted_ips.list / trusted_ips.map), and a per-site opt-out
+    # via /etc/haproxy/wpadmin_gate_exempt.list (operator-managed, seeded
+    # empty by start-up.sh) for sites where a plugin legitimately serves
+    # unauthenticated visitors from a /wp-admin/ URL outside this allowlist.
+    # EVERY ACL BELOW MATCHES THE NORMALISED PATH. The normalize-uri chain at
+    # the top of this frontend has already merged duplicate slashes, resolved
+    # "." / ".." segments (including percent-encoded ones) and decoded
+    # unreserved escapes by the time these run, so these patterns only have to
+    # describe the ONE canonical spelling the backend will resolve -- they do
+    # not have to anticipate every encoding of it. That is the whole point of
+    # the normalisation block; do not "harden" these regexes by re-adding
+    # encoding variants, fix the normalisation instead.
+    #
+    # wp_admin_safe_path guards against an OPEN REDIRECT this gate would
+    # otherwise introduce. The redirect target below is built by rewriting
+    # `path` with regsub -- regsub only replaces the matched substring, so
+    # everything BEFORE the matched "/wp-admin/" survives untouched in the
+    # output. `path` is not guaranteed to be a clean site-relative string;
+    # three concrete requests turn that survival into an off-site
+    # `Location:` header:
+    #   //evil.example.com/wp-admin/x.php        -> //evil.example.com/wp-login.php
+    #     (protocol-relative -- browsers resolve "//host/path" to
+    #     "https://host/path", so this redirects off-site with no scheme
+    #     needed). NOW NEUTRALISED UPSTREAM: path-merge-slashes rewrites this
+    #     to /evil.example.com/wp-admin/x.php before any ACL sees it, so the
+    #     Location becomes the same-origin /evil.example.com/wp-login.php.
+    #     Verified live.
+    #   /\evil.example.com/wp-admin/x.php        -> /\evil.example.com/wp-login.php
+    #     (browsers normalise a leading "/\" the same as "//"). STILL LIVE
+    #     after normalisation -- a backslash is not a slash, so no normalizer
+    #     touches it. This ACL is the only thing that stops it.
+    #   https://evil.example.com/wp-admin/x.php  -> https://evil.example.com/wp-login.php
+    #     (RFC 7230 absolute-form request targets can make HAProxy's `path`
+    #     fetch return a full URI, not just the path component). HAProxy's own
+    #     H1 parser answers 400 on this frontend; this ACL is the backstop.
+    # So wp_admin_safe_path is NOT redundant with the normalisation and must
+    # not be deleted as such -- one of its three vectors survives normalisation
+    # untouched.
+    #
+    # It is used TWO ways, and the pair matters:
+    #   - as a POSITIVE condition on the redirect, so a pathological path can
+    #     never produce a `Location:` header at all; and
+    #   - as an explicit deny, so such a path is not merely un-redirected.
+    # The deny is what closes the failure mode the positive-condition form
+    # introduced on its own: "not redirected" used to mean "falls through to
+    # the backend UNGATED", i.e. the exact PHP-booting request this gate
+    # exists to stop, reachable by prefixing "//" (that specific spelling is
+    # now normalised away, but "/\" is not). Post-normalisation the only
+    # paths that reach the deny are "/\..." ones, which cannot resolve to a
+    # real file on any tier, so nothing legitimate is denied.
+    # -i (case-insensitive) and the matching ",i" flag on the regsub below are
+    # a PAIR -- adding one without the other produces an infinite redirect
+    # loop, because a case-sensitive regsub finds no "/wp-admin/" in
+    # "/WP-ADMIN/plugins.php", returns `path` UNCHANGED, and the Location then
+    # points at the request's own URL. Verified live that the pair is correct:
+    # /WP-ADMIN/plugins.php -> /wp-login.php and /blog/WP-Admin/plugins.php ->
+    # /blog/wp-login.php.
+    #
+    # On this fleet's Linux backends /WP-ADMIN/plugins.php 404s without booting
+    # PHP, so this is hardening rather than a live-bypass fix; it matters if a
+    # docroot ever sits on a case-insensitive mount, where that same request
+    # WOULD boot PHP. The cost is that a site with a real directory literally
+    # named e.g. /docs/WP-Admin/ now gets gated -- the same false positive the
+    # lowercase pattern already has, which is what the per-site exempt list
+    # exists to resolve.
+    acl wp_admin_path      path_reg -i (^|/)wp-admin/
+    # Four literal backslashes here is NOT a typo. HAProxy's config-line word
+    # parser treats backslash as its OWN escape character before the value
+    # ever reaches the regex engine: "\\" (two backslashes) in the config
+    # collapses to one literal backslash by the time PCRE compiles it, which
+    # leaves an unterminated character class ("[^/\]") and fails with
+    # "missing terminating ] for character class" -- verified against real
+    # HAProxy 3.0.11. Four backslashes ("\\\\") collapse to two ("\\"),
+    # which PCRE then reads as a single escaped-backslash class member --
+    # the intended "reject a literal backslash" semantics.
+    acl wp_admin_safe_path path_reg ^/[^/\\\\]
+    acl wp_admin_allowed   path_end /wp-admin/admin-ajax.php /wp-admin/admin-post.php /wp-admin/load-styles.php /wp-admin/load-scripts.php
+    # The (?!.*\.php) lookahead is DEFENCE IN DEPTH, not the primary fix. This
+    # ACL grants an un-gated bypass to everything under wp-admin/css|js|images,
+    # and it used to anchor its prefix but not its suffix, so
+    # /wp-admin/css/../plugins.php took the bypass and the backend then
+    # resolved ".." and booted plugins.php. path-strip-dotdot now rewrites that
+    # to /wp-admin/plugins.php before this ACL runs, which is the real fix; the
+    # lookahead additionally makes the bypass structurally incapable of
+    # covering a PHP entrypoint even if a future encoding trick survives
+    # normalisation. It excludes ".php" ONLY -- no static asset contains that
+    # substring, so it cannot cause the silent "login page renders unstyled"
+    # regression that an extension allowlist would risk. Requires PCRE2, which
+    # both the Debian (deployed) and Alpine haproxy builds have (+PCRE2).
+    #
+    # THE "--" IS LOAD-BEARING, DO NOT DELETE IT. HAProxy warns on any pattern
+    # whose first character is "(", because it cannot tell an intended regex
+    # from a fetch-argument list someone typo'd a space into:
+    #   parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
+    #   '(^|/)wp-admin/...' is likely a mistake and probably not what you want.
+    #   Maybe you need to remove the extraneous space before '('.
+    # "--" is HAProxy's documented end-of-flags marker and is the fix it names
+    # itself ("please insert '--' between the match and the pattern"). It changes
+    # no matching semantics -- verified live on whp02: /wp-admin/css/login.min.css
+    # still passes and /wp-admin/css/x.php is still gated, before and after.
+    # Left unsilenced this fires on EVERY config load and reload on every host,
+    # where it trains operators to skim past warnings and can bury a real one.
+    # wp_admin_path escapes the warning only because its "-i" flag happens to
+    # consume the flag slot first; it is not otherwise special.
+    acl wp_admin_asset     path_reg -- (^|/)wp-admin/(css|js|images)/(?!.*\.php).*$
+    acl wp_gate_exempt     hdr(host),lower -f /etc/haproxy/wpadmin_gate_exempt.list
+    # ENCODED SEPARATOR. percent-decode-unreserved deliberately does NOT decode
+    # %2F -- "/" is a reserved character, and decoding it in the normalizer
+    # would change the path's structure (it would invent new segments), which
+    # is precisely why HAProxy refuses to. But OpenLiteSpeed DOES decode it and
+    # then serves the file: /wp-admin%2Fplugins.php was measured returning 302
+    # from a real WordPress site on the OLS tier, i.e. full PHP boot, while
+    # matching none of the ACLs above. Apache returns 404 for the same request
+    # (AllowEncodedSlashes Off), so this is an OLS-tier defect -- and OLS is the
+    # tier currently saturating.
+    #
+    # DENY, not "treat it as a wp-admin path and redirect". Two reasons:
+    #   1. The redirect target is computed by regsub(/wp-admin/.*) which finds
+    #      no "/wp-admin/" in "/wp-admin%2Fplugins.php", so `path` would come
+    #      back UNCHANGED and the Location would point at the request's own
+    #      URL -- an infinite redirect loop, not a gate.
+    #   2. Nothing legitimate emits it. A path segment cannot contain a literal
+    #      "/", so %2F inside a path is always either a probe or a proxy-
+    #      confusion attempt, and the Apache tier has been 404ing it all along,
+    #      so no site on the fleet can depend on it.
+    #
+    # SCOPED to paths that mention wp-admin, not all paths. A blanket "deny any
+    # %2F in any path" would also hit REST/API-style routes on non-WordPress
+    # customer apps that legitimately pass an encoded slash inside a path
+    # parameter. Scoping keeps the blast radius inside the attack surface this
+    # gate owns.
+    #
+    # Matching is on the SUBSTRING, not an anchored pattern, on purpose:
+    # /blog%2Fwp-admin/plugins.php hides the separator BEFORE "wp-admin", where
+    # an anchored (^|/)wp-admin/ never matches, and OLS still resolves it to
+    # /blog/wp-admin/plugins.php. Substring matching catches the separator
+    # wherever it is. percent-to-uppercase has already folded %2f into %2F;
+    # the -i is belt and braces so this rule stands on its own if the
+    # normalizer is ever reordered.
+    #
+    # %5C (encoded backslash) is denied on the same terms. On this fleet's
+    # Linux backends a backslash is an ordinary filename character, so
+    # /wp-admin%5Cplugins.php 404s rather than booting PHP -- measured, it is
+    # not a live bypass today. It is included because it is the same
+    # encoded-separator trick against a backend that happens to treat "\" as
+    # one, it costs nothing, and no legitimate path contains it.
+    acl wp_admin_word         path -i -m sub wp-admin
+    acl path_has_encoded_sep  path -i -m sub %2f %5c
+    http-request deny deny_status 403 if wp_admin_word path_has_encoded_sep !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+    http-request deny deny_status 403 if wp_admin_path !wp_admin_safe_path !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+    http-request set-var(txn.wp_login_url) path,regsub(/wp-admin/.*,/wp-login.php,i) if wp_admin_path
+    http-request redirect code 302 location %[var(txn.wp_login_url)]?redirect_to=%[path,url_enc] if wp_admin_path wp_admin_safe_path !wp_admin_allowed !wp_admin_asset !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+
     # IP blocking using map file (manual blocks only)
     # Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
-    # Runtime updates: echo "add map #0 IP_ADDRESS 1" | socat stdio /var/run/haproxy.sock
+    # Runtime updates (worker command, map referenced by PATH -- "#<id>" ids
+    # move on every config regeneration and "#0" silently adds nothing):
+    #   echo "@1 add map /etc/haproxy/blocked_ips.map IP_ADDRESS 1" | socat stdio /tmp/haproxy-cli
     # Checks the real client IP (from headers if present, otherwise src)
     # map_ip() converter supports both single IPs and CIDR ranges (e.g., 192.168.1.0/24)
     acl is_blocked_ip var(txn.real_ip),map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0
@@ -154,7 +622,74 @@ frontend web
     # guards (no point asking the WAF about requests we're already dropping)
     # and AFTER the real-client-IP resolution (so Coraza sees the right src).
     filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
+
+    # --- Make Coraza actually see the request body -------------------------
+    # send-spoe-group sends `req.body` = whatever body bytes are in the buffer
+    # AT THAT MOMENT. Without buffering that is usually NOTHING: measured on
+    # haproxy 3.0.11 + coraza-spoa v0.7.1, a 14-byte `content=$a($b)` POST
+    # whose body was written in a second send() right after the headers
+    # passed 933180 (an enforced rule) uninspected, while the identical bytes
+    # in one send() were blocked. A body sent 2s late, or after an
+    # `Expect: 100-continue`, was never inspected at all.
+    #
+    # `option http-buffer-request` makes HAProxy wait, before evaluating the
+    # http-request rules, until the WHOLE body is in or the buffer is full,
+    # bounded by `timeout http-request` (30s, hap_header.tpl). It answers
+    # `Expect: 100-continue` itself so the client actually sends the body.
+    # It does NOT hold uploads: once one buffer (tune.bufsize, 16 KB) is full
+    # the request is forwarded and the rest streams through as before -- a
+    # 50 MB upload is not stored at the edge.
+    option http-buffer-request
+
+    # Exactly one Content-Type, or 400. With two Content-Type lines Coraza and
+    # the backend disagree about what the body IS: Coraza evaluates every value
+    # (`text/xml` on either line selects its XML processor, which accepts a
+    # form body without error), while Apache merges the lines with ", " and
+    # PHP cuts the media type at the first ',' / ';'. Measured (Coraza v3.7.0
+    # + PHP 8.3): `application/x-www-form-urlencoded` + `text/xml` puts the
+    # payload in $_POST with nothing inspected. req.fhdr_cnt counts header
+    # LINES, so a single comma-joined value (which browsers produce when a
+    # script calls setRequestHeader twice) is not affected here; that form is
+    # parsed consistently enough by both sides (see the report's matrix).
+    # Browsers, curl, PHP-curl and the common HTTP libraries send one line.
+    http-request deny deny_status 400 if { req.fhdr_cnt(content-type) gt 1 }
+
+    # Tell Coraza whether it got the whole body. Coraza cannot tell on its
+    # own (REQUEST_BODY_LENGTH is only filled by the urlencoded/raw
+    # processors), and the difference matters: a TRUNCATED JSON body, or a
+    # multipart body cut inside a boundary/part header, fails to parse by
+    # construction, so the body-parse-error rules (200002/200003) may only
+    # deny a COMPLETE body. WHP's generated Coraza rules
+    # (regenerate-coraza-overrides.sh) key on this header:
+    #   complete -> 200002/200003 deny on multipart/JSON (enforce mode)
+    #   partial  -> 200002/200003 removed for this request (every mode)
+    #
+    # With a Content-Length: complete iff req.body_len (buffered) equals
+    # req.body_size (= Content-Length).
+    # Without one (HTTP/1.1 chunked, HTTP/2-3 without content-length) HAProxy
+    # 3.0 has no "end of message" fetch and req.body_size is just what is
+    # buffered. But http-buffer-request only releases the request early when
+    # the body ENDED or the buffer FILLED (a stalled body gets a 408 instead),
+    # and body + headers well below one buffer (12 KB of the ~15 KB usable)
+    # cannot have filled it. So that is complete, anything else partial.
+    # Without this branch an attacker only had to add
+    # `Transfer-Encoding: chunked` to escape enforcement (measured).
+    #
+    # Client-supplied copies are deleted FIRST so it cannot be forged, and it
+    # is deleted again after the SPOE call so backends never see it.
+    # txn.waf_body is also logged (`wafb=` in the log-format above) so the
+    # complete/partial mix is measurable per host.
+    http-request del-header X-WHP-WAF-Body
+    http-request set-var(txn.waf_body_size) req.body_size if { req.hdr_cnt(content-length) eq 1 }
+    http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_body_size) -m found } { req.body_len,sub(txn.waf_body_size) eq 0 }
+    http-request set-var(txn.waf_body) str(partial) if { var(txn.waf_body_size) -m found } !{ req.body_len,sub(txn.waf_body_size) eq 0 }
+    http-request set-var(txn.waf_hdrs_len) req.hdrs,length if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 }
+    http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_hdrs_len) -m found } { req.body_len,add(txn.waf_hdrs_len) lt 12288 }
+    http-request set-var(txn.waf_body) str(partial) if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 } !{ var(txn.waf_body) -m found }
+    http-request set-header X-WHP-WAF-Body %[var(txn.waf_body)] if { var(txn.waf_body) -m found }
+
     http-request send-spoe-group coraza coraza-req
+    http-request del-header X-WHP-WAF-Body
 
     # Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to
     # "deny" / "drop" / "redirect" when a rule with the corresponding

templates/hap_security_tables.tpl

@@ -13,4 +13,19 @@ frontend stats
 # sc0 connection/rate table so the login-attempt threshold is independent of
 # the (much higher) flood thresholds.
 backend wp_bruteforce
+    stick-table type ip size 100k expire 30m store http_req_rate(60s)
+
+# Dedicated stick-table for POST /xmlrpc.php flood tracking.
+# Tracked via track-sc2 from the `web` frontend (hap_listener.tpl); counts
+# only xmlrpc POSTs per real client IP over a 60s window. This is a SEPARATE
+# table/counter from wp_bruteforce (sc1) rather than a shared one: both are
+# machine-to-machine WordPress endpoints an attacker could hit from the same
+# IP, and sharing a counter would let one endpoint's traffic inflate the
+# other's rate -- an IP credential-stuffing wp-login while also flooding
+# xmlrpc would trip the wp-login threshold early on xmlrpc volume alone (or
+# vice versa). track-sc1 (wp-login) and track-sc2 (xmlrpc) are each gated on
+# mutually exclusive path ACLs, so at most one of them ever fires per
+# request -- HAProxy's "one track-sc<N> per counter per request" limit is
+# never in play here since they're different counters anyway.
+backend xmlrpc_bruteforce
     stick-table type ip size 100k expire 30m store http_req_rate(60s)

trusted_proxies.list

@@ -0,0 +1,13 @@
+# Additional trusted reverse proxies — peers permitted to set CF-Connecting-IP,
+# X-Real-IP and X-Forwarded-For. Anything NOT matched here or in
+# cloudflare_ips.list has those headers stripped before real-IP resolution.
+#
+# Referenced by templates/hap_listener.tpl.
+#
+# Leave EMPTY unless a real proxy sits in front of HAProxy on this host. Adding
+# a range here lets that peer assert any client identity, which bypasses rate
+# limits, IP blocks and the WAF for it.
+#
+# Do NOT commit real IPs — this repo is mirrored publicly. Add entries directly
+# on the server; the file lives in the /etc/haproxy named volume and persists
+# across container recreates.

wpadmin_gate_exempt.list

@@ -0,0 +1,16 @@
+# Per-site opt-out from the WordPress admin edge gate.
+#
+# Hostnames listed here are EXEMPT: unauthenticated /wp-admin/* requests for
+# these sites pass through to PHP instead of being redirected to wp-login.php.
+# One hostname per line, lowercase. Matched against the Host header.
+#
+# Referenced by templates/hap_listener.tpl:
+#   acl wp_gate_exempt hdr(host),lower -f /etc/haproxy/wpadmin_gate_exempt.list
+#
+# Add a site here when a plugin legitimately serves unauthenticated visitors
+# from a /wp-admin/ URL that is not in the rule's allowlist. Symptom: "my
+# plugin's admin page redirects to login".
+#
+# Do NOT commit real customer domains — this repo is mirrored publicly. Add
+# entries directly on the server; the file lives in the /etc/haproxy named
+# volume and persists across container recreates.