feat(waf): log why Coraza acted (wrule/wrules/wscore); VERSION 2026.10.4

coraza-spoa v0.7.1 already returns the interrupting rule id
(txn.coraza.ruleid) and, when the request message sends
exportRuleIDs=bool(true), the comma list of every matched CRS attack
rule (txn.coraza.rule_ids) plus the inbound anomaly score. Send the arg
and append all three to the access log so a block's cause (e.g.
920440,949110 score 5) is visible in haproxy.log without opening the
Coraza audit log. Fields are appended last; existing ones unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAUDE.md

@@ -8,6 +8,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
 - **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
 - **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
+- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
+- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
+- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
+- **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine.
+- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
 - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
 
 ### Reading stick tables (and why it is easy to get silently wrong)
@@ -31,6 +36,29 @@ reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
 stick table has ever stored — for their entire existence. See the header of
 `haproxy_tarpit_config.txt` and the contract test.
 
+### Changing a runtime map (`add map` / `del map`)
+
+Same socket, two more ways to fail silently — and both were live in
+`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
+existence:
+
+* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
+  config-parse time and move on every config regeneration (on whp01
+  `blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
+  `add map /etc/haproxy/blocked_ips.map <ip> 1`.
+* **A mutation answers NOTHING on success**, so an empty body is the only
+  success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
+  *also* answers nothing and adds nothing, so the body cannot prove an add
+  worked. **Read it back** with `@1 get map <path> <key>`.
+* Entries must carry the value `1`; haproxy.cfg matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
+
+In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
+and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
+only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
+re-reads it on reload, so a failed runtime command must degrade to
+"enforced on reload" and be reported, never swallowed.
+
 ### Running the Application
 - **Docker Build**: `docker build -t haproxy-manager .`
 - **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)
@@ -141,6 +169,8 @@ stick table has ever stored — for their entire existence. See the header of
 - Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
 - Stats page on port 8404 (localhost only inside container)
 - Management interface on port 8000 should be firewall-protected in production
-- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s`
+- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
+- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
+- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
 - Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
 - HAProxy is version 3.0.11

Dockerfile

@@ -44,6 +44,9 @@ WORKDIR /haproxy
 COPY ./templates /haproxy/templates
 COPY requirements.txt /haproxy/
 COPY haproxy_manager.py /haproxy/
+COPY haproxy_gunicorn_worker.py /haproxy/
+# Read by GET /api/capabilities (WHP gates whole-site re-bundling on it).
+COPY VERSION /haproxy/VERSION
 COPY scripts /haproxy/scripts
 COPY trusted_ips.list /etc/haproxy/trusted_ips.list
 COPY trusted_ips.map /etc/haproxy/trusted_ips.map
@@ -92,6 +95,8 @@ RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
  && python3 /haproxy/scripts/test-config-rollback.py \
  && python3 /haproxy/scripts/test-cert-write-safety.py \
  && python3 /haproxy/scripts/test-cert-scripts.py \
+ && python3 /haproxy/scripts/test-waf-body-inspection.py \
+ && python3 /haproxy/scripts/test-gunicorn-socket-timeouts.py \
  && python3 /haproxy/scripts/validate-rendered-config.py
 # Create log directories
 RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
@@ -105,7 +110,10 @@ RUN mkdir -p /var/spool/cron/crontabs && \
 # 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
 # must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
 EXPOSE 80 443 443/udp 8000
-# Add health check
+# Health check: management API (:8000, short timeout) AND HAProxy (:80).
+# See scripts/healthcheck.sh. NOTE: WHP's lib-edge-container.sh passes its own
+# --health-cmd, which REPLACES this one on deployed hosts; it must be pointed
+# at /haproxy/scripts/healthcheck.sh for the fleet to get the :8000 probe.
 HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
-    CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1
+    CMD /haproxy/scripts/healthcheck.sh
 CMD ["/haproxy/scripts/start-up.sh"]

IP_BLOCKING_API.md

@@ -508,20 +508,45 @@ curl -X POST http://localhost:8000/api/blocked-ips/sync \
 
 For advanced users, you can interact directly with HAProxy's runtime API:
 
+Three things about these commands are easy to get wrong, and each one fails
+**silently** (socat exits 0 either way — the rejection, if any, is only in the
+response body):
+
+* `/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Map commands are
+  worker commands and need the `@1` prefix. Without it the reply is
+  `Unknown command: 'add', ...`.
+* Reference the map by its **file path**, never by `#<id>`. Ids are assigned at
+  config-parse time and move on every config regeneration (on a live edge,
+  `blocked_ips.map` is id 37, `trusted_ips.map` is 10 — there is no id 0).
+  Worse, `@1 add map #0 <ip> 1` returns an **empty** reply and adds nothing.
+* Entries must carry the value `1`. `haproxy.cfg` matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block. (`add map`
+  with no value is rejected: `'add map' expects three parameters ...`.)
+
 ```bash
+MAP=/etc/haproxy/blocked_ips.map
+
 # Add IP to runtime (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map $MAP 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP from runtime
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
+
+# Confirm what actually happened (do not trust the exit status)
+echo "@1 get map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
 
 # Clear all blocked IPs from runtime
-echo "clear map #0" | socat stdio /var/run/haproxy.sock
+echo "@1 clear map $MAP" | socat stdio /tmp/haproxy-cli
 
-# Show all runtime map entries
-echo "show map #0" | socat stdio /var/run/haproxy.sock
+# Show all runtime map entries, and the map ids currently in use
+echo "@1 show map $MAP" | socat stdio /tmp/haproxy-cli
+echo "@1 show map" | socat stdio /tmp/haproxy-cli
 ```
 
+The runtime map is a **fast path only**. `/etc/haproxy/blocked_ips.map` is
+authoritative: HAProxy re-reads it on reload, so a failed runtime command
+delays a block until the next reload rather than losing it.
+
 ## Migration from ACL Method
 
 If you're upgrading from the old ACL-based method:

MIGRATION_GUIDE.md

@@ -50,14 +50,22 @@ http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }
 - **Graceful error handling**
 
 ### 2. Runtime IP Management
+Map commands go to a **worker** (`@1`), reference the map by **file path**
+(ids move between config regenerations, and `#0` silently adds nothing), and
+carry the value `1` that `map_ip(...,0) -m int gt 0` matches on:
+
 ```bash
 # Add IP without reload (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map /etc/haproxy/blocked_ips.map 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP without reload
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map /etc/haproxy/blocked_ips.map 192.168.1.100" | socat stdio /tmp/haproxy-cli
 ```
 
+socat exits 0 even when HAProxy rejects the command, so read the response body
+(or read the entry back with `@1 get map ...`) rather than the exit status.
+See IP_BLOCKING_API.md for the full set.
+
 ### 3. New API Endpoints
 
 #### Safe Config Reload

VERSION

@@ -1 +1 @@
-2026.08.9
+2026.10.4

coraza-spoa/config.yaml

@@ -26,6 +26,37 @@ applications:
       # CRS-bundled defaults: recommended Coraza settings + CRS setup +
       # the rule pack itself (~16 MB of rules embedded in the binary).
       Include @coraza.conf-recommended
+
+      # Request-body limits, set explicitly instead of inheriting the
+      # recommended file's 12.5 MB / Reject. HAProxy only ever hands us one
+      # buffer of body (tune.bufsize 16384 in haproxy-manager's
+      # hap_header.tpl), and the SPOP library refuses frames over 65535
+      # bytes, so no body we receive can exceed 64 KB. The invariant is:
+      #     tune.bufsize (16 KB) < SPOP frame cap (65535) < SecRequestBodyLimit
+      # Breaking it is NOT harmless: SecRequestBodyLimitAction Reject
+      # interrupts with 413 even under DetectionOnly (Coraza
+      # setAndReturnBodyLimitInterruption ignores the engine mode), which the
+      # edge turns into a 403 on every large upload. ProcessPartial is the
+      # belt to that brace. InMemoryLimit == BodyLimit keeps the body buffer
+      # off disk (this image is distroless). Pinned by
+      # haproxy-manager-base/scripts/test-waf-body-inspection.py.
+      SecRequestBodyLimit 131072
+      SecRequestBodyInMemoryLimit 131072
+      SecRequestBodyLimitAction ProcessPartial
+
+      # JSON nesting limit (the recommended file sets 1024). Coraza's JSON
+      # flattener is quadratic in nesting: a 601-byte `[[[...]]]` of depth 300
+      # cost 297 ms and depth 1000 (2 KB) 3.1 s of CPU -- past the 100 ms SPOE
+      # timeout, so the request passed UNINSPECTED (fail-open) while the WAF
+      # burned CPU. Over the limit the body fails to parse (REQBODY_ERROR,
+      # 0.1 ms), which WHP's enforce-mode body-integrity rule denies on a
+      # complete body. Legitimate depth, measured on the JSON shipped in
+      # WordPress 6.x core, Gutenberg, WooCommerce and Elementor (theme.json,
+      # styles/*.json, block.json, checkout schemas): max 8, so a global-styles
+      # save is ~9; 64 leaves ~7x headroom for Divi/Elementor-style module
+      # trees that could not be sampled. Depth 64 itself costs ~21 ms.
+      SecRequestBodyJsonDepthLimit 64
+
       Include @crs-setup.conf.example
 
       # Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.

coraza-spoa/overrides.conf

@@ -47,7 +47,17 @@ SecRuleUpdateActionById 933170-933200 "ctl:ruleEngine=On"
 # vulnerable, but blocking at the edge keeps logs clean and protects
 # any future Java workloads.
 # ---------------------------------------------------------------------------
-SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
+# 944110 and 944120 are NOT promoted (changed 2026-10-09, 2026.10.2). Both are
+# CHAINS whose first link matches bare words anywhere in the request --
+# `runtime|processbuilder` and `getproperty|forclosure|filewriter|...` -- and
+# Coraza runs an appended ctl:ruleEngine=On on that first link even when the
+# chain then fails. Measured (Coraza v3.7.0 + CRS 4.25.0): in detect_only,
+# `?q=runtime&id=1' or '1'='1` was 403 while the SQLi alone was 200. Same
+# chain-starter defect as 920440. They still add to the anomaly score.
+# Log4Shell (944150/944151/944152, ...) stays promoted. WHP mirrors this split
+# in coraza_baked_promotion_ranges() (web-files/libs/coraza_blocking_set.php).
+SecRuleUpdateActionById 944100 "ctl:ruleEngine=On"
+SecRuleUpdateActionById 944130-944300 "ctl:ruleEngine=On"
 
 # ---------------------------------------------------------------------------
 # 920440 — URL file extension restricted by policy
@@ -59,7 +69,26 @@ SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
 # single GCP-hosted scanner. Zero false positives observed; standard
 # WP/WooCommerce/Divi/HPR URLs do not end in these extensions.
 # ---------------------------------------------------------------------------
-SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
+# HOW IT IS PROMOTED (changed 2026-10-09, haproxy-manager-base 2026.10.2).
+# This used to be `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"`, which
+# was WRONG: 920440 is a CHAIN whose first link is
+# `REQUEST_BASENAME "@rx \.([^.]+)$"` -- it matches EVERY URL with a file
+# extension -- and Coraza runs actions appended to a chain starter when that
+# first link matches, even though the chain then fails. Measured (Coraza
+# v3.7.0 + CRS 4.25.0, in-process and live): every *.php / *.css / ...
+# request ran with the engine On, so the whole CRS anomaly score (949110) and
+# 200002 enforced there -- in detect_only mode too, since SecRuleEngine
+# DetectionOnly cannot override a runtime ctl.
+#
+# Now: 920440 itself is untouched (it still sets tx.extension in its first
+# link and adds the anomaly score in its second), and this separate phase-1
+# rule, which runs after it, flips the engine only when the extension really
+# is in tx.restricted_extensions -- the original intent. Removing 920440
+# (break-glass, a per-host exception) also removes tx.extension, so this rule
+# goes quiet with it. Id 992920440 is outside WHP's generated range
+# (990000000-991999999).
+SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \
+    "id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"
 
 # ---------------------------------------------------------------------------
 # 930130 — Restricted File Access Attempt

haproxy_gunicorn_worker.py

@@ -0,0 +1,502 @@
+"""gunicorn gthread worker that cannot be wedged by silent or slow clients.
+
+Incident 2026-10-09 (sdbees), and very likely the 2026-07-07 whp01 one too
+-------------------------------------------------------------------------
+Both :8000 API workers had 7 of 8 pool threads parked in the kernel in
+``tcp_recvmsg -> sk_wait_data``: a blocking socket read with no timeout.
+/health and /api/blocked-ips hung for 40s+, the container stayed "healthy".
+
+haproxy_manager.py makes NO in-process outbound network reads (no requests,
+urllib, http.client, raw socket or ssl client anywhere; every ACME/HAProxy
+interaction is a subprocess, already bounded since 2026.07.3). The reads that
+blocked were gunicorn's own reads of the CLIENT socket:
+
+    gunicorn/workers/gthread.py  TConn.init():  self.sock.setblocking(True)
+
+Once the main loop sees a connection become readable it hands it to a pool
+thread and turns the socket fully blocking -- timeout None -- before parsing
+the request. Any peer that sends a few bytes and then waits (a TLS ClientHello
+sent to this plain-HTTP port, a port scanner, a half-written request) pins
+that pool thread forever. :8000 was published on 0.0.0.0 by the deploy
+(`-p 8000:8000`, which bypasses firewalld) and internet scanners were measured
+connecting to it. 4 such connections exhaust a worker's pool.
+
+gunicorn's --timeout does not catch this: for gthread it only checks that the
+worker's MAIN thread keeps heart-beating, and the main thread keeps polling
+happily while every pool thread is stuck. --max-requests does not help either:
+a worker whose threads never finish a request never reaches the limit.
+socket.setdefaulttimeout() does not help: setblocking(True) explicitly resets
+the timeout to None, overriding any default.
+
+Stock gthread has two more exhaustion paths on the same port (review F1/F2):
+
+* A connection that NEVER sends a byte never becomes readable, so it is never
+  handed to a thread and never gets any deadline -- but it holds one of the
+  worker's ``worker_connections`` (1000) slots. At 1000 the run loop stops
+  polling altogether and only waits on futures; none exist, so even the
+  attacker's FINs are never seen and the worker is wedged PERMANENTLY, still
+  heart-beating, with no request in flight for a watchdog to notice.
+* Slowloris: one header byte every <timeout seconds never trips a per-recv
+  timeout, so a socket timeout alone does not bound header-read time.
+
+What this worker changes
+------------------------
+1. CLIENT_TIMEOUT (HAPROXY_MGR_CLIENT_TIMEOUT, default 20s): every client
+   socket gets an explicit socket timeout at the exact point gthread makes it
+   blocking (re-applied on every keep-alive reuse). Python applies it to each
+   recv() call and to each WHOLE sendall() call -- gunicorn writes a response
+   chunk with one sendall, so any single chunk that takes longer than this to
+   transmit is cut off. Responses here are KB-sized JSON/PEMs to localhost, so
+   that never happens in practice. It does NOT bound how long the app takes to
+   handle a request (a long certbot run is a subprocess with its own timeout
+   and touches the client socket only once it is done).
+
+2. FIRST_BYTE_TIMEOUT (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, default 10s): an
+   accepted connection that has not become readable within this is closed by
+   the main loop, freeing its slot (review F1).
+
+3. HEADER_DEADLINE (HAPROXY_MGR_HEADER_DEADLINE, default 10s): total time a
+   client may take to deliver the request line + headers, counted from when
+   the connection was ENQUEUED for a thread, with a floor of
+   HAPROXY_MGR_HEADER_PICKUP_GRACE (1s) after a thread picks it up:
+   deadline = max(enqueue + 10s, pickup + 1s). Disarmed in handle_request()
+   (which gunicorn calls only once the parser has returned a complete
+   request). A 1s monitor loop shuts down (SHUT_RDWR, not close -- no fd-reuse
+   race with the blocked recv) the socket of any connection past it; the
+   blocked recv returns EOF and the thread is freed (review F2). A legit
+   request queued behind busy threads (certbot runs) already has its headers
+   in the kernel buffer and parses in milliseconds inside the 1s grace; a
+   slow client that sat in the queue gets only the grace, so reconnecting
+   drippers cannot starve the pool by queueing (re-review). Bodies are only
+   read by routes behind require_api_key, so the unauthenticated surface is
+   exactly this phase.
+
+4. Wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS, default 3600s; 0 disables). If
+   EVERY pool thread has been inside the same request for longer than that the
+   worker logs every thread's Python stack to the error log and exits; the
+   gunicorn master respawns it in about a second, without touching HAProxy or
+   the edge. 3600s because a multi-domain /api/certificates/request runs one
+   certbot (up to 180s) per domain; with (1)-(3) closing the known client-side
+   sources, this is only a backstop for blocking sources not yet found.
+
+5. Peer filter (re-review R1). haproxy-manager shares the client-net bridge
+   with every tenant container, so tenant PHP could reach haproxy-manager:8000
+   directly; the 127.0.0.1 publish only removes the internet path. On the
+   ports in HAPROXY_MGR_PEER_FILTER_PORTS (default "8000" -- NOT the :8080
+   default-page server, which HAProxy feeds over loopback) accept() closes
+   any connection whose peer is not in HAPROXY_MGR_API_ALLOWED_PEERS (default
+   "127.0.0.1,::1,gateway"; IPs/CIDRs; "gateway" = this container's default
+   gateway, which is where host callers arrive from through docker-proxy).
+   Refused before a thread or connection slot is used; logged at most once a
+   minute as a per-peer summary. Effective only once the edge publishes
+   127.0.0.1:8000 -- with a 0.0.0.0 publish a tenant can hairpin through the
+   host and arrive from the gateway too.
+
+6. pool_status() exposes busy/total threads and the oldest in-flight age so
+   /health can report saturation.
+
+This overrides gthread internals (enqueue_req, accept,
+on_client_socket_readable, murder_keepalived, handle, handle_request) as they
+are in gunicorn 23.0.0, which requirements.txt pins. A warning is logged if a
+different version is loaded; scripts/test-gunicorn-socket-timeouts.py runs
+real gunicorn in the Docker build gate and fails on behavioural drift.
+"""
+
+import errno
+import ipaddress
+import logging
+import os
+import selectors
+import socket
+import sys
+import threading
+import time
+import traceback
+from collections import deque
+from functools import partial
+
+import gunicorn
+from gunicorn.workers.gthread import TConn, ThreadWorker
+
+
+def _env_float(name, default):
+    try:
+        return float(os.environ.get(name, default))
+    except (TypeError, ValueError):
+        return float(default)
+
+
+CLIENT_TIMEOUT = _env_float('HAPROXY_MGR_CLIENT_TIMEOUT', 20)
+FIRST_BYTE_TIMEOUT = _env_float('HAPROXY_MGR_FIRST_BYTE_TIMEOUT', 10)
+HEADER_DEADLINE = _env_float('HAPROXY_MGR_HEADER_DEADLINE', 10)
+WEDGE_SECONDS = _env_float('HAPROXY_MGR_WEDGE_SECONDS', 3600)
+WATCHDOG_INTERVAL = _env_float('HAPROXY_MGR_WATCHDOG_INTERVAL', 15)
+MONITOR_TICK = 1.0
+# Once a thread picks a queued connection up it gets at least this long to
+# finish its headers, even if its enqueue-based deadline has already passed.
+HEADER_PICKUP_GRACE = _env_float('HAPROXY_MGR_HEADER_PICKUP_GRACE', 1)
+
+# Peer filter (review R1). Listener ports it applies to (comma-separated;
+# empty disables) and the peers allowed through. "gateway" means this
+# container's default gateway on its bridge network -- the address host-side
+# callers arrive from via the 127.0.0.1:8000 publish (docker-proxy) and via a
+# direct host->container-IP connect. Tenant containers on the same bridge
+# (client-net) arrive from their OWN addresses and are refused.
+PEER_FILTER_PORTS = os.environ.get('HAPROXY_MGR_PEER_FILTER_PORTS', '8000')
+ALLOWED_PEERS_SPEC = os.environ.get('HAPROXY_MGR_API_ALLOWED_PEERS', '127.0.0.1,::1,gateway')
+PEER_REJECT_LOG_INTERVAL = 60.0
+
+
+def _default_gateways(route_file='/proc/net/route'):
+    """IPv4 default gateway(s) from the kernel routing table."""
+    gws = []
+    try:
+        with open(route_file) as fh:
+            next(fh, None)
+            for line in fh:
+                f = line.split()
+                if len(f) >= 4 and f[1] == '00000000' and int(f[3], 16) & 0x2:
+                    gws.append(str(ipaddress.IPv4Address(int(f[2], 16).to_bytes(4, 'little'))))
+    except (OSError, ValueError):
+        pass
+    return gws
+
+
+def parse_allowed_peers(spec, gateways=None):
+    """Return (networks, problems). Tokens: IPs, CIDRs, or the word
+    'gateway'. Unparseable tokens are reported, never silently widened."""
+    nets, problems = [], []
+    for tok in (t.strip() for t in (spec or '').split(',')):
+        if not tok:
+            continue
+        if tok.lower() == 'gateway':
+            gws = _default_gateways() if gateways is None else gateways
+            if not gws:
+                problems.append('gateway: no IPv4 default route found')
+            for g in gws:
+                nets.append(ipaddress.ip_network(g + '/32'))
+            continue
+        try:
+            nets.append(ipaddress.ip_network(tok, strict=False))
+        except ValueError:
+            problems.append('unparseable entry %r' % tok)
+    return nets, problems
+
+
+def peer_allowed(addr, nets):
+    try:
+        ip = ipaddress.ip_address(addr.split('%', 1)[0])
+    except (ValueError, AttributeError):
+        return False
+    if ip.version == 6 and ip.ipv4_mapped is not None:
+        ip = ip.ipv4_mapped
+    return any(ip.version == n.version and ip in n for n in nets)
+
+TESTED_GUNICORN_VERSION = '23.0.0'
+
+# Exit status for a watchdog-killed worker. Must NOT be 3 (WORKER_BOOT_ERROR)
+# or 4 (APP_LOAD_ERROR): on those the gunicorn master halts the whole server
+# instead of respawning the worker.
+WEDGED_EXIT_CODE = 1
+
+_current_worker = None
+
+
+def pool_status():
+    """Busy/total threads of the worker serving the caller, or None when the
+    app is not running under this worker class (e.g. tests, dev server)."""
+    w = _current_worker
+    if w is None:
+        return None
+    return w.pool_status()
+
+
+class _QuietClientTimeout(logging.Filter):
+    """gthread logs any OSError without an errno it recognises as
+    log.exception("Socket error processing request.") -- a full traceback.
+    A client timeout is now an expected, bounded event (scanners hit this port
+    constantly), so reduce it to one line instead of flooding the error log."""
+
+    def filter(self, record):
+        exc = record.exc_info[1] if record.exc_info else None
+        if isinstance(exc, TimeoutError) and \
+                str(record.msg).startswith('Socket error processing request'):
+            record.msg = ('Closed client connection idle for more than %ss '
+                          '(HAPROXY_MGR_CLIENT_TIMEOUT)' % CLIENT_TIMEOUT)
+            record.args = ()
+            record.exc_info = None
+            record.exc_text = None
+            record.levelno = logging.INFO
+            record.levelname = 'INFO'
+        return True
+
+
+class _InFlight:
+    __slots__ = ('conn', 'start', 'header_deadline')
+
+    def __init__(self, conn, start, header_deadline):
+        self.conn = conn
+        self.start = start
+        self.header_deadline = header_deadline
+
+
+class TimeoutThreadWorker(ThreadWorker):
+    """gthread worker with client-socket timeouts, a first-byte deadline, a
+    total header deadline and a wedge watchdog. See module docstring."""
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._inflight = {}          # thread ident -> _InFlight
+        self._inflight_lock = threading.Lock()
+        self._pending = deque()      # accepted, not yet readable (FIFO by deadline)
+        self._filter_ports = set()
+        for tok in PEER_FILTER_PORTS.split(','):
+            tok = tok.strip()
+            if tok.isdigit():
+                self._filter_ports.add(int(tok))
+        self._allowed_nets = []
+        self._rejected = {}          # peer ip -> count since last log line
+        self._rejected_lock = threading.Lock()
+        self._last_reject_log = 0.0
+
+    # -- (1) client socket timeout ------------------------------------------
+    def enqueue_req(self, conn):
+        # Same three steps as ThreadWorker.enqueue_req, with the timeout set
+        # right after conn.init() -- init() is what calls setblocking(True)
+        # (== settimeout(None)), and it runs again on every keep-alive reuse,
+        # so the timeout must be re-applied here each time, not at accept().
+        conn.init()
+        if CLIENT_TIMEOUT > 0:
+            conn.sock.settimeout(CLIENT_TIMEOUT)
+        conn.whp_enqueued = time.monotonic()   # header deadline counts from here
+        fs = self.tpool.submit(self.handle, conn)
+        self._wrap_future(fs, conn)
+
+    # -- (2) first-byte deadline for never-readable connections ---------------
+    def accept(self, server, listener):
+        # ThreadWorker.accept (23.0.0) plus: a deadline, and the conn is
+        # tracked in self._pending so murder_keepalived() can reap it.
+        try:
+            sock, client = listener.accept()
+            if self._filter_ports and self._server_port(server) in self._filter_ports \
+                    and not peer_allowed(client[0] if isinstance(client, tuple) else '',
+                                         self._allowed_nets):
+                # Refused before it costs a TConn, a connection slot, a poller
+                # registration or a thread.
+                sock.close()
+                self._note_rejected(client[0] if isinstance(client, tuple) else str(client))
+                return
+            conn = TConn(self.cfg, sock, client, server)
+            conn.timeout = time.time() + FIRST_BYTE_TIMEOUT
+
+            self.nr_conns += 1
+            with self._lock:
+                self._pending.append(conn)
+                self.poller.register(conn.sock, selectors.EVENT_READ,
+                                     partial(self.on_client_socket_readable, conn))
+        except OSError as e:
+            if e.errno not in (errno.EAGAIN, errno.ECONNABORTED,
+                               errno.EWOULDBLOCK):
+                raise
+
+    @staticmethod
+    def _server_port(server):
+        try:
+            return int(server[1])
+        except (TypeError, IndexError, ValueError):
+            return None
+
+    def _note_rejected(self, peer):
+        with self._rejected_lock:
+            self._rejected[peer] = self._rejected.get(peer, 0) + 1
+        if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
+            self._flush_rejected()
+
+    def _flush_rejected(self):
+        with self._rejected_lock:
+            if not self._rejected:
+                return
+            items, self._rejected = self._rejected, {}
+            self._last_reject_log = time.monotonic()
+        total = sum(items.values())
+        top = sorted(items.items(), key=lambda kv: -kv[1])[:5]
+        self.log.warning(
+            'Refused %d connection(s) to the management API from %d non-allowed peer(s) '
+            '(HAPROXY_MGR_API_ALLOWED_PEERS=%s): %s%s', total, len(items), ALLOWED_PEERS_SPEC,
+            ', '.join('%s x%d' % kv for kv in top), ' ...' if len(items) > 5 else '')
+
+    def on_client_socket_readable(self, conn, client):
+        # ThreadWorker version only knows about self._keep; a first request
+        # comes off self._pending instead.
+        with self._lock:
+            self.poller.unregister(client)
+            queue = self._keep if conn.initialized else self._pending
+            try:
+                queue.remove(conn)
+            except ValueError:
+                # already reaped (race with murder_keepalived)
+                return
+        self.enqueue_req(conn)
+
+    def murder_keepalived(self):
+        # Runs on every pass of ThreadWorker.run() -- including the branch
+        # taken when nr_conns >= worker_connections, where the poller is not
+        # serviced. Reaping here is what breaks that permanent deadlock.
+        super().murder_keepalived()
+        if FIRST_BYTE_TIMEOUT <= 0:
+            return
+        now = time.time()
+        while True:
+            with self._lock:
+                if not self._pending or self._pending[0].timeout > now:
+                    break
+                conn = self._pending.popleft()
+                try:
+                    self.poller.unregister(conn.sock)
+                except (OSError, KeyError, ValueError):
+                    pass
+            self.nr_conns -= 1
+            conn.close()
+
+    # -- (3) header deadline + busy tracking ----------------------------------
+    def handle(self, conn):
+        ident = threading.get_ident()
+        now = time.monotonic()
+        deadline = None
+        if HEADER_DEADLINE > 0:
+            # max(enqueue + D, pickup + grace): a legit request that waited in
+            # the queue behind busy threads already has its headers in the
+            # kernel buffer and parses in milliseconds; a slow client that
+            # waited gets only the grace, not a fresh full deadline (review:
+            # 40 reconnecting drippers otherwise starved legit requests).
+            enq = getattr(conn, 'whp_enqueued', now)
+            deadline = max(enq + HEADER_DEADLINE, now + HEADER_PICKUP_GRACE)
+        entry = _InFlight(conn, now, deadline)
+        with self._inflight_lock:
+            self._inflight[ident] = entry
+        try:
+            return super().handle(conn)
+        finally:
+            with self._inflight_lock:
+                self._inflight.pop(ident, None)
+
+    def handle_request(self, req, conn):
+        # The parser has produced a complete request line + headers.
+        entry = self._inflight.get(threading.get_ident())
+        if entry is not None:
+            entry.header_deadline = None
+        return super().handle_request(req, conn)
+
+    def _sweep_header_deadlines(self):
+        now = time.monotonic()
+        with self._inflight_lock:
+            late = [e for e in self._inflight.values()
+                    if e.header_deadline is not None and e.header_deadline < now]
+        for e in late:
+            e.header_deadline = None   # one shutdown per connection
+            try:
+                # shutdown, not close: wakes the blocked recv() with EOF
+                # without releasing the fd while another thread is in it.
+                e.conn.sock.shutdown(socket.SHUT_RDWR)
+            except OSError:
+                pass
+        if late:
+            self.log.info('Cut %d client connection(s) still sending request headers '
+                          'after %ss (HAPROXY_MGR_HEADER_DEADLINE)', len(late), HEADER_DEADLINE)
+
+    def pool_status(self):
+        now = time.monotonic()
+        with self._inflight_lock:
+            starts = [e.start for e in self._inflight.values()]
+        return {
+            'pid': os.getpid(),
+            'threads': self.cfg.threads,
+            'busy': len(starts),
+            'oldest_busy_seconds': round(now - min(starts), 1) if starts else 0.0,
+            'client_timeout_seconds': CLIENT_TIMEOUT,
+        }
+
+    # -- (4) watchdog ---------------------------------------------------------
+    def _wedged_for(self):
+        """Seconds the pool has been completely wedged (every thread busy in
+        its current request), measured from the YOUNGEST in-flight request.
+        0 when at least one thread is free."""
+        with self._inflight_lock:
+            starts = [e.start for e in self._inflight.values()]
+        if len(starts) < self.cfg.threads:
+            return 0.0
+        return time.monotonic() - max(starts)
+
+    def _dump_stacks(self):
+        frames = sys._current_frames()
+        names = {t.ident: t.name for t in threading.enumerate()}
+        out = []
+        for ident, frame in frames.items():
+            out.append('--- thread %s (%s)\n%s' % (
+                ident, names.get(ident, '?'), ''.join(traceback.format_stack(frame))))
+        return '\n'.join(out)
+
+    def _check_wedged(self):
+        wedged = self._wedged_for()
+        if WEDGE_SECONDS > 0 and wedged > WEDGE_SECONDS:
+            try:
+                self.log.critical(
+                    'WEDGED WORKER (pid %s): all %s threads busy in the same request '
+                    'for %.0fs > HAPROXY_MGR_WEDGE_SECONDS=%s. Exiting so the master '
+                    'respawns it. Thread stacks:\n%s',
+                    os.getpid(), self.cfg.threads, wedged, WEDGE_SECONDS,
+                    self._dump_stacks())
+                for h in logging.getLogger('gunicorn.error').handlers:
+                    try:
+                        h.flush()
+                    except Exception:
+                        pass
+            finally:
+                # os._exit, not SIGTERM: a graceful exit joins the (stuck,
+                # non-daemon) pool threads at interpreter shutdown and would
+                # hang until the master's --timeout escalates to SIGKILL.
+                os._exit(WEDGED_EXIT_CODE)
+
+    def _monitor(self):
+        """One daemon thread: header-deadline sweep every second, wedge check
+        every WATCHDOG_INTERVAL. Never allowed to die silently."""
+        last_wedge_check = time.monotonic()
+        while self.alive:
+            time.sleep(MONITOR_TICK)
+            try:
+                if HEADER_DEADLINE > 0:
+                    self._sweep_header_deadlines()
+                if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
+                    self._flush_rejected()
+                if time.monotonic() - last_wedge_check >= WATCHDOG_INTERVAL:
+                    last_wedge_check = time.monotonic()
+                    self._check_wedged()
+            except Exception:
+                try:
+                    self.log.exception('haproxy_gunicorn_worker monitor error (continuing)')
+                except Exception:
+                    pass
+
+    def init_process(self):
+        global _current_worker
+        _current_worker = self
+        try:
+            self.log.error_log.addFilter(_QuietClientTimeout())
+        except AttributeError:
+            pass
+        if gunicorn.__version__ != TESTED_GUNICORN_VERSION:
+            self.log.warning(
+                'haproxy_gunicorn_worker overrides gthread internals as of gunicorn %s; '
+                'running %s -- re-run scripts/test-gunicorn-socket-timeouts.py',
+                TESTED_GUNICORN_VERSION, gunicorn.__version__)
+        if self._filter_ports:
+            self._allowed_nets, problems = parse_allowed_peers(ALLOWED_PEERS_SPEC)
+            for prob in problems:
+                self.log.error('HAPROXY_MGR_API_ALLOWED_PEERS: %s -- that entry allows nothing '
+                               '(fail closed)', prob)
+            self.log.info('Management API peer filter on port(s) %s: allowing %s',
+                          ','.join(str(p) for p in sorted(self._filter_ports)),
+                          ', '.join(str(n) for n in self._allowed_nets) or 'NOTHING')
+        threading.Thread(target=self._monitor, name='worker-monitor',
+                         daemon=True).start()
+        super().init_process()   # runs the event loop; does not return

scripts/cert-publish-lib.sh

@@ -203,6 +203,82 @@ cert_bundle_valid() {
     return 0
 }
 
+# cert_is_expired CERT_FILE
+#
+# Returns 0 only when openssl positively reports that the first certificate in
+# CERT_FILE has expired. Anything else (not expired, unreadable, no openssl)
+# returns 1; unreadable material is rejected with a better message by
+# cert_bundle_valid().
+#
+# Why: HAProxy loads the crt directory wholesale and serves the FIRST file that
+# claims an SNI name. An expired bundle in there is never useful and can only
+# shadow a valid one - on 2026-10-07 renew-certificates.sh kept re-publishing
+# the expired brain-jar.com lineage every 12 hours, and HAProxy served it for
+# brainjar.net instead of the valid brainjar.net bundle next to it.
+cert_is_expired() {
+    local file="$1" out
+    [ -f "$file" ] || return 1
+    command -v openssl >/dev/null 2>&1 || return 1
+    out="$(openssl x509 -checkend 0 -noout -in "$file" 2>/dev/null </dev/null)"
+    [ $? -eq 1 ] && [[ "$out" == *"will expire"* ]]
+}
+
+# cert_not_after CERT_FILE - the notAfter date string, or empty.
+cert_not_after() {
+    openssl x509 -enddate -noout -in "$1" 2>/dev/null </dev/null | sed 's/^notAfter=//'
+}
+
+cert_retired_dir() {
+    if [ -n "${CERT_RETIRED_DIR:-}" ]; then
+        echo "$CERT_RETIRED_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-retired"
+    fi
+}
+
+# cert_retire_expired_pems
+#
+# Move every EXPIRED *.pem out of $SSL_CERTS_DIR into cert-retired/ (a sibling,
+# never inside the crt dir). Sets CERT_RETIRED_EXPIRED to the number moved.
+#
+# Not publishing expired lineages stops the re-publish loop, but a pem that was
+# published while valid and has expired since stays in the crt dir and keeps
+# shadowing - that was exactly brain-jar.com.pem (review M2). An expired
+# certificate is never a valid certificate for anyone, so it can always go: the
+# only-valid-cert safety rule cannot be violated by moving it. Two exceptions:
+#   * default_self_signed_cert.pem - HAProxy's fallback, regenerated separately;
+#   * the LAST .pem in the directory - a crt directory with no certificate at
+#     all fails the whole :443 bind, which is worse than serving an expired one.
+# Nothing is deleted; reverse with mv.
+cert_retire_expired_pems() {
+    local dir="${SSL_CERTS_DIR:-/etc/haproxy/certs}" retired f base dest remaining
+    CERT_RETIRED_EXPIRED=0
+    retired="$(cert_retired_dir)"
+    for f in "$dir"/*.pem; do
+        [ -f "$f" ] || continue
+        base="$(basename "$f")"
+        [ "$base" = "default_self_signed_cert.pem" ] && continue
+        cert_is_expired "$f" || continue
+        remaining="$(find "$dir" -maxdepth 1 -type f -name '*.pem' | wc -l)"
+        if [ "$remaining" -le 1 ]; then
+            log_error "Expired certificate $f is the only certificate left in $dir - NOT retiring it (an empty crt directory fails the :443 bind)"
+            continue
+        fi
+        if ! mkdir -p "$retired"; then
+            log_error "cannot create $retired - expired $f left in place"
+            continue
+        fi
+        dest="$retired/$base"
+        [ -e "$dest" ] && dest="$dest.$(date -u +%Y%m%dT%H%M%SZ)"
+        if mv -- "$f" "$dest"; then
+            log_error "Retired EXPIRED certificate $base (notAfter $(cert_not_after "$dest")) from $dir to $dest - it could only shadow valid certificates"
+            CERT_RETIRED_EXPIRED=$((CERT_RETIRED_EXPIRED + 1))
+        else
+            log_error "failed to move expired $f to $dest"
+        fi
+    done
+}
+
 # cert_publish CERT_FILE KEY_FILE DEST_FILE
 #
 # Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
@@ -227,6 +303,11 @@ cert_publish() {
         log_error "cert_publish: private key $key_file is missing or empty"
         return 1
     fi
+    # (a2) never publish an expired certificate (see cert_is_expired).
+    if cert_is_expired "$cert_file"; then
+        log_error "cert_publish: certificate $cert_file expired ($(cert_not_after "$cert_file")) - refusing to publish $dest_file"
+        return 1
+    fi
 
     # (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
     #     scans wholesale.

scripts/healthcheck.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Container liveness probe: BOTH planes must answer quickly.
+#
+#   1. Management API (:8000) -- GET /health with a short --max-time. On
+#      2026-10-09 (sdbees) the API workers' thread pools were wedged for long
+#      enough that /health hung 40s+, while the container reported "healthy"
+#      because the deployed --health-cmd only probed :80. A hung API is a dead
+#      control plane (every WHP site update times out), so it must turn the
+#      container unhealthy.
+#   2. HAProxy data plane (:80) -- any HTTP answer at all (the default page,
+#      a redirect, a 4xx/5xx) proves HAProxy accepted and answered. Only a
+#      connect/timeout failure fails it, which is why there is no -f here.
+#
+# 127.0.0.1, not localhost: the API binds IPv4 only, and resolving localhost
+# to ::1 first costs a failed connect on every probe.
+#
+# WHP's lib-edge-container.sh overrides the image HEALTHCHECK with its own
+# --health-cmd; point that at this script so the fleet gets the same probe.
+set -u
+API_TIMEOUT="${HAPROXY_MGR_HEALTH_API_TIMEOUT:-4}"
+EDGE_TIMEOUT="${HAPROXY_MGR_HEALTH_EDGE_TIMEOUT:-5}"
+
+if ! curl -sf -o /dev/null --max-time "$API_TIMEOUT" http://127.0.0.1:8000/health; then
+    echo "management API :8000 /health failed or took longer than ${API_TIMEOUT}s"
+    exit 1
+fi
+if ! curl -s -o /dev/null --max-time "$EDGE_TIMEOUT" http://127.0.0.1/; then
+    echo "HAProxy :80 did not answer within ${EDGE_TIMEOUT}s"
+    exit 1
+fi
+exit 0

scripts/renew-certificates.sh

@@ -65,13 +65,22 @@ fi
 # Copy certificates for each domain
 UPDATED=0
 FAILED=0
+SKIPPED_EXPIRED=0
 
 while read -r domain; do
     CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
     KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
-    if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
+    if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
+        # Never publish an expired lineage. It cannot help any client and,
+        # because HAProxy serves the first file claiming an SNI name, it can
+        # shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
+        # Counted separately from FAILED: a dead lineage that keeps expiring is
+        # reported on every run (ERROR log) without making every run "fail".
+        log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
+        SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
+    elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
         # Assemble in a staging dir and rename into place. NEVER redirect into
         # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
         # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -89,10 +98,15 @@ while read -r domain; do
     fi
 done <<< "$DOMAINS"
 
-log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
+# Already-published pems that have expired since are moved out too (they
+# can only shadow; see cert_retire_expired_pems).
+cert_retire_expired_pems
+RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
+
+log_info "Certificate update completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
 
 # Reload HAProxy if any certificates were updated
-if [ $UPDATED -gt 0 ]; then
+if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
     # Never reload onto unvalidated material: a reload that fails to load the
     # certs directory drops HTTPS for every site on this host.
     if ! haproxy_config_ok; then

scripts/start-up.sh

@@ -77,7 +77,7 @@ HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
 ) &
 
 # Phase 2: WSGI servers
-# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
+# Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
 # (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
 # 1000 — worker recycle frequency).
 #
@@ -91,12 +91,27 @@ API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
 MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
 MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
 
+# Worker class (incident 2026-10-09): stock gthread makes every client socket
+# fully blocking with NO timeout, so any peer that sends a few bytes and goes
+# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
+# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. Stock gthread
+# also never reaps connections that send nothing (2x1000 of them wedge it
+# permanently) and has no total header-read bound (slowloris). This subclass
+# adds: a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, 20s), a
+# first-byte deadline (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s), a total
+# request-line+headers deadline (HAPROXY_MGR_HEADER_DEADLINE, 10s), and a
+# watchdog that exits a worker whose threads have ALL been stuck for
+# HAPROXY_MGR_WEDGE_SECONDS (3600) so the master respawns it.
+# See haproxy_gunicorn_worker.py. --timeout below does NOT cover any of these:
+# for gthread it only watches the main thread's heartbeat.
+WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
+
 # Default page server on :8080. Stays in the background.
 # --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
 # without forking. --max-requests recycles the worker to bound memory drift.
 gunicorn \
     --bind 0.0.0.0:8080 \
-    --workers 1 --threads 4 --worker-class gthread \
+    --workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout 30 \
     --access-logfile - --error-logfile - --log-level info \
@@ -108,7 +123,7 @@ gunicorn \
 # up). Longer --timeout because cert issuance hits ACME and can take a while.
 exec gunicorn \
     --bind 0.0.0.0:8000 \
-    --workers "${API_WORKERS}" --threads 4 --worker-class gthread \
+    --workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout "${API_TIMEOUT}" \
     --access-logfile - --error-logfile - --log-level info \

scripts/sync-certificates.sh

@@ -47,13 +47,22 @@ fi
 # Copy certificates for each domain
 UPDATED=0
 FAILED=0
+SKIPPED_EXPIRED=0
 
 while read -r domain; do
     CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
     KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
-    if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
+    if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
+        # Never publish an expired lineage. It cannot help any client and,
+        # because HAProxy serves the first file claiming an SNI name, it can
+        # shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
+        # Counted separately from FAILED: a dead lineage that keeps expiring is
+        # reported on every run (ERROR log) without making every run "fail".
+        log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
+        SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
+    elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
         # Assemble in a staging dir and rename into place. NEVER redirect into
         # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
         # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -71,10 +80,15 @@ while read -r domain; do
     fi
 done <<< "$DOMAINS"
 
-log_info "Certificate sync completed: $UPDATED updated, $FAILED failed"
+# Already-published pems that have expired since are moved out too (they
+# can only shadow; see cert_retire_expired_pems).
+cert_retire_expired_pems
+RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
+
+log_info "Certificate sync completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
 
 # Reload HAProxy if any certificates were updated
-if [ $UPDATED -gt 0 ]; then
+if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
     # Never reload onto unvalidated material: a reload that fails to load the
     # certs directory drops HTTPS for every site on this host.
     if ! haproxy_config_ok; then

scripts/test-cert-scripts.py

@@ -73,6 +73,15 @@ LIB = os.path.join(SCRIPTS_DIR, 'cert-publish-lib.sh')
 BROKEN_TOKEN = '__BROKEN__'
 DOMAIN = 'test.example.com'
 
+# Real SAN certificates (see scripts/test-fixtures/san-certs/). expired_old is
+# the brain-jar.com shape: a multi-SAN certificate that expired in 2020.
+SAN_FIXTURES = os.path.join(SCRIPTS_DIR, 'test-fixtures', 'san-certs')
+
+
+def san_fixture(name):
+    with open(os.path.join(SAN_FIXTURES, name)) as fh:
+        return fh.read()
+
 # --- test key material -------------------------------------------------------
 # openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \
 #         -subj /CN=test.example.com
@@ -618,6 +627,72 @@ class CertScriptBehaviour:
         self.assertFalse(self.reload_attempted())
 
 
+    # -- never publish an expired certificate (brainjar.net, 2026-10-07) ----
+    def test_expired_lineage_is_skipped_and_reported(self):
+        """renew-certificates.sh used to re-publish EVERY live/ lineage every 12h,
+        expired or not; the expired brain-jar.com bundle then shadowed the valid
+        brainjar.net bundle for the names they shared."""
+        dead = os.path.join(self.le_live, 'brain-jar.com')
+        os.makedirs(dead)
+        write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
+        write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
+
+        result = self.run_it()
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertFalse(
+            os.path.exists(os.path.join(self.certs_dir, 'brain-jar.com.pem')),
+            'an expired lineage was published into the crt directory')
+        # The healthy lineage next to it is still published.
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+        self.assertIn('Skipping EXPIRED certificate lineage brain-jar.com', self.logs())
+        self.assertIn('1 updated, 0 failed, 1 expired lineage(s) skipped', self.logs())
+        self.assertIn('brain-jar.com', read(self.error_log),
+                      'the skip must land in the monitored error log')
+
+    def test_expired_lineage_does_not_replace_an_existing_pem(self):
+        dead = os.path.join(self.le_live, 'brain-jar.com')
+        os.makedirs(dead)
+        write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
+        write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
+        existing = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'), OTHER_BUNDLE)
+
+        self.run_it()
+
+        self.assertEqual(read(existing), OTHER_BUNDLE,
+                         'an expired lineage overwrote the file in the crt dir')
+
+    def test_already_published_expired_pem_is_retired(self):
+        """Review M2: skipping publication is not enough - an expired pem that
+        is ALREADY in the crt dir (brain-jar.com.pem) keeps shadowing."""
+        expired = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'),
+                        san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertFalse(os.path.exists(expired), 'expired pem still in the crt dir')
+        retired = os.path.join(self.haproxy_dir, 'cert-retired', 'brain-jar.com.pem')
+        self.assertTrue(os.path.exists(retired), 'expired pem was not moved to cert-retired/ (never delete)')
+        self.assertIn('Retired EXPIRED certificate brain-jar.com.pem', self.logs())
+        self.assertIn('1 expired pem(s) retired', self.logs())
+        self.assertTrue(self.reload_attempted())
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE, 'valid neighbours untouched')
+        self.assert_certs_dir_is_clean()
+
+    def test_retiring_an_expired_pem_alone_triggers_a_reload(self):
+        self.seed_previous_bundle(NEW_BUNDLE)
+        shutil.rmtree(self.domain_dir)
+        other = os.path.join(self.le_live, 'other.example')
+        os.makedirs(other)
+        write(os.path.join(other, 'fullchain.pem'), san_fixture('expired_old.crt'))
+        write(os.path.join(other, 'privkey.pem'), san_fixture('expired_old.key'))
+        write(os.path.join(self.certs_dir, 'other.example.pem'),
+              san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertIn('0 updated', self.logs())
+        self.assertTrue(self.reload_attempted(), 'HAProxy must reload to drop the retired pem')
+
+
 class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture):
     SCRIPT = 'renew-certificates.sh'
 
@@ -638,6 +713,52 @@ class TestCertPublishLibrary(CertScriptFixture):
         path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
         self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0)
 
+    def test_cert_is_expired_reports_both_directions(self):
+        expired = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
+        valid = write(os.path.join(self.tmp, 'valid.crt'), san_fixture('new_bundle.crt'))
+        self.assertEqual(0, self.call('cert_is_expired "$1"', expired).returncode,
+                         'an expired certificate was not recognised as expired')
+        self.assertEqual(1, self.call('cert_is_expired "$1"', valid).returncode,
+                         'a valid certificate was reported as expired')
+        self.assertEqual(1, self.call('cert_is_expired "$1"',
+                                      os.path.join(self.tmp, 'missing')).returncode)
+
+    def test_expired_sweep_keeps_the_last_pem_and_the_default_cert(self):
+        exp = san_fixture('expired_old.crt') + san_fixture('expired_old.key')
+        only = write(os.path.join(self.certs_dir, 'only.pem'), exp)
+        r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
+        self.assertIn('n=0', r.stdout)
+        self.assertTrue(os.path.exists(only), 'emptied the crt dir: that fails the whole :443 bind')
+        dflt = write(os.path.join(self.certs_dir, 'default_self_signed_cert.pem'), exp)
+        r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
+        self.assertIn('n=1', r.stdout)
+        self.assertTrue(os.path.exists(dflt), 'the default self-signed cert is not ours to retire')
+        self.assertFalse(os.path.exists(only))
+
+    def test_expired_sweep_never_touches_a_valid_pem(self):
+        valid = write(os.path.join(self.certs_dir, 'a.pem'), NEW_BUNDLE)
+        write(os.path.join(self.certs_dir, 'b.pem'), OTHER_BUNDLE)
+        r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
+        self.assertIn('n=0', r.stdout)
+        self.assertEqual(read(valid), NEW_BUNDLE)
+
+    def test_cert_publish_refuses_an_expired_certificate(self):
+        cert = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
+        key = write(os.path.join(self.tmp, 'expired.key'), san_fixture('expired_old.key'))
+        dest = os.path.join(self.certs_dir, 'brain-jar.com.pem')
+
+        result = self.call('cert_publish "$1" "$2" "$3"', cert, key, dest)
+
+        self.assert_rejected(result, because='expired')
+        self.assertFalse(os.path.exists(dest), 'expired certificate was published')
+        # And the same material with a valid certificate IS publishable, so the
+        # refusal above is about expiry, not about the fixture.
+        good_cert = write(os.path.join(self.tmp, 'good.crt'), san_fixture('new_bundle.crt'))
+        good_key = write(os.path.join(self.tmp, 'good.key'), san_fixture('new_bundle.key'))
+        ok = self.call('cert_publish "$1" "$2" "$3"', good_cert, good_key, dest)
+        self.assertEqual(0, ok.returncode, ok.stdout + ok.stderr)
+        self.assertTrue(os.path.exists(dest))
+
     def test_empty_and_missing_bundles_rejected(self):
         empty = write(os.path.join(self.tmp, 'empty.pem'), '')
         self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),

scripts/test-cert-write-safety.py

@@ -27,6 +27,8 @@ These tests pin the invariants:
   * nothing is published that is not a complete, validated cert+key pair;
   * no old certificate file is removed and no lineage deleted until the
     replacement is validated, in place, and actually loaded by HAProxy;
+  * removing ONE domain never unlinks a .pem, or deletes a certbot lineage,
+    that other still-configured domains are being served from;
   * only final .pem files ever exist in the crt directory.
 
 Running
@@ -55,6 +57,7 @@ import subprocess
 import sys
 import tempfile
 import textwrap
+import time
 import unittest
 
 BROKEN_TOKEN = '__BROKEN__'
@@ -252,6 +255,9 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
 
 # certbot stub: succeeds, announces a renewal, and records every invocation so
 # tests can assert that `certbot delete` did or did not run.
+#   TEST_CERTBOT_INSTALL_SRC/_DIR: on certonly, copy fullchain.pem + privkey.pem
+#       from SRC into DIR (a real issuance changes the lineage's certificate).
+#   TEST_CERTBOT_NOT_DUE: on certonly, print certbot's "not yet due" wording.
 FAKE_CERTBOT = textwrap.dedent("""\
     #!/bin/sh
     if [ -n "$TEST_CERTBOT_LOG" ]; then
@@ -260,6 +266,14 @@ FAKE_CERTBOT = textwrap.dedent("""\
     case "$1" in
       renew) echo "Congratulations, all renewals succeeded" ;;
       delete) [ -n "$TEST_CERTBOT_DELETE_FAILS" ] && exit 1 ;;
+      certonly)
+        if [ -n "$TEST_CERTBOT_INSTALL_SRC" ]; then
+          mkdir -p "$TEST_CERTBOT_INSTALL_DIR"
+          cp "$TEST_CERTBOT_INSTALL_SRC/fullchain.pem" "$TEST_CERTBOT_INSTALL_SRC/privkey.pem" "$TEST_CERTBOT_INSTALL_DIR/"
+        fi
+        if [ -n "$TEST_CERTBOT_NOT_DUE" ]; then
+          echo "Certificate not yet due for renewal; no action taken."
+        fi ;;
     esac
     exit 0
 """)
@@ -329,6 +343,7 @@ class CertPublishTestCase(unittest.TestCase):
             'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'),
             'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'),
             'SSL_CERTS_DIR': self.certs,
+            'LETSENCRYPT_DIR': os.path.join(self.tmp, 'letsencrypt'),
             'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'),
             'API_KEY': None,
         }
@@ -572,7 +587,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
 
     @FIX_ONLY
     @NEEDS_OPENSSL
-    def test_lineage_is_deleted_only_after_haproxy_loads_the_bundle(self):
+    def test_lineage_is_untouched_when_haproxy_does_not_load_the_bundle(self):
         """A reload failure must leave the lineage intact and the file recoverable."""
         old_path = self._setup_supersede(broken=False)
         # Make generate_config() produce a config the validator rejects, so the
@@ -589,7 +604,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
             [], self.certbot_deletes(),
             'the lineage must not be deleted when HAProxy did not reload')
         self.assertFalse(os.path.exists(old_path))
-        quarantined = os.path.join(hm.cert_backup_dir(),
+        quarantined = os.path.join(hm.cert_retired_dir(),
                                    os.path.basename(old_path))
         self.assertTrue(os.path.exists(quarantined),
                         'the superseded file must be recoverable by hand')
@@ -597,8 +612,9 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
 
     @FIX_ONLY
     @NEEDS_OPENSSL
-    def test_successful_bundle_still_supersedes_and_deletes(self):
-        """The cleanup must still do its job on the happy path."""
+    def test_successful_bundle_still_supersedes_without_deleting(self):
+        """The cleanup must still do its job on the happy path - by moving, never
+        by `certbot delete` (irreversible; recovery is a rate-limited ACME order)."""
         old_path = self._setup_supersede(broken=False)
 
         resp = self.client.post('/api/ssl/bundle', json={
@@ -610,12 +626,381 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
         self.assertFalse(os.path.exists(old_path),
                          'the superseded file must leave the crt directory or '
                          'it keeps shadowing the new bundle')
-        self.assertEqual(['delete --cert-name www.test.example.com -n'],
-                         self.certbot_deletes())
+        self.assertTrue(os.path.exists(os.path.join(
+            hm.cert_retired_dir(), os.path.basename(old_path))),
+            'the superseded file must be moved aside, not destroyed')
+        self.assertEqual([], self.certbot_deletes(),
+                         'retirement must never `certbot delete` anything')
         self.assertTrue(self.edge_would_start())
         self.assert_only_final_pems_in_certs_dir()
 
 
+
+# --- SAN-overlap retirement (brainjar.net, 2026-10-07) -----------------------
+SAN_FIXTURES = os.path.join(MODULE_DIR, 'scripts', 'test-fixtures', 'san-certs')
+
+
+def san(name):
+    with open(os.path.join(SAN_FIXTURES, name)) as fh:
+        return fh.read()
+
+
+def make_certbot_lineage(le_dir, lineage, cert, key):
+    """archive/<x>/{cert,chain,fullchain,privkey}1.pem + live/<x>/*.pem
+    symlinks (../../archive/<x>/...1.pem) + renewal/<x>.conf, like certbot."""
+    archive = os.path.join(le_dir, 'archive', lineage)
+    live = os.path.join(le_dir, 'live', lineage)
+    os.makedirs(archive, exist_ok=True)
+    os.makedirs(live, exist_ok=True)
+    for kind, body in (('cert', cert), ('chain', cert), ('fullchain', cert), ('privkey', key)):
+        with open(os.path.join(archive, f'{kind}1.pem'), 'w') as fh:
+            fh.write(body)
+        link = os.path.join(live, f'{kind}.pem')
+        if os.path.lexists(link):
+            os.remove(link)
+        os.symlink(f'../../archive/{lineage}/{kind}1.pem', link)
+    os.makedirs(os.path.join(le_dir, 'renewal'), exist_ok=True)
+    with open(os.path.join(le_dir, 'renewal', f'{lineage}.conf'), 'w') as fh:
+        fh.write(f'archive_dir = {archive}\ncert = {live}/cert.pem\n')
+    return live
+
+
+def retired_path(le_dir, part, name):
+    hits = sorted(__import__('glob').glob(os.path.join(le_dir, 'retired', '*', part, name)))
+    return hits[-1] if hits else os.path.join(le_dir, 'retired', '<none>', part, name)
+
+
+FIX_ONLY_RETIRE = unittest.skipIf(
+    TESTING_FOREIGN_TREE and not hasattr(hm, 'retirement_blockers'),
+    'HAPROXY_MANAGER_DIR points at a tree without SAN-overlap retirement')
+
+
+class TestRetirementRules(unittest.TestCase):
+    """retirement_blockers() is the shared-PEM safety rule, tested directly."""
+
+    NOW = 2_000_000_000
+
+    def cert(self, names, days_left, fp):
+        return {'names': set(names), 'fingerprint': fp,
+                'not_after': self.NOW + days_left * 86400, 'cn': sorted(names)[0]}
+
+    @FIX_ONLY_RETIRE
+    def test_expired_candidate_never_blocks(self):
+        old = self.cert(['brain-jar.com', 'brainjar.net'], -2, 'A')
+        self.assertEqual([], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'brain-jar.com', 'brainjar.net'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_valid_candidate_that_is_last_cert_for_a_served_name_is_kept(self):
+        old = self.cert(['shared.example', 'brainjar.net', 'other.example'], 40, 'A')
+        self.assertEqual(['other.example'], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'brainjar.net', 'other.example'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_unreachable_names_do_not_block(self):
+        old = self.cert(['dead.example', 'brainjar.net'], 40, 'A')
+        self.assertEqual([], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'brainjar.net', 'dead.example'},
+            ['dead.example'], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_name_covered_by_another_valid_cert_does_not_block(self):
+        old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
+        other = self.cert(['other.example'], 60, 'B')
+        self.assertEqual([], hm.retirement_blockers(
+            old, [other], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_cover_by_an_expired_or_identical_cert_does_not_count(self):
+        old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
+        expired = self.cert(['other.example'], -1, 'B')
+        same = self.cert(['other.example'], 60, 'A')
+        self.assertEqual(['other.example'], hm.retirement_blockers(
+            old, [expired, same], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_unknown_served_set_keeps_every_valid_name(self):
+        old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
+        self.assertEqual(['other.example'], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, None, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    def test_wildcard_coverage(self):
+        self.assertTrue(hm.cert_name_covers('*.example.com', 'a.example.com'))
+        self.assertFalse(hm.cert_name_covers('*.example.com', 'a.b.example.com'))
+        self.assertFalse(hm.cert_name_covers('*.example.com', 'example.com'))
+        old = self.cert(['*.example.com', 'brainjar.net'], 40, 'A')
+        self.assertEqual(['shop.example.com'], hm.retirement_blockers(
+            old, [], {'brainjar.net'}, {'shop.example.com'}, [], now=self.NOW))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_read_cert_info_reads_sans_and_expiry(self):
+        tmp = tempfile.mkdtemp(prefix='cert-info-')
+        self.addCleanup(shutil.rmtree, tmp, True)
+        path = os.path.join(tmp, 'c.pem')
+        with open(path, 'w') as fh:
+            fh.write(san('expired_old.crt'))
+        info = hm.read_cert_info(path)
+        self.assertEqual({'brain-jar.com', 'www.brain-jar.com', 'brainjar.net',
+                          'www.brainjar.net'}, info['names'])
+        self.assertEqual('brain-jar.com', info['cn'])
+        self.assertEqual(1585699200, info['not_after'])  # 2020-04-01T00:00:00Z
+        self.assertTrue(info['fingerprint'])
+
+
+class TestSanOverlapRetirement(CertPublishTestCase):
+    """The new bundle must not stay shadowed by an older cert sharing a SAN."""
+
+    def setUp(self):
+        super().setUp()
+        self.le = os.path.join(self.tmp, 'letsencrypt')
+        self.new_src = os.path.join(self.tmp, 'issued', 'brainjar.net')
+        self.write(os.path.join(self.new_src, 'fullchain.pem'), san('new_bundle.crt'))
+        self.write(os.path.join(self.new_src, 'privkey.pem'), san('new_bundle.key'))
+        # certbot "issues" brainjar.net into live/ when called.
+        os.environ['TEST_CERTBOT_INSTALL_SRC'] = self.new_src
+        os.environ['TEST_CERTBOT_INSTALL_DIR'] = os.path.join(self.le_live, 'brainjar.net')
+        for name in ('brainjar.net', 'www.brainjar.net'):
+            self.add_domain(name, name.replace('.', '_'))
+
+    def old_lineage(self, lineage, fixture, publish=True):
+        """A lineage laid out the way certbot does it: real files in
+        archive/<x>/*1.pem, relative symlinks in live/<x>/, a renewal conf."""
+        live = make_certbot_lineage(self.le, lineage, san(f'{fixture}.crt'),
+                                    san(f'{fixture}.key'))
+        conf = os.path.join(self.le, 'renewal', f'{lineage}.conf')
+        if not publish:
+            return live, conf, None
+        pem = self.write(os.path.join(self.certs, f'{lineage}.pem'),
+                         san(f'{fixture}.crt') + san(f'{fixture}.key'))
+        return live, conf, pem
+
+    def bundle(self, **extra):
+        body = {'primary': 'brainjar.net', 'sans': ['www.brainjar.net']}
+        body.update(extra)
+        resp = self.client.post('/api/ssl/bundle', json=body)
+        self.assertEqual(200, resp.status_code, resp.get_data(as_text=True))
+        return resp.get_json()
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_expired_overlapping_cert_with_a_different_cn_is_retired(self):
+        """The incident: CN brain-jar.com, SANs include brainjar.net."""
+        self.add_domain('brain-jar.com', 'bjc')
+        self.add_domain('www.brain-jar.com', 'wbjc')
+        live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
+
+        data = self.bundle(unreachable_names=['brain-jar.com', 'www.brain-jar.com'])
+
+        self.assertFalse(os.path.exists(pem),
+                         'the expired SAN-overlapping pem is still shadowing the bundle')
+        retired = [e['file'] for e in data['cleanup']['removed']]
+        self.assertEqual(['brain-jar.com.pem'], retired)
+        self.assertTrue(os.path.exists(os.path.join(hm.cert_retired_dir(), 'brain-jar.com.pem')))
+        # Lineage retired, not deleted: renew-certificates.sh can no longer
+        # re-publish it and certbot can no longer try to renew it.
+        self.assertFalse(os.path.exists(live))
+        self.assertFalse(os.path.exists(conf))
+        # live, archive AND renewal leave together (review H2), into one tree.
+        self.assertFalse(os.path.exists(os.path.join(self.le, 'archive', 'brain-jar.com')))
+        self.assertTrue(os.path.isdir(retired_path(self.le, 'live', 'brain-jar.com')))
+        self.assertTrue(os.path.isdir(retired_path(self.le, 'archive', 'brain-jar.com')))
+        self.assertTrue(os.path.exists(retired_path(self.le, 'renewal', 'brain-jar.com.conf')))
+        # Relative symlinks still resolve in the retired tree (restore = mv back).
+        self.assertIn('BEGIN CERTIFICATE', self.read(
+            os.path.join(retired_path(self.le, 'live', 'brain-jar.com'), 'cert.pem')))
+        self.assertEqual([], self.certbot_deletes())
+        self.assertTrue(os.path.exists(os.path.join(self.certs, 'brainjar.net.pem')))
+        self.assertTrue(self.edge_would_start())
+        self.assert_only_final_pems_in_certs_dir()
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_expired_overlap_is_retired_even_without_the_unreachable_hint(self):
+        self.add_domain('brain-jar.com', 'bjc')
+        live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
+        self.bundle()
+        self.assertFalse(os.path.exists(pem))
+        self.assertFalse(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_shared_pem_still_serving_another_site_is_kept(self):
+        """The shared-PEM hazard: never retire the only valid cert for a served name."""
+        self.add_domain('other-site.example', 'other_site')
+        live, conf, pem = self.old_lineage('shared-old.example', 'shared_old')
+
+        data = self.bundle()
+
+        self.assertTrue(os.path.exists(pem),
+                        'retired the only valid certificate for other-site.example')
+        self.assertTrue(os.path.exists(live))
+        self.assertTrue(os.path.exists(conf))
+        kept = {e['file']: e['blockers'] for e in data['cleanup']['kept']}
+        self.assertEqual({'shared-old.example.pem': ['other-site.example']}, kept)
+        self.assertEqual(['other-site.example'],
+                         data['cleanup']['lineages']['kept'][0]['blockers'])
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_valid_overlap_whose_other_names_are_unreachable_is_retired(self):
+        self.add_domain('dead.example', 'dead')
+        live, conf, pem = self.old_lineage('dead.example', 'dead_old')
+
+        self.bundle(unreachable_names=['dead.example'])
+
+        self.assertFalse(os.path.exists(pem))
+        self.assertFalse(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_valid_overlap_still_served_without_the_hint_is_kept(self):
+        """Same files as above, but nobody said dead.example is gone: keep it."""
+        self.add_domain('dead.example', 'dead')
+        live, conf, pem = self.old_lineage('dead.example', 'dead_old')
+
+        self.bundle()
+
+        self.assertTrue(os.path.exists(pem))
+        self.assertTrue(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_non_overlapping_certificates_are_untouched(self):
+        live, conf, pem = self.old_lineage('unrelated.example', 'unrelated')
+        self.bundle()
+        self.assertTrue(os.path.exists(pem), 'retired a cert sharing no name with the bundle')
+        self.assertTrue(os.path.exists(live))
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_response_distinguishes_issuance_from_not_yet_due(self):
+        issued = self.bundle()
+        self.assertTrue(issued['renewed'], 'a fresh issuance was not reported as one')
+        self.assertEqual(4922899200, issued['not_after'])  # 2126-01-01T00:00:00Z
+
+        # Lineage now exists with the same names; certbot keeps it.
+        del os.environ['TEST_CERTBOT_INSTALL_SRC']
+        os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
+        noop = self.bundle()
+        self.assertFalse(noop['renewed'], 'a certbot no-op was reported as a renewal')
+        self.assertTrue(noop['certbot_not_due'])
+        self.assertEqual(4922899200, noop['not_after'])
+        self.assertIn('--keep-until-expiring', self.certbot_invocations()[-1])
+
+
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_not_due_noop_never_retires_its_own_cert_name_lineage(self):
+        """Review H1: find_certbot_live_dir() returns the NEWEST of brainjar.net
+        and brainjar.net-0001. After a certbot no-op that was the stale -0001:
+        it got published as brainjar.net.pem and the real brainjar.net lineage
+        was retired as "superseded"."""
+        del os.environ['TEST_CERTBOT_INSTALL_SRC']
+        os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
+        real = make_certbot_lineage(self.le, 'brainjar.net', san('new_bundle.crt'), san('new_bundle.key'))
+        stale = make_certbot_lineage(self.le, 'brainjar.net-0001', san('dead_old.crt'), san('dead_old.key'))
+        later = time.time() + 60
+        os.utime(os.path.join(self.le, 'archive', 'brainjar.net-0001', 'fullchain1.pem'), (later, later))
+        # Production behaviour of find_certbot_live_dir(): newest wins.
+        hm.find_certbot_live_dir = lambda base: stale if base == 'brainjar.net' else None
+
+        data = self.bundle()
+
+        self.assertTrue(os.path.isdir(real), 'the --cert-name lineage itself was retired')
+        self.assertTrue(os.path.exists(os.path.join(self.le, 'renewal', 'brainjar.net.conf')))
+        self.assertNotIn('brainjar.net', [e['lineage'] for e in data['cleanup']['lineages']['retired']])
+        published = self.read(os.path.join(self.certs, 'brainjar.net.pem'))
+        self.assertIn(san('new_bundle.crt').strip().splitlines()[1], published,
+                      'published the stale -0001 certificate instead of live/brainjar.net')
+        self.assertFalse(data['renewed'])
+
+    @FIX_ONLY_RETIRE
+    def test_bundle_reports_capabilities(self):
+        data = self.bundle()
+        self.assertIn('san_overlap_retirement', data.get('capabilities', []))
+        caps = self.client.get('/api/capabilities').get_json()
+        self.assertIn('unreachable_names', caps['capabilities'])
+
+
+class TestRetiredLineageCanBeReissued(CertPublishTestCase):
+    """Review H2: retirement must leave certbot able to issue the name again."""
+
+    def setUp(self):
+        super().setUp()
+        self.le = hm.LETSENCRYPT_DIR
+
+    @FIX_ONLY_RETIRE
+    def test_retirement_removes_every_part_certbot_checks(self):
+        make_certbot_lineage(self.le, 'brain-jar.com', san('expired_old.crt'), san('expired_old.key'))
+        out = hm.retire_lineage_tree('brain-jar.com')
+        # certbot new_lineage() refuses if any of these exist for the name.
+        for part in (('live', 'brain-jar.com'), ('archive', 'brain-jar.com'),
+                     ('renewal', 'brain-jar.com.conf')):
+            self.assertFalse(os.path.lexists(os.path.join(self.le, *part)), part)
+            self.assertTrue(os.path.lexists(os.path.join(out['retired_to'], *part)), part)
+        self.assertIn('BEGIN CERTIFICATE',
+                      self.read(os.path.join(out['retired_to'], 'live', 'brain-jar.com', 'fullchain.pem')))
+
+    @FIX_ONLY_RETIRE
+    def test_second_retirement_of_the_same_name_does_not_collide(self):
+        make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
+        a = hm.retire_lineage_tree('x.example', stamp='S')
+        make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
+        b = hm.retire_lineage_tree('x.example', stamp='S')
+        self.assertNotEqual(a['retired_to'], b['retired_to'])
+        self.assertTrue(os.path.isdir(os.path.join(a['retired_to'], 'archive', 'x.example')))
+
+    @FIX_ONLY_RETIRE
+    def test_certbot_can_issue_the_name_again_after_retirement(self):
+        """Against certbot's own storage code (skipped where certbot is not
+        importable, e.g. the image's app python)."""
+        # Import AND initialisation are both guarded: a half-installed or
+        # version-skewed certbot can fail with far more than ImportError (a
+        # missing transitive dependency, a pkg_resources/entry-point error, a
+        # moved internal). Any of those means "cannot run this check here",
+        # not "retirement is broken". new_lineage() itself is NOT inside the
+        # guard - that is the assertion.
+        try:
+            import types
+            from certbot._internal import storage
+            real_rv = storage.relevant_values
+            new_lineage = storage.RenewableCert.new_lineage
+            cfg = types.SimpleNamespace(
+                config_dir=self.le, renewal_configs_dir=os.path.join(self.le, 'renewal'),
+                default_archive_dir=os.path.join(self.le, 'archive'),
+                live_dir=os.path.join(self.le, 'live'))
+        except Exception as e:  # noqa: BLE001 - deliberately broad, see above
+            self.skipTest(f'certbot unusable from this python ({type(e).__name__}: {e})')
+        storage.relevant_values = lambda c: {'server': 'https://acme.invalid/directory'}
+        self.addCleanup(setattr, storage, 'relevant_values', real_rv)
+        cert, key = san('new_bundle.crt').encode(), san('new_bundle.key').encode()
+
+        first = new_lineage('brain-jar.com', cert, key, cert, cfg)
+        self.assertEqual('brain-jar.com', first.lineagename)
+        hm.retire_lineage_tree('brain-jar.com')
+        again = new_lineage('brain-jar.com', cert, key, cert, cfg)
+        self.assertEqual('brain-jar.com', again.lineagename,
+                         'certbot could not re-create the lineage under the same name')
+        self.assertTrue(os.path.getsize(os.path.join(self.le, 'renewal', 'brain-jar.com.conf')) > 0)
+
+class TestExpiredCertificateIsNeverPublished(CertPublishTestCase):
+
+    @FIX_ONLY_RETIRE
+    @NEEDS_OPENSSL
+    def test_publisher_refuses_an_expired_leaf(self):
+        src = os.path.join(self.tmp, 'expired')
+        cert = self.write(os.path.join(src, 'fullchain.pem'), san('expired_old.crt'))
+        key = self.write(os.path.join(src, 'privkey.pem'), san('expired_old.key'))
+        live = self.publish_live_bundle('brain-jar.com')
+        with self.assertRaises(hm.CertificatePublishError) as ctx:
+            hm.publish_pem_bundle(live, [cert, key])
+        self.assertIn('expired', str(ctx.exception))
+        self.assertEqual(PREVIOUS_BUNDLE, self.read(live))
+
+
 class TestBundleValidation(CertPublishTestCase):
     """What may and may not be published."""
 
@@ -894,6 +1279,184 @@ class TestBundleValidation(CertPublishTestCase):
                          'a private key must never be group/world writable')
 
 
+class TestSharedCertificateSurvivesDomainRemoval(CertPublishTestCase):
+    """Bug 5: DELETE /api/domain unlinked a PEM other live sites were served from.
+
+    `domains.domain` is UNIQUE; `domains.ssl_cert_path` is not, and nothing
+    ever made it so. request_ssl_bundle() issues one SAN certificate and points
+    every included name's row at the same /etc/haproxy/certs/<primary>.pem, so
+    sharing is not an edge case - it is the normal shape of the table. Measured
+    on production the day this was written: 39 of 71 distinct cert paths on one
+    host were referenced by more than one domain row (118 of 150 SSL-enabled
+    rows), and one shared path was
+    /etc/haproxy/certs/threeworldsoneheart.org.pem, referenced by the live
+    apex, its www, and a mail.* alias.
+
+    remove_domain() did, unconditionally:
+
+        os.remove(ssl_cert_path)
+        certbot delete --cert-name <domain>
+
+    Removing the mail.* alias would therefore have deleted the PEM the apex was
+    serving on, and (had the alias been the bundle primary) the lineage behind
+    it - HTTPS down for every other name in the bundle, with no local copy and
+    only a fresh, rate-limited ACME order to recover from.
+
+    These tests assert file-system and certbot-invocation outcomes, not which
+    branch was taken.
+    """
+
+    BUNDLE = '/etc/haproxy/certs'  # documentation only; SSL_CERTS_DIR is stubbed
+
+    def _cert_for(self, primary):
+        """Publish a bundle for `primary` and return its path."""
+        return self.publish_live_bundle(primary)
+
+    def remove(self, domain):
+        return self.client.delete('/api/domain', json={'domain': domain})
+
+    # -- last reference: the cleanup must still happen --------------------
+
+    def test_last_reference_removal_unlinks_the_pem(self):
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_example', ssl_cert_path=cert)
+
+        resp = self.remove('example.com')
+
+        self.assertEqual(200, resp.status_code, resp.data)
+        self.assertFalse(
+            os.path.exists(cert),
+            'nothing else referenced this bundle - it must be cleaned up, or '
+            'the crt directory accumulates certs for domains that are gone')
+
+    def test_last_reference_removal_deletes_the_lineage(self):
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_example', ssl_cert_path=cert)
+
+        self.remove('example.com')
+
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes(),
+            'the last name on a lineage went away - the lineage should go too')
+
+    def test_lineage_deleted_is_the_bundles_not_the_domains_own_name(self):
+        """Removing a SAN member must target the lineage that issued the file."""
+        cert = self._cert_for('example.com')
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('www.example.com')
+
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes(),
+            'the lineage is named after the bundle primary (--cert-name), not '
+            'after whichever SAN happened to be removed last')
+
+    # -- shared reference: nothing may be destroyed -----------------------
+
+    def test_shared_pem_survives_removal_of_one_name(self):
+        cert = self._cert_for('example.com')
+        before = self.read(cert)
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        resp = self.remove('www.example.com')
+
+        self.assertEqual(200, resp.status_code, resp.data)
+        self.assertTrue(
+            os.path.exists(cert),
+            'example.com is still configured and still served from this file')
+        self.assertEqual(before, self.read(cert),
+                         'the surviving bundle must be byte-for-byte intact')
+        self.assertTrue(self.edge_would_start(),
+                        'the edge must still load the crt directory')
+
+    def test_shared_pem_survives_removal_of_the_bundle_primary(self):
+        """The worst shape: the name being removed IS the lineage/file name."""
+        cert = self._cert_for('example.com')
+        before = self.read(cert)
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('example.com')
+
+        self.assertTrue(
+            os.path.exists(cert),
+            'www.example.com is still configured and is served from this exact '
+            'file - removing the apex must not unlink it')
+        self.assertEqual(before, self.read(cert))
+        self.assertTrue(self.edge_would_start())
+
+    def test_shared_lineage_is_not_certbot_deleted(self):
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('example.com')
+
+        self.assertEqual(
+            [], self.certbot_deletes(),
+            'certbot delete destroys archive, live and renewal config for a '
+            'lineage that is still renewing the certificate www.example.com '
+            'is served with')
+
+    def test_production_shape_mail_alias_removal(self):
+        """The exact row that stopped a production cleanup.
+
+        mail.threeworldsoneheart.org carried
+        ssl_cert_path=/etc/haproxy/certs/threeworldsoneheart.org.pem - the live
+        PEM of a different, serving site.
+        """
+        cert = self._cert_for('threeworldsoneheart.org')
+        before = self.read(cert)
+        self.add_domain('threeworldsoneheart.org', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.threeworldsoneheart.org', 'be_www', ssl_cert_path=cert)
+        self.add_domain('mail.threeworldsoneheart.org', 'be_mail', ssl_cert_path=cert)
+
+        self.remove('mail.threeworldsoneheart.org')
+
+        self.assertTrue(os.path.exists(cert),
+                        'two sites are still served from this bundle')
+        self.assertEqual(before, self.read(cert))
+        self.assertEqual([], self.certbot_deletes())
+        self.assertTrue(self.edge_would_start())
+
+    def test_removing_every_name_eventually_cleans_up(self):
+        """The guard defers cleanup, it does not cancel it."""
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('www.example.com')
+        self.assertTrue(os.path.exists(cert), 'apex still needs it')
+
+        self.remove('example.com')
+        self.assertFalse(os.path.exists(cert),
+                         'the last reference is gone - now it may be removed')
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes())
+
+    def test_unrelated_domains_certificate_is_untouched(self):
+        """Sharing is by exact path; different bundles stay independent."""
+        mine = self._cert_for('example.com')
+        theirs = self._cert_for('other.example')
+        theirs_before = self.read(theirs)
+        self.add_domain('example.com', 'be_mine', ssl_cert_path=mine)
+        self.add_domain('other.example', 'be_theirs', ssl_cert_path=theirs)
+
+        self.remove('example.com')
+
+        self.assertFalse(os.path.exists(mine))
+        self.assertTrue(os.path.exists(theirs),
+                        'a different bundle must not be collateral damage')
+        self.assertEqual(theirs_before, self.read(theirs))
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes())
+
+
 @unittest.skipIf(TESTING_FOREIGN_TREE,
                  'HAPROXY_MANAGER_DIR points at another tree')
 class TestPublisherApiIsPresent(unittest.TestCase):

scripts/test-fixtures/san-certs/dead_old.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC9DCCAdygAwIBAgICEAMwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMZGVh
+ZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
+EwYDVQQDDAxkZWFkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQCumL3T27a3D6fZbIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnO
+abvnQNxEVqyYtTJOn9taitVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BU
+SpW96rK40KVftzzDEUnH+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0
+seOIdyP/n3T2cX1wsCa/F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9b
+PBd2cBaBwTe/3TICu0RnvvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2
+wzG4zlKU31fdLvQOzMGH1W5H91sBAgMBAAGjSDBGMCUGA1UdEQQeMByCDGRlYWQu
+ZXhhbXBsZYIMYnJhaW5qYXIubmV0MB0GA1UdDgQWBBSPDJiv0Z2YpuloexOfvpIQ
+UrWufTANBgkqhkiG9w0BAQsFAAOCAQEAJwBtN3Qw41XvYuSa8Mjn869QOTKEgkaR
+EuNQ8WXVBOTRJdxaLHDunGO4ExjHHswCWzXkefYOoIClAU9qLnk9M667BSNxOcz0
+VcG/DUMd0tO8X4Zm0zOMjt4VC7whgj7MdYtYVg2+Jhb4sMSx6OqmCUAD4X6ULSTn
+lZK2holEC/H3bKt8Q6cecgeubgWojP1bWmWK7jQL5NRIzxQ0KzFxgAf3t2Z05nX8
+loABkBiR7vDthkQseUW6RQMayDyzhVqORLt+gZ9D5T3hT1M7VaDc4Twr7UbWPFcR
+GuQ7RwKplPIIbPe7+c91pm+agdSDjnV5HmKcAMCwf7bWGQ9/Xc0BFg==
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/dead_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCumL3T27a3D6fZ
+bIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnOabvnQNxEVqyYtTJOn9ta
+itVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BUSpW96rK40KVftzzDEUnH
++3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0seOIdyP/n3T2cX1wsCa/
+F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9bPBd2cBaBwTe/3TICu0Rn
+vvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2wzG4zlKU31fdLvQOzMGH
+1W5H91sBAgMBAAECggEADUvYBFLn6gPWSZj55nxOqR+2Lmtwh04omgbu2fpWhPTE
+ujgKd/23srnnMZXnR6qZQQIPGwJiOPnJuRjiu/PMZSj8j54bbwfA6XmbH9SCIzm0
+la50UGBSkw0T50jBI76vICMm6PBkf0rT1oKN9VOd/qlkJoCUiFcGCh177Fbt07oY
+R3iAKu7/+yEANX0iEcxbO330TPW3oI0ah5OryfKVoNvxpF8NUfV9pWY66APkOYMJ
+zsYvIMvzV/KIgfrtyxJjSjgd7Ud8uictYG1BrZBXpIuGxuxyKUJ7OXOvMMVNnEME
+2uzk/aL75UjpurJJU0LCywdUOGwRqS2q4PYQiTeDLQKBgQDjdywH7RODoYeH05Rd
+sa2MkNx0Day1i3A4IBds55mzn8jxpT55HlqvHWhNA30kEw01H8+6sB7iMX04xQtS
+u3N2opR1VlCvMJhGyRUYUtfu4NnAi/HSu7mCOFGAwaLclI7BAI2or2UrclrlUhGb
+CYIBKfEr6Tn6b6AffWaOruCbtQKBgQDEf70R922fnJK+//QuiYoaESwJd3/k10yJ
+nH0ZAgbnFcNLT9YVFxHvlt4RHGIotaBjfN+FKMn045jeekgeVdX+uF8+7FwEazAH
+zOfxswOeKd8EOooqCD7n7cW3Ma6ev8oWmXwJsz2coRW7Z5zCGQWSzCEDB4yFU9Mt
+0z6aN9SJnQKBgQCvBZSIqL+6KgD53zArA7sRDKnEVn1M13TKis7bij02/DyGyuoO
+EfAfDBakI4oXxt0Dwm15LWkmDIRYuS0kRH3si96FhpvgoRiaZsDnjV7/PF3wjP3O
+c4mjDCiSVOw2o8xP71gQspy62GveRc6sJ52zvY0Hh5aH/3oKoNfX8/F5fQKBgALQ
+vbnDLMvrp2KbMjl0eQkUk+fy0V9LAmq6VkSEderbsfejRjL5KKWhKL9gA7wHliu2
+v1EeG36pwCf34o6ktjM3Gw5AvrTFgGOsOOqKoROkD/D0kcQ2aAhUrM+KI7FcmB2o
+x1EwQ0K6QCB2RJW3XBAVYqezdPz4DX+LlksJX781AoGBAISGOer3s4+oRjO/2VuT
+hnMQrET3A0GTol8cgYL2N9wfx7jy5bhSfiJIX/dkh5IrPGYoBdWM87gcUgNQ8AZ7
+il5dVeG80oc7KFU/QltZJu9VS32vwXnNsnrXupQ5Ee0ztTLciTMWU5jxvB0ZJHtK
+StgYepr4IPDUaObqrPx4Jvfm
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/expired_old.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDGjCCAgKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwGDEWMBQGA1UEAwwNYnJh
+aW4tamFyLmNvbTAeFw0yMDAxMDEwMDAwMDBaFw0yMDA0MDEwMDAwMDBaMBgxFjAU
+BgNVBAMMDWJyYWluLWphci5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQC9SJUMsK5HhVEATu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9s
+i5KzgYp0K2z899wQAKuLzZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ
+7jax2QWFMQVhyy6M5QTCHUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyN
+N8eolukhvFxQNU3E7oo1jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSy
+UEc+Gje6dJLZ9CdJtSQSNGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8
+FHC7atJOe5EoaekaAXw5Tq+yB+ODAgMBAAGjbjBsMEsGA1UdEQREMEKCDWJyYWlu
+LWphci5jb22CEXd3dy5icmFpbi1qYXIuY29tggxicmFpbmphci5uZXSCEHd3dy5i
+cmFpbmphci5uZXQwHQYDVR0OBBYEFAWqbBh2PDUvqQzZoyh960cpuhcOMA0GCSqG
+SIb3DQEBCwUAA4IBAQCaovtrXl56YcFlXWMrW/XZBe2/fXI14860qyiR5rb07lS6
+CbKOsnRyQdxZ3cdfRwTN1UMmz5IPOZ6fhdo3IPaS2Wae1OdeupRaRWr1SAJtpVZ3
+WWugDaumsWV5eWVxVTf93yDEkMERWtTGMviCKBeeHjzHIFiD/EjZznE87V5DYwuf
+guOd1F4wXCMRFu8q7sBdPXjQ8rZMlQT+dQXzMST9Ee5PXiInmCAA5nT8vMInAwd2
+rP6FxuHJB75ekYzs57CJ82BKKboOCESjUbm54wmVV5pBL305/EjxGf+VovNlRktg
+t1znolntIIofE25UCbVNPdrtuDkZgKT3Rv8tCdTP
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/expired_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC9SJUMsK5HhVEA
+Tu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9si5KzgYp0K2z899wQAKuL
+zZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ7jax2QWFMQVhyy6M5QTC
+HUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyNN8eolukhvFxQNU3E7oo1
+jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSyUEc+Gje6dJLZ9CdJtSQS
+NGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8FHC7atJOe5EoaekaAXw5
+Tq+yB+ODAgMBAAECggEABAeOCL71eYH4jKRMOBbXu98483u1CxnTEa8klnw9ZVES
+JctJjY96B/EMwv+g/V+gxhgAkiK9VWfg6bEplOVDWgR9fRBoP57mVkOv3QPn5fcR
+xlt9LQQZ+pULQRZoUAFEkVEkV1sfIBJmDLMufOnbAHCKa9+hVlrlCqu0lJgM8put
+2bkpSAqwroHVfmD0SmKFAZggBhFwXFJzwmUf3WL+N25f4RZpJsX17FOJ216mlhQ5
+edje6dcDpmko2VQtGJXYr8Ge02ZYeRyKGcTsgUZlarB4xUZq1pcnUUSy0Ez7LVbk
+tKjPcUpZLlnAA/a2pDgHmwN1+fv/X+Wwv/vZaQMBTQKBgQDuZFyfDRSHCssHtQ4+
+Ezxg4WEjwXkq58WxL0zTtLTcquZfQZ1VXgLwAylVwver0I4/UgCfW/AYY+2eLrba
+FhuYOosd/3rsUQqf9y+TkSkhe1uVS2/1zjzC1G/5coleLddjzd+YBpEtVxfxDRbA
+rVePY3GxkJiTrjAg5OETk7GYpwKBgQDLQ6bPhaocJPQmDi3hKyW8zPJ8wAJNyI0c
+RWt9gjeidjmMvwCKtJbBDcV5V0oQ5SB2M5E49mT58B62totAD7FS18c4a4133haV
+a1pLBX0U524eQgDzMJ6YA6N9bHdXOH4FZd+Pw90bHetbT1HcKpOdXv4SF/XITju5
+Ro5xiymdxQKBgBB4fq2+iZyK8lwl9kXzrkhDSiLeSzDbspz7V6OI6LYWWLRcJDKx
+vCI5FbIHNnV4g1t8nu5ju8LzpIc5UwjeXim2YcZA0I1jTAihW3gmn2Dg1QMLo0Co
+bypG4zWo5DTNlgQfRUOLsJwhWFYfQWMuls4hYMmT4/bQIiMjrSldGvghAoGALpaB
+7DyZobvCjdpgH6TREHX0+izUGlnmJd2vZmPrNnxccmrwx9neV789TLN6EWdCynTF
+S0YGHNcxYvVx8yvlm7RDQCY+lYO8r2CcoGdQpPLqZVsBdrvlDbh/x2aN79rN35Ta
+6cudqPJxmEsU2x6TGNXF7AW72r47qBqI2XNpBXUCgYAFQdHRRxa2E5rgV0WkL7PF
+vXok3isAzpnKAsb5oxPZbhIdLo5EPUGCslsTE1Na8NBdUSoyjhgkMELEDx/ceGL7
+IIlNrc/4j71+IahljzaX/D5b5qaW+RnoTVAlbMRR7L/ecbkoovhQR9d37QdoiD/t
+B9si92+XB4HYDAjXFxTK3w==
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/new_bundle.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC+DCCAeCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMYnJh
+aW5qYXIubmV0MCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
+EwYDVQQDDAxicmFpbmphci5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQCyhL5YcFhPEekc/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdb
+l4twhlkcyvyaEbcHIgLEgz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgU
+QcAAvdAyP77jDF6lX11oCjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zz
+XRVp4l2WyqEUJmKxcpppAnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU
+0v78QfI+ueCw261UWZ+iXwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQ
+dFOfKjH0QGInyByji9eAaXlmvCKtAgMBAAGjTDBKMCkGA1UdEQQiMCCCDGJyYWlu
+amFyLm5ldIIQd3d3LmJyYWluamFyLm5ldDAdBgNVHQ4EFgQUdDxfZKX96j61mW7O
+NiQpN4T+sZ4wDQYJKoZIhvcNAQELBQADggEBAJE1lP62caByZT7GkKtJLfOj84Ih
+MeP751w2gdU8ZfiO8GxXXiR3ZjYzbJgsEi9LiNT9jqHeBf3pv/mJLjPVZ+U557ws
+Gq3V5A3uKlRKGcejXlHaWnExhWqLyIaLTeLKstbJyiB/ja4yeTL2TYcOvvlu6weh
+ERqwx0p6r7LSQ3nRfTdjEhNq3FVUmqQ8neBP/vXLp2hKr2ACZndGyQOiv0oZWzud
+part6RMd8gRNq1Tctrl0YqRYoLRIddBrnTtUtac4FLqxLSBNc03/I9ZgF56Yf+Sl
+M1ikmyDJ2ZYJFU6v42Og4/8o3fpq0ttP6pKPi8fInGaFQ+KXO1NWFf1ogD0=
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/new_bundle.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCyhL5YcFhPEekc
+/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdbl4twhlkcyvyaEbcHIgLE
+gz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgUQcAAvdAyP77jDF6lX11o
+CjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zzXRVp4l2WyqEUJmKxcppp
+AnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU0v78QfI+ueCw261UWZ+i
+XwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQdFOfKjH0QGInyByji9eA
+aXlmvCKtAgMBAAECggEAEQi8lvjFXGojmco3H7PVD7SZftLOFidVh9Fo8/JAe+2Z
+LQM/hIjZ1i7sNverbQXMzAQMfXESS2Jx7Clv0mZ0aRgE0hq7wXrabyHfFUPOtqDa
+6cPHsz8YvwAOmcQ6W3fBPYI6pzSfdV03rv1GGeCC7ETI3mIupwgUbANDuiDcVY1q
++VuK/hDzPJ5KslDHUNR42MgWacIftq99fV9jN0zQQw1EcyO8QwpXzTSEbdyiAkT+
+n/tR9He2MYhgRhjyWPMaxmY6XQpv1h5ASq1yT1kI2oNTaRT3xvd40sLkbfwpZnxi
+HnqzCkPptEpWeCLrme+r71+Hxu/+Bph7tPtcuK7jgQKBgQD8XCoPRiWbgCl6kM2G
+E3GeM6fftTieTIs7sFVte7wutIKlsec4vizc94ipGaoT7yGEG0T8g9+5hAMbebc2
+zzhxvJMB2s18Z/He+Ujv1L1AhBBNVY3zCAxfdq8/n5MGXppEdVRo9QOQndoBfuAV
+Qw2Vp4bcy728n4T7sBugW0Ad7QKBgQC1F+utuiF8XI2yx0mX0+Fbty6o+odyM0eb
+G/QQDlcGpY0K7V3WAFMESNfytig8OhidzRfWn8lNoqhd80SJfCuiYjiGJ8tjBzcp
++Vqrd/KAxJySJ3NJ2MwyAp7W3l3HAEZLGVYVCankoa4O3qR8YqaIZwc2nk5+8KnG
+nBYFlSF/wQKBgQCqaRYh0E2O7wGTeXq/ohukJ9+hqf88rhWLTeyyDM8rfsuGVlb8
+MzeaUzEed20pHt60FbIFDowh+yoaAJwez9mevCdNJlKCQrLKze1Zvn+5tDRJ9I7x
+YSrxv/Rhc0FWGMX1moFQirtIib9wKRvqX6gpucHREWfjmpizWN2e8F+glQKBgAOp
+/jy/nVnUzZEmcxi9Urv2Ut1cZE0tOG+s+x0wDRa3Vsk30EQ6MXPOvX/Nq7a8Pi9D
+LQ5MoVgOyHVOLL7TYlagwiAEwhq41KbmwqzhemickcZrQevmlQsV/ny+dwtFuCZg
+1tsKHQPhOZJXcuCJZnEwFHdIcLIk0GqSUee/ArSBAoGALOD+Opk+0j4ObNcqtrpP
+UcEYqhOuAscbxLoUv53wnOvc7XzMG8JCK1y5LthB2Xanr1JSlheMRrvGxC+WlDmc
+5v0Nz71cq5QQ1p8AXnosLRjuy/+edMvGIxxwFMEJBO2ra2Jizpdwd34bjpQ42+CQ
+6pVwFQX0oet8SI5xWUWx7NU=
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/shared_old.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDGjCCAgKgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwHTEbMBkGA1UEAwwSc2hh
+cmVkLW9sZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAw
+WjAdMRswGQYDVQQDDBJzaGFyZWQtb2xkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEB
+AQUAA4IBDwAwggEKAoIBAQCBg7iYuvjhIG7nlFeja5Lyv2t2fdtNvxzUib7ygejC
++WQvNdH4k12q8vCXNYuadbGAceSejG4bazpjo5/ym2KFZPwiSZoqKUugi9lCoKuL
+2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMHYFwURjARRAku3a/GJUHQR0C572mW
+mt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64fGeYSOaYOHEOp/VqIRlzsy7eGQh3
+xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2JbqFxvijCLnBz5WsrS/S/VWOELNEfGr
+84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLfWuSgp5hRAgMBAAGjYjBgMD8GA1Ud
+EQQ4MDaCEnNoYXJlZC1vbGQuZXhhbXBsZYIMYnJhaW5qYXIubmV0ghJvdGhlci1z
+aXRlLmV4YW1wbGUwHQYDVR0OBBYEFP+AdZ9MkBmFjAQTgLnWZxbJtnvkMA0GCSqG
+SIb3DQEBCwUAA4IBAQA32Xl/1IrOyESd2UQcWp/fuHvkivLgamJrheKvKP0Y0vMV
+atRu0D0GhqMOr79OT60H/H1FFW7bWQ+UPcZpnuXec40/PcrHxwaSoiLoIUVWhB4t
+BOhtTDReX29gEjnB7OqBCJu4ub3HnpFTflUZd7n5A4EcetBsowc5EZ9UAtfvkgGD
+nK9ONvNVGurk/V8xgtMJ877ZK1fn+UtgvIWp/F1Gc/taLhe+oCZOPkOqvvqUnS8q
+QyIhqmMWRUkBP+jwRA4TP4pj7OH38VY/Ae408Xe4Y97GF7NbOjAL0bkX/bHkLnSG
+WzoGO0TK7pCemeFQ+Ghic8Yl0eDynoP7rZIhCKS0
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/shared_old.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCBg7iYuvjhIG7n
+lFeja5Lyv2t2fdtNvxzUib7ygejC+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpj
+o5/ym2KFZPwiSZoqKUugi9lCoKuL2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMH
+YFwURjARRAku3a/GJUHQR0C572mWmt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64
+fGeYSOaYOHEOp/VqIRlzsy7eGQh3xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2Jbq
+FxvijCLnBz5WsrS/S/VWOELNEfGr84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLf
+WuSgp5hRAgMBAAECggEACB1tvTwciKvp5wliGF45feDv5c5q+vXE462h0OYXjyfl
+1sNuMuoLjhYyFvdcQO2rlTBjB8Ald1hXoeidsVYlF9XvqmJRjjS6MAazMpEzTnba
+66V6U1j5gOQ6pi3SjfBJGFYo8L+cy/NqUIBWz0+6hp/BsjW3uW2aMGer2Y/zs2Oi
+s0kUNqqUWQpGpEDxbDnUAzto4OOcTOO4A5NB5Oe2ifzFiXCWtTNqFIC0ae9tiqGy
+m0sggF2hzmlH1WC54rbv7xujYT6tOAEi1OKp/Cpj8XVLbjaqju5avRl1t2nDRQri
+9vnOI9P4bnJKbIYzwJVyy8B1M5iK+W2K7wUQ4ZMetQKBgQC2zwtywQ4jUduLWf4f
+6HLUIvmcHdJI0YBHS1+4fKc7tsakRo/0FHxNdUlBhqkYTyp57h6/R1T2gG70Kla1
+MarGIgyPIlUXsRojlUcNyKXqSVs+kejftIClb+VDXWrpKtGtyxuESsMjaJBsBBAN
+hg3yUx2NL2ct4nnlpl1iq3pH9wKBgQC1XkrjntuzjVYlFeg2ih5va3PUD2qmH9Ej
+leNXTGhzZL1eQWz4GaRCgzJH8q41pXBxcXQ8hMXZCWdW5GHBYfVqjTRKMUPdI6FI
+ypwMbbTBfQfw1kz5VHfLH3romxikNQXeorZkIZqv7/aLPsxLY78XdW8nSDIsT9ZP
+mh7It+Tf9wKBgHT7VSYf6D4t2cH42qfshd6+auC+LJywg9gboatbCWEq+z6QcPoO
+SeT/Utt9oqONtW7FNCPPL812LopMtaPJnrCfhU3q+GzqxgFTO17UsWeVfbAyisTs
+qw4y5pEPLKT846TFPlyPwwPbm9+jUep1iGMzUMvSM8KnFL4ikcvcU6nNAoGBAJQI
+MVHV0HadzokYd19EnpDMgE6+goqy/21vIo6XY82DlPjAp+NccnCDSt/kmQJaVQuS
+0Y+vwxhhj4IhiBRiEwBz3hEMYtq1ASg7juCqDlW66mdNUqhSpDAbMqN2K8ejK3ZL
+EllqACrHzlBa4j5KXKcWB7JlP+ZhB4gTQDsNtXaDAoGBAJCktIdyM3b6UtM5OofB
+lzn6/2uT5u46ECBRybk/J+Y2gfcS2cFzkH179SUB4WgfR0h21FlLmjns5ZvHVk0p
+/U4aY3/J42JU6lCYrsyV5BBH+O0stSmrUG/4JdKRe6Mq/pB321RsWD06LVJrBDpi
+6NUPW03tNLc+Qe+cnQu0c7y+
+-----END PRIVATE KEY-----

scripts/test-fixtures/san-certs/unrelated.crt

@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC8zCCAdugAwIBAgICEAQwDQYJKoZIhvcNAQELBQAwHDEaMBgGA1UEAwwRdW5y
+ZWxhdGVkLmV4YW1wbGUwHhcNMjAwMTAxMDAwMDAwWhcNMjAwNDAxMDAwMDAwWjAc
+MRowGAYDVQQDDBF1bnJlbGF0ZWQuZXhhbXBsZTCCASIwDQYJKoZIhvcNAQEBBQAD
+ggEPADCCAQoCggEBALEpfqdRPBSrbzixAxD2WhsO1AOTUd/Gguf3zjDZ3TUmWJKb
+OhlFthtU6PJVhMdAqOk+hiz9Coo/UngWjykyNMOj1YIpq3M4/P8wBTNxkJd/z2x0
+iAQBKvWlB6hxf9c1kenu/ea1iMIThA/uudtX3g8k39Asj/37+o070cdBJExWGnbG
+lz103FCO6kBzlJPGIVQwzKH6OKthTzi7Uxy5Fxc/ymS9gc43F0tyY6892Wyl8Y62
+a3r1PpZoWR0GDFxPIyCS7NP44T15DSL/2RBi4YnWXOC8QMvPnHbSKjoQ6GLQtGkO
+1NO4Uy3XbwDFanYXv6Wxv0oUmZMuY7CRTOMdxCMCAwEAAaM/MD0wHAYDVR0RBBUw
+E4IRdW5yZWxhdGVkLmV4YW1wbGUwHQYDVR0OBBYEFBh0L/5U5f94TylibBCgS9pD
+/VXkMA0GCSqGSIb3DQEBCwUAA4IBAQB5MLqNjTqIdyztjJsKPdYUkA27NpfdqqY6
+Yprv5N7U0slsGUqFIDrSuJ1mMMwTpt02IAFyCOWMUtze+/0PqM9k9TqxJH2cTODp
+0ET34Qh8NTo0B534EQUWhny+EzgNAzWK02wbpgm88+Mp/t//sHzhWzOm7uk1dlkq
++5ty/8OveWwAtqAWlDtcsRhkEjzynZKUCAF1extwUSuapFSm20mJ4ivt7Ff5MkTI
+fq8P+3XAxWTaBW8DzBIzYd1mwFHK2pcklLMRRTW3hukPxEcTivBgJdwsHMskM3UO
+/ejy5EHem5ohCMbwMybvUqLS4xdoqXV36Sj2I1zv0IjgO0wgdK1r
+-----END CERTIFICATE-----

scripts/test-fixtures/san-certs/unrelated.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCxKX6nUTwUq284
+sQMQ9lobDtQDk1HfxoLn984w2d01JliSmzoZRbYbVOjyVYTHQKjpPoYs/QqKP1J4
+Fo8pMjTDo9WCKatzOPz/MAUzcZCXf89sdIgEASr1pQeocX/XNZHp7v3mtYjCE4QP
+7rnbV94PJN/QLI/9+/qNO9HHQSRMVhp2xpc9dNxQjupAc5STxiFUMMyh+jirYU84
+u1McuRcXP8pkvYHONxdLcmOvPdlspfGOtmt69T6WaFkdBgxcTyMgkuzT+OE9eQ0i
+/9kQYuGJ1lzgvEDLz5x20io6EOhi0LRpDtTTuFMt128AxWp2F7+lsb9KFJmTLmOw
+kUzjHcQjAgMBAAECggEAJjBKo+iH6fC2beMCRTLFY5XDr+2etX0GFNi+8prT6zQT
+3a5mTLJROf0Ipcw3bfgLMRQQOjxlfD2AGnDKdVfkNdPoH0G6OoErUBra1CcDwxSC
+8t7/vtU9nGvxhabJputeQ64HvyacjzFDw2dzWsvLCzqooCgq9dXOQYmGg8wP9gAx
+pxfKnd84h41C3pXERdufi6nJHPjZPRW+dKsQ1NMLmtgO5YQnTt3LfVOkZaRa7Izq
+ZFWZCBfdsNHHuXCYj3FtqYrqSxELuZAxJ62hxpizrfQdKixguGTObHhC4muIo9fR
+eIMCvrAzrs4Fs8nM/K/IPyDU9wPwINK6Z+sxJkhM+QKBgQDaL3Yt3T/LE6bTYyb0
+EjVcXoybnuAjFQAenCTIsBRJv21Yv2ddqjq6rOTD64hFuFWPMEOi1TJxIPsFFDMw
+NeAwF41fiSrtU8awXs0hTKRQeHjElSBckumOl1DrHu2oKuwWXLniViD62vMsWmpZ
+89KT2rgFQuMPvsD3pps8l6W2HwKBgQDP3eTwdVJTdRYlZonqd0+LHG7h715TT1OE
+QJOdN9vK5g/9GOCOGHoUxB3IxJVmvz57PrfWaD2rSZ6JMs4TcAY5a0PbJEpgqoJY
+kIm/EDnSrFxu2L8DNsAzuThfzXrwWc7PzajYJctYtEfvxCyfo/GdZahg2xaEU0Le
+dp+7fxtJfQKBgGVdLWtLYarH+3Yzl1JG71F1f5+A5SKHTCHsas5tt083xuY3Lh9+
+yYAWeMVotEunwuAeDR4mLCQQrW4Dn3BjZz8QghaDVFUj2afwsx4oTCIyhfDWU02O
+10DunaJOIR1Cq6IL/DPhzbqdY+IUM1qNAyS7tUiEdZbICUBEv9Qbo9oDAoGAORnY
+cSGZaumoGsAPNMRvqmQ2JYSO4c3RwGvI4RBeJV+LlthFn2Mj7h5tnCA8dkwkytBQ
+yeo6vpnrnf7B018bppxLuKBAekt8oP2biUvYxoGopp6N4kEhDVRdIxwcIHhbMoRp
+VG8VbBmiBUBbjcMIdVdeqLSme9UvclMdz/C7uzUCgYEAzD2dvoJSu9zfo3fACw1X
+te53Rvr6Kq6DTw0SS4UrAnxlhX9TwD9gymIrR9rxnC+UPwfMwWMdpYqY+Da4sF1G
+uwd+nyZ4iFNq+lbLii9rmJ58h03+uo4lUXWTrAik6qOF1ztIfYKWFrOyBQmxjWwo
+HnUinudalENFJYZRwSUAtXo=
+-----END PRIVATE KEY-----

scripts/test-gunicorn-socket-timeouts.py

@@ -0,0 +1,631 @@
+#!/usr/bin/env python3
+"""Regression tests: a silent peer must not be able to wedge the API's threads.
+
+Why this file exists
+--------------------
+On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool threads
+blocked in the kernel in tcp_recvmsg -> sk_wait_data, /health hung 40s+, and the
+container stayed "healthy". The app makes no in-process outbound socket reads;
+the blocked reads were gunicorn gthread reading the CLIENT socket, which it
+makes fully blocking with no timeout (TConn.init -> setblocking(True)).
+
+These tests run REAL gunicorn processes (no mocks) against tiny WSGI apps and
+pin:
+
+  * the failure mode itself, on the stock gthread worker (control);
+  * that haproxy_gunicorn_worker.TimeoutThreadWorker frees the thread when a
+    client stalls before / during the request line, mid-body, or on a reused
+    keep-alive connection;
+  * that a legitimately SLOW request (handling time > client timeout) is not
+    cut off -- the timeout bounds socket I/O, not request duration;
+  * the watchdog: a worker whose every thread is stuck is replaced;
+  * review round 1, at production shape (2 workers x 4 threads): 2050
+    connections that never send a byte are reaped (F1, stock gthread wedges
+    permanently), and slowloris drip connections are cut by the total header
+    deadline while /health stays responsive (F2);
+  * the timeout is logged as one line, not a traceback per scanner;
+  * haproxy_manager's socket.setdefaulttimeout: an in-process client call to a
+    stalled upstream (accepts, never replies) times out instead of hanging;
+  * why that default alone is not enough (setblocking(True) clears it).
+
+Running
+-------
+    python3 scripts/test-gunicorn-socket-timeouts.py
+
+stdlib unittest, like the other scripts/test-*.py; needs only the app's own
+requirements (gunicorn, Flask...). Runs in the Docker build gate. ~70s.
+"""
+
+import os
+import socket
+import subprocess
+import sys
+import tempfile
+import textwrap
+import threading
+import time
+import unittest
+import urllib.request
+
+REPO_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')))
+
+try:
+    import gunicorn  # noqa: F401
+except ImportError:
+    sys.stderr.write('SKIP: gunicorn not installed (pip install -r requirements.txt)\n')
+    raise SystemExit(77)
+
+WSGI_APP = textwrap.dedent('''\
+    import os, time
+
+    def app(environ, start_response):
+        path = environ.get('PATH_INFO', '/')
+        if path == '/sleep':
+            time.sleep(float(environ.get('QUERY_STRING') or 3))
+        elif path == '/hang':
+            time.sleep(3600)
+        elif path == '/body':
+            n = int(environ.get('CONTENT_LENGTH') or 0)
+            environ['wsgi.input'].read(n)
+        body = ('pid=%d' % os.getpid()).encode()
+        start_response('200 OK', [('Content-Type', 'text/plain'),
+                                  ('Content-Length', str(len(body)))])
+        return [body]
+''')
+
+
+def free_port():
+    s = socket.socket()
+    s.bind(('127.0.0.1', 0))
+    port = s.getsockname()[1]
+    s.close()
+    return port
+
+
+def http_get(port, path='/', timeout=5):
+    with urllib.request.urlopen('http://127.0.0.1:%d%s' % (port, path),
+                                timeout=timeout) as r:
+        return r.read().decode()
+
+
+class Gunicorn:
+    """A real gunicorn master on a free port, with its output captured."""
+
+    def __init__(self, worker_class, threads=1, env=None, workers=1, extra_ports=()):
+        self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
+        with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
+            fh.write(WSGI_APP)
+        self.port = free_port()
+        self.log_path = os.path.join(self.tmp, 'gunicorn.log')
+        e = dict(os.environ)
+        e.update(env or {})
+        self.log = open(self.log_path, 'w')
+        self.proc = subprocess.Popen(
+            [sys.executable, '-m', 'gunicorn',
+             '--bind', '127.0.0.1:%d' % self.port,
+             *sum((['--bind', '127.0.0.1:%d' % p] for p in extra_ports), []),
+             '--workers', str(workers), '--threads', str(threads),
+             '--worker-class', worker_class,
+             '--timeout', '120', '--graceful-timeout', '2',
+             '--log-level', 'info', '--error-logfile', '-',
+             '--pythonpath', '%s,%s' % (REPO_DIR, self.tmp),
+             'tapp:app'],
+            stdout=self.log, stderr=subprocess.STDOUT, env=e)
+        deadline = time.time() + 15
+        while time.time() < deadline:
+            try:
+                http_get(self.port, '/', timeout=1)
+                return
+            except Exception:
+                time.sleep(0.2)
+        self.stop()
+        raise RuntimeError('gunicorn did not come up:\n' + self.output())
+
+    def output(self):
+        self.log.flush()
+        with open(self.log_path) as fh:
+            return fh.read()
+
+    def stop(self):
+        if self.proc.poll() is None:
+            self.proc.terminate()
+            try:
+                self.proc.wait(5)
+            except subprocess.TimeoutExpired:
+                self.proc.kill()
+                self.proc.wait()
+        self.log.close()
+
+
+def stall_client(port, payload=b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03'):
+    """Connect and send a few bytes that are not a complete HTTP request (by
+    default the start of a TLS ClientHello -- what an HTTPS scanner sends to
+    this plain-HTTP port), then go silent and keep the connection open."""
+    s = socket.create_connection(('127.0.0.1', port))
+    if payload:
+        s.sendall(payload)
+    return s
+
+
+def wait_closed_by_server(sock, within):
+    """True if the server closed (EOF/RST) the connection within `within` s."""
+    sock.settimeout(within)
+    try:
+        while True:
+            data = sock.recv(4096)
+            if not data:
+                return True
+    except (ConnectionResetError, BrokenPipeError):
+        return True
+    except socket.timeout:
+        return False
+
+
+STOCK = 'gthread'
+FIXED = 'haproxy_gunicorn_worker.TimeoutThreadWorker'
+FAST = {'HAPROXY_MGR_CLIENT_TIMEOUT': '1'}
+
+
+class StockGthreadControl(unittest.TestCase):
+    """Proves the failure mode is real, so the fixed-worker tests mean something."""
+
+    def test_stock_gthread_is_wedged_by_one_silent_client(self):
+        g = Gunicorn(STOCK, threads=1)
+        self.addCleanup(g.stop)
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        time.sleep(0.3)
+        with self.assertRaises(Exception, msg='stock gthread should be wedged'):
+            http_get(g.port, '/', timeout=3)
+
+
+class TimeoutWorkerTests(unittest.TestCase):
+
+    def start(self, threads=1, env=None):
+        g = Gunicorn(FIXED, threads=threads, env=env or FAST)
+        self.addCleanup(g.stop)
+        return g
+
+    def test_silent_client_is_dropped_and_thread_freed(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        t0 = time.time()
+        # Only one thread: this request can only be served once the stalled
+        # connection has released it.
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertLess(time.time() - t0, 4)
+        self.assertTrue(wait_closed_by_server(s, 3),
+                        'server should have closed the stalled connection')
+
+    def test_connect_and_send_nothing_does_not_hold_a_thread(self):
+        # Never readable -> never handed to a pool thread (main loop only).
+        g = self.start()
+        s = stall_client(g.port, payload=b'')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
+
+    def test_half_written_headers(self):
+        g = self.start()
+        s = stall_client(g.port, payload=b'GET /health HTTP/1.1\r\nHost: x\r\n')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_body_shorter_than_content_length(self):
+        g = self.start()
+        s = stall_client(g.port, payload=(
+            b'POST /body HTTP/1.1\r\nHost: x\r\nContent-Length: 100\r\n\r\n0123456789'))
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+
+    def test_timeout_reapplied_on_keepalive_reuse(self):
+        g = self.start()
+        s = socket.create_connection(('127.0.0.1', g.port))
+        self.addCleanup(s.close)
+        s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
+        s.settimeout(3)
+        self.assertIn(b'200 OK', s.recv(4096))
+        # Same connection, now stall mid-request. gthread calls conn.init()
+        # (setblocking(True)) again for the reused connection.
+        s.sendall(b'GET / HTT')
+        time.sleep(0.3)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_slow_legit_request_is_not_cut_off(self):
+        # Handling takes 2.5s, client timeout is 1s: must still succeed.
+        g = self.start()
+        self.assertTrue(http_get(g.port, '/sleep?2.5', timeout=10).startswith('pid='))
+
+    def test_timeout_logged_as_one_line_not_traceback(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        http_get(g.port, '/', timeout=5)
+        time.sleep(0.3)
+        out = g.output()
+        self.assertIn('Closed client connection idle', out)
+        self.assertNotIn('Traceback', out)
+
+    def test_watchdog_replaces_fully_wedged_worker(self):
+        g = self.start(env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                            'HAPROXY_MGR_WEDGE_SECONDS': '2',
+                            'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.5'})
+        pid_before = http_get(g.port, '/')
+        # Wedge the only thread inside the app (not a socket read).
+        hung = threading.Thread(target=lambda: self._swallow(g.port), daemon=True)
+        hung.start()
+        time.sleep(0.5)
+        deadline = time.time() + 12
+        pid_after = None
+        while time.time() < deadline:
+            try:
+                pid_after = http_get(g.port, '/', timeout=2)
+                if pid_after != pid_before:
+                    break
+            except Exception:
+                pass
+            time.sleep(0.3)
+        self.assertIsNotNone(pid_after)
+        self.assertNotEqual(pid_before, pid_after, 'worker was not replaced')
+        out = g.output()
+        self.assertIn('WEDGED WORKER', out)
+        self.assertIn('tapp.py', out)  # the stuck thread's stack was dumped
+        self.assertIsNone(g.proc.poll(), 'master must keep running')
+
+    def test_watchdog_leaves_partially_busy_pool_alone(self):
+        # 2 threads, one hung forever: the pool still serves, so no restart.
+        g = self.start(threads=2, env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                                       'HAPROXY_MGR_WEDGE_SECONDS': '1',
+                                       'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.3'})
+        pid_before = http_get(g.port, '/')
+        threading.Thread(target=lambda: self._swallow(g.port), daemon=True).start()
+        time.sleep(3)
+        self.assertEqual(pid_before, http_get(g.port, '/', timeout=3))
+        self.assertNotIn('WEDGED WORKER', g.output())
+
+    @staticmethod
+    def _swallow(port):
+        try:
+            http_get(port, '/hang', timeout=60)
+        except Exception:
+            pass
+
+
+def raise_nofile(n):
+    try:
+        import resource
+        soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
+        if soft < n:
+            resource.setrlimit(resource.RLIMIT_NOFILE, (min(n, hard), hard))
+        return resource.getrlimit(resource.RLIMIT_NOFILE)[0]
+    except (ImportError, ValueError, OSError):
+        return None
+
+
+class ProductionShapeExhaustion(unittest.TestCase):
+    """Review round 1 (F1/F2), at the production shape: 2 workers x 4 threads,
+    default worker_connections=1000."""
+
+    def start(self, env):
+        g = Gunicorn(FIXED, threads=4, workers=2, env=env)
+        self.addCleanup(g.stop)
+        return g
+
+    def test_f1_2050_idle_conns_reaped_and_health_recovers(self):
+        # Stock gthread: 2x1000 never-readable conns fill every slot, the run
+        # loop stops polling, and the worker stays dead even after the
+        # attacker closes everything. Here they must be reaped after
+        # FIRST_BYTE_TIMEOUT -- even while the attacker still holds them open.
+        if (raise_nofile(4500) or 0) < 4300:
+            self.skipTest('RLIMIT_NOFILE too low for 2050 client sockets')
+        g = self.start({'HAPROXY_MGR_FIRST_BYTE_TIMEOUT': '3'})
+        socks = []
+        self.addCleanup(lambda: [x.close() for x in socks])
+        for _ in range(2050):
+            s = socket.socket()
+            s.settimeout(2)
+            s.connect(('127.0.0.1', g.port))
+            socks.append(s)
+        # Attacker keeps every socket open. Within FIRST_BYTE_TIMEOUT plus the
+        # time to drain the backlog, the API must answer again.
+        deadline = time.time() + 20
+        ok = None
+        while time.time() < deadline:
+            try:
+                ok = http_get(g.port, '/', timeout=3)
+                break
+            except Exception:
+                time.sleep(0.5)
+        self.assertIsNotNone(ok, '/health never recovered while idle conns were held')
+        held = time.time()
+        # The reaped sockets were closed by the server.
+        closed = sum(1 for x in socks[:1500] if wait_closed_by_server(x, 0.01))
+        self.assertGreater(closed, 1000, 'server did not close idle conns (%d)' % closed)
+        for x in socks:
+            x.close()
+        socks.clear()
+        time.sleep(1)
+        for _ in range(5):
+            self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
+        self.assertLess(held, time.time())
+
+    def test_f2_drip_connections_cut_and_health_responsive_throughout(self):
+        # 8 connections (= every thread at production shape) each drip one
+        # header byte per second -- far under the 20s per-recv timeout, so only
+        # the total HEADER_DEADLINE can cut them.
+        g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '3'})   # CLIENT_TIMEOUT stays 20
+        stop = threading.Event()
+        cut_after = []
+
+        def drip():
+            s = socket.create_connection(('127.0.0.1', g.port))
+            t0 = time.time()
+            req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 1000
+            try:
+                for i in range(len(req)):
+                    if stop.is_set():
+                        return
+                    s.send(req[i:i + 1])
+                    time.sleep(1.0)
+                    s.settimeout(0.01)
+                    try:
+                        if s.recv(1) == b'':
+                            raise ConnectionResetError
+                    except socket.timeout:
+                        pass
+            except OSError:
+                cut_after.append(time.time() - t0)
+            finally:
+                s.close()
+
+        drippers = [threading.Thread(target=drip, daemon=True) for _ in range(8)]
+        for t in drippers:
+            t.start()
+        self.addCleanup(stop.set)
+        time.sleep(0.5)
+        worst = 0.0
+        t_end = time.time() + 12
+        while time.time() < t_end:
+            t0 = time.time()
+            self.assertTrue(http_get(g.port, '/', timeout=6).startswith('pid='))
+            worst = max(worst, time.time() - t0)
+            time.sleep(0.5)
+        stop.set()
+        self.assertEqual(len(cut_after), 8, 'not every drip connection was cut: %r' % cut_after)
+        # The deadline is armed when a THREAD picks the connection up, not at
+        # accept: a request queued behind busy threads (e.g. legit certbot
+        # runs) must not be cut for waiting. So a drip conn that was queued
+        # (the 2 workers rarely split 8 conns 4/4) is cut at up to about
+        # 2 x deadline; one that got a thread at once within ~deadline + 1s.
+        self.assertLessEqual(min(cut_after), 3 + 2, cut_after)
+        self.assertTrue(all(c < 2 * 3 + 2 for c in cut_after), cut_after)
+        self.assertLess(worst, 5.0)
+        self.assertIn('HAPROXY_MGR_HEADER_DEADLINE', g.output())
+
+    def test_f2_slow_but_complete_headers_within_deadline_are_served(self):
+        g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '4'})
+        s = socket.create_connection(('127.0.0.1', g.port))
+        self.addCleanup(s.close)
+        for part in (b'GET / HTTP/1.1\r\n', b'Host: x\r\n', b'\r\n'):
+            s.sendall(part)
+            time.sleep(1)
+        s.settimeout(5)
+        self.assertIn(b'200 OK', s.recv(4096))
+
+
+def refused(port, wait=2.0):
+    """True if the server closes a fresh connection without answering a
+    complete request (peer filter)."""
+    s = socket.create_connection(('127.0.0.1', port), timeout=wait)
+    try:
+        try:
+            s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
+        except OSError:
+            return True
+        s.settimeout(wait)
+        try:
+            return s.recv(4096) == b''
+        except (ConnectionResetError, BrokenPipeError):
+            return True
+        except socket.timeout:
+            return False
+    finally:
+        s.close()
+
+
+class PeerFilter(unittest.TestCase):
+    """Re-review R1: tenant containers share client-net with haproxy-manager,
+    so the API port must refuse peers that are not loopback / the host path."""
+
+    @classmethod
+    def setUpClass(cls):
+        sys.path.insert(0, REPO_DIR)
+        import haproxy_gunicorn_worker as w
+        cls.w = w
+
+    def test_allowlist_parsing_and_matching(self):
+        nets, problems = self.w.parse_allowed_peers('127.0.0.1,::1,gateway', gateways=['172.18.0.1'])
+        self.assertEqual(problems, [])
+        ok = self.w.peer_allowed
+        self.assertTrue(ok('127.0.0.1', nets))
+        self.assertTrue(ok('::1', nets))
+        self.assertTrue(ok('::ffff:127.0.0.1', nets))
+        self.assertTrue(ok('172.18.0.1', nets))           # docker-proxy / host
+        self.assertFalse(ok('172.18.0.7', nets))          # a tenant container
+        self.assertFalse(ok('127.0.0.2', nets))
+        self.assertFalse(ok('', nets))
+        self.assertFalse(ok('garbage', nets))
+
+    def test_bad_entries_fail_closed_and_are_reported(self):
+        nets, problems = self.w.parse_allowed_peers('nonsense,gateway', gateways=[])
+        self.assertEqual(nets, [])
+        self.assertEqual(len(problems), 2, problems)
+        nets, _ = self.w.parse_allowed_peers('10.0.0.0/8')
+        self.assertTrue(self.w.peer_allowed('10.1.2.3', nets))
+
+    def test_default_gateway_parsed_from_route_table(self):
+        fn = os.path.join(tempfile.mkdtemp(), 'route')
+        with open(fn, 'w') as fh:
+            fh.write('Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\n'
+                     'eth0\t00000000\t010012AC\t0003\t0\t0\t0\t00000000\n'
+                     'eth0\t000012AC\t00000000\t0001\t0\t0\t0\t0000FFFF\n')
+        self.assertEqual(self.w._default_gateways(fn), ['172.18.0.1'])
+
+    def test_filtered_port_refuses_unlisted_peer_other_port_unaffected(self):
+        other = free_port()
+        g = Gunicorn(FIXED, threads=1, extra_ports=(other,), env={
+            'HAPROXY_MGR_PEER_FILTER_PORTS': str(other),
+            'HAPROXY_MGR_API_ALLOWED_PEERS': '10.255.255.1'})
+        self.addCleanup(g.stop)
+        # g.port (unfiltered) works; `other` refuses 127.0.0.1 (not listed).
+        self.assertTrue(http_get(g.port, '/').startswith('pid='))
+        for _ in range(5):
+            self.assertTrue(refused(other), 'filtered port answered an unlisted peer')
+        self.assertTrue(http_get(g.port, '/').startswith('pid='))
+        out = g.output()
+        self.assertIn('Refused 1 connection(s) to the management API', out)  # first one, immediately
+        self.assertEqual(out.count('Refused '), 1, 'reject logging is not rate-limited')
+
+    def test_filtered_port_serves_listed_peer(self):
+        port = free_port()
+        g = Gunicorn(FIXED, threads=1, extra_ports=(port,), env={
+            'HAPROXY_MGR_PEER_FILTER_PORTS': str(port),
+            'HAPROXY_MGR_API_ALLOWED_PEERS': '127.0.0.1,::1'})
+        self.addCleanup(g.stop)
+        self.assertTrue(http_get(port, '/').startswith('pid='))
+        self.assertNotIn('Refused', g.output())
+
+
+class QueuedDripStarvation(unittest.TestCase):
+    """Re-review: with the deadline armed at thread pickup, 40 reconnecting
+    drippers queued behind each other starved legit requests (every probe
+    >30s). Armed as max(enqueue + D, pickup + 1s) they must not."""
+
+    def test_40_reconnecting_drippers_do_not_starve_legit_requests(self):
+        g = Gunicorn(FIXED, threads=4, workers=2, env={})   # production defaults
+        self.addCleanup(g.stop)
+        stop = threading.Event()
+        self.addCleanup(stop.set)
+
+        def drip():
+            while not stop.is_set():
+                try:
+                    s = socket.create_connection(('127.0.0.1', g.port))
+                except OSError:
+                    time.sleep(0.2)
+                    continue
+                try:
+                    req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 5000
+                    for i in range(len(req)):
+                        if stop.is_set():
+                            break
+                        s.send(req[i:i + 1])
+                        time.sleep(1.0)
+                except OSError:
+                    pass
+                finally:
+                    s.close()
+
+        for _ in range(40):
+            threading.Thread(target=drip, daemon=True).start()
+        time.sleep(2)
+        lat = []
+        t_end = time.time() + 25
+        while time.time() < t_end:
+            t0 = time.time()
+            try:
+                http_get(g.port, '/', timeout=30)
+                lat.append(time.time() - t0)
+            except Exception:
+                lat.append(99.0)
+            time.sleep(1)
+        stop.set()
+        lat.sort()
+        median = lat[len(lat) // 2]
+        sys.stderr.write('\n  40 drippers: probes=%d median=%.2fs worst=%.2fs\n'
+                         % (len(lat), median, lat[-1]))
+        # Armed at pickup, every probe here exceeded 30s (total starvation).
+        # With max(enqueue + D, pickup + 1s) queued drippers get ~1s of thread
+        # time, so legit requests keep getting through. They still queue: the
+        # equilibrium wait is ~(drippers/threads) x hold time, about 9s at 40
+        # drippers on 8 threads -- measured, see the round-2 report. The
+        # remaining fix for that is not admitting partial requests to a thread
+        # at all; the peer filter makes it a local-only scenario.
+        # The property under test is "no starvation" (every probe answered
+        # inside 30s). The latency bounds depend on host load: a shared CI
+        # runner measured median 14.97s on 2 probes, so they are only enforced
+        # when HAPROXY_MGR_STRICT_TIMING=1 (quiet local box).
+        self.assertNotIn(99.0, lat, 'a legit request was starved for 30s+')
+        if os.environ.get('HAPROXY_MGR_STRICT_TIMING') == '1':
+            self.assertLess(median, 12.0, lat)
+            self.assertLess(lat[-1], 20.0, lat)
+
+
+class StalledUpstreamDefaultTimeout(unittest.TestCase):
+    """haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
+    client call to an upstream that accepts and never replies must time out."""
+
+    def test_in_process_client_to_stalled_upstream_times_out(self):
+        srv = socket.socket()
+        srv.bind(('127.0.0.1', 0))
+        srv.listen(5)
+        self.addCleanup(srv.close)
+        port = srv.getsockname()[1]
+        accepted = []
+        threading.Thread(target=lambda: accepted.append(srv.accept()),
+                         daemon=True).start()
+        # Import the real module in a child (it changes process-global socket
+        # state and opens its log files at import), then make a client call
+        # with NO explicit timeout -- what a careless future handler would do.
+        probe = textwrap.dedent('''\
+            import logging, os, sys, tempfile, time, urllib.request
+            d = tempfile.mkdtemp()
+            real = logging.FileHandler
+            logging.FileHandler = lambda fn, *a, **k: real(os.path.join(d, os.path.basename(fn)), *a, **k)
+            os.chdir(sys.argv[1]); sys.path.insert(0, sys.argv[1])
+            import haproxy_manager
+            logging.FileHandler = real
+            import socket
+            print('default', socket.getdefaulttimeout())
+            t0 = time.time()
+            try:
+                urllib.request.urlopen('http://127.0.0.1:%s/' % sys.argv[2]).read()
+                print('result returned')
+            except Exception as e:
+                print('result', type(e).__name__, round(time.time() - t0, 1))
+        ''')
+        env = dict(os.environ, HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT='1.5')
+        r = subprocess.run([sys.executable, '-c', probe, REPO_DIR, str(port)],
+                           capture_output=True, text=True, timeout=30, env=env)
+        if 'No module named' in r.stderr and 'flask' in r.stderr.lower():
+            self.skipTest('Flask not installed')
+        self.assertIn('default 1.5', r.stdout, r.stderr)
+        line = [l for l in r.stdout.splitlines() if l.startswith('result')][0]
+        self.assertTrue(line.startswith('result TimeoutError '), line)
+        elapsed = float(line.split()[-1])
+        self.assertLess(elapsed, 5)
+        self.assertTrue(accepted, 'upstream never saw the connection')
+        accepted[0][0].close()
+
+    def test_default_timeout_alone_does_not_cover_gthread_client_sockets(self):
+        # Documents why the custom worker exists: gthread's setblocking(True)
+        # resets any timeout -- including the process default -- to None.
+        old = socket.getdefaulttimeout()
+        socket.setdefaulttimeout(60)
+        try:
+            a, b = socket.socketpair()
+        finally:
+            socket.setdefaulttimeout(old)
+        self.addCleanup(a.close)
+        self.addCleanup(b.close)
+        self.assertEqual(a.gettimeout(), 60)
+        a.setblocking(True)
+        self.assertIsNone(a.gettimeout())
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-runtime-map-contract.py

@@ -0,0 +1,397 @@
+#!/usr/bin/env python3
+"""Contract test: the runtime-map fast path (blocked IPs).
+
+Why this file exists
+--------------------
+`add_ip_to_runtime_map()` and `remove_ip_from_runtime_map()` spent their whole
+existence sending
+
+    add map #0 <ip> 1
+    del map #0 <ip>
+
+to `/tmp/haproxy-cli` and returning True whenever socat exited 0. Neither
+command has ever worked. Two independent defects:
+
+  * **No `@1` prefix.** `/tmp/haproxy-cli` is HAProxy's MASTER CLI socket; map
+    commands are worker commands. The master answers `Unknown command: 'add',
+    but maybe one of the following ones is a better match: ...` -- and **socat
+    still exits 0**, so `result.returncode == 0` was true and the function
+    logged "Added IP x to runtime map".
+  * **`#0` is not a valid map id.** Ids are assigned at config-parse time and
+    move on every config regeneration; on the live edge `blocked_ips.map` is
+    id 37 and `trusted_ips.map` is 10. There is no id 0. Any hardcoded number
+    is wrong -- the map must be referenced by its FILE PATH, which is what
+    haproxy.cfg itself names in `map_ip(/etc/haproxy/blocked_ips.map,0)`.
+
+And a third silence that makes a response-body check alone insufficient:
+`@1 add map #0 <ip> 1` returns an **empty body**, exit 0, and adds nothing
+anywhere -- while `@1 del map #0 <ip>` answers `Unknown map identifier.`. The
+add path can therefore only be trusted after reading the entry back.
+
+IP blocking still worked, because `update_blocked_ips_map()` rewrites
+`/etc/haproxy/blocked_ips.map` and the callers reload HAProxy, which re-reads
+it. The FILE is authoritative; this is the no-reload fast path, and it has
+never once run while reporting that it did.
+
+What it enforces
+----------------
+  1. The command strings actually sent: `@1` prefix first, map referenced by
+     PATH and never by `#<id>`, and the value `1` that
+     `map_ip(...,0) -m int gt 0` requires.
+  2. Every captured rejection is classified as FAILURE (returns False), not
+     success -- including the two that carry no error text at all.
+  3. Success is only reported when the entry reads back in the state asked
+     for. Not the exit status, not an empty reply.
+  4. No source in this repo builds a map command with a `#<id>` reference.
+     Comments may describe the old form; code may not use it.
+
+Runs fully offline: `_cli_send()` is replaced, so no socket, no socat, no
+HAProxy, no network.
+
+Running
+-------
+    python3 scripts/test-runtime-map-contract.py
+"""
+
+import ast
+import io
+import os
+import re
+import sys
+import glob
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+logging.getLogger().setLevel(logging.CRITICAL)
+
+MAP = haproxy_manager.BLOCKED_IPS_MAP_PATH
+IP = '192.0.2.77'
+
+# ---------------------------------------------------------------------------
+# Responses captured VERBATIM from the haproxy-manager container on the live
+# edge (HAProxy 3.0.11, 2026-08-22). socat exited 0 for every single one of
+# them, which is the entire reason none of this is decided on exit status.
+# ---------------------------------------------------------------------------
+
+# What the MASTER socket answers to an unprefixed `add map ...` -- i.e. the
+# reply the old code read as success.
+MASTER_REJECTS_ADD = """\
+Unknown command: 'add', but maybe one of the following ones is a better match:
+  @!<pid>                                 : send a command to the <pid> process
+  @master                                 : send a command to the master process
+  hard-reload                             : achieve a hard-reload (-st) of haproxy
+  reload                                  : achieve a soft-reload (-sf) of haproxy
+  user                                    : lower the level of the current CLI session to user
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+MASTER_REJECTS_DEL = MASTER_REJECTS_ADD.replace("'add'", "'del'")
+
+# `@1 del map #0 <ip>` / `@1 get map /etc/haproxy/nope.map <ip>`.
+UNKNOWN_MAP_IDENTIFIER = 'Unknown map identifier. Please use #<id> or <file>.\n'
+
+# `@1 add map <map> <ip>` with the value omitted (as the old docs showed).
+ADD_MAP_MISSING_VALUE = (
+    "'add map' expects three parameters (map identifier, key and value) or one "
+    "parameter (map identifier) and a payload\n")
+
+# `@1 del map <map> <ip>` for a key the runtime map does not hold.
+KEY_NOT_FOUND = 'Key not found.\n'
+
+# A successful mutation. THIS IS THE WHOLE PROBLEM: it is byte-for-byte what
+# `@1 add map #0 <ip> 1` also returns while adding nothing at all.
+MUTATION_OK = ''
+
+GET_FOUND = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+             'value="1", type="str"\n' % IP)
+GET_NOT_FOUND = 'type=ip, case=sensitive, found=no\n'
+# `@1 get map #0 <ip>` -- "found", but with no value. haproxy.cfg matches with
+# `-m int gt 0`, so an entry like this does NOT block.
+GET_FOUND_NO_VALUE = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+                      'value=none\n' % IP)
+
+SHOW_MAP_SAMPLE = (
+    '0x7f6e5a788700 101.36.109.130 1\n'
+    '0x7f6e5a788780 101.47.140.218 1\n'
+    '0x7f6e5a1fd500 %s 1\n' % IP)
+
+
+class FakeSocket(object):
+    """Replaces _cli_send(). Records every command, answers from a script.
+
+    `script` is a list of (substring, response) pairs, consulted in order; the
+    first whose substring appears in the command wins. Anything unmatched is
+    an explicit test bug, not a silent default.
+    """
+
+    def __init__(self, script):
+        self.script = script
+        self.sent = []
+
+    def __call__(self, command, socket_path, timeout):
+        self.sent.append(command)
+        for needle, response in self.script:
+            if needle in command:
+                return response
+        raise AssertionError('test script has no response for %r' % command)
+
+
+def with_socket(script):
+    """Install a FakeSocket for the duration of a `with` block."""
+    class _Ctx(object):
+        def __enter__(self):
+            self.fake = FakeSocket(script)
+            self._real = haproxy_manager._cli_send
+            haproxy_manager._cli_send = self.fake
+            return self.fake
+
+        def __exit__(self, *exc):
+            haproxy_manager._cli_send = self._real
+            return False
+    return _Ctx()
+
+
+# Every command the happy path needs, in the order the code issues them.
+HAPPY_ADD = [('get map', GET_FOUND), ('add map', MUTATION_OK)]
+HAPPY_DEL = [('get map', GET_NOT_FOUND), ('del map', MUTATION_OK)]
+
+
+class CommandsAreWellFormed(unittest.TestCase):
+    """Guard 1: the exact bytes on the wire.
+
+    Both original defects are visible here and nowhere else -- a `#0` map
+    reference and a missing `@1` are perfectly ordinary-looking Python.
+    """
+
+    def test_add_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_ADD) as fake:
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 add map %s %s 1' % (MAP, IP))
+
+    def test_del_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_DEL) as fake:
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 del map %s %s' % (MAP, IP))
+
+    def test_every_command_is_tried_on_the_worker_first(self):
+        """`/tmp/haproxy-cli` is the MASTER socket; bare map commands 404."""
+        for run in (lambda: haproxy_manager.add_ip_to_runtime_map(IP),
+                    lambda: haproxy_manager.remove_ip_from_runtime_map(IP)):
+            with with_socket(HAPPY_ADD + HAPPY_DEL) as fake:
+                run()
+            for command in fake.sent:
+                self.assertTrue(command.startswith('@1 '),
+                                '%r is missing the @1 worker prefix' % command)
+
+    def test_no_command_references_a_map_by_id(self):
+        """Map ids move on every config regeneration. Path, always."""
+        script = HAPPY_ADD + HAPPY_DEL + [('show map', SHOW_MAP_SAMPLE)]
+        with with_socket(script) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+            haproxy_manager.remove_ip_from_runtime_map(IP)
+            haproxy_manager.runtime_map_keys(MAP)
+        for command in fake.sent:
+            self.assertNotRegex(
+                command, r'\bmap\s+#',
+                '%r references a map by id; ids are not stable' % command)
+            self.assertIn(MAP, command,
+                          '%r does not name the map file' % command)
+
+    def test_add_carries_the_value_the_config_matches_on(self):
+        """`map_ip(...,0) -m int gt 0`: a valueless entry does not block."""
+        self.assertEqual(haproxy_manager.BLOCKED_IPS_MAP_VALUE, '1')
+        with with_socket(HAPPY_ADD) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+        self.assertTrue(fake.sent[0].endswith(' %s 1' % IP),
+                        '%r has no value; HAProxy rejects it' % fake.sent[0])
+
+
+class RejectionIsFailure(unittest.TestCase):
+    """Guard 2+3: nothing may report success unless the map really changed.
+
+    Every response below was returned by the live socket with **exit code 0**.
+    The old code returned True for all of them.
+    """
+
+    def test_master_socket_rejection_of_add_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_master_socket_rejection_of_del_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MASTER_REJECTS_DEL)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_unknown_map_identifier_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket([('get map', GET_FOUND), ('del map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_missing_value_rejection_is_failure(self):
+        """Carries no marker word at all -- caught by 'a mutation says nothing'."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', ADD_MAP_MISSING_VALUE)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_silent_noop_add_is_failure(self):
+        """The `#0` failure mode: accepted, empty reply, nothing added.
+
+        Nothing in the response distinguishes this from success. Only the
+        read-back does -- which is why the read-back is not optional.
+        """
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_add_that_lands_without_a_value_is_failure(self):
+        with with_socket([('get map', GET_FOUND_NO_VALUE), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_del_that_leaves_the_key_behind_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_key_not_found_on_del_is_the_requested_end_state(self):
+        """Not a failure: the runtime map already lacks the key."""
+        with with_socket([('get map', GET_NOT_FOUND), ('del map', KEY_NOT_FOUND)]):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_verified_success_is_reported_as_success(self):
+        with with_socket(HAPPY_ADD):
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket(HAPPY_DEL):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_a_runtime_failure_never_raises_into_the_request_handler(self):
+        """The map file + reload still enforces the block; degrade, don't 500."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertIs(haproxy_manager.add_ip_to_runtime_map(IP), False)
+
+    def test_mutations_are_checked_for_an_empty_body_not_a_marker_list(self):
+        """_HAPROXY_CLI_ERROR_MARKERS can only know rejections already seen."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            with with_socket([('add map', 'something nobody has ever seen\n')]):
+                haproxy_manager.haproxy_cli('add map %s x 1' % MAP,
+                                            worker=True, expect_empty=True)
+
+    def test_the_new_error_markers_are_recognised(self):
+        for response in (UNKNOWN_MAP_IDENTIFIER, KEY_NOT_FOUND,
+                         MASTER_REJECTS_ADD):
+            self.assertTrue(haproxy_manager._cli_response_is_error(response),
+                            '%r must be classified as an error' % response[:40])
+        for response in (GET_FOUND, GET_NOT_FOUND, SHOW_MAP_SAMPLE):
+            self.assertFalse(haproxy_manager._cli_response_is_error(response),
+                             '%r is data, not an error' % response[:40])
+
+    def test_socat_exit_zero_carries_no_information(self):
+        """FakeSocket never signals failure any other way, and neither did socat."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]) as fake:
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertTrue(fake.sent, 'the command was sent and "succeeded" at the '
+                                   'process level; only the body says otherwise')
+
+
+class ReadBack(unittest.TestCase):
+    """The read-back primitives the guarantees above rest on."""
+
+    def test_lookup_reports_found_with_value(self):
+        with with_socket([('get map', GET_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (True, '1'))
+
+    def test_lookup_reports_not_found(self):
+        with with_socket([('get map', GET_NOT_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (False, None))
+
+    def test_lookup_raises_on_a_rejected_reference(self):
+        with with_socket([('get map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_lookup(MAP, IP)
+
+    def test_keys_parses_show_map_output(self):
+        with with_socket([('show map', SHOW_MAP_SAMPLE)]):
+            self.assertEqual(
+                haproxy_manager.runtime_map_keys(MAP),
+                {'101.36.109.130', '101.47.140.218', IP})
+
+    def test_empty_map_is_not_a_rejection(self):
+        with with_socket([('show map', '')]):
+            self.assertEqual(haproxy_manager.runtime_map_keys(MAP), set())
+
+    def test_rejected_show_map_still_raises(self):
+        with with_socket([('show map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_keys(MAP)
+
+
+MAP_BY_ID_RE = re.compile(r'\b(?:add|del|clear|show|get)\s+map\s+#')
+
+
+class NoSourceBuildsAMapIdCommand(unittest.TestCase):
+    """Guard 4: `map #<id>` may be described in comments, never executed.
+
+    Scanning string literals rather than raw text is deliberate -- the whole
+    reason this bug is documented at length in the source is so the next reader
+    does not reintroduce it, and a plain grep would fail on those comments.
+    """
+
+    def test_no_python_string_literal_builds_a_map_id_command(self):
+        tree = ast.parse(io.open('haproxy_manager.py', encoding='utf-8').read())
+        offenders = [
+            node.value for node in ast.walk(tree)
+            if isinstance(node, ast.Constant) and isinstance(node.value, str)
+            and MAP_BY_ID_RE.search(node.value)
+            and not (ast.get_docstring(tree) == node.value)
+        ]
+        # Docstrings are string literals too; exclude any literal that is a
+        # docstring of a module/class/function.
+        docstrings = set()
+        for node in ast.walk(tree):
+            if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef,
+                                 ast.AsyncFunctionDef)):
+                doc = ast.get_docstring(node, clean=False)
+                if doc:
+                    docstrings.add(doc)
+        offenders = [o for o in offenders if o not in docstrings]
+        self.assertEqual(offenders, [],
+                         'these string literals build a map command with an '
+                         'unstable #<id> reference')
+
+    def test_no_shell_or_template_code_line_uses_a_map_id(self):
+        targets = (glob.glob('scripts/*.sh') + glob.glob('templates/*.tpl')
+                   + ['Dockerfile'])
+        offenders = []
+        for path in targets:
+            if not os.path.exists(path):
+                continue
+            for lineno, line in enumerate(
+                    io.open(path, encoding='utf-8').read().splitlines(), 1):
+                if line.lstrip().startswith('#'):
+                    continue  # a comment describing the old form is fine
+                if MAP_BY_ID_RE.search(line):
+                    offenders.append('%s:%d: %s' % (path, lineno, line.strip()))
+        self.assertEqual(offenders, [],
+                         'map ids are assigned at config-parse time and move; '
+                         'reference the map file by path')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-waf-body-inspection.py

@@ -0,0 +1,382 @@
+#!/usr/bin/env python3
+"""Regression tests for WAF request-body inspection (Coraza via SPOE).
+
+Why this file exists
+--------------------
+Measured 2026-10-09 on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0
+(report: waf-body-hardening-report.md):
+
+  * Without `option http-buffer-request`, send-spoe-group fires at header
+    time and `req.body` holds only what happened to arrive with the headers.
+    A 14-byte `content=$a($b)` POST (CRS 933180, an ENFORCED rule) passed
+    uninspected when its body was written in a second send(); a body sent 2s
+    late, or after `Expect: 100-continue`, was never inspected at all.
+  * Coraza+CRS costs ~3.3 ms per KB of body. The SPOE `timeout processing`
+    is 100 ms and FAILS OPEN, so raising tune.bufsize to 32 KB turned a
+    payload at byte 0 of a 30 KB base64 body into a bypass. bufsize is a
+    security limit, not a performance knob.
+  * coraza-spoa's SPOP library refuses frames > 65535 bytes, and Coraza's
+    SecRequestBodyLimitAction Reject 413s even under DetectionOnly, so the
+    three limits must stay ordered.
+  * The X-WHP-WAF-Body stamp drives enforcement of body-parse errors
+    (200002/200003) in WHP's generated Coraza rules. A client must never be
+    able to set it, and a backend must never see it.
+
+These tests pin the rendered config. That HAProxy ACCEPTS it is
+validate-rendered-config.py's job; that it BEHAVES is the live measurement.
+
+Running
+-------
+    python3 scripts/test-waf-body-inspection.py
+"""
+
+import logging
+import os
+import re
+import sys
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+SPOA_FRAME_CAP = 65535          # dropmorepackets/haproxy-go v0.0.8 maxFrameSize
+CORAZA_CONFIG = os.path.join(MODULE_DIR, 'coraza-spoa', 'config.yaml')
+
+
+def render(template, coraza):
+    kw = dict(crt_path='/etc/haproxy/certs', suspension_enabled=False,
+              coraza_spoe_backend='coraza-spoa:9000' if coraza else None,
+              cluster_secret=None, syslog_target='172.18.0.1:514')
+    return haproxy_manager.template_env.get_template(template).render(**kw)
+
+
+def active_lines(text):
+    """Config lines with comments and blanks stripped (comments mention the
+    directives by name, so a raw substring check would pass on prose)."""
+    out = []
+    for line in text.splitlines():
+        s = line.strip()
+        if s and not s.startswith('#'):
+            out.append(s)
+    return out
+
+
+def index_of(lines, pattern):
+    for i, line in enumerate(lines):
+        if re.search(pattern, line):
+            return i
+    return -1
+
+
+class ListenerWithCoraza(unittest.TestCase):
+
+    def setUp(self):
+        self.lines = active_lines(render('hap_listener.tpl', coraza=True))
+
+    def test_body_is_buffered_before_inspection(self):
+        self.assertIn('option http-buffer-request', self.lines)
+
+    def test_stamp_order_strip_set_inspect_strip(self):
+        strip1 = index_of(self.lines, r'^http-request del-header X-WHP-WAF-Body$')
+        size = index_of(self.lines, r'^http-request set-var\(txn\.waf_body_size\) req\.body_size if \{ req\.hdr_cnt\(content-length\) eq 1 \}$')
+        complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
+        partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if \{ var\(txn\.waf_body_size\) -m found \} !\{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
+        hdrs_len = index_of(self.lines, r'^http-request set-var\(txn\.waf_hdrs_len\) req\.hdrs,length if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \}$')
+        nocl_complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_hdrs_len\) -m found \} \{ req\.body_len,add\(txn\.waf_hdrs_len\) lt (\d+) \}$')
+        nocl_partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \} !\{ var\(txn\.waf_body\) -m found \}$')
+        stamp = index_of(self.lines, r'^http-request set-header X-WHP-WAF-Body %\[var\(txn\.waf_body\)\] if \{ var\(txn\.waf_body\) -m found \}$')
+        spoe = index_of(self.lines, r'^http-request send-spoe-group coraza coraza-req$')
+        deny = index_of(self.lines, r'^http-request return status 403 .*var\(txn\.coraza\.action\) -m str deny')
+        dup_ct = index_of(self.lines, r'^http-request deny deny_status 400 if \{ req\.fhdr_cnt\(content-type\) gt 1 \}$')
+        self.assertGreaterEqual(dup_ct, 0, 'duplicate Content-Type must be rejected at the edge')
+        self.assertLess(dup_ct, spoe, 'and before Coraza is asked')
+        for name, idx in (('client-copy strip', strip1), ('size var', size), ('complete', complete),
+                          ('partial', partial), ('no-CL header length', hdrs_len),
+                          ('no-CL complete', nocl_complete),
+                          ('no-CL partial', nocl_partial), ('stamp', stamp),
+                          ('send-spoe-group', spoe), ('deny', deny)):
+            self.assertGreaterEqual(idx, 0, f'{name} rule missing from the rendered listener')
+        # Client copies stripped BEFORE we stamp, stamp BEFORE the SPOE call.
+        self.assertLess(strip1, size)
+        self.assertLess(size, complete)
+        # The no-CL `partial` fallback relies on "-m found" of an earlier
+        # `complete`: order is load-bearing.
+        self.assertLess(size, hdrs_len)
+        self.assertLess(hdrs_len, nocl_complete)
+        self.assertLess(nocl_complete, nocl_partial)
+        self.assertLess(max(complete, partial, nocl_partial), stamp)
+        self.assertLess(stamp, spoe)
+        # Stripped again AFTER the SPOE call so no backend ever sees it.
+        strip2 = [i for i, l in enumerate(self.lines)
+                  if l == 'http-request del-header X-WHP-WAF-Body' and i > spoe]
+        self.assertTrue(strip2, 'X-WHP-WAF-Body must be deleted again after send-spoe-group')
+        self.assertLess(strip2[0], deny)
+
+    def test_no_content_length_complete_cannot_have_filled_the_buffer(self):
+        """Without a Content-Length, `complete` is inferred from "the buffer
+        cannot have been full". body + headers thresholds must sit well below
+        bufsize - maxrewrite (1024), or a truncated chunked body would be
+        stamped complete and a parse error on it would 403."""
+        line = [l for l in self.lines if 'str(complete) if { var(txn.waf_hdrs_len)' in l][0]
+        total_lt = int(re.search(r'add\(txn\.waf_hdrs_len\) lt (\d+)', line).group(1))
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
+        self.assertLessEqual(total_lt, (bufsize - 1024) * 0.85)
+
+    def test_complete_and_partial_are_mutually_exclusive_conditions(self):
+        """With a Content-Length, `complete` and `partial` must be the same
+        condition, negated -- an inverted or loosened `complete` would enforce
+        parse errors on truncated bodies (403 on large legitimate posts)."""
+        c = [l for l in self.lines if 'str(complete) if { var(txn.waf_body_size)' in l]
+        p = [l for l in self.lines if 'str(partial) if { var(txn.waf_body_size)' in l]
+        self.assertEqual((len(c), len(p)), (1, 1))
+        c, p = c[0], p[0]
+        self.assertEqual(c.split(' if ', 1)[1].replace('!{', '{'),
+                         p.split(' if ', 1)[1].replace('!{', '{'))
+        self.assertIn('!{ req.body_len', p)
+        self.assertNotIn('!{', c)
+
+    def test_stamp_never_set_from_client_input(self):
+        """The only set-header for the stamp is from our own var."""
+        sets = [l for l in self.lines if re.search(r'set-header\s+X-WHP-WAF-Body', l, re.I)]
+        self.assertEqual(len(sets), 1)
+        self.assertIn('%[var(txn.waf_body)]', sets[0])
+        self.assertFalse([l for l in self.lines if re.search(r'add-header\s+X-WHP-WAF-Body', l, re.I)])
+
+    def test_access_log_carries_coverage_fields(self):
+        lf = [l for l in self.lines if l.startswith('log-format ')]
+        self.assertEqual(len(lf), 1)
+        self.assertIn('wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)]', lf[0])
+
+    def test_access_log_carries_waf_verdict_fields(self):
+        """wrule/wrules/wscore are appended LAST so any consumer anchored on
+        the earlier fields keeps working."""
+        lf = [l for l in self.lines if l.startswith('log-format ')]
+        self.assertEqual(len(lf), 1)
+        self.assertTrue(lf[0].endswith(
+            'wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] '
+            'wscore=%[var(txn.coraza.anomaly_score)]"'))
+
+    def test_fail_open_preserved(self):
+        """Nothing may deny on var(txn.coraza.error): an SPOA outage must not
+        503 customer sites."""
+        self.assertFalse([l for l in self.lines if 'txn.coraza.error' in l and
+                          re.search(r'\b(deny|reject|return|silent-drop|tarpit)\b', l)])
+
+
+class ListenerWithoutCoraza(unittest.TestCase):
+
+    def test_waf_off_config_has_no_body_buffering(self):
+        lines = active_lines(render('hap_listener.tpl', coraza=False))
+        self.assertNotIn('option http-buffer-request', lines)
+        self.assertFalse([l for l in lines if 'X-WHP-WAF-Body' in l])
+        self.assertFalse([l for l in lines if 'send-spoe-group' in l])
+
+    def test_waf_off_header_keeps_default_bufsize(self):
+        lines = active_lines(render('hap_header.tpl', coraza=False))
+        self.assertFalse([l for l in lines if l.startswith('tune.bufsize')])
+
+
+class BufsizeCeilings(unittest.TestCase):
+
+    def setUp(self):
+        lines = active_lines(render('hap_header.tpl', coraza=True))
+        found = [l for l in lines if l.startswith('tune.bufsize')]
+        self.assertEqual(len(found), 1, 'tune.bufsize must be pinned when Coraza is on')
+        self.bufsize = int(found[0].split()[1])
+        spoe = active_lines(haproxy_manager.template_env.get_template(
+            'hap_coraza_spoe_engine.tpl').render())
+        m = [re.match(r'timeout\s+processing\s+(\d+)ms$', l) for l in spoe]
+        m = [x for x in m if x]
+        self.assertEqual(len(m), 1)
+        self.processing_ms = int(m[0].group(1))
+
+    def test_bufsize_fits_the_spoe_time_budget(self):
+        """~3.3 ms/KB measured; 31 KB took 104 ms against a 100 ms timeout and
+        FAILED OPEN. Keep the worst case at about half the budget."""
+        worst_ms = (self.bufsize / 1024.0) * 3.3
+        self.assertLessEqual(worst_ms, self.processing_ms * 0.6,
+                             f'tune.bufsize {self.bufsize} => ~{worst_ms:.0f} ms of Coraza time '
+                             f'against a {self.processing_ms} ms fail-open timeout')
+
+    def test_bufsize_below_spop_frame_cap(self):
+        self.assertLess(self.bufsize, SPOA_FRAME_CAP)
+
+    def test_body_arg_is_the_full_buffered_body(self):
+        """The message must keep sending req.body unsliced: the `complete`
+        stamp compares req.body_len to Content-Length, so a slice would make
+        a truncated body look complete to Coraza."""
+        spoe = haproxy_manager.template_env.get_template('hap_coraza_spoe_engine.tpl').render()
+        args = [l.strip() for l in spoe.splitlines() if l.strip().startswith('args ')]
+        self.assertEqual(len(args), 1)
+        self.assertTrue(args[0].endswith(' body=req.body'))
+
+
+class CorazaTransactionIdIsTheRequestReference(unittest.TestCase):
+    """Coraza's transaction ID must be HAProxy's unique-id, so the reference a
+    blocked customer quotes (403-waf.html / X-Request-Reference / `id=` in
+    haproxy.log) is the `unique_id` in /var/log/coraza/audit.log and
+    spoa.log. Without `id=` coraza-spoa v0.7.1 invents a random 16-letter ID
+    (internal/application.go HandleRequest) that joins to nothing."""
+
+    def setUp(self):
+        spoe = haproxy_manager.template_env.get_template(
+            'hap_coraza_spoe_engine.tpl').render()
+        lines = active_lines(spoe)
+        self.messages = {}
+        current = None
+        for line in lines:
+            m = re.match(r'^spoe-message\s+(\S+)$', line)
+            if m:
+                current = m.group(1)
+                continue
+            if line.startswith('spoe-') or line.startswith('['):
+                current = None
+            elif current and line.startswith('args '):
+                self.messages[current] = line.split()[1:]
+
+    def args_of(self, message):
+        return dict(a.split('=', 1) for a in self.messages[message])
+
+    def test_request_message_sends_unique_id(self):
+        self.assertIn('coraza-req', self.messages)
+        self.assertEqual(self.args_of('coraza-req').get('id'), 'unique-id')
+
+    def test_request_message_exports_rule_ids(self):
+        """Without exportRuleIDs coraza-spoa never sets txn.coraza.rule_ids
+        and the access log's wrules= is always '-'."""
+        self.assertEqual(self.args_of('coraza-req').get('exportRuleIDs'), 'bool(true)')
+
+    def test_app_is_still_the_first_arg(self):
+        """agent.go HandleSPOE Panics unless the FIRST kv entry is `app`."""
+        for name, args in self.messages.items():
+            self.assertTrue(args[0].startswith('app='),
+                            f'{name}: first arg must be app=, got {args[0]}')
+
+    def test_every_response_message_carries_the_same_id(self):
+        """HandleResponse finds the request transaction by `id`; a coraza-res
+        without it errors ("response id is empty") on every response."""
+        for name in self.messages:
+            if name != 'coraza-req':
+                self.assertEqual(self.args_of(name).get('id'), 'unique-id', name)
+
+    def test_unique_id_is_the_header_reference(self):
+        """The fetch is only meaningful while unique-id-format is the UUID that
+        is also emitted as X-Request-Reference and on the 403 page."""
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        self.assertIn('unique-id-format %[uuid()]', [re.sub(r'\s+', ' ', l) for l in header])
+        self.assertIn('unique-id-header X-Request-Reference', [re.sub(r'\s+', ' ', l) for l in header])
+        listener = render('hap_listener.tpl', coraza=True)
+        self.assertIn('x-request-reference "%[unique-id]"', listener)
+
+
+CORAZA_OVERRIDES = os.path.join(MODULE_DIR, 'coraza-spoa', 'overrides.conf')
+
+
+@unittest.skipUnless(os.path.exists(CORAZA_OVERRIDES),
+                     'coraza-spoa/overrides.conf is not shipped in this image; '
+                     'this class runs from a repo checkout')
+class BakedPromotions(unittest.TestCase):
+    """920440 is a CHAIN whose first link matches every URL with a file
+    extension. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on that
+    first link, so every *.php / *.css request was evaluated with the engine
+    On (full CRS anomaly enforcement, in detect_only mode too). Measured
+    in-process and live, 2026-10-09; fixed by keying the flip on the chain's
+    real condition."""
+
+    def setUp(self):
+        with open(CORAZA_OVERRIDES) as fh:
+            self.lines = active_lines(fh.read())
+
+    def test_920440_is_not_promoted_on_its_chain_starter(self):
+        self.assertFalse([l for l in self.lines
+                          if re.match(r'SecRuleUpdateActionById\s+(\S*-)?920440\b', l)],
+                         '920440 promoted via SecRuleUpdateActionById again')
+
+    def test_no_chain_starter_is_promoted(self):
+        """Chain-starter rules in the promoted ranges (CRS 4.25.0): 920440,
+        944110, 944120. None may be covered by a SecRuleUpdateActionById."""
+        chain_starters = (920440, 944110, 944120)
+        promoted = []
+        for l in self.lines:
+            m = re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l)
+            if m:
+                lo = int(m.group(1))
+                promoted.append((lo, int(m.group(2) or lo)))
+        self.assertTrue(promoted, 'no promotions parsed -- the test is blind')
+        for rid in chain_starters:
+            self.assertFalse([r for r in promoted if r[0] <= rid <= r[1]],
+                             f'{rid} is a CRS chain starter and must not be promoted')
+
+    def test_log4shell_range_still_promoted(self):
+        for rid in (944100, 944130, 944150, 944151, 944152, 944300):
+            self.assertTrue([1 for l in self.lines
+                             if (m := re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l))
+                             and int(m.group(1)) <= rid <= int(m.group(2) or m.group(1))],
+                            f'{rid} lost its promotion')
+
+    def test_920440_flip_keys_on_the_restricted_extension(self):
+        i = self.lines.index('SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \\')
+        self.assertEqual(self.lines[i + 1],
+                         '"id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"')
+
+
+@unittest.skipUnless(os.path.exists(CORAZA_CONFIG),
+                     'coraza-spoa/config.yaml is not shipped in this image; '
+                     'this class runs from a repo checkout')
+class CorazaBodyLimits(unittest.TestCase):
+
+    def setUp(self):
+        with open(CORAZA_CONFIG) as fh:
+            self.cfg = active_lines(fh.read())
+        header = active_lines(render('hap_header.tpl', coraza=True))
+        self.bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
+
+    def directive(self, name):
+        vals = [l.split(None, 1)[1] for l in self.cfg if l.split()[0] == name]
+        self.assertEqual(len(vals), 1, f'{name} must be set exactly once in config.yaml')
+        return vals[0]
+
+    def test_limit_strictly_above_anything_haproxy_can_send(self):
+        limit = int(self.directive('SecRequestBodyLimit'))
+        self.assertGreater(limit, SPOA_FRAME_CAP)
+        self.assertGreater(limit, self.bufsize)
+
+    def test_over_limit_never_rejects(self):
+        """Reject interrupts even under DetectionOnly -> 403 on uploads."""
+        self.assertEqual(self.directive('SecRequestBodyLimitAction'), 'ProcessPartial')
+
+    def test_body_buffer_stays_in_memory(self):
+        self.assertGreaterEqual(int(self.directive('SecRequestBodyInMemoryLimit')),
+                                int(self.directive('SecRequestBodyLimit')))
+
+    def test_json_depth_limit_pinned(self):
+        """Quadratic JSON flattening: depth 300 = 297 ms, past the fail-open
+        timeout. Legitimate WordPress JSON measured at depth <= 9."""
+        depth = int(self.directive('SecRequestBodyJsonDepthLimit'))
+        self.assertGreaterEqual(depth, 32, 'too tight for real block/theme payloads')
+        self.assertLessEqual(depth, 64, 'depth 100 already costs ~40 ms')
+
+    def test_limits_are_set_after_the_recommended_file(self):
+        inc = self.cfg.index('Include @coraza.conf-recommended')
+        for name in ('SecRequestBodyLimit ', 'SecRequestBodyJsonDepthLimit '):
+            lim = [i for i, l in enumerate(self.cfg) if l.startswith(name)][0]
+            self.assertLess(inc, lim, f'the recommended file would override {name.strip()}')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

templates/hap_backend.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -29,7 +29,7 @@ backend {{ name }}-sse-backend
 
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_http_check.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option httpchk
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     {% if ssl_enabled %}http-request set-header X-Forwarded-Proto https if { ssl_fc }{% endif %}

templates/hap_backend_longlived.tpl

@@ -14,7 +14,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -31,7 +31,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_websocket.tpl

@@ -7,7 +7,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -24,7 +24,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_coraza_spoe_engine.tpl

@@ -10,9 +10,9 @@
 # config.yaml.
 #
 # Reference: this config follows the shape from coraza-spoa's upstream
-# example/haproxy/coraza.cfg (v0.7.1). Arg names + ordering are required by
-# Coraza-SPOA exactly as specified — DO NOT reorder or rename without
-# coordinating with the agent.
+# example/haproxy/coraza.cfg (v0.7.1). Arg NAMES are a contract with
+# coraza-spoa — DO NOT rename without coordinating with the agent. Ordering:
+# only `app` is positional (it must be FIRST); the rest are matched by name.
 
 [coraza]
 
@@ -56,9 +56,41 @@ spoe-agent coraza
 
 # Per-request inspection message. No `event` directive — fires only when
 # explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.
-# Arg order/names are mandatory: Coraza-SPOA parses positionally and renames
-# break the agent. `app=str(haproxy)` is the literal application name from
-# coraza-spoa's config.yaml `applications:` block.
+#
+# How coraza-spoa v0.7.1 reads these args (measured from source, not inferred):
+#   * `app` MUST be the first arg. internal/agent.go HandleSPOE reads exactly
+#     one KV entry and Panics if its name is not "app".
+#   * Every other arg is matched BY NAME in a switch (internal/application.go
+#     HandleRequest); order is irrelevant and unknown names are ignored with a
+#     debug log. So a typo'd name silently drops that input — renames still
+#     break inspection, they just don't error.
+# `app=str(haproxy)` is the literal application name from coraza-spoa's
+# config.yaml `applications:` block.
+#
+# id=unique-id makes Coraza's transaction ID the SAME UUID HAProxy generates
+# via `unique-id-format %[uuid()]` (hap_header.tpl) — the value shown on the
+# 403-waf.html page, sent as X-Request-Reference, and logged as `id=` in
+# /var/log/haproxy.log. coraza-spoa passes it to waf.NewTransactionWithID(),
+# so it becomes `transaction.id` in /var/log/coraza/audit.log and
+# `match.unique_id` in /var/log/coraza/spoa.log: a customer-quoted reference
+# greps straight to the rule that fired. Without it coraza-spoa invents a
+# random 16-letter A-Z ID that appears nowhere on the HAProxy side. If the
+# fetch ever came back empty the agent falls back to that random ID, so this
+# cannot break inspection.
+#
+# exportRuleIDs=bool(true) makes coraza-spoa (application.go exportWAFMetrics)
+# set var(txn.coraza.rule_ids) = comma list of every matched CRS attack rule
+# (910000-959999 + 190000-199999; control rules such as our 99xxxxxxx host
+# exceptions and 2000xx body rules are NOT listed), alongside rules_hit and
+# anomaly_score which it sets regardless. var(txn.coraza.ruleid) is set on
+# every interruption (agent.go) -- for anomaly blocks that is 949110, so
+# rule_ids is what actually says WHY. All three are logged in the access
+# log (hap_listener.tpl log-format: wrule= wrules= wscore=).
+#
+# Response inspection (a `coraza-res` message) is NOT configured:
+# config.yaml has response_check: false. If it is ever enabled, coraza-res
+# MUST also send `id=unique-id` -- HandleResponse
+# looks the request transaction up by that ID and errors without it.
 #
 # src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
 # top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
@@ -66,7 +98,7 @@ spoe-agent coraza
 # when no proxy headers are present. This is what shows up as `client_ip`
 # in /var/log/coraza/audit.log and the rule manager's view-matches panel.
 spoe-message coraza-req
-    args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
+    args app=str(haproxy) id=unique-id exportRuleIDs=bool(true) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
 
 # Group binding for send-spoe-group invocation in the frontend. One group,
 # one message; could add more in the future (e.g. coraza-res for response

templates/hap_header.tpl

@@ -106,6 +106,41 @@ global
 
     # Stats persistence for zero-downtime reloads
     stats-file /var/lib/haproxy/stats.dat
+{%- if coraza_spoe_backend %}
+
+    # WAF BODY INSPECTION LIMIT. Only emitted when Coraza is wired in, so a
+    # WAF-off / standalone config is unchanged byte-for-byte. 16384 is also
+    # HAProxy's default: this line changes no behaviour or memory, it pins
+    # the value so nobody raises it without reading this.
+    #
+    # The request body Coraza sees is whatever `req.body` holds when
+    # send-spoe-group fires, and with `option http-buffer-request`
+    # (hap_listener.tpl) that is "the whole body, or as much as fits in ONE
+    # buffer". So bufsize IS the inspection limit: a body up to roughly
+    # bufsize - maxrewrite (1024) - the request headers (~15 KB) is inspected
+    # whole and stamped `complete`; anything larger is forwarded intact, its
+    # first ~15 KB inspected, and stamped `partial`. Large uploads are never
+    # rejected or held for it.
+    #
+    # DO NOT RAISE THIS WITHOUT RE-MEASURING -- three ceilings, all measured:
+    #   * TIME. Coraza+CRS 4.25 costs ~3.3 ms per KB of body (linear; 31 KB
+    #     base64 = 104 ms, 15 KB = ~50 ms on a 32-core box). The SPOE
+    #     `timeout processing` is 100 ms (hap_coraza_spoe_engine.tpl) and a
+    #     timeout FAILS OPEN: with tune.bufsize 32768 a payload at byte 0 of a
+    #     30 KB base64 multipart body passed 933180 uninspected. A larger
+    #     buffer is a bypass, not more coverage, unless the timeout (and CPU
+    #     budget) grows with it.
+    #   * FRAME. coraza-spoa's SPOP library (dropmorepackets/haproxy-go
+    #     v0.0.8) refuses a HELLO whose max-frame-size exceeds 65535; HAProxy
+    #     derives its offer from bufsize. bufsize >= 64 KB = every SPOE
+    #     handshake fails = WAF silently off for all traffic.
+    #   * MEMORY. maxconn 4000 x 2 buffers x bufsize, inside the 1 GB edge
+    #     container.
+    # Coraza's SecRequestBodyLimit (coraza-spoa/config.yaml, 131072) must stay
+    # above anything this can deliver. Pinned by
+    # scripts/test-waf-body-inspection.py.
+    tune.bufsize 16384
+{%- endif %}
 
 #---------------------------------------------------------------------
 # DNS resolver for Docker container name resolution

templates/hap_listener.tpl

@@ -63,7 +63,18 @@ frontend web
     #   host=/ua= identify the vhost and client; %ST/%B/%tsc give status,
     #        bytes and the termination state that distinguishes a rate-limit
     #        deny (PR--) from a tarpit (PT--) from a normal close.
-    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major]"
+    #   wafb=/wafe= WAF body coverage: wafb is complete|partial|- (did Coraza
+    #        get the whole request body, see the Coraza block below);
+    #        wafe is var(txn.coraza.error), set when the SPOE call failed or
+    #        timed out and the request went through UNINSPECTED (fail-open).
+    #        Both are "-" when the WAF is off or the request had no body.
+    #   wrule=/wrules=/wscore= WHY Coraza acted, without opening audit.log:
+    #        wrule is the interrupting rule (949110 for anomaly blocks),
+    #        wrules every matched CRS attack rule (e.g. 920440,949110),
+    #        wscore the inbound anomaly score. Set by coraza-spoa via the
+    #        exportRuleIDs SPOE arg (hap_coraza_spoe_engine.tpl). All "-"
+    #        when the WAF is off; wrule is "-" unless Coraza interrupted.
+    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major] wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)] wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] wscore=%[var(txn.coraza.anomaly_score)]"
 
     # --- URI normalisation (MUST be the first path-touching block here) ---
     # Every path-based control in this frontend (the ACME health-check bypass,
@@ -582,7 +593,9 @@ frontend web
 
     # IP blocking using map file (manual blocks only)
     # Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
-    # Runtime updates: echo "add map #0 IP_ADDRESS 1" | socat stdio /var/run/haproxy.sock
+    # Runtime updates (worker command, map referenced by PATH -- "#<id>" ids
+    # move on every config regeneration and "#0" silently adds nothing):
+    #   echo "@1 add map /etc/haproxy/blocked_ips.map IP_ADDRESS 1" | socat stdio /tmp/haproxy-cli
     # Checks the real client IP (from headers if present, otherwise src)
     # map_ip() converter supports both single IPs and CIDR ranges (e.g., 192.168.1.0/24)
     acl is_blocked_ip var(txn.real_ip),map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0
@@ -609,7 +622,74 @@ frontend web
     # guards (no point asking the WAF about requests we're already dropping)
     # and AFTER the real-client-IP resolution (so Coraza sees the right src).
     filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
+
+    # --- Make Coraza actually see the request body -------------------------
+    # send-spoe-group sends `req.body` = whatever body bytes are in the buffer
+    # AT THAT MOMENT. Without buffering that is usually NOTHING: measured on
+    # haproxy 3.0.11 + coraza-spoa v0.7.1, a 14-byte `content=$a($b)` POST
+    # whose body was written in a second send() right after the headers
+    # passed 933180 (an enforced rule) uninspected, while the identical bytes
+    # in one send() were blocked. A body sent 2s late, or after an
+    # `Expect: 100-continue`, was never inspected at all.
+    #
+    # `option http-buffer-request` makes HAProxy wait, before evaluating the
+    # http-request rules, until the WHOLE body is in or the buffer is full,
+    # bounded by `timeout http-request` (30s, hap_header.tpl). It answers
+    # `Expect: 100-continue` itself so the client actually sends the body.
+    # It does NOT hold uploads: once one buffer (tune.bufsize, 16 KB) is full
+    # the request is forwarded and the rest streams through as before -- a
+    # 50 MB upload is not stored at the edge.
+    option http-buffer-request
+
+    # Exactly one Content-Type, or 400. With two Content-Type lines Coraza and
+    # the backend disagree about what the body IS: Coraza evaluates every value
+    # (`text/xml` on either line selects its XML processor, which accepts a
+    # form body without error), while Apache merges the lines with ", " and
+    # PHP cuts the media type at the first ',' / ';'. Measured (Coraza v3.7.0
+    # + PHP 8.3): `application/x-www-form-urlencoded` + `text/xml` puts the
+    # payload in $_POST with nothing inspected. req.fhdr_cnt counts header
+    # LINES, so a single comma-joined value (which browsers produce when a
+    # script calls setRequestHeader twice) is not affected here; that form is
+    # parsed consistently enough by both sides (see the report's matrix).
+    # Browsers, curl, PHP-curl and the common HTTP libraries send one line.
+    http-request deny deny_status 400 if { req.fhdr_cnt(content-type) gt 1 }
+
+    # Tell Coraza whether it got the whole body. Coraza cannot tell on its
+    # own (REQUEST_BODY_LENGTH is only filled by the urlencoded/raw
+    # processors), and the difference matters: a TRUNCATED JSON body, or a
+    # multipart body cut inside a boundary/part header, fails to parse by
+    # construction, so the body-parse-error rules (200002/200003) may only
+    # deny a COMPLETE body. WHP's generated Coraza rules
+    # (regenerate-coraza-overrides.sh) key on this header:
+    #   complete -> 200002/200003 deny on multipart/JSON (enforce mode)
+    #   partial  -> 200002/200003 removed for this request (every mode)
+    #
+    # With a Content-Length: complete iff req.body_len (buffered) equals
+    # req.body_size (= Content-Length).
+    # Without one (HTTP/1.1 chunked, HTTP/2-3 without content-length) HAProxy
+    # 3.0 has no "end of message" fetch and req.body_size is just what is
+    # buffered. But http-buffer-request only releases the request early when
+    # the body ENDED or the buffer FILLED (a stalled body gets a 408 instead),
+    # and body + headers well below one buffer (12 KB of the ~15 KB usable)
+    # cannot have filled it. So that is complete, anything else partial.
+    # Without this branch an attacker only had to add
+    # `Transfer-Encoding: chunked` to escape enforcement (measured).
+    #
+    # Client-supplied copies are deleted FIRST so it cannot be forged, and it
+    # is deleted again after the SPOE call so backends never see it.
+    # txn.waf_body is also logged (`wafb=` in the log-format above) so the
+    # complete/partial mix is measurable per host.
+    http-request del-header X-WHP-WAF-Body
+    http-request set-var(txn.waf_body_size) req.body_size if { req.hdr_cnt(content-length) eq 1 }
+    http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_body_size) -m found } { req.body_len,sub(txn.waf_body_size) eq 0 }
+    http-request set-var(txn.waf_body) str(partial) if { var(txn.waf_body_size) -m found } !{ req.body_len,sub(txn.waf_body_size) eq 0 }
+    http-request set-var(txn.waf_hdrs_len) req.hdrs,length if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 }
+    http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_hdrs_len) -m found } { req.body_len,add(txn.waf_hdrs_len) lt 12288 }
+    http-request set-var(txn.waf_body) str(partial) if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 } !{ var(txn.waf_body) -m found }
+    http-request set-header X-WHP-WAF-Body %[var(txn.waf_body)] if { var(txn.waf_body) -m found }
+
     http-request send-spoe-group coraza coraza-req
+    http-request del-header X-WHP-WAF-Body
 
     # Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to
     # "deny" / "drop" / "redirect" when a rule with the corresponding