chore: bump VERSION to 2026.09.1

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CLAUDE.md

@@ -7,8 +7,55 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ### Testing
 - **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
 - **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
+- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
+- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
+- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
 - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
 
+### Reading stick tables (and why it is easy to get silently wrong)
+
+`/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Worker commands
+(`show table`, `show map`, `add map`, ...) need an `@1` prefix. Without it
+HAProxy answers `Unknown command: 'show', ...` **and socat still exits 0** — so
+an exit-status check passes and the help text gets parsed as data. Always use
+`haproxy_cli(cmd, worker=True)` in Python, which inspects the response body.
+
+Stick-table entries are `name=value` / `name(window_ms)=value` pairs, not fixed
+columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
+by NAME, and treat a missing field as an ERROR — never default it to `0`. The
+`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
+`http_err_rate`; it holds **no history and no counter of past blocks**. What was
+actually denied/tarpitted is in the edge access log on the **host** at
+`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table.
+
+This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
+reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
+stick table has ever stored — for their entire existence. See the header of
+`haproxy_tarpit_config.txt` and the contract test.
+
+### Changing a runtime map (`add map` / `del map`)
+
+Same socket, two more ways to fail silently — and both were live in
+`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
+existence:
+
+* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
+  config-parse time and move on every config regeneration (on whp01
+  `blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
+  `add map /etc/haproxy/blocked_ips.map <ip> 1`.
+* **A mutation answers NOTHING on success**, so an empty body is the only
+  success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
+  *also* answers nothing and adds nothing, so the body cannot prove an add
+  worked. **Read it back** with `@1 get map <path> <key>`.
+* Entries must carry the value `1`; haproxy.cfg matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
+
+In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
+and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
+only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
+re-reads it on reload, so a failed runtime command must degrade to
+"enforced on reload" and be reported, never swallowed.
+
 ### Running the Application
 - **Docker Build**: `docker build -t haproxy-manager .`
 - **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)

Dockerfile

@@ -23,7 +23,23 @@ LABEL org.opencontainers.image.title="haproxy-manager-base" \
       org.opencontainers.image.version="${VERSION}" \
       org.opencontainers.image.licenses="MIT"
 
-RUN apt update -y && apt dist-upgrade -y && apt install socat haproxy cron certbot curl jq net-tools -y && apt clean && rm -rf /var/lib/apt/lists/*
+# haproxy is PINNED. It was previously unpinned, so the binary could move under
+# us at Debian's timing — an upstream release that rejected our config would have
+# broken an unrelated commit's build, or worse, shipped an edge that refuses to
+# start (see the `haproxy -c` gate below for why that matters: a config HAProxy
+# rejects leaves the container Up with 80/443 unbound and /health still 200).
+#
+# Pinning does NOT make the gate redundant, and the gate does NOT make pinning
+# unnecessary — they compose. Pinned means the version moves deliberately; the
+# gate then answers immediately whether the new binary still accepts fleet config.
+# It also makes the image reproducible, which it previously was not.
+#
+# To move it: bump the version here, rebuild, and let the gate verify. If Debian
+# security-updates the package (e.g. -1+deb13u4) the build FAILS until this pin is
+# updated — that failure is the point, not a bug. Check availability with:
+#   apt-cache policy haproxy
+ARG HAPROXY_VERSION=3.0.11-1+deb13u3
+RUN apt update -y && apt dist-upgrade -y && apt install socat "haproxy=${HAPROXY_VERSION}" cron certbot curl jq net-tools -y && apt-mark hold haproxy && apt clean && rm -rf /var/lib/apt/lists/*
 WORKDIR /haproxy
 COPY ./templates /haproxy/templates
 COPY requirements.txt /haproxy/
@@ -32,12 +48,51 @@ COPY scripts /haproxy/scripts
 COPY trusted_ips.list /etc/haproxy/trusted_ips.list
 COPY trusted_ips.map /etc/haproxy/trusted_ips.map
 # /etc/haproxy is a named volume in deployed containers, so baked-in files
-# under that path get shadowed by the volume on existing deployments.
+# under that path get shadowed by the volume on existing deployments. The
+# trusted_ips.* pair above predates that discovery and is handled by the
+# older start-up.sh guard (out of scope here). cloudflare_ips.list and
+# trusted_proxies.list are staged under /haproxy/defaults instead, so
+# start-up.sh can always read the image's baked copy regardless of what the
+# volume shadows /etc/haproxy with.
+COPY cloudflare_ips.list /haproxy/defaults/cloudflare_ips.list
+COPY trusted_proxies.list /haproxy/defaults/trusted_proxies.list
+COPY wpadmin_gate_exempt.list /haproxy/defaults/wpadmin_gate_exempt.list
 # Place errorfiles outside the volumed path; the HAProxy config references
 # them by absolute path.
 COPY errors /haproxy/errors
 RUN chmod +x /haproxy/scripts/*
 RUN pip install -r requirements.txt
+# ---------------------------------------------------------------------------
+# Build gate: no image ships unless the real haproxy binary accepts the config
+# this image's templates actually produce.
+#
+# On 2026-08-14 a template change rendered fine, passed all 13 unit tests, and
+# was rejected by HAProxy ("invalid arg 2 in converter 'regsub'"). It was only
+# caught because someone built an image by hand and ran `haproxy -c`. Nothing
+# in the build or in CI would have stopped it: .gitea/workflows/build-push.yaml
+# is checkout -> build -> push, and test-config-rollback.py's "haproxy" is a
+# shell stub that only rejects a sentinel token. In production an invalid
+# haproxy.cfg means init.py refuses to start HAProxy while the container stays
+# Up - ports 80/443 unbound, every site on the host down, /health still 200.
+#
+# This lives in the Dockerfile rather than in the workflow deliberately:
+#   * it cannot be skipped, and it protects local `docker build` too;
+#   * no workflow restructuring (build-push-action builds and pushes in one
+#     step, so gating in CI would mean splitting build from push);
+#   * it validates against the EXACT haproxy binary in this image. Line 26
+#     installs haproxy unpinned, so that binary moves between builds - this
+#     turns "the new haproxy rejects our config" from a silent production
+#     risk into a build failure.
+#
+# The unit suites run here too. They had never run anywhere automated either,
+# and they cost a few seconds.
+RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
+ && python3 /haproxy/scripts/test-trusted-proxy-gate.py \
+ && python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \
+ && python3 /haproxy/scripts/test-config-rollback.py \
+ && python3 /haproxy/scripts/test-cert-write-safety.py \
+ && python3 /haproxy/scripts/test-cert-scripts.py \
+ && python3 /haproxy/scripts/validate-rendered-config.py
 # Create log directories
 RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
 RUN chmod 755 /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log

IP_BLOCKING_API.md

@@ -508,20 +508,45 @@ curl -X POST http://localhost:8000/api/blocked-ips/sync \
 
 For advanced users, you can interact directly with HAProxy's runtime API:
 
+Three things about these commands are easy to get wrong, and each one fails
+**silently** (socat exits 0 either way — the rejection, if any, is only in the
+response body):
+
+* `/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Map commands are
+  worker commands and need the `@1` prefix. Without it the reply is
+  `Unknown command: 'add', ...`.
+* Reference the map by its **file path**, never by `#<id>`. Ids are assigned at
+  config-parse time and move on every config regeneration (on a live edge,
+  `blocked_ips.map` is id 37, `trusted_ips.map` is 10 — there is no id 0).
+  Worse, `@1 add map #0 <ip> 1` returns an **empty** reply and adds nothing.
+* Entries must carry the value `1`. `haproxy.cfg` matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block. (`add map`
+  with no value is rejected: `'add map' expects three parameters ...`.)
+
 ```bash
+MAP=/etc/haproxy/blocked_ips.map
+
 # Add IP to runtime (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map $MAP 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP from runtime
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
+
+# Confirm what actually happened (do not trust the exit status)
+echo "@1 get map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
 
 # Clear all blocked IPs from runtime
-echo "clear map #0" | socat stdio /var/run/haproxy.sock
+echo "@1 clear map $MAP" | socat stdio /tmp/haproxy-cli
 
-# Show all runtime map entries
+# Show all runtime map entries, and the map ids currently in use
-echo "show map #0" | socat stdio /var/run/haproxy.sock
+echo "@1 show map $MAP" | socat stdio /tmp/haproxy-cli
+echo "@1 show map" | socat stdio /tmp/haproxy-cli
 ```
 
+The runtime map is a **fast path only**. `/etc/haproxy/blocked_ips.map` is
+authoritative: HAProxy re-reads it on reload, so a failed runtime command
+delays a block until the next reload rather than losing it.
+
 ## Migration from ACL Method
 
 If you're upgrading from the old ACL-based method:

MIGRATION_GUIDE.md

@@ -50,14 +50,22 @@ http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }
 - **Graceful error handling**
 
 ### 2. Runtime IP Management
+Map commands go to a **worker** (`@1`), reference the map by **file path**
+(ids move between config regenerations, and `#0` silently adds nothing), and
+carry the value `1` that `map_ip(...,0) -m int gt 0` matches on:
+
 ```bash
 # Add IP without reload (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map /etc/haproxy/blocked_ips.map 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP without reload
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map /etc/haproxy/blocked_ips.map 192.168.1.100" | socat stdio /tmp/haproxy-cli
 ```
 
+socat exits 0 even when HAProxy rejects the command, so read the response body
+(or read the entry back with `@1 get map ...`) rather than the exit status.
+See IP_BLOCKING_API.md for the full set.
+
 ### 3. New API Endpoints
 
 #### Safe Config Reload

VERSION

@@ -1 +1 @@
-2026.08.1
+2026.09.1

cloudflare_ips.list

@@ -0,0 +1,34 @@
+# Cloudflare edge ranges — peers allowed to set CF-Connecting-IP / X-Real-IP /
+# X-Forwarded-For. Referenced by templates/hap_listener.tpl:
+#   acl from_trusted_proxy src -f /etc/haproxy/cloudflare_ips.list -f /etc/haproxy/trusted_proxies.list
+#
+# PUBLIC DATA — safe to commit. Source: https://www.cloudflare.com/ips-v4
+# and https://www.cloudflare.com/ips-v6. Keep in sync with the IPv4 snapshot in
+# WHP's ssl_renewal_orchestrator.php::isCloudflareIP() (that one is IPv4-only).
+# Refresh at release time; Cloudflare has not added a range since 2021.
+#
+# IPv4
+173.245.48.0/20
+103.21.244.0/22
+103.22.200.0/22
+103.31.4.0/22
+141.101.64.0/18
+108.162.192.0/18
+190.93.240.0/20
+188.114.96.0/20
+197.234.240.0/22
+198.41.128.0/17
+162.158.0.0/15
+104.16.0.0/13
+104.24.0.0/14
+172.64.0.0/13
+131.0.72.0/22
+#
+# IPv6
+2400:cb00::/32
+2606:4700::/32
+2803:f800::/32
+2405:b500::/32
+2405:8100::/32
+2a06:98c0::/29
+2c0f:f248::/32

haproxy_tarpit_config.txt

@@ -1,3 +1,37 @@
+# =============================================================================
+# NOT IMPLEMENTED. THIS FILE IS A DESIGN SKETCH THAT WAS NEVER SHIPPED.
+# =============================================================================
+#
+# Nothing here is deployed, has ever been deployed, or is rendered into
+# haproxy.cfg. The real edge config is generated from templates/*.tpl. Compare:
+#
+#   THIS FILE proposes:  store gpc0,gpc1,gpc2,http_err_rate(30s),...
+#                        plus sc-inc-gpc0/1/2 scan-escalation rules
+#   templates/hap_listener.tpl ACTUALLY has:
+#                        store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(30s)
+#
+# There is no gpc0, no gpc1, no gpc2, and no scan-escalation state anywhere on
+# the edge, and never has been.
+#
+# WHY THE BANNER. This file was mistaken for the shipped config. Two consumers
+# -- /api/security/stats in haproxy_manager.py and scripts/show-tarpit-ips.sh --
+# were written to parse gpc0/gpc1 out of `show table web`, defaulting missing
+# fields to 0. The result was a "Scan Count" column and BLOCKED/TARPITTED
+# statuses that were pure fabrication, presented to an operator as fact, for
+# the entire life of both tools. Fixed 2026-08-22; scripts/test-stick-table-
+# contract.py now fails if any consumer's field expectations and the templates'
+# `store` clauses ever drift apart again.
+#
+# IF YOU WANT TO REVIVE ANY OF THIS: the counters must be added to a template
+# `store` clause and to STICK_TABLE_FIELD_CONTRACT in haproxy_manager.py first.
+# Adding them to a consumer alone produces confident zeros, not data. Note also
+# that sc0/sc1/sc2 are all in use and HAProxy's tune.stick-counters defaults to
+# 3, and that since 2026.08.8 the edge has real per-request access logging on
+# the host at /var/log/haproxy.log -- which records what was actually denied,
+# tarpitted and rate-limited, with request references. That log is a better
+# source for most of what this sketch was reaching for.
+# =============================================================================
+
 global
     daemon
     log stdout local0 info

scripts/cert-publish-lib.sh

@@ -0,0 +1,371 @@
+# shellcheck shell=bash
+# cert-publish-lib.sh - safe publication of HAProxy certificate bundles.
+#
+# This file is SOURCED, never executed (hence no shebang / no exec bit).
+#
+# Why this exists
+# ---------------
+# renew-certificates.sh and sync-certificates.sh used to publish a bundle with
+#
+#     cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"
+#
+# where $COMBINED_FILE is the LIVE pem HAProxy is serving right now. The shell
+# truncates the destination to zero bytes when it sets up the redirect, BEFORE
+# cat ever runs, so any failure after that point (unreadable source, ENOSPC,
+# container killed mid-write) leaves a zero-length or key-less pem in place.
+# Checking cat's exit status does not help: the damage is already done.
+#
+# That matters more here than for an ordinary file because HAProxy loads
+# $SSL_CERTS_DIR as a DIRECTORY:
+#
+#     bind 0.0.0.0:443 ssl crt /etc/haproxy/certs
+#
+# It tries to load *every* file in that directory, and one unloadable file
+# fails the whole bind - i.e. HTTPS goes down for every customer on the host.
+#
+# Two consequences drive the design below:
+#   1. Assemble somewhere else and rename into place, so the live pem is either
+#      the old bundle or the new one and never a half-written one.
+#   2. NEVER create a temp file, .tmp, .backup or any other non-final file
+#      inside $SSL_CERTS_DIR. Staging and backups live in SIBLING directories.
+#
+# Directory layout (kept identical to the Python half in haproxy_manager.py):
+#   staging: $(dirname $SSL_CERTS_DIR)/cert-staging   [$CERT_STAGING_DIR]
+#   backups: $(dirname $SSL_CERTS_DIR)/cert-backups   [$CERT_BACKUP_DIR]
+# Both siblings of the certs dir, so they are on the same filesystem and the
+# final mv is a rename(2) - atomic. There is deliberately no "just write it
+# directly into the certs dir" fallback path.
+#
+# The same-filesystem property is CHECKED, not assumed (see cert_publish step
+# (c)). An earlier revision of this comment claimed a cross-device mv would
+# "fail loudly and leave the live pem alone". It does not: GNU mv falls back to
+# copy-then-unlink across filesystems, so it OPENS THE DESTINATION FOR WRITING
+# and only then discovers it cannot finish - e.g. with a full destination
+# filesystem the live pem is already overwritten when mv reports failure. That
+# is precisely the truncation this library exists to prevent, so the device
+# numbers of the staging dir and the certs dir are compared with stat(1) before
+# anything is written, and a mismatch aborts the publish. Both directories are
+# env-overridable ($CERT_STAGING_DIR / $SSL_CERTS_DIR), so "they are siblings"
+# is not something the code can take on faith. This mirrors the explicit
+# st_dev check in publish_pem_bundle() on the Python side.
+
+# Logging: the callers define their own log_info/log_error. Only provide
+# fallbacks so this library is usable standalone (e.g. from a test or a shell).
+declare -F log_info >/dev/null || log_info() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [INFO] $*"
+}
+declare -F log_error >/dev/null || log_error() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" >&2
+}
+# log_warn is not part of the callers' vocabulary; route it through log_info
+# with a loud prefix so it lands in the main log without tripping the
+# error-log monitors (scripts/monitor-errors.sh) for non-fatal conditions.
+declare -F log_warn >/dev/null || log_warn() {
+    log_info "WARNING: $*"
+}
+
+cert_staging_dir() {
+    if [ -n "${CERT_STAGING_DIR:-}" ]; then
+        echo "$CERT_STAGING_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-staging"
+    fi
+}
+
+cert_backup_dir() {
+    if [ -n "${CERT_BACKUP_DIR:-}" ]; then
+        echo "$CERT_BACKUP_DIR"
+    else
+        echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-backups"
+    fi
+}
+
+# cert_bundle_valid FILE
+#
+# Returns 0 if FILE is publishable as an HAProxy pem bundle.
+#
+# Layer 1 (structure, pure shell/grep): covers the truncation / partial-write /
+#         key-less failure modes this library exists to prevent.
+# Layer 2 (cryptographic pairing via the openssl CLI): covers what layer 1
+#         cannot see. Structural checks are weak on their own - a bundle of
+#         EMPTY pem blocks ("-----BEGIN CERTIFICATE-----" immediately followed
+#         by "-----END CERTIFICATE-----") satisfies every grep below and is
+#         caught only by openssl.
+#
+# BOTH LAYERS ARE MANDATORY. An absent openssl binary is a hard failure, not a
+# skip.
+#
+# The previous revision made layer 2 best-effort and justified it with "the
+# container image installs haproxy, certbot and socat, but not necessarily the
+# openssl CLI". That premise is false. openssl 3.x is present in the image: it
+# is a dependency of ca-certificates, which certbot needs, and
+# generate_self_signed_cert() in haproxy_manager.py shells out to `openssl req`
+# with check=True during first-run setup, so a container that reached the point
+# of publishing a bundle has always had it. The "unavailable" branch therefore
+# never fired in production, which means the fail-open was safe only by
+# accident - and a comment that justifies a decision on a false premise is
+# worse than no comment, because the next person extends the reasoning.
+#
+# Making it mandatory does mean a hypothetical image without openssl stops
+# publishing renewals. That is the right trade: it fails immediately and
+# loudly, into the monitored error log, on the first renewal run, whereas
+# publishing an unpaired or empty-block bundle takes the whole :443 bind (i.e.
+# every site on the host) down at the next reload. There is deliberately no
+# python `cryptography` fallback: the app runs on /usr/local/bin/python3 (the
+# base image's 3.12), where cryptography is NOT importable - it is installed
+# for Debian's /usr/bin/python3 as a certbot dependency. Coding to a hardcoded
+# /usr/bin/python3 would just be a second unverified premise.
+cert_bundle_valid() {
+    local file="$1"
+
+    if [ -z "$file" ]; then
+        log_error "cert_bundle_valid: no file given"
+        return 1
+    fi
+    if [ ! -f "$file" ]; then
+        log_error "Certificate bundle $file does not exist (or is not a regular file)"
+        return 1
+    fi
+
+    # openssl is checked BEFORE any content check so a broken image is reported
+    # as a broken image rather than as a bad certificate.
+    if ! command -v openssl >/dev/null 2>&1; then
+        log_error "openssl binary not found - REFUSING to publish $file." \
+                  "The cert/key pairing check (openssl x509 -pubkey vs openssl pkey -pubout)" \
+                  "is mandatory; structural checks alone cannot tell a real bundle from" \
+                  "empty pem blocks. Install openssl in this image."
+        return 1
+    fi
+
+    # Read the file ONCE and run every check against that snapshot.
+    #
+    # This used to open $file six times (two [ ] tests, three greps, two
+    # openssl invocations). cert_bundle_valid() is called on the LIVE pem in
+    # cert_publish() step (d), where a concurrent publisher can replace it
+    # between two of those opens - each open then sees a different file. The
+    # observable symptom was a spurious "private key does not match the
+    # certificate" ERROR in the monitored error log for a pair that was fine:
+    # openssl x509 read the old bundle and openssl pkey the new one.
+    local content
+    if ! content="$(cat -- "$file" 2>/dev/null)"; then
+        log_error "Certificate bundle $file could not be read"
+        return 1
+    fi
+    if [ -z "$content" ]; then
+        log_error "Certificate bundle $file is empty"
+        return 1
+    fi
+
+    # --- layer 1: structure -------------------------------------------------
+    if ! grep -qF -- '-----BEGIN CERTIFICATE-----' <<< "$content"; then
+        log_error "Certificate bundle $file contains no certificate block"
+        return 1
+    fi
+    if ! grep -qF -- '-----END CERTIFICATE-----' <<< "$content"; then
+        log_error "Certificate bundle $file has an unterminated certificate block (truncated?)"
+        return 1
+    fi
+
+    local key_begin key_end
+    key_begin="$(grep -m1 -oE -- '-----BEGIN (RSA |EC )?PRIVATE KEY-----' <<< "$content")"
+    if [ -z "$key_begin" ]; then
+        log_error "Certificate bundle $file contains no private key block"
+        return 1
+    fi
+    key_end="${key_begin/BEGIN/END}"
+    if ! grep -qF -- "$key_end" <<< "$content"; then
+        log_error "Certificate bundle $file has an unterminated private key block (truncated?)"
+        return 1
+    fi
+
+    # --- layer 2: cert/key pairing ------------------------------------------
+    # Fed from the same snapshot on stdin (openssl reads stdin when -in is
+    # omitted) rather than re-opening $file, so layer 2 judges exactly the
+    # bytes layer 1 judged. -passin pass: means an encrypted key fails fast
+    # instead of prompting - a passphrase prompt in a cron job is a hang, not
+    # an error.
+    local cert_pub key_pub
+    if ! cert_pub="$(openssl x509 -noout -pubkey 2>/dev/null <<< "$content")" \
+       || [ -z "$cert_pub" ]; then
+        log_error "Certificate bundle $file: openssl could not read the certificate"
+        return 1
+    fi
+    if ! key_pub="$(openssl pkey -pubout -passin pass: 2>/dev/null <<< "$content")" \
+       || [ -z "$key_pub" ]; then
+        log_error "Certificate bundle $file: openssl could not read the private key"
+        return 1
+    fi
+    if [ "$cert_pub" != "$key_pub" ]; then
+        log_error "Certificate bundle $file: private key does not match the certificate"
+        return 1
+    fi
+
+    return 0
+}
+
+# cert_publish CERT_FILE KEY_FILE DEST_FILE
+#
+# Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
+# partially written DEST_FILE to HAProxy. Returns 0 on success.
+#
+# On ANY failure DEST_FILE is left exactly as it was.
+cert_publish() {
+    if [ $# -ne 3 ]; then
+        log_error "cert_publish: expected 3 arguments (cert key dest), got $#"
+        return 1
+    fi
+
+    local cert_file="$1" key_file="$2" dest_file="$3"
+    local staging_dir backup_dir dest_dir tmp base
+
+    # (a) sources must exist and be non-empty before we touch anything.
+    if [ ! -s "$cert_file" ]; then
+        log_error "cert_publish: certificate $cert_file is missing or empty"
+        return 1
+    fi
+    if [ ! -s "$key_file" ]; then
+        log_error "cert_publish: private key $key_file is missing or empty"
+        return 1
+    fi
+
+    # (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
+    #     scans wholesale.
+    staging_dir="$(cert_staging_dir)"
+    dest_dir="$(dirname "$dest_file")"
+    if ! mkdir -p "$staging_dir" || ! mkdir -p "$dest_dir"; then
+        log_error "cert_publish: cannot create staging directory $staging_dir or destination directory $dest_dir"
+        return 1
+    fi
+
+    # (c) the final swap is `mv`, which is only a rename(2) - and therefore only
+    #     atomic - within one filesystem. Across filesystems GNU mv copies:
+    #     it truncates and writes the DESTINATION, then unlinks the source, so a
+    #     failure part-way through (ENOSPC is the realistic one) leaves exactly
+    #     the half-written live pem this library exists to prevent. Both paths
+    #     are env-overridable, so check instead of assuming. Same check as the
+    #     st_dev comparison in publish_pem_bundle() on the Python side.
+    local staging_dev dest_dev
+    staging_dev="$(stat -Lc '%d' "$staging_dir" 2>/dev/null)"
+    dest_dev="$(stat -Lc '%d' "$dest_dir" 2>/dev/null)"
+    if [ -z "$staging_dev" ] || [ -z "$dest_dev" ]; then
+        log_error "cert_publish: cannot stat $staging_dir and/or $dest_dir - refusing to publish $dest_file"
+        return 1
+    fi
+    if [ "$staging_dev" != "$dest_dev" ]; then
+        log_error "cert_publish: staging dir $staging_dir and destination dir $dest_dir" \
+                  "are on different filesystems, so the bundle cannot be swapped in atomically." \
+                  "Refusing to publish $dest_file (live file left untouched);" \
+                  "point CERT_STAGING_DIR at a directory on the same filesystem as $dest_dir"
+        return 1
+    fi
+
+    # Sweep temps orphaned by a kill -9 / OOM in an earlier run. Two shapes,
+    # because two publishers share this directory: mktemp's six-X suffix from
+    # this library, and `<name>.<random>.tmp` from write_config_atomically() on
+    # the Python side (tempfile.mkstemp(prefix=name + '.', suffix='.tmp')).
+    # Matching only the mktemp shape - as this did - left every Python-side
+    # temp behind forever.
+    find "$staging_dir" -maxdepth 1 -type f \
+        \( -name '*.??????' -o -name '*.tmp' \) -mmin +1440 -delete 2>/dev/null
+
+    base="$(basename "$dest_file")"
+    tmp="$(mktemp "${staging_dir}/${base}.XXXXXX" 2>/dev/null)"
+    if [ -z "$tmp" ] || [ ! -f "$tmp" ]; then
+        log_error "cert_publish: cannot create a staging file in $staging_dir"
+        return 1
+    fi
+    # 0600 while the temp file holds a private key; the final mode is matched
+    # to the file being replaced just before the swap (see below).
+    chmod 600 "$tmp" 2>/dev/null
+
+    if ! cat "$cert_file" "$key_file" > "$tmp"; then
+        log_error "cert_publish: failed to assemble $cert_file + $key_file (live $dest_file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+    if [ ! -s "$tmp" ]; then
+        log_error "cert_publish: assembled bundle for $dest_file is empty (live file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    # (d) never promote something HAProxy would choke on.
+    if ! cert_bundle_valid "$tmp"; then
+        log_error "cert_publish: assembled bundle for $dest_file failed validation (live file left untouched)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    # (e) back up the bundle we are about to replace - but only if it is itself
+    #     valid. Overwriting a good backup with garbage would turn "restore the
+    #     backup" into "restore a different broken file". Same semantics as
+    #     create_backup(require_valid=True) in haproxy_manager.py.
+    if [ -e "$dest_file" ]; then
+        backup_dir="$(cert_backup_dir)"
+        if cert_bundle_valid "$dest_file"; then
+            if mkdir -p "$backup_dir"; then
+                if ! cp -p "$dest_file" "${backup_dir}/${base}"; then
+                    log_warn "could not back up $dest_file to ${backup_dir}/${base}; publishing anyway"
+                fi
+            else
+                log_warn "could not create backup directory $backup_dir; publishing without a backup"
+            fi
+        else
+            log_warn "existing $dest_file is not a valid bundle - KEEPING the previous backup" \
+                     "in $backup_dir rather than overwriting it with an unusable one"
+        fi
+    fi
+
+    # Preserve the mode of the bundle being replaced (0644 by default, which is
+    # what `cat > file` produced under the standard umask). mktemp gives 0600,
+    # and mv carries the temp file's mode onto the destination, so without this
+    # every publish would silently tighten the live pem's permissions. Changing
+    # who can read these files is not something a write-safety fix should do as
+    # a side effect - and it must match write_config_atomically() on the Python
+    # side, which preserves the mode the same way.
+    #
+    # -L (follow symlinks) matters: stat without it reports the mode of the
+    # SYMLINK, which is 0777 on Linux and is not a permission at all. A live
+    # pem that is a symlink therefore produced a world-WRITABLE 0777 private
+    # key sitting in the crt directory. With -L we copy the mode of the file
+    # the link points at, which is the mode an operator actually chose.
+    local mode
+    mode="$(stat -Lc '%a' "$dest_file" 2>/dev/null)"
+    [ -n "$mode" ] || mode=644
+    chmod "$mode" "$tmp" 2>/dev/null
+
+    # (f) atomic swap. Same filesystem, verified in (c); if it still fails,
+    #     stop - do not fall back to writing into the certs dir.
+    if ! mv -f "$tmp" "$dest_file"; then
+        log_error "cert_publish: failed to move $tmp into place as $dest_file" \
+                  "(live file left untouched; NOT falling back to a direct write)"
+        rm -f "$tmp"
+        return 1
+    fi
+
+    return 0
+}
+
+# haproxy_config_ok
+#
+# Gate a reload on `haproxy -c`. Returns 0 if the config validates, or if we
+# cannot check (no haproxy binary) - a missing checker must not block a reload
+# that is otherwise needed, but a checker that says "no" always wins.
+haproxy_config_ok() {
+    local cfg="${HAPROXY_CONFIG:-/etc/haproxy/haproxy.cfg}"
+    local out rc
+
+    if ! command -v haproxy >/dev/null 2>&1; then
+        log_warn "haproxy binary not found - skipping 'haproxy -c' validation before reload"
+        return 0
+    fi
+
+    out="$(haproxy -c -f "$cfg" 2>&1 </dev/null)"
+    rc=$?
+    if [ $rc -eq 0 ]; then
+        return 0
+    fi
+
+    log_error "haproxy -c -f $cfg failed (exit $rc): $out"
+    return 1
+}

scripts/monitor-attacks.sh

@@ -1,136 +1,148 @@
-#!/bin/bash
+#!/usr/bin/env bash
+#
+# monitor-attacks.sh — HAProxy edge activity monitor.
+#
+# Two sections, both fed from real data:
+#   1. Current per-IP rates, from the `web` stick table (delegated to
+#      show-edge-ip-rates.sh — there is exactly one stick-table parser).
+#   2. Recent enforcement events, from the HAProxy access log.
+#
+# HISTORY / WHY THIS IS SHORTER THAN IT USED TO BE
+#   The previous version printed a "Threat Intelligence Dashboard" with
+#   fourteen categories (auth_fail, authz_fail, scanner, sql_inj, traversal,
+#   wp_brute, admin_scan, shell_att, repeat_off, manual_bl, auto_bl,
+#   glitch_rate, ...) and a composite "threat score", all parsed out of
+#   gpc(0), gpc(1), gpc(3), gpc(12), gpc(13) and glitch_rate(300s). NONE of
+#   those fields exist: the `web` table stores only conn_cur, conn_rate,
+#   http_req_rate and http_err_rate. Every category was permanently 0 and the
+#   whole dashboard printed nothing while implying it was watching. All of it
+#   has been deleted rather than "fixed" — there was no data source to fix it
+#   against.
+#
+# Usage: monitor-attacks.sh [live]
+# Env:   LOG_FILE=<path>  access log to read (default /var/log/haproxy.log)
+#        LOG_LINES=<n>    how many trailing log lines to scan (default 500)
 
-# Real-time attack monitoring for HAProxy
+set -uo pipefail
-# Shows blocked requests and suspicious activity
 
-LOG_FILE="/var/log/haproxy.log"
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-SOCKET="/tmp/haproxy-cli"
+LOG_FILE="${LOG_FILE:-/var/log/haproxy.log}"
+LOG_LINES="${LOG_LINES:-500}"
 
-echo "==================================================="
+# --- Section 1: current rates (real stick-table data) -----------------------
-echo "HAProxy Security Monitor - Real-time Attack Detection"
+show_rates() {
-echo "==================================================="
+    "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
-echo ""
-
-# Function to show current threats with HAProxy 3.0.11 metrics
-show_threats() {
-    echo "HAProxy 3.0.11 Threat Intelligence Dashboard:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 {
-            # Parse the stick table output for array-based GPC values
-            ip = $1
-            # Look for GPC array values in the data
-            auth_fail = 0
-            authz_fail = 0
-            rate_viol = 0
-            scanner = 0
-            sql_inj = 0
-            traversal = 0
-            wp_brute = 0
-            admin_scan = 0
-            shell_att = 0
-            repeat_off = 0
-            manual_bl = 0
-            auto_bl = 0
-            glitch_rate = 0
-            threat_score = 0
-
-            # Extract relevant metrics (simplified parsing)
-            if ($0 ~ /gpc\(0\)=([0-9]+)/) {
-                match($0, /gpc\(0\)=([0-9]+)/, arr); auth_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(1\)=([0-9]+)/) {
-                match($0, /gpc\(1\)=([0-9]+)/, arr); authz_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(3\)=([0-9]+)/) {
-                match($0, /gpc\(3\)=([0-9]+)/, arr); scanner = arr[1]
-            }
-            if ($0 ~ /gpc\(12\)=([0-9]+)/) {
-                match($0, /gpc\(12\)=([0-9]+)/, arr); repeat_off = arr[1]
-            }
-            if ($0 ~ /gpc\(13\)=([0-9]+)/) {
-                match($0, /gpc\(13\)=([0-9]+)/, arr); manual_bl = arr[1]
-            }
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr); glitch_rate = arr[1]
-            }
-
-            # Calculate composite threat score (simplified)
-            threat_score = auth_fail*10 + authz_fail*8 + scanner*12 + repeat_off*25 + manual_bl*100
-
-            # Only show IPs with significant threat indicators
-            if (auth_fail > 0 || authz_fail > 0 || scanner > 0 || repeat_off > 0 || manual_bl > 0 || glitch_rate > 0) {
-                threat_level = "LOW"
-                if (threat_score >= 100) threat_level = "CRITICAL"
-                else if (threat_score >= 50) threat_level = "HIGH"
-                else if (threat_score >= 20) threat_level = "MEDIUM"
-
-                printf "%-15s [%8s] Score:%-3d Auth:%-2d Authz:%-2d Scanner:%-1d Repeat:%-1d Glitch:%-2d\n",
-                       ip, threat_level, threat_score, auth_fail, authz_fail, scanner, repeat_off, glitch_rate
-            }
-        }' | head -15
-
-    echo ""
-    echo "Top HTTP/2 Protocol Violators:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 && $0 ~ /glitch/ {
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr)
-                if (arr[1] > 2) {
-                    printf "%-15s glitch_rate:%-3s\n", $1, arr[1]
-                }
-            }
-        }' | head -5
-    echo "---------------------------------------------------"
 }
 
-# Function to show recent blocks
+# --- Section 2: recent enforcement events (real access-log data) ------------
 show_recent_blocks() {
-    echo "Recent Blocked Requests:"
+    echo "Recent enforcement events (last $LOG_LINES log lines):"
-    tail -100 "$LOG_FILE" 2>/dev/null | \
+    echo
-        grep -E "(bot_scanner|scan_admin|scan_shells|sql_injection|directory_traversal|rate_abuse|tarpit|denied|403)" | \
+
-        tail -10 | \
+    if [ ! -f "$LOG_FILE" ] || [ ! -r "$LOG_FILE" ]; then
-        awk '{
+        cat <<MSG
-            if (match($0, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+/)) {
+  Access log not readable at: $LOG_FILE
-                ip = substr($0, RSTART, RLENGTH)
+
-                gsub(/:.*/, "", ip)
+  This is expected INSIDE the haproxy-manager container: HAProxy logs to
-                reason = ""
+  syslog on the DOCKER HOST, and the file lives on the host, not in here.
-                if ($0 ~ /bot_scanner/) reason = "BOT_SCANNER"
+  Read it from the host instead:
-                else if ($0 ~ /scan_admin/) reason = "ADMIN_SCAN"
+
-                else if ($0 ~ /scan_shells/) reason = "SHELL_SCAN"
+    grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
-                else if ($0 ~ /sql_injection/) reason = "SQL_INJECTION"
+    grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limited / blocked
-                else if ($0 ~ /directory_traversal/) reason = "DIR_TRAVERSAL"
+    grep -a 'cip=<IP>'  /var/log/haproxy.log | tail -50     # one client IP
-                else if ($0 ~ /rate_abuse/) reason = "RATE_ABUSE"
+    grep -a 'id=<uuid>' /var/log/haproxy.log                # one request reference
-                else if ($0 ~ /tarpit/) reason = "TARPIT"
+                                                            # (the UUID on the block page)
-                else if ($0 ~ /denied/) reason = "DENIED"
+    tail -f /var/log/haproxy.log | grep -aE ' (PT|PR)--'    # live
-                else if ($0 ~ /403/) reason = "BLOCKED"
+
-                printf "[%s] %-15s %s\n", strftime("%H:%M:%S"), ip, reason
+  Or point this script at a copy:  LOG_FILE=/path/to/haproxy.log $0
+MSG
+        echo
+        return 0
+    fi
+
+    printf "%-15s %-16s %-4s %-5s %-28s %s\n" "TIME" "CLIENT IP" "CODE" "TERM" "HOST" "REQUEST / REQUEST-ID"
+    printf "%s\n" "-----------------------------------------------------------------------------------------------------"
+
+    local found
+    found=$(tail -n "$LOG_LINES" "$LOG_FILE" 2>/dev/null | awk '
+    {
+        status = ""; term = ""; cip = ""; host = ""; id = ""; ts = ""; req = ""
+
+        # %tr is bracketed: [22/Aug/2026:10:11:12.345] -> keep HH:MM:SS
+        # No {n} interval expressions here: not every awk in a slim Debian
+        # image supports them. Spelled out instead.
+        if (match($0, /\[[0-9][0-9]\/[A-Za-z][A-Za-z][A-Za-z]\/[0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/)) {
+            ts = substr($0, RSTART + 13, 8)
+        }
+
+        # Anchor on the %TR/%Tw/%Tc/%Tr/%Ta timers block: %ST follows it, and
+        # the termination state (%tsc) is 4 fields further on (%B %CC %CS %tsc).
+        for (i = 1; i <= NF; i++) {
+            if ($i ~ /^[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+$/) {
+                status = $(i + 1)
+                term = $(i + 5)
+                break
             }
-        }'
+        }
-    echo ""
+
+        # Only enforcement outcomes: tarpit (PT--), deny (PR--), 403, 429.
+        if (!(term ~ /^PT/ || term ~ /^PR/ || status == "403" || status == "429")) next
+
+        for (i = 1; i <= NF; i++) {
+            if (substr($i, 1, 4) == "cip=")  cip  = substr($i, 5)
+            if (substr($i, 1, 5) == "host=") host = substr($i, 6)
+            if (substr($i, 1, 3) == "id=")   id   = substr($i, 4)
+        }
+
+        if (match($0, /"[A-Z]+ [^"]*"/)) {
+            req = substr($0, RSTART + 1, RLENGTH - 2)
+            if (length(req) > 42) req = substr(req, 1, 41) "..."
+        }
+
+        if (cip == "") cip = "-"
+        if (host == "") host = "-"
+        if (term == "") term = "-"
+        if (status == "") status = "-"
+        printf "%-15s %-16s %-4s %-5s %-28s %s\n", ts, cip, status, term, host, req
+        if (id != "" && id != "-") printf "%-15s %s\n", "", "  id=" id
+        n++
+    }
+    END { if (n == 0) print "(no tarpit/deny/403/429 events in the scanned window)" }
+    ')
+    printf '%s\n' "$found"
+    echo
+    echo "TERM = HAProxy termination state: PT-- tarpit, PR-- deny (incl. WAF/rate limit)."
+    echo "id=  = request reference; it is printed on the block page and is how a"
+    echo "       customer support ticket correlates to an exact request here."
 }
 
-# Monitor mode selection
+banner() {
-if [ "$1" == "live" ]; then
+    echo "==================================================="
-    echo "Live monitoring mode - Press Ctrl+C to exit"
+    echo "HAProxy Edge Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
-    echo ""
+    echo "==================================================="
+    echo
+}
 
+if [ "${1:-}" = "live" ]; then
+    echo "Live monitoring mode - Press Ctrl+C to exit"
     while true; do
         clear
-        echo "==================================================="
+        banner
-        echo "HAProxy Security Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
+        show_rates || true
-        echo "==================================================="
+        echo
-        echo ""
-        show_threats
-        echo ""
         show_recent_blocks
         sleep 5
     done
 else
-    # Single run mode
+    banner
-    show_threats
+    rc=0
-    echo ""
+    show_rates || rc=$?
+    echo
     show_recent_blocks
-    echo ""
+    echo
-    echo "Tip: Run with 'live' parameter for continuous monitoring"
+    echo "Tip: run with 'live' for a refreshing view."
     echo "Usage: $0 [live]"
-fi
+    # Propagate a stick-table read failure: if the rates section could not be
+    # produced, this run did NOT report what it claims to report.
+    exit "$rc"
+fi

scripts/renew-certificates.sh

@@ -8,6 +8,7 @@ LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
 ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
 DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
 SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
+LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
 
 # Logging functions
 log_info() {
@@ -18,6 +19,18 @@ log_error() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
 }
 
+# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
+# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
+# uses this script's logging rather than its own fallbacks.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=cert-publish-lib.sh
+if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
+    . "${SCRIPT_DIR}/cert-publish-lib.sh"
+else
+    log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
+    exit 1
+fi
+
 log_info "Starting certificate renewal process"
 
 # Run certbot renewal — don't exit on failure, some certs may have
@@ -42,7 +55,7 @@ fi
 mkdir -p "$SSL_CERTS_DIR"
 
 # Get all SSL-enabled domains from database
-DOMAINS=$(find /etc/letsencrypt/live/ -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
+DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
 
 if [ -z "$DOMAINS" ]; then
     log_info "No SSL-enabled domains found"
@@ -54,13 +67,16 @@ UPDATED=0
 FAILED=0
 
 while read -r domain; do
-    CERT_FILE="/etc/letsencrypt/live/${domain}/fullchain.pem"
+    CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
-    KEY_FILE="/etc/letsencrypt/live/${domain}/privkey.pem"
+    KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
     if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
-        # Combine cert and key into single file for HAProxy
+        # Assemble in a staging dir and rename into place. NEVER redirect into
-        if cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"; then
+        # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
+        # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
+        # takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
+        if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
             log_info "Updated certificate for $domain"
             UPDATED=$((UPDATED + 1))
         else
@@ -77,6 +93,13 @@ log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
 
 # Reload HAProxy if any certificates were updated
 if [ $UPDATED -gt 0 ]; then
+    # Never reload onto unvalidated material: a reload that fails to load the
+    # certs directory drops HTTPS for every site on this host.
+    if ! haproxy_config_ok; then
+        log_error "HAProxy configuration does not validate - refusing to reload after certificate renewal"
+        exit 1
+    fi
+
     if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
         log_info "HAProxy reloaded successfully"
     else
@@ -85,5 +108,18 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# A per-domain publication failure is a real failure and must be reported as
+# one. This script used to `exit 0` no matter how many domains failed, so
+# "0 updated, 12 failed" - a host that has completely stopped publishing
+# renewals - looked identical to a clean run to cron, to
+# host-renew-certificates.sh (which branches on this exit code) and to any
+# external monitoring. The first visible symptom would have been certificates
+# expiring. The loop above deliberately continues past a failed domain so the
+# others still get published; the status is reported here instead.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate renewal process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate renewal process completed"
 exit 0

scripts/show-edge-ip-rates.sh

@@ -0,0 +1,314 @@
+#!/usr/bin/env bash
+#
+# show-edge-ip-rates.sh — real, current per-IP rate counters from the HAProxy
+# `web` stick table.
+#
+# WHAT THIS CAN TELL YOU
+#   The `web` stick table (templates/hap_listener.tpl) stores exactly four
+#   counters per client IP:
+#       conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
+#   Those are INSTANTANEOUS values — the current concurrency and the current
+#   sliding-window rates. This script prints them, and nothing else.
+#
+# WHAT THIS CANNOT TELL YOU
+#   * Who has been tarpitted, denied, or rate-limited. The stick table stores
+#     NO history and NO counter of past enforcement actions. It has no gpc0 /
+#     gpc1 / gpc(N) / gpc_rate / glitch_rate columns at all — any tool that
+#     claims to read them from this table is fabricating numbers.
+#   * Anything about an IP that has gone quiet: entries expire after 10m.
+#
+#   Real enforcement events live in the HAProxy ACCESS LOG, which is on the
+#   DOCKER HOST at /var/log/haproxy.log (it does NOT exist inside this
+#   container). The log-format carries the HAProxy termination state plus
+#   cip= (real client IP), host=, ua= and id= (the request UUID shown on the
+#   block page, which correlates with customer support tickets).
+#
+#   Ready to run ON THE HOST:
+#       # last 20 tarpitted (PT--) or denied (PR--) requests
+#       grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+#       # everything HAProxy answered 429/403 to, newest last
+#       grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20
+#       # everything for one client IP
+#       grep -a 'cip=203.0.113.7' /var/log/haproxy.log | tail -50
+#       # look up one request reference from a support ticket
+#       grep -a 'id=<uuid-from-the-block-page>' /var/log/haproxy.log
+#
+# USAGE
+#   show-edge-ip-rates.sh [-a|--all]
+#     -a, --all   also show rows whose counters are all zero (off by default:
+#                 a table with hundreds of idle entries is pure noise)
+#
+# ENVIRONMENT
+#   SHOW_ALL=1               same as --all
+#   HAPROXY_SOCKET=<path>    override the CLI socket (default /tmp/haproxy-cli)
+#   HAPROXY_TABLE_DUMP=<file>
+#                            parse a previously captured `show table web` dump
+#                            from a file instead of talking to the socket.
+#                            Supported seam for offline analysis of a captured
+#                            support bundle, and for testing this parser.
+#
+# NOTE ON THE SOCKET
+#   /tmp/haproxy-cli is HAProxy's MASTER CLI socket, so worker commands need an
+#   `@1` prefix. Without it HAProxy answers "Unknown command: 'show' ..." AND
+#   socat still exits 0 — so exit status is worthless here and this script
+#   inspects the RESPONSE BODY instead.
+
+set -euo pipefail
+
+SOCKET="${HAPROXY_SOCKET:-/tmp/haproxy-cli}"
+TABLE="web"
+
+# Fields this script expects the `web` stick table to store. Keep on ONE line
+# in this exact NAME=(a b c d) shape — the contract test greps for it, and the
+# parser below is driven entirely by it.
+EXPECTED_FIELDS=(conn_cur conn_rate http_req_rate http_err_rate)
+
+# Which of EXPECTED_FIELDS to sort on (descending). Falls back to the first
+# field if this name is not in the list.
+SORT_FIELD="http_req_rate"
+
+SHOW_ALL="${SHOW_ALL:-0}"
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -a|--all) SHOW_ALL=1 ;;
+        -h|--help) sed -n '2,60p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
+        *) echo "Unknown argument: $1" >&2; echo "Usage: $0 [-a|--all]" >&2; exit 2 ;;
+    esac
+    shift
+done
+
+die() { echo "ERROR: $*" >&2; exit 1; }
+
+# First non-blank line of a blob.
+#
+# Deliberately NOT `printf ... | sed -n '/./{p;q;}'`. sed quits after the first
+# match and closes the pipe; on a real 550-entry table dump printf is still
+# writing and takes SIGPIPE, so under `set -o pipefail` the whole command
+# substitution returns 141 and `set -e` kills the script -- silently, with no
+# output at all. That is the same class of failure this script exists to stop
+# hiding, so it does not get to happen here. A plain read loop has no pipeline
+# and no early close.
+first_nonblank() {
+    local line
+    while IFS= read -r line || [ -n "$line" ]; do
+        case "$line" in
+            *[![:space:]]*) printf '%s\n' "$line"; return 0 ;;
+        esac
+    done <<EOF
+$1
+EOF
+    return 0
+}
+
+# Return 0 if the CLI response body is a rejection rather than table data.
+# Checked on the body because socat's exit status is 0 either way.
+body_is_rejected() {
+    local first
+    first=$(first_nonblank "$1")
+    case "$first" in
+        "Unknown command"*|"No such table"*|"Permission denied"*) return 0 ;;
+        *) return 1 ;;
+    esac
+}
+
+send_cmd() {
+    printf '%s\n' "$1" | socat stdio "$SOCKET" 2>/dev/null
+}
+
+# ---------------------------------------------------------------- fetch data
+BODY=""
+SOURCE=""
+if [ -n "${HAPROXY_TABLE_DUMP:-}" ]; then
+    [ -r "$HAPROXY_TABLE_DUMP" ] || die "HAPROXY_TABLE_DUMP is set but '$HAPROXY_TABLE_DUMP' is not readable."
+    BODY=$(cat "$HAPROXY_TABLE_DUMP")
+    SOURCE="file $HAPROXY_TABLE_DUMP"
+else
+    [ -S "$SOCKET" ] || die "HAProxy CLI socket not found at $SOCKET (is HAProxy running, and are you inside the haproxy-manager container?)"
+    command -v socat >/dev/null 2>&1 || die "socat is not installed; cannot talk to $SOCKET"
+
+    # Master socket form first, then the plain stats-socket form.
+    BODY=$(send_cmd "@1 show table $TABLE" || true)
+    SOURCE="socket $SOCKET (@1 show table $TABLE)"
+    if [ -z "${BODY//[[:space:]]/}" ] || body_is_rejected "$BODY"; then
+        FALLBACK=$(send_cmd "show table $TABLE" || true)
+        if [ -n "${FALLBACK//[[:space:]]/}" ] && ! body_is_rejected "$FALLBACK"; then
+            BODY="$FALLBACK"
+            SOURCE="socket $SOCKET (show table $TABLE)"
+        else
+            echo "ERROR: HAProxy rejected BOTH '@1 show table $TABLE' and 'show table $TABLE'." >&2
+            echo "  @1 response : $(first_nonblank "$BODY")" >&2
+            echo "  bare response: $(first_nonblank "$FALLBACK")" >&2
+            echo "  Check the socket is HAProxy's CLI and that the table '$TABLE' exists" >&2
+            echo "  (a config reload without the frontend would drop it)." >&2
+            exit 1
+        fi
+    fi
+fi
+
+# ------------------------------------------------------------- header checks
+HEADER=$(first_nonblank "$BODY")
+case "$HEADER" in
+    "# table: $TABLE,"*) : ;;
+    *)
+        echo "ERROR: unexpected first line from '$SOURCE'." >&2
+        echo "  expected it to start with: # table: $TABLE," >&2
+        echo "  got                      : $HEADER" >&2
+        exit 1
+        ;;
+esac
+
+TBL_SIZE=$(printf '%s\n' "$HEADER" | sed -n 's/.*size:\([0-9]*\).*/\1/p')
+TBL_USED=$(printf '%s\n' "$HEADER" | sed -n 's/.*used:\([0-9]*\).*/\1/p')
+[ -n "$TBL_SIZE" ] || TBL_SIZE="?"
+[ -n "$TBL_USED" ] || TBL_USED="?"
+
+# ------------------------------------------------------------------- parsing
+# awk emits:
+#   W \t <field>:<window-seconds-or-dash> ...       (one line, from first row)
+#   R \t <sortkey> \t <ip> \t <value per EXPECTED_FIELDS in order>
+# and exits 1 after reporting any row missing an expected field.
+PARSED=""
+if ! PARSED=$(printf '%s\n' "$BODY" | awk -v fieldlist="${EXPECTED_FIELDS[*]}" -v sortfield="$SORT_FIELD" '
+BEGIN {
+    nf = split(fieldlist, F, " ")
+    sortidx = 1
+    for (i = 1; i <= nf; i++) if (F[i] == sortfield) sortidx = i
+    wprinted = 0
+}
+/^#/ { next }
+!/key=/ { next }
+{
+    split("", val, " "); split("", win, " ")
+    for (i = 1; i <= NF; i++) {
+        tok = $i
+        p = index(tok, "=")
+        if (p == 0) continue
+        lhs = substr(tok, 1, p - 1)
+        rhs = substr(tok, p + 1)
+        b = index(lhs, "(")
+        if (b > 0) {
+            nm = substr(lhs, 1, b - 1)
+            win[nm] = substr(lhs, b + 1, length(lhs) - b - 1)
+        } else {
+            nm = lhs
+            win[nm] = ""
+        }
+        val[nm] = rhs
+    }
+
+    missing = ""
+    for (i = 1; i <= nf; i++) if (!(F[i] in val)) missing = missing (missing == "" ? "" : ", ") F[i]
+    if (missing != "") {
+        printf "ERROR: stick table row is missing expected field(s): %s\n", missing > "/dev/stderr"
+        printf "  offending row: %s\n", $0 > "/dev/stderr"
+        printf "  this script expects the web table to store: %s\n", fieldlist > "/dev/stderr"
+        print  "  Those expectations and the templates/hap_listener.tpl `store` clause have DRIFTED." > "/dev/stderr"
+        print  "  Fix one or the other; refusing to print 0 for a counter HAProxy never reported." > "/dev/stderr"
+        exit 1
+    }
+    if (!("key" in val)) {
+        printf "ERROR: stick table row has no key= field: %s\n", $0 > "/dev/stderr"
+        exit 1
+    }
+
+    if (!wprinted) {
+        line = "W"
+        for (i = 1; i <= nf; i++) {
+            w = win[F[i]]
+            if (w ~ /^[0-9]+$/) w = sprintf("%g", w / 1000); else w = "-"
+            line = line "\t" F[i] ":" w
+        }
+        print line
+        wprinted = 1
+    }
+
+    nonzero = 0
+    row = ""
+    for (i = 1; i <= nf; i++) {
+        v = val[F[i]]
+        if (v + 0 != 0) nonzero = 1
+        row = row "\t" v
+    }
+    printf "R\t%s\t%s\t%d%s\n", val[F[sortidx]] + 0, val["key"], nonzero, row
+}
+'); then
+    exit 1
+fi
+
+# ------------------------------------------------------------------ printing
+WINSPEC=$(printf '%s\n' "$PARSED" | sed -n 's/^W\t//p' || true)
+
+echo "==================================================================="
+echo "  HAProxy edge IP rates — table '$TABLE' (current values only)"
+echo "==================================================================="
+echo "Source     : $SOURCE"
+echo "Tracked    : ${TBL_USED} of ${TBL_SIZE} slots in use"
+if [ "$SHOW_ALL" = "1" ]; then
+    echo "Filter     : showing ALL tracked IPs"
+else
+    echo "Filter     : showing only IPs with a non-zero counter (use --all for every row)"
+fi
+echo
+
+# Column headers, with each counter's window rendered in SECONDS (HAProxy
+# reports the window in milliseconds, e.g. conn_rate(10000) = 10s).
+HDR=$(printf "%-18s" "IP Address")
+i=0
+for f in "${EXPECTED_FIELDS[@]}"; do
+    w=$(printf '%s\n' "$WINSPEC" | tr '\t' '\n' | sed -n "s/^${f}://p")
+    if [ -n "$w" ] && [ "$w" != "-" ]; then
+        label="${f}/${w}s"
+    else
+        label="$f"
+    fi
+    HDR="$HDR $(printf '%18s' "$label")"
+    i=$((i + 1))
+done
+echo "$HDR"
+printf '%s\n' "$HDR" | sed 's/./-/g'
+
+ROWS=$(printf '%s\n' "$PARSED" | sed -n 's/^R\t//p' || true)
+shown=0
+if [ -n "$ROWS" ]; then
+    while IFS=$'\t' read -r sortkey ip nonzero rest; do
+        [ -n "${ip:-}" ] || continue
+        if [ "$SHOW_ALL" != "1" ] && [ "$nonzero" = "0" ]; then
+            continue
+        fi
+        line=$(printf "%-18s" "$ip")
+        oldifs="$IFS"; IFS=$'\t'
+        # shellcheck disable=SC2086
+        set -- $rest
+        IFS="$oldifs"
+        for v in "$@"; do
+            line="$line $(printf '%18s' "$v")"
+        done
+        echo "$line"
+        shown=$((shown + 1))
+    done < <(printf '%s\n' "$ROWS" | sort -t"$(printf '\t')" -k1,1nr)
+fi
+
+if [ "$shown" -eq 0 ]; then
+    if [ "$SHOW_ALL" = "1" ]; then
+        echo "(no IPs currently tracked)"
+    else
+        echo "(no IP currently has a non-zero counter — re-run with --all to list idle entries)"
+    fi
+fi
+
+echo
+echo "==================================================================="
+echo "These are CURRENT values. The table keeps no history and no record of"
+echo "past tarpits/denials. For actual enforcement events, read the access"
+echo "log ON THE DOCKER HOST (it does not exist in this container):"
+echo "  grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20     # tarpit / deny"
+echo "  grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20    # rate-limit / block"
+echo "  grep -a 'cip=<IP>' /var/log/haproxy.log | tail -50        # one client"
+echo "  grep -a 'id=<uuid>' /var/log/haproxy.log                  # one request reference"
+echo
+echo "Operator actions (via the MASTER CLI socket — the @1 prefix is required):"
+echo "  printf '@1 show table $TABLE key <IP>\\n'  | socat stdio $SOCKET"
+echo "  printf '@1 set table $TABLE key <IP> data.http_req_rate 0\\n' | socat stdio $SOCKET"
+echo "  printf '@1 clear table $TABLE key <IP>\\n' | socat stdio $SOCKET   # drop one entry"
+echo "  printf '@1 clear table $TABLE\\n'          | socat stdio $SOCKET   # drop ALL entries"
+echo "==================================================================="

scripts/show-tarpit-ips.sh

@@ -1,123 +1,35 @@
-#!/bin/bash
+#!/usr/bin/env bash
-
-# Script to display IPs that have been tarpitted by HAProxy 3.0
-# Uses HAProxy stats socket to query stick-table data
 #
-# Usage in Docker container:
+# DEPRECATED SHIM — kept so existing docs/runbooks/muscle memory keep working.
-#   docker exec -it haproxy-manager /haproxy/scripts/show-tarpit-ips.sh
+#
+# This script used to print a "Tarpitted IPs Report" with a "Scan Count" and a
+# BLOCKED / SILENT-DROP / TARPIT status per IP, all derived from gpc0 and gpc1
+# stick-table columns. Those columns DO NOT EXIST: the `web` table
+# (templates/hap_listener.tpl) stores only conn_cur, conn_rate, http_req_rate
+# and http_err_rate. The old parser defaulted every missing field to 0, so the
+# whole report was fabricated — every IP showed "Scan Count 0 / Normal"
+# regardless of what it was actually doing.
+#
+# The stick table also keeps NO history, so nothing in it can identify who was
+# tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST
+# at /var/log/haproxy.log (it does not exist inside this container):
+#   grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+#   grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limit / block
+#
+# What IS knowable from the stick table — the current per-IP rates — is printed
+# by show-edge-ip-rates.sh, which this shim now runs.
 
-SOCKET="/tmp/haproxy-cli"
+set -euo pipefail
 
-# Check if socket exists
+cat >&2 <<'NOTE'
-if [ ! -S "$SOCKET" ]; then
+NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits.
-    echo "Error: HAProxy socket not found at $SOCKET"
+      The HAProxy stick table stores no history and no gpc0/gpc1 counters, so
-    echo "Make sure HAProxy is running with stats socket enabled"
+      the old "Scan Count"/"BLOCKED" columns were fabricated numbers.
-    exit 1
+      Actual tarpit/deny events are in /var/log/haproxy.log ON THE HOST:
-fi
+        grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+      Running show-edge-ip-rates.sh instead (current rates, real values):
 
-echo "==================================================================="
+NOTE
-echo "                    HAProxy Tarpitted IPs Report                   "
-echo "==================================================================="
-echo
-echo "Showing IPs tracked in the stick-table with scan detection counters:"
-echo "(gpc0 = total scan attempts, gpc1 = escalation level)"
-echo
 
-# In HAProxy 3.0, we need to use the proper process prefix
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-# The web frontend table is in the worker process, not master
+exec "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
-# First check which process has the table
-# Note: grep for actual worker line, not the header
-PROCESS_ID=$(echo "show proc" | socat stdio "$SOCKET" 2>/dev/null | grep -E '^[0-9]+.*worker' | awk '{print $1}' | head -1)
-
-if [ -z "$PROCESS_ID" ]; then
-    echo "Error: Could not find HAProxy worker process"
-    echo "Try: echo 'show proc' | socat stdio $SOCKET"
-    exit 1
-fi
-
-# Show stick-table entries from the web frontend using the worker process
-# Use printf to avoid bash history expansion issues with !
-printf "@!%s show table web\n" "${PROCESS_ID}" | socat stdio "$SOCKET" 2>/dev/null | {
-    # Skip the header line
-    read header
-    
-    # Check if we got an error or empty response
-    if echo "$header" | grep -q "No such table"; then
-        echo "Error: Table 'web' not found. HAProxy may need to be reloaded."
-        exit 1
-    fi
-    
-    has_data=false
-    echo "IP Address           | Scan Count | Level | HTTP Err Rate | Status"
-    echo "---------------------|------------|-------|---------------|------------------"
-    
-    # Process each line
-    while IFS= read -r line; do
-        # Skip empty lines and comments
-        if [ -z "$line" ] || echo "$line" | grep -q "^#"; then
-            continue
-        fi
-        
-        # HAProxy 3.0 format: 0x... key=<ip> use=... exp=... gpc0=... gpc1=... http_err_rate(10s)=...
-        if echo "$line" | grep -q "key="; then
-            has_data=true
-            
-            # Extract IP and counters
-            ip=$(echo "$line" | grep -o 'key=[^ ]*' | cut -d'=' -f2)
-            gpc0=$(echo "$line" | grep -o 'gpc0=[0-9]*' | cut -d'=' -f2)
-            gpc1=$(echo "$line" | grep -o 'gpc1=[0-9]*' | cut -d'=' -f2)
-            err_rate=$(echo "$line" | grep -o 'http_err_rate([^)]*=[0-9]*' | grep -o '[0-9]*$')
-            
-            # Set defaults if values are empty
-            gpc0=${gpc0:-0}
-            gpc1=${gpc1:-0}
-            err_rate=${err_rate:-0}
-            
-            # Determine status based on scan count and escalation
-            status=""
-            if [ "$gpc0" -ge 100 ]; then
-                status="BLOCKED (429)"
-            elif [ "$gpc0" -ge 60 ]; then
-                status="SILENT-DROP"
-            elif [ "$gpc0" -ge 40 ]; then
-                if [ "$gpc1" -ge 2 ]; then
-                    status="SILENT-DROP (repeat)"
-                else
-                    status="TARPIT 10s"
-                fi
-            elif [ "$gpc0" -ge 25 ]; then
-                status="TARPIT 10s"
-            else
-                status="Normal"
-            fi
-            
-            # Format output
-            printf "%-20s | %10s | %5s | %13s | %s\n" "$ip" "$gpc0" "$gpc1" "$err_rate/10s" "$status"
-        fi
-    done
-    
-    if [ "$has_data" = false ]; then
-        echo "(No IPs currently tracked - table is empty)"
-    fi
-}
-
-echo
-echo "==================================================================="
-echo "Legend:"
-echo "  - Scan Count 25-39: Low scanner → TARPIT 10s delay"
-echo "  - Scan Count 40-59: Medium scanner → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "  - Scan Count 60-99: High scanner → SILENT-DROP (immediate disconnect)"
-echo "  - Scan Count 100+:  Critical scanner → BLOCKED (429 response)"
-echo "  - Burst (5+ in 10s): → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "==================================================================="
-echo "Note: Only counts suspicious scripts/configs, NOT missing images/fonts/CSS"
-echo "Note: IPs are tracked for 1 hour since last activity"
-echo
-echo "To clear a specific IP from the table:"
-echo "  printf '@!${PROCESS_ID} del table web key <IP>\\n' | socat stdio $SOCKET"
-echo
-echo "To clear all entries:"
-echo "  printf '@!${PROCESS_ID} clear table web\\n' | socat stdio $SOCKET"
-echo
-echo "Debug: Worker PID is ${PROCESS_ID}"
-echo

scripts/start-up.sh

@@ -22,6 +22,39 @@ mkdir -p /etc/haproxy
 [ -f /etc/haproxy/trusted_ips.list ] || : > /etc/haproxy/trusted_ips.list
 [ -f /etc/haproxy/trusted_ips.map ]  || : > /etc/haproxy/trusted_ips.map
 
+# cloudflare_ips.list is SHIPPED DATA: it must always match what this image
+# bakes in (/haproxy/defaults), so a Cloudflare range refresh actually reaches
+# existing hosts instead of being permanently shadowed by the volume.
+# Overwrite it from the baked copy on every start.
+#
+# trusted_proxies.list and wpadmin_gate_exempt.list are OPERATOR DATA:
+# operators add entries directly on the server and those must survive
+# restarts/recreates. Seed each from the baked copy only when it's missing;
+# never overwrite an existing one.
+#
+# All branches fall back to an empty file if the baked default is somehow
+# absent, because "acl ... -f <missing file>" is a fatal HAProxy config
+# error -- the list files must exist unconditionally by the time HAProxy starts.
+if [ -f /haproxy/defaults/cloudflare_ips.list ]; then
+    cp /haproxy/defaults/cloudflare_ips.list /etc/haproxy/cloudflare_ips.list
+else
+    [ -f /etc/haproxy/cloudflare_ips.list ] || : > /etc/haproxy/cloudflare_ips.list
+fi
+if [ ! -f /etc/haproxy/trusted_proxies.list ]; then
+    if [ -f /haproxy/defaults/trusted_proxies.list ]; then
+        cp /haproxy/defaults/trusted_proxies.list /etc/haproxy/trusted_proxies.list
+    else
+        : > /etc/haproxy/trusted_proxies.list
+    fi
+fi
+if [ ! -f /etc/haproxy/wpadmin_gate_exempt.list ]; then
+    if [ -f /haproxy/defaults/wpadmin_gate_exempt.list ]; then
+        cp /haproxy/defaults/wpadmin_gate_exempt.list /etc/haproxy/wpadmin_gate_exempt.list
+    else
+        : > /etc/haproxy/wpadmin_gate_exempt.list
+    fi
+fi
+
 cron &
 
 # Phase 1: container init

scripts/sync-certificates.sh

@@ -8,6 +8,7 @@ LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
 ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
 DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
 SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
+LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
 
 # Logging functions
 log_info() {
@@ -18,13 +19,25 @@ log_error() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
 }
 
+# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
+# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
+# uses this script's logging rather than its own fallbacks.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=cert-publish-lib.sh
+if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
+    . "${SCRIPT_DIR}/cert-publish-lib.sh"
+else
+    log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
+    exit 1
+fi
+
 log_info "Starting certificate sync process"
 
 # Ensure SSL certs directory exists
 mkdir -p "$SSL_CERTS_DIR"
 
 # Get all SSL-enabled domains from database
-DOMAINS=$(find /etc/letsencrypt/live/ -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
+DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
 
 if [ -z "$DOMAINS" ]; then
     log_info "No SSL-enabled domains found"
@@ -36,13 +49,16 @@ UPDATED=0
 FAILED=0
 
 while read -r domain; do
-    CERT_FILE="/etc/letsencrypt/live/${domain}/fullchain.pem"
+    CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
-    KEY_FILE="/etc/letsencrypt/live/${domain}/privkey.pem"
+    KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
     COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
 
     if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
-        # Combine cert and key into single file for HAProxy
+        # Assemble in a staging dir and rename into place. NEVER redirect into
-        if cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"; then
+        # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
+        # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
+        # takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
+        if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
             log_info "Updated certificate for $domain"
             UPDATED=$((UPDATED + 1))
         else
@@ -59,6 +75,13 @@ log_info "Certificate sync completed: $UPDATED updated, $FAILED failed"
 
 # Reload HAProxy if any certificates were updated
 if [ $UPDATED -gt 0 ]; then
+    # Never reload onto unvalidated material: a reload that fails to load the
+    # certs directory drops HTTPS for every site on this host.
+    if ! haproxy_config_ok; then
+        log_error "HAProxy configuration does not validate - refusing to reload after certificate sync"
+        exit 1
+    fi
+
     if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
         log_info "HAProxy reloaded successfully"
     else
@@ -67,5 +90,12 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# See the matching block in renew-certificates.sh: a run in which every domain
+# failed to publish must not look like a clean run to its caller.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate sync process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate sync process completed"
 exit 0

scripts/test-cert-scripts.py

@@ -0,0 +1,958 @@
+#!/usr/bin/env python3
+"""Regression tests for the certificate publication shell scripts.
+
+Why this file exists
+--------------------
+renew-certificates.sh and sync-certificates.sh published a bundle with
+
+    cat "$CERT_FILE" "$KEY_FILE" > "$COMBINED_FILE"
+
+where $COMBINED_FILE is the pem HAProxy is serving *right now*. The shell
+truncates the destination when it opens the redirect, before cat runs, so any
+failure after that point - unreadable source key, ENOSPC, container killed
+mid-write - left a zero-length or key-less pem behind. The exit status of cat
+was checked, but by then the live file was already destroyed.
+
+HAProxy loads $SSL_CERTS_DIR as a DIRECTORY (`bind :443 ssl crt /etc/haproxy/
+certs`) and tries to load every file in it, so a single unloadable file fails
+the whole bind: HTTPS down for every customer on the host.
+
+The fix (scripts/cert-publish-lib.sh) assembles into a sibling staging dir,
+validates, backs up the outgoing bundle into a sibling backup dir, and renames
+into place. These tests pin the observable guarantees:
+
+  * a successful publish replaces the live pem and archives the old one;
+  * a FAILED publish leaves the previous, still-valid pem byte-for-byte intact;
+  * nothing that is not a final *.pem ever appears in the certs directory;
+  * HAProxy is not reloaded when `haproxy -c` rejects the configuration.
+
+Running
+-------
+    python3 scripts/test-cert-scripts.py            # tests the repo checkout
+    HAPROXY_MANAGER_DIR=/some/other/tree \
+        python3 scripts/test-cert-scripts.py        # tests another tree
+
+Self-contained stdlib unittest - no pytest, no venv, no bats, and nothing is
+imported from the application. The scripts are driven as subprocesses with
+every path they touch redirected by environment variable (SSL_CERTS_DIR,
+LETSENCRYPT_LIVE_DIR, CERT_STAGING_DIR, CERT_BACKUP_DIR, LOG_FILE,
+ERROR_LOG_FILE, HAPROXY_CONFIG) and stub `certbot`, `socat` and `haproxy`
+binaries on PATH.
+
+The certificate material below is a real self-signed test certificate with its
+matching key (plus a second, unrelated key for the mismatch case), embedded as
+constants so the tests need no openssl to *create* material. openssl IS needed
+to run them: the library's cert/key pairing check is mandatory (it is the only
+layer that can reject a bundle of empty pem blocks), so without the binary
+every publish is refused by design. The image ships openssl 3.x.
+
+The acceptance bar for this file is a green run INSIDE the built image, as
+root, which is where these scripts actually execute - not on a workstation.
+Several failure modes are invisible outside the container (root ignores the
+directory permissions one test used to rely on) and one was actively
+destructive there; see _cleanup_tmp().
+"""
+
+import os
+import re
+import shutil
+import stat
+import subprocess
+import tempfile
+import textwrap
+import time
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+SCRIPTS_DIR = os.path.join(MODULE_DIR, 'scripts')
+LIB = os.path.join(SCRIPTS_DIR, 'cert-publish-lib.sh')
+
+BROKEN_TOKEN = '__BROKEN__'
+DOMAIN = 'test.example.com'
+
+# --- test key material -------------------------------------------------------
+# openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \
+#         -subj /CN=test.example.com
+TEST_CERT = """\
+-----BEGIN CERTIFICATE-----
+MIIDFzCCAf+gAwIBAgIUeaz/lNOESOTHsB3Y97+Xja7Fy4gwDQYJKoZIhvcNAQEL
+BQAwGzEZMBcGA1UEAwwQdGVzdC5leGFtcGxlLmNvbTAeFw0yNjA4MDYxNTQyMTda
+Fw0zNjA4MDMxNTQyMTdaMBsxGTAXBgNVBAMMEHRlc3QuZXhhbXBsZS5jb20wggEi
+MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDF5GL7Gjn+UnPFy5sqP2k4XHth
+mkWFZj+mjK6cDhbBXYt60NrwVdrrgOFydMC75VeUceFxG/5GD7wrXZP23xzbnWKm
+7FxfOSmr4y+1rVEZwi8IeWEz3W6C6y5rjZsCI+pBgdna+aJSpTQZHPfDpNtQm5vl
+enj5BfizYixinORxm9kvXMGXV+Cw1CJkqB3mzScwWt40EtQoVxekebf8B7i4ZHyx
+xT6/xwF+WY8OliZkY1pdqncoTLUAYcaE/HR/ojJKmSVIq1GswZE/y3E56LIwq+wJ
+eGbgH46a+86z+VO2UX1jbad1kWKBCsRoOpaybZDEWAYTOqahW7kOH7umTUsRAgMB
+AAGjUzBRMB0GA1UdDgQWBBTIqa3BNEjjcxkhXqnRwInrLM9yijAfBgNVHSMEGDAW
+gBTIqa3BNEjjcxkhXqnRwInrLM9yijAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
+DQEBCwUAA4IBAQB/lYzXb5PI3magMz/IXmwTsMCrVSdaUYEIKLEJggmbGxqpwO1a
+iYagWZ/5H3B9KDvNQA+L4FkMJ726ZkdGEH/vkwvTAuhwU2NSWcbRJ8DK5u3Q4rnJ
+VswPcW5njUF9mQq0NPX/PMCeOoFDEI8+RrgQZxtHhopwuKOgVA6HRBINKdEZJlrp
+oLLQrHDNVLMYTclNHG6kBg0lOHUV31TgkJQ8kMgtq0WQX7RseKR10QKgN5iOBomU
+3+y713Ibpac5B1zw5l3LjE/59xFteFbDENr2+A5VGhVNVZC+bs+YTYTzeAsGB0e3
+MQ+XJMJq3kaJmQ+QcTrRaKMtoMz2h1AIRbLd
+-----END CERTIFICATE-----
+"""
+
+# The private key that matches TEST_CERT.
+TEST_KEY = """\
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDF5GL7Gjn+UnPF
+y5sqP2k4XHthmkWFZj+mjK6cDhbBXYt60NrwVdrrgOFydMC75VeUceFxG/5GD7wr
+XZP23xzbnWKm7FxfOSmr4y+1rVEZwi8IeWEz3W6C6y5rjZsCI+pBgdna+aJSpTQZ
+HPfDpNtQm5vlenj5BfizYixinORxm9kvXMGXV+Cw1CJkqB3mzScwWt40EtQoVxek
+ebf8B7i4ZHyxxT6/xwF+WY8OliZkY1pdqncoTLUAYcaE/HR/ojJKmSVIq1GswZE/
+y3E56LIwq+wJeGbgH46a+86z+VO2UX1jbad1kWKBCsRoOpaybZDEWAYTOqahW7kO
+H7umTUsRAgMBAAECggEAA9SnFFqGfR1yO4XUlfmmgyZqJoJmnl2TdZlDT4bHyrwx
+dSIKHO0iiNzEsHMhYHnA62EVdruunUNUdofwE28v9zHZnSbV5mt8OqUyERue5mdf
+gvPbjXYXu63LBx61fZH9qME3WwFqUpx7UNGiW62LJ8ktWEC5ywNCNFG+D3YfR3Iu
+v3PdIFXwkpJMaXO+42JSoSVoiqxlONqNiqcdQj64iYgCYdNxcgdLy+mHGrf1hAZs
+dom2Jp0oEM4ZdOQu3Z6uyEyPsECiz1PdQjzagaEfPWtKCQk0kY8DDCn5X2xQif9w
+3xeaISqho7bQgSo4IEgWHTP6V0v7brTP7VcK+gyMyQKBgQD15FrE4XAL89NBQ6iX
+m2l6quZv5tIUtreuYxsOpFIMU22zfsRZOvA7sJR3ufOJ+b7tFHuJ8DAJUpK08Vvg
+A930/LW9wUsY42d54XKIO/8DTsIrmjppoGdshs3axJQkqfN0zQkhqfbwaXRJ9X2k
+Fvax+5jftaIaw0hQdWLnfTmGmQKBgQDOBue89D8THYl/LvDGg6WoVyK3ljBMK2s3
+4BljeZCJdWAjtido13Mltubc6YVHScmVoIZKTmx+fCjdQ3y1t92vZZSBeLsXhfFr
+N+IOGZu3ZmJu64x3OukSYQ7x6agi5yP3+7k0siZgxOMXJRQDYZUcHxIHDrSGLeLZ
+sj7LnbvLOQKBgE8murE1gEPYsOAJT3O96y45ZQQQYP+Z8XaJIGSOMHsXP/DPlZTD
+jCEqrh/8E5EOe48FUN8OGehmVCM6rkBl/kSmNDpoxiu0x9JL5/pClcwSxh4S/0qQ
+/7nHiuwo6ycCLgQjHBViCMNKrsw/4bm4SqDwRD1+0jebNOPxZWzuul3BAoGAZij0
+ZjSyxhbCZEdxau5CiYvTkjct8cch3k4IKNRRwGdsaajcN9eFqHDeXzKIPQYwqDo1
+/MiQcdO9K6JYR39JtLxo/B5Sn2JyiJjoRdea6EEjlB7GwyR6B/wKvhf/oHb+1euD
+NccU0q0ucf6XwulzV8NsXAWFrHc6YnpJOwwW37kCgYAkEyjUc73jImiTyf/IXVOD
+UHlRXZPvwtZUuPGe4RI0Gds97tKnvXnvFsPIRCOGfVzZ8z79DGiQ8TR2a0hgZec1
+Mo3J2dCjlv4Q6ACjHkCA1cmi13OHUPnpaeesrOk+SpEVJfR2k4qRWH4z3oxbPO/q
+TDFnjWHSgjkDMrxVHZ8wPg==
+-----END PRIVATE KEY-----
+"""
+
+# A perfectly valid RSA key that has nothing to do with TEST_CERT.
+UNRELATED_KEY = """\
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDE7mosrsdBhdn1
+ZIErYmMuPU69ws51JdyqwtRZlV2uLLby0dOfJpFKZPuBPEqTqDimd9N4FSL0CGjG
+zXG23RYd9AbdSoorFORrUxeiPzFnbz4v38srGHckOS9ozKAbPLACUEjWMX1NwAZ+
+wGlSz3cWYTWVztFCUIxpvLVR87PSpTnpdCXIj0EABOc6WoLBLE+v2knZOa63LKg6
+GryInD43CnWFBKpH0gdgWqh+ie3NFMumLR8M3lZq2Mk0EFgVWrxPnJobYvOmavNp
+NBR3ZugR4X57c1YFyLkYXQIdhxuYV1ZAY6NgmAIsaiCKFqdksF5AJSV7FZNBM43r
+33ae/R5hAgMBAAECggEAIgNYsL9+OEWtU8ooWi0r2q5plXJaVNb1gkPUx+U5sS3V
+amKNxbj0UrBW1Scr7U1afXwIOP8TkqkKKb4NpCsS2RkO/3USoKbC3fuTwyjdeFM5
+Hy0sytR2rXm4A8aF57ZnYvrpXZ9eGEnwhT9n4Y7mL2YaSnXWZDkDy3Z1rcIk/p5P
+jUo4UQyzD/9Gab1stcBbGv+66B3mlVdRVJor5+tGn6zmr4TjvqupAexuXd+Q6+1L
+OG4e2bAUmuOVOa+w8Xo4vwkiXSDLVEsl1z1x4Bcrv61bIg0rbZAeqcZ4EUPyrEyR
+RpaaOLAiwdzjOj5pQvHAF2q/++IlY6xPhotTr11tvQKBgQD37C2UQRGRo622+2kx
+qgsdA6jPM9vCE9S4aS4qaOj8XSMMCu6bQW5lU48DaONvXxDmT6x9IrgR3QN1Iz/t
+17kUCCghuviEP1RjnahBRQqZR38KdDzKhaWt9jgdjCun9MzUXIaQRTFlFGA1z1pz
+apZ9a8/ehYPSk4pv9h06/B3IjQKBgQDLWPBuhj203QOmwVF1v1k4yyuJ7UBqtRaY
+I9jMW93sB2hPs6Se10UroQHlF95IHeRXvNKH/UXuAILIJMN8oTI4uU7t60shJHvI
+o14RzEZoUQjES7BBVhePglndJmYIKoKKAX5lSFe9Lk0ei9B+1Dle5Q+9ZeP09cp4
+vVcGvOgqJQKBgQCtxe1srO8TlhZ821uwY+/GNnpsQX0XW68OUyr4rvAfc2jNWBxG
+1mX6v8bOLQa9WXUO+Wl9jIhYfQGfaUW2AC7Jy63VdqgaigksiaUVmr8DEQoK2c6C
+ZYrrlFlg3I78+qlXcEMhfF5S6yVEkkJkA6HX52mcHxl2z9OJBokWfwChQQKBgAzH
+xzy7FS/D4FHfvpXu89Wc91yQ28aZIRVo01xsvbLy+DxiJwuQrhlC4lKawG656jsV
+dAn2AiomQBICNYMkwnpMM0jCzBMGLv16PxRRSW+PAEUOGMLSfWKYp7s9iZYjzdaM
+p3wIIvOR8GjmErGV9xEexnF58OzZceNKyyhyQQk9AoGAbvjJvYLOhQpAbbqErNNv
+LK1P+TngKpukRHXjiUpPVEGNhN6krBBJCBWzY7ucrIy6jz8UBy7SbITBy7qIKhxZ
+PVP7WVATMWEeW1AfdBfCYDI4jFKAD8SLECby45nRBuBllYdQnW1gBzLcCulCwB+w
+FV0RvuQPDYkqsx8ibqpSv7c=
+-----END PRIVATE KEY-----
+"""
+
+# A SECOND, unrelated but internally consistent pair. Needed by the
+# concurrency test: two bundles that are each perfectly valid but whose keys
+# differ, so a validator that reads the certificate from one and the key from
+# the other reports a mismatch. Two bundles sharing key material - which is
+# what PREVIOUS_BUNDLE and NEW_BUNDLE are - cannot detect that at all.
+# openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
+#         -subj /CN=second.example.com
+TEST_CERT_2 = """\
+-----BEGIN CERTIFICATE-----
+MIIDGzCCAgOgAwIBAgIUKQAAvrVkden7gIg2zYMLb6dO1c8wDQYJKoZIhvcNAQEL
+BQAwHTEbMBkGA1UEAwwSc2Vjb25kLmV4YW1wbGUuY29tMB4XDTI2MDgwNjE2NTcz
+NVoXDTM2MDgwMzE2NTczNVowHTEbMBkGA1UEAwwSc2Vjb25kLmV4YW1wbGUuY29t
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv2pfTOvN2zrZJ6d+biiD
+VOczMqanuHPBCy2CnTBif+7VWf8AaywRzQ3ShfhmalVRNEFMn0MSO+GocmH71Ve1
+nl89oGiJPmvX1lpncbgM692ddhP9ez4xUeNj+QAWp9VBZhInNuM4Pawv5BPngtpj
+2MGXf3ZlBSli8Ng7jBo1fTMT3bh8GcE1rIPRvmUuQwFIt2eGnLR8jQd+xGelhAjG
+nnXtlc+ebo4r2OjljNgvtdUknBZdpiZXmjdFzyClYTeMuEen2uwMpJNc0wLbRjcU
+khVF3nw4jUnkOhWH3JYGAoWslJyEqZSANwt/eOHwXgyVuxg31bCl297iskW0IRZL
+wQIDAQABo1MwUTAdBgNVHQ4EFgQU7HH8GacJzc2j9s2UJCPwykgrIckwHwYDVR0j
+BBgwFoAU7HH8GacJzc2j9s2UJCPwykgrIckwDwYDVR0TAQH/BAUwAwEB/zANBgkq
+hkiG9w0BAQsFAAOCAQEAYE5jHX1dK091jVsFSZDdiw9AU5rrk8XpF1yuPmDisRnE
+dJ4QQq3dzWXRnp0bzZnq7fdfiEz1m39zVixov7WFp24QhenD2n5K7/wew7RpXTnA
+pAGBEdsGvBJ+3MgkRYklXCM9f9f4z21xXRNZ+BwBcM25D+gR4b+PRMQR6BhZx5R+
+y2jQsoM68cFjRApFWgmji4pBjg/eOaZMBCfVTjP+npVyqG7UtV5EyYXwPgPa/rm2
+FoP+eftzP6dszBEonkIVyyvkdscI4Wkr8hw3S0R/TP8l9lTnvN1HC3o7Es5VY53R
+swNAXWBlgm0N7A96ISLtQjgvOfeMRTCSjxW9pm0wJA==
+-----END CERTIFICATE-----
+"""
+
+TEST_KEY_2 = """\
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC/al9M683bOtkn
+p35uKINU5zMypqe4c8ELLYKdMGJ/7tVZ/wBrLBHNDdKF+GZqVVE0QUyfQxI74ahy
+YfvVV7WeXz2gaIk+a9fWWmdxuAzr3Z12E/17PjFR42P5ABan1UFmEic24zg9rC/k
+E+eC2mPYwZd/dmUFKWLw2DuMGjV9MxPduHwZwTWsg9G+ZS5DAUi3Z4actHyNB37E
+Z6WECMaede2Vz55ujivY6OWM2C+11SScFl2mJleaN0XPIKVhN4y4R6fa7Aykk1zT
+AttGNxSSFUXefDiNSeQ6FYfclgYChayUnISplIA3C3944fBeDJW7GDfVsKXb3uKy
+RbQhFkvBAgMBAAECggEABQLqy6eedRPt31sLOMFEDkzbZdFHOMpZeuqThsNmjo7z
+t9diokgeD4ZQXimbEQqsZYDAtmFGdiEp86I9JQBk/4TUiqFhHrOADBe1jQAptjfs
+iyIb51gIPnjZ1RBA5Al8zohy28T9h1+Z7+/OeCvcgLyVAixf/U9pU8D1El9cu9zv
+4q4WJPB5Tgkq+YwcmeuT8LzsKoSDmPQjFVY9v+gz6hoVUVyP6gswnlFnKjNcmfZU
+0CKP90sCAc5mKZv7RyGG920LDU4u2ggnQoK05GhXK8R3amJmoAF3i+xGeTRvNEKe
+wDC7NinTG2WDVo6y/FCvFKs0+qqlKww1u39fq166rQKBgQDtjPjvkDkSYFcWFznn
+sfsxN5R1cLpxdutLZhfvtpHIj5NkQbmOk0r3LbDAwU+UNVQ6F8jWQVHUsni9jAY6
+3RNRF4ZabC51aqg/Ssj7d5mE4kj7y6Ch/nnSXIogRxLG4bWo2rtG1M2Dt3Ef08R4
+6gjcp9ZmELyVx7H4yXFJGIaDdQKBgQDOSCHFK/ggZzzy4iC7DOq1lrbR+jyk95rh
+F5EGzyAkgJ4uYc9TCPkUDxXjWL17r61/obfbW4znaV9fHEKpz3R54osDXln3oLea
+BBWkJI3ANe8iNrGDE4FN9to4DdUMFsWX/WEiRyBPIqDy9DTyadblMLTNZjNPuLmg
+ZJOB3tRZnQKBgQCjeglya9Eq2Uv1MuSxk2VnmHU9YOed4BXLHKZKXFz1JgFr1GNL
+QAguFK536FDIkO62z9lxwR/8fRnkb7F13uBFRSg7oAlU2qKQc/nePI9UyJkrVxXj
+hYn2f6K61c6ROZFXc7e/5gDMrXhXS9gA0iZpG8PLF6eAeB39NTwV7p/bZQKBgQCV
+v5eEY58FJu0ABVhtcbsRiA+/70EHIRi2Pz1xC/vxg81RLoArb2AiR7FEEa+8kpQJ
+C4VFIPjxJXWuvf1G+Os9cFAqadw1/95JWJ29QywEVSL8W2gSF57O0l0oRCJdXEql
+Q7O4BppV2HWu6clmEZ+HUgxu77pgLWHUJi9PIExXoQKBgEq6rYBXmsSQermnPTOn
+YFx7c2ns97hsjYIbs497+gPW4/xQWwsN76t60SWqjXV4DHJCpi5Tnjo2fk5/OzK+
+HfshC9CKFDk8T0KGQWwaPwqP/OYbqOA88IlJ7xbPdSuJkjefHCtVVEtao6+HNgzm
+lniLtDMpU0MLPgB98ClQ4HDA
+-----END PRIVATE KEY-----
+"""
+
+# The bundle already on disk when a run starts. Same key material, plus a
+# trailing marker so "the live file was replaced" and "the old file was
+# archived" can be told apart byte-for-byte.
+PREVIOUS_BUNDLE = TEST_CERT + TEST_KEY + '# previous bundle\n'
+NEW_BUNDLE = TEST_CERT + TEST_KEY
+OTHER_BUNDLE = TEST_CERT_2 + TEST_KEY_2
+
+# --- stub binaries -----------------------------------------------------------
+STUB_CERTBOT = """\
+#!/bin/sh
+# Test stub for certbot: pretend there was nothing to renew.
+echo "No renewals were attempted."
+exit 0
+"""
+
+STUB_SOCAT = """\
+#!/bin/sh
+# Test stub for socat: record that a reload was attempted, and what was sent.
+{ printf 'socat %s <<' "$*"; cat; printf '>>\\n'; } >> "$SOCAT_LOG"
+exit 0
+"""
+
+STUB_HAPROXY = """\
+#!/bin/sh
+# Test stub for the haproxy binary: `haproxy -c -f FILE` rejects any config
+# containing %(token)s, which is how the tests inject an invalid config.
+cfg=""
+while [ $# -gt 0 ]; do
+  case "$1" in -f) cfg="$2"; shift ;; esac
+  shift
+done
+if [ -n "$cfg" ] && grep -q '%(token)s' "$cfg" 2>/dev/null; then
+  echo "[ALERT] parsing [$cfg:1] : unknown keyword '%(token)s'" >&2
+  exit 1
+fi
+exit 0
+""" % {'token': BROKEN_TOKEN}
+
+GOOD_HAPROXY_CFG = textwrap.dedent("""\
+    global
+        daemon
+    defaults
+        mode http
+    frontend fe
+        bind 0.0.0.0:443 ssl crt /etc/haproxy/certs
+""")
+
+
+def write(path, content, mode=None):
+    with open(path, 'w') as fh:
+        fh.write(content)
+    if mode is not None:
+        os.chmod(path, mode)
+    return path
+
+
+def read(path):
+    with open(path) as fh:
+        return fh.read()
+
+
+class CertScriptFixture(unittest.TestCase):
+    """An isolated fake /etc/haproxy + /etc/letsencrypt plus stub binaries."""
+
+    def setUp(self):
+        # Without the library every "the library rejects X" assertion in this
+        # file can be satisfied by bash exiting 127, so make its absence a
+        # failure of every test rather than a silent pass of several.
+        self.assertTrue(os.path.isfile(LIB),
+                        f'{LIB} is missing - nothing below tests anything')
+
+        self.tmp = tempfile.mkdtemp(prefix='haproxy-cert-test-')
+        self.addCleanup(self._cleanup_tmp)
+
+        self.bindir = os.path.join(self.tmp, 'bin')
+        os.makedirs(self.bindir)
+        write(os.path.join(self.bindir, 'certbot'), STUB_CERTBOT, 0o755)
+        write(os.path.join(self.bindir, 'socat'), STUB_SOCAT, 0o755)
+        write(os.path.join(self.bindir, 'haproxy'), STUB_HAPROXY, 0o755)
+        self.socat_log = os.path.join(self.tmp, 'socat-invocations.log')
+
+        # Mirrors the real layout: certs dir, with staging/backups as SIBLINGS.
+        self.haproxy_dir = os.path.join(self.tmp, 'etc', 'haproxy')
+        self.certs_dir = os.path.join(self.haproxy_dir, 'certs')
+        self.staging_dir = os.path.join(self.haproxy_dir, 'cert-staging')
+        self.backup_dir = os.path.join(self.haproxy_dir, 'cert-backups')
+        os.makedirs(self.certs_dir)
+
+        self.le_live = os.path.join(self.tmp, 'etc', 'letsencrypt', 'live')
+        self.domain_dir = os.path.join(self.le_live, DOMAIN)
+        os.makedirs(self.domain_dir)
+        self.src_cert = write(os.path.join(self.domain_dir, 'fullchain.pem'), TEST_CERT)
+        self.src_key = write(os.path.join(self.domain_dir, 'privkey.pem'), TEST_KEY)
+
+        self.live_pem = os.path.join(self.certs_dir, DOMAIN + '.pem')
+        self.haproxy_cfg = write(os.path.join(self.haproxy_dir, 'haproxy.cfg'),
+                                 GOOD_HAPROXY_CFG)
+        self.log_file = os.path.join(self.tmp, 'haproxy-manager.log')
+        self.error_log = os.path.join(self.tmp, 'haproxy-manager-errors.log')
+
+    def _cleanup_tmp(self):
+        # A test may have chmod 000'd a fixture file, which would stop rmtree.
+        #
+        # SYMLINKS ARE SKIPPED, and that is not a nicety. os.chmod() FOLLOWS
+        # symlinks, and the openssl-availability tests below build a stripped
+        # PATH directory out of symlinks to real system binaries (/usr/bin/cat,
+        # /usr/bin/chmod, ...). Walking those with chmod 0600 as root - which is
+        # how this container runs - stripped the exec bit from a dozen core
+        # binaries of the machine running the tests, chmod itself included, so
+        # it could not even be undone from inside the container: every later
+        # test failed with "/usr/bin/grep: Permission denied" and certificate
+        # publishing stayed dead until the container was recreated. It never
+        # showed up on a workstation because an unprivileged chmod of a
+        # root-owned file fails EPERM and was swallowed by `except OSError`.
+        # This file ships in the image (COPY scripts /haproxy/scripts), so
+        # running it in place is a thing an operator will do.
+        for root, dirs, files in os.walk(self.tmp):
+            for name in files:
+                path = os.path.join(root, name)
+                if os.path.islink(path):
+                    continue
+                try:
+                    os.chmod(path, 0o600)
+                except OSError:
+                    pass
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    # -- helpers ---------------------------------------------------------
+    def env(self, **overrides):
+        env = dict(os.environ)
+        env.update({
+            'PATH': self.bindir + os.pathsep + os.environ['PATH'],
+            'SSL_CERTS_DIR': self.certs_dir,
+            'LETSENCRYPT_LIVE_DIR': self.le_live,
+            'CERT_STAGING_DIR': self.staging_dir,
+            'CERT_BACKUP_DIR': self.backup_dir,
+            'LOG_FILE': self.log_file,
+            'ERROR_LOG_FILE': self.error_log,
+            'HAPROXY_CONFIG': self.haproxy_cfg,
+            'SOCAT_LOG': self.socat_log,
+        })
+        env.update(overrides)
+        return env
+
+    def run_script(self, name, **env_overrides):
+        path = os.path.join(SCRIPTS_DIR, name)
+        self.assertTrue(os.path.exists(path), f'{path} does not exist')
+        return subprocess.run(['bash', path], env=self.env(**env_overrides),
+                              capture_output=True, text=True, timeout=120)
+
+    def bundle_is_valid(self, path):
+        """Ask the shipped library whether HAProxy could use this bundle."""
+        return subprocess.run(
+            ['bash', '-c', '. "$1"; cert_bundle_valid "$2"', '_', LIB, path],
+            env=self.env(), capture_output=True, text=True).returncode == 0
+
+    def reload_attempted(self):
+        return os.path.exists(self.socat_log) and os.path.getsize(self.socat_log) > 0
+
+    def logs(self):
+        text = ''
+        for path in (self.log_file, self.error_log):
+            if os.path.exists(path):
+                text += read(path)
+        return text
+
+    def seed_previous_bundle(self, content=PREVIOUS_BUNDLE):
+        return write(self.live_pem, content)
+
+    def assert_certs_dir_is_clean(self):
+        """HAProxy loads this directory wholesale: only final *.pem may be here."""
+        entries = sorted(os.listdir(self.certs_dir))
+        strays = [e for e in entries if not e.endswith('.pem')]
+        self.assertEqual(strays, [],
+                         f'non-.pem files left in the certs directory HAProxy '
+                         f'loads wholesale: {strays} (dir: {entries})')
+
+    def assert_no_staging_leftovers(self):
+        if os.path.isdir(self.staging_dir):
+            self.assertEqual(sorted(os.listdir(self.staging_dir)), [],
+                             'staging file was not cleaned up')
+
+    def assert_rejected(self, result, because):
+        """The library refused, FOR THE STATED REASON.
+
+        `assertNotEqual(rc, 0)` on its own proves nothing about this library.
+        Delete cert-publish-lib.sh and `. "$1"` fails, cert_bundle_valid is
+        never defined, bash exits 127 - and a bare rc!=0 assertion passes. Four
+        tests in TestCertPublishLibrary were doing exactly that; they were
+        pinning "some bash pipeline failed", not "the bundle was rejected".
+        """
+        output = result.stdout + result.stderr
+        self.assertNotIn('command not found', output,
+                         'the shell could not find the function under test - '
+                         'this asserts nothing about the library')
+        self.assertNotEqual(127, result.returncode,
+                            f'exit 127 means "no such command", not "rejected": {output}')
+        self.assertNotEqual(0, result.returncode,
+                            f'expected a rejection, got success: {output}')
+        self.assertIn(because, output,
+                      f'rejected, but not for the expected reason '
+                      f'({because!r} not in output): {output}')
+
+
+class CertScriptBehaviour:
+    """Behaviour shared by renew-certificates.sh and sync-certificates.sh.
+
+    A mixin rather than a TestCase so the cases are collected once per concrete
+    script, not a third time for the base class.
+    """
+
+    SCRIPT = None
+
+    def run_it(self, **env_overrides):
+        return self.run_script(self.SCRIPT, **env_overrides)
+
+    # -- happy path ------------------------------------------------------
+    def test_publish_replaces_live_pem_and_archives_the_previous_one(self):
+        self.seed_previous_bundle()
+        result = self.run_it()
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE,
+                         'live pem is not the newly assembled cert+key')
+        backup = os.path.join(self.backup_dir, DOMAIN + '.pem')
+        self.assertTrue(os.path.exists(backup),
+                        f'previous bundle was not archived to {backup}')
+        self.assertEqual(read(backup), PREVIOUS_BUNDLE,
+                         'the archived bundle is not the one that was replaced')
+        self.assertIn('1 updated, 0 failed', self.logs())
+        self.assertTrue(self.reload_attempted(),
+                        'HAProxy was never reloaded after a successful update')
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+
+    def test_first_publish_works_with_no_previous_bundle(self):
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+        self.assert_certs_dir_is_clean()
+
+    # -- THE HEADLINE ----------------------------------------------------
+    def test_unreadable_source_key_leaves_the_previous_bundle_intact(self):
+        """The bug this whole change exists for.
+
+        Pre-fix: `cat cert key > live.pem` truncated live.pem before cat ran,
+        so an unreadable key left a key-less (or empty) pem in the directory
+        HAProxy loads wholesale -> the :443 bind fails -> every site on the
+        host loses HTTPS. Checking cat's exit status did not undo that.
+        """
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        self.assertTrue(self.bundle_is_valid(self.live_pem),
+                        'fixture precondition: the seeded bundle must be valid')
+
+        how = self.break_source_key()
+
+        result = self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         f'the live pem was damaged by a failed publish ({how}); '
+                         f'HAProxy would fail to load the certs directory')
+        self.assertTrue(self.bundle_is_valid(self.live_pem),
+                        'the pem left on disk is no longer a usable bundle')
+        logs = self.logs()
+        self.assertIn(f'Failed to combine certificate for {DOMAIN}', logs)
+        self.assertIn('0 updated, 1 failed', logs)
+        self.assertFalse(self.reload_attempted(),
+                         'HAProxy was reloaded even though nothing was updated')
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+        # This used to assert returncode == 0 with the comment "a per-domain
+        # failure should not change the exit code", codifying the script's
+        # `exit 0`. That is wrong and it is the dangerous kind of wrong: a host
+        # where EVERY domain fails to publish looked, to cron and to
+        # host-renew-certificates.sh (which branches on this exit code),
+        # exactly like a clean run. Nothing would notice until the certificates
+        # expired. Continuing past a failed domain so the others still get
+        # published is right; reporting success afterwards is not.
+        self.assertNotEqual(result.returncode, 0,
+                            'a domain that failed to publish must be reported '
+                            'in the exit code, not just in the log')
+
+    def break_source_key(self):
+        """Make reading the source key fail, however this environment allows.
+
+        chmod 000 is the faithful reproduction (file present, `-f` true, cat
+        fails), but it is a no-op for root, so as root we truncate the key
+        instead: pre-fix that is even nastier, because `cat` then *succeeds*
+        and silently publishes a key-less pem.
+        """
+        if os.geteuid() == 0:
+            write(self.src_key, '')
+            return 'zero-length source key (running as root)'
+        os.chmod(self.src_key, 0o000)
+        return 'unreadable source key (chmod 000)'
+
+    # -- other ways to end up with an unusable bundle --------------------
+    def test_cert_only_bundle_is_rejected(self):
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(self.src_key, TEST_CERT)  # no private key block at all
+
+        self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         'a key-less bundle was published over the live pem')
+        self.assertIn('0 updated, 1 failed', self.logs())
+        self.assertFalse(self.reload_attempted())
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+
+    def test_truncated_certificate_block_is_rejected(self):
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(self.src_cert, TEST_CERT.split('\n')[0] + '\nMIIDFzCCAf+gAwIBA\n')
+
+        self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         'a truncated certificate was published over the live pem')
+        self.assertIn('0 updated, 1 failed', self.logs())
+        self.assert_certs_dir_is_clean()
+
+    def test_mismatched_key_is_rejected(self):
+        if shutil.which('openssl') is None:
+            self.skipTest('openssl CLI not available: without it every publish '
+                          'is refused, so this test could not tell a pairing '
+                          'rejection from a missing-checker rejection')
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(self.src_key, UNRELATED_KEY)
+
+        self.run_it()
+
+        self.assertEqual(read(self.live_pem), before,
+                         'a bundle whose key does not match the cert was published')
+        self.assertIn('does not match the certificate', self.logs())
+        self.assertIn('0 updated, 1 failed', self.logs())
+        self.assert_certs_dir_is_clean()
+
+    # -- the certs directory is HAProxy's, not ours ----------------------
+    def test_no_stray_files_in_certs_dir_after_success_or_failure(self):
+        self.seed_previous_bundle()
+        self.run_it()
+        self.assert_certs_dir_is_clean()
+        self.assertEqual(sorted(os.listdir(self.certs_dir)), [DOMAIN + '.pem'])
+
+        self.break_source_key()
+        self.run_it()
+        self.assert_certs_dir_is_clean()
+        self.assertEqual(sorted(os.listdir(self.certs_dir)), [DOMAIN + '.pem'])
+        self.assert_no_staging_leftovers()
+
+    def test_staging_and_backup_dirs_are_outside_the_certs_dir(self):
+        """Belt and braces: even with the defaults, nothing lands under certs/."""
+        self.seed_previous_bundle()
+        # Drop the explicit overrides so the derived defaults are exercised.
+        env = {'CERT_STAGING_DIR': '', 'CERT_BACKUP_DIR': ''}
+        self.run_it(**env)
+
+        self.assert_certs_dir_is_clean()
+        self.assertEqual(sorted(os.listdir(self.certs_dir)), [DOMAIN + '.pem'])
+        self.assertTrue(
+            os.path.exists(os.path.join(self.haproxy_dir, 'cert-staging')),
+            'default staging dir is not the documented sibling of the certs dir')
+        self.assertTrue(
+            os.path.exists(os.path.join(self.haproxy_dir, 'cert-backups',
+                                        DOMAIN + '.pem')),
+            'default backup dir is not the documented sibling of the certs dir')
+
+    # -- reload gating ---------------------------------------------------
+    def test_reload_is_not_attempted_when_haproxy_config_is_invalid(self):
+        write(self.haproxy_cfg, GOOD_HAPROXY_CFG + BROKEN_TOKEN + '\n')
+
+        result = self.run_it()
+
+        self.assertFalse(self.reload_attempted(),
+                         'HAProxy was reloaded with a configuration that '
+                         '`haproxy -c` rejects')
+        self.assertNotEqual(result.returncode, 0,
+                            'refusing to reload must be a loud, non-zero exit')
+        self.assertIn('does not validate', self.logs())
+
+    def test_reload_happens_when_the_config_validates(self):
+        self.run_it()
+        self.assertTrue(self.reload_attempted())
+        self.assertIn('reload', read(self.socat_log))
+
+    def test_reload_is_not_attempted_when_nothing_was_updated(self):
+        shutil.rmtree(self.domain_dir)
+        result = self.run_it()
+        self.assertEqual(result.returncode, 0)
+        self.assertFalse(self.reload_attempted())
+
+
+class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture):
+    SCRIPT = 'renew-certificates.sh'
+
+
+class TestSyncCertificates(CertScriptBehaviour, CertScriptFixture):
+    SCRIPT = 'sync-certificates.sh'
+
+
+class TestCertPublishLibrary(CertScriptFixture):
+    """Unit-level checks on cert-publish-lib.sh itself."""
+
+    def call(self, snippet, *args, **env_overrides):
+        return subprocess.run(
+            ['bash', '-c', '. "$1"; shift; ' + snippet, '_', LIB, *args],
+            env=self.env(**env_overrides), capture_output=True, text=True)
+
+    def test_valid_bundle_accepted(self):
+        path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
+        self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0)
+
+    def test_empty_and_missing_bundles_rejected(self):
+        empty = write(os.path.join(self.tmp, 'empty.pem'), '')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),
+                             'is empty')
+        missing = os.path.join(self.tmp, 'nope.pem')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', missing),
+                             'does not exist')
+
+    def test_key_without_end_marker_rejected(self):
+        truncated = write(os.path.join(self.tmp, 'cut.pem'),
+                          TEST_CERT + '-----BEGIN PRIVATE KEY-----\nMIIEvAIB\n')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', truncated),
+                             'unterminated private key block')
+
+    def test_empty_pem_blocks_are_rejected(self):
+        """Why the pairing check is mandatory rather than best-effort.
+
+        Every structural check in the library passes on this file: a complete
+        CERTIFICATE block and a complete PRIVATE KEY block, both with nothing
+        between BEGIN and END. Only openssl can tell it is not a certificate.
+        """
+        hollow = write(os.path.join(self.tmp, 'hollow.pem'),
+                       '-----BEGIN CERTIFICATE-----\n'
+                       '-----END CERTIFICATE-----\n'
+                       '-----BEGIN PRIVATE KEY-----\n'
+                       '-----END PRIVATE KEY-----\n')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', hollow),
+                             'openssl could not read the certificate')
+
+    def test_a_broken_live_pem_does_not_overwrite_a_good_backup(self):
+        """Mirrors create_backup(require_valid=True) in haproxy_manager.py.
+
+        If the pem currently on disk is already garbage, archiving it would
+        replace a restorable backup with an unusable one.
+        """
+        os.makedirs(self.backup_dir)
+        good_backup = write(os.path.join(self.backup_dir, DOMAIN + '.pem'),
+                            PREVIOUS_BUNDLE)
+        write(self.live_pem, 'garbage, not a pem at all\n')
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+        self.assertEqual(read(good_backup), PREVIOUS_BUNDLE,
+                         'a good backup was overwritten with an unusable pem')
+
+    def test_publish_fails_loudly_when_the_rename_cannot_happen(self):
+        """No silent fallback to writing straight into the certs dir.
+
+        The failure is injected with a stub `mv` that refuses, rather than by
+        chmod 0500 on the certs dir: the container these scripts run in is
+        root, root ignores directory permissions, so the chmod version skipped
+        itself exactly where it matters and only ever ran on a workstation.
+        """
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+        write(os.path.join(self.bindir, 'mv'),
+              "#!/bin/sh\n"
+              "echo \"mv: cannot move '$2': Permission denied\" >&2\n"
+              "exit 1\n", 0o755)
+        self.addCleanup(os.unlink, os.path.join(self.bindir, 'mv'))
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+
+        self.assert_rejected(result, 'NOT falling back to a direct write')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was damaged by a failed rename')
+        self.assert_no_staging_leftovers()
+
+    def test_published_pem_keeps_the_mode_of_the_file_it_replaces(self):
+        """A write-safety fix must not silently re-permission private keys.
+
+        Both directions matter: mktemp stages at 0600, so without the explicit
+        chmod every publish would tighten a 0644 bundle; and the mode must not
+        be copied from a symlink (see the next test).
+        """
+        for mode in (0o644, 0o640, 0o600):
+            with self.subTest(oct(mode)):
+                write(self.live_pem, PREVIOUS_BUNDLE, mode)
+                result = self.call('cert_publish "$1" "$2" "$3"',
+                                   self.src_cert, self.src_key, self.live_pem)
+                self.assertEqual(result.returncode, 0,
+                                 result.stdout + result.stderr)
+                self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+                self.assertEqual(
+                    stat.S_IMODE(os.stat(self.live_pem).st_mode), mode,
+                    'publishing changed who can read the private key')
+
+    def test_symlinked_live_pem_does_not_become_world_writable(self):
+        """`stat -c %a` on a symlink reports 0777 - the LINK's mode, not a
+        permission. Copying that onto the staged bundle put a world-writable
+        private key in the directory HAProxy serves from. -L is what makes the
+        preserved mode the mode of the file an operator actually chose.
+        """
+        target = write(os.path.join(self.tmp, 'real-bundle.pem'),
+                       PREVIOUS_BUNDLE, 0o640)
+        os.symlink(target, self.live_pem)
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        mode = stat.S_IMODE(os.stat(self.live_pem).st_mode)
+        self.assertEqual(
+            0, mode & 0o022,
+            f'published bundle is group/world writable ({oct(mode)}) - the '
+            f'symlink mode was copied onto a real private key')
+        self.assertEqual(0o640, mode)
+
+    def test_publish_refuses_when_staging_is_on_another_filesystem(self):
+        """The header used to claim a cross-device mv "fails loudly and leaves
+        the live pem alone". GNU mv does no such thing: across filesystems it
+        copies, so it truncates and writes the DESTINATION first and only then
+        discovers it cannot finish (ENOSPC being the realistic case) - the very
+        truncation this library exists to prevent. Both directories are
+        env-overridable, so the device numbers have to be checked up front.
+        """
+        staging = os.path.join(self.a_dir_on_another_filesystem(),
+                               'cert-staging')
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem,
+                           CERT_STAGING_DIR=staging)
+
+        self.assert_rejected(result, 'different filesystems')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was disturbed by a refused publish')
+        self.assert_certs_dir_is_clean()
+
+    def test_stale_python_side_staging_temps_are_reaped(self):
+        """The staging dir has two writers.
+
+        write_config_atomically() on the Python side stages as
+        `<name>.<random>.tmp` (tempfile.mkstemp(prefix=name + '.',
+        suffix='.tmp')). The reaper matched only mktemp's `*.??????` shape, so
+        every temp leaked by a SIGKILL on the Python side stayed there forever.
+        """
+        os.makedirs(self.staging_dir, exist_ok=True)
+        stale_py = write(os.path.join(self.staging_dir,
+                                      DOMAIN + '.pem.ab12cd34.tmp'), 'stale\n')
+        stale_sh = write(os.path.join(self.staging_dir,
+                                      DOMAIN + '.pem.AbCdEf'), 'stale\n')
+        fresh = write(os.path.join(self.staging_dir,
+                                   'recent.pem.zz99yy.tmp'), 'fresh\n')
+        old = time.time() - 3 * 24 * 3600
+        for path in (stale_py, stale_sh):
+            os.utime(path, (old, old))
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+
+        self.assertFalse(os.path.exists(stale_py),
+                         'a stale Python-side staging temp was never reaped')
+        self.assertFalse(os.path.exists(stale_sh),
+                         'a stale shell-side staging temp was never reaped')
+        self.assertTrue(os.path.exists(fresh),
+                        'the reaper deleted a temp a concurrent publisher may '
+                        'still be writing')
+
+    def test_validation_of_a_live_pem_being_republished_is_not_spurious(self):
+        """cert_bundle_valid must judge ONE snapshot of the file.
+
+        cert_publish() calls cert_bundle_valid() on the LIVE pem (step (e), to
+        decide whether it is worth backing up) while another publisher may be
+        renaming a new bundle over it. The function used to open the file six
+        times, so `openssl x509` could read the outgoing bundle and `openssl
+        pkey` the incoming one - and report "private key does not match the
+        certificate" about two files that were each perfectly fine. That ERROR
+        goes into the log monitor-errors.sh watches, which makes it a page.
+
+        The two bundles alternated below are each internally valid but carry
+        DIFFERENT key material. That matters: NEW_BUNDLE and PREVIOUS_BUNDLE
+        share a cert and a key, so alternating those two could never produce a
+        mismatch no matter how badly the reads were interleaved - the test
+        would model a world in which the bug cannot happen and pass forever.
+        """
+        write(self.live_pem, NEW_BUNDLE)
+        a = write(os.path.join(self.tmp, 'churn-a.pem'), NEW_BUNDLE)
+        b = write(os.path.join(self.tmp, 'churn-b.pem'), OTHER_BUNDLE)
+
+        # A publisher renaming over the live pem as fast as it can. Staged
+        # outside the certs dir, then renamed, exactly like cert_publish().
+        churn = subprocess.Popen(
+            ['bash', '-c',
+             'end=$((SECONDS+8)); s="$4"; while [ $SECONDS -lt $end ]; do '
+             '  cp "$1" "$s"; mv -f "$s" "$2"; '
+             '  cp "$3" "$s"; mv -f "$s" "$2"; '
+             'done', '_', a, self.live_pem, b,
+             os.path.join(self.tmp, 'churn-staged.pem')],
+            env=self.env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+        def _stop():
+            churn.kill()
+            churn.wait()
+        self.addCleanup(_stop)
+
+        result = subprocess.run(
+            ['bash', '-c',
+             '. "$1"; for i in $(seq 1 200); do cert_bundle_valid "$2" || exit 1; done',
+             '_', LIB, self.live_pem],
+            env=self.env(), capture_output=True, text=True, timeout=120)
+        _stop()
+
+        output = result.stdout + result.stderr
+        self.assertNotIn('does not match the certificate', output,
+                         'two valid bundles were reported as a mismatched pair '
+                         'because the checks read different files')
+        self.assertEqual(0, result.returncode,
+                         f'a concurrent republish made validation fail: {output}')
+
+    def a_dir_on_another_filesystem(self):
+        certs_dev = os.stat(self.certs_dir).st_dev
+        for candidate in ('/dev/shm', '/run', '/var/tmp', '/tmp', '/'):
+            try:
+                if (os.path.isdir(candidate)
+                        and os.access(candidate, os.W_OK)
+                        and os.stat(candidate).st_dev != certs_dev):
+                    path = tempfile.mkdtemp(prefix='cert-xdev-', dir=candidate)
+                    self.addCleanup(shutil.rmtree, path, True)
+                    return path
+            except OSError:
+                continue
+        self.skipTest('no writable directory on a second filesystem available')
+
+    def test_haproxy_config_ok_follows_the_validator(self):
+        self.assertEqual(self.call('haproxy_config_ok').returncode, 0)
+        write(self.haproxy_cfg, GOOD_HAPROXY_CFG + BROKEN_TOKEN + '\n')
+        self.assertNotEqual(self.call('haproxy_config_ok').returncode, 0)
+
+    def _openssl_free_path(self, name):
+        """A PATH directory with the library's tools but no openssl.
+
+        Symlinks, so this directory must never be walked with a chmod that
+        follows them - see _cleanup_tmp().
+        """
+        fake_path = os.path.join(self.tmp, name)
+        os.makedirs(fake_path)
+        for tool in ('cat', 'grep', 'mktemp', 'mv', 'cp', 'rm', 'mkdir',
+                     'basename', 'dirname', 'find', 'date', 'chmod', 'stat'):
+            real = shutil.which(tool)
+            if real:
+                os.symlink(real, os.path.join(fake_path, tool))
+        return fake_path
+
+    def test_missing_openssl_is_a_hard_failure(self):
+        """The pairing check is MANDATORY: no openssl, no publication.
+
+        This used to assert the opposite - that a missing openssl warns and
+        publishes anyway - justified by "the image does not necessarily install
+        the openssl CLI". The image does: openssl 3.x arrives with
+        ca-certificates, which certbot needs, and generate_self_signed_cert()
+        already runs `openssl req` with check=True at first-run setup. So the
+        fail-open never actually fired, and structural checks alone accept a
+        bundle of empty pem blocks (see test_empty_pem_blocks_are_rejected).
+        """
+        fake_path = self._openssl_free_path('no-openssl-bin')
+        path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
+
+        # bash by absolute path: the stripped PATH cannot resolve it.
+        result = subprocess.run(
+            [shutil.which('bash'), '-c', '. "$1"; cert_bundle_valid "$2"',
+             '_', LIB, path],
+            env=self.env(PATH=fake_path), capture_output=True, text=True)
+
+        self.assert_rejected(result, 'openssl binary not found')
+        self.assertIn('REFUSING', result.stdout + result.stderr,
+                      'a broken image must be reported as a broken image')
+
+    def test_missing_openssl_stops_a_publish_rather_than_weakening_it(self):
+        """cert_publish must inherit the refusal, and not touch the live pem."""
+        fake_path = self._openssl_free_path('no-openssl-bin2')
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+
+        result = subprocess.run(
+            [shutil.which('bash'), '-c',
+             '. "$1"; cert_publish "$2" "$3" "$4"',
+             '_', LIB, self.src_cert, self.src_key, self.live_pem],
+            env=self.env(PATH=fake_path), capture_output=True, text=True)
+
+        self.assert_rejected(result, 'openssl binary not found')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was disturbed by a refused publish')
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
+
+
+class TestScriptsAreSane(unittest.TestCase):
+    """Cheap static guards against the failure mode coming back."""
+
+    SHELL_FILES = ('renew-certificates.sh', 'sync-certificates.sh',
+                   'cert-publish-lib.sh')
+
+    def test_shell_files_parse(self):
+        for name in self.SHELL_FILES:
+            path = os.path.join(SCRIPTS_DIR, name)
+            result = subprocess.run(['bash', '-n', path],
+                                    capture_output=True, text=True)
+            self.assertEqual(result.returncode, 0,
+                             f'{name}: {result.stderr}')
+
+    def test_no_script_redirects_into_the_live_pem(self):
+        pattern = re.compile(r'>\s*"?\$\{?COMBINED_FILE')
+        for name in ('renew-certificates.sh', 'sync-certificates.sh'):
+            body = read(os.path.join(SCRIPTS_DIR, name))
+            self.assertIsNone(pattern.search(body),
+                              f'{name} still redirects output straight into the '
+                              f'live pem HAProxy is serving')
+
+
+if __name__ == '__main__':
+    print(f'testing scripts from: {SCRIPTS_DIR}')
+    unittest.main(verbosity=2)

scripts/test-config-rollback.py

@@ -31,8 +31,10 @@ __BROKEN__, which is how the tests inject an invalid configuration.
 """
 
 import os
+import re
 import sys
 import shutil
+import inspect
 import sqlite3
 import logging
 import tempfile
@@ -412,12 +414,79 @@ class TestBackupPrimitives(RollbackTestCase):
         self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
 
     def test_backup_set_covers_every_file_generate_config_writes(self):
-        pairs = dict(hm._config_backup_pairs())
+        """Derived, not restated.
-        for path in (hm.HAPROXY_CONFIG_PATH, hm.BLOCKED_IPS_MAP_PATH,
+
-                     hm.CORAZA_SPOE_CONFIG_PATH):
+        An earlier version of this test listed the three files it expected and
-            self.assertIn(path, pairs,
+        checked they were in the backup set, so it could never have noticed a
-                          f'{path} is written by generate_config() but is not '
+        FOURTH file being added. Here the set of files generate_config() writes
-                          'part of the backed-up config set')
+        is observed (and, for env-gated branches this fixture cannot safely
+        execute, read out of the source), and anything not backed up has to be
+        on the documented exclusion list below.
+        """
+        # Written by generate_config() but deliberately NOT restorable, with
+        # the reason. Everything else must be in the backup set: `haproxy -c`
+        # validates the config as a set, so a file it loads that is not
+        # restored alongside haproxy.cfg breaks rollback.
+        excluded = {
+            # Only ever created empty-when-missing (haproxy refuses to start
+            # with an ACL -f pointing at a missing file); its contents are
+            # owned by the /suspended API, not by generate_config(), so there
+            # is nothing here for a config rollback to undo.
+            'suspended_domains.list',
+            # Generated once and then read, never rewritten with new content.
+            # Its value is rendered INTO haproxy.cfg, so restoring an older
+            # haproxy.cfg alongside the current secret file is consistent.
+            'cluster-secret',
+        }
+        backed_up = {os.path.basename(p) for p, _ in hm._config_backup_pairs()}
+
+        # 1. Observed: run a generation with every patchable optional branch on
+        #    and see what actually changed on disk.
+        self.add_domain('derive.example.com', 'derive_backend')
+        os.environ['HAPROXY_CORAZA_SPOE_BACKEND'] = '127.0.0.1:9000'
+        self.addCleanup(os.environ.pop, 'HAPROXY_CORAZA_SPOE_BACKEND', None)
+        before = self._snapshot_etc()
+        hm.generate_config()
+        after = self._snapshot_etc()
+        touched = {name for name, blob in after.items()
+                   if before.get(name) != blob}
+        self.assertIn('coraza-spoe.cfg', touched,
+                      'fixture precondition: the Coraza branch did not run')
+
+        # 2. Read out of the source: branches this fixture must not execute
+        #    (suspension writes a hardcoded /etc/haproxy path that no test
+        #    global can redirect) still have to be accounted for.
+        touched |= {
+            os.path.basename(m)
+            for m in re.findall(r"'(/etc/haproxy/[\w.+-]+)'",
+                                inspect.getsource(hm.generate_config))
+        }
+
+        unaccounted = touched - backed_up - excluded
+        self.assertEqual(
+            unaccounted, set(),
+            f'generate_config() writes {sorted(unaccounted)}, which is neither '
+            'in the backup set nor on the documented exclusion list - a '
+            'rollback would restore a mixed-vintage config set')
+
+    def _snapshot_etc(self):
+        """Contents of every plain file in the fake /etc/haproxy.
+
+        Skips the backup halves (they are the thing being maintained), the
+        SQLite database and its journals, and the stats socket.
+        """
+        skip_prefixes = (os.path.basename(hm.DB_FILE),
+                         os.path.basename(hm.HAPROXY_SOCKET_PATH))
+        state = {}
+        for name in os.listdir(self.etc):
+            path = os.path.join(self.etc, name)
+            if not os.path.isfile(path) or name.endswith('.backup'):
+                continue
+            if name.startswith(skip_prefixes):
+                continue
+            with open(path, 'rb') as fh:
+                state[name] = fh.read()
+        return state
 
     def test_coraza_spoe_config_round_trips(self):
         self.generate_good_config()
@@ -452,8 +521,12 @@ class TestAtomicWrite(RollbackTestCase):
             fh.write('old content\n')
 
         # Anything that makes f.write() blow up mid-flight stands in for a full
-        # disk / killed container.
+        # disk / killed container. TypeError specifically, not Exception: a
-        with self.assertRaises(Exception):
+        # bare assertRaises(Exception) also swallows the AttributeError raised
+        # when write_config_atomically does not exist at all, so this test
+        # passed against the pre-fix tree and would keep passing if the
+        # function were deleted.
+        with self.assertRaises(TypeError):
             hm.write_config_atomically(path, object())
 
         self.assertEqual(self.read(path), 'old content\n',
@@ -462,6 +535,241 @@ class TestAtomicWrite(RollbackTestCase):
         self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
 
 
+class TestBackupFailureGuard(RollbackTestCase):
+    """generate_config() refuses to write when no rollback target could be taken."""
+
+    def test_a_failed_backup_stops_the_config_from_being_written(self):
+        good = self.generate_good_config()
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        real_create_backup = hm.create_backup
+        hm.create_backup = lambda *a, **kw: (False, 'error')
+        self.addCleanup(setattr, hm, 'create_backup', real_create_backup)
+
+        self.add_domain('second.example.com', 'second_backend', '10.0.0.3')
+        with self.assertRaises(Exception) as ctx:
+            hm.generate_config()
+
+        self.assertIn('Refusing to regenerate', str(ctx.exception),
+                      'a backup failure was not reported as a refusal')
+        self.assertEqual(
+            self.read(hm.HAPROXY_CONFIG_PATH), good,
+            'a new config was written even though the snapshot failed - a bad '
+            'change could not have been rolled back')
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+            'the blocked IPs map was rewritten even though the snapshot failed')
+
+
+class TestValidatorAvailability(RollbackTestCase):
+    """'the validator could not run' is not the same as 'the config is bad'."""
+
+    def _hide_the_haproxy_binary(self):
+        empty = os.path.join(self.tmp, 'empty-bin')
+        os.makedirs(empty, exist_ok=True)
+        os.environ['PATH'] = empty
+        # setUp's cleanup restores the original PATH.
+
+    def test_a_missing_validator_is_unavailable_not_invalid(self):
+        good = self.generate_good_config()
+        self._hide_the_haproxy_binary()
+
+        status, message = hm.validate_config_file(hm.HAPROXY_CONFIG_PATH)
+        self.assertEqual(status, 'unavailable',
+                         'a validator that could not run was reported as a '
+                         f'verdict on the config ({status}: {message})')
+
+        # And the consequence create_backup() draws from it: a config it could
+        # not check is still snapshotted, because refusing would leave the box
+        # with no rollback target at all. Contrast
+        # test_a_broken_current_config_does_not_replace_a_good_backup, where a
+        # real 'invalid' verdict yields 'kept_previous'.
+        with open(hm.HAPROXY_CONFIG_PATH, 'w') as fh:
+            fh.write('hand written, unverifiable\n')
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(status, 'created',
+                         'an unverifiable config was treated as a rejected one')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH),
+                         'hand written, unverifiable\n')
+        self.assertNotEqual(self.read(hm.HAPROXY_BACKUP_PATH), good)
+
+
+class TestFileComparison(RollbackTestCase):
+    """The fast path compares bytes, not sizes."""
+
+    def test_same_size_different_content_is_not_identical(self):
+        a = os.path.join(self.etc, 'a')
+        b = os.path.join(self.etc, 'b')
+        with open(a, 'w') as fh:
+            fh.write('aaaa\n')
+        with open(b, 'w') as fh:
+            fh.write('aaba\n')
+        self.assertEqual(os.path.getsize(a), os.path.getsize(b),
+                         'fixture: the two files must be the same size')
+        self.assertFalse(hm._files_identical(a, b),
+                         'two same-size files with different bytes compared equal')
+
+    def test_a_same_size_drifted_config_still_hits_the_validation_gate(self):
+        """The case the byte-compare exists for.
+
+        A config edited in place without changing its length (one character
+        swapped, a hostname replaced by another of the same width) must not be
+        mistaken for the known-good backup and waved through.
+        """
+        good = self.generate_good_config()
+        # Sized in BYTES, not characters: the rendered config contains
+        # non-ASCII (em dashes in template comments), so len(str) would be
+        # smaller than the file and this test would pass for the wrong reason.
+        size = os.path.getsize(hm.HAPROXY_CONFIG_PATH)
+        broken = f'# {BROKEN_TOKEN}\n'.encode()
+        broken += b'#' * (size - len(broken) - 1) + b'\n'
+        with open(hm.HAPROXY_CONFIG_PATH, 'wb') as fh:
+            fh.write(broken)
+        self.assertEqual(os.path.getsize(hm.HAPROXY_CONFIG_PATH), size,
+                         'fixture: the drifted config must be the same size')
+
+        ok, status = hm.create_backup()
+        self.assertTrue(ok)
+        self.assertEqual(
+            status, 'kept_previous',
+            'a same-size broken config was accepted as unchanged and skipped '
+            'the validation gate')
+        self.assertEqual(self.read(hm.HAPROXY_BACKUP_PATH), good,
+                         'the known-good backup was overwritten')
+
+
+class TestBlockedIpsMapWrites(RollbackTestCase):
+    """blocked_ips.map is loaded by `haproxy -c`, so it gets the same care.
+
+    Verified against HAProxy 2.8: with the map referenced by
+    map_ip(/etc/haproxy/blocked_ips.map,0), a half-written final line makes the
+    WHOLE configuration invalid ("'198.51.10' is not a valid IPv4 or IPv6
+    address at line 2 of file ..."), not merely a dropped entry.
+    """
+
+    def test_the_map_goes_through_the_atomic_writer(self):
+        seen = []
+        real_write = hm.write_config_atomically
+
+        def spy(path, content, *args, **kwargs):
+            seen.append(path)
+            return real_write(path, content, *args, **kwargs)
+
+        hm.write_config_atomically = spy
+        self.addCleanup(setattr, hm, 'write_config_atomically', real_write)
+
+        self.block_ip('192.0.2.10')
+        self.assertTrue(hm.update_blocked_ips_map())
+        self.assertIn(hm.BLOCKED_IPS_MAP_PATH, seen,
+                      'the blocked IPs map was written without the atomic '
+                      'writer - a truncated map is a fatal config')
+
+    def test_a_crash_before_the_rename_leaves_the_old_map_intact(self):
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_PATH)
+
+        real_replace = os.replace
+
+        def boom(src, dst, *args, **kwargs):
+            if dst == hm.BLOCKED_IPS_MAP_PATH:
+                raise OSError('simulated crash between write and rename')
+            return real_replace(src, dst, *args, **kwargs)
+
+        os.replace = boom
+        self.addCleanup(setattr, os, 'replace', real_replace)
+
+        self.block_ip('198.51.100.20')
+        self.assertFalse(hm.update_blocked_ips_map(),
+                         'a failed map write was reported as success')
+        os.replace = real_replace
+
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_PATH), good_map,
+            'an interrupted map write clobbered the map HAProxy is running')
+        leftovers = [n for n in os.listdir(self.etc) if n.endswith('.tmp')]
+        self.assertEqual(leftovers, [], f'temp files left behind: {leftovers}')
+
+    def test_a_malformed_map_is_not_recorded_as_known_good(self):
+        """The map backup must stay something HAProxy would actually load."""
+        self.generate_good_config()
+        good_map = self.read(hm.BLOCKED_IPS_MAP_BACKUP_PATH)
+
+        # Straight into the table, the way a bad row gets there in the first
+        # place - the API route is not the only writer.
+        self.block_ip('not-an-ip')
+        hm.update_blocked_ips_map()
+
+        self.assertEqual(
+            self.read(hm.BLOCKED_IPS_MAP_BACKUP_PATH), good_map,
+            'a map HAProxy cannot parse was promoted to the rollback target')
+
+    def test_no_map_backup_is_fabricated_before_a_config_exists(self):
+        """Nothing to stay in step with means nothing to write."""
+        self.block_ip('192.0.2.10')
+        self.assertTrue(hm.update_blocked_ips_map())
+        self.assertFalse(
+            os.path.exists(hm.BLOCKED_IPS_MAP_BACKUP_PATH),
+            'a rollback target was invented out of a map write alone')
+
+
+class TestValidationCost(RollbackTestCase):
+    """`haproxy -c` runs are the customer-facing cost of a config change.
+
+    generate_config() runs synchronously inside the API call that adds a
+    domain, and on an edge with hundreds of certificates `haproxy -c` is the
+    expensive part. These counts are the contract; changing them should be a
+    deliberate decision, not a side effect.
+    """
+
+    def _count_validations(self, action):
+        calls = []
+        real_validate = hm.validate_config_file
+
+        def spy(path):
+            calls.append(path)
+            return real_validate(path)
+
+        hm.validate_config_file = spy
+        try:
+            action()
+        finally:
+            hm.validate_config_file = real_validate
+        return len(calls)
+
+    def test_blocking_an_ip_does_not_add_a_validation_to_the_next_change(self):
+        self.generate_good_config()
+
+        def add(domain, backend, address):
+            self.add_domain(domain, backend, address)
+            hm.generate_config()
+
+        steady = self._count_validations(
+            lambda: add('a.example.com', 'a_backend', '10.0.0.4'))
+        self.assertEqual(
+            steady, 1,
+            'a steady-state config change should cost exactly one `haproxy -c` '
+            '(the pre-reload gate); the known-good fast path should skip the '
+            f'other one, but {steady} ran')
+
+        # What POST /api/blocked-ips does: rewrite the map outside
+        # generate_config(). This fleet blocks IPs automatically, so it happens
+        # between most config changes.
+        self.block_ip('192.0.2.10')
+        hm.update_blocked_ips_map()
+
+        after_block = self._count_validations(
+            lambda: add('b.example.com', 'b_backend', '10.0.0.5'))
+        self.assertEqual(
+            after_block, steady,
+            'an IP block left the map out of step with its backup, so the next '
+            f'domain add paid {after_block} `haproxy -c` runs instead of '
+            f'{steady} - on the customer-facing call')
+
+
 if __name__ == '__main__':
     print(f"testing haproxy_manager from: {MODULE_DIR}")
     unittest.main(verbosity=2)

scripts/test-runtime-map-contract.py

@@ -0,0 +1,397 @@
+#!/usr/bin/env python3
+"""Contract test: the runtime-map fast path (blocked IPs).
+
+Why this file exists
+--------------------
+`add_ip_to_runtime_map()` and `remove_ip_from_runtime_map()` spent their whole
+existence sending
+
+    add map #0 <ip> 1
+    del map #0 <ip>
+
+to `/tmp/haproxy-cli` and returning True whenever socat exited 0. Neither
+command has ever worked. Two independent defects:
+
+  * **No `@1` prefix.** `/tmp/haproxy-cli` is HAProxy's MASTER CLI socket; map
+    commands are worker commands. The master answers `Unknown command: 'add',
+    but maybe one of the following ones is a better match: ...` -- and **socat
+    still exits 0**, so `result.returncode == 0` was true and the function
+    logged "Added IP x to runtime map".
+  * **`#0` is not a valid map id.** Ids are assigned at config-parse time and
+    move on every config regeneration; on the live edge `blocked_ips.map` is
+    id 37 and `trusted_ips.map` is 10. There is no id 0. Any hardcoded number
+    is wrong -- the map must be referenced by its FILE PATH, which is what
+    haproxy.cfg itself names in `map_ip(/etc/haproxy/blocked_ips.map,0)`.
+
+And a third silence that makes a response-body check alone insufficient:
+`@1 add map #0 <ip> 1` returns an **empty body**, exit 0, and adds nothing
+anywhere -- while `@1 del map #0 <ip>` answers `Unknown map identifier.`. The
+add path can therefore only be trusted after reading the entry back.
+
+IP blocking still worked, because `update_blocked_ips_map()` rewrites
+`/etc/haproxy/blocked_ips.map` and the callers reload HAProxy, which re-reads
+it. The FILE is authoritative; this is the no-reload fast path, and it has
+never once run while reporting that it did.
+
+What it enforces
+----------------
+  1. The command strings actually sent: `@1` prefix first, map referenced by
+     PATH and never by `#<id>`, and the value `1` that
+     `map_ip(...,0) -m int gt 0` requires.
+  2. Every captured rejection is classified as FAILURE (returns False), not
+     success -- including the two that carry no error text at all.
+  3. Success is only reported when the entry reads back in the state asked
+     for. Not the exit status, not an empty reply.
+  4. No source in this repo builds a map command with a `#<id>` reference.
+     Comments may describe the old form; code may not use it.
+
+Runs fully offline: `_cli_send()` is replaced, so no socket, no socat, no
+HAProxy, no network.
+
+Running
+-------
+    python3 scripts/test-runtime-map-contract.py
+"""
+
+import ast
+import io
+import os
+import re
+import sys
+import glob
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+logging.getLogger().setLevel(logging.CRITICAL)
+
+MAP = haproxy_manager.BLOCKED_IPS_MAP_PATH
+IP = '192.0.2.77'
+
+# ---------------------------------------------------------------------------
+# Responses captured VERBATIM from the haproxy-manager container on the live
+# edge (HAProxy 3.0.11, 2026-08-22). socat exited 0 for every single one of
+# them, which is the entire reason none of this is decided on exit status.
+# ---------------------------------------------------------------------------
+
+# What the MASTER socket answers to an unprefixed `add map ...` -- i.e. the
+# reply the old code read as success.
+MASTER_REJECTS_ADD = """\
+Unknown command: 'add', but maybe one of the following ones is a better match:
+  @!<pid>                                 : send a command to the <pid> process
+  @master                                 : send a command to the master process
+  hard-reload                             : achieve a hard-reload (-st) of haproxy
+  reload                                  : achieve a soft-reload (-sf) of haproxy
+  user                                    : lower the level of the current CLI session to user
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+MASTER_REJECTS_DEL = MASTER_REJECTS_ADD.replace("'add'", "'del'")
+
+# `@1 del map #0 <ip>` / `@1 get map /etc/haproxy/nope.map <ip>`.
+UNKNOWN_MAP_IDENTIFIER = 'Unknown map identifier. Please use #<id> or <file>.\n'
+
+# `@1 add map <map> <ip>` with the value omitted (as the old docs showed).
+ADD_MAP_MISSING_VALUE = (
+    "'add map' expects three parameters (map identifier, key and value) or one "
+    "parameter (map identifier) and a payload\n")
+
+# `@1 del map <map> <ip>` for a key the runtime map does not hold.
+KEY_NOT_FOUND = 'Key not found.\n'
+
+# A successful mutation. THIS IS THE WHOLE PROBLEM: it is byte-for-byte what
+# `@1 add map #0 <ip> 1` also returns while adding nothing at all.
+MUTATION_OK = ''
+
+GET_FOUND = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+             'value="1", type="str"\n' % IP)
+GET_NOT_FOUND = 'type=ip, case=sensitive, found=no\n'
+# `@1 get map #0 <ip>` -- "found", but with no value. haproxy.cfg matches with
+# `-m int gt 0`, so an entry like this does NOT block.
+GET_FOUND_NO_VALUE = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+                      'value=none\n' % IP)
+
+SHOW_MAP_SAMPLE = (
+    '0x7f6e5a788700 101.36.109.130 1\n'
+    '0x7f6e5a788780 101.47.140.218 1\n'
+    '0x7f6e5a1fd500 %s 1\n' % IP)
+
+
+class FakeSocket(object):
+    """Replaces _cli_send(). Records every command, answers from a script.
+
+    `script` is a list of (substring, response) pairs, consulted in order; the
+    first whose substring appears in the command wins. Anything unmatched is
+    an explicit test bug, not a silent default.
+    """
+
+    def __init__(self, script):
+        self.script = script
+        self.sent = []
+
+    def __call__(self, command, socket_path, timeout):
+        self.sent.append(command)
+        for needle, response in self.script:
+            if needle in command:
+                return response
+        raise AssertionError('test script has no response for %r' % command)
+
+
+def with_socket(script):
+    """Install a FakeSocket for the duration of a `with` block."""
+    class _Ctx(object):
+        def __enter__(self):
+            self.fake = FakeSocket(script)
+            self._real = haproxy_manager._cli_send
+            haproxy_manager._cli_send = self.fake
+            return self.fake
+
+        def __exit__(self, *exc):
+            haproxy_manager._cli_send = self._real
+            return False
+    return _Ctx()
+
+
+# Every command the happy path needs, in the order the code issues them.
+HAPPY_ADD = [('get map', GET_FOUND), ('add map', MUTATION_OK)]
+HAPPY_DEL = [('get map', GET_NOT_FOUND), ('del map', MUTATION_OK)]
+
+
+class CommandsAreWellFormed(unittest.TestCase):
+    """Guard 1: the exact bytes on the wire.
+
+    Both original defects are visible here and nowhere else -- a `#0` map
+    reference and a missing `@1` are perfectly ordinary-looking Python.
+    """
+
+    def test_add_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_ADD) as fake:
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 add map %s %s 1' % (MAP, IP))
+
+    def test_del_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_DEL) as fake:
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 del map %s %s' % (MAP, IP))
+
+    def test_every_command_is_tried_on_the_worker_first(self):
+        """`/tmp/haproxy-cli` is the MASTER socket; bare map commands 404."""
+        for run in (lambda: haproxy_manager.add_ip_to_runtime_map(IP),
+                    lambda: haproxy_manager.remove_ip_from_runtime_map(IP)):
+            with with_socket(HAPPY_ADD + HAPPY_DEL) as fake:
+                run()
+            for command in fake.sent:
+                self.assertTrue(command.startswith('@1 '),
+                                '%r is missing the @1 worker prefix' % command)
+
+    def test_no_command_references_a_map_by_id(self):
+        """Map ids move on every config regeneration. Path, always."""
+        script = HAPPY_ADD + HAPPY_DEL + [('show map', SHOW_MAP_SAMPLE)]
+        with with_socket(script) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+            haproxy_manager.remove_ip_from_runtime_map(IP)
+            haproxy_manager.runtime_map_keys(MAP)
+        for command in fake.sent:
+            self.assertNotRegex(
+                command, r'\bmap\s+#',
+                '%r references a map by id; ids are not stable' % command)
+            self.assertIn(MAP, command,
+                          '%r does not name the map file' % command)
+
+    def test_add_carries_the_value_the_config_matches_on(self):
+        """`map_ip(...,0) -m int gt 0`: a valueless entry does not block."""
+        self.assertEqual(haproxy_manager.BLOCKED_IPS_MAP_VALUE, '1')
+        with with_socket(HAPPY_ADD) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+        self.assertTrue(fake.sent[0].endswith(' %s 1' % IP),
+                        '%r has no value; HAProxy rejects it' % fake.sent[0])
+
+
+class RejectionIsFailure(unittest.TestCase):
+    """Guard 2+3: nothing may report success unless the map really changed.
+
+    Every response below was returned by the live socket with **exit code 0**.
+    The old code returned True for all of them.
+    """
+
+    def test_master_socket_rejection_of_add_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_master_socket_rejection_of_del_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MASTER_REJECTS_DEL)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_unknown_map_identifier_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket([('get map', GET_FOUND), ('del map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_missing_value_rejection_is_failure(self):
+        """Carries no marker word at all -- caught by 'a mutation says nothing'."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', ADD_MAP_MISSING_VALUE)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_silent_noop_add_is_failure(self):
+        """The `#0` failure mode: accepted, empty reply, nothing added.
+
+        Nothing in the response distinguishes this from success. Only the
+        read-back does -- which is why the read-back is not optional.
+        """
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_add_that_lands_without_a_value_is_failure(self):
+        with with_socket([('get map', GET_FOUND_NO_VALUE), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_del_that_leaves_the_key_behind_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_key_not_found_on_del_is_the_requested_end_state(self):
+        """Not a failure: the runtime map already lacks the key."""
+        with with_socket([('get map', GET_NOT_FOUND), ('del map', KEY_NOT_FOUND)]):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_verified_success_is_reported_as_success(self):
+        with with_socket(HAPPY_ADD):
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket(HAPPY_DEL):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_a_runtime_failure_never_raises_into_the_request_handler(self):
+        """The map file + reload still enforces the block; degrade, don't 500."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertIs(haproxy_manager.add_ip_to_runtime_map(IP), False)
+
+    def test_mutations_are_checked_for_an_empty_body_not_a_marker_list(self):
+        """_HAPROXY_CLI_ERROR_MARKERS can only know rejections already seen."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            with with_socket([('add map', 'something nobody has ever seen\n')]):
+                haproxy_manager.haproxy_cli('add map %s x 1' % MAP,
+                                            worker=True, expect_empty=True)
+
+    def test_the_new_error_markers_are_recognised(self):
+        for response in (UNKNOWN_MAP_IDENTIFIER, KEY_NOT_FOUND,
+                         MASTER_REJECTS_ADD):
+            self.assertTrue(haproxy_manager._cli_response_is_error(response),
+                            '%r must be classified as an error' % response[:40])
+        for response in (GET_FOUND, GET_NOT_FOUND, SHOW_MAP_SAMPLE):
+            self.assertFalse(haproxy_manager._cli_response_is_error(response),
+                             '%r is data, not an error' % response[:40])
+
+    def test_socat_exit_zero_carries_no_information(self):
+        """FakeSocket never signals failure any other way, and neither did socat."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]) as fake:
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertTrue(fake.sent, 'the command was sent and "succeeded" at the '
+                                   'process level; only the body says otherwise')
+
+
+class ReadBack(unittest.TestCase):
+    """The read-back primitives the guarantees above rest on."""
+
+    def test_lookup_reports_found_with_value(self):
+        with with_socket([('get map', GET_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (True, '1'))
+
+    def test_lookup_reports_not_found(self):
+        with with_socket([('get map', GET_NOT_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (False, None))
+
+    def test_lookup_raises_on_a_rejected_reference(self):
+        with with_socket([('get map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_lookup(MAP, IP)
+
+    def test_keys_parses_show_map_output(self):
+        with with_socket([('show map', SHOW_MAP_SAMPLE)]):
+            self.assertEqual(
+                haproxy_manager.runtime_map_keys(MAP),
+                {'101.36.109.130', '101.47.140.218', IP})
+
+    def test_empty_map_is_not_a_rejection(self):
+        with with_socket([('show map', '')]):
+            self.assertEqual(haproxy_manager.runtime_map_keys(MAP), set())
+
+    def test_rejected_show_map_still_raises(self):
+        with with_socket([('show map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_keys(MAP)
+
+
+MAP_BY_ID_RE = re.compile(r'\b(?:add|del|clear|show|get)\s+map\s+#')
+
+
+class NoSourceBuildsAMapIdCommand(unittest.TestCase):
+    """Guard 4: `map #<id>` may be described in comments, never executed.
+
+    Scanning string literals rather than raw text is deliberate -- the whole
+    reason this bug is documented at length in the source is so the next reader
+    does not reintroduce it, and a plain grep would fail on those comments.
+    """
+
+    def test_no_python_string_literal_builds_a_map_id_command(self):
+        tree = ast.parse(io.open('haproxy_manager.py', encoding='utf-8').read())
+        offenders = [
+            node.value for node in ast.walk(tree)
+            if isinstance(node, ast.Constant) and isinstance(node.value, str)
+            and MAP_BY_ID_RE.search(node.value)
+            and not (ast.get_docstring(tree) == node.value)
+        ]
+        # Docstrings are string literals too; exclude any literal that is a
+        # docstring of a module/class/function.
+        docstrings = set()
+        for node in ast.walk(tree):
+            if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef,
+                                 ast.AsyncFunctionDef)):
+                doc = ast.get_docstring(node, clean=False)
+                if doc:
+                    docstrings.add(doc)
+        offenders = [o for o in offenders if o not in docstrings]
+        self.assertEqual(offenders, [],
+                         'these string literals build a map command with an '
+                         'unstable #<id> reference')
+
+    def test_no_shell_or_template_code_line_uses_a_map_id(self):
+        targets = (glob.glob('scripts/*.sh') + glob.glob('templates/*.tpl')
+                   + ['Dockerfile'])
+        offenders = []
+        for path in targets:
+            if not os.path.exists(path):
+                continue
+            for lineno, line in enumerate(
+                    io.open(path, encoding='utf-8').read().splitlines(), 1):
+                if line.lstrip().startswith('#'):
+                    continue  # a comment describing the old form is fine
+                if MAP_BY_ID_RE.search(line):
+                    offenders.append('%s:%d: %s' % (path, lineno, line.strip()))
+        self.assertEqual(offenders, [],
+                         'map ids are assigned at config-parse time and move; '
+                         'reference the map file by path')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-stick-table-contract.py

@@ -0,0 +1,475 @@
+#!/usr/bin/env python3
+"""Contract test: what the stick tables STORE vs what the consumers READ.
+
+Why this file exists
+--------------------
+`/api/security/stats` and `scripts/show-tarpit-ips.sh` spent their whole
+existence reporting "Scan Count", "offense count" and "BLOCKED" figures parsed
+out of `gpc0` and `gpc1`. **No stick table in this repo has ever stored a
+general-purpose counter.** Every one of those numbers was fabricated, and an
+operator was making decisions on them.
+
+Nothing caught it, because each layer failed silently in a different way:
+
+  * `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
+    and the loop `continue`d -- so the endpoint answered `active_threats: 0`
+    with an empty list. "No threats" and "the parser is broken" looked
+    identical.
+  * The command went to the MASTER CLI socket without the `@1` worker prefix.
+    HAProxy answered `Unknown command: 'show', but maybe one of the following
+    ones is a better match: ...` and **socat still exited 0**, so the
+    `returncode != 0` guard never fired. The reported `total_tracked_ips` was
+    the line count of that help text (8) while the real table held 388 entries.
+  * The shell consumers wrote `gpc0=${gpc0:-0}`, so a field that does not exist
+    rendered as a confident zero.
+
+The durable fix is not "parse better" -- it is making the template and its
+consumers unable to drift apart without something going red. That is this file.
+
+What it enforces
+----------------
+  1. `STICK_TABLE_FIELD_CONTRACT` in haproxy_manager.py equals, exactly and in
+     both directions, the `store` clauses in the rendered templates.
+  2. Every shell consumer's `EXPECTED_FIELDS=(...)` array equals the contract.
+  3. A captured sample of REAL `show table` output from the live edge parses to
+     exactly the contract's fields plus the entry metadata -- so the contract
+     describes reality, not just itself.
+  4. The loud-failure behaviour: `read_stick_table()` RAISES on a rejected
+     command, on a non-table response, and on a row missing a contract field.
+     It must never answer zeros. Guard 4 is the one that would have caught the
+     original bug on day one.
+  5. No `store` clause names a general-purpose counter, and no template tracks
+     one -- the state this repo is actually in, asserted rather than assumed.
+
+Assertions about the TEMPLATES go through `rule_lines()`, which strips comments
+before matching. These templates quote their own rules in prose at length; a
+bare `assertIn` over the rendered text passes just as happily against a rule
+that has been commented out. Same lesson, and same helper, as
+scripts/test-wpadmin-gate.py.
+
+Runs fully offline -- no HAProxy, no socket, no network.
+
+Running
+-------
+    python3 scripts/test-stick-table-contract.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# A captured sample of REAL output, so this test can assert against reality
+# without a live socket.
+#
+# Provenance: `echo "@1 show table web" | socat stdio /tmp/haproxy-cli` inside
+# the haproxy-manager container on whp01, 2026-08-22, HAProxy 3.0.11. Rows
+# trimmed for length; format byte-for-byte as emitted.
+#
+# Note what is and is NOT here: no gpc0, no gpc1, no gpc(N), no gpc_rate, no
+# glitch_rate. Note also that field windows come back in MILLISECONDS
+# (`conn_rate(10000)`), not the `10s` the template writes -- a consumer that
+# labels the raw number "10s" is off by a factor of 1000.
+# ---------------------------------------------------------------------------
+LIVE_TABLE_SAMPLE = """\
+# table: web, type: ip, size:204800, used:388
+0x7f6e5447e728: key=17.58.57.102 use=0 exp=368842 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e541cf7c8: key=43.173.182.9 use=0 exp=171404 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e5547f528: key=95.129.255.180 use=0 exp=590639 shard=0 conn_rate(10000)=1 conn_cur=0 http_req_rate(10000)=1 http_err_rate(30000)=0
+0x7f6e5447e488: key=5.9.105.254 use=0 exp=556277 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=1
+"""
+
+# The verbatim reply the MASTER CLI socket gives to an unprefixed worker
+# command. Captured the same way. socat exits 0 on this -- which is the entire
+# reason haproxy_cli() inspects the body.
+MASTER_SOCKET_REJECTION = """\
+Unknown command: 'show', but maybe one of the following ones is a better match:
+  show cli level                          : display the level of the current CLI session
+  show cli sockets                        : dump list of cli sockets
+  show proc                               : show processes status
+  show startup-logs                       : report logs emitted during HAProxy startup
+  show version                            : show version of the current process
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+# Anything matching these in a `store` clause is a general-purpose counter.
+GPC_PATTERN = re.compile(r'\bgpc|glitch')
+
+# Shell consumers that must declare their field expectations as a single
+# EXPECTED_FIELDS array, and the table each one reads.
+SHELL_CONSUMERS = {
+    'scripts/show-edge-ip-rates.sh': 'web',
+}
+
+EXPECTED_FIELDS_RE = re.compile(r'^\s*EXPECTED_FIELDS=\(([^)]*)\)\s*$', re.M)
+
+
+def rule_lines(cfg, needle):
+    """Comment-stripped config lines containing `needle`.
+
+    A line that is entirely a comment is dropped; a line mixing config with a
+    trailing comment is truncated at the first ' #' before matching. Without
+    this, every assertion below would pass against a rule that had been
+    commented out but whose text survived in the surrounding prose -- and these
+    templates quote their own rules in prose constantly. See
+    scripts/test-wpadmin-gate.py, where a mutation audit proved the point.
+    """
+    out = []
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if stripped and not stripped.startswith('#'):
+            code = stripped.split(' #', 1)[0].rstrip()
+            if code and needle in code:
+                out.append(code)
+    return out
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_security_tables():
+    return haproxy_manager.template_env.get_template('hap_security_tables.tpl').render()
+
+
+def stick_tables_from(cfg):
+    """{table name: (field, ...)} for every stick-table declared in `cfg`.
+
+    A stick-table takes the name of the frontend/backend/listen section that
+    declares it -- that name is what `show table <name>` wants, so the section
+    header is part of the contract, not incidental. Section headers and
+    stick-table lines are both read comment-stripped.
+    """
+    tables = {}
+    section = None
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if not stripped or stripped.startswith('#'):
+            continue
+        code = stripped.split(' #', 1)[0].rstrip()
+        header = re.match(r'^(frontend|backend|listen)\s+(\S+)', code)
+        if header:
+            section = header.group(2)
+            continue
+        if code.startswith('stick-table'):
+            store = re.search(r'\bstore\s+(\S+)', code)
+            if not store:
+                raise AssertionError(
+                    'stick-table in section %r has no `store` clause: %r'
+                    % (section, code))
+            if section is None:
+                raise AssertionError(
+                    'stick-table declared outside any section: %r' % code)
+            # store is a comma-separated list; each item is name or name(window)
+            fields = tuple(item.split('(')[0]
+                           for item in store.group(1).split(','))
+            tables[section] = fields
+    return tables
+
+
+def all_template_stick_tables():
+    tables = {}
+    for cfg in (render_listener(), render_security_tables()):
+        for name, fields in stick_tables_from(cfg).items():
+            if name in tables:
+                raise AssertionError('stick table %r declared twice' % name)
+            tables[name] = fields
+    return tables
+
+
+class StickTableContract(unittest.TestCase):
+    """Guard 1 + 5: the templates and the Python contract, held together."""
+
+    def setUp(self):
+        self.templates = all_template_stick_tables()
+        self.contract = haproxy_manager.STICK_TABLE_FIELD_CONTRACT
+
+    def test_templates_actually_declare_stick_tables(self):
+        """Guard the guard: an empty parse would make every other check vacuous."""
+        self.assertTrue(self.templates,
+                        'parsed no stick tables out of the templates at all -- '
+                        'stick_tables_from() is broken, not the templates')
+
+    def test_same_table_names(self):
+        self.assertEqual(
+            sorted(self.templates), sorted(self.contract),
+            'STICK_TABLE_FIELD_CONTRACT and the templates disagree on WHICH '
+            'stick tables exist. Add/remove the table in both places.')
+
+    def test_same_fields_per_table(self):
+        for table in sorted(self.templates):
+            with self.subTest(table=table):
+                self.assertEqual(
+                    sorted(self.templates[table]),
+                    sorted(self.contract.get(table, ())),
+                    "stick table %r stores %s but STICK_TABLE_FIELD_CONTRACT "
+                    "claims %s. Whichever is wrong, a consumer is about to read "
+                    "a field that is never populated -- which is the bug this "
+                    "test exists for." % (table,
+                                          list(self.templates[table]),
+                                          list(self.contract.get(table, ()))))
+
+    def test_web_table_is_the_one_the_api_reads(self):
+        self.assertIn('web', self.contract)
+        self.assertIn('web', self.templates)
+
+    def test_no_general_purpose_counters_are_stored(self):
+        """The state of the world today, asserted rather than assumed.
+
+        If a gpc/glitch counter is ever genuinely added to a template, this
+        test is the place to update -- and updating it forces whoever does so
+        to also add the field to STICK_TABLE_FIELD_CONTRACT (test_same_fields_
+        per_table) and to the shell consumers (test_shell_consumers_match_
+        contract). That chain is the point: a counter cannot appear in a
+        consumer without existing in the table, and cannot appear in the table
+        without the consumers being updated.
+        """
+        for table, fields in self.templates.items():
+            for field in fields:
+                self.assertIsNone(
+                    GPC_PATTERN.search(field),
+                    'stick table %r now stores %r. Update this test, '
+                    'STICK_TABLE_FIELD_CONTRACT, and every consumer.'
+                    % (table, field))
+
+    def test_track_sc_counters_have_a_table_each(self):
+        """Every `track-scN ... table X` names a table that really exists.
+
+        A typo here is invisible to `haproxy -c` only in the sense that it is
+        NOT -- but it is invisible to the consumers, which would query a table
+        that is never written.
+        """
+        cfg = render_listener()
+        for line in rule_lines(cfg, 'track-sc'):
+            named = re.search(r'\btable\s+(\S+)', line)
+            if named:
+                self.assertIn(
+                    named.group(1), self.templates,
+                    'track-sc rule references undeclared table %r: %r'
+                    % (named.group(1), line))
+
+    def test_sc_counter_indices_fit_haproxys_limit(self):
+        """sc0/sc1/sc2 are all HAProxy gives us by default.
+
+        `tune.stick-counters` defaults to 3. A `track-sc3` without raising it
+        is a config-time failure, and the templates' own comments assume the
+        limit -- so assert it rather than leaving it as folklore.
+        """
+        cfg = render_listener() + '\n' + render_security_tables()
+        raised = rule_lines(cfg, 'tune.stick-counters')
+        limit = 3
+        if raised:
+            limit = int(re.search(r'(\d+)', raised[-1]).group(1))
+        for line in rule_lines(cfg, 'track-sc'):
+            idx = int(re.search(r'track-sc(\d+)', line).group(1))
+            self.assertLess(
+                idx, limit,
+                'track-sc%d exceeds tune.stick-counters (%d): %r'
+                % (idx, limit, line))
+
+
+class ShellConsumerContract(unittest.TestCase):
+    """Guard 2: the shell consumers cannot drift from the contract."""
+
+    def test_shell_consumers_match_contract(self):
+        for path, table in sorted(SHELL_CONSUMERS.items()):
+            with self.subTest(script=path):
+                full = os.path.join(MODULE_DIR, path)
+                self.assertTrue(os.path.exists(full),
+                                '%s is missing; it is a declared consumer of '
+                                'stick table %r' % (path, table))
+                with open(full) as fh:
+                    src = fh.read()
+                m = EXPECTED_FIELDS_RE.search(src)
+                self.assertIsNotNone(
+                    m, '%s must declare its field expectations once as a '
+                       'single-line `EXPECTED_FIELDS=(a b c)` array so this '
+                       'test can hold it to the template' % path)
+                declared = sorted(m.group(1).split())
+                self.assertEqual(
+                    declared,
+                    sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table]),
+                    '%s reads %s but stick table %r stores %s'
+                    % (path, declared, table,
+                       sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table])))
+
+    def test_retired_script_no_longer_parses_phantom_counters(self):
+        """show-tarpit-ips.sh may explain gpc0/gpc1; it may not extract them.
+
+        The shim is allowed -- encouraged -- to name the fields in prose so an
+        operator who runs it learns why its numbers went away. What it must not
+        do is go back to pulling values out of them.
+        """
+        path = os.path.join(MODULE_DIR, 'scripts/show-tarpit-ips.sh')
+        if not os.path.exists(path):
+            self.skipTest('show-tarpit-ips.sh has been removed outright')
+        with open(path) as fh:
+            lines = fh.readlines()
+        for raw in lines:
+            stripped = raw.strip()
+            if not stripped or stripped.startswith('#'):
+                continue
+            code = stripped.split(' #', 1)[0]
+            self.assertIsNone(
+                re.search(r"grep -o ['\"]?gpc|gpc[0-9]*=\$|sc_get_gpc|sc-inc-gpc", code),
+                'show-tarpit-ips.sh is extracting a general-purpose counter '
+                'again: %r' % stripped)
+
+
+class LiveSampleParses(unittest.TestCase):
+    """Guard 3: the contract describes real HAProxy output, not just itself."""
+
+    def test_header_parses(self):
+        header, entries = self._read()
+        self.assertEqual(header['name'], 'web')
+        self.assertEqual(header['type'], 'ip')
+        self.assertEqual(header['size'], 204800)
+        self.assertEqual(header['used'], 388)
+        self.assertEqual(len(entries), 4)
+
+    def test_sample_fields_are_exactly_contract_plus_metadata(self):
+        expected = set(haproxy_manager.STICK_TABLE_FIELD_CONTRACT['web'])
+        expected |= set(haproxy_manager.STICK_TABLE_ENTRY_META)
+        _, entries = self._read()
+        for line, fields in entries:
+            self.assertEqual(
+                set(fields), expected,
+                'real `show table web` output carries %s, contract+metadata '
+                'expects %s. Row: %r'
+                % (sorted(fields), sorted(expected), line))
+
+    def test_key_is_the_ip_not_the_allocation_pointer(self):
+        """The original bug read parts[0] -- the `0x...:` pointer -- as the IP."""
+        _, entries = self._read()
+        ips = [f['key']['value'] for _, f in entries]
+        self.assertIn('95.129.255.180', ips)
+        for ip in ips:
+            self.assertFalse(ip.startswith('0x'),
+                             'parsed a memory address as an IP: %r' % ip)
+
+    def test_windows_are_milliseconds(self):
+        """HAProxy reports `conn_rate(10000)` for a `conn_rate(10s)` store.
+
+        Asserted because labelling that raw 10000 as "10s" (or as seconds) is
+        an easy and completely silent way to be wrong by 1000x in the panel.
+        """
+        _, entries = self._read()
+        _, fields = entries[0]
+        self.assertEqual(fields['conn_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_req_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_err_rate']['window_ms'], 30000)
+        self.assertIsNone(fields['conn_cur']['window_ms'],
+                          'conn_cur is a gauge, not a rate; it has no window')
+
+    def test_values_are_the_real_ones(self):
+        _, entries = self._read()
+        by_ip = {f['key']['value']: f for _, f in entries}
+        self.assertEqual(by_ip['95.129.255.180']['http_req_rate']['value'], '1')
+        self.assertEqual(by_ip['5.9.105.254']['http_err_rate']['value'], '1')
+        self.assertEqual(by_ip['17.58.57.102']['http_req_rate']['value'], '0')
+
+    def _read(self):
+        return _read_table_from(LIVE_TABLE_SAMPLE)
+
+
+class FailsLoudly(unittest.TestCase):
+    """Guard 4: every way this can go wrong must raise, never return zeros.
+
+    This is the guard that would have caught the original bug immediately. Each
+    case below is a real response the old code accepted silently.
+    """
+
+    def test_master_socket_rejection_is_not_data(self):
+        """The exact reply that used to be reported as `total_tracked_ips: 8`."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(MASTER_SOCKET_REJECTION)
+        self.assertIn('not a stick-table dump', str(ctx.exception))
+
+    def test_socat_exit_zero_does_not_mean_success(self):
+        """haproxy_cli() must reject on the BODY, not the exit status.
+
+        socat returns 0 for every response above -- the rejection is only ever
+        visible in the text.
+        """
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error(MASTER_SOCKET_REJECTION))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('No such table\n'))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('Permission denied\n'))
+        self.assertFalse(
+            haproxy_manager._cli_response_is_error(LIVE_TABLE_SAMPLE))
+
+    def test_missing_contract_field_raises_and_names_it(self):
+        """A field the table stopped storing must not silently become 0."""
+        degraded = LIVE_TABLE_SAMPLE.replace(' http_err_rate(30000)=0', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(degraded)
+        msg = str(ctx.exception)
+        self.assertIn('http_err_rate', msg,
+                      'the error must name the missing field')
+        self.assertIn('drifted', msg,
+                      'the error must say what actually went wrong')
+
+    def test_empty_response_raises(self):
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            _read_table_from('')
+
+    def test_row_without_key_raises(self):
+        broken = LIVE_TABLE_SAMPLE.replace('key=17.58.57.102 ', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(broken)
+        self.assertIn('no key=', str(ctx.exception))
+
+    def test_unknown_table_raises_before_touching_the_socket(self):
+        """Querying a table with no contract is a programming error, not a 0."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            haproxy_manager.read_stick_table('does_not_exist')
+        self.assertIn('no field contract', str(ctx.exception))
+
+    def test_empty_table_is_not_an_error(self):
+        """A table with zero entries is a legitimate, distinguishable result."""
+        header, entries = _read_table_from(
+            '# table: web, type: ip, size:204800, used:0\n')
+        self.assertEqual(header['used'], 0)
+        self.assertEqual(entries, [])
+
+
+def _read_table_from(response, table='web'):
+    """Run read_stick_table() against a canned response instead of a socket."""
+    real = haproxy_manager.haproxy_cli
+    haproxy_manager.haproxy_cli = lambda cmd, worker=False, timeout=None: response
+    try:
+        return haproxy_manager.read_stick_table(table)
+    finally:
+        haproxy_manager.haproxy_cli = real
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-trusted-proxy-gate.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Regression tests for the trusted-proxy header gate in hap_listener.tpl.
+
+Why this file exists
+--------------------
+txn.real_ip was derived from CF-Connecting-IP / X-Real-IP / X-Forwarded-For
+with no check on the peer, so any direct client could assert any client IP.
+That variable drives rate limiting, the trusted-IP whitelist, the wp-login
+brute-force table and cookie challenge, the wp-json/batch/v1 virtual patch,
+IP blocking, and Coraza's src-ip -- so a spoofed header bypassed all of them.
+
+These tests pin the invariant that the header strip is rendered BEFORE any
+real-IP resolution. Ordering is the whole fix: a del-header emitted after the
+set-var chain would parse fine, validate fine, and do nothing.
+
+Running
+-------
+    python3 scripts/test-trusted-proxy-gate.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+GATED_HEADERS = ('CF-Connecting-IP', 'X-Real-IP', 'X-Forwarded-For')
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+class TrustedProxyGate(unittest.TestCase):
+
+    def setUp(self):
+        self.cfg = render_listener()
+
+    def test_trusted_proxy_acl_is_defined(self):
+        self.assertIn('acl from_trusted_proxy src', self.cfg)
+        self.assertIn('/etc/haproxy/cloudflare_ips.list', self.cfg)
+        self.assertIn('/etc/haproxy/trusted_proxies.list', self.cfg)
+
+    def test_each_header_is_deleted_for_untrusted_peers(self):
+        for header in GATED_HEADERS:
+            with self.subTest(header=header):
+                pattern = (r'http-request\s+del-header\s+%s\s+if\s+!from_trusted_proxy'
+                           % re.escape(header))
+                self.assertRegex(self.cfg, pattern)
+
+    def test_strip_precedes_real_ip_resolution(self):
+        """The ordering invariant. A strip after the set-var chain is a no-op."""
+        last_strip = max(
+            self.cfg.index('del-header %s' % h) for h in GATED_HEADERS
+        )
+        first_setvar = self.cfg.index('set-var(txn.real_ip)')
+        self.assertLess(
+            last_strip, first_setvar,
+            'del-header rules must be rendered before set-var(txn.real_ip)')
+
+    def test_src_fallback_still_present(self):
+        """Direct clients must fall through to the real TCP peer."""
+        self.assertIn('set-var(txn.real_ip) src', self.cfg)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-wpadmin-gate.py

@@ -0,0 +1,530 @@
+#!/usr/bin/env python3
+"""Regression tests for the WordPress admin edge gate in hap_listener.tpl.
+
+Why this file exists
+--------------------
+Unauthenticated GETs to /wp-admin/* were reaching PHP, booting WordPress just to
+produce a login redirect and exhausting lsphp pools under a distributed
+low-and-slow attack. The gate redirects them at the edge instead.
+
+Two properties are easy to get wrong and invisible to `haproxy -c`:
+
+  * ORDERING. `is_whitelisted` reads var(txn.real_ip). If the rule renders before
+    the set-var chain, the whitelist evaluates against an unset variable.
+  * THE ALLOWLIST. wp-login.php loads its OWN css/js from /wp-admin/. Dropping
+    those entries leaves every login page on the fleet unstyled, while still
+    returning 200 -- a silent regression.
+
+A THIRD property, added after an adversarial mutation audit: every assertion
+here must be scoped to the CODE, not the surrounding prose. This file's own
+comment blocks quote ACL names, rule fragments and even whole rules to explain
+them -- which means a bare `assertIn` / `re.search` / `str.index` run over the
+raw rendered config passes just as happily when the real rule has been deleted
+(or merely commented out) and only its explanation survives. The audit proved
+this concretely: commenting out the entire redirect rule, or the
+`acl wp_admin_allowed` line, or all five normalizers, left the previous version
+of this file at 26/26 PASS. See `rule_lines()` below, and use it (or one of the
+guarded helpers built on it) for every assertion about whether a rule exists,
+what it says, or where it sits relative to another rule. Do not add a new
+`self.cfg.index(...)`, `self.assertIn(x, self.cfg)`, or `re.search(pattern,
+self.cfg)` to this file -- none of them can tell code from comment.
+
+Running
+-------
+    python3 scripts/test-wpadmin-gate.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+ALLOWLIST = ('/admin-ajax.php', '/admin-post.php',
+             '/load-styles.php', '/load-scripts.php')
+EXCLUSIONS = ('!wp_admin_allowed', '!wp_admin_asset', '!has_wp_logged_in',
+              '!wp_gate_exempt', '!is_local', '!is_trusted_ip', '!is_whitelisted')
+
+# The normalizer set, in the order it MUST render. Decoding has to precede the
+# path walkers or "%2e%2e" is decoded to ".." only after path-strip-dotdot has
+# already run, leaving the ".." unresolved -- measured against real HAProxy
+# 3.0.11, both orders side by side.
+NORMALIZERS = ('percent-to-uppercase',
+               'percent-decode-unreserved',
+               'path-merge-slashes',
+               'path-strip-dot',
+               'path-strip-dotdot full')
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_header():
+    return haproxy_manager.template_env.get_template('hap_header.tpl').render(
+        cluster_secret=None,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Comment-safe config inspection
+#
+# Every helper below operates on `rule_positions()`'s output, never on the raw
+# rendered string. That is the one rule this whole module exists to enforce on
+# itself: a mutation audit found that commenting out a real rule (prefixing it
+# with '#', or -- more slyly -- deleting it and appending its own text as a
+# TRAILING comment on the line above) left the previous version of these tests
+# fully green, because plain `str.index` / `assertIn` / `re.search` over
+# `self.cfg` cannot distinguish code from a comment that merely quotes it.
+# ---------------------------------------------------------------------------
+
+def rule_positions(cfg, needle):
+    """[(comment-stripped line, char offset in cfg)] for every non-comment
+    line containing `needle`, in document order.
+
+    Two things a bare substring/regex search over `self.cfg` gets wrong, both
+    fixed here:
+
+      1. A line that is ENTIRELY a comment (starts with '#' once stripped) is
+         dropped. This is necessary but not sufficient -- see (2).
+      2. A line that MIXES real config with a trailing comment
+         (`live-code  # note`, or the decoy `live-code  # was: <the other
+         rule's exact text>`) is truncated at the first ' #' before matching,
+         so text stuffed into a trailing comment cannot masquerade as the
+         rule itself. A real HAProxy comment always starts at a '#' preceded
+         by whitespace here -- none of these templates use bare '#' as a
+         value character -- so this truncation does not clip real rules.
+
+    Returning (line, position) pairs together -- rather than making callers
+    re-derive one from the other with a second `cfg.index(line)` -- also
+    avoids a subtler bug: if the same comment-stripped line occurs twice
+    (e.g. a duplicated rule), re-deriving the position with `str.index` always
+    finds the FIRST copy regardless of which one you meant. Walking the file
+    once and recording positions as we go keeps first/last unambiguous.
+    """
+    out = []
+    pos = 0
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if stripped and not stripped.startswith('#'):
+            code = stripped.split(' #', 1)[0].rstrip()
+            if code and needle in code:
+                out.append((code, pos))
+        pos += len(raw) + 1  # +1 for the '\n' split() consumed
+    return out
+
+
+def rule_lines(cfg, needle):
+    """Comment-stripped rule lines containing `needle` (text only, no
+    position). See `rule_positions()` for what this guards against. Every
+    assertion about a RULE's presence or content must go through this (or
+    `rule_positions`/the guarded helpers below) -- never a bare
+    `needle in cfg` or `re.search(pattern, cfg)`.
+    """
+    return [line for line, _ in rule_positions(cfg, needle)]
+
+
+def require_rule(cfg, needle, what=None):
+    """The single rule line containing `needle`.
+
+    Raises a plain AssertionError naming what was being looked for -- not
+    IndexError from an unguarded `rule_lines(...)[0]`, and not
+    `ValueError: substring not found` from a bare `cfg.index(...)` -- when
+    the rule is missing. A missing rule and a broken test harness must not
+    look identical in a failure report.
+
+    Raises the same way if `needle` is ambiguous (matches more than one rule
+    line): silently taking the first match in that case would hide the
+    ambiguity instead of surfacing it.
+    """
+    label = what or needle
+    lines = rule_lines(cfg, needle)
+    if not lines:
+        raise AssertionError('no rule found for %r (expected: %s)' % (needle, label))
+    if len(lines) > 1:
+        raise AssertionError(
+            '%r matched %d rule lines, expected exactly one (%s): %r'
+            % (needle, len(lines), label, lines))
+    return lines[0]
+
+
+def require_position(cfg, needle, what=None, last=False):
+    """(line, char offset) for an ordering assertion, guarded the same way as
+    `require_rule` -- but tolerant of the needle matching multiple lines
+    (e.g. a multi-line set-var "chain"), since ordering checks often want the
+    first or last of several. Pass last=True for the last occurrence.
+    """
+    label = what or needle
+    positions = rule_positions(cfg, needle)
+    if not positions:
+        raise AssertionError(
+            'no rule found for %r, cannot check ordering (expected: %s)' % (needle, label))
+    return positions[-1] if last else positions[0]
+
+
+class WpAdminGate(unittest.TestCase):
+
+    def setUp(self):
+        self.cfg = render_listener()
+
+    def test_wp_admin_path_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_path', 'wp_admin_path ACL')
+        self.assertIn('path_reg', line)
+
+    def test_wp_admin_asset_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertIn('path_reg', line)
+
+    def test_wp_admin_allowed_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        self.assertIn('path_end', line)
+
+    def test_wp_gate_exempt_acl_declared(self):
+        """Scoped to the ACL line itself, not `self.cfg` as a whole -- the
+        surrounding prose (see hap_listener.tpl's "per-site opt-out" comment)
+        also spells out /etc/haproxy/wpadmin_gate_exempt.list verbatim, so an
+        unscoped `assertIn` would still pass with the real ACL deleted.
+        """
+        line = require_rule(self.cfg, 'acl wp_gate_exempt', 'wp_gate_exempt ACL')
+        self.assertIn('/etc/haproxy/wpadmin_gate_exempt.list', line)
+
+    def test_allowlist_entries_present(self):
+        """wp-login.php loads its own css/js from /wp-admin/ -- see module docstring."""
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        for entry in ALLOWLIST:
+            with self.subTest(entry=entry):
+                self.assertIn(entry, line)
+
+    def test_static_asset_dirs_allowed(self):
+        line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertRegex(line, r'path_reg.*\(css\|js\|images\)')
+
+    def test_install_php_is_NOT_allowlisted(self):
+        """install.php is deliberately gated -- a takeover vector on abandoned installs."""
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        self.assertNotIn('install.php', line)
+
+    def test_redirect_rule_has_all_exclusions(self):
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        for excl in EXCLUSIONS:
+            with self.subTest(exclusion=excl):
+                self.assertIn(excl, rule)
+
+    def test_rule_renders_after_real_ip_resolution(self):
+        """is_whitelisted reads txn.real_ip; before the set-var chain it is unset."""
+        _, setvar_pos = require_position(self.cfg, 'set-var(txn.real_ip)',
+                                          'real_ip set-var chain')
+        _, rule_pos = require_position(self.cfg, 'wp_admin_path', 'wp_admin_path ACL/rule')
+        self.assertLess(setvar_pos, rule_pos,
+                         'wp_admin_path renders before txn.real_ip is resolved')
+
+    def test_rule_renders_after_has_wp_logged_in_declared(self):
+        """HAProxy resolves ACLs as it parses; use-before-declare fails."""
+        _, decl_pos = require_position(self.cfg, 'acl has_wp_logged_in',
+                                        'has_wp_logged_in ACL declaration')
+        _, rule_pos = require_position(self.cfg, 'wp_admin_path', 'wp_admin_path ACL/rule')
+        self.assertLess(decl_pos, rule_pos,
+                         'wp_admin_path renders before has_wp_logged_in is declared')
+
+    def test_only_one_has_wp_logged_in_declaration(self):
+        self.assertEqual(len(rule_lines(self.cfg, 'acl has_wp_logged_in')), 1)
+
+    def test_allowlist_entries_are_anchored_to_wp_admin(self):
+        """Bare `path_end /admin-ajax.php` also matches
+        /wp-admin/evil/admin-ajax.php, which ALSO matches wp_admin_path
+        (path_reg only requires /wp-admin/ to appear somewhere) -- an
+        attacker-inserted path segment would then sail through the
+        allowlist ungated. Entries must be anchored to sit directly under
+        wp-admin/. Scoped to the captured ACL line only, since the
+        surrounding comment block also mentions these bare filenames.
+        """
+        line = require_rule(self.cfg, 'acl wp_admin_allowed', 'wp_admin_allowed ACL')
+        for entry in ALLOWLIST:
+            with self.subTest(entry=entry):
+                self.assertIn('/wp-admin' + entry, line)
+                self.assertNotRegex(
+                    line, r'(?<!wp-admin)' + re.escape(entry) + r'(?!\S)',
+                    'found a bare, unanchored allowlist entry: ' + entry)
+
+    def test_wp_login_url_setvar_renders_in_correct_order(self):
+        """The inline regsub-in-`location` form is rejected by real HAProxy
+        3.0.11 (invalid arg 2 in converter 'regsub'), so the regsub is
+        computed in its own set-var line instead. That set-var must render
+        after the set-var(txn.real_ip) chain (it must not disturb that
+        load-bearing chain) and before the redirect rule that consumes it.
+        """
+        _, last_real_ip_pos = require_position(self.cfg, 'set-var(txn.real_ip)',
+                                                'real_ip set-var chain', last=True)
+        _, wp_login_pos = require_position(self.cfg, 'set-var(txn.wp_login_url)',
+                                            'wp_login_url set-var')
+        _, redirect_pos = require_position(
+            self.cfg, 'http-request redirect code 302 location %[var(txn.wp_login_url)]',
+            'wp-admin redirect rule')
+        self.assertLess(last_real_ip_pos, wp_login_pos,
+                         'wp_login_url set-var must render after the real_ip set-var chain')
+        self.assertLess(wp_login_pos, redirect_pos,
+                         'wp_login_url set-var must render before the redirect rule that uses it')
+
+    def test_redirect_rule_uses_setvar_not_inline_regsub(self):
+        """Guards against reintroducing the rejected inline form."""
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        self.assertIn('%[var(txn.wp_login_url)]', rule)
+        self.assertNotIn('regsub', rule)
+
+    def test_redirect_rule_requires_safe_path(self):
+        """OPEN REDIRECT guard. The redirect target is built by rewriting
+        `path` with regsub, which only replaces the matched substring --
+        everything before "/wp-admin/" survives untouched in the output.
+        Three concrete requests turn that into an off-site `Location:`
+        header: "//evil.example.com/wp-admin/x.php" (protocol-relative,
+        browsers resolve "//host/path" to "https://host/path"),
+        "/\\evil.example.com/wp-admin/x.php" (browsers normalise a leading
+        "/\\" the same as "//"), and an RFC 7230 absolute-form request
+        target ("https://evil.example.com/wp-admin/x.php") which can make
+        HAProxy's `path` fetch return a full URI. wp_admin_safe_path
+        (requiring a well-formed absolute path) must be a POSITIVE
+        condition on the redirect rule -- scoped to the captured rule line
+        only, since the surrounding comment block also mentions this ACL
+        name and a bare substring match would pass even if the condition
+        were dropped from the rule itself.
+        """
+        rule = require_rule_by_predicate(
+            self.cfg, 'http-request redirect', lambda ln: 'wp_admin_path' in ln,
+            'wp-admin redirect rule')
+        self.assertIn('wp_admin_safe_path', rule)
+        self.assertNotIn('!wp_admin_safe_path', rule,
+                          'wp_admin_safe_path must be a positive condition, not negated')
+
+    def test_wp_admin_safe_path_acl_declared(self):
+        line = require_rule(self.cfg, 'acl wp_admin_safe_path', 'wp_admin_safe_path ACL')
+        self.assertIn('path_reg', line)
+
+    def test_unsafe_wp_admin_path_is_denied_not_passed_through(self):
+        """wp_admin_safe_path being a POSITIVE condition on the redirect means
+        a path that fails it is simply not redirected -- which used to mean it
+        fell through to the backend UNGATED, i.e. exactly the PHP-booting
+        request the gate exists to stop. Normalisation removes the "//"
+        spelling of that, but not "/\\", so the fall-through must be closed
+        with an explicit deny rather than left implicit.
+        """
+        denies = rule_lines(self.cfg, '!wp_admin_safe_path')
+        self.assertTrue(denies,
+                        'no rule denies a wp-admin path that fails wp_admin_safe_path')
+        self.assertTrue(any(d.startswith('http-request deny') for d in denies),
+                        'the !wp_admin_safe_path rule must be a deny: %r' % denies)
+
+
+def require_rule_by_predicate(cfg, needle, predicate, what):
+    """Like require_rule(), but for rules identified by needle + a predicate
+    over the comment-stripped line (e.g. "the `http-request redirect` line
+    that also mentions wp_admin_path", since the frontend has more than one
+    `http-request redirect`). Raises a clear AssertionError, not IndexError
+    or a silently-empty match, if no line satisfies both.
+    """
+    candidates = [ln for ln in rule_lines(cfg, needle) if predicate(ln)]
+    if not candidates:
+        raise AssertionError('no rule found matching %s' % what)
+    if len(candidates) > 1:
+        raise AssertionError('%s matched more than one rule line: %r' % (what, candidates))
+    return candidates[0]
+
+
+class UriNormalisation(unittest.TestCase):
+    """The gate matches the RAW path; the backend normalises and decodes it.
+    Every gap between those is a bypass -- five were found this way. These
+    tests pin the normalisation that closes the gap as a class.
+
+    NOTE: these are config-TEXT assertions. They are necessary but NOT
+    sufficient: the previous revision of this file passed while five live
+    bypasses shipped. The real evidence is the behavioural matrix run against
+    real haproxy 3.0.11 with raw sockets -- see
+    .superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-4-normalize-report.md.
+    """
+
+    def setUp(self):
+        self.cfg = render_listener()
+        self.header = render_header()
+
+    def test_experimental_directives_exposed_in_global(self):
+        """normalize-uri is experimental in 3.0; without this HAProxy refuses
+        to start (`haproxy -c` exits 1, ALERT). That does NOT crash-loop the
+        container, though -- see hap_header.tpl's comment and
+        haproxy_manager.py's start_haproxy()/do_initial_setup(): the failure
+        is swallowed, and the container comes up with haproxy simply never
+        running. This test exists so that silent-outage mode is never
+        reintroduced by dropping this line.
+        """
+        lines = rule_lines(self.header, 'expose-experimental-directives')
+        matches = [ln for ln in lines if ln == 'expose-experimental-directives']
+        self.assertTrue(
+            matches, 'expose-experimental-directives missing from the global section')
+
+    def test_all_normalizers_render(self):
+        for norm in NORMALIZERS:
+            with self.subTest(normalizer=norm):
+                self.assertTrue(
+                    rule_lines(self.cfg, 'normalize-uri ' + norm),
+                    'missing normalizer: ' + norm)
+
+    def test_normalizer_order_decode_before_path_walkers(self):
+        """Reverse this order and /wp-admin/js/%2e%2e/plugins.php reaches the
+        ACLs as /wp-admin/js/../plugins.php -- decoded but unresolved.
+        """
+        lines = [ln for ln in rule_lines(self.cfg, 'http-request normalize-uri')
+                 if ln.startswith('http-request normalize-uri')]
+        names = [ln.split('normalize-uri ', 1)[1] for ln in lines]
+        self.assertEqual(
+            names, list(NORMALIZERS),
+            'normalize-uri directives are missing, reordered, or duplicated: %r' % names)
+
+    def test_normalisation_precedes_every_path_based_rule(self):
+        """A normalizer placed after a path rule normalises nothing for it."""
+        norm_positions = []
+        for n in NORMALIZERS:
+            _, pos = require_position(self.cfg, 'http-request normalize-uri ' + n,
+                                       'normalizer: ' + n)
+            norm_positions.append(pos)
+        last_norm = max(norm_positions)
+        for marker in ('acl is_health_check', 'acl wp_login_path',
+                       'acl xmlrpc_path', 'acl wp_batch_path',
+                       'acl wp_admin_path', 'http-request set-path'):
+            with self.subTest(rule=marker):
+                _, marker_pos = require_position(self.cfg, marker, marker)
+                self.assertLess(last_norm, marker_pos,
+                                 marker + ' renders before URI normalisation')
+
+    def test_query_sort_by_name_is_not_enabled(self):
+        """query-sort-by-name reorders query parameters, which would break
+        anything that signs or caches on the exact query string. This is NOT
+        because the enabled normalizers already leave the query alone --
+        percent-to-uppercase and percent-decode-unreserved rewrite the WHOLE
+        request-target, query string included (see hap_listener.tpl's BLAST
+        RADIUS comment) -- it is a deliberate line between "case-fold /
+        decode" (no-ops under RFC 3986) and "reorder" (not a no-op for a
+        signed/cached query string).
+        """
+        self.assertFalse(rule_lines(self.cfg, 'normalize-uri query-sort-by-name'))
+
+    def test_dotdot_normalizer_uses_full(self):
+        """Without "full", ".." segments that climb above the root are left in
+        place and /../../wp-admin/plugins.php survives -- measured.
+        """
+        lines = rule_lines(self.cfg, 'normalize-uri path-strip-dotdot')
+        self.assertTrue(lines, 'missing normalizer: path-strip-dotdot')
+        for ln in lines:
+            self.assertTrue(ln.endswith('path-strip-dotdot full'), ln)
+
+    def test_encoded_separator_on_wp_admin_is_denied(self):
+        """percent-decode-unreserved deliberately leaves %2F encoded ("/" is
+        reserved), but OpenLiteSpeed decodes it and serves the file --
+        /wp-admin%2Fplugins.php was measured booting PHP on the OLS tier while
+        matching no wp-admin ACL. Normalisation cannot close this; it needs its
+        own rule.
+        """
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        acl_line = require_rule(self.cfg, 'acl path_has_encoded_sep', 'path_has_encoded_sep ACL')
+        self.assertIn('%2f', acl_line.lower())
+
+    def test_encoded_separator_deny_is_scoped_to_wp_admin(self):
+        """A blanket "deny any %2F in any path" would break non-WordPress
+        customer apps that legitimately pass an encoded slash in a path
+        parameter. The deny must be conditioned on the path mentioning
+        wp-admin.
+        """
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        for d in denies:
+            self.assertIn('wp_admin_word', d)
+
+    def test_encoded_separator_deny_honors_the_same_whitelist(self):
+        denies = [ln for ln in rule_lines(self.cfg, 'path_has_encoded_sep')
+                  if ln.startswith('http-request deny')]
+        self.assertTrue(denies, 'no deny rule for encoded separators')
+        for d in denies:
+            for excl in ('!has_wp_logged_in', '!wp_gate_exempt', '!is_local',
+                         '!is_trusted_ip', '!is_whitelisted'):
+                with self.subTest(rule=d, exclusion=excl):
+                    self.assertIn(excl, d)
+
+    def test_encoded_separator_acl_matches_a_substring_not_a_prefix(self):
+        """/blog%2Fwp-admin/plugins.php hides the separator BEFORE "wp-admin",
+        where an anchored pattern never matches, and OLS still resolves it.
+        """
+        acl_line = require_rule(self.cfg, 'acl path_has_encoded_sep', 'path_has_encoded_sep ACL')
+        self.assertIn('-m sub', acl_line)
+
+    def test_wp_admin_asset_bypass_cannot_cover_a_php_entrypoint(self):
+        """The asset bypass anchored its prefix but not its suffix, so
+        /wp-admin/css/../plugins.php took it and the backend then resolved
+        ".." and booted plugins.php. path-strip-dotdot is the real fix; this
+        keeps the bypass structurally incapable of covering PHP.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertIn('.php', acl_line,
+                      'wp_admin_asset must exclude .php explicitly')
+
+    def test_wp_admin_asset_pattern_is_end_of_flags_guarded(self):
+        """A pattern starting with "(" makes HAProxy warn on EVERY load/reload:
+
+            parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
+            '(^|/)wp-admin/...' is likely a mistake ... Maybe you need to
+            remove the extraneous space before '('.
+
+        "--" is the end-of-flags marker HAProxy itself names as the fix. It is
+        cosmetic to matching but not to operations: an unsilenced warning on
+        every reload on every host trains people to skim past warnings, which
+        is how a real one gets missed. Assert the pattern is still the one we
+        think it is, so this can never pass by the pattern having been changed.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_asset', 'wp_admin_asset ACL')
+        self.assertRegex(
+            acl_line, r'path_reg\s+--\s+\(',
+            'wp_admin_asset pattern begins with "(" and MUST be preceded by "--"')
+        self.assertIn('(^|/)wp-admin/(css|js|images)/', acl_line)
+
+    def test_case_insensitive_acl_and_regsub_are_kept_in_sync(self):
+        """A case-insensitive wp_admin_path with a case-sensitive regsub is an
+        INFINITE REDIRECT LOOP: regsub finds no "/wp-admin/" in
+        "/WP-ADMIN/plugins.php", returns `path` unchanged, and the Location
+        then points at the request's own URL.
+        """
+        acl_line = require_rule(self.cfg, 'acl wp_admin_path', 'wp_admin_path ACL')
+        setvar_line = require_rule(self.cfg, 'set-var(txn.wp_login_url)', 'wp_login_url set-var')
+        acl_ci = bool(re.search(r'path_reg\s+-i\s', acl_line))
+        regsub_ci = bool(re.search(r'regsub\([^)]*,\s*i\)', setvar_line))
+        self.assertEqual(
+            acl_ci, regsub_ci,
+            'wp_admin_path case-sensitivity (%s) and regsub flags (%s) disagree'
+            % (acl_line, setvar_line))
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-xmlrpc-rate-limit.py

@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""Regression tests for per-client-IP rate limiting on POST /xmlrpc.php.
+
+Why this file exists
+--------------------
+POST /xmlrpc.php floods were unthrottled fleet-wide. The generic frontend
+rate limits (hap_listener.tpl) trigger at 3000/5000 req/10s -- i.e. 300-500
+req/s -- but the observed floods run at a few req/s for hours, well under
+that ceiling. The existing wp_bruteforce mechanism (dedicated stick-table,
+60s window, per real client IP) solves exactly this shape of problem for
+POST /wp-login.php; this change adds an equivalent dedicated table/rule pair
+for POST /xmlrpc.php.
+
+This is only safe to key on var(txn.real_ip) because of the trusted-proxy
+gate added earlier (release 2026.08.3, see test-trusted-proxy-gate.py) --
+before that fix, a direct client could spoof any client IP via
+X-Forwarded-For and evade all per-IP tracking.
+
+These tests pin:
+  - a dedicated stick-table for xmlrpc tracking exists in
+    hap_security_tables.tpl (own sc slot / own counter -- not sharing the
+    wp_bruteforce counter, so a wp-login brute-force run and an xmlrpc flood
+    from the same IP don't inflate each other's rate)
+  - the tracking rule only fires on POST /xmlrpc.php (path_end, so
+    subdirectory WP installs are covered)
+  - the limiting rule tarpits over the chosen threshold
+  - the limiting rule honors the same whitelist as every other rule in the
+    file (!is_local !is_trusted_ip !is_whitelisted)
+  - xmlrpc is not blocked outright -- only the rate-limit ACL is present,
+    there's no blanket deny of the path
+
+Running
+-------
+    python3 scripts/test-xmlrpc-rate-limit.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_security_tables():
+    return haproxy_manager.template_env.get_template(
+        'hap_security_tables.tpl').render()
+
+
+class XmlrpcRateLimit(unittest.TestCase):
+
+    def setUp(self):
+        self.listener_cfg = render_listener()
+        self.tables_cfg = render_security_tables()
+
+    def test_dedicated_stick_table_defined(self):
+        """A dedicated table (not wp_bruteforce) tracks xmlrpc requests, so a
+        wp-login brute-force run and an xmlrpc flood from the same IP can't
+        inflate each other's rate counter."""
+        self.assertRegex(
+            self.tables_cfg,
+            r'backend\s+xmlrpc_bruteforce\s*\n\s*stick-table\s+type\s+ip\b.*store.*http_req_rate',
+        )
+        # Must not be the same table wp-login already uses.
+        self.assertNotIn('backend wp_bruteforce\n    stick-table type ip size 100k expire 30m store http_req_rate(60s)\nbackend xmlrpc_bruteforce', self.tables_cfg)
+
+    def test_xmlrpc_path_acl_uses_path_end(self):
+        """path_end (not path_beg) so subdirectory WP installs are covered,
+        matching the wp-login rule's reasoning."""
+        self.assertRegex(
+            self.listener_cfg,
+            r'acl\s+xmlrpc_path\s+path_end\s+/xmlrpc\.php',
+        )
+
+    def test_tracking_rule_only_fires_on_post_xmlrpc(self):
+        self.assertRegex(
+            self.listener_cfg,
+            r'http-request\s+track-sc2\s+var\(txn\.real_ip\)\s+table\s+xmlrpc_bruteforce\s+if\s+METH_POST\s+xmlrpc_path',
+        )
+
+    def test_limiting_rule_tarpits_over_threshold_with_whitelist(self):
+        pattern = (
+            r'http-request\s+tarpit\s+deny_status\s+429\s+if\s+METH_POST\s+xmlrpc_path\s+'
+            r'\{\s*sc_http_req_rate\(2\)\s+gt\s+(\d+)\s*\}\s+'
+            r'!is_local\s+!is_trusted_ip\s+!is_whitelisted'
+        )
+        match = re.search(pattern, self.listener_cfg)
+        self.assertIsNotNone(
+            match, 'expected a tarpit rule tracking sc2 with the full whitelist')
+        threshold = int(match.group(1))
+        self.assertGreater(threshold, 0)
+
+    def test_xmlrpc_not_blocked_outright(self):
+        """The endpoint must remain functional for clients under the
+        threshold -- only a rate-limit ACL, no blanket deny of the path."""
+        self.assertNotRegex(
+            self.listener_cfg,
+            r'http-request\s+deny\s+deny_status\s+\d+\s+if\s+(?:METH_POST\s+)?xmlrpc_path\s*(?:!is_local|\n)',
+        )
+
+    def test_rule_order_after_wp_login_block(self):
+        """Not load-bearing for correctness (mutually exclusive paths), but
+        keep the new block grouped with the other WordPress-specific rules
+        rather than scattered elsewhere in the file."""
+        wp_login_idx = self.listener_cfg.index('wp_login_path')
+        xmlrpc_idx = self.listener_cfg.index('xmlrpc_path')
+        self.assertLess(wp_login_idx, xmlrpc_idx)
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/validate-rendered-config.py

@@ -0,0 +1,423 @@
+#!/usr/bin/env python3
+"""Build gate: render the real HAProxy config and hand it to the real `haproxy -c`.
+
+Why this file exists
+--------------------
+On 2026-08-14 a change to hap_listener.tpl rendered perfectly, passed every
+unit test in scripts/ (13 green), and was then rejected outright by HAProxy:
+
+    [ALERT] config : parsing [/etc/haproxy/haproxy.cfg:...] :
+            invalid arg 2 in converter 'regsub' : ... unexpected empty
+            replacement string
+
+Nothing between "commit" and "running in production" would have caught it.
+The unit tests assert on the *text* of the rendered config with regexes, which
+tells you what the template says, never whether HAProxy will accept it. And
+scripts/test-config-rollback.py stubs the `haproxy` binary with a shell script
+that only rejects a literal sentinel token, so its "validation" has never
+parsed a single line of real HAProxy syntax.
+
+The failure mode this guards is not cosmetic. When haproxy.cfg is invalid,
+scripts/init.py refuses to start HAProxy but the container still comes up:
+ports 80/443 are unbound, every site on the host is down, and /health keeps
+answering 200 because the Flask API is fine.
+
+So: render the config through the SAME code path production uses
+(haproxy_manager.generate_config(), templates and all), then run the actual
+`haproxy -c` against the result and gate on its exit code.
+
+This runs as a RUN step in the Dockerfile, which means it also validates
+against the exact haproxy binary that ships in the image being built - note
+that the Dockerfile installs haproxy UNPINNED, so that binary can move under
+us between builds. Any syntax the new binary rejects now fails the build
+instead of failing at 3am on an edge node.
+
+What it covers
+--------------
+  * every template generate_config() touches, assembled in the real order
+  * a domain with SSL + a backend, a wildcard domain, a cert-only domain with
+    no backend, and two template_override backends
+  * blocked-IP map entries (single IP and CIDR)
+  * BOTH sides of the two conditional blocks in hap_listener.tpl -
+    {%- if suspension_enabled %} and {%- if coraza_spoe_backend %} - because a
+    syntax error inside a conditional ships undetected otherwise. Scenario
+    "full" turns both on; scenario "default" leaves both off, which is the
+    byte-identical-to-standalone shape.
+
+Warnings vs failures
+--------------------
+`haproxy -c` emits warnings on a clean config here (at minimum "Can't load
+stats file" because /var/lib/haproxy/stats.dat doesn't exist at build time,
+plus assorted path_reg/ACL advisories). Those are NOT failures. This gate keys
+on the process EXIT CODE only, and dumps the full output when it is non-zero.
+
+Running
+-------
+    python3 scripts/validate-rendered-config.py
+
+Needs the real `haproxy` binary, the application's Python dependencies, and
+write access to /etc/haproxy (several templates reference files there by
+absolute path - see _REAL_PATH_NOTE below). Inside the image build all three
+hold. On a workstation, run it in the container instead.
+"""
+
+import logging
+import os
+import re
+import shutil
+import sqlite3
+import subprocess
+import sys
+import tempfile
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+# Same trick the other suites use: haproxy_manager configures logging at import
+# time against /var/log/haproxy-manager.log. Redirect the handlers so this runs
+# without root and without polluting the image's log files.
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-validate-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+try:
+    import haproxy_manager as hm  # noqa: E402
+finally:
+    logging.FileHandler = _real_file_handler
+
+# The application logs a lot at INFO during a render, and legitimately logs at
+# ERROR about things that are only true in this harness ("no existing HAProxy
+# config on disk ... ROLLBACK IS NOT AVAILABLE" - correct, there is no live
+# config during a build). Silence it so the build log carries the gate's own
+# verdict and haproxy's output, which is what matters.
+logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
+
+
+# _REAL_PATH_NOTE
+# ---------------
+# Most paths haproxy_manager writes to are module-level constants and are
+# redirected into a temp dir below. Two cannot be:
+#
+#   /etc/haproxy/blocked_ips.map   - hardcoded inside hap_listener.tpl's
+#                                    map_ip() converter
+#   /etc/haproxy/coraza-spoe.cfg   - hardcoded in the `filter spoe engine`
+#                                    line, and parsed by haproxy -c
+#
+# Redirecting the constants without editing the templates would just make
+# haproxy read a different (missing) file, so those two are left at their real
+# paths. Everything this script creates under /etc/haproxy is removed again on
+# exit; pre-existing files (the baked trusted_ips.*) are never touched.
+ETC_HAPROXY = '/etc/haproxy'
+
+# Files referenced with `-f` / map_ip() from the templates. A missing `-f` file
+# is a FATAL haproxy error, so a gate that didn't create these would fail for
+# reasons that have nothing to do with the config being tested.
+STUB_FILES = {
+    os.path.join(ETC_HAPROXY, 'trusted_ips.list'): '# validation stub\n203.0.113.10\n',
+    os.path.join(ETC_HAPROXY, 'trusted_ips.map'): '# validation stub\n203.0.113.11 1\n',
+    os.path.join(ETC_HAPROXY, 'cloudflare_ips.list'): '# validation stub\n198.51.100.0/24\n',
+    os.path.join(ETC_HAPROXY, 'trusted_proxies.list'): '# validation stub\n192.0.2.0/24\n',
+    os.path.join(ETC_HAPROXY, 'wpadmin_gate_exempt.list'): '# validation stub\nexempt.example.test\n',
+    os.path.join(ETC_HAPROXY, 'suspended_domains.list'): 'suspended.example.test\n',
+    # `lf-file` on the Coraza deny rule; loaded at parse time. Present in the
+    # image (COPY errors /haproxy/errors), stubbed for anything else.
+    '/haproxy/errors/403-waf.html': '<html><body>blocked %[unique-id]</body></html>\n',
+}
+
+# (suspension_enabled, coraza_spoe_backend) combinations to render + validate.
+SCENARIOS = (
+    ('default', {}),
+    ('full', {
+        'HAPROXY_SUSPENSION_ENABLED': 'true',
+        'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
+    }),
+)
+
+
+def log(msg):
+    sys.stdout.write(f'[validate-config] {msg}\n')
+    sys.stdout.flush()
+
+
+def fail(msg):
+    sys.stderr.write(f'[validate-config] FAIL: {msg}\n')
+    sys.stderr.flush()
+    raise SystemExit(1)
+
+
+class CreatedFiles:
+    """Tracks what we put on disk outside the temp dir so it can be removed.
+
+    Two sources: files we create explicitly, and files generate_config() itself
+    writes into /etc/haproxy (blocked_ips.map, coraza-spoe.cfg and their
+    .backup copies). The latter are caught by diffing the directory listing,
+    which also picks up anything a future change starts writing there.
+    """
+
+    def __init__(self):
+        self.explicit = []
+        self.etc_before = self._listdir(ETC_HAPROXY)
+
+    @staticmethod
+    def _listdir(path):
+        try:
+            return set(os.listdir(path))
+        except OSError:
+            return set()
+
+    def ensure(self, path, content):
+        """Create path with content if it does not already exist."""
+        if os.path.exists(path):
+            return
+        os.makedirs(os.path.dirname(path), exist_ok=True)
+        with open(path, 'w') as fh:
+            fh.write(content)
+        os.chmod(path, 0o644)
+        self.explicit.append(path)
+
+    def cleanup(self):
+        for path in self.explicit:
+            try:
+                os.unlink(path)
+            except OSError:
+                pass
+        for name in self._listdir(ETC_HAPROXY) - self.etc_before:
+            try:
+                os.unlink(os.path.join(ETC_HAPROXY, name))
+            except OSError:
+                pass
+
+
+def require_haproxy_binary():
+    """Fail closed. A gate that skips itself when the binary is missing is a
+    gate that would have let the 2026-08-14 change through."""
+    path = shutil.which('haproxy')
+    if not path:
+        fail('no `haproxy` binary on PATH - this gate cannot validate anything. '
+             'Run it inside the image (the Dockerfile installs haproxy).')
+    version = subprocess.run([path, '-v'], capture_output=True, text=True)
+    log(f'using {path}: {version.stdout.strip().splitlines()[0] if version.stdout else "unknown version"}')
+
+
+def make_self_signed_cert(certs_dir):
+    """HAProxy loads every file in the `bind ... ssl crt <dir>` directory at
+    parse time, so the directory has to hold a real, loadable bundle."""
+    os.makedirs(certs_dir, exist_ok=True)
+    cert = os.path.join(certs_dir, 'cert.tmp')
+    key = os.path.join(certs_dir, 'key.tmp')
+    subprocess.run(
+        ['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes',
+         '-keyout', key, '-out', cert, '-days', '1',
+         '-subj', '/CN=validate.example.test'],
+        check=True, capture_output=True,
+    )
+    bundle = os.path.join(certs_dir, 'validate.example.test.pem')
+    with open(bundle, 'w') as out:
+        for part in (cert, key):
+            with open(part) as fh:
+                out.write(fh.read())
+    os.unlink(cert)
+    os.unlink(key)
+    os.chmod(bundle, 0o600)
+
+
+def seed_database(db_path):
+    """A representative fleet: SSL + backend, wildcard, cert-only, overrides."""
+    hm.DB_FILE = db_path
+    hm.init_db()
+    with sqlite3.connect(db_path) as conn:
+        cur = conn.cursor()
+
+        def add_site(domain, backend, ssl_enabled=1, wildcard=0, override=None,
+                     servers=(('web1', '10.0.0.10', 8080, 'check'),)):
+            cur.execute(
+                'INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
+                'template_override, is_wildcard) VALUES (?, ?, ?, ?, ?)',
+                (domain, ssl_enabled, f'/etc/haproxy/certs/{domain}.pem',
+                 override, wildcard),
+            )
+            domain_id = cur.lastrowid
+            cur.execute('INSERT INTO backends (name, domain_id, settings) '
+                        'VALUES (?, ?, ?)', (backend, domain_id, None))
+            backend_id = cur.lastrowid
+            for name, addr, port, opts in servers:
+                cur.execute(
+                    'INSERT INTO backend_servers (backend_id, server_name, '
+                    'server_address, server_port, server_options) '
+                    'VALUES (?, ?, ?, ?, ?)',
+                    (backend_id, name, addr, port, opts),
+                )
+
+        add_site('site-one.example.test', 'site-one',
+                 servers=(('web1', '10.0.0.10', 8080, 'check'),
+                          ('web2', '10.0.0.11', 8080, 'check backup')))
+        add_site('site-two.example.test', 'site-two', ssl_enabled=0)
+        add_site('*.wildcard.example.test', 'wildcard-site', wildcard=1)
+        add_site('ws.example.test', 'ws-site', override='hap_backend_websocket')
+        add_site('sse.example.test', 'sse-site', override='hap_backend_longlived')
+
+        # Cert/management-only domain: registered for certificates, no backend.
+        # generate_config() has an explicit branch for this.
+        cur.execute(
+            'INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
+            'template_override, is_wildcard) VALUES (?, ?, ?, ?, ?)',
+            ('panel.example.test', 1, '/etc/haproxy/certs/panel.pem', None, 0),
+        )
+
+        # Both map_ip() shapes: a single address and a CIDR.
+        for ip in ('203.0.113.66', '198.51.100.0/24'):
+            cur.execute('INSERT INTO blocked_ips (ip_address, reason, blocked_by) '
+                        'VALUES (?, ?, ?)', (ip, 'validation fixture', 'gate'))
+        conn.commit()
+
+
+def render(scenario_name, env_overrides, workdir):
+    """Render via generate_config() and return the path to the assembled config.
+
+    generate_config() is the real production entry point: it takes the rollback
+    snapshot, writes the blocked-IP map, renders every template, and writes
+    haproxy.cfg. Only the reload is stubbed - there is no HAProxy process to
+    reload during a build, and validating the file is the whole point.
+    """
+    scenario_dir = os.path.join(workdir, scenario_name)
+    certs_dir = os.path.join(scenario_dir, 'certs')
+    os.makedirs(scenario_dir)
+    make_self_signed_cert(certs_dir)
+
+    hm.HAPROXY_CONFIG_PATH = os.path.join(scenario_dir, 'haproxy.cfg')
+    hm.HAPROXY_BACKUP_PATH = os.path.join(scenario_dir, 'haproxy.cfg.backup')
+    hm.CLUSTER_SECRET_PATH = os.path.join(scenario_dir, 'cluster-secret')
+    hm.HAPROXY_SOCKET_PATH = os.path.join(scenario_dir, 'haproxy.sock')
+    hm.SSL_CERTS_DIR = certs_dir
+    seed_database(os.path.join(scenario_dir, 'haproxy_config.db'))
+
+    saved_env = {}
+    for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND'):
+        saved_env[key] = os.environ.pop(key, None)
+    os.environ.update(env_overrides)
+
+    real_reload = hm.reload_haproxy_safely
+    hm.reload_haproxy_safely = lambda *a, **kw: (True, 'reload skipped: build-time validation')
+    try:
+        hm.generate_config()
+    finally:
+        hm.reload_haproxy_safely = real_reload
+        for key, value in saved_env.items():
+            os.environ.pop(key, None)
+            if value is not None:
+                os.environ[key] = value
+
+    return hm.HAPROXY_CONFIG_PATH
+
+
+def assert_scenario_branches(scenario_name, config_path, env_overrides):
+    """Cheap sanity check that the conditional blocks actually rendered.
+
+    Without this, a template refactor that silently stopped emitting the
+    suspension or Coraza block would leave the gate passing while covering
+    less than it claims to.
+    """
+    with open(config_path) as fh:
+        text = fh.read()
+    expected = {
+        'suspension': ('acl is_suspended_domain',
+                       'HAPROXY_SUSPENSION_ENABLED' in env_overrides),
+        'coraza': ('filter spoe engine coraza',
+                   'HAPROXY_CORAZA_SPOE_BACKEND' in env_overrides),
+    }
+    for label, (needle, should_be_present) in expected.items():
+        present = needle in text
+        if present != should_be_present:
+            fail(f'[{scenario_name}] {label} block {"missing" if should_be_present else "unexpectedly present"} '
+                 f'in the rendered config (looked for {needle!r}). The gate is '
+                 f'not covering what it thinks it is.')
+
+
+def _dump_context(config_path, output):
+    """Print the rendered lines HAProxy complained about.
+
+    The temp dir is deleted on the way out, so the build log has to carry the
+    evidence. HAProxy reports `parsing [<file>:<line>]`; show a window around
+    each reported line rather than dumping ~1500 lines of config.
+    """
+    line_numbers = sorted({
+        int(n) for n in re.findall(
+            r'parsing \[' + re.escape(config_path) + r':(\d+)\]', output)
+    })
+    if not line_numbers:
+        return
+    with open(config_path) as fh:
+        lines = fh.read().splitlines()
+    sys.stderr.write('[validate-config] --- rendered config around the error ---\n')
+    for number in line_numbers:
+        start = max(1, number - 6)
+        end = min(len(lines), number + 4)
+        for index in range(start, end + 1):
+            marker = '>>' if index == number else '  '
+            sys.stderr.write(f'{marker}{index:6d}| {lines[index - 1]}\n')
+        sys.stderr.write('[validate-config] ---\n')
+
+
+def validate(scenario_name, config_path):
+    result = subprocess.run(['haproxy', '-c', '-f', config_path],
+                            capture_output=True, text=True)
+    if result.returncode != 0:
+        output = result.stdout + result.stderr
+        sys.stderr.write(
+            f'\n[validate-config] ===== {scenario_name}: haproxy REJECTED the '
+            f'rendered configuration (exit {result.returncode}) =====\n')
+        sys.stderr.write(output if output.endswith('\n') else output + '\n')
+        _dump_context(config_path, output)
+        sys.stderr.write(
+            '[validate-config] This is a real HAProxy parse failure. Shipping it '
+            'would leave the container Up with ports 80/443 unbound and every '
+            'site on the host down, while /health still returns 200.\n')
+        sys.stderr.flush()
+        raise SystemExit(1)
+
+    # Exit code 0 is the verdict. Warnings are expected and are NOT failures:
+    # "Can't load stats file" always fires at build time, and HAProxy emits
+    # path_reg/ACL advisories on a perfectly valid config.
+    noise = (result.stdout + result.stderr).strip()
+    log(f'{scenario_name}: haproxy -c OK (exit 0)')
+    if noise:
+        for line in noise.splitlines():
+            log(f'  {scenario_name}: haproxy said: {line}')
+
+
+def main():
+    require_haproxy_binary()
+
+    if not os.path.isdir(ETC_HAPROXY) or not os.access(ETC_HAPROXY, os.W_OK):
+        fail(f'{ETC_HAPROXY} must exist and be writable - several templates '
+             'reference files there by absolute path. Run this inside the image.')
+
+    created = CreatedFiles()
+    workdir = tempfile.mkdtemp(prefix='haproxy-validate-')
+    try:
+        for path, content in STUB_FILES.items():
+            created.ensure(path, content)
+
+        for scenario_name, env_overrides in SCENARIOS:
+            log(f'rendering scenario "{scenario_name}" '
+                f'({env_overrides or "no optional features"})')
+            config_path = render(scenario_name, env_overrides, workdir)
+            assert_scenario_branches(scenario_name, config_path, env_overrides)
+            validate(scenario_name, config_path)
+    finally:
+        created.cleanup()
+        shutil.rmtree(workdir, ignore_errors=True)
+        shutil.rmtree(_LOG_DIR, ignore_errors=True)
+
+    log('all scenarios accepted by the real haproxy binary')
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

templates/hap_backend.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -29,7 +29,7 @@ backend {{ name }}-sse-backend
 
     option forwardfor
     # Pass the real client IP to backend (from proxy headers or direct connection)
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_http_check.tpl

@@ -4,7 +4,7 @@ backend {{ name }}-backend
     option httpchk
     # Pass the real client IP to backend (from proxy headers or direct connection)
     # This is crucial for container-level logging and security tools
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     {% if ssl_enabled %}http-request set-header X-Forwarded-Proto https if { ssl_fc }{% endif %}

templates/hap_backend_longlived.tpl

@@ -14,7 +14,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -31,7 +31,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_backend_websocket.tpl

@@ -7,7 +7,7 @@ backend {{ name }}-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -24,7 +24,7 @@ backend {{ name }}-sse-backend
     timeout tunnel 6h
     timeout http-keep-alive 6h
     option forwardfor
-    http-request add-header X-CLIENT-IP %[var(txn.real_ip)]
+    http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
     http-request set-header X-Real-IP %[var(txn.real_ip)]
     http-request set-header X-Forwarded-For %[var(txn.real_ip)]
     http-request set-header X-Forwarded-Proto https if { ssl_fc }

templates/hap_coraza_spoe_engine.tpl

@@ -37,7 +37,22 @@ spoe-agent coraza
     timeout     processing  100ms
 
     use-backend coraza-spoa-backend
-    log         global
+
+    # NO `log global` here, deliberately.
+    #
+    # `log global` in a spoe-agent emits one line PER INSPECTED REQUEST, e.g.
+    #   SPOE: [coraza] <GROUP:coraza-req> sid=537 st=0 0/0/0/0/0 32/32 0/0 0/467
+    # Measured on whp01 immediately after access logging started working:
+    # 618 SPOE lines vs 669 real access lines -- it was ~48% of the log volume,
+    # i.e. it would roughly DOUBLE the edge's log footprint (~400 MB/day extra)
+    # to record `st=0` over and over.
+    #
+    # It carries nothing incident response needs: the WAF's verdict is already
+    # visible in the access log line (status 403 + the `id=` UUID, which joins
+    # to /var/log/coraza/audit.log for the rule_id), and per-transaction WAF
+    # detail is written by the SPOA itself to /var/log/coraza/spoa.log.
+    # Agent-level failures still surface via `option set-on-error error` ->
+    # var(txn.coraza.error) and the fail-open path in hap_listener.tpl.
 
 # Per-request inspection message. No `event` directive — fires only when
 # explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.

templates/hap_header.tpl

@@ -2,20 +2,46 @@
 # Global settings
 #---------------------------------------------------------------------
 global
-    # to have these messages end up in /var/log/haproxy.log you will
+    # ACCESS LOG DESTINATION.
-    # need to:
     #
-    # 1) configure syslog to accept network log events.  This is done
+    # This used to be `log 127.0.0.1 local2`, which was a silent black hole:
-    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
+    # 127.0.0.1 is the CONTAINER's own loopback, nothing has ever listened on
-    #    /etc/sysconfig/syslog
+    # udp/514 in the container netns, and there is no /dev/log in the image.
+    # Every access log line -- ~1.5M/day across the whole edge -- was written
+    # to a socket with no receiver and dropped. Nothing errored, nothing
+    # warned, and `haproxy -c` was perfectly happy. The cost only shows up
+    # during an incident: per-IP 429s, tarpits, wp-admin gate redirects,
+    # WAF 403s and `silent-drop`s left no record anywhere, so the edge could
+    # not be asked what it had rejected. Only aggregate stick-table counters
+    # survived.
     #
-    # 2) configure local2 events to go to the /var/log/haproxy.log
+    # Now points at the DOCKER BRIDGE GATEWAY, where the host's rsyslog has an
-    #   file. A line like the following can be added to
+    # imudp listener bound (installed idempotently by WHP's
-    #   /etc/sysconfig/syslog
+    # setup-haproxy-logrotate.sh, which also writes the logrotate stanza).
+    # The host writes local2 to /var/log/haproxy.log and stops it there, so it
+    # does not also flood /var/log/messages or the Graylog forwarder.
     #
-    #    local2.*                       /var/log/haproxy.log
+    # WHY NOT `log stdout format raw local0`: it is INCOMPATIBLE with the
+    # `daemon` keyword below, and incompatible SILENTLY. Verified on the
+    # pinned 3.0.11 binary: with `daemon` set, a `log stdout` config serves
+    # traffic normally and emits ZERO log lines, and `haproxy -c` returns 0
+    # with no error and no warning -- so the CI config gate
+    # (scripts/validate-rendered-config.py) cannot catch it either. Making it
+    # work means dropping `daemon` / adding -db so haproxy stays in the
+    # foreground, which in turn breaks the three synchronous
+    # `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites in
+    # haproxy_manager.py (they would block until the 180s timeout and then be
+    # killed). That is a change to the exact code path whose failure mode is
+    # "container Up, ports 80/443 never bound, every site down, /health still
+    # 200". Not worth it for a logging change.
     #
-    log         127.0.0.1 local2
+    # UDP means a dead listener degrades to dropped log lines, never to a
+    # stalled or failing request path -- the correct failure direction for an
+    # edge fronting ~60 customer sites.
+    #
+    # len 2048 accommodates the enriched log-format in hap_listener.tpl
+    # (URL + User-Agent + UUID); the default 1024 would truncate long ones.
+    log         {{ syslog_target }} len 2048 format rfc5424 local2 info
 
     chroot      /var/lib/haproxy
     pidfile     /var/run/haproxy.pid
@@ -27,6 +53,36 @@ global
     # SSL and Performance
     tune.ssl.default-dh-param 2048
 
+    # Required by the `http-request normalize-uri` chain at the top of the
+    # `web` frontend (hap_listener.tpl). normalize-uri is still flagged
+    # EXPERIMENTAL in HAProxy 3.0, and HAProxy REFUSES TO START without this
+    # opt-in -- not a warning, a fatal:
+    #   [ALERT] config : parsing [...] : 'normalize-uri' action is
+    #   experimental, must be allowed via a global
+    #   'expose-experimental-directives'
+    # (verified against real haproxy 3.0.11-9e587df: `haproxy -c` exits 1).
+    # So this line and the normalize-uri rules must be added/removed together.
+    #
+    # Dropping this one alone does NOT crash-loop the container -- the truth
+    # is worse: it is a SILENT TOTAL OUTAGE that nothing escalates. Container
+    # init (scripts/init.py -> haproxy_manager.do_initial_setup()) calls
+    # generate_config() (which still succeeds -- Jinja doesn't validate
+    # HAProxy semantics) and then start_haproxy(), which runs `haproxy -c`,
+    # sees it fail, logs an error, and RETURNS WITHOUT RAISING. init.py exits
+    # 0. scripts/start-up.sh then execs gunicorn as PID 1 regardless. Result:
+    # the container stays "Up", ports 80/443 are never bound, EVERY SITE ON
+    # THE HOST IS DOWN, and the in-container supervisor loop
+    # (ensure_haproxy.py, every HAPROXY_SUPERVISOR_INTERVAL seconds) retries
+    # the identical failing render forever without ever escalating. Worse
+    # still, GET /health keeps returning HTTP 200 -- health_check() only
+    # answers 500 on a database error; a dead haproxy just flips the JSON
+    # body's "haproxy_status" to "stopped" while the status code a naive
+    # monitor checks never changes. Do not trust /health alone to catch this.
+    #
+    # This exposes ONLY the experimental directives that are actually used --
+    # it does not change the behaviour of anything else in this file.
+    expose-experimental-directives
+
     # HTTP/3 over QUIC. The Debian haproxy package is built against system
     # OpenSSL via the compatibility shim (USE_QUIC_OPENSSL_COMPAT), which is
     # not a native QUIC TLS stack. HAProxy therefore rejects `quic*@` binds
@@ -92,7 +148,11 @@ defaults
     maxconn                 3000
 
     # Per-request unique reference, used:
-    #   - in the log line (httplog includes %ID)
+    #   - in the access log line, as the `id=` field of the custom log-format
+    #     in hap_listener.tpl. NOTE: `option httplog` does NOT include %ID
+    #     (verified against haproxy 3.0.11) -- this comment used to claim it
+    #     did, which made the support workflow below look supported when it
+    #     was not. The explicit log-format is what actually carries it.
     #   - echoed to clients in the X-Request-Reference response header on
     #     WAF blocks so a customer can quote it when opening a support ticket
     #   - embedded in /etc/haproxy/errors/403-waf.html so a blocked visitor

templates/hap_listener.tpl

@@ -19,9 +19,171 @@ frontend web
     # response, including haproxy-generated ones (blocks, default page).
     http-after-response set-header alt-svc "h3=\":443\"; ma=86400"
 
-    # Capture Host header so it appears in httplog output (in %hr field)
+    # Capture Host header so it appears in httplog output (in %hr field).
+    # ORDER IS LOAD-BEARING: this is capture slot 0, referenced by the
+    # access log-format below as %[capture.req.hdr(0)].
     http-request capture req.hdr(Host) len 64
 
+    # Capture slot 1 = User-Agent. Incident response needs it to tell a
+    # scanner from a browser, and it is not in `option httplog` output.
+    # Any new capture MUST be appended AFTER this line, never inserted
+    # above it, or the slot indices in the log-format silently shift and
+    # the access log starts attributing the wrong string to the wrong field.
+    # req.fhdr(), NOT req.hdr(): req.hdr() treats the header as a comma-
+    # separated list and returns only the LAST element. Real User-Agent strings
+    # contain commas -- "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
+    # AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
+    # captured with req.hdr() logs as just "like Gecko) Chrome/131.0.0.0
+    # Safari/537.36", silently losing the platform half -- which is exactly the
+    # half you need to tell a spoofed crawler from a real browser.
+    # Observed in production on whp01 before this was corrected.
+    http-request capture req.fhdr(User-Agent) len 200
+
+    # --- Access logging -----------------------------------------------------
+    # Scoped to THIS frontend on purpose: it references capture slots and
+    # var(txn.real_ip), which only exist here. Putting it in `defaults` would
+    # apply it to the stats frontend and every backend too, where those
+    # samples are undefined.
+    #
+    # `log-format` overrides `option httplog` for this proxy (haproxy emits a
+    # harmless warning saying so). The first 16 fields are byte-identical to
+    # the 3.0 httplog default, so anything that already parses httplog keeps
+    # working; the `key=value` tail is additive.
+    #
+    # Why the default httplog is not enough for incident response:
+    #   %ci  is the PROXY's address for Cloudflare-fronted sites, not the
+    #        visitor. The real client is var(txn.real_ip), resolved further
+    #        down from CF-Connecting-IP / X-Real-IP / X-Forwarded-For and only
+    #        honoured from trusted proxies. BOTH are logged: cip= is who to
+    #        rate-limit or block, %ci is which edge it arrived through.
+    #   %ID  is NOT included by `option httplog` (verified against
+    #        haproxy 3.0.11). Without it the documented support workflow
+    #        -- X-Request-Reference -> access log -> coraza audit.log -> rule_id
+    #        -- cannot be completed. id= is what makes that join possible.
+    #   host=/ua= identify the vhost and client; %ST/%B/%tsc give status,
+    #        bytes and the termination state that distinguishes a rate-limit
+    #        deny (PR--) from a tarpit (PT--) from a normal close.
+    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major]"
+
+    # --- URI normalisation (MUST be the first path-touching block here) ---
+    # Every path-based control in this frontend (the ACME health-check bypass,
+    # wp-login, xmlrpc, the wp-json/batch virtual patch, the wp-admin gate, the
+    # blocked-IP and suspension set-path rules, and everything Coraza inspects)
+    # matched the RAW request-target while the backend NORMALISED and DECODED
+    # it before resolving a file. Every gap between those two behaviours is a
+    # bypass, and each one had to be patched individually. Five were found in
+    # the wp-admin gate alone, all the same class:
+    #
+    #   //wp-admin/plugins.php            raw path starts "//" -> the safe-path
+    #                                     guard failed -> request fell through
+    #                                     ungated, backend served it
+    #   /wp-admin/css/../plugins.php      matched the css/js/images asset
+    #                                     bypass; backend resolved ".." and
+    #                                     booted plugins.php
+    #   /wp-admin/js/%2e%2e/plugins.php   same, with the ".." percent-encoded
+    #   /wp%2Dadmin/plugins.php           "wp-admin" spelled with %2D never
+    #                                     matched any wp-admin ACL at all
+    #   /wp-admin%2Fplugins.php           encoded separator; see the dedicated
+    #                                     rule in the wp-admin gate below
+    #
+    # Rather than keep bolting a counter-pattern onto each rule, normalise the
+    # URI once, here, so every rule below matches the SAME string the backend
+    # will resolve. HAProxy rewrites the request-target in place, so the
+    # backend receives the normalised form too.
+    #
+    # ORDER IS LOAD-BEARING and was determined empirically against real
+    # haproxy 3.0.11, not from the docs. The decoders must run BEFORE the path
+    # walkers: with the reverse order, /wp-admin/js/%2e%2e/plugins.php ends up
+    # as /wp-admin/js/../plugins.php -- decoded, but the ".." left unresolved,
+    # because path-strip-dotdot had already run by the time the "%2e%2e"
+    # became "..". Verified both directions side by side.
+    #
+    #   percent-to-uppercase       %2f -> %2F. Canonicalises the spelling of
+    #                              whatever stays encoded, so downstream rules
+    #                              need one case of each escape, not two.
+    #   percent-decode-unreserved  Decodes ONLY RFC 3986 unreserved chars
+    #                              (A-Za-z0-9-._~). This is what turns %2e%2e
+    #                              into .. and wp%2Dadmin into wp-admin.
+    #                              Reserved escapes are deliberately left
+    #                              alone -- %2F in particular, which is why
+    #                              the wp-admin gate needs its own encoded-
+    #                              slash rule (see below).
+    #   path-merge-slashes         //x -> /x. Also removes the entire class of
+    #                              "leading // defeats an anchored regex".
+    #   path-strip-dot             /a/./b -> /a/b.
+    #   path-strip-dotdot full     /a/b/../c -> /a/c. "full" additionally
+    #                              resolves ".." segments that would climb
+    #                              above the root (/../../wp-admin/x.php ->
+    #                              /wp-admin/x.php); without "full" HAProxy
+    #                              leaves those in place and the vector
+    #                              survives -- measured, both forms tested.
+    #
+    # DELIBERATELY NOT ENABLED: query-sort-by-name. Reordering query-string
+    # parameters would silently break anything that signs or caches on the
+    # exact query string (signed asset URLs, HMAC'd callbacks, CDN cache
+    # keys). This is NOT because the enabled normalizers already leave the
+    # query alone -- see BLAST RADIUS just below, they don't -- it is a
+    # deliberate line drawn between "case-fold / decode", which RFC 3986
+    # defines as no-ops, and "reorder", which is not a no-op for a caller
+    # treating the query as an opaque signed string.
+    #
+    # BLAST RADIUS: this block applies to EVERY request for EVERY site on
+    # EVERY tier, so the decoding was kept minimal on purpose -- and it is
+    # NOT scoped to the path. percent-to-uppercase and percent-decode-
+    # unreserved normalise the WHOLE request-target as HAProxy parses it --
+    # query string included, not just the path component the ACLs below
+    # match on -- and the BACKEND receives the rewritten query on the wire,
+    # not just an internal haproxy view of it. Measured against real HAProxy
+    # 3.0.11:
+    #   /a?sig=%2babc%2fdef  -> /a?sig=%2Babc%2Fdef   (percent-hex upper-cased)
+    #   /a?b=%41%42%43       -> /a?b=ABC               (unreserved chars decoded)
+    #   /a?tok=%7e%2d%5f%2e  -> /a?tok=~-_.            (unreserved chars decoded)
+    # Only parameter ORDER is preserved -- that guarantee is exactly why
+    # query-sort-by-name above is the one normalizer in this family left
+    # disabled. Per RFC 3986 both enabled rewrites are defined as the same
+    # URI (case in a percent-escape, and an unreserved character vs. its
+    # escape, carry no distinct meaning), but "the same URI" is not "the
+    # same bytes": an application that HMACs or otherwise signs the RAW
+    # query string, rather than parsing it first, could see a mutated value
+    # and fail to verify an otherwise-legitimate request. Checked against
+    # this fleet: no .NET backends (.NET's UrlEncode emits lowercase
+    # percent-hex, which percent-to-uppercase would rewrite) and no
+    # URL-in-path proxies, so there is no known victim today -- but do not
+    # assume "path only" from this block; that was the actual bug in an
+    # earlier draft of this comment.
+    #
+    # normalize-uri is EXPERIMENTAL in 3.0 and requires
+    # `expose-experimental-directives` in the global section
+    # (hap_header.tpl). Without it HAProxy does not start. Remove one and you
+    # must remove the other.
+    http-request normalize-uri percent-to-uppercase
+    http-request normalize-uri percent-decode-unreserved
+    http-request normalize-uri path-merge-slashes
+    http-request normalize-uri path-strip-dot
+    http-request normalize-uri path-strip-dotdot full
+
+    # --- Trusted-proxy gate (MUST precede real-IP resolution below) ---
+    # CF-Connecting-IP / X-Real-IP / X-Forwarded-For are client-supplied. Any
+    # peer that is not a known reverse proxy gets them stripped, so the
+    # set-var chain below falls through to `src` -- the real TCP peer.
+    #
+    # Without this, a direct client dictates txn.real_ip, and every control
+    # keyed on that variable trusts the attacker's own claim: rate limiting
+    # (track-sc0), the trusted-IP whitelist, the wp-login brute-force table and
+    # cookie challenge, the wp-json/batch/v1 virtual patch, IP blocking, and
+    # Coraza's src-ip. Spoofing a whitelisted IP bypassed all of them.
+    #
+    # ORDER MATTERS: these must come before the set-var lines. HAProxy applies
+    # http-request rules in file order, so a strip placed afterwards would
+    # validate cleanly and accomplish nothing.
+    #
+    # Cloudflare-fronted domains keep working: CF's edge matches
+    # from_trusted_proxy, so its CF-Connecting-IP survives.
+    acl from_trusted_proxy src -f /etc/haproxy/cloudflare_ips.list -f /etc/haproxy/trusted_proxies.list
+    http-request del-header CF-Connecting-IP if !from_trusted_proxy
+    http-request del-header X-Real-IP        if !from_trusted_proxy
+    http-request del-header X-Forwarded-For  if !from_trusted_proxy
+
     # Detect real client IP from proxy headers if they exist
     # Priority: CF-Connecting-IP (Cloudflare) > X-Real-IP > X-Forwarded-For > src
     acl has_cf_connecting_ip req.hdr(CF-Connecting-IP) -m found
@@ -96,6 +258,42 @@ frontend web
     acl has_login_cookie req.cook(whplc) -m found
     http-request deny deny_status 403 if METH_POST wp_login_path !has_login_cookie !is_local !is_trusted_ip !is_whitelisted
 
+    # --- WordPress xmlrpc.php flood protection ---
+    # xmlrpc.php floods are a common, sustained abuse pattern that the generic
+    # limits above don't catch: those trigger at 3000/5000 req/10s (300-500
+    # req/s, sized for media-heavy pageloads), while observed xmlrpc floods run
+    # at just a few req/s for hours -- comfortably under that ceiling but still
+    # enough to pin PHP-FPM workers and show up as 503s for the rest of the
+    # site. Same shape of problem as wp-login credential stuffing, so it gets
+    # the same fix: track POSTs to xmlrpc.php per real client IP in a DEDICATED
+    # 60s table (sc2 / backend xmlrpc_bruteforce, defined in
+    # hap_security_tables.tpl -- kept separate from wp_bruteforce so the two
+    # endpoints' traffic can't inflate each other's counter, see that file for
+    # the reasoning) and tarpit once an IP exceeds the threshold.
+    #
+    # Threshold is 60/min (double wp-login's 30/min), not because xmlrpc abuse
+    # is less severe but because legitimate traffic here is machine-to-machine
+    # rather than a human filling out a form: Jetpack sync, the WordPress
+    # mobile app, and remote-publishing clients (e.g. an offline blog editor)
+    # can legitimately burst several xmlrpc calls in quick succession. 60/min
+    # (1 req/s average over the window) comfortably absorbs that burst while
+    # still tripping well before an hours-long few-req/s flood does real
+    # damage -- at 2 req/s sustained the 60s counter clears the threshold in
+    # under a minute.
+    #
+    # Tarpit (not deny), matching the wp-login rule: this is per-IP tracking
+    # of a bounded set of offenders, not the wp-login cookie challenge's
+    # distributed hundreds-of-thousands-of-IPs scenario where holding
+    # connections would exhaust HAProxy itself, so tying up the flooding IP's
+    # connections is the cheaper and more effective response. path_end (not
+    # path_beg) covers subdirectory WP installs, same reasoning as wp-login.
+    # Honors the same whitelist (RFC1918 / trusted_ips.list / trusted_ips.map)
+    # so health checks and trusted infrastructure are unaffected, and legit
+    # clients under the threshold are never blocked outright.
+    acl xmlrpc_path path_end /xmlrpc.php
+    http-request track-sc2 var(txn.real_ip) table xmlrpc_bruteforce if METH_POST xmlrpc_path
+    http-request tarpit deny_status 429 if METH_POST xmlrpc_path { sc_http_req_rate(2) gt 60 } !is_local !is_trusted_ip !is_whitelisted
+
     # WordPress REST batch endpoint lockdown ("wp2shell": CVE-2026-63030 +
     # CVE-2026-60137). Chaining a core SQL injection with REST batch-route
     # confusion gives unauthenticated RCE on WP 6.9.0-6.9.4 and 7.0.0-7.0.1
@@ -125,9 +323,268 @@ frontend web
     http-request deny deny_status 403 if wp_batch_route !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
     http-request deny deny_status 403 if wp_batch_route_enc !has_wp_logged_in !is_local !is_trusted_ip !is_whitelisted
 
+    # --- WordPress admin edge gate ---
+    # Measured on whp02, 2026-08-14: distributed unauthenticated GETs booting
+    # WordPress just to bounce back a login redirect --
+    #   2801 GET /wp-admin/profile.php   2781 GET /wp-admin/edit.php   2761 GET /wp-admin/plugins.php
+    # -- spread across many source IPs at roughly 4 req/min per IP, each hit
+    # burning a PHP-FPM/lsphp worker. Two sites absorbed 1,243 resulting 503s
+    # in nine hours as their pools saturated.
+    #
+    # IDENTITY, NOT RATE. This is the one rule in this file that tracks
+    # nothing and has no stick-table counter or threshold. Every rate-based
+    # control above (the generic limits, wp_bruteforce, xmlrpc_bruteforce) is
+    # per-IP, and per-IP rate is exactly what this attack is engineered to
+    # stay under: ~4 req/min from any single IP is indistinguishable from a
+    # slow human, and the source set is large enough that no threshold can be
+    # lowered to catch it without also catching real visitors. There is also
+    # no free stick-table slot left to try anyway -- sc0/sc1/sc2 are already
+    # used above and HAProxy's default tune.stick-counters is 3, so a fourth
+    # tracked counter is not an option here. DO NOT "simplify" this into a
+    # rate/threshold rule later: the whole point is that a threshold cannot
+    # see this traffic. Instead we gate on identity -- a real logged-in
+    # WordPress user always carries a wordpress_logged_in_* cookie (the same
+    # ACL the wp2shell block above already declares), and an unauthenticated
+    # request to a wp-admin page has no legitimate reason to boot PHP at all.
+    #
+    # 302, not 403. WordPress itself redirects an unauthenticated /wp-admin/
+    # request to wp-login.php, so replicating that at the edge means an admin
+    # whose session merely expired lands on the normal login screen instead
+    # of an error page -- we are not trading a bot problem for a support
+    # ticket. Bots get a cheap redirect they ignore.
+    #
+    # path_reg, not path_beg. A subdirectory install at /blog/wp-admin/ slips
+    # past a prefix match; path_reg with an optional leading "/" catches both
+    # root and subdirectory installs, same reasoning as the wp-login and
+    # xmlrpc path_end rules above.
+    #
+    # regsub rewrites the redirect target itself, so /blog/wp-admin/x.php
+    # redirects to /blog/wp-login.php rather than 404ing at the site root.
+    #
+    # DO NOT write regsub's regex argument with a capturing group / literal
+    # parentheses, e.g. regsub((^|/)wp-admin/.*,\1wp-login.php) -- neither
+    # inlined into the redirect's `location` nor in a standalone set-var.
+    # HAProxy 3.0.11's converter-argument parser counts parens to find the
+    # end of the regsub(...) call itself, so the *inner* "(^|/)" grouping
+    # parens are misread as closing the outer call early -- it does not
+    # matter whether the argument is quoted ("...": still fails) or the
+    # parens are backslash-escaped (\(...\): still fails). Every such form
+    # was verified against real HAProxy 3.0.11-1+deb13u3 and all produce the
+    # same ALERT: "invalid arg 2 in converter 'regsub' : missing arguments
+    # (got 1/2)". This is a converter-argument-parsing limitation, not a
+    # log-format/`%[...]` issue -- the identical failure reproduces in a
+    # plain set-var (outside any log-format string), which rules out the
+    # `location` value's log-format context as the cause.
+    #
+    # The fix sidesteps groups/backreferences entirely: HTTP paths always
+    # start with "/", so the leading "(^|/)" alternation is redundant --
+    # matching the literal substring "/wp-admin/" (both slashes, no group)
+    # is sufficient to anchor to a real path segment (a false match like
+    # "/somewp-admin/" doesn't contain "/wp-admin/" as a substring, since
+    # there's no "/" directly before "wp-admin"). No backreference is
+    # needed either: regsub only replaces the matched substring, so
+    # replacing "/wp-admin/.*" with a literal "/wp-login.php" leaves
+    # whatever precedes it (the subdirectory-install prefix, if any)
+    # untouched. Computed in its own set-var so it is a plain sample
+    # expression, not something baked into the redirect's log-format
+    # string. Behaviorally verified live against real HAProxy 3.0.11:
+    # /wp-admin/edit.php -> /wp-login.php and
+    # /blog/wp-admin/plugins.php -> /blog/wp-login.php, both with
+    # redirect_to preserved. See
+    # .superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-3b-report.md.
+    #
+    # redirect_to carries the path only (%[path,url_enc]), not the query
+    # string -- deliberate, see design spec. An admin bounced off
+    # post.php?post=123&action=edit lands back on a blank post.php rather
+    # than that exact post. Capturing the full URI needs capture.req.uri
+    # (extra config, and the captured value is length-capped) for a benefit
+    # that only matters on session expiry, so it was not worth it.
+    #
+    # THE ALLOWLIST IS MEASURED, NOT GUESSED -- taken from actual fleet
+    # traffic returning 200 on /wp-admin/*. admin-ajax.php and admin-post.php
+    # are the standard front-end AJAX/form-handler endpoints real themes and
+    # plugins call while logged out. Critically, wp-login.php loads its OWN
+    # css/js FROM /wp-admin/ (load-styles.php, load-scripts.php, and the
+    # static wp_admin_asset dirs below) -- miss those and every login page on
+    # the fleet renders unstyled with a broken password-strength meter, while
+    # wp-login.php itself still returns 200, making it a silent regression
+    # that "looks like" the gate is working.
+    #
+    # Each allowlist entry is anchored to /wp-admin/<file>, not a bare
+    # filename suffix. A bare `path_end /admin-ajax.php` also matches
+    # /wp-admin/evil/admin-ajax.php -- which ALSO matches wp_admin_path
+    # (path_reg only requires /wp-admin/ to appear somewhere), so an
+    # attacker-inserted path segment would sail through this allowlist
+    # ungated and boot full WordPress, exactly the resource exhaustion this
+    # gate exists to stop. Anchoring still covers subdirectory installs via
+    # suffix matching (/blog/wp-admin/admin-ajax.php ends with
+    # /wp-admin/admin-ajax.php) while rejecting an inserted directory.
+    #
+    # install.php is DELIBERATELY NOT allowlisted. It is legitimately
+    # reachable without a cookie during a fresh install, but it is also a
+    # standing scanner target and a real takeover vector on a site that was
+    # half-installed and then abandoned. Anyone genuinely installing uses the
+    # per-site exempt-list opt-out below instead.
+    #
+    # Honors the same whitelist as every other rule in this frontend
+    # (RFC1918 / trusted_ips.list / trusted_ips.map), and a per-site opt-out
+    # via /etc/haproxy/wpadmin_gate_exempt.list (operator-managed, seeded
+    # empty by start-up.sh) for sites where a plugin legitimately serves
+    # unauthenticated visitors from a /wp-admin/ URL outside this allowlist.
+    # EVERY ACL BELOW MATCHES THE NORMALISED PATH. The normalize-uri chain at
+    # the top of this frontend has already merged duplicate slashes, resolved
+    # "." / ".." segments (including percent-encoded ones) and decoded
+    # unreserved escapes by the time these run, so these patterns only have to
+    # describe the ONE canonical spelling the backend will resolve -- they do
+    # not have to anticipate every encoding of it. That is the whole point of
+    # the normalisation block; do not "harden" these regexes by re-adding
+    # encoding variants, fix the normalisation instead.
+    #
+    # wp_admin_safe_path guards against an OPEN REDIRECT this gate would
+    # otherwise introduce. The redirect target below is built by rewriting
+    # `path` with regsub -- regsub only replaces the matched substring, so
+    # everything BEFORE the matched "/wp-admin/" survives untouched in the
+    # output. `path` is not guaranteed to be a clean site-relative string;
+    # three concrete requests turn that survival into an off-site
+    # `Location:` header:
+    #   //evil.example.com/wp-admin/x.php        -> //evil.example.com/wp-login.php
+    #     (protocol-relative -- browsers resolve "//host/path" to
+    #     "https://host/path", so this redirects off-site with no scheme
+    #     needed). NOW NEUTRALISED UPSTREAM: path-merge-slashes rewrites this
+    #     to /evil.example.com/wp-admin/x.php before any ACL sees it, so the
+    #     Location becomes the same-origin /evil.example.com/wp-login.php.
+    #     Verified live.
+    #   /\evil.example.com/wp-admin/x.php        -> /\evil.example.com/wp-login.php
+    #     (browsers normalise a leading "/\" the same as "//"). STILL LIVE
+    #     after normalisation -- a backslash is not a slash, so no normalizer
+    #     touches it. This ACL is the only thing that stops it.
+    #   https://evil.example.com/wp-admin/x.php  -> https://evil.example.com/wp-login.php
+    #     (RFC 7230 absolute-form request targets can make HAProxy's `path`
+    #     fetch return a full URI, not just the path component). HAProxy's own
+    #     H1 parser answers 400 on this frontend; this ACL is the backstop.
+    # So wp_admin_safe_path is NOT redundant with the normalisation and must
+    # not be deleted as such -- one of its three vectors survives normalisation
+    # untouched.
+    #
+    # It is used TWO ways, and the pair matters:
+    #   - as a POSITIVE condition on the redirect, so a pathological path can
+    #     never produce a `Location:` header at all; and
+    #   - as an explicit deny, so such a path is not merely un-redirected.
+    # The deny is what closes the failure mode the positive-condition form
+    # introduced on its own: "not redirected" used to mean "falls through to
+    # the backend UNGATED", i.e. the exact PHP-booting request this gate
+    # exists to stop, reachable by prefixing "//" (that specific spelling is
+    # now normalised away, but "/\" is not). Post-normalisation the only
+    # paths that reach the deny are "/\..." ones, which cannot resolve to a
+    # real file on any tier, so nothing legitimate is denied.
+    # -i (case-insensitive) and the matching ",i" flag on the regsub below are
+    # a PAIR -- adding one without the other produces an infinite redirect
+    # loop, because a case-sensitive regsub finds no "/wp-admin/" in
+    # "/WP-ADMIN/plugins.php", returns `path` UNCHANGED, and the Location then
+    # points at the request's own URL. Verified live that the pair is correct:
+    # /WP-ADMIN/plugins.php -> /wp-login.php and /blog/WP-Admin/plugins.php ->
+    # /blog/wp-login.php.
+    #
+    # On this fleet's Linux backends /WP-ADMIN/plugins.php 404s without booting
+    # PHP, so this is hardening rather than a live-bypass fix; it matters if a
+    # docroot ever sits on a case-insensitive mount, where that same request
+    # WOULD boot PHP. The cost is that a site with a real directory literally
+    # named e.g. /docs/WP-Admin/ now gets gated -- the same false positive the
+    # lowercase pattern already has, which is what the per-site exempt list
+    # exists to resolve.
+    acl wp_admin_path      path_reg -i (^|/)wp-admin/
+    # Four literal backslashes here is NOT a typo. HAProxy's config-line word
+    # parser treats backslash as its OWN escape character before the value
+    # ever reaches the regex engine: "\\" (two backslashes) in the config
+    # collapses to one literal backslash by the time PCRE compiles it, which
+    # leaves an unterminated character class ("[^/\]") and fails with
+    # "missing terminating ] for character class" -- verified against real
+    # HAProxy 3.0.11. Four backslashes ("\\\\") collapse to two ("\\"),
+    # which PCRE then reads as a single escaped-backslash class member --
+    # the intended "reject a literal backslash" semantics.
+    acl wp_admin_safe_path path_reg ^/[^/\\\\]
+    acl wp_admin_allowed   path_end /wp-admin/admin-ajax.php /wp-admin/admin-post.php /wp-admin/load-styles.php /wp-admin/load-scripts.php
+    # The (?!.*\.php) lookahead is DEFENCE IN DEPTH, not the primary fix. This
+    # ACL grants an un-gated bypass to everything under wp-admin/css|js|images,
+    # and it used to anchor its prefix but not its suffix, so
+    # /wp-admin/css/../plugins.php took the bypass and the backend then
+    # resolved ".." and booted plugins.php. path-strip-dotdot now rewrites that
+    # to /wp-admin/plugins.php before this ACL runs, which is the real fix; the
+    # lookahead additionally makes the bypass structurally incapable of
+    # covering a PHP entrypoint even if a future encoding trick survives
+    # normalisation. It excludes ".php" ONLY -- no static asset contains that
+    # substring, so it cannot cause the silent "login page renders unstyled"
+    # regression that an extension allowlist would risk. Requires PCRE2, which
+    # both the Debian (deployed) and Alpine haproxy builds have (+PCRE2).
+    #
+    # THE "--" IS LOAD-BEARING, DO NOT DELETE IT. HAProxy warns on any pattern
+    # whose first character is "(", because it cannot tell an intended regex
+    # from a fetch-argument list someone typo'd a space into:
+    #   parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
+    #   '(^|/)wp-admin/...' is likely a mistake and probably not what you want.
+    #   Maybe you need to remove the extraneous space before '('.
+    # "--" is HAProxy's documented end-of-flags marker and is the fix it names
+    # itself ("please insert '--' between the match and the pattern"). It changes
+    # no matching semantics -- verified live on whp02: /wp-admin/css/login.min.css
+    # still passes and /wp-admin/css/x.php is still gated, before and after.
+    # Left unsilenced this fires on EVERY config load and reload on every host,
+    # where it trains operators to skim past warnings and can bury a real one.
+    # wp_admin_path escapes the warning only because its "-i" flag happens to
+    # consume the flag slot first; it is not otherwise special.
+    acl wp_admin_asset     path_reg -- (^|/)wp-admin/(css|js|images)/(?!.*\.php).*$
+    acl wp_gate_exempt     hdr(host),lower -f /etc/haproxy/wpadmin_gate_exempt.list
+    # ENCODED SEPARATOR. percent-decode-unreserved deliberately does NOT decode
+    # %2F -- "/" is a reserved character, and decoding it in the normalizer
+    # would change the path's structure (it would invent new segments), which
+    # is precisely why HAProxy refuses to. But OpenLiteSpeed DOES decode it and
+    # then serves the file: /wp-admin%2Fplugins.php was measured returning 302
+    # from a real WordPress site on the OLS tier, i.e. full PHP boot, while
+    # matching none of the ACLs above. Apache returns 404 for the same request
+    # (AllowEncodedSlashes Off), so this is an OLS-tier defect -- and OLS is the
+    # tier currently saturating.
+    #
+    # DENY, not "treat it as a wp-admin path and redirect". Two reasons:
+    #   1. The redirect target is computed by regsub(/wp-admin/.*) which finds
+    #      no "/wp-admin/" in "/wp-admin%2Fplugins.php", so `path` would come
+    #      back UNCHANGED and the Location would point at the request's own
+    #      URL -- an infinite redirect loop, not a gate.
+    #   2. Nothing legitimate emits it. A path segment cannot contain a literal
+    #      "/", so %2F inside a path is always either a probe or a proxy-
+    #      confusion attempt, and the Apache tier has been 404ing it all along,
+    #      so no site on the fleet can depend on it.
+    #
+    # SCOPED to paths that mention wp-admin, not all paths. A blanket "deny any
+    # %2F in any path" would also hit REST/API-style routes on non-WordPress
+    # customer apps that legitimately pass an encoded slash inside a path
+    # parameter. Scoping keeps the blast radius inside the attack surface this
+    # gate owns.
+    #
+    # Matching is on the SUBSTRING, not an anchored pattern, on purpose:
+    # /blog%2Fwp-admin/plugins.php hides the separator BEFORE "wp-admin", where
+    # an anchored (^|/)wp-admin/ never matches, and OLS still resolves it to
+    # /blog/wp-admin/plugins.php. Substring matching catches the separator
+    # wherever it is. percent-to-uppercase has already folded %2f into %2F;
+    # the -i is belt and braces so this rule stands on its own if the
+    # normalizer is ever reordered.
+    #
+    # %5C (encoded backslash) is denied on the same terms. On this fleet's
+    # Linux backends a backslash is an ordinary filename character, so
+    # /wp-admin%5Cplugins.php 404s rather than booting PHP -- measured, it is
+    # not a live bypass today. It is included because it is the same
+    # encoded-separator trick against a backend that happens to treat "\" as
+    # one, it costs nothing, and no legitimate path contains it.
+    acl wp_admin_word         path -i -m sub wp-admin
+    acl path_has_encoded_sep  path -i -m sub %2f %5c
+    http-request deny deny_status 403 if wp_admin_word path_has_encoded_sep !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+    http-request deny deny_status 403 if wp_admin_path !wp_admin_safe_path !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+    http-request set-var(txn.wp_login_url) path,regsub(/wp-admin/.*,/wp-login.php,i) if wp_admin_path
+    http-request redirect code 302 location %[var(txn.wp_login_url)]?redirect_to=%[path,url_enc] if wp_admin_path wp_admin_safe_path !wp_admin_allowed !wp_admin_asset !has_wp_logged_in !wp_gate_exempt !is_local !is_trusted_ip !is_whitelisted
+
     # IP blocking using map file (manual blocks only)
     # Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
-    # Runtime updates: echo "add map #0 IP_ADDRESS 1" | socat stdio /var/run/haproxy.sock
+    # Runtime updates (worker command, map referenced by PATH -- "#<id>" ids
+    # move on every config regeneration and "#0" silently adds nothing):
+    #   echo "@1 add map /etc/haproxy/blocked_ips.map IP_ADDRESS 1" | socat stdio /tmp/haproxy-cli
     # Checks the real client IP (from headers if present, otherwise src)
     # map_ip() converter supports both single IPs and CIDR ranges (e.g., 192.168.1.0/24)
     acl is_blocked_ip var(txn.real_ip),map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0

templates/hap_security_tables.tpl

@@ -13,4 +13,19 @@ frontend stats
 # sc0 connection/rate table so the login-attempt threshold is independent of
 # the (much higher) flood thresholds.
 backend wp_bruteforce
+    stick-table type ip size 100k expire 30m store http_req_rate(60s)
+
+# Dedicated stick-table for POST /xmlrpc.php flood tracking.
+# Tracked via track-sc2 from the `web` frontend (hap_listener.tpl); counts
+# only xmlrpc POSTs per real client IP over a 60s window. This is a SEPARATE
+# table/counter from wp_bruteforce (sc1) rather than a shared one: both are
+# machine-to-machine WordPress endpoints an attacker could hit from the same
+# IP, and sharing a counter would let one endpoint's traffic inflate the
+# other's rate -- an IP credential-stuffing wp-login while also flooding
+# xmlrpc would trip the wp-login threshold early on xmlrpc volume alone (or
+# vice versa). track-sc1 (wp-login) and track-sc2 (xmlrpc) are each gated on
+# mutually exclusive path ACLs, so at most one of them ever fires per
+# request -- HAProxy's "one track-sc<N> per counter per request" limit is
+# never in play here since they're different counters anyway.
+backend xmlrpc_bruteforce
     stick-table type ip size 100k expire 30m store http_req_rate(60s)

trusted_proxies.list

@@ -0,0 +1,13 @@
+# Additional trusted reverse proxies — peers permitted to set CF-Connecting-IP,
+# X-Real-IP and X-Forwarded-For. Anything NOT matched here or in
+# cloudflare_ips.list has those headers stripped before real-IP resolution.
+#
+# Referenced by templates/hap_listener.tpl.
+#
+# Leave EMPTY unless a real proxy sits in front of HAProxy on this host. Adding
+# a range here lets that peer assert any client identity, which bypasses rate
+# limits, IP blocks and the WAF for it.
+#
+# Do NOT commit real IPs — this repo is mirrored publicly. Add entries directly
+# on the server; the file lives in the /etc/haproxy named volume and persists
+# across container recreates.

wpadmin_gate_exempt.list

@@ -0,0 +1,16 @@
+# Per-site opt-out from the WordPress admin edge gate.
+#
+# Hostnames listed here are EXEMPT: unauthenticated /wp-admin/* requests for
+# these sites pass through to PHP instead of being redirected to wp-login.php.
+# One hostname per line, lowercase. Matched against the Host header.
+#
+# Referenced by templates/hap_listener.tpl:
+#   acl wp_gate_exempt hdr(host),lower -f /etc/haproxy/wpadmin_gate_exempt.list
+#
+# Add a site here when a plugin legitimately serves unauthenticated visitors
+# from a /wp-admin/ URL that is not in the rule's allowlist. Symptom: "my
+# plugin's admin page redirects to login".
+#
+# Do NOT commit real customer domains — this repo is mirrored publicly. Add
+# entries directly on the server; the file lives in the /etc/haproxy named
+# volume and persists across container recreates.