req.body_size is "buffered + htx->extra". In HAProxy 3.0 only the H1 and
H2 muxes set htx->extra from Content-Length; the H3 mux never does, so
over HTTP/3 req.body_size == req.body_len and any body cut at one buffer
(~15 KB) was stamped `complete`. Coraza then failed to parse the
truncated JSON (200002) and WHP's 991900020 returned a 403.
Measured on whp01, Oct 6-19 2026: 73,795 H3 `complete` stamps and 0
`partial`; 7 legitimate requests 403'd (6 Divi sync-to-server saves of
265-417 KB, 1 Groundhogg preview of 18 KB). Reproduced in a lab on
haproxy 3.0.11 with QUIC. H1/H2/H3 all reject a Content-Length that
doesn't match the DATA received, so the header is a protocol-independent
body length. No change on H1/H2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coraza-spoa v0.7.1 already returns the interrupting rule id
(txn.coraza.ruleid) and, when the request message sends
exportRuleIDs=bool(true), the comma list of every matched CRS attack
rule (txn.coraza.rule_ids) plus the inbound anomaly score. Send the arg
and append all three to the access log so a block's cause (e.g.
920440,949110 score 5) is visible in haproxy.log without opening the
Coraza audit log. Fields are appended last; existing ones unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Send `id=unique-id` in the coraza-req SPOE message so coraza-spoa calls
waf.NewTransactionWithID() with HAProxy's unique-id (the %[uuid()] shown
on 403-waf.html, sent as X-Request-Reference and logged as `id=` in
haproxy.log). A customer-quoted reference now greps directly to
transaction.id in /var/log/coraza/audit.log and match.unique_id in
spoa.log. Previously coraza-spoa invented a random 16-letter A-Z ID
that joined to nothing on the HAProxy side.
Measured against coraza-spoa v0.7.1 source (no version bump needed):
- internal/agent.go HandleSPOE: only `app` is positional (must be the
first KV, else Panic); the template comment claiming all args are
parsed positionally was wrong and is corrected.
- internal/application.go HandleRequest: every other arg is matched by
name; `id` is recognised and passed to NewTransactionWithID. Empty/
NULL id falls back to the random ID, so this cannot break inspection.
- coraza v3.7.0 places no format/length constraint on the ID.
response_check is false so there is no coraza-res message; the template
and a test now require any future coraza-res to send the same id
(HandleResponse looks the request transaction up by it).
Tests: new CorazaTransactionIdIsTheRequestReference class in
test-waf-body-inspection.py (fails without the change). Full in-image
suite + validate-rendered-config (haproxy 3.0.11 -c) pass; a negative
control with a bogus fetch name is rejected by haproxy -c.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI runner measured median 14.97s on 2 probes (load-dependent), failing the
build gate on a timing bound rather than the property under test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skip guard caught only ImportError; a half-installed or version-skewed
certbot can fail on import with other exceptions (verified with a stub that
raises RuntimeError: previously an ERROR, now a skip). new_lineage() itself
stays outside the guard - it is the assertion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-review R1: haproxy-manager shares client-net with every tenant
container, so tenant PHP could reach haproxy-manager:8000 directly (the
127.0.0.1 publish only removed the internet path). On the ports in
HAPROXY_MGR_PEER_FILTER_PORTS (default 8000; NOT :8080, fed by HAProxy
over loopback) the overridden accept() closes any peer not in
HAPROXY_MGR_API_ALLOWED_PEERS (default 127.0.0.1,::1,gateway), before a
TConn, slot, poller registration or thread is used. "gateway" is read
from /proc/net/route: measured, host callers via the 127.0.0.1 publish
arrive from the bridge gateway (docker-proxy). Bad entries fail closed
and are logged; refusals are logged at most once a minute as a per-peer
summary.
Header deadline is now max(enqueue + 10s, pickup + 1s)
(HAPROXY_MGR_HEADER_PICKUP_GRACE): 40 reconnecting drippers no longer
starve legit requests (before: every probe >30s; after: median ~9s).
Tests: 21 (was 15): allowlist parsing/matching incl. v4-mapped, fail
closed, /proc/net/route parsing, real-gunicorn filtered vs unfiltered
port with rate-limited logging, listed peer served, 40 reconnecting
drippers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner ruling (same mode contract as 920440: no accidental enforcement from
bare words). 944110 and 944120 are CRS chains whose first link matches
`runtime|processbuilder` / `getproperty|forclosure|filewriter|...`
anywhere in the request; the range promotion 944100-944300 put
ctl:ruleEngine=On on those starters, so any request containing the word
ran with full anomaly enforcement, in detect_only too.
Promotion is now 944100 and 944130-944300. Live (haproxy-manager-base
:wafbody-r3 + coraza-spoa :wafbody-r3), both modes: `?q=runtime&<SQLi>`
and `?q=getproperty&<SQLi>` 403 -> 200 on detect_only; Log4Shell in the
query (plain and ${lower:j} obfuscated), User-Agent and a JSON body still
403.
test-waf-body-inspection.py: 21 tests (+2: no CRS chain starter
920440/944110/944120 is covered by a promotion; the Log4Shell ids keep
theirs). Red on round 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on
920440's chain STARTER (`REQUEST_BASENAME "@rx \.([^.]+)$"`), which matches
every URL with a file extension, so every *.php/*.css request was evaluated
with the engine On: full CRS anomaly enforcement (949110) and 200002, in
detect_only mode too.
Replaced by `SecRule TX:EXTENSION "@within %{tx.restricted_extensions}"
... ctl:ruleEngine=On` (id 992920440, phase 1, after 920440). CRS 4.25.0
sets tx.extension in 920440's first link (REQUEST-920...conf:667, the
coreruleset version coraza-spoa v0.7.1 pins). Live, both modes:
.bak/.sql/.save/.old/.swp probes still 403; benign .php/.css 200; a
942100 SQLi on /x.php and the onclick / "select ... union" admin saves are
no longer 403 in detect_only.
test-waf-body-inspection.py: 19 tests (2 new, repo-checkout only).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- H1: request_ssl_bundle uses live/<primary> exactly (certbot was given
--cert-name <primary>) instead of find_certbot_live_dir(), which returns the
newest of <primary>/<primary>-NNNN. After a "not yet due" no-op that
published a stale -0001 and retired the real <primary> lineage. keep_lineage
is now the primary; the bundle's own lineage is never a retirement candidate.
- H2: retire_lineage_tree() moves live/<x>, archive/<x> and renewal/<x>.conf
together to /etc/letsencrypt/retired/<stamp>/{live,archive,renewal}/ (same
depth, relative symlinks still resolve, restore is three mv's). Leaving
archive/<x> behind made certbot's new_lineage() refuse the name forever
(CertStorageError) and leave an empty renewal conf. Test re-issues the name
through certbot's own storage code after retirement.
- M2: cert_retire_expired_pems moves already-published EXPIRED pems out of
/etc/haproxy/certs to cert-retired/ (never the last pem, never the default
self-signed cert) and the scripts reload when anything was retired.
- GET /api/capabilities + `capabilities` in the bundle response, and VERSION is
copied into the image, so WHP can refuse whole-site re-bundling against an
older edge.
Tests: test-cert-write-safety.py 47 -> 52, test-cert-scripts.py 44 -> 50; each
fix reverted in a scratch copy turns its suite red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1 on 0ce144d (FIX-FIRST):
F1: stock gthread never reaps a connection that never sends a byte; it
holds a worker_connections slot forever and at 1000 per worker the run
loop stops polling, wedging the worker permanently (survives the
attacker disconnecting). accept/on_client_socket_readable/
murder_keepalived now track such conns in a FIFO with a deadline
(HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s) and close expired ones on every
loop pass, including the at-capacity branch.
F2: slowloris (1 header byte per <20s) never tripped the per-recv
timeout. A total request-line+headers deadline
(HAPROXY_MGR_HEADER_DEADLINE, 10s) is armed in handle() and cleared in
handle_request(); a 1s monitor loop shutdown(SHUT_RDWR)s expired
sockets. Armed per thread, not at accept, so legit requests queued
behind busy threads are never cut for waiting.
F4: monitor loop wrapped in try/except with logging; wedge threshold
1200s -> 3600s (multi-domain cert requests run 180s per domain).
F5: docstring now says the timeout bounds each whole sendall().
F6: warn when gunicorn != 23.0.0 (worker overrides gthread internals).
API key compared with hmac.compare_digest.
Tests: +3 production-shape (2x4) tests: 2050 idle conns reaped and
/health recovers while still held; 8 drip conns cut within the deadline
with /health responsive; slow-but-complete headers still served.
15 tests, in the Docker build gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hap_listener.tpl (Coraza block): `http-request deny deny_status 400 if
{ req.fhdr_cnt(content-type) gt 1 }`. With two Content-Type lines Coraza
evaluates every value (text/xml on either line selects its XML processor,
which accepts a form body without error) while Apache merges the lines
and PHP parses the body as a form. Measured with Coraza v3.7.0 + PHP 8.3:
`application/x-www-form-urlencoded` + `text/xml` put the payload in
$_POST uninspected. Live: two lines -> 400 over HTTP/1.1 and HTTP/2; one
comma-joined line (browser XHR double-set) is unaffected.
- coraza-spoa/config.yaml: SecRequestBodyJsonDepthLimit 64 (was 1024 from
the recommended file). Coraza's JSON flattening is quadratic in depth:
depth 300 (601 B) = 297 ms, depth 1000 (2 KB) = 3.1 s, past the 100 ms
fail-open timeout. Over the limit it is a 0.1 ms parse error, which WHP's
enforce-mode rule denies on a complete body (live: depth 65/300/1000 ->
403, depth 64 and a depth-9 global-styles body -> 200). Legitimate depth
measured on WordPress core, Gutenberg, WooCommerce and Elementor JSON:
max 8.
- test-waf-body-inspection.py: 17 tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brainjar.net served an expired certificate for two days (2026-10-07..09):
the old brain-jar.com bundle (CN brain-jar.com, SANs incl. all eight
brainjar names) stayed in /etc/haproxy/certs and won the SNI match over the
valid brainjar.net bundle, because superseded-cert cleanup matched on CN
only, and renew-certificates.sh re-published the expired lineage every 12h.
- Retirement now matches on ANY shared name (CN or SAN). The overlapping pem
is moved to cert-retired/ and its lineage's live/ dir and renewal conf to
live-retired/ and renewal-retired/ (after HAProxy loads the new bundle).
Nothing is deleted any more: `certbot delete` is gone from this path.
- Shared-PEM safety: a candidate is kept when it is still the only valid
certificate for a name in the domains table (retirement_blockers()). Expired
certs never block; names the caller lists in the new optional
`unreachable_names` body field do not block.
- /api/ssl/bundle passes --keep-until-expiring and reports `renewed`
(fingerprint before vs after), `certbot_not_due` and the certificate's
`not_after`, so WHP can stop recording certbot no-ops as renewals.
- Never publish an expired certificate: renew-certificates.sh and
sync-certificates.sh skip expired lineages (ERROR-logged, counted
separately), and cert_publish / publish_pem_bundle refuse expired leaves.
- VERSION 2026.10.3.
Tests: test-cert-write-safety.py 31 -> 47, test-cert-scripts.py 38 -> 44,
each new group verified to fail against a reverted fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0 (local
haproxy-manager + coraza-spoa + echo backend):
- send-spoe-group fired at header time, so req.body held only what arrived
with the headers. A 933180 payload whose body came in a second send(),
2s late, or after Expect: 100-continue passed uninspected; 8 parallel
14 KB payload-first POSTs: 6/40 blocked before, 40/40 after.
- Add `option http-buffer-request` (Coraza block only). Uploads are not
held: one buffer is inspected, the rest streams (50 MB upload: sha-ok,
same timing before/after).
- Stamp X-WHP-WAF-Body: complete|partial for Coraza (client copies
stripped first, stripped again before the backend). Content-Length:
exact compare; chunked/no-CL: body+headers < 12 KB => complete. WHP's
generated rules use it to deny body-parse errors (200002/200003) only on
complete bodies.
- Pin tune.bufsize 16384 (= default) when Coraza is on. Coraza costs
~3.3 ms/KB of body and the 100 ms SPOE timeout fails open: at 32 KB a
payload at byte 0 of a 30 KB base64 body bypassed. SPOP frames > 65535
are refused by the agent library.
- coraza-spoa/config.yaml: SecRequestBodyLimit 131072 / InMemory 131072 /
ProcessPartial (Reject 413s even under DetectionOnly).
- Access log gains wafb= (complete|partial) and wafe= (SPOE error =
request passed uninspected).
- overrides.conf: document (comment only) that the 920440 promotion turns
the engine On for every URL with a file extension.
- scripts/test-waf-body-inspection.py (16 tests), wired into the build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.
The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).
- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
* per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
re-applied on keep-alive reuse; bounds socket I/O, not handling time
* watchdog: if every pool thread has been in the same request for more
than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
all thread stacks and exit so the master respawns the worker
* client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
60s) as defence in depth for any future in-process client; /health now
reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
added to the Docker build gate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
add-header let a client-supplied X-CLIENT-IP survive next to ours, so a
backend reading the first occurrence got the attacker's value. set-header
replaces any inbound copy. Refs #10.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DELETE /api/domain ran, unconditionally, for any ssl_enabled row:
os.remove(ssl_cert_path)
certbot delete --cert-name <domain> --non-interactive
`domains.domain` is UNIQUE. `domains.ssl_cert_path` is not, and nothing
anywhere guarded against two rows naming the same file. Sharing is not an
edge case -- it is the normal shape of the table, because
request_ssl_bundle() deliberately creates it: one SAN certificate is issued
as `--cert-name <primary>`, published once to
/etc/haproxy/certs/<primary>.pem, and then EVERY included name's row is
pointed at that same path ("Mark every name in the bundle as ssl_enabled,
all pointing at the same combined .pem").
Measured read-only on the live SQLite in the haproxy-manager containers on
2026-08-23:
* whp01: 157 domain rows, 150 ssl_enabled, 71 distinct cert paths.
39 of those paths are referenced by MORE THAN ONE row, covering 118 of
the 150 SSL-enabled rows. Worst cases: brain-jar.com.pem and
arclightcourt.com.pem with 10 domains each, hackerpublicradio.org.pem
with 6, anhonesthost.com.pem with 5.
* whp02: 33 rows, 31 ssl_enabled, 15 distinct paths, 11 shared across 27
rows -- including threeworldsoneheart.org.pem, referenced by the apex,
its www, and mail.threeworldsoneheart.org. A production cleanup of that
mail.* row was stopped short precisely because removing it would have
unlinked the PEM the serving site is using.
So removing one domain unlinked a file up to nine other configured domains
were being served from. HAProxy binds the crt directory
(`bind ... ssl crt /etc/haproxy/certs`), so the loss is not noticed until
the next reload or restart, at which point the listener refuses to come up
or those names fall back to the wrong certificate.
`certbot delete` is the worse half. It destroys the lineage's archive, live
symlinks and renewal config; recovery is a fresh, rate-limited ACME order.
The old code passed `--cert-name <domain>`, which is also simply the wrong
lineage for a SAN member: 81 of whp01's 150 SSL-enabled rows have a cert
path whose basename is not their own domain, so for those the call was a
silent no-op -- while for a bundle PRIMARY it deleted the one lineage still
renewing the certificate every other name in the bundle is served with.
The fix refcounts, after the row is deleted so the query answers "who else
still needs this":
* lineage_name_for_cert_path() -- the lineage is the published bundle's
basename minus .pem, the same derivation _quarantine_superseded_certs()
already uses, not the domain being removed.
* domains_referencing_cert_path() / domains_referencing_lineage() -- the
remaining rows that name that file, and that lineage.
* remove_domain() unlinks only when the list is empty, `certbot delete`s
only when the list is empty, logs the retained names explicitly when it
skips, and reports them as certificate_retained_for /
lineage_retained_for in the API response.
ssl_enabled is deliberately not filtered on in the refcount: the two
mistakes are not symmetric. A stale PEM left in the crt directory costs a
few kilobytes; an unlinked live one is HTTPS down for every name it serves.
Cleanup is deferred, not cancelled -- removing the last name on a bundle
still unlinks the file and deletes the lineage.
No row on either production host has ssl_enabled=1 with an empty
ssl_cert_path, so the "no path, no attributable lineage" branch changes
nothing on the current fleet.
Tests (scripts/test-cert-write-safety.py, +9, suite now 31, all offline):
last reference -> file unlinked and lineage deleted; shared file survives
removal of a SAN member AND of the bundle primary, byte-for-byte, with the
edge still starting; shared lineage is not certbot-deleted; the production
mail.* shape; removing every name eventually cleans up; an unrelated
bundle is never collateral damage. Assertions are on os.path.exists, file
contents and the recorded certbot argv, never on which branch ran.
Mutation-tested, all five mutants killed:
1. guard absent entirely (suite run with HAPROXY_MANAGER_DIR pointed at
main) -> 6 failures, incl. "example.com is still configured and still
served from this file".
2. refcount taken before the row is deleted -> 5 failures, incl. "the
last reference is gone - now it may be removed".
3. certbot guard removed, file guard kept -> 3 failures, incl.
[] != ['delete --cert-name example.com --non-interactive'].
4. lineage taken from the domain name instead of the cert path -> 3
failures, incl. 'delete --cert-name example.com' != 'delete
--cert-name www.example.com'.
5. file-unlink guard removed, certbot guard kept -> 4 failures, incl.
"two sites are still served from this bundle".
Other suites unchanged and green: test-config-rollback, test-cert-scripts,
test-stick-table-contract, test-runtime-map-contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent
`add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True
whenever socat exited 0. Neither command has ever worked, on any deployment,
for the entire life of the feature -- while logging "Added IP x to runtime map"
every single time. Two independent defects:
* NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map
commands are worker commands. Captured verbatim on whp01:
$ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli
Unknown command: 'add', but maybe one of the following ones is a better match:
@!<pid> : send a command to the <pid> process
...
$ echo $?
0
socat exits 0 on the rejection, so `result.returncode == 0` was true. Same
silence PR #7 fixed on the `show table` path.
* `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move
on every config regeneration -- `@1 show map` on whp01 reports
blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0.
Hardcoding any number is wrong; the map is referenced by FILE PATH, which
is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0).
And a third silence, which is why a response-body check alone is not enough
here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to
any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer
`Unknown map identifier.`. On the add path the reply is byte-for-byte identical
to success. Only reading the entry back can tell them apart.
IP blocking itself was never broken: update_blocked_ips_map() rewrites
/etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it.
That path is untouched and stays authoritative. What was broken is the
no-reload fast path, plus every report that it had worked.
* haproxy_manager.py: both functions send `@1 add|del map
/etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with
`get map` before returning True. runtime_map_lookup()/runtime_map_keys()
are the read-back primitives. `sync_blocked_ips` loses `clear map #0`
(which the master socket rejected just as loudly and just as invisibly) and
verifies the whole set with one `show map` instead of counting commands
that did not visibly complain; it answers 207 + `runtime_map_synced: false`
when the runtime map does not match the database.
* haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy
answers those with nothing on success, so an empty body is the success and
ANY non-empty body is a rejection. That is stricter than the marker list on
purpose -- markers only recognise rejections someone has already seen, and
it catches `'add map' expects three parameters ...`, which matches nothing.
HaproxyCliError carries `.responses` so `del map` answering `Key not found.`
(the requested end state) is told apart from a real failure without regex.
* The four callers capture the boolean instead of discarding it and report
`runtime_map_updated` / `runtime_map_failures` in the API response and the
operation log. A runtime failure degrades to "enforced on the reload that
already happens two lines later" -- never to an unblocked IP, never to a
500.
* scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes
on the wire (`@1` first, map by path, value `1`), classifies every captured
response, and scans the repo's Python string literals and shell/template
code lines for `#<id>` map references -- comments may describe the old
form, code may not use it. Verified to fail on each defect reintroduced
separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the-
reply (1).
* The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md
and the comment in templates/hap_listener.tpl -- where every copy of it
additionally omitted the `1`, which `-m int gt 0` needs to match.
The only template change is a comment; `haproxy -c` on the live rendered config
with it applied is clean (HAProxy 3.0.11, warnings unchanged).
Verified on whp01 against the running container (docker cp + SIGHUP, no
recreate). Before: both functions returned True and logged success while
`@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed
add lands with value "1" and the remove takes it out again; the old command
form is now classified as a failure; a `#0` map reference returns False via the
read-back. End to end through the API, `runtime_map_updated: true`, and
/api/blocked-ips/sync -- which used to be a no-op reporting a full sync --
reports 264/264 verified present.
The runtime path was isolated from the reload that normally follows it: with
NO map-file write and NO reload (same haproxy worker pid throughout), adding
100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not
in the is_local ranges) to the runtime map alone flipped a live site from
HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path
working for the first time. All test IPs were removed afterwards: 0 rows in
blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer
sites, the panel /health and `haproxy -c` are byte-identical to the baseline
taken before the change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/api/security/stats and scripts/show-tarpit-ips.sh reported "Scan Count",
"offense count" and BLOCKED/TARPITTED status parsed from gpc0/gpc1. No stick
table in this repo has ever stored a general-purpose counter -- the `web` table
stores conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s), and
the two brute-force tables store http_req_rate(60s). Every one of those figures
was fabricated, and an operator was making decisions on them.
Three independent silences kept it alive:
* `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
and the loop `continue`d -- so the endpoint always answered
`active_threats: 0` with an empty list. Live on whp01 it also reported
parts[0], the `0x...:` allocation pointer, as the source IP.
* The command was sent to /tmp/haproxy-cli WITHOUT the `@1` worker prefix.
That is the MASTER CLI socket, which answers "Unknown command: 'show' ..."
-- and socat still exits 0, so the `returncode != 0` guard never fired.
`total_tracked_ips` was the line count of that help text (8) while the real
table held 388 entries.
* The shell consumers wrote `gpc0=${gpc0:-0}`, rendering a field that does
not exist as a confident zero.
Report what the tables actually store, rather than adding gpc counters to make
the old semantics real. Adding them would mean editing hap_listener.tpl -- the
one change here with a silent-total-outage failure mode -- to rebuild
enforcement history that the edge access log (shipped 2026.08.8, on the host at
/var/log/haproxy.log) already records per request, with status codes,
termination states and request references the stick table could never hold.
* haproxy_manager.py: STICK_TABLE_FIELD_CONTRACT names what each table
stores. haproxy_cli() sends worker commands with `@1`, falls back to the
bare form for a plain stats socket, and inspects the RESPONSE BODY because
socat's exit status is worthless here. parse_stick_table_entry() reads
name=value / name(window_ms)=value pairs by NAME, never by position.
read_stick_table() RAISES -- naming the field -- when a row is missing a
contract field, instead of defaulting it to 0.
* /api/security/stats returns the four real counters with their windows, the
true `used:` count, and no invented threat_level/blocked/offense_count.
Fewer numbers, all of them real.
* scripts/show-edge-ip-rates.sh replaces the fabricated report; the four
expected fields are declared once as EXPECTED_FIELDS and drive the parser.
show-tarpit-ips.sh becomes a shim that explains why its numbers are gone
and points at where tarpit events actually live.
* monitor-attacks.sh loses fourteen fabricated "threat" categories and a
composite threat score, all permanently zero; its access-log section now
says the log is on the host instead of silently printing nothing.
* haproxy_tarpit_config.txt -- the never-shipped design sketch these counters
were copied from -- gets a NOT IMPLEMENTED banner.
* scripts/test-stick-table-contract.py (offline, 21 tests) holds the
templates' `store` clauses, STICK_TABLE_FIELD_CONTRACT and every consumer
to each other, and asserts each loud-failure path against the real captured
responses. Template and consumers can no longer drift apart quietly.
No template is touched, so haproxy.cfg is unchanged.
Verified on whp01: total_tracked_ips now tracks `used:` exactly (511 vs the
table's 511, was 8 vs 388), and per-IP values match `show table web key <ip>`
field for field. haproxy PIDs unmoved, `haproxy -c` warnings unchanged, five
customer sites HTTP 200.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects caught by watching the real production access log on whp01 in the
minutes after 2026.08.7 made access logging work for the first time.
1. User-Agent was being truncated to its tail.
`http-request capture req.hdr(User-Agent)` treats the header as a
comma-separated list and returns only the LAST element. Real User-Agent
strings contain commas, so
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
logged as
ua=like Gecko) Chrome/131.0.0.0 Safari/537.36
losing the platform half -- exactly the half needed to tell a spoofed
crawler from a real browser, which is one of the main reasons the field was
added. Switched to req.fhdr(), which returns the full unsplit header value.
2. SPOE was writing one log line per inspected request.
`log global` inside the spoe-agent block emitted
SPOE: [coraza] <GROUP:coraza-req> sid=537 st=0 0/0/0/0/0 32/32 0/0 0/467
for every single request. Measured on whp01: 618 SPOE lines against 669 real
access lines -- ~48% of the log volume, roughly doubling the edge's log
footprint (~400 MB/day extra) to record `st=0` over and over.
It carries nothing incident response needs. The WAF verdict is already in
the access line (status 403 plus the id= UUID, which joins to
/var/log/coraza/audit.log for the rule_id), and per-transaction WAF detail
is written by the SPOA itself to /var/log/coraza/spoa.log. Agent-level
failures still surface through `option set-on-error error` ->
var(txn.coraza.error) and the fail-open path in hap_listener.tpl.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; wp-admin gate,
trusted-proxy gate and xmlrpc rate-limit suites all still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.
The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.
Changes:
* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
/var/log/messages or the Graylog forwarder, bound to the bridge IP rather
than 0.0.0.0.
* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
standalone/home deployments on a different bridge subnet can retarget it.
* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
not sufficient for incident response -- it omits %ID entirely (verified
against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
for CF-fronted sites. The new format keeps the first 16 fields byte-identical
to the httplog default (so existing parsers still work) and appends
cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
Adds a User-Agent capture in slot 1 to feed it.
* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
It does not, which made the documented support-correlation workflow
(X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
supported when it could never have worked.
Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".
UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:
<150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
[22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HAProxy warns on any pattern whose first character is "(", because it
cannot distinguish an intended regex from a fetch-argument list with a
stray space:
parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
'(^|/)wp-admin/...' is likely a mistake and probably not what you want.
"--" is HAProxy's documented end-of-flags marker and is the remedy the
warning itself names. Cosmetic to matching, but not to operations: left
unsilenced it fires on every config load and every reload on every host,
which trains operators to skim past warnings and gives a real one
somewhere to hide.
Matching semantics are unchanged, verified rather than assumed. Both
forms were run side by side as two frontends under real HAProxy
3.0.11-1+deb13u3 and gave identical verdicts on all 8 vectors:
/wp-admin/css/login.min.css MATCH / MATCH
/wp-admin/js/user-profile.min.js MATCH / MATCH
/wp-admin/images/x.png MATCH / MATCH
/wp-admin/css/sub/deep.css MATCH / MATCH
/blog/wp-admin/css/a.css MATCH / MATCH
/wp-admin/css/x.php NOMATCH / NOMATCH
/wp-admin/plugins.php NOMATCH / NOMATCH
/some--path/file.css NOMATCH / NOMATCH
The last vector is the one that matters: it proves HAProxy consumed "--"
as end-of-flags rather than adopting it as the pattern. Had it done the
latter, the ACL would have matched paths containing "--" and stopped
matching css/js -- gating every login page's own stylesheets while the
page itself still returned 200.
wp_admin_path needs no "--" only because its "-i" flag already occupies
the flag slot; it is not otherwise special.
Adds a regression test asserting both the "--" and the pattern it
guards, so this cannot pass by the pattern having been changed. Verified
to fail when the "--" is removed.
Previously unpinned, so the binary could move at Debian's timing and break an
unrelated commit's build -- or ship an edge that refuses to start. Pinning and
the haproxy -c gate compose: pinned means the version moves deliberately, and
the gate then answers whether the new binary still accepts fleet config. Also
makes the image reproducible.
A config change can render perfectly, pass every unit test in scripts/, and
still be rejected outright by HAProxy. That happened on 2026-08-14: an inline
`regsub((^|/)wp-admin/.*,\1wp-login.php)` in a redirect location produced
"invalid arg 2 in converter 'regsub'". Thirteen tests were green. It was only
caught because someone built an image by hand and ran `haproxy -c`.
Nothing between commit and production would have stopped it. The unit suites
assert on the TEXT of the rendered config with regexes, which says what the
template emits, never whether HAProxy accepts it. test-config-rollback.py's
"validation" stubs the haproxy binary with a shell script that rejects one
sentinel token and has never parsed a line of real syntax. And
.gitea/workflows/build-push.yaml is checkout -> build -> push, with no tests
at all.
The production consequence is not a broken deploy, it is a silent outage:
init.py refuses to start HAProxy on an invalid config while the container
still comes up, so ports 80/443 are unbound, every site on the host is down,
and /health keeps answering 200.
scripts/validate-rendered-config.py renders the config through the real
generate_config() - every template, real order, both conditional branches
({%- if suspension_enabled %} and {%- if coraza_spoe_backend %}) rendered on
in one scenario and off in the other - creates the stub files the config
loads via `-f` (a missing one is a FATAL haproxy error and would be a false
failure), then runs `haproxy -c` and gates on its EXIT CODE. Warnings are
expected on a clean config ("Can't load stats file", path_reg advisories) and
are not failures; on a real failure the full haproxy output plus the offending
config lines go to the build log.
It runs as a Dockerfile RUN rather than a CI step so it cannot be skipped, so
it protects local builds too, and - the reason that matters most - so it
validates against the EXACT haproxy binary in the image being built. The
Dockerfile installs haproxy unpinned, so that binary moves between builds;
this turns "the new haproxy rejects our config" from a silent production risk
into a build failure. Gating in CI instead would also have meant splitting
build-push-action's single build-and-push step.
The six existing unit suites run in the same step. They had never run
anywhere automated either, and they cost about five seconds.
Verified both ways: the clean build passes and the gate's output appears in
the log; reintroducing the known-bad regsub into a copy of the template fails
the build with HAProxy's own "invalid arg 2 in converter 'regsub' : missing
arguments (got 1/2)".
Adversarial mutation audit found the wp-admin gate test suite (26 tests, all
green) did not actually test the feature: 14 of 26 assertions ran bare
str.index/assertIn/re.search over the full rendered config, so they matched
this file's own explanatory comment blocks (which quote ACL names and whole
rules) just as happily as the real rule. Deleting the entire redirect rule,
or `acl wp_admin_allowed`, or all five normalizers, left the old suite at
26/26 PASS. rule_lines() also only stripped whole-comment lines, so a
trailing " # decoy" comment on a surviving line could impersonate a deleted
one, and one ordering test used bare cfg.index() which still "finds" a
normalize-uri directive that has been fully commented out (the substring
survives after the '#').
Rewrites every rule-presence/content/ordering assertion to go through
rule_lines()/rule_positions(), now truncating each line at the first ' #'
before matching, and adds require_rule()/require_position() guards so a
missing rule raises a named AssertionError instead of IndexError or
"substring not found". Adds dedicated declared-ACL tests for wp_admin_path,
wp_admin_asset, wp_admin_allowed and wp_gate_exempt so each has its own
direct, comment-safe check. 29 tests now (was 26).
Proved via a mutation harness (copy templates to a scratch dir, mutate the
copy, run the suite via HAPROXY_MANAGER_DIR, restore): commenting out the
redirect rule, either deny rule, any of the four wp_admin_* ACLs, any one of
the five normalize-uri lines, or expose-experimental-directives now reddens
the suite -- 13/13 required mutations caught, plus the exact trailing-comment
decoy and "all five normalizers commented at once" cases from the audit.
Also corrects two doc claims the audit found factually wrong:
- hap_listener.tpl: normalize-uri's percent-to-uppercase and
percent-decode-unreserved rewrite the WHOLE request-target, not just the
path -- measured examples included, and the query-sort-by-name rejection
reasoning ("every rule matches path") was a non-sequitur given that. Real
reason to leave it off: reordering would break signed/cached URLs. Fleet
checked: no .NET backends, no URL-in-path proxies, no known victim today.
- hap_header.tpl: dropping expose-experimental-directives does not
crash-loop the container. do_initial_setup() swallows the `haproxy -c`
failure and start_haproxy() returns without raising, so start-up.sh execs
gunicorn as PID 1 anyway -- a silent total outage (ports 80/443 unbound,
every site down) that ensure_haproxy.py retries forever without
escalating, while GET /health keeps answering 200.
No HAProxy rule, ACL, or normalizer changed -- comments and tests only.
Verified: all 5 required suites green, and `haproxy -c` against the real
haproxy 3.0.11 (Debian package) still exits 0 with only the same pre-existing
warnings as before (wp_admin_asset path_reg advisory, stats file).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wp-admin edge gate matched the RAW request path while the backend
normalised and decoded it before resolving a file. Every gap between those
two behaviours was a bypass, and five had already been patched individually:
//wp-admin/plugins.php fell through ungated
/wp-admin/css/../plugins.php took the static-asset bypass
/wp-admin/js/%2e%2e/plugins.php same, percent-encoded
/wp%2Dadmin/plugins.php matched no wp-admin ACL at all
/wp-admin%2Fplugins.php encoded separator, served by OLS
Stop patching vectors and normalise once, first, so every path-based rule in
the frontend sees the same string the backend will resolve:
percent-to-uppercase
percent-decode-unreserved
path-merge-slashes
path-strip-dot
path-strip-dotdot full
Order was determined empirically against real haproxy 3.0.11, not from the
docs: the decoders MUST precede the path walkers, or %2e%2e is decoded to ..
only after path-strip-dotdot has already run and the traversal survives. Plain
path-strip-dotdot also leaves /../../ untouched -- "full" is required.
query-sort-by-name is deliberately not enabled; it reorders query parameters
and would break anything signing or caching on the exact query string.
normalize-uri is experimental in 3.0, so global gains
expose-experimental-directives -- without it haproxy does not start at all.
The two must be added and removed together.
%2F cannot be closed by normalisation ("/" is reserved, so decoding it is
correctly refused), so it gets its own deny, scoped to paths mentioning
wp-admin so non-WordPress apps that pass encoded slashes in path parameters
keep working. Deny rather than redirect: regsub finds no "/wp-admin/" in
"/wp-admin%2F...", so a redirect would point at the request's own URL.
Gate changes:
* wp_admin_safe_path KEPT -- merge-slashes kills its "//" vector but not
"/\", which no normalizer touches. Its failure mode (unsafe path is not
redirected, therefore falls through UNGATED -- the original C1) is now
closed by an explicit deny instead of being left implicit.
* wp_admin_asset now excludes .php, so the asset bypass cannot cover a PHP
entrypoint even if an encoding trick ever survives normalisation.
* wp_admin_path is case-insensitive, paired with a matching regsub flag --
adding either alone is an infinite redirect loop.
Verified behaviourally against real haproxy 3.0.11 with raw sockets (curl
normalises client-side and hides these), run twice: once against the rendered
templates and once against the haproxy.cfg generated by a real, healthy
container. 12/12 gated, 19/19 passed through, 7/7 with no off-site Location,
plus ~30 adversarial vectors. haproxy -c exits 0 and the container reaches
healthy. Blast radius measured on a 40-URL production-shaped corpus: 4
rewritten, all RFC-equivalent (%7E->~, /./ , //); query strings and all
non-unreserved escapes byte-identical.
Full evidence:
.superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-4-normalize-report.md
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The redirect target is built by regsub-rewriting `path`, which only
replaces the matched "/wp-admin/.*" substring -- anything before it
survives untouched. Three request forms turn that survival into an
off-site Location header: a protocol-relative "//evil/wp-admin/x.php",
a browser-normalized "/\evil/wp-admin/x.php", and an RFC 7230
absolute-form request target. Without this gate those paths simply
404 against WordPress; the gate itself is what would have exposed a
fleet-wide phishing primitive.
Adds a positive wp_admin_safe_path ACL (path_reg ^/[^/\\]) requiring a
well-formed absolute path, required alongside the existing conditions
on the redirect rule. A path that fails it is simply not redirected
and falls through to the backend -- pre-gate behavior, so no
regression. set-var is left unguarded since it only computes a
variable; the redirect is what emits the header, so guarding it is
sufficient.
Verified against real HAProxy 3.0.11: the naive two-backslash form
fails to compile (config-line word parsing collapses "\\" to one
backslash before PCRE sees it, leaving an unterminated class); four
backslashes are required in the template so PCRE receives the
intended single-backslash class member. Confirmed live, via a
differential test against the pre-fix rule, that both the // and /\
vectors previously produced off-site Location headers and now do not,
while normal root and subdirectory-install redirects are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects in the wp-admin edge gate (2171bed, 704be38):
1. HAProxy 3.0.11 rejects the inline regsub redirect
(regsub((^|/)wp-admin/.*,\1wp-login.php)) with "invalid arg 2 in
converter 'regsub': missing arguments". Verified this is a
converter-argument-parenthesis-counting limitation -- the inner
"(^|/)" grouping parens are misread as closing the outer regsub()
call, and neither quoting nor backslash-escaping the parens helps.
Since HTTP paths always start with "/", the group is unnecessary:
compute the login URL in its own set-var, matching the literal
substring "/wp-admin/" (no group, no backreference) and replacing
it with the literal "/wp-login.php" -- regsub only replaces the
matched substring, so a subdirectory-install prefix survives
untouched.
2. wp_admin_allowed used a bare path_end suffix match
(/admin-ajax.php etc), so /wp-admin/evil/admin-ajax.php matched
both wp_admin_path and the allowlist and sailed through the gate
ungated. Anchored each entry to /wp-admin/<file>.
Verified against real HAProxy 3.0.11-1+deb13u3: haproxy -c exit 0,
and live curl against the real generated config's literal lines
confirms root-install and subdirectory-install redirects, the
anchored-allowlist fix, cookie exemption, and non-wp-admin passthrough
all behave correctly.
Extends scripts/test-wpadmin-gate.py with regression tests for the
anchored allowlist and the set-var ordering/no-inline-regsub guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Redirect /wp-admin/* to the site's login page when no wordpress_logged_in_
cookie is present, so unauthenticated requests never boot PHP. Identity-based
rather than rate-based, so it is unaffected by how widely an attack is
distributed. Allowlists the paths that legitimately serve unauthenticated
visitors, including the css/js the login page itself loads.
Mirrors the existing wp-login.php brute-force protection. Generic frontend
limits trigger at 300-500 req/s (sized for media-heavy pageloads), but
observed xmlrpc floods run at just a few req/s for hours -- well under that
ceiling while still pinning PHP-FPM workers and driving 503s fleet-wide
(1,011 in one day on a single site).
Adds a dedicated stick-table (xmlrpc_bruteforce, sc2) rather than reusing
wp_bruteforce: sharing a counter would let wp-login and xmlrpc traffic from
the same IP inflate each other's rate. Tarpits at 60 req/min/IP (double
wp-login's 30, since xmlrpc is machine-to-machine and legitimately bursts --
Jetpack sync, mobile app, remote publishing). Honors the same whitelist as
every other rule in the file and does not block the endpoint outright.
Only safe to key on var(txn.real_ip) because of the trusted-proxy header
gate shipped earlier today (2026.08.3) -- before that, per-IP tracking was
trivially evaded via a spoofed X-Forwarded-For.
Adds scripts/test-xmlrpc-rate-limit.py (stdlib unittest, no pytest in this
repo) pinning the tracking rule, the tarpit threshold, the path_end ACL, and
the whitelist exclusions. Existing trusted-proxy-gate, config-rollback, and
cert-write-safety regression suites all still pass unmodified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/etc/haproxy is a named volume in deployed containers, so the baked-in
cloudflare_ips.list and trusted_proxies.list COPYed there in the prior
task never actually reached hosts with a pre-existing volume -- the
start-up.sh guard then found them "missing" and created them empty.
With both lists empty, the from_trusted_proxy ACL in hap_listener.tpl
matched nothing, so CF-Connecting-IP / X-Real-IP / X-Forwarded-For got
stripped from every peer, including Cloudflare's own edge. Confirmed
live: image shipped 34/13 lines, running container had 0/0.
Fix: stage both files under /haproxy/defaults (outside the volume) and
apply their ownership rule in start-up.sh instead of a blind
"create if missing":
- cloudflare_ips.list is shipped data -- always refresh it from the
baked default so Cloudflare range updates reach existing hosts.
- trusted_proxies.list is operator data -- seed it from the baked
default only when missing, and never overwrite what an operator
added on the server.
Both branches fall back to creating an empty file if the baked default
is somehow absent, since a missing "-f" target is a fatal HAProxy
config error.
Verified against a volume pre-populated to shadow the image (mimicking
a real host): cloudflare_ips.list repopulates with all 15 IPv4 + 7
IPv6 ranges even after being truncated and restarted; a distinctive
operator entry appended to trusted_proxies.list survives a restart
untouched; haproxy -c still validates cleanly.
Release-worthy fix for a defect from the just-released 2026.08.2 build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restrict CF-Connecting-IP, X-Real-IP and X-Forwarded-For to peers matching
cloudflare_ips.list or trusted_proxies.list; other peers fall through to src.
Adds a regression test pinning the strip-before-resolve ordering.
Two stacked fixes for the edge that terminates every customer's HTTPS and
routes every customer's traffic. They ship together because deploying
either means recreating haproxy-manager on every host.
1. Config rollback was a no-op. generate_config() wrote the new config
BEFORE create_backup() copied it, so restore_backup() restored the
identical broken file while logging 'Backups restored successfully'.
The backup is now taken before the first write, refuses to promote a
config haproxy already rejects, and covers blocked_ips.map and
coraza-spoe.cfg as one restorable set. blocked_ips.map is now written
atomically too - haproxy -c parses it, so a truncated map is a fatal
config.
2. Certificate publishing truncated the live PEM. Six sites opened the
bundle HAProxy is serving in truncate mode, then read the source; any
failure between left a key-less or partial PEM, unrecoverable by config
rollback. The shell path was worse: with a zero-length source key, cat
exits 0, so renew-certificates.sh logged 'Updated certificate', reported
'0 failed', and reloaded HAProxy onto a key-less bundle (measured:
2912 -> 1208 bytes, key gone, exit 0). Publishing is now assemble ->
validate -> back up -> os.replace(), staged in a SIBLING directory
because HAProxy loads the crt path as a directory and would parse a
stray .tmp. certbot delete no longer runs before the replacement has
validated and loaded.
Also fixed: the shipped test suite stripped the exec bit from twelve system
binaries - chmod included - when run as root inside the container, because
os.chmod follows symlinks. It reported 32 OK on a workstation and 18
failures in the image it ships to.
Behaviour changes: renewals exit non-zero when any domain fails to publish
(previously always 0, and a test codified that); a host without openssl
refuses to publish rather than trusting structural checks alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shadowdaoandClaude Opus 5.5
jknapp