Two defects caught by watching the real production access log on whp01 in the
minutes after 2026.08.7 made access logging work for the first time.
1. User-Agent was being truncated to its tail.
`http-request capture req.hdr(User-Agent)` treats the header as a
comma-separated list and returns only the LAST element. Real User-Agent
strings contain commas, so
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
logged as
ua=like Gecko) Chrome/131.0.0.0 Safari/537.36
losing the platform half -- exactly the half needed to tell a spoofed
crawler from a real browser, which is one of the main reasons the field was
added. Switched to req.fhdr(), which returns the full unsplit header value.
2. SPOE was writing one log line per inspected request.
`log global` inside the spoe-agent block emitted
SPOE: [coraza] <GROUP:coraza-req> sid=537 st=0 0/0/0/0/0 32/32 0/0 0/467
for every single request. Measured on whp01: 618 SPOE lines against 669 real
access lines -- ~48% of the log volume, roughly doubling the edge's log
footprint (~400 MB/day extra) to record `st=0` over and over.
It carries nothing incident response needs. The WAF verdict is already in
the access line (status 403 plus the id= UUID, which joins to
/var/log/coraza/audit.log for the rule_id), and per-transaction WAF detail
is written by the SPOA itself to /var/log/coraza/spoa.log. Agent-level
failures still surface through `option set-on-error error` ->
var(txn.coraza.error) and the fail-open path in hap_listener.tpl.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; wp-admin gate,
trusted-proxy gate and xmlrpc rate-limit suites all still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.
The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.
Changes:
* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
/var/log/messages or the Graylog forwarder, bound to the bridge IP rather
than 0.0.0.0.
* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
standalone/home deployments on a different bridge subnet can retarget it.
* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
not sufficient for incident response -- it omits %ID entirely (verified
against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
for CF-fronted sites. The new format keeps the first 16 fields byte-identical
to the httplog default (so existing parsers still work) and appends
cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
Adds a User-Agent capture in slot 1 to feed it.
* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
It does not, which made the documented support-correlation workflow
(X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
supported when it could never have worked.
Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".
UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:
<150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
[22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/etc/haproxy is a named volume in deployed containers, so the baked-in
cloudflare_ips.list and trusted_proxies.list COPYed there in the prior
task never actually reached hosts with a pre-existing volume -- the
start-up.sh guard then found them "missing" and created them empty.
With both lists empty, the from_trusted_proxy ACL in hap_listener.tpl
matched nothing, so CF-Connecting-IP / X-Real-IP / X-Forwarded-For got
stripped from every peer, including Cloudflare's own edge. Confirmed
live: image shipped 34/13 lines, running container had 0/0.
Fix: stage both files under /haproxy/defaults (outside the volume) and
apply their ownership rule in start-up.sh instead of a blind
"create if missing":
- cloudflare_ips.list is shipped data -- always refresh it from the
baked default so Cloudflare range updates reach existing hosts.
- trusted_proxies.list is operator data -- seed it from the baked
default only when missing, and never overwrite what an operator
added on the server.
Both branches fall back to creating an empty file if the baked default
is somehow absent, since a missing "-f" target is a fatal HAProxy
config error.
Verified against a volume pre-populated to shadow the image (mimicking
a real host): cloudflare_ips.list repopulates with all 15 IPv4 + 7
IPv6 ranges even after being truncated and restarted; a distinctive
operator entry appended to trusted_proxies.list survives a restart
untouched; haproxy -c still validates cleanly.
Release-worthy fix for a defect from the just-released 2026.08.2 build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
generate_config() wrote /etc/haproxy/haproxy.cfg and only then called
reload_haproxy_safely(), which called create_backup(). The "backup" was
therefore a copy of the config that had just been written, so on a validation
failure restore_backup() restored the identical broken bytes: the advertised
rollback was a no-op and a fatal haproxy.cfg stayed on disk, where
start_haproxy() refuses to launch. Same shape as the June 2026 incident where
a missing template produced a fatal config and took an edge down.
Reproduced end to end before the fix (invalid config generated -> "Backups
created successfully" -> "Backups restored successfully" -> haproxy.cfg on
disk still invalid, `haproxy -c` rc=1).
Changes:
* create_backup() is now called by generate_config() BEFORE the first write,
which also covers blocked_ips.map (rewritten early in generate_config) and
coraza-spoe.cfg - both previously written before the backup and, for the
SPOE file, never backed up at all even though `haproxy -c` parses it.
* create_backup() refuses to promote a config HAProxy already rejects, so a
broken file on disk cannot overwrite a known-good backup ("rollback" must
not mean "restore a different broken config"). It returns (ok, status) so
the caller knows whether a rollback target exists.
* promote_current_config_to_backup() records the config as known-good only
after it has validated AND loaded, so a box whose first generation succeeded
has a rollback target immediately, and a config that never loaded is never
promoted.
* restore_backup() returns (restored, message) and distinguishes "no backup
available" from "restored". Every caller now surfaces the difference; a
failed rollback is logged CRITICAL and reported as ROLLBACK FAILED in the
API error message instead of silently looking like a successful recovery.
* reload_haproxy_safely(backup_status=...) no longer takes its own backup - it
runs after the write, where a backup is meaningless. Called without a status
it logs the contract violation rather than overwriting a good backup.
* validate_config_file() separates "config is invalid" from "validator could
not run" so a missing haproxy binary is not read as a bad config.
* Config writes are atomic (temp file + fsync + os.replace, mode preserved);
a truncated haproxy.cfg is as fatal as an invalid one. Removes the dead
temp_config_path variable whose comment claimed this already happened.
* Fast path: if the live config set is already byte-identical to the backup
(the normal case after a successful reload), skip the re-validation and the
copy, so this adds no `haproxy -c` latency to customer-facing API calls.
Tests: scripts/test-config-rollback.py - 17 self-contained stdlib-unittest
tests, no new dependencies (the repo has no Python test framework; the
existing scripts/test-*.sh are curl integration scripts). A stub `haproxy`
binary stands in for the validator. 12 of the 17 fail against the previous
code; every assertion was mutation-proven (9 mutations, each reddening only
the tests that cover it).
No template, QUIC or HTTP/3 changes.
The management API wedged on whp01 2026-07-07: every panel call to the
manager (config regenerate during a WHP site update, SSL, even /health)
timed out at 30s while customer sites stayed up. Root cause: all four
gunicorn gthread worker threads were permanently blocked in socket reads
inside untimed subprocess.run() calls (certbot ACME / socat reloads). A
stalled external command holds its worker thread forever; gunicorn
--timeout can't rescue it (gthread only kills a worker whose main thread
stops heart-beating, and ours kept polling). Stalled calls accumulated
until the 4-thread pool was exhausted and the whole API went dark.
- Wrap subprocess.run with a default timeout (HAPROXY_MGR_SUBPROCESS_TIMEOUT,
180s) so every external command is bounded and releases its thread on
expiry via the existing per-endpoint try/except. Bounding by default
covers all ~30 call sites and any future one.
- certbot renew keeps an explicit 900s timeout (walks every lineage).
- API_WORKERS default 1 -> 2: a single worker made a thread-pool wedge a
total outage; a second worker keeps the API answering while one recycles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
generate_config emits "Skipping domain <host> - no backend name" on every run
for domains registered without a proxy backend — most commonly the panel's own
hostname (whpNN.cloud-hosting.io), which lives in the DB only for certificate
management and intentionally has no backend. Logging it at WARNING tripped the
WHP AI log monitor as a recurring error and prompted a bogus "restart
haproxy-manager" remediation. It's expected, benign, and recurs by design.
Log it at INFO instead (consistent with the sibling per-domain "Added ACL for
domain" INFO lines) with a clearer message ("no proxy backend
(cert/management-only)"). Verified against the WHP monitor's ErrorClassifier:
the old WARNING line classified as non_critical (captured); the new INFO line
classifies as None (skipped) — so it no longer shows up in reports, while
remaining visible in container logs for manual routing debugging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
haproxy runs as a background child of PID 1 (gunicorn) with nothing
watching it after init. If the haproxy master dies mid-life (observed
2026-07-01 on whp01: SIGABRT -> exit 134, reaped by gunicorn and logged
as "Worker (pid:22) exited"), the container stays "up", Docker's
--restart never fires, and haproxy is down until the external host
watchdog full-restarts the whole container minutes later (dropping every
connection).
Add an in-container supervisor loop in start-up.sh (Phase 1.5) that runs
scripts/ensure_haproxy.py every HAPROXY_SUPERVISOR_INTERVAL (default 15s).
ensure_haproxy.py calls the existing, idempotent start_haproxy() only when
haproxy isn't running (psutil guard), reviving it in place within one
interval with no container restart. Same entrypoint-supervision pattern
shipped for cac-litespeed.
Validated locally: killing haproxy -> revived with new PIDs in ~one
interval, container stayed healthy, no spurious restarts while healthy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-IP throttle can't see distributed attacks (observed 76k–289k UNIQUE
IPs hitting wp-login.php, each low-and-slow). But those bots POST straight to
wp-login.php without GETting the form (~15:1 POST:GET on attacked sites). So:
hand out a `whplc` cookie on GET of the login form (set-var at request time +
http-after-response add-header — request fetches don't evaluate in the response
phase) and DENY 403 on login POSTs that lack it. Direct-POST bots are dropped
at the edge before reaching PHP; real logins are unaffected (WP login already
requires loading the page + cookies). Immediate deny, not tarpit, to avoid
connection exhaustion under a 300k-POST flood. Honors the whitelist.
Validated locally: GET /wp-login.php emits whplc; other paths don't; config OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The generic rate-limits are tuned high for media-heavy sites, so slow
credential-stuffing on wp-login.php slips under them. Add a dedicated sc1
stick-table (backend wp_bruteforce, 60s window) that counts POSTs to
wp-login.php per real client IP and tarpits once an IP exceeds 30/min.
Only login POSTs are counted (browsing + the login form GET + a legit user's
few attempts are unaffected); an offending IP can still browse, just not keep
hammering login. Honors the existing whitelist (RFC1918 / trusted_ips.list /
trusted_ips.map) and the already-resolved CF/proxy real IP. path_end also
covers subdirectory WP installs. Stops attacks at the edge before they reach
PHP/WordPress, on all edges regardless of Coraza mode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HTTP/3 is config-only — the Debian haproxy package is built +QUIC via the
OpenSSL compat shim. Changes:
- hap_header.tpl: `limited-quic` (required to enable QUIC binds under the
compat layer) + self-healing `cluster-secret` for QUIC token derivation.
- hap_listener.tpl: `bind quic4@:443 ... alpn h3` in the shared frontend (so
real-IP/rate-limit/IP-block/Coraza rules apply to H3 too) + alt-svc header.
- Dockerfile/README: publish/document 443/udp; stamp image.version from VERSION.
- CI: tag :latest + :<VERSION> + :<sha> so there's a pinnable rollback target.
No 0-RTT (compat-layer limitation). Validated end-to-end on a standalone edge:
config parses, UDP/443 binds, alt-svc advertised, real curl --http3 -> HTTP/3.
Container must run with `-p 443:443/udp` + host UDP/443 open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude
shadowdao