add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent
`add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True
whenever socat exited 0. Neither command has ever worked, on any deployment,
for the entire life of the feature -- while logging "Added IP x to runtime map"
every single time. Two independent defects:
* NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map
commands are worker commands. Captured verbatim on whp01:
$ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli
Unknown command: 'add', but maybe one of the following ones is a better match:
@!<pid> : send a command to the <pid> process
...
$ echo $?
0
socat exits 0 on the rejection, so `result.returncode == 0` was true. Same
silence PR #7 fixed on the `show table` path.
* `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move
on every config regeneration -- `@1 show map` on whp01 reports
blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0.
Hardcoding any number is wrong; the map is referenced by FILE PATH, which
is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0).
And a third silence, which is why a response-body check alone is not enough
here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to
any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer
`Unknown map identifier.`. On the add path the reply is byte-for-byte identical
to success. Only reading the entry back can tell them apart.
IP blocking itself was never broken: update_blocked_ips_map() rewrites
/etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it.
That path is untouched and stays authoritative. What was broken is the
no-reload fast path, plus every report that it had worked.
* haproxy_manager.py: both functions send `@1 add|del map
/etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with
`get map` before returning True. runtime_map_lookup()/runtime_map_keys()
are the read-back primitives. `sync_blocked_ips` loses `clear map #0`
(which the master socket rejected just as loudly and just as invisibly) and
verifies the whole set with one `show map` instead of counting commands
that did not visibly complain; it answers 207 + `runtime_map_synced: false`
when the runtime map does not match the database.
* haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy
answers those with nothing on success, so an empty body is the success and
ANY non-empty body is a rejection. That is stricter than the marker list on
purpose -- markers only recognise rejections someone has already seen, and
it catches `'add map' expects three parameters ...`, which matches nothing.
HaproxyCliError carries `.responses` so `del map` answering `Key not found.`
(the requested end state) is told apart from a real failure without regex.
* The four callers capture the boolean instead of discarding it and report
`runtime_map_updated` / `runtime_map_failures` in the API response and the
operation log. A runtime failure degrades to "enforced on the reload that
already happens two lines later" -- never to an unblocked IP, never to a
500.
* scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes
on the wire (`@1` first, map by path, value `1`), classifies every captured
response, and scans the repo's Python string literals and shell/template
code lines for `#<id>` map references -- comments may describe the old
form, code may not use it. Verified to fail on each defect reintroduced
separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the-
reply (1).
* The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md
and the comment in templates/hap_listener.tpl -- where every copy of it
additionally omitted the `1`, which `-m int gt 0` needs to match.
The only template change is a comment; `haproxy -c` on the live rendered config
with it applied is clean (HAProxy 3.0.11, warnings unchanged).
Verified on whp01 against the running container (docker cp + SIGHUP, no
recreate). Before: both functions returned True and logged success while
`@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed
add lands with value "1" and the remove takes it out again; the old command
form is now classified as a failure; a `#0` map reference returns False via the
read-back. End to end through the API, `runtime_map_updated: true`, and
/api/blocked-ips/sync -- which used to be a no-op reporting a full sync --
reports 264/264 verified present.
The runtime path was isolated from the reload that normally follows it: with
NO map-file write and NO reload (same haproxy worker pid throughout), adding
100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not
in the is_local ranges) to the runtime map alone flipped a live site from
HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path
working for the first time. All test IPs were removed afterwards: 0 rows in
blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer
sites, the panel /health and `haproxy -c` are byte-identical to the baseline
taken before the change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects caught by watching the real production access log on whp01 in the
minutes after 2026.08.7 made access logging work for the first time.
1. User-Agent was being truncated to its tail.
`http-request capture req.hdr(User-Agent)` treats the header as a
comma-separated list and returns only the LAST element. Real User-Agent
strings contain commas, so
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
logged as
ua=like Gecko) Chrome/131.0.0.0 Safari/537.36
losing the platform half -- exactly the half needed to tell a spoofed
crawler from a real browser, which is one of the main reasons the field was
added. Switched to req.fhdr(), which returns the full unsplit header value.
2. SPOE was writing one log line per inspected request.
`log global` inside the spoe-agent block emitted
SPOE: [coraza] <GROUP:coraza-req> sid=537 st=0 0/0/0/0/0 32/32 0/0 0/467
for every single request. Measured on whp01: 618 SPOE lines against 669 real
access lines -- ~48% of the log volume, roughly doubling the edge's log
footprint (~400 MB/day extra) to record `st=0` over and over.
It carries nothing incident response needs. The WAF verdict is already in
the access line (status 403 plus the id= UUID, which joins to
/var/log/coraza/audit.log for the rule_id), and per-transaction WAF detail
is written by the SPOA itself to /var/log/coraza/spoa.log. Agent-level
failures still surface through `option set-on-error error` ->
var(txn.coraza.error) and the fail-open path in hap_listener.tpl.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; wp-admin gate,
trusted-proxy gate and xmlrpc rate-limit suites all still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.
The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.
Changes:
* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
/var/log/messages or the Graylog forwarder, bound to the bridge IP rather
than 0.0.0.0.
* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
standalone/home deployments on a different bridge subnet can retarget it.
* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
not sufficient for incident response -- it omits %ID entirely (verified
against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
for CF-fronted sites. The new format keeps the first 16 fields byte-identical
to the httplog default (so existing parsers still work) and appends
cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
Adds a User-Agent capture in slot 1 to feed it.
* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
It does not, which made the documented support-correlation workflow
(X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
supported when it could never have worked.
Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".
UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:
<150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
[22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HAProxy warns on any pattern whose first character is "(", because it
cannot distinguish an intended regex from a fetch-argument list with a
stray space:
parsing acl 'wp_admin_asset' : matching 'path_reg' for pattern
'(^|/)wp-admin/...' is likely a mistake and probably not what you want.
"--" is HAProxy's documented end-of-flags marker and is the remedy the
warning itself names. Cosmetic to matching, but not to operations: left
unsilenced it fires on every config load and every reload on every host,
which trains operators to skim past warnings and gives a real one
somewhere to hide.
Matching semantics are unchanged, verified rather than assumed. Both
forms were run side by side as two frontends under real HAProxy
3.0.11-1+deb13u3 and gave identical verdicts on all 8 vectors:
/wp-admin/css/login.min.css MATCH / MATCH
/wp-admin/js/user-profile.min.js MATCH / MATCH
/wp-admin/images/x.png MATCH / MATCH
/wp-admin/css/sub/deep.css MATCH / MATCH
/blog/wp-admin/css/a.css MATCH / MATCH
/wp-admin/css/x.php NOMATCH / NOMATCH
/wp-admin/plugins.php NOMATCH / NOMATCH
/some--path/file.css NOMATCH / NOMATCH
The last vector is the one that matters: it proves HAProxy consumed "--"
as end-of-flags rather than adopting it as the pattern. Had it done the
latter, the ACL would have matched paths containing "--" and stopped
matching css/js -- gating every login page's own stylesheets while the
page itself still returned 200.
wp_admin_path needs no "--" only because its "-i" flag already occupies
the flag slot; it is not otherwise special.
Adds a regression test asserting both the "--" and the pattern it
guards, so this cannot pass by the pattern having been changed. Verified
to fail when the "--" is removed.
Adversarial mutation audit found the wp-admin gate test suite (26 tests, all
green) did not actually test the feature: 14 of 26 assertions ran bare
str.index/assertIn/re.search over the full rendered config, so they matched
this file's own explanatory comment blocks (which quote ACL names and whole
rules) just as happily as the real rule. Deleting the entire redirect rule,
or `acl wp_admin_allowed`, or all five normalizers, left the old suite at
26/26 PASS. rule_lines() also only stripped whole-comment lines, so a
trailing " # decoy" comment on a surviving line could impersonate a deleted
one, and one ordering test used bare cfg.index() which still "finds" a
normalize-uri directive that has been fully commented out (the substring
survives after the '#').
Rewrites every rule-presence/content/ordering assertion to go through
rule_lines()/rule_positions(), now truncating each line at the first ' #'
before matching, and adds require_rule()/require_position() guards so a
missing rule raises a named AssertionError instead of IndexError or
"substring not found". Adds dedicated declared-ACL tests for wp_admin_path,
wp_admin_asset, wp_admin_allowed and wp_gate_exempt so each has its own
direct, comment-safe check. 29 tests now (was 26).
Proved via a mutation harness (copy templates to a scratch dir, mutate the
copy, run the suite via HAPROXY_MANAGER_DIR, restore): commenting out the
redirect rule, either deny rule, any of the four wp_admin_* ACLs, any one of
the five normalize-uri lines, or expose-experimental-directives now reddens
the suite -- 13/13 required mutations caught, plus the exact trailing-comment
decoy and "all five normalizers commented at once" cases from the audit.
Also corrects two doc claims the audit found factually wrong:
- hap_listener.tpl: normalize-uri's percent-to-uppercase and
percent-decode-unreserved rewrite the WHOLE request-target, not just the
path -- measured examples included, and the query-sort-by-name rejection
reasoning ("every rule matches path") was a non-sequitur given that. Real
reason to leave it off: reordering would break signed/cached URLs. Fleet
checked: no .NET backends, no URL-in-path proxies, no known victim today.
- hap_header.tpl: dropping expose-experimental-directives does not
crash-loop the container. do_initial_setup() swallows the `haproxy -c`
failure and start_haproxy() returns without raising, so start-up.sh execs
gunicorn as PID 1 anyway -- a silent total outage (ports 80/443 unbound,
every site down) that ensure_haproxy.py retries forever without
escalating, while GET /health keeps answering 200.
No HAProxy rule, ACL, or normalizer changed -- comments and tests only.
Verified: all 5 required suites green, and `haproxy -c` against the real
haproxy 3.0.11 (Debian package) still exits 0 with only the same pre-existing
warnings as before (wp_admin_asset path_reg advisory, stats file).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wp-admin edge gate matched the RAW request path while the backend
normalised and decoded it before resolving a file. Every gap between those
two behaviours was a bypass, and five had already been patched individually:
//wp-admin/plugins.php fell through ungated
/wp-admin/css/../plugins.php took the static-asset bypass
/wp-admin/js/%2e%2e/plugins.php same, percent-encoded
/wp%2Dadmin/plugins.php matched no wp-admin ACL at all
/wp-admin%2Fplugins.php encoded separator, served by OLS
Stop patching vectors and normalise once, first, so every path-based rule in
the frontend sees the same string the backend will resolve:
percent-to-uppercase
percent-decode-unreserved
path-merge-slashes
path-strip-dot
path-strip-dotdot full
Order was determined empirically against real haproxy 3.0.11, not from the
docs: the decoders MUST precede the path walkers, or %2e%2e is decoded to ..
only after path-strip-dotdot has already run and the traversal survives. Plain
path-strip-dotdot also leaves /../../ untouched -- "full" is required.
query-sort-by-name is deliberately not enabled; it reorders query parameters
and would break anything signing or caching on the exact query string.
normalize-uri is experimental in 3.0, so global gains
expose-experimental-directives -- without it haproxy does not start at all.
The two must be added and removed together.
%2F cannot be closed by normalisation ("/" is reserved, so decoding it is
correctly refused), so it gets its own deny, scoped to paths mentioning
wp-admin so non-WordPress apps that pass encoded slashes in path parameters
keep working. Deny rather than redirect: regsub finds no "/wp-admin/" in
"/wp-admin%2F...", so a redirect would point at the request's own URL.
Gate changes:
* wp_admin_safe_path KEPT -- merge-slashes kills its "//" vector but not
"/\", which no normalizer touches. Its failure mode (unsafe path is not
redirected, therefore falls through UNGATED -- the original C1) is now
closed by an explicit deny instead of being left implicit.
* wp_admin_asset now excludes .php, so the asset bypass cannot cover a PHP
entrypoint even if an encoding trick ever survives normalisation.
* wp_admin_path is case-insensitive, paired with a matching regsub flag --
adding either alone is an infinite redirect loop.
Verified behaviourally against real haproxy 3.0.11 with raw sockets (curl
normalises client-side and hides these), run twice: once against the rendered
templates and once against the haproxy.cfg generated by a real, healthy
container. 12/12 gated, 19/19 passed through, 7/7 with no off-site Location,
plus ~30 adversarial vectors. haproxy -c exits 0 and the container reaches
healthy. Blast radius measured on a 40-URL production-shaped corpus: 4
rewritten, all RFC-equivalent (%7E->~, /./ , //); query strings and all
non-unreserved escapes byte-identical.
Full evidence:
.superpowers/sdd/2026-08-14-wpadmin-edge-gate/task-4-normalize-report.md
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The redirect target is built by regsub-rewriting `path`, which only
replaces the matched "/wp-admin/.*" substring -- anything before it
survives untouched. Three request forms turn that survival into an
off-site Location header: a protocol-relative "//evil/wp-admin/x.php",
a browser-normalized "/\evil/wp-admin/x.php", and an RFC 7230
absolute-form request target. Without this gate those paths simply
404 against WordPress; the gate itself is what would have exposed a
fleet-wide phishing primitive.
Adds a positive wp_admin_safe_path ACL (path_reg ^/[^/\\]) requiring a
well-formed absolute path, required alongside the existing conditions
on the redirect rule. A path that fails it is simply not redirected
and falls through to the backend -- pre-gate behavior, so no
regression. set-var is left unguarded since it only computes a
variable; the redirect is what emits the header, so guarding it is
sufficient.
Verified against real HAProxy 3.0.11: the naive two-backslash form
fails to compile (config-line word parsing collapses "\\" to one
backslash before PCRE sees it, leaving an unterminated class); four
backslashes are required in the template so PCRE receives the
intended single-backslash class member. Confirmed live, via a
differential test against the pre-fix rule, that both the // and /\
vectors previously produced off-site Location headers and now do not,
while normal root and subdirectory-install redirects are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects in the wp-admin edge gate (2171bed, 704be38):
1. HAProxy 3.0.11 rejects the inline regsub redirect
(regsub((^|/)wp-admin/.*,\1wp-login.php)) with "invalid arg 2 in
converter 'regsub': missing arguments". Verified this is a
converter-argument-parenthesis-counting limitation -- the inner
"(^|/)" grouping parens are misread as closing the outer regsub()
call, and neither quoting nor backslash-escaping the parens helps.
Since HTTP paths always start with "/", the group is unnecessary:
compute the login URL in its own set-var, matching the literal
substring "/wp-admin/" (no group, no backreference) and replacing
it with the literal "/wp-login.php" -- regsub only replaces the
matched substring, so a subdirectory-install prefix survives
untouched.
2. wp_admin_allowed used a bare path_end suffix match
(/admin-ajax.php etc), so /wp-admin/evil/admin-ajax.php matched
both wp_admin_path and the allowlist and sailed through the gate
ungated. Anchored each entry to /wp-admin/<file>.
Verified against real HAProxy 3.0.11-1+deb13u3: haproxy -c exit 0,
and live curl against the real generated config's literal lines
confirms root-install and subdirectory-install redirects, the
anchored-allowlist fix, cookie exemption, and non-wp-admin passthrough
all behave correctly.
Extends scripts/test-wpadmin-gate.py with regression tests for the
anchored allowlist and the set-var ordering/no-inline-regsub guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Redirect /wp-admin/* to the site's login page when no wordpress_logged_in_
cookie is present, so unauthenticated requests never boot PHP. Identity-based
rather than rate-based, so it is unaffected by how widely an attack is
distributed. Allowlists the paths that legitimately serve unauthenticated
visitors, including the css/js the login page itself loads.
Mirrors the existing wp-login.php brute-force protection. Generic frontend
limits trigger at 300-500 req/s (sized for media-heavy pageloads), but
observed xmlrpc floods run at just a few req/s for hours -- well under that
ceiling while still pinning PHP-FPM workers and driving 503s fleet-wide
(1,011 in one day on a single site).
Adds a dedicated stick-table (xmlrpc_bruteforce, sc2) rather than reusing
wp_bruteforce: sharing a counter would let wp-login and xmlrpc traffic from
the same IP inflate each other's rate. Tarpits at 60 req/min/IP (double
wp-login's 30, since xmlrpc is machine-to-machine and legitimately bursts --
Jetpack sync, mobile app, remote publishing). Honors the same whitelist as
every other rule in the file and does not block the endpoint outright.
Only safe to key on var(txn.real_ip) because of the trusted-proxy header
gate shipped earlier today (2026.08.3) -- before that, per-IP tracking was
trivially evaded via a spoofed X-Forwarded-For.
Adds scripts/test-xmlrpc-rate-limit.py (stdlib unittest, no pytest in this
repo) pinning the tracking rule, the tarpit threshold, the path_end ACL, and
the whitelist exclusions. Existing trusted-proxy-gate, config-rollback, and
cert-write-safety regression suites all still pass unmodified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restrict CF-Connecting-IP, X-Real-IP and X-Forwarded-For to peers matching
cloudflare_ips.list or trusted_proxies.list; other peers fall through to src.
Adds a regression test pinning the strip-before-resolve ordering.
Adds a frontend virtual patch denying /wp-json/batch/v1 and the
?rest_route=/batch/v1 fallback (including the %2F-encoded spelling) for
requests without a wordpress_logged_in_* cookie.
wp2shell chains CVE-2026-60137 (core SQL injection) with CVE-2026-63030
(REST batch-route confusion) into unauthenticated RCE on WP 6.9.0-6.9.4
and 7.0.0-7.0.1. Exploits are public and were used against this fleet on
2026-07-19/20; one site was compromised through this path, including a
re-injection of wp-includes/plugin.php nine minutes after it was patched.
Anonymous-only by design: batch/v1 is used legitimately by the block
editor for multi-entity saves, so a blanket deny would break wp-admin.
Placed ahead of the SPOE handoff so blocked requests never cost a WAF
round-trip, and it works regardless of Coraza mode (whp02 and sdbees run
detect_only, where a Coraza rule would log but not block).
This is a virtual patch, not a fix. It removes reachability only, and
stays until every site is confirmed on a fixed release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-IP throttle can't see distributed attacks (observed 76k–289k UNIQUE
IPs hitting wp-login.php, each low-and-slow). But those bots POST straight to
wp-login.php without GETting the form (~15:1 POST:GET on attacked sites). So:
hand out a `whplc` cookie on GET of the login form (set-var at request time +
http-after-response add-header — request fetches don't evaluate in the response
phase) and DENY 403 on login POSTs that lack it. Direct-POST bots are dropped
at the edge before reaching PHP; real logins are unaffected (WP login already
requires loading the page + cookies). Immediate deny, not tarpit, to avoid
connection exhaustion under a 300k-POST flood. Honors the whitelist.
Validated locally: GET /wp-login.php emits whplc; other paths don't; config OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The generic rate-limits are tuned high for media-heavy sites, so slow
credential-stuffing on wp-login.php slips under them. Add a dedicated sc1
stick-table (backend wp_bruteforce, 60s window) that counts POSTs to
wp-login.php per real client IP and tarpits once an IP exceeds 30/min.
Only login POSTs are counted (browsing + the login form GET + a legit user's
few attempts are unaffected); an offending IP can still browse, just not keep
hammering login. Honors the existing whitelist (RFC1918 / trusted_ips.list /
trusted_ips.map) and the already-resolved CF/proxy real IP. path_end also
covers subdirectory WP installs. Stops attacks at the edge before they reach
PHP/WordPress, on all edges regardless of Coraza mode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generic long-lived backend (template_override='hap_backend_longlived') for apps
whose primary path holds connections open: streaming, large up/downloads,
persistent sessions. Both primary and SSE backends tuned long-lived (no
http-server-close, http-no-delay, 6h server/tunnel/keep-alive). Differs from
hap_backend_websocket (which only long-lives the SSE variant).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HTTP/3 is config-only — the Debian haproxy package is built +QUIC via the
OpenSSL compat shim. Changes:
- hap_header.tpl: `limited-quic` (required to enable QUIC binds under the
compat layer) + self-healing `cluster-secret` for QUIC token derivation.
- hap_listener.tpl: `bind quic4@:443 ... alpn h3` in the shared frontend (so
real-IP/rate-limit/IP-block/Coraza rules apply to H3 too) + alt-svc header.
- Dockerfile/README: publish/document 443/udp; stamp image.version from VERSION.
- CI: tag :latest + :<VERSION> + :<sha> so there's a pinnable rollback target.
No 0-RTT (compat-layer limitation). Validated end-to-end on a standalone edge:
config parses, UDP/443 binds, alt-svc advertised, real curl --http3 -> HTTP/3.
Container must run with `-p 443:443/udp` + host UDP/443 open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously a Coraza block returned an empty 403 with only the
`waf-block: request` header — a legitimate site owner caught in a
false-positive had no idea what happened or how to get help.
Now:
- hap_header.tpl: every request gets a unique-id (uuid()) and that ID
is injected back into the request as X-Request-Reference for the
backend, so upstream Apache/PHP logs can correlate too.
- hap_listener.tpl: on a request-phase Coraza deny we use
`http-request return` with `lf-file` instead of `http-request deny`,
so HAProxy renders the new errors/403-waf.html page with the
request reference substituted in. The page tells the visitor a
request was blocked, displays the reference, and points site owners
to https://secure.anhonesthost.com/submitticket.php to open a ticket
rather than exposing a public email address (avoids giving
attackers a flood target).
- The waf-block header and x-request-reference header are still set
on the response so curl / monitoring clients can pick them up
without rendering HTML.
- Response-phase deny stays as the bare 403 — outbound blocks are
rare in our config and an HTML body could land mid-stream.
Errorfile lives at /haproxy/errors/403-waf.html (NOT under
/etc/haproxy/, because that path is a named volume in deployed
containers and would shadow baked-in files on existing deployments).
Support workflow: visitor quotes the reference → support greps
/var/log/haproxy.log for the uuid → gets timestamp + client IP +
Host + URI → greps /var/log/coraza/audit.log for the matching
transaction → reads the rule_id that fired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous design used a separate whp-suspended container (nginx:alpine
serving a static 503 page) reachable via a dedicated bk_suspended backend.
That was over-engineered — haproxy-manager-base already ships a default-app
Flask server on :8080 that serves /default-page and /blocked-ip via
path-rewrite ACLs. Mirroring that pattern lets the suspension page live
in the SAME container, no extra image to build, no extra container to
run/health-monitor.
Changes:
- Add /suspended Flask route on default_app returning 503 + suspended_page.html
- Add templates/suspended_page.html (dark-themed 503 page)
- hap_listener.tpl: 'http-request set-path /suspended' + 'use_backend
default-backend' when host is in suspended_domains.list (same pattern
as is_blocked_ip)
- Rename env var from HAPROXY_SUSPENSION_BACKEND (a target hostport) to
HAPROXY_SUSPENSION_ENABLED (a bool); accepts 1/true/yes/on (case-insensitive)
- Remove hap_suspended_backend.tpl and its rendering in generate_config
Non-WHP deployments (env var unset) see byte-identical haproxy.cfg as before
(verified via jinja2 render diff).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If the upstream container isn't up when haproxy-manager starts (e.g. when
haproxy is recreated before whp-suspended), the default `init-addr libc` mode
makes haproxy refuse to start — taking down the whole proxy. Switched to
`init-addr last,none` (use last known address, fall back to 0.0.0.0 = DOWN)
and added `resolvers docker_dns` (defined in hap_header.tpl) so the real IP
is picked up once DNS becomes resolvable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a new env var HAPROXY_SUSPENSION_BACKEND (default unset). When set
(e.g. "whp-suspended:80"), generate_config() renders:
- A bk_suspended backend pointing at the configured upstream
- An ACL `acl is_suspended_domain hdr(host),lower -f /etc/haproxy/suspended_domains.list`
+ `use_backend bk_suspended if is_suspended_domain` in the frontend,
sitting after IP-blocking and before any per-domain routing
- An empty /etc/haproxy/suspended_domains.list if missing (haproxy refuses
to start with -f pointing at a non-existent file)
External tooling (e.g. WHP's site_disable.php) maintains the list via
`docker cp` and HUP-reloads the container.
Non-WHP deployments (home networks, standalone use) leave the env var
unset and see byte-identical haproxy.cfg output. Same opt-in shape as
the existing HAPROXY_CORAZA_SPOE_BACKEND integration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fixes that complete the SPOE enforcement path:
1. Listener was sending requests to Coraza for inspection but never reading
the result. Coraza-SPOA sets var(txn.coraza.action) to "deny" / "drop"
/ "redirect" when a rule with that disruptive action fires; HAProxy
needs explicit rules that READ the variable and apply the action.
Without them, the audit log shows "Access denied" but the request
still gets HTTP 200 (verified on staging: sqlmap/JNDI/shellinj all
detected, all returned 200).
Added the standard six rules from upstream's example/haproxy/haproxy.cfg
covering http-request + http-response phases for each of deny/drop/
redirect. Same set the upstream Coraza-SPOA docs recommend.
Intentionally did NOT add the upstream's fail-CLOSED rule
`http-request deny deny_status 500 if { var(txn.coraza.error) -m int gt 0 }`
— for a hosting platform we want fail-open. Documented inline.
2. Backend health check switched from plain TCP `check` to `option
spop-check`. The spop-check actually negotiates a SPOE session against
the agent, so HAProxy detects a half-broken SPOA that's listening on
:9000 but failing protocol handshakes. Plain `check` would miss that.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three real bugs in the SPOE config caught when HAProxy validated the
generated file:
1. spoe-agent must declare `groups` not `messages`. The `messages` form
doesn't make the message reachable via `send-spoe-group`; HAProxy
complained:
unable to find SPOE group 'coraza-check' into SPOE engine 'coraza'
2. send-spoe-group references a spoe-GROUP name, which needs its own
block. Added `spoe-group coraza-req { messages coraza-req }` as
the indirection layer.
3. Arg names + ORDER are required to match what Coraza-SPOA parses
positionally. My version had `dest-ip`/`dest-port`; upstream's
example/haproxy/coraza.cfg (v0.7.1) uses `dst-ip`/`dst-port`.
Renamed and reordered to match upstream verbatim, including the
`app=str(haproxy)` literal that matches our config.yaml application
name.
Also corrected misleading comment about `set-on-error continue`: that
option actually sets a variable on error; the fail-open behavior comes
from us deliberately NOT adding a `http-request deny if errored` rule
in the frontend. Renamed the variable to `error` (matching upstream)
and updated comments to be accurate.
Listener template's send-spoe-group action updated to reference the
new group name `coraza-req`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two HAProxy parse errors caught in staging functional test:
1. coraza-spoe.cfg:39 'args': missing fetch method
The args directive had backslash line continuations. HAProxy doesn't
support those in SPOE configs — args must be one physical line.
Collapsed to a single line.
2. coraza-spoe.cfg:50 Missing LF on last line
Same trailing-LF issue we hit on haproxy.cfg one commit ago. The
Jinja2 template ends with content rather than a newline, and write()
doesn't add one. Belt-and-suspenders: explicitly append '\n' before
writing if not already there.
After this commit HAProxy validates the generated config cleanly. Will
verify on staging now (combined SPOE injection + fail-open + active
attack-detection tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Default Jinja2 {% if %}{% endif %} block syntax leaves a trailing newline
even when the conditional doesn't render. Staging verification of PR 2
showed the resulting haproxy.cfg differed from the pre-PR2 version by
exactly 1 blank line — semantically identical but not byte-identical,
which violates the design promise that haproxy-manager-base's default
output stays unchanged for home/standalone deployments.
Use {%- if -%}/{%- endif %} (the whitespace-stripping variants) so the
block contributes zero bytes when coraza_spoe_backend is unset.
Verified locally: without env var = 55 lines, ends cleanly on the
is_blocked_ip rule. With env var = 62 lines, +7 for the SPOE block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the plumbing that lets haproxy-manager talk to the coraza-spoa sidecar
added in PR 1, while keeping the default behavior bit-identical for any
deployment that doesn't set the new env var (the home network / standalone
use cases).
Single gate: HAPROXY_CORAZA_SPOE_BACKEND env var on the haproxy-manager
container. Unset (default) = generate_config() renders zero SPOE-related
output. Set (e.g. "coraza-spoa:9000") = three things happen at config
generation time:
1. hap_listener.tpl injects 5 lines at the end of the frontend block:
filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
http-request send-spoe-group coraza coraza-check
...placed AFTER rate-limit and IP-block guards so we don't waste WAF
calls on requests we were going to drop anyway.
2. A new TCP backend (hap_coraza_spoa_backend.tpl) is appended:
backend coraza-spoa-backend
mode tcp
server coraza-spoa <env-var-target> check ...
3. The SPOE engine config (hap_coraza_spoe_engine.tpl) is rendered and
written to /etc/haproxy/coraza-spoe.cfg, defining the spoe-agent
"coraza" + spoe-message "coraza-check". This sets:
- option set-on-error continue (FAIL-OPEN if SPOA is unreachable)
- timeout processing 100ms (per-request inspection budget)
- app=str(haproxy) (matches sidecar's application name)
Verification (template render only, before staging deploy):
- hap_listener.tpl with no env var: 55 lines, zero SPOE references
- hap_listener.tpl with env var: 62 lines, filter + send-spoe-group present
- Engine cfg + backend block render with correct agent_target substitution
Next: PR 3 wires this into WHP (sidecar deploy via container-manager.sh
extension, server-settings UI for on/off, AI Monitor source for the audit
log). Staging verification of PR 1 + PR 2 together happens after PR 3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The resolvers section was inserted inside the global section, causing
HAProxy to parse global directives (pidfile, maxconn, etc.) as
resolver keywords. Moved resolvers to its own top-level section
between global and defaults where HAProxy expects it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Docker containers restart, they can get new IPs on the bridge
network. HAProxy caches DNS at config load time, so stale IPs cause
503s until config is regenerated.
Added a 'docker_dns' resolvers section pointing to Docker's embedded
DNS (127.0.0.11) with 10s hold time. Backend servers now use
'resolvers docker_dns init-addr last,libc,none' so HAProxy:
- Re-resolves container names every 10 seconds
- Falls back to last known IP if DNS is temporarily unavailable
- Starts even if a backend can't be resolved yet (init-addr none)
This eliminates 503s from container restarts, scaling, and recreation
without requiring a HAProxy config regeneration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generous thresholds that accommodate sites with many images/assets
while still catching obvious automated floods:
- Request rate: tarpit at 300 req/s, block at 500 req/s
- Connection rate: 500/10s
- Concurrent connections: 500
- Error rate: 100/30s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous thresholds (200/500 req/10s) were too aggressive — WordPress
login pages with their CSS/JS/image assets can easily burst 30-50
requests per page load, triggering tarpits and blocks on legitimate
users.
New thresholds:
- Request rate: tarpit at 1000/10s (100 req/s), block at 2000/10s (200 req/s)
- Connection rate: 300/10s (was 150)
- Concurrent connections: 200 (was 100)
- Error rate: 50/30s (was 20)
These still catch real floods and scanners while giving normal web
traffic plenty of headroom.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds trusted_ips.list and trusted_ips.map files that exempt specific
IPs from all rate limiting rules. Supports both direct source IP
matching (is_trusted_ip) and proxy-header real IP matching
(is_whitelisted). Files are baked into the image and can be updated
by editing and rebuilding.
Adds phone system IP 172.116.197.166 to the whitelist.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gives more headroom for customers with code that makes frequent
callbacks to itself, while still catching connection floods.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Activate HAProxy's built-in attack prevention to stop floods that cause
the container to become unresponsive:
- Stick table tracks per-IP: conn_cur, conn_rate, http_req_rate, http_err_rate
- Rate limit rules: deny at 50 req/s, tarpit at 20 req/s, connection
rate limit at 60/10s, concurrent connection cap at 100, error rate
tarpit at 20 errors/30s
- Harden timeouts: http-request 300s→30s, connect 120s→10s, client
10m→5m, keep-alive 120s→30s
- HTTP/2 Rapid Reset protection (CVE-2023-44487): stream and glitch limits
- Stats frontend on localhost:8404 for monitoring
- HEALTHCHECK now validates both port 80 (HAProxy) and 8000 (API)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Captures the Host header in HAProxy httplog output so high-connection
alerts can be correlated to specific domains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Support wildcard domains (*.domain.tld) in HAProxy config generation
with exact-match ACLs prioritized over wildcard ACLs. Add DNS-01
challenge endpoints that coordinate with certbot via auth/cleanup
hook scripts for wildcard SSL certificate issuance.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Changes:
- Detect SSE via Accept header (text/event-stream) or ?action=stream parameter
- Disable http-server-close to allow long-lived SSE connections
- Enable http-no-delay for immediate event delivery
- Set 1-hour timeouts for SSE support (also fine for normal requests)
- Force Connection: keep-alive for detected SSE requests
Benefits:
- SSE now works automatically without special backend configuration
- Fixes transcription server display disconnection issues
- Normal HTTP requests still work perfectly
- No need for separate SSE-specific backends
Fixes: Server-Sent Events timing out through HAProxy
- Update map file format to include value (IP/CIDR 1)
- Fix HAProxy template to use map_ip() for CIDR support
- Update runtime map commands to include value
- Document CIDR range blocking in API documentation
- Support blocking entire network ranges (e.g., 192.168.1.0/24)
This allows blocking compromised ISP ranges and other large-scale attacks.
This commit simplifies the HAProxy configuration by removing automatic
threat detection and blocking rules while preserving essential functionality.
Changes:
- Removed all automatic ACL-based security rules (SQL injection detection,
scanner detection, rate limiting, brute force protection, etc.)
- Removed complex stick-table tracking with 15 GPC counters
- Removed graduated threat response system (tarpit, deny based on threat scores)
- Removed HTTP/2 security tuning parameters specific to threat detection
- Commented out IP header forwarding in hap_backend_basic.tpl
Preserved functionality:
- Real client IP detection from proxy headers (CF-Connecting-IP, X-Real-IP,
X-Forwarded-For) with proper fallback to source IP
- Manual IP blocking via map file (/etc/haproxy/blocked_ips.map)
- Runtime map updates for immediate blocking without reload
- Backend IP forwarding capabilities (available in hap_backend.tpl)
The configuration now focuses on manual IP blocking only, which can be
managed through the API endpoints (/api/blocked-ips).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove reference to non-existent security_blacklist table
- Use single table tracking with consolidated array-based GPC system
- Remove res.hdr(X-Threat-Level) from log-format as response headers not available in request phase
- Maintains threat intelligence logging with available request-phase data
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace compound ACL xmlrpc_abuse with separate conditions
- Use xmlrpc_rate_abuse for rate detection and combine with is_xmlrpc in http-request rule
- Prevents ACL-to-ACL reference which is not supported in HAProxy 3.0.11
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add -m int matcher for all var(txn.threat_score) comparisons
- Fix set-header, tarpit, deny, and set-log-level conditions
- Ensures proper variable type matching for HAProxy 3.0.11
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix tune.h2.fe-max-total-streams parameter name in global config
- Fix stick-table multiline syntax by removing line continuations
- Replace sc0_get_gpc with sc_get_gpc for proper 3.0.11 syntax
- Replace sc-set-gpc with sc-set-gpt for value assignments
- Update ACL definitions to use correct GPT fetch methods
- Simplify threat scoring to avoid unsupported add-var operations
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
shadowdaoandClaude Opus 5