Two stacked fixes for the edge that terminates every customer's HTTPS and
routes every customer's traffic. They ship together because deploying
either means recreating haproxy-manager on every host.
1. Config rollback was a no-op. generate_config() wrote the new config
BEFORE create_backup() copied it, so restore_backup() restored the
identical broken file while logging 'Backups restored successfully'.
The backup is now taken before the first write, refuses to promote a
config haproxy already rejects, and covers blocked_ips.map and
coraza-spoe.cfg as one restorable set. blocked_ips.map is now written
atomically too - haproxy -c parses it, so a truncated map is a fatal
config.
2. Certificate publishing truncated the live PEM. Six sites opened the
bundle HAProxy is serving in truncate mode, then read the source; any
failure between left a key-less or partial PEM, unrecoverable by config
rollback. The shell path was worse: with a zero-length source key, cat
exits 0, so renew-certificates.sh logged 'Updated certificate', reported
'0 failed', and reloaded HAProxy onto a key-less bundle (measured:
2912 -> 1208 bytes, key gone, exit 0). Publishing is now assemble ->
validate -> back up -> os.replace(), staged in a SIBLING directory
because HAProxy loads the crt path as a directory and would parse a
stray .tmp. certbot delete no longer runs before the replacement has
validated and loaded.
Also fixed: the shipped test suite stripped the exec bit from twelve system
binaries - chmod included - when run as root inside the container, because
os.chmod follows symlinks. It reported 32 OK on a workstation and 18
failures in the image it ships to.
Behaviour changes: renewals exit non-zero when any domain fails to publish
(previously always 0, and a test codified that); a host without openssl
refuses to publish rather than trusting structural checks alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gate findings on fix/cert-write-safety. One blocker, one latent truncation
path, a false premise in a load-bearing comment, and a set of tests that were
passing without testing anything.
THE BLOCKER - scripts/test-cert-scripts.py bricked the production container
------------------------------------------------------------------------
The openssl-availability tests build a stripped PATH out of SYMLINKS to real
system binaries (cat, grep, mktemp, mv, cp, rm, mkdir, basename, dirname, find,
date, chmod). _cleanup_tmp() then walked the temp tree calling os.chmod(p,
0o600) - and os.chmod FOLLOWS SYMLINKS. Run once as root in the real image,
which is where this file ships (COPY scripts /haproxy/scripts) and where an
operator would most plausibly run it after deploying a cert fix, it stripped
the exec bit off twelve core binaries INCLUDING chmod itself, so it could not
be undone from inside the container:
/haproxy/scripts/cert-publish-lib.sh: line 109: /usr/bin/grep: Permission denied
bash: /usr/bin/chmod: Permission denied
Certificate publishing stayed dead until the container was recreated. It was
invisible on a workstation because an unprivileged chmod of a root-owned file
fails EPERM straight into `except OSError: pass` - which is also why the
advertised "32 tests pass" was only ever true off-container. In the image the
shipped file measured FAILED (failures=22, skipped=1). Cleanup now skips
symlinks; the suites are green in the image and the binaries survive.
S1 - the shell half had no same-filesystem guard
------------------------------------------------
The header claimed a cross-device mv would "FAIL LOUDLY and leave the live pem
alone". GNU mv does the opposite: across filesystems it copies, so it opens and
truncates the DESTINATION and only then discovers it cannot finish - measured
as a 204800-byte partial live.pem, sentinel gone, before mv reported ENOSPC.
cert_publish() now compares stat -Lc %d of the staging and certs dirs before
writing anything, mirroring the st_dev check the Python half already had, and
the comment says what mv actually does. Latent today (both dirs share a device
on all five hosts) but both are env-overridable.
openssl is present - correct the premise, make the check mandatory
------------------------------------------------------------------
Both halves justified a fail-open with "the image does not necessarily install
the openssl CLI". It does: openssl 3.5.6 in the running container, pulled in by
ca-certificates which certbot needs, and generate_self_signed_cert() already
shells to `openssl req` with check=True during setup. The 'unavailable' branch
never fired, so the pairing check has always run - and that, not the stated
reasoning, is what made the fail-open harmless. Structural validation alone is
weak: a bundle of EMPTY pem blocks passes every structural rule and is caught
only by openssl. The check is now mandatory in both halves and a missing binary
is a loud refusal. No `cryptography` fallback: the app runs on
/usr/local/bin/python3 (3.12) where it is not importable - it belongs to
Debian's /usr/bin/python3 - and reaching for that would be a second unverified
premise.
Smaller items
-------------
* cert_bundle_valid() read the file six times; a concurrent swap between two of
them made openssl x509 and openssl pkey judge different files and log a bogus
"private key does not match the certificate" into the monitored error log. It
now reads one snapshot and feeds openssl from it on stdin.
* except OSError -> except (OSError, UnicodeDecodeError): a BINARY-corrupt live
pem made backup_existing_pem() raise out of publish_pem_bundle() entirely, so
the republish that would have healed the host was the one thing that could
not run. The shell half recovers fine.
* stat -c %a on a symlinked live pem reports the LINK's 0777 and produced a
world-writable private key in the crt directory; now stat -Lc.
* The staging reaper's '*.??????' glob matched mktemp names but not the Python
side's '<name>.<random>.tmp', so those leaked forever. Matches both now.
* renew-certificates.sh and sync-certificates.sh exited 0 even when every
domain failed to publish, so "0 updated, 12 failed" looked identical to a
clean run to cron, to host-renew-certificates.sh (which branches on it) and
to monitoring - a host could silently stop publishing renewals until the
certificates expired. They now exit 1 if any domain failed, still after
publishing the ones that worked.
Test-quality
------------
Four TestCertPublishLibrary tests passed with cert-publish-lib.sh DELETED -
they asserted only rc != 0, and `command not found` is 127. All four now assert
the rejection REASON via assert_rejected(), and setUp() fails if the library is
missing. test_missing_openssl_still_rejects... is replaced by
test_empty_pem_blocks_are_rejected, which pins the case that makes the pairing
check necessary.
Also: the staging-containment test used startswith(certs + os.sep), so
cert_staging_dir() returning the crt directory ITSELF - the exact hazard -
still passed; test_successful_renewal_still_publishes built its "renewed" cert
with a no-op .replace() and could not tell a renewal that published nothing;
test_no_temp_file_survives_a_failed_publish failed before the staging dir
existed and asserted [] == []; FIX_ONLY was skipUnless(hasattr(hm,
'publish_pem_bundle')), so renaming that function turned 10 of 17 tests into
skips while the run still printed OK. Each is fixed and each fix is
mutation-proved: the mutation that the old assertion waved through now fails.
File mode is pinned in both suites (it was pinned nowhere), and the rename
failure is injected with a stub mv instead of chmod 0500, which root ignored -
so that test no longer skips itself precisely where it matters.
Verification
------------
IN THE BUILT IMAGE, as root (the acceptance bar):
scripts/test-cert-scripts.py 38 tests, OK, 0 skipped
scripts/test-cert-write-safety.py 22 tests, OK, 0 skipped
scripts/test-config-rollback.py 17 tests, OK (neighbour, unchanged)
Workstation: 38/38 OK for the shell suite; the Python suite needs flask.
Against the pre-fix tree (main): shell 38 failures; python failures=4, errors=1,
skipped=15 - the FIX_ONLY skips are the 14 fix-only tests plus the API guard.
The old python suite against the pre-fix tree measures skipped=10, confirming
the gate's count.
32 mutation checks, all behaving as intended: every fix breaks a test when
reverted, and every rewritten test fails under the mutation its predecessor
passed. py_compile clean, bash -n clean, shellcheck clean, no new pyflakes
warnings (same 4 pre-existing).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every path that refreshed a combined certificate did
with open(combined_path, 'w') as combined: # TRUNCATES
subprocess.run(['cat', cert, key], stdout=combined) # rc ignored
where combined_path is the bundle HAProxy is currently serving. The live file
was emptied before any source material had been read and the cat status was
never checked, so an unreadable source, a zero-length privkey, a full disk or a
killed container left a truncated or key-less PEM in place. HAProxy loads
/etc/haproxy/certs as a directory, so one unusable file there fails the whole
ssl bind - HTTPS down for every site on the host, and unlike a bad haproxy.cfg
it is not recoverable by config rollback.
The bundle endpoint compounded it: superseded .pem files were unlinked and
their lineages `certbot delete`d before anything had checked the replacement,
destroying both copies of a working certificate. Recovery there means fresh,
rate-limited ACME orders.
Bundles are now assembled in a staging directory beside the crt directory (never
inside it - HAProxy would try to load a temp file), validated there, and swapped
in with os.replace(). Validation is mandatory structural checks in pure Python
plus a best-effort openssl key/leaf pairing check; a missing openssl warns
loudly and does not silently pass. The previous bundle is copied to
/etc/haproxy/cert-backups first. Superseded certs are moved aside rather than
deleted, and `certbot delete` runs only after HAProxy has reloaded onto the
replacement. The same guarantees are implemented for the cron/renewal shell
path in scripts/cert-publish-lib.sh, which also gates the reload on `haproxy -c`.
Reuses write_config_atomically() from the config-rollback fix (extended with
staging_dir/validate) rather than adding a second atomic writer.
Also heals a zero-byte QUIC cluster-secret file, which previously made
get_or_create_cluster_secret() return '' forever.
Tests: scripts/test-cert-write-safety.py (17) and scripts/test-cert-scripts.py
(32) in the existing stdlib-unittest, stub-binary convention. Both bug classes
reproduce against the pre-fix tree via HAPROXY_MANAGER_DIR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to 9d16151 (take the rollback backup BEFORE writing the new config).
Review findings, none of them blockers, plus two corrections to the record.
blocked_ips.map was still written with a plain open(path,'w'). `haproxy -c`
LOADS that file - hap_listener.tpl matches on
map_ip(/etc/haproxy/blocked_ips.map,0) - and a half-written final line is a
FATAL config error, not one dropped entry. Verified against HAProxy 2.8: a
truncated "198.51.10" gives "is not a valid IPv4 or IPv6 address at line 2 of
file ..." and the whole configuration is rejected. That is precisely the
failure shape the backup-ordering fix exists to prevent, on a different file,
reachable from all five /api/blocked-ips routes. It now goes through the same
write_config_atomically() as haproxy.cfg and coraza-spoe.cfg. (A truncation
that happens to land on a line boundary is not fatal - it silently drops
blocks - which is its own reason to write the file atomically.)
The previous commit message claimed the fast path adds "no `haproxy -c`
latency to customer-facing API calls". That was measured on an idle box and is
false in this fleet's normal pattern: update_blocked_ips_map() is called from
those five routes OUTSIDE generate_config(), so the live map drifts from its
backup and the NEXT config change misses create_backup()'s fast path. Measured
`haproxy -c` runs per domain add: 1 steady state, 2 after an IP block. This
fleet blocks IPs automatically, so the 2x recurred on the customer-facing call
indefinitely. update_blocked_ips_map() now promotes the map it just wrote to
its backup, restoring the steady state - guarded twice: nothing is promoted
unless a config backup set already exists (never fabricate a rollback target),
and not unless the map parses as IPs/CIDRs, so promotion cannot leave a
"rollback target" HAProxy would refuse to load. generate_config() passes
promote_backup=False: it took the snapshot moments earlier and the map is part
of the not-yet-validated change, so refreshing the backup there would be the
original bug again. The remaining 2 is the first generation after this upgrade
(coraza-spoe.cfg has no backup yet); that is once per host, by construction.
Two claims in 9d16151's message are wrong and are corrected here rather than by
rewriting a pushed commit:
* "12 of the 17 fail against the previous code" - it is 14 of 17 (6 failures +
8 errors). 12 was measured before two fast-path tests were added and never
re-measured.
* "a missing haproxy binary is not read as a bad config" - true of
create_backup() only. validate_haproxy_config() collapses both 'invalid' and
'unavailable' to False, so in the reload path a missing validator still
triggers a full rollback labelled "Config validation failed". The behaviour
is right (without a working validator we cannot claim the new config is
safe, and the reload path is where guessing wrong takes the edge down); the
sentence was broader than the code. Now documented on the function.
Tests: 26 (was 17), all green; 22 fail against main. New coverage closes the
review's mutation survivors:
* the "Refusing to regenerate config" guard, previously entirely uncovered;
* the invalid/unavailable split, previously zero coverage - both the verdict
and the consequence create_backup() draws from it;
* the byte-compare loop, with a same-size-different-content config, which is
the exact case the "not filecmp.cmp" rationale exists for. Building that
fixture found a bug in the test itself: sizing the drifted config with
len(str) instead of bytes made it pass for the wrong reason, because the
rendered config contains non-ASCII.
* test_failed_write_leaves_the_previous_file_intact was vacuous: its bare
assertRaises(Exception) swallowed the AttributeError from
write_config_atomically not existing, so it passed against main and would
have kept passing if the function were deleted. Narrowed to TypeError -
proven by deleting the function and watching it go red.
* test_backup_set_covers_every_file_generate_config_writes restated the three
files it expected, so it could never have noticed a fourth. It now derives
the set - observed on disk for the branches the fixture can execute, read
out of generate_config()'s source for the env-gated one that writes a
hardcoded /etc/haproxy path - and requires anything unbacked-up to be on a
documented exclusion list (suspended_domains.list, cluster-secret, each with
its reason). Proven by adding a fourth written file and watching it fail.
Every change above was mutation-proved: 11 mutations, 0 survivors, each
reddening only the tests that cover it. Two review items were confirmed
untestable in-process and are deliberately skipped: the fsync (M11) and a
log-line-only mutation (M15).
Left alone deliberately, all pre-existing on main and unchanged here: the
outer `except Exception` in reload_haproxy_safely() does not roll back (narrow
window, now commented at the site); stale .tmp files after SIGKILL are never
swept (verified inert - nothing globs /etc/haproxy, and the only
directory-wide load is `crt /etc/haproxy/certs`, which nothing here writes to);
a partial backup-copy failure can leave a mixed-vintage backup set (very
narrow, and generate_config() correctly refuses to write).
VERSION stays 2026.08.1. NOTE: fix/cert-write-safety, which is stacked on this
branch, carries the same 2026.08.1. If both land on main as separate commits,
CI pushes :2026.08.1 twice with different content. Either that branch moves to
2026.08.2 or the two land as a single merge - not decided here.
No template, QUIC or HTTP/3 changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
generate_config() wrote /etc/haproxy/haproxy.cfg and only then called
reload_haproxy_safely(), which called create_backup(). The "backup" was
therefore a copy of the config that had just been written, so on a validation
failure restore_backup() restored the identical broken bytes: the advertised
rollback was a no-op and a fatal haproxy.cfg stayed on disk, where
start_haproxy() refuses to launch. Same shape as the June 2026 incident where
a missing template produced a fatal config and took an edge down.
Reproduced end to end before the fix (invalid config generated -> "Backups
created successfully" -> "Backups restored successfully" -> haproxy.cfg on
disk still invalid, `haproxy -c` rc=1).
Changes:
* create_backup() is now called by generate_config() BEFORE the first write,
which also covers blocked_ips.map (rewritten early in generate_config) and
coraza-spoe.cfg - both previously written before the backup and, for the
SPOE file, never backed up at all even though `haproxy -c` parses it.
* create_backup() refuses to promote a config HAProxy already rejects, so a
broken file on disk cannot overwrite a known-good backup ("rollback" must
not mean "restore a different broken config"). It returns (ok, status) so
the caller knows whether a rollback target exists.
* promote_current_config_to_backup() records the config as known-good only
after it has validated AND loaded, so a box whose first generation succeeded
has a rollback target immediately, and a config that never loaded is never
promoted.
* restore_backup() returns (restored, message) and distinguishes "no backup
available" from "restored". Every caller now surfaces the difference; a
failed rollback is logged CRITICAL and reported as ROLLBACK FAILED in the
API error message instead of silently looking like a successful recovery.
* reload_haproxy_safely(backup_status=...) no longer takes its own backup - it
runs after the write, where a backup is meaningless. Called without a status
it logs the contract violation rather than overwriting a good backup.
* validate_config_file() separates "config is invalid" from "validator could
not run" so a missing haproxy binary is not read as a bad config.
* Config writes are atomic (temp file + fsync + os.replace, mode preserved);
a truncated haproxy.cfg is as fatal as an invalid one. Removes the dead
temp_config_path variable whose comment claimed this already happened.
* Fast path: if the live config set is already byte-identical to the backup
(the normal case after a successful reload), skip the re-validation and the
copy, so this adds no `haproxy -c` latency to customer-facing API calls.
Tests: scripts/test-config-rollback.py - 17 self-contained stdlib-unittest
tests, no new dependencies (the repo has no Python test framework; the
existing scripts/test-*.sh are curl integration scripts). A stub `haproxy`
binary stands in for the validator. 12 of the 17 fail against the previous
code; every assertion was mutation-proven (9 mutations, each reddening only
the tests that cover it).
No template, QUIC or HTTP/3 changes.
Adds a frontend virtual patch denying /wp-json/batch/v1 and the
?rest_route=/batch/v1 fallback (including the %2F-encoded spelling) for
requests without a wordpress_logged_in_* cookie.
wp2shell chains CVE-2026-60137 (core SQL injection) with CVE-2026-63030
(REST batch-route confusion) into unauthenticated RCE on WP 6.9.0-6.9.4
and 7.0.0-7.0.1. Exploits are public and were used against this fleet on
2026-07-19/20; one site was compromised through this path, including a
re-injection of wp-includes/plugin.php nine minutes after it was patched.
Anonymous-only by design: batch/v1 is used legitimately by the block
editor for multi-entity saves, so a blanket deny would break wp-admin.
Placed ahead of the SPOE handoff so blocked requests never cost a WAF
round-trip, and it works regardless of Coraza mode (whp02 and sdbees run
detect_only, where a Coraza rule would log but not block).
This is a virtual patch, not a fix. It removes reachability only, and
stays until every site is confirmed on a fixed release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The management API wedged on whp01 2026-07-07: every panel call to the
manager (config regenerate during a WHP site update, SSL, even /health)
timed out at 30s while customer sites stayed up. Root cause: all four
gunicorn gthread worker threads were permanently blocked in socket reads
inside untimed subprocess.run() calls (certbot ACME / socat reloads). A
stalled external command holds its worker thread forever; gunicorn
--timeout can't rescue it (gthread only kills a worker whose main thread
stops heart-beating, and ours kept polling). Stalled calls accumulated
until the 4-thread pool was exhausted and the whole API went dark.
- Wrap subprocess.run with a default timeout (HAPROXY_MGR_SUBPROCESS_TIMEOUT,
180s) so every external command is bounded and releases its thread on
expiry via the existing per-endpoint try/except. Bounding by default
covers all ~30 call sites and any future one.
- certbot renew keeps an explicit 900s timeout (walks every lineage).
- API_WORKERS default 1 -> 2: a single worker made a thread-pool wedge a
total outage; a second worker keeps the API answering while one recycles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
generate_config emits "Skipping domain <host> - no backend name" on every run
for domains registered without a proxy backend — most commonly the panel's own
hostname (whpNN.cloud-hosting.io), which lives in the DB only for certificate
management and intentionally has no backend. Logging it at WARNING tripped the
WHP AI log monitor as a recurring error and prompted a bogus "restart
haproxy-manager" remediation. It's expected, benign, and recurs by design.
Log it at INFO instead (consistent with the sibling per-domain "Added ACL for
domain" INFO lines) with a clearer message ("no proxy backend
(cert/management-only)"). Verified against the WHP monitor's ErrorClassifier:
the old WARNING line classified as non_critical (captured); the new INFO line
classifies as None (skipped) — so it no longer shows up in reports, while
remaining visible in container logs for manual routing debugging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
haproxy runs as a background child of PID 1 (gunicorn) with nothing
watching it after init. If the haproxy master dies mid-life (observed
2026-07-01 on whp01: SIGABRT -> exit 134, reaped by gunicorn and logged
as "Worker (pid:22) exited"), the container stays "up", Docker's
--restart never fires, and haproxy is down until the external host
watchdog full-restarts the whole container minutes later (dropping every
connection).
Add an in-container supervisor loop in start-up.sh (Phase 1.5) that runs
scripts/ensure_haproxy.py every HAPROXY_SUPERVISOR_INTERVAL (default 15s).
ensure_haproxy.py calls the existing, idempotent start_haproxy() only when
haproxy isn't running (psutil guard), reviving it in place within one
interval with no container restart. Same entrypoint-supervision pattern
shipped for cac-litespeed.
Validated locally: killing haproxy -> revived with new PIDs in ~one
interval, container stayed healthy, no spurious restarts while healthy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-IP throttle can't see distributed attacks (observed 76k–289k UNIQUE
IPs hitting wp-login.php, each low-and-slow). But those bots POST straight to
wp-login.php without GETting the form (~15:1 POST:GET on attacked sites). So:
hand out a `whplc` cookie on GET of the login form (set-var at request time +
http-after-response add-header — request fetches don't evaluate in the response
phase) and DENY 403 on login POSTs that lack it. Direct-POST bots are dropped
at the edge before reaching PHP; real logins are unaffected (WP login already
requires loading the page + cookies). Immediate deny, not tarpit, to avoid
connection exhaustion under a 300k-POST flood. Honors the whitelist.
Validated locally: GET /wp-login.php emits whplc; other paths don't; config OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The generic rate-limits are tuned high for media-heavy sites, so slow
credential-stuffing on wp-login.php slips under them. Add a dedicated sc1
stick-table (backend wp_bruteforce, 60s window) that counts POSTs to
wp-login.php per real client IP and tarpits once an IP exceeds 30/min.
Only login POSTs are counted (browsing + the login form GET + a legit user's
few attempts are unaffected); an offending IP can still browse, just not keep
hammering login. Honors the existing whitelist (RFC1918 / trusted_ips.list /
trusted_ips.map) and the already-resolved CF/proxy real IP. path_end also
covers subdirectory WP installs. Stops attacks at the edge before they reach
PHP/WordPress, on all edges regardless of Coraza mode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generic long-lived backend (template_override='hap_backend_longlived') for apps
whose primary path holds connections open: streaming, large up/downloads,
persistent sessions. Both primary and SSE backends tuned long-lived (no
http-server-close, http-no-delay, 6h server/tunnel/keep-alive). Differs from
hap_backend_websocket (which only long-lives the SSE variant).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HTTP/3 is config-only — the Debian haproxy package is built +QUIC via the
OpenSSL compat shim. Changes:
- hap_header.tpl: `limited-quic` (required to enable QUIC binds under the
compat layer) + self-healing `cluster-secret` for QUIC token derivation.
- hap_listener.tpl: `bind quic4@:443 ... alpn h3` in the shared frontend (so
real-IP/rate-limit/IP-block/Coraza rules apply to H3 too) + alt-svc header.
- Dockerfile/README: publish/document 443/udp; stamp image.version from VERSION.
- CI: tag :latest + :<VERSION> + :<sha> so there's a pinnable rollback target.
No 0-RTT (compat-layer limitation). Validated end-to-end on a standalone edge:
config parses, UDP/443 binds, alt-svc advertised, real curl --http3 -> HTTP/3.
Container must run with `-p 443:443/udp` + host UDP/443 open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds (Dockerfile) and updates (coraza-spoa/Dockerfile) the OCI
image.source label to point at github.com/shadowdao/haproxy-manager-base.
ghcr.io auto-links a package to a GitHub repo when this label resolves
to a github.com URL whose owner+name match the package's owner — that
makes the published packages show up on the GitHub repo sidebar and
inherit its collaborator settings.
Gitea's registry ignores image.source, so changing the value away from
the previous Gitea URL costs nothing on that side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- trusted_ips.{list,map}: replace home IP with 127.0.0.1 + usage notes
- skill: resolve deploy host from gitignored target-host.local, ask if unset
(no hardcoded server FQDN); customer host in WAF test -> <live-vhost>
- README / coraza README: registry FQDN in run examples -> placeholder
- 403 block page: drop hardcoded support link -> contact provider support
- CLAUDE.md: note whitelist files ship without real IPs
- .gitignore: ignore target-host.local and *.local
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a second registry login + tag to both build-push workflows so each
build publishes to ghcr.io alongside the in-house Gitea registry. Single
build, two destinations — docker/build-push-action handles the multi-tag
push in one step.
Requires Gitea Actions secret GHCR_TOKEN (a classic PAT with
write:packages on the shadowdao user).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Procedural discipline for shipping haproxy-manager-base changes.
The flow differs from WHP's (Gitea Actions auto-build vs.
build-release.sh, docker pull + recreate vs. update.sh) and has
its own foot-guns worth codifying:
- /etc/haproxy is a named volume → baked-in image files under that
path are shadowed on existing deployments; use /haproxy/ instead
- HAProxy lf-file expansion eats single % → literal CSS percentages
must be doubled (100%%)
- WAF-block synthetic test ACL must be injected AFTER send-spoe-group
or the SPOE call overwrites the forced action
- coraza-spoa is distroless (no sh); peek inside with docker create
+ docker cp rather than docker exec sh
Both build paths (build-push.yaml for haproxy-manager-base, build-
push-coraza.yaml for coraza-spoa) are surfaced so a contributor
knows which CI run to watch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end test of the 403 page showed CSS `100%` rendering as `100`
and gradient stops `0%, 100%` rendering as `0, 100` — HAProxy's
`lf-file` directive runs log-format expansion over the file content,
and `%` is the format-escape character. Single `%` is consumed by
the expander.
Doubled every literal CSS percentage (`100%%`, `0%%`, etc.) so HAProxy
emits a single `%` in the rendered body. Format expressions like
`%[unique-id]` and `%[req.hdr(host)]` stay single-`%` — those are the
substitutions we want.
Added a comment block at the top of the file documenting the gotcha for
future editors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously a Coraza block returned an empty 403 with only the
`waf-block: request` header — a legitimate site owner caught in a
false-positive had no idea what happened or how to get help.
Now:
- hap_header.tpl: every request gets a unique-id (uuid()) and that ID
is injected back into the request as X-Request-Reference for the
backend, so upstream Apache/PHP logs can correlate too.
- hap_listener.tpl: on a request-phase Coraza deny we use
`http-request return` with `lf-file` instead of `http-request deny`,
so HAProxy renders the new errors/403-waf.html page with the
request reference substituted in. The page tells the visitor a
request was blocked, displays the reference, and points site owners
to https://secure.anhonesthost.com/submitticket.php to open a ticket
rather than exposing a public email address (avoids giving
attackers a flood target).
- The waf-block header and x-request-reference header are still set
on the response so curl / monitoring clients can pick them up
without rendering HTML.
- Response-phase deny stays as the bare 403 — outbound blocks are
rare in our config and an HTML body could land mid-stream.
Errorfile lives at /haproxy/errors/403-waf.html (NOT under
/etc/haproxy/, because that path is a named volume in deployed
containers and would shadow baked-in files on existing deployments).
Support workflow: visitor quotes the reference → support greps
/var/log/haproxy.log for the uuid → gets timestamp + client IP +
Host + URI → greps /var/log/coraza/audit.log for the matching
transaction → reads the rule_id that fired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
25h whp01 burn-in (2026-05-13) found ~11% FP rate on rule 913100:
ActivityPub federation pulls (Mastodon UA "...Bot" on hackerpublicradio.org
and blog.anti-social.online) and SiteLockSpider scans (a customer-paid
security service hitting greggfranklin.com + suchascream.net). The other
six promoted rule families (930120, 932100-160, 933170-200, 944100-300,
920440, 930130) showed zero FPs across the same window and stay enforced.
Detection-only still feeds the anomaly score, so we lose ~no real
blocking value by demoting this family.
The previous design used a separate whp-suspended container (nginx:alpine
serving a static 503 page) reachable via a dedicated bk_suspended backend.
That was over-engineered — haproxy-manager-base already ships a default-app
Flask server on :8080 that serves /default-page and /blocked-ip via
path-rewrite ACLs. Mirroring that pattern lets the suspension page live
in the SAME container, no extra image to build, no extra container to
run/health-monitor.
Changes:
- Add /suspended Flask route on default_app returning 503 + suspended_page.html
- Add templates/suspended_page.html (dark-themed 503 page)
- hap_listener.tpl: 'http-request set-path /suspended' + 'use_backend
default-backend' when host is in suspended_domains.list (same pattern
as is_blocked_ip)
- Rename env var from HAPROXY_SUSPENSION_BACKEND (a target hostport) to
HAPROXY_SUSPENSION_ENABLED (a bool); accepts 1/true/yes/on (case-insensitive)
- Remove hap_suspended_backend.tpl and its rendering in generate_config
Non-WHP deployments (env var unset) see byte-identical haproxy.cfg as before
(verified via jinja2 render diff).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If the upstream container isn't up when haproxy-manager starts (e.g. when
haproxy is recreated before whp-suspended), the default `init-addr libc` mode
makes haproxy refuse to start — taking down the whole proxy. Switched to
`init-addr last,none` (use last known address, fall back to 0.0.0.0 = DOWN)
and added `resolvers docker_dns` (defined in hap_header.tpl) so the real IP
is picked up once DNS becomes resolvable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a new env var HAPROXY_SUSPENSION_BACKEND (default unset). When set
(e.g. "whp-suspended:80"), generate_config() renders:
- A bk_suspended backend pointing at the configured upstream
- An ACL `acl is_suspended_domain hdr(host),lower -f /etc/haproxy/suspended_domains.list`
+ `use_backend bk_suspended if is_suspended_domain` in the frontend,
sitting after IP-blocking and before any per-domain routing
- An empty /etc/haproxy/suspended_domains.list if missing (haproxy refuses
to start with -f pointing at a non-existent file)
External tooling (e.g. WHP's site_disable.php) maintains the list via
`docker cp` and HUP-reloads the container.
Non-WHP deployments (home networks, standalone use) leave the env var
unset and see byte-identical haproxy.cfg output. Same opt-in shape as
the existing HAPROXY_CORAZA_SPOE_BACKEND integration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After ~30 min of detect-only on whp01 we have actionable data on what
fires against legitimate customer traffic vs. attacker recon. Two rules
demonstrably catch only the latter and earn promotion to the day-one
enforce list:
920440 — URL file extension restricted by policy
Caught 124 events in the sample window, ALL backup/config-file
disclosure probes (`/wp-config.php.old`, `/db_backup.sql`,
`/.env.save`, `/releases.sql` ...) from a single GCP-hosted scanner
hammering joshuaknapp.net. Match patterns: .sql (×62), .bak (×5),
.old (×3), .save (×2), .backup, .dist. No legitimate URL on
WP/WooCommerce/Divi/HPR ends in these.
930130 — Restricted File Access Attempt
Caught 117 events, ALL dotfile/VCS/config-disclosure probes
(`/.env`, `/.env.local`, `/.env.bak`, `/.git/config`, `/config.php`,
`/admin/.env`, `/backend/.env` ...). Spread across joshuaknapp.net,
cgdannyb.com, onlinesupplements.net. Notably, HPR's
`/ccdn.php?filename=/eps/...` legitimate audio-delivery URL does NOT
trigger this rule — verified empirically.
Also documented in the "intentionally detect-only" comment block: 933150
fires on WooCommerce checkout when literal `session_start` appears in
billing form data (alphaoneaminos.com saw 2 such events). That's a
canonical CRS false positive on WooCommerce; left detect-only.
Net effect: existing detect_only deployments stay detect-only (the WHP
apply script bind-mounts an empty overrides over the baked-in file).
When operators next flip a server to enforce, these two extra ranges
activate alongside the original day-one list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fixes that complete the SPOE enforcement path:
1. Listener was sending requests to Coraza for inspection but never reading
the result. Coraza-SPOA sets var(txn.coraza.action) to "deny" / "drop"
/ "redirect" when a rule with that disruptive action fires; HAProxy
needs explicit rules that READ the variable and apply the action.
Without them, the audit log shows "Access denied" but the request
still gets HTTP 200 (verified on staging: sqlmap/JNDI/shellinj all
detected, all returned 200).
Added the standard six rules from upstream's example/haproxy/haproxy.cfg
covering http-request + http-response phases for each of deny/drop/
redirect. Same set the upstream Coraza-SPOA docs recommend.
Intentionally did NOT add the upstream's fail-CLOSED rule
`http-request deny deny_status 500 if { var(txn.coraza.error) -m int gt 0 }`
— for a hosting platform we want fail-open. Documented inline.
2. Backend health check switched from plain TCP `check` to `option
spop-check`. The spop-check actually negotiates a SPOE session against
the agent, so HAProxy detects a half-broken SPOA that's listening on
:9000 but failing protocol handshakes. Plain `check` would miss that.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three real bugs in the SPOE config caught when HAProxy validated the
generated file:
1. spoe-agent must declare `groups` not `messages`. The `messages` form
doesn't make the message reachable via `send-spoe-group`; HAProxy
complained:
unable to find SPOE group 'coraza-check' into SPOE engine 'coraza'
2. send-spoe-group references a spoe-GROUP name, which needs its own
block. Added `spoe-group coraza-req { messages coraza-req }` as
the indirection layer.
3. Arg names + ORDER are required to match what Coraza-SPOA parses
positionally. My version had `dest-ip`/`dest-port`; upstream's
example/haproxy/coraza.cfg (v0.7.1) uses `dst-ip`/`dst-port`.
Renamed and reordered to match upstream verbatim, including the
`app=str(haproxy)` literal that matches our config.yaml application
name.
Also corrected misleading comment about `set-on-error continue`: that
option actually sets a variable on error; the fail-open behavior comes
from us deliberately NOT adding a `http-request deny if errored` rule
in the frontend. Renamed the variable to `error` (matching upstream)
and updated comments to be accurate.
Listener template's send-spoe-group action updated to reference the
new group name `coraza-req`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two HAProxy parse errors caught in staging functional test:
1. coraza-spoe.cfg:39 'args': missing fetch method
The args directive had backslash line continuations. HAProxy doesn't
support those in SPOE configs — args must be one physical line.
Collapsed to a single line.
2. coraza-spoe.cfg:50 Missing LF on last line
Same trailing-LF issue we hit on haproxy.cfg one commit ago. The
Jinja2 template ends with content rather than a newline, and write()
doesn't add one. Belt-and-suspenders: explicitly append '\n' before
writing if not already there.
After this commit HAProxy validates the generated config cleanly. Will
verify on staging now (combined SPOE injection + fail-open + active
attack-detection tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The SPOE backend block from hap_coraza_spoa_backend.tpl was being appended
last to config_parts. The template's render output doesn't end with a
newline (and config_parts is joined with '\n' BETWEEN elements, not after
the last one), so the resulting haproxy.cfg ended on `server coraza-spoa
...` with no trailing LF. HAProxy refuses to parse such files:
[ALERT] config: parsing [/etc/haproxy/haproxy.cfg:288]: Missing LF
on last line, file might have been truncated at position 70.
Match the existing pattern at the previous-last config_parts.append
(line 1850 uses `'\n'.join(config_backends) + '\n'`) and add an explicit
'\n' on the coraza block append.
Caught immediately on staging: HTTP 000 to localhost:80 because HAProxy
never started; gunicorn/management API kept serving on :8000 fine.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Default Jinja2 {% if %}{% endif %} block syntax leaves a trailing newline
even when the conditional doesn't render. Staging verification of PR 2
showed the resulting haproxy.cfg differed from the pre-PR2 version by
exactly 1 blank line — semantically identical but not byte-identical,
which violates the design promise that haproxy-manager-base's default
output stays unchanged for home/standalone deployments.
Use {%- if -%}/{%- endif %} (the whitespace-stripping variants) so the
block contributes zero bytes when coraza_spoe_backend is unset.
Verified locally: without env var = 55 lines, ends cleanly on the
is_blocked_ip rule. With env var = 62 lines, +7 for the SPOE block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the plumbing that lets haproxy-manager talk to the coraza-spoa sidecar
added in PR 1, while keeping the default behavior bit-identical for any
deployment that doesn't set the new env var (the home network / standalone
use cases).
Single gate: HAPROXY_CORAZA_SPOE_BACKEND env var on the haproxy-manager
container. Unset (default) = generate_config() renders zero SPOE-related
output. Set (e.g. "coraza-spoa:9000") = three things happen at config
generation time:
1. hap_listener.tpl injects 5 lines at the end of the frontend block:
filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
http-request send-spoe-group coraza coraza-check
...placed AFTER rate-limit and IP-block guards so we don't waste WAF
calls on requests we were going to drop anyway.
2. A new TCP backend (hap_coraza_spoa_backend.tpl) is appended:
backend coraza-spoa-backend
mode tcp
server coraza-spoa <env-var-target> check ...
3. The SPOE engine config (hap_coraza_spoe_engine.tpl) is rendered and
written to /etc/haproxy/coraza-spoe.cfg, defining the spoe-agent
"coraza" + spoe-message "coraza-check". This sets:
- option set-on-error continue (FAIL-OPEN if SPOA is unreachable)
- timeout processing 100ms (per-request inspection budget)
- app=str(haproxy) (matches sidecar's application name)
Verification (template render only, before staging deploy):
- hap_listener.tpl with no env var: 55 lines, zero SPOE references
- hap_listener.tpl with env var: 62 lines, filter + send-spoe-group present
- Engine cfg + backend block render with correct agent_target substitution
Next: PR 3 wires this into WHP (sidecar deploy via container-manager.sh
extension, server-settings UI for on/off, AI Monitor source for the audit
log). Staging verification of PR 1 + PR 2 together happens after PR 3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cloudflare's bot-management incident on 2026-05-12 took out docker.io blob
pulls twice in one day — first for python:3.12-slim (mirrored in e11d8c4),
then again for golang:1.25 when the PR 1 coraza-spoa build hit the same
R2-via-Cloudflare failure on the build stage's base image.
Restructure .gitea/workflows/mirror-base-image.yaml into a matrix that
iterates over a list of (src, dst_path, tag) entries. Adding a new base
image is now a one-line matrix entry. fail-fast: false so one image's
upstream being down doesn't block refreshing the others.
Switch coraza-spoa/Dockerfile's build stage FROM to the in-house golang
mirror. Runtime FROM (gcr.io/distroless/static-debian12:nonroot) stays
on upstream — distroless is on Google's registry, separate from Docker
Hub's Cloudflare R2 setup, and didn't fail during today's incident.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Self-contained sidecar that runs Coraza-SPOA v0.7.1 (latest upstream as of
2026-05-08, with OWASP CRS bundled in the binary). HAProxy will consult it
per-request via SPOE in PR 2; for now this PR ships the image only.
Defines:
- coraza-spoa/Dockerfile — multi-stage build (golang:1.25 -> distroless),
pinned to v0.7.1, ARG-overridable
- coraza-spoa/config.yaml — single application "haproxy", JSON audit log
to /var/log/coraza/audit.log, SecRuleEngine
DetectionOnly globally
- coraza-spoa/overrides.conf — day-one enforce list: scanner UAs (913xxx),
RCE shell injection (932100-932160),
webshell paths (933170-933200), targeted LFI
(930120), Log4Shell/JNDI (944100-944300).
Rationale per-range documented inline.
Detect-only for XSS/SQLi/protocol (high FP
on WP/WooCommerce/Divi customer mix).
- coraza-spoa/README.md — deployment shape, audit log location, pin
upgrade procedure, false-positive tuning.
- .gitea/workflows/build-push-coraza.yaml — Gitea Action triggered on
coraza-spoa/** changes, publishes
repo.anhonesthost.net/cloud-hosting-platform/
coraza-spoa:latest. Path-scoped so it
doesn't fire on every haproxy-manager push.
No changes to haproxy-manager-base itself in this PR — the existing image
stays bit-identical, used standalone in home networks and other projects
without dependency on this sidecar. PR 2 will add the OPT-IN template
plumbing that lets haproxy-manager call out to this agent when an env var
is set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to the Dockerfile change in e11d8c4. The previous manual refresh
note in the Dockerfile becomes automated: a workflow_dispatch + weekly cron
that pulls python:3.12-slim from docker.io and re-pushes it to
repo.anhonesthost.net/cloud-hosting-platform/python:3.12-slim.
Workflow can also be triggered manually from the Gitea UI when Python
publishes patches between cron firings. Logs the upstream and mirror digests
so it's easy to verify "did the mirror really update" after a run.
If more base images need mirroring later (haproxy itself, alpine, etc.),
this workflow should be promoted to a matrix or moved to a dedicated infra
repo — keeping it co-located with haproxy-manager-base for now since it's
the only consumer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docker.io serves image blobs from Cloudflare R2. The 2026-05-12 Cloudflare
incident took out blob pulls for hours and broke this image's Gitea CI
build mid-way through the haproxy-manager gunicorn migration (commit
c22d2cd). With the base image mirrored at repo.anhonesthost.net,
CI builds no longer depend on docker.io reachability.
Refresh procedure documented in the Dockerfile comment block. Manual
re-push monthly or when Python patches drop. A future Gitea Action could
automate the pull-tag-push so we always have a current base.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related fixes for the issues the AI Monitor surfaced on whp01 on
2026-05-12 (haproxy-manager going "healthy but stalled" after long
uptime, and noise from POST /blocked-ip returning 405):
1. Production WSGI server. The Flask app was running on werkzeug's
built-in dev server (the one that prints "WARNING: This is a
development server" on every startup). werkzeug is single-threaded
and accumulates worker state over long uptimes; after ~24h on whp01
the health endpoint stops responding while the container still
reports "healthy" because Docker's HEALTHCHECK uses an HTTP probe
from inside the same werkzeug process that's stalled.
Replace with gunicorn (gthread worker class, --max-requests=1000
with jitter so workers recycle periodically). Two gunicorn instances,
one per Flask app — port 8000 for the management API, port 8080 for
the default/blocked-ip page server. Both lift their app objects from
the haproxy_manager module so gunicorn can import them.
Required structural change: default_app was created INSIDE the
__name__ == '__main__' block at module bottom, where gunicorn could
never reach it. Moved to module level. The __main__ block now stays
only for `python haproxy_manager.py` local-dev workflow.
Container init (init_db, certbot register, generate_config,
start_haproxy) extracted into a do_initial_setup() function called
from a new scripts/init.py. start-up.sh runs init.py to completion
before either gunicorn binds, which keeps HAProxy startup off the
WSGI workers' fork paths (no race between workers all trying to
start_haproxy() at once).
2. /blocked-ip and / accept ALL methods. HAProxy proxies blocked-IP
traffic to default_app preserving the original verb, so a blocked
POST request used to hit Flask's GET-only route and get a 405 +
the AI Monitor flagged the noise. Adding the full method list lets
the 403 page render regardless of verb.
Gunicorn settings tunable via env (workers, timeout, max-requests).
API gets --timeout 120 because ACME cert issuance can be slow; the
default page server stays on the gunicorn default 30s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
certbot uses fasteners (fcntl-based locking) to serialize concurrent
invocations. The kernel auto-releases fcntl locks when the holding
process exits, but the .certbot.lock FILES persist on disk — and we've
seen real cases where subsequent runs report "Another instance of
Certbot is already running" even when no certbot process is alive.
Observed during the 2026-05-09 bundling rollout when a hung worker
held a lock across container-internal Python crashes.
When SSL is blocked on a customer site, this is high-impact: the
certbot lock can sit stale until somebody manually deletes it.
clear_stale_certbot_locks():
- probes each known lock path with fcntl.LOCK_NB
- if the lock is unheld → file is stale → delete it
- if the lock IS held → leave it alone (real certbot is running)
Wired in:
- container startup (init block)
- /api/ssl single-domain handler
- /api/ssl/bundle handler
- /api/certificates/renew handler
Safe to call repeatedly; never deletes a lock a real process holds, so
can never trigger concurrent certbot runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bundle endpoint correctly issued multi-SAN certs but left old
single-SAN .pem files (e.g. <name>-0001.pem) in /etc/haproxy/certs/.
HAProxy's `bind ... ssl crt /etc/haproxy/certs` loads everything in the
directory and picked the alphabetically-first matching file — typically
the older single-SAN one — so the new bundle had no effect on what was
served. Repro on peptidesaver.net: bundle covered 4 SANs but HAProxy
kept serving peptidesaver.net-0001.pem (single SAN, April-issued).
After a successful bundle write, walk SSL_CERTS_DIR and remove any
.pem whose CN is in the new bundle's name list (excluding the bundle's
own combined file). Drop the matching certbot lineage with
`certbot delete --cert-name <X> -n` so `certbot renew` stops touching
the dead lineage too.
Returns a `cleanup` summary in the API response so callers can log /
display what was deleted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WHP's renewal orchestrator now bundles a site's domains into one cert
covering all SANs, instead of N separate single-domain orders. Single
ACME order = better behavior under Let's Encrypt's 50/hour orders limit
when many domains need attention at once.
Endpoint: POST /api/ssl/bundle
Body: {"primary": "example.com", "sans": ["www.example.com", ...]}
- Uses --cert-name <primary> so the lineage stays stable across renewals
(no -0001/-0002 proliferation seen with the legacy single-domain flow).
- Single combined .pem at /etc/haproxy/certs/<primary>.pem; HAProxy SNI-
matches against the cert's SAN list, so one file serves all included
hostnames.
- Updates the domains table for every SAN in the bundle.
- Hard cap at 100 SANs (LE limit).
Existing /api/ssl single-domain endpoint kept for backwards compat.
The WHP haproxy_manager::bundleSSL() helper falls back to a per-domain
loop if /api/ssl/bundle returns 404, so the WHP side keeps working
during the rolling image upgrade window.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Volume-mounted /etc/haproxy can shadow the image-baked
trusted_ips.list/trusted_ips.map, causing HAProxy to fail
config validation with "failed to open pattern file" on
non-WHP deployments. Touch empty files if they don't exist
so the ACLs always parse.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The resolvers section was inserted inside the global section, causing
HAProxy to parse global directives (pidfile, maxconn, etc.) as
resolver keywords. Moved resolvers to its own top-level section
between global and defaults where HAProxy expects it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Docker containers restart, they can get new IPs on the bridge
network. HAProxy caches DNS at config load time, so stale IPs cause
503s until config is regenerated.
Added a 'docker_dns' resolvers section pointing to Docker's embedded
DNS (127.0.0.11) with 10s hold time. Backend servers now use
'resolvers docker_dns init-addr last,libc,none' so HAProxy:
- Re-resolves container names every 10 seconds
- Falls back to last known IP if DNS is temporarily unavailable
- Starts even if a backend can't be resolved yet (init-addr none)
This eliminates 503s from container restarts, scaling, and recreation
without requiring a HAProxy config regeneration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The renewal script was exiting immediately when certbot returned a
non-zero exit code, which happens when ANY cert fails to renew. A
single dead domain (e.g., DNS no longer pointed here) would block
ALL other certificates from being processed and combined for HAProxy.
Now logs the failures but continues to copy/combine successfully
renewed certificates and reload HAProxy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents HAProxy health checks, watchdog, rate limiting, trusted IP
whitelist, timeout hardening, HTTP/2 protection, and the AI-powered
log monitor system with two-tier analysis, auto-remediation, and
notification support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generous thresholds that accommodate sites with many images/assets
while still catching obvious automated floods:
- Request rate: tarpit at 300 req/s, block at 500 req/s
- Connection rate: 500/10s
- Concurrent connections: 500
- Error rate: 100/30s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous thresholds (200/500 req/10s) were too aggressive — WordPress
login pages with their CSS/JS/image assets can easily burst 30-50
requests per page load, triggering tarpits and blocks on legitimate
users.
New thresholds:
- Request rate: tarpit at 1000/10s (100 req/s), block at 2000/10s (200 req/s)
- Connection rate: 300/10s (was 150)
- Concurrent connections: 200 (was 100)
- Error rate: 50/30s (was 20)
These still catch real floods and scanners while giving normal web
traffic plenty of headroom.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds trusted_ips.list and trusted_ips.map files that exempt specific
IPs from all rate limiting rules. Supports both direct source IP
matching (is_trusted_ip) and proxy-header real IP matching
(is_whitelisted). Files are baked into the image and can be updated
by editing and rebuilding.
Adds phone system IP 172.116.197.166 to the whitelist.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gives more headroom for customers with code that makes frequent
callbacks to itself, while still catching connection floods.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Activate HAProxy's built-in attack prevention to stop floods that cause
the container to become unresponsive:
- Stick table tracks per-IP: conn_cur, conn_rate, http_req_rate, http_err_rate
- Rate limit rules: deny at 50 req/s, tarpit at 20 req/s, connection
rate limit at 60/10s, concurrent connection cap at 100, error rate
tarpit at 20 errors/30s
- Harden timeouts: http-request 300s→30s, connect 120s→10s, client
10m→5m, keep-alive 120s→30s
- HTTP/2 Rapid Reset protection (CVE-2023-44487): stream and glitch limits
- Stats frontend on localhost:8404 for monitoring
- HEALTHCHECK now validates both port 80 (HAProxy) and 8000 (API)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Captures the Host header in HAProxy httplog output so high-connection
alerts can be correlated to specific domains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add find_certbot_live_dir() helper to locate the most recent certbot live
directory for a domain, handling -NNNN suffixed dirs from repeated requests.
Fix combined cert filename from *.domain.pem to _wildcard_.domain.pem.
Apply the helper across all SSL endpoints (request, renew, verify, download).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hook scripts are at /haproxy/scripts/ inside the container (per
Dockerfile COPY), not /app/scripts/. Also added logging of certbot
stdout/stderr so failures are visible in haproxy-manager.log.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Support wildcard domains (*.domain.tld) in HAProxy config generation
with exact-match ACLs prioritized over wildcard ACLs. Add DNS-01
challenge endpoints that coordinate with certbot via auth/cleanup
hook scripts for wildcard SSL certificate issuance.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Changes:
- Detect SSE via Accept header (text/event-stream) or ?action=stream parameter
- Disable http-server-close to allow long-lived SSE connections
- Enable http-no-delay for immediate event delivery
- Set 1-hour timeouts for SSE support (also fine for normal requests)
- Force Connection: keep-alive for detected SSE requests
Benefits:
- SSE now works automatically without special backend configuration
- Fixes transcription server display disconnection issues
- Normal HTTP requests still work perfectly
- No need for separate SSE-specific backends
Fixes: Server-Sent Events timing out through HAProxy
Improved certificate renewal and sync scripts to be more resilient:
- Removed 'set -e' to prevent silent failures when individual domains error
- Scripts now continue processing remaining domains even if one fails
- Replaced database queries with direct filesystem scanning of /etc/letsencrypt/live/
- Uses 'find' command to discover all domains with Let's Encrypt certificates
- More reliable as it works even if database is out of sync
Benefits:
- No silent failures - errors are logged but don't stop the entire process
- Works independently of database state
- Simpler and more straightforward
- All domains with certificates get processed regardless of database
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Simplified all certificate renewal scripts to be more straightforward and reliable:
- Scripts now just run certbot renew and copy cert+key files to HAProxy format
- Removed overly complex retry logic and error handling
- Both in-container and host-side scripts work with cron scheduling
Added automatic certbot cleanup when domains are removed:
- When a domain is deleted via API, certbot certificate is also removed
- Prevents renewal errors for domains that no longer exist in HAProxy
- Cleans up both HAProxy combined cert and Let's Encrypt certificate
Script changes:
- renew-certificates.sh: Simplified to 87 lines (from 215)
- sync-certificates.sh: Simplified to 79 lines (from 200+)
- host-renew-certificates.sh: Simplified to 36 lines (from 40)
- All scripts use same pattern: query DB, copy certs, reload HAProxy
Python changes:
- remove_domain() now calls 'certbot delete' to remove certificates
- Prevents orphaned certificates from causing renewal failures
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Write combined certificates to temporary file first
- Verify file is not empty before moving to final location
- Use atomic mv operation to prevent HAProxy from reading partial files
- Add proper cleanup of temporary files on all error paths
- Matches robust patterns from haproxy_manager.py
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Update map file format to include value (IP/CIDR 1)
- Fix HAProxy template to use map_ip() for CIDR support
- Update runtime map commands to include value
- Document CIDR range blocking in API documentation
- Support blocking entire network ranges (e.g., 192.168.1.0/24)
This allows blocking compromised ISP ranges and other large-scale attacks.
After certbot renews certificates, the separate fullchain.pem and privkey.pem
files must be combined into a single .pem file for HAProxy. The renewal script
was missing this critical step, causing HAProxy to continue using old certificates.
Changes:
- Add update_combined_certificates() function to renew-certificates.sh
- Query database for all SSL-enabled domains
- Combine Let's Encrypt cert + key files using cat (matches haproxy_manager.py pattern)
- Always update combined certs after renewal, even if certbot says no renewal needed
- Add new sync-certificates.sh script for syncing all existing certificates
- Smart update detection in sync script (only updates when source is newer)
This ensures HAProxy always gets properly formatted certificate files after renewal.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit simplifies the HAProxy configuration by removing automatic
threat detection and blocking rules while preserving essential functionality.
Changes:
- Removed all automatic ACL-based security rules (SQL injection detection,
scanner detection, rate limiting, brute force protection, etc.)
- Removed complex stick-table tracking with 15 GPC counters
- Removed graduated threat response system (tarpit, deny based on threat scores)
- Removed HTTP/2 security tuning parameters specific to threat detection
- Commented out IP header forwarding in hap_backend_basic.tpl
Preserved functionality:
- Real client IP detection from proxy headers (CF-Connecting-IP, X-Real-IP,
X-Forwarded-For) with proper fallback to source IP
- Manual IP blocking via map file (/etc/haproxy/blocked_ips.map)
- Runtime map updates for immediate blocking without reload
- Backend IP forwarding capabilities (available in hap_backend.tpl)
The configuration now focuses on manual IP blocking only, which can be
managed through the API endpoints (/api/blocked-ips).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed crontab permissions (600) and ownership for proper cron execution
- Added PATH environment variable to crontab to prevent command not found issues
- Created dedicated renewal script with comprehensive logging and error handling
- Added retry logic (3 attempts) for HAProxy reload with socket health checks
- Implemented host-side renewal script for external cron scheduling via docker exec
- Added crontab configuration examples for various renewal schedules
- Updated README with detailed certificate renewal documentation
This resolves issues where the cron job would not run or hang during execution.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove semicolons from variable initialization in AWK scripts
- Each variable now on separate line to prevent syntax errors
- Fixes "syntax error at or near ," in monitor-attacks.sh and manage-blocked-ips.sh
- Scripts now properly parse HAProxy 3.0.11 threat intelligence data
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove reference to non-existent security_blacklist table
- Use single table tracking with consolidated array-based GPC system
- Remove res.hdr(X-Threat-Level) from log-format as response headers not available in request phase
- Maintains threat intelligence logging with available request-phase data
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace compound ACL xmlrpc_abuse with separate conditions
- Use xmlrpc_rate_abuse for rate detection and combine with is_xmlrpc in http-request rule
- Prevents ACL-to-ACL reference which is not supported in HAProxy 3.0.11
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add -m int matcher for all var(txn.threat_score) comparisons
- Fix set-header, tarpit, deny, and set-log-level conditions
- Ensures proper variable type matching for HAProxy 3.0.11
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix tune.h2.fe-max-total-streams parameter name in global config
- Fix stick-table multiline syntax by removing line continuations
- Replace sc0_get_gpc with sc_get_gpc for proper 3.0.11 syntax
- Replace sc-set-gpc with sc-set-gpt for value assignments
- Update ACL definitions to use correct GPT fetch methods
- Simplify threat scoring to avoid unsupported add-var operations
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Escape inner quotes in the certbot renewal cron job to properly
send reload command to HAProxy via socat after certificate renewal.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Removed all 40X error tracking and rate limiting from HAProxy templates
- Preserved critical IP forwarding headers (X-CLIENT-IP, X-Real-IP, X-Forwarded-For)
- Kept stick table and IP blocking infrastructure for potential future use
- Rate limiting can now be implemented at container level with proper context
This change prevents legitimate developers from being rate-limited during
normal development activities while maintaining proper client IP forwarding
for container-level security and logging.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove invalid ACL combination syntax (can't use 'or' to combine ACLs)
- Use multiple http-response lines instead (each line is OR'd together)
- Each line checks specific scan pattern with 404 AND not legitimate assets
- Simplify logic to be HAProxy 3.0 compatible
This fixes the config parsing errors while maintaining the same
detection logic - only counting suspicious script/config 404s, not
missing assets.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Handle common missing files (favicon.ico, robots.txt) without counting as errors
- Return 404 directly from frontend for these files (bypasses backend counting)
- Add clear-ip.sh script to remove specific IPs from stick-table
- Keep trusted networks whitelist for local/private IPs
This prevents legitimate users from being blocked due to browser
requests for common files that don't exist.
Usage: ./scripts/clear-ip.sh <IP_ADDRESS>
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove unsupported set-timeout tarpit directives
- Use fixed 30s global tarpit timeout (reduced from 60s)
- Keep escalation tracking via gpc1 for monitoring repeat offenders
- HAProxy 3.0 doesn't support variable tarpit timeouts per request
The escalation level (gpc1) is still tracked and visible in monitoring
but all tarpits use the same 30s delay.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Extends the tarpit protection and real IP handling to all backend templates,
ensuring consistent behavior across different backend configurations.
Changes to all backend templates:
- Pass real client IP via X-CLIENT-IP and X-Real-IP headers
- Use var(txn.real_ip) which contains the actual client IP (from proxy headers or direct)
- Add scan attempt detection (400/401/403/404 errors)
- Track suspicious paths (admin panels, config files, etc.)
- Increment error counters for tarpit decisions
Updated templates:
- hap_backend.tpl: Main backend template
- hap_backend_http_check.tpl: Backend with HTTP health checks
- hap_backend_basic.tpl: Minimal backend configuration
Benefits:
- Backend applications receive the real client IP, not proxy IPs
- All backend types now contribute to scan detection
- Consistent security across different backend configurations
- Works seamlessly with Cloudflare and other CDNs
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements smart client IP detection to handle Cloudflare and other reverse
proxies correctly, preventing legitimate traffic from being tarpited when
behind a shared proxy IP.
Changes:
- Detect real client IP from proxy headers with priority order:
1. CF-Connecting-IP (Cloudflare)
2. X-Real-IP (common proxy header)
3. X-Forwarded-For (standard proxy header)
4. src (fallback to source IP if no headers)
- Track real client IP in stick-table instead of proxy IP
- Check real client IP for blocking rules
- No need to maintain proxy IP lists - works automatically
This ensures that:
- Cloudflare and other CDN traffic is tracked per real client
- Each actual user gets their own tarpit counter
- Legitimate users aren't affected by attackers on the same proxy
- Works automatically with any proxy that sets standard headers
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Corrected the tarpit logic flow to work as intended:
1. Backend tracks 400/401/403/404 error responses via http-response
2. Counter increments AFTER the backend responds with an error
3. Frontend checks counter on SUBSEQUENT requests
4. Tarpit/blocking only applies after error thresholds are reached:
- 5+ errors: Potential scanner (no action yet)
- 15+ errors: Likely scanner (tarpit if also burst traffic)
- 30+ errors: Confirmed scanner (always tarpit)
- 50+ errors: Aggressive scanner (block with 429)
This ensures:
- Normal traffic is never delayed
- First requests always go through normally
- Only clients that accumulate errors get progressively slowed/blocked
- The tarpit is a response to bad behavior, not a preemptive measure
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The previous configuration was tarpiting all connections because the ACLs
were overlapping (e.g., low_threat >= 3 would match everything above 3).
Changes:
- Add proper range checks for threat levels (e.g., >= 3 AND < 10 for low)
- Simplify tarpit logic to only apply when scan attempts are detected
- Remove complex escalation levels (not working properly in HAProxy 3.0)
- Only tarpit connections with 3+ scan attempts or burst attacks
- Critical threats (50+ attempts) get immediate 429 block
This ensures normal traffic flows through without delay while actual
scanners and attackers get tarpited based on their behavior.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
shadowdao
