- H1: request_ssl_bundle uses live/<primary> exactly (certbot was given
--cert-name <primary>) instead of find_certbot_live_dir(), which returns the
newest of <primary>/<primary>-NNNN. After a "not yet due" no-op that
published a stale -0001 and retired the real <primary> lineage. keep_lineage
is now the primary; the bundle's own lineage is never a retirement candidate.
- H2: retire_lineage_tree() moves live/<x>, archive/<x> and renewal/<x>.conf
together to /etc/letsencrypt/retired/<stamp>/{live,archive,renewal}/ (same
depth, relative symlinks still resolve, restore is three mv's). Leaving
archive/<x> behind made certbot's new_lineage() refuse the name forever
(CertStorageError) and leave an empty renewal conf. Test re-issues the name
through certbot's own storage code after retirement.
- M2: cert_retire_expired_pems moves already-published EXPIRED pems out of
/etc/haproxy/certs to cert-retired/ (never the last pem, never the default
self-signed cert) and the scripts reload when anything was retired.
- GET /api/capabilities + `capabilities` in the bundle response, and VERSION is
copied into the image, so WHP can refuse whole-site re-bundling against an
older edge.
Tests: test-cert-write-safety.py 47 -> 52, test-cert-scripts.py 44 -> 50; each
fix reverted in a scratch copy turns its suite red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0 (local
haproxy-manager + coraza-spoa + echo backend):
- send-spoe-group fired at header time, so req.body held only what arrived
with the headers. A 933180 payload whose body came in a second send(),
2s late, or after Expect: 100-continue passed uninspected; 8 parallel
14 KB payload-first POSTs: 6/40 blocked before, 40/40 after.
- Add `option http-buffer-request` (Coraza block only). Uploads are not
held: one buffer is inspected, the rest streams (50 MB upload: sha-ok,
same timing before/after).
- Stamp X-WHP-WAF-Body: complete|partial for Coraza (client copies
stripped first, stripped again before the backend). Content-Length:
exact compare; chunked/no-CL: body+headers < 12 KB => complete. WHP's
generated rules use it to deny body-parse errors (200002/200003) only on
complete bodies.
- Pin tune.bufsize 16384 (= default) when Coraza is on. Coraza costs
~3.3 ms/KB of body and the 100 ms SPOE timeout fails open: at 32 KB a
payload at byte 0 of a 30 KB base64 body bypassed. SPOP frames > 65535
are refused by the agent library.
- coraza-spoa/config.yaml: SecRequestBodyLimit 131072 / InMemory 131072 /
ProcessPartial (Reject 413s even under DetectionOnly).
- Access log gains wafb= (complete|partial) and wafe= (SPOE error =
request passed uninspected).
- overrides.conf: document (comment only) that the 920440 promotion turns
the engine On for every URL with a file extension.
- scripts/test-waf-body-inspection.py (16 tests), wired into the build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.
The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).
- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
* per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
re-applied on keep-alive reuse; bounds socket I/O, not handling time
* watchdog: if every pool thread has been in the same request for more
than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
all thread stacks and exit so the master respawns the worker
* client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
60s) as defence in depth for any future in-process client; /health now
reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
added to the Docker build gate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Previously unpinned, so the binary could move at Debian's timing and break an
unrelated commit's build -- or ship an edge that refuses to start. Pinning and
the haproxy -c gate compose: pinned means the version moves deliberately, and
the gate then answers whether the new binary still accepts fleet config. Also
makes the image reproducible.
A config change can render perfectly, pass every unit test in scripts/, and
still be rejected outright by HAProxy. That happened on 2026-08-14: an inline
`regsub((^|/)wp-admin/.*,\1wp-login.php)` in a redirect location produced
"invalid arg 2 in converter 'regsub'". Thirteen tests were green. It was only
caught because someone built an image by hand and ran `haproxy -c`.
Nothing between commit and production would have stopped it. The unit suites
assert on the TEXT of the rendered config with regexes, which says what the
template emits, never whether HAProxy accepts it. test-config-rollback.py's
"validation" stubs the haproxy binary with a shell script that rejects one
sentinel token and has never parsed a line of real syntax. And
.gitea/workflows/build-push.yaml is checkout -> build -> push, with no tests
at all.
The production consequence is not a broken deploy, it is a silent outage:
init.py refuses to start HAProxy on an invalid config while the container
still comes up, so ports 80/443 are unbound, every site on the host is down,
and /health keeps answering 200.
scripts/validate-rendered-config.py renders the config through the real
generate_config() - every template, real order, both conditional branches
({%- if suspension_enabled %} and {%- if coraza_spoe_backend %}) rendered on
in one scenario and off in the other - creates the stub files the config
loads via `-f` (a missing one is a FATAL haproxy error and would be a false
failure), then runs `haproxy -c` and gates on its EXIT CODE. Warnings are
expected on a clean config ("Can't load stats file", path_reg advisories) and
are not failures; on a real failure the full haproxy output plus the offending
config lines go to the build log.
It runs as a Dockerfile RUN rather than a CI step so it cannot be skipped, so
it protects local builds too, and - the reason that matters most - so it
validates against the EXACT haproxy binary in the image being built. The
Dockerfile installs haproxy unpinned, so that binary moves between builds;
this turns "the new haproxy rejects our config" from a silent production risk
into a build failure. Gating in CI instead would also have meant splitting
build-push-action's single build-and-push step.
The six existing unit suites run in the same step. They had never run
anywhere automated either, and they cost about five seconds.
Verified both ways: the clean build passes and the gate's output appears in
the log; reintroducing the known-bad regsub into a copy of the template fails
the build with HAProxy's own "invalid arg 2 in converter 'regsub' : missing
arguments (got 1/2)".
/etc/haproxy is a named volume in deployed containers, so the baked-in
cloudflare_ips.list and trusted_proxies.list COPYed there in the prior
task never actually reached hosts with a pre-existing volume -- the
start-up.sh guard then found them "missing" and created them empty.
With both lists empty, the from_trusted_proxy ACL in hap_listener.tpl
matched nothing, so CF-Connecting-IP / X-Real-IP / X-Forwarded-For got
stripped from every peer, including Cloudflare's own edge. Confirmed
live: image shipped 34/13 lines, running container had 0/0.
Fix: stage both files under /haproxy/defaults (outside the volume) and
apply their ownership rule in start-up.sh instead of a blind
"create if missing":
- cloudflare_ips.list is shipped data -- always refresh it from the
baked default so Cloudflare range updates reach existing hosts.
- trusted_proxies.list is operator data -- seed it from the baked
default only when missing, and never overwrite what an operator
added on the server.
Both branches fall back to creating an empty file if the baked default
is somehow absent, since a missing "-f" target is a fatal HAProxy
config error.
Verified against a volume pre-populated to shadow the image (mimicking
a real host): cloudflare_ips.list repopulates with all 15 IPv4 + 7
IPv6 ranges even after being truncated and restarted; a distinctive
operator entry appended to trusted_proxies.list survives a restart
untouched; haproxy -c still validates cleanly.
Release-worthy fix for a defect from the just-released 2026.08.2 build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HTTP/3 is config-only — the Debian haproxy package is built +QUIC via the
OpenSSL compat shim. Changes:
- hap_header.tpl: `limited-quic` (required to enable QUIC binds under the
compat layer) + self-healing `cluster-secret` for QUIC token derivation.
- hap_listener.tpl: `bind quic4@:443 ... alpn h3` in the shared frontend (so
real-IP/rate-limit/IP-block/Coraza rules apply to H3 too) + alt-svc header.
- Dockerfile/README: publish/document 443/udp; stamp image.version from VERSION.
- CI: tag :latest + :<VERSION> + :<sha> so there's a pinnable rollback target.
No 0-RTT (compat-layer limitation). Validated end-to-end on a standalone edge:
config parses, UDP/443 binds, alt-svc advertised, real curl --http3 -> HTTP/3.
Container must run with `-p 443:443/udp` + host UDP/443 open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds (Dockerfile) and updates (coraza-spoa/Dockerfile) the OCI
image.source label to point at github.com/shadowdao/haproxy-manager-base.
ghcr.io auto-links a package to a GitHub repo when this label resolves
to a github.com URL whose owner+name match the package's owner — that
makes the published packages show up on the GitHub repo sidebar and
inherit its collaborator settings.
Gitea's registry ignores image.source, so changing the value away from
the previous Gitea URL costs nothing on that side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously a Coraza block returned an empty 403 with only the
`waf-block: request` header — a legitimate site owner caught in a
false-positive had no idea what happened or how to get help.
Now:
- hap_header.tpl: every request gets a unique-id (uuid()) and that ID
is injected back into the request as X-Request-Reference for the
backend, so upstream Apache/PHP logs can correlate too.
- hap_listener.tpl: on a request-phase Coraza deny we use
`http-request return` with `lf-file` instead of `http-request deny`,
so HAProxy renders the new errors/403-waf.html page with the
request reference substituted in. The page tells the visitor a
request was blocked, displays the reference, and points site owners
to https://secure.anhonesthost.com/submitticket.php to open a ticket
rather than exposing a public email address (avoids giving
attackers a flood target).
- The waf-block header and x-request-reference header are still set
on the response so curl / monitoring clients can pick them up
without rendering HTML.
- Response-phase deny stays as the bare 403 — outbound blocks are
rare in our config and an HTML body could land mid-stream.
Errorfile lives at /haproxy/errors/403-waf.html (NOT under
/etc/haproxy/, because that path is a named volume in deployed
containers and would shadow baked-in files on existing deployments).
Support workflow: visitor quotes the reference → support greps
/var/log/haproxy.log for the uuid → gets timestamp + client IP +
Host + URI → greps /var/log/coraza/audit.log for the matching
transaction → reads the rule_id that fired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docker.io serves image blobs from Cloudflare R2. The 2026-05-12 Cloudflare
incident took out blob pulls for hours and broke this image's Gitea CI
build mid-way through the haproxy-manager gunicorn migration (commit
c22d2cd). With the base image mirrored at repo.anhonesthost.net,
CI builds no longer depend on docker.io reachability.
Refresh procedure documented in the Dockerfile comment block. Manual
re-push monthly or when Python patches drop. A future Gitea Action could
automate the pull-tag-push so we always have a current base.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds trusted_ips.list and trusted_ips.map files that exempt specific
IPs from all rate limiting rules. Supports both direct source IP
matching (is_trusted_ip) and proxy-header real IP matching
(is_whitelisted). Files are baked into the image and can be updated
by editing and rebuilding.
Adds phone system IP 172.116.197.166 to the whitelist.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Activate HAProxy's built-in attack prevention to stop floods that cause
the container to become unresponsive:
- Stick table tracks per-IP: conn_cur, conn_rate, http_req_rate, http_err_rate
- Rate limit rules: deny at 50 req/s, tarpit at 20 req/s, connection
rate limit at 60/10s, concurrent connection cap at 100, error rate
tarpit at 20 errors/30s
- Harden timeouts: http-request 300s→30s, connect 120s→10s, client
10m→5m, keep-alive 120s→30s
- HTTP/2 Rapid Reset protection (CVE-2023-44487): stream and glitch limits
- Stats frontend on localhost:8404 for monitoring
- HEALTHCHECK now validates both port 80 (HAProxy) and 8000 (API)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fixed crontab permissions (600) and ownership for proper cron execution
- Added PATH environment variable to crontab to prevent command not found issues
- Created dedicated renewal script with comprehensive logging and error handling
- Added retry logic (3 attempts) for HAProxy reload with socket health checks
- Implemented host-side renewal script for external cron scheduling via docker exec
- Added crontab configuration examples for various renewal schedules
- Updated README with detailed certificate renewal documentation
This resolves issues where the cron job would not run or hang during execution.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Escape inner quotes in the certbot renewal cron job to properly
send reload command to HAProxy via socat after certificate renewal.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
shadowdaoandClaude Opus 5.5