feat/default-splash
66
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  shadowdaoandClaude Sonnet 5.5
|
0075b2e0de |
feat(edge): unrouted hosts 404+noindex everywhere; access log moves into the container (2026.11.2)
Default splash (Josh 2026-11-08): unrouted hosts are 404 on EVERY path incl. "/" (same static page), with cache-control: no-store, x-robots-tag: noindex, nofollow, noarchive and nosniff; /robots.txt answers 200 "Disallow: /". So the server IP/hostname is never indexed as a site. Routed sites, ACME, blocked IPs, suspended hosts, rate limits, wp-* gates and WAF are unchanged. Access log: the old target 172.18.0.1:514 (host rsyslog) had no listener on the fleet. HAProxy now logs to a UNIX datagram socket in its chroot (`log /log.sock`, i.e. /var/lib/haproxy/log.sock); scripts/haproxy-log-receiver.py (started before init.py, supervised by a restart loop in start-up.sh) writes /var/log/haproxy/access.log rotated at 40 MB + 4 gzip generations (<= 200 MB). No network listener; sends are non-blocking, so a dead/stalled receiver or a full disk only drops lines (DroppedLogs). Nothing is printed to docker logs (WHP's AI monitor reads them). Same log-format (cip/host/status/Coraza fields). HAPROXY_SYSLOG_TARGET still overrides. show-tarpit-ips.sh / monitor-attacks.sh point at the new file. Tests: test-default-splash.py (19, incl. log target), new test-access-log-receiver.py (10), both in the image build; validate-rendered- config.py scenarios all accepted by haproxy -c. Lab (pre-populated volume, recreate): 404+headers on hostname/IP/unknown, robots 200, routed site 200, attack on site 403, ACME 200, suspended 503; 2000/2000 burst lines logged; receiver kill -9 -> requests 200, receiver back <3s; SIGSTOP + 2500 requests -> all 200, 1475 DroppedLogs; rotation at 64 KB cap kept 1+2 gz; logs survive recreate on the haproxy-logs volume. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb |
||
|
shadowdaoandClaude Sonnet 5.5
|
8787e0d99e |
feat(edge): static default splash for unrouted hosts, answered before the WAF (2026.11.1)
Visiting a server's own hostname (or its IP, or any unrouted name) on 80/443 could show the Coraza WAF block page. `use_backend default-backend` does not short-circuit http-request rules, so the catch-all was WAF-inspected with the full server-wide rule set like customer traffic; a browser on the server hostname also carries the control panel's cookies (not port-scoped). Unrouted requests are now answered by HAProxy itself with a tiny static page (errors/default-splash.html via `http-request return ... file`; 200 on "/", 404 elsewhere) placed before the SPOE call. "Routed" comes from routed_hosts.list / routed_wildcards.list, written by generate_config() from the same rows as the routing ACLs, port-stripped + lowercased (superset of the raw-Host routing ACLs), and part of the rollback set. ACME http-01, blocked IPs and suspended hosts keep their existing handling; rate limits and gates still run first. HAPROXY_DEFAULT_SPLASH=off restores the old behaviour. Page and template live in the image outside the /etc/haproxy named volume; the lists are generated on start, so image bump + recreate is the deploy. Tests: scripts/test-default-splash.py (in the image build), new splash-off scenario in validate-rendered-config.py. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb |
||
|
shadowdaoandClaude Opus 5.5
|
765b2ce00a |
Merge fix/ssl-renewal-durable into release/2026.10.3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 5.5
|
3712860429 |
Merge feat/waf-body-hardening into release/2026.10.3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 5.5
|
8c7fd93387 |
fix(ssl): review round 1 - exact --cert-name lineage, retire whole lineage tree, sweep expired pems
- H1: request_ssl_bundle uses live/<primary> exactly (certbot was given
--cert-name <primary>) instead of find_certbot_live_dir(), which returns the
newest of <primary>/<primary>-NNNN. After a "not yet due" no-op that
published a stale -0001 and retired the real <primary> lineage. keep_lineage
is now the primary; the bundle's own lineage is never a retirement candidate.
- H2: retire_lineage_tree() moves live/<x>, archive/<x> and renewal/<x>.conf
together to /etc/letsencrypt/retired/<stamp>/{live,archive,renewal}/ (same
depth, relative symlinks still resolve, restore is three mv's). Leaving
archive/<x> behind made certbot's new_lineage() refuse the name forever
(CertStorageError) and leave an empty renewal conf. Test re-issues the name
through certbot's own storage code after retirement.
- M2: cert_retire_expired_pems moves already-published EXPIRED pems out of
/etc/haproxy/certs to cert-retired/ (never the last pem, never the default
self-signed cert) and the scripts reload when anything was retired.
- GET /api/capabilities + `capabilities` in the bundle response, and VERSION is
copied into the image, so WHP can refuse whole-site re-bundling against an
older edge.
Tests: test-cert-write-safety.py 47 -> 52, test-cert-scripts.py 44 -> 50; each
fix reverted in a scratch copy turns its suite red.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 5.5
|
fd06b30d78 |
fix(api): reap never-readable conns, bound total header time, harden watchdog
Review round 1 on
|
||
|
shadowdaoandClaude Opus 5.5
|
e58996d6c9 |
fix(ssl): retire SAN-overlapping certs, never publish expired lineages, report no-op renewals
brainjar.net served an expired certificate for two days (2026-10-07..09): the old brain-jar.com bundle (CN brain-jar.com, SANs incl. all eight brainjar names) stayed in /etc/haproxy/certs and won the SNI match over the valid brainjar.net bundle, because superseded-cert cleanup matched on CN only, and renew-certificates.sh re-published the expired lineage every 12h. - Retirement now matches on ANY shared name (CN or SAN). The overlapping pem is moved to cert-retired/ and its lineage's live/ dir and renewal conf to live-retired/ and renewal-retired/ (after HAProxy loads the new bundle). Nothing is deleted any more: `certbot delete` is gone from this path. - Shared-PEM safety: a candidate is kept when it is still the only valid certificate for a name in the domains table (retirement_blockers()). Expired certs never block; names the caller lists in the new optional `unreachable_names` body field do not block. - /api/ssl/bundle passes --keep-until-expiring and reports `renewed` (fingerprint before vs after), `certbot_not_due` and the certificate's `not_after`, so WHP can stop recording certbot no-ops as renewals. - Never publish an expired certificate: renew-certificates.sh and sync-certificates.sh skip expired lineages (ERROR-logged, counted separately), and cert_publish / publish_pem_bundle refuse expired leaves. - VERSION 2026.10.3. Tests: test-cert-write-safety.py 31 -> 47, test-cert-scripts.py 38 -> 44, each new group verified to fail against a reverted fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 5.5
|
4f21bb2f76 |
feat(waf): make Coraza see request bodies; stamp complete/partial; 2026.10.2
Measured on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0 (local haproxy-manager + coraza-spoa + echo backend): - send-spoe-group fired at header time, so req.body held only what arrived with the headers. A 933180 payload whose body came in a second send(), 2s late, or after Expect: 100-continue passed uninspected; 8 parallel 14 KB payload-first POSTs: 6/40 blocked before, 40/40 after. - Add `option http-buffer-request` (Coraza block only). Uploads are not held: one buffer is inspected, the rest streams (50 MB upload: sha-ok, same timing before/after). - Stamp X-WHP-WAF-Body: complete|partial for Coraza (client copies stripped first, stripped again before the backend). Content-Length: exact compare; chunked/no-CL: body+headers < 12 KB => complete. WHP's generated rules use it to deny body-parse errors (200002/200003) only on complete bodies. - Pin tune.bufsize 16384 (= default) when Coraza is on. Coraza costs ~3.3 ms/KB of body and the 100 ms SPOE timeout fails open: at 32 KB a payload at byte 0 of a 30 KB base64 body bypassed. SPOP frames > 65535 are refused by the agent library. - coraza-spoa/config.yaml: SecRequestBodyLimit 131072 / InMemory 131072 / ProcessPartial (Reject 413s even under DetectionOnly). - Access log gains wafb= (complete|partial) and wafe= (SPOE error = request passed uninspected). - overrides.conf: document (comment only) that the 920440 promotion turns the engine On for every URL with a file extension. - scripts/test-waf-body-inspection.py (16 tests), wired into the build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 5.5
|
0ce144d2fb |
fix(api): stop silent clients from wedging the gunicorn thread pool
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.
The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).
- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
* per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
re-applied on keep-alive reuse; bounds socket I/O, not handling time
* watchdog: if every pool thread has been in the same request for more
than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
all thread stacks and exit so the master respawns the worker
* client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
60s) as defence in depth for any future in-process client; /health now
reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
added to the Docker build gate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 5
|
b072786192 |
fix(domain-removal): stop deleting certificates other live sites are served from
DELETE /api/domain ran, unconditionally, for any ssl_enabled row:
os.remove(ssl_cert_path)
certbot delete --cert-name <domain> --non-interactive
`domains.domain` is UNIQUE. `domains.ssl_cert_path` is not, and nothing
anywhere guarded against two rows naming the same file. Sharing is not an
edge case -- it is the normal shape of the table, because
request_ssl_bundle() deliberately creates it: one SAN certificate is issued
as `--cert-name <primary>`, published once to
/etc/haproxy/certs/<primary>.pem, and then EVERY included name's row is
pointed at that same path ("Mark every name in the bundle as ssl_enabled,
all pointing at the same combined .pem").
Measured read-only on the live SQLite in the haproxy-manager containers on
2026-08-23:
* whp01: 157 domain rows, 150 ssl_enabled, 71 distinct cert paths.
39 of those paths are referenced by MORE THAN ONE row, covering 118 of
the 150 SSL-enabled rows. Worst cases: brain-jar.com.pem and
arclightcourt.com.pem with 10 domains each, hackerpublicradio.org.pem
with 6, anhonesthost.com.pem with 5.
* whp02: 33 rows, 31 ssl_enabled, 15 distinct paths, 11 shared across 27
rows -- including threeworldsoneheart.org.pem, referenced by the apex,
its www, and mail.threeworldsoneheart.org. A production cleanup of that
mail.* row was stopped short precisely because removing it would have
unlinked the PEM the serving site is using.
So removing one domain unlinked a file up to nine other configured domains
were being served from. HAProxy binds the crt directory
(`bind ... ssl crt /etc/haproxy/certs`), so the loss is not noticed until
the next reload or restart, at which point the listener refuses to come up
or those names fall back to the wrong certificate.
`certbot delete` is the worse half. It destroys the lineage's archive, live
symlinks and renewal config; recovery is a fresh, rate-limited ACME order.
The old code passed `--cert-name <domain>`, which is also simply the wrong
lineage for a SAN member: 81 of whp01's 150 SSL-enabled rows have a cert
path whose basename is not their own domain, so for those the call was a
silent no-op -- while for a bundle PRIMARY it deleted the one lineage still
renewing the certificate every other name in the bundle is served with.
The fix refcounts, after the row is deleted so the query answers "who else
still needs this":
* lineage_name_for_cert_path() -- the lineage is the published bundle's
basename minus .pem, the same derivation _quarantine_superseded_certs()
already uses, not the domain being removed.
* domains_referencing_cert_path() / domains_referencing_lineage() -- the
remaining rows that name that file, and that lineage.
* remove_domain() unlinks only when the list is empty, `certbot delete`s
only when the list is empty, logs the retained names explicitly when it
skips, and reports them as certificate_retained_for /
lineage_retained_for in the API response.
ssl_enabled is deliberately not filtered on in the refcount: the two
mistakes are not symmetric. A stale PEM left in the crt directory costs a
few kilobytes; an unlinked live one is HTTPS down for every name it serves.
Cleanup is deferred, not cancelled -- removing the last name on a bundle
still unlinks the file and deletes the lineage.
No row on either production host has ssl_enabled=1 with an empty
ssl_cert_path, so the "no path, no attributable lineage" branch changes
nothing on the current fleet.
Tests (scripts/test-cert-write-safety.py, +9, suite now 31, all offline):
last reference -> file unlinked and lineage deleted; shared file survives
removal of a SAN member AND of the bundle primary, byte-for-byte, with the
edge still starting; shared lineage is not certbot-deleted; the production
mail.* shape; removing every name eventually cleans up; an unrelated
bundle is never collateral damage. Assertions are on os.path.exists, file
contents and the recorded certbot argv, never on which branch ran.
Mutation-tested, all five mutants killed:
1. guard absent entirely (suite run with HAPROXY_MANAGER_DIR pointed at
main) -> 6 failures, incl. "example.com is still configured and still
served from this file".
2. refcount taken before the row is deleted -> 5 failures, incl. "the
last reference is gone - now it may be removed".
3. certbot guard removed, file guard kept -> 3 failures, incl.
[] != ['delete --cert-name example.com --non-interactive'].
4. lineage taken from the domain name instead of the cert path -> 3
failures, incl. 'delete --cert-name example.com' != 'delete
--cert-name www.example.com'.
5. file-unlink guard removed, certbot guard kept -> 4 failures, incl.
"two sites are still served from this bundle".
Other suites unchanged and green: test-config-rollback, test-cert-scripts,
test-stick-table-contract, test-runtime-map-contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 5
|
c8d16b6990 |
fix(ip-blocking): the runtime map fast path has never once run
add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent `add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True whenever socat exited 0. Neither command has ever worked, on any deployment, for the entire life of the feature -- while logging "Added IP x to runtime map" every single time. Two independent defects: * NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map commands are worker commands. Captured verbatim on whp01: $ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli Unknown command: 'add', but maybe one of the following ones is a better match: @!<pid> : send a command to the <pid> process ... $ echo $? 0 socat exits 0 on the rejection, so `result.returncode == 0` was true. Same silence PR #7 fixed on the `show table` path. * `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move on every config regeneration -- `@1 show map` on whp01 reports blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0. Hardcoding any number is wrong; the map is referenced by FILE PATH, which is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0). And a third silence, which is why a response-body check alone is not enough here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer `Unknown map identifier.`. On the add path the reply is byte-for-byte identical to success. Only reading the entry back can tell them apart. IP blocking itself was never broken: update_blocked_ips_map() rewrites /etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it. That path is untouched and stays authoritative. What was broken is the no-reload fast path, plus every report that it had worked. * haproxy_manager.py: both functions send `@1 add|del map /etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with `get map` before returning True. runtime_map_lookup()/runtime_map_keys() are the read-back primitives. `sync_blocked_ips` loses `clear map #0` (which the master socket rejected just as loudly and just as invisibly) and verifies the whole set with one `show map` instead of counting commands that did not visibly complain; it answers 207 + `runtime_map_synced: false` when the runtime map does not match the database. * haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy answers those with nothing on success, so an empty body is the success and ANY non-empty body is a rejection. That is stricter than the marker list on purpose -- markers only recognise rejections someone has already seen, and it catches `'add map' expects three parameters ...`, which matches nothing. HaproxyCliError carries `.responses` so `del map` answering `Key not found.` (the requested end state) is told apart from a real failure without regex. * The four callers capture the boolean instead of discarding it and report `runtime_map_updated` / `runtime_map_failures` in the API response and the operation log. A runtime failure degrades to "enforced on the reload that already happens two lines later" -- never to an unblocked IP, never to a 500. * scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes on the wire (`@1` first, map by path, value `1`), classifies every captured response, and scans the repo's Python string literals and shell/template code lines for `#<id>` map references -- comments may describe the old form, code may not use it. Verified to fail on each defect reintroduced separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the- reply (1). * The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md and the comment in templates/hap_listener.tpl -- where every copy of it additionally omitted the `1`, which `-m int gt 0` needs to match. The only template change is a comment; `haproxy -c` on the live rendered config with it applied is clean (HAProxy 3.0.11, warnings unchanged). Verified on whp01 against the running container (docker cp + SIGHUP, no recreate). Before: both functions returned True and logged success while `@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed add lands with value "1" and the remove takes it out again; the old command form is now classified as a failure; a `#0` map reference returns False via the read-back. End to end through the API, `runtime_map_updated: true`, and /api/blocked-ips/sync -- which used to be a no-op reporting a full sync -- reports 264/264 verified present. The runtime path was isolated from the reload that normally follows it: with NO map-file write and NO reload (same haproxy worker pid throughout), adding 100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not in the is_local ranges) to the runtime map alone flipped a live site from HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path working for the first time. All test IPs were removed afterwards: 0 rows in blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer sites, the panel /health and `haproxy -c` are byte-identical to the baseline taken before the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 5
|
b6a62e7f9f |
fix(security-stats): stop reporting counters the stick tables never stored
/api/security/stats and scripts/show-tarpit-ips.sh reported "Scan Count",
"offense count" and BLOCKED/TARPITTED status parsed from gpc0/gpc1. No stick
table in this repo has ever stored a general-purpose counter -- the `web` table
stores conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s), and
the two brute-force tables store http_req_rate(60s). Every one of those figures
was fabricated, and an operator was making decisions on them.
Three independent silences kept it alive:
* `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
and the loop `continue`d -- so the endpoint always answered
`active_threats: 0` with an empty list. Live on whp01 it also reported
parts[0], the `0x...:` allocation pointer, as the source IP.
* The command was sent to /tmp/haproxy-cli WITHOUT the `@1` worker prefix.
That is the MASTER CLI socket, which answers "Unknown command: 'show' ..."
-- and socat still exits 0, so the `returncode != 0` guard never fired.
`total_tracked_ips` was the line count of that help text (8) while the real
table held 388 entries.
* The shell consumers wrote `gpc0=${gpc0:-0}`, rendering a field that does
not exist as a confident zero.
Report what the tables actually store, rather than adding gpc counters to make
the old semantics real. Adding them would mean editing hap_listener.tpl -- the
one change here with a silent-total-outage failure mode -- to rebuild
enforcement history that the edge access log (shipped 2026.08.8, on the host at
/var/log/haproxy.log) already records per request, with status codes,
termination states and request references the stick table could never hold.
* haproxy_manager.py: STICK_TABLE_FIELD_CONTRACT names what each table
stores. haproxy_cli() sends worker commands with `@1`, falls back to the
bare form for a plain stats socket, and inspects the RESPONSE BODY because
socat's exit status is worthless here. parse_stick_table_entry() reads
name=value / name(window_ms)=value pairs by NAME, never by position.
read_stick_table() RAISES -- naming the field -- when a row is missing a
contract field, instead of defaulting it to 0.
* /api/security/stats returns the four real counters with their windows, the
true `used:` count, and no invented threat_level/blocked/offense_count.
Fewer numbers, all of them real.
* scripts/show-edge-ip-rates.sh replaces the fabricated report; the four
expected fields are declared once as EXPECTED_FIELDS and drive the parser.
show-tarpit-ips.sh becomes a shim that explains why its numbers are gone
and points at where tarpit events actually live.
* monitor-attacks.sh loses fourteen fabricated "threat" categories and a
composite threat score, all permanently zero; its access-log section now
says the log is on the host instead of silently printing nothing.
* haproxy_tarpit_config.txt -- the never-shipped design sketch these counters
were copied from -- gets a NOT IMPLEMENTED banner.
* scripts/test-stick-table-contract.py (offline, 21 tests) holds the
templates' `store` clauses, STICK_TABLE_FIELD_CONTRACT and every consumer
to each other, and asserts each loud-failure path against the real captured
responses. Template and consumers can no longer drift apart quietly.
No template is touched, so haproxy.cfg is unchanged.
Verified on whp01: total_tracked_ips now tracks `used:` exactly (511 vs the
table's 511, was 8 vs 388), and per-IP values match `show table web key <ip>`
field for field. haproxy PIDs unmoved, `haproxy -c` warnings unchanged, five
customer sites HTTP 200.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Claude
|
711c670319 |
fix(logging): stop silently discarding every access log line
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m46s
haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.
The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.
Changes:
* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
/var/log/messages or the Graylog forwarder, bound to the bridge IP rather
than 0.0.0.0.
* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
standalone/home deployments on a different bridge subnet can retarget it.
* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
not sufficient for incident response -- it omits %ID entirely (verified
against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
for CF-fronted sites. The new format keeps the first 16 fields byte-identical
to the httplog default (so existing parsers still work) and appends
cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
Adds a User-Agent capture in slot 1 to feed it.
* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
It does not, which made the documented support-correlation workflow
(X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
supported when it could never have worked.
Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".
UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:
<150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
[22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 5
|
4f0949d534 |
fix(certs): stop the test suite bricking the container; make the pairing check real
Gate findings on fix/cert-write-safety. One blocker, one latent truncation
path, a false premise in a load-bearing comment, and a set of tests that were
passing without testing anything.
THE BLOCKER - scripts/test-cert-scripts.py bricked the production container
------------------------------------------------------------------------
The openssl-availability tests build a stripped PATH out of SYMLINKS to real
system binaries (cat, grep, mktemp, mv, cp, rm, mkdir, basename, dirname, find,
date, chmod). _cleanup_tmp() then walked the temp tree calling os.chmod(p,
0o600) - and os.chmod FOLLOWS SYMLINKS. Run once as root in the real image,
which is where this file ships (COPY scripts /haproxy/scripts) and where an
operator would most plausibly run it after deploying a cert fix, it stripped
the exec bit off twelve core binaries INCLUDING chmod itself, so it could not
be undone from inside the container:
/haproxy/scripts/cert-publish-lib.sh: line 109: /usr/bin/grep: Permission denied
bash: /usr/bin/chmod: Permission denied
Certificate publishing stayed dead until the container was recreated. It was
invisible on a workstation because an unprivileged chmod of a root-owned file
fails EPERM straight into `except OSError: pass` - which is also why the
advertised "32 tests pass" was only ever true off-container. In the image the
shipped file measured FAILED (failures=22, skipped=1). Cleanup now skips
symlinks; the suites are green in the image and the binaries survive.
S1 - the shell half had no same-filesystem guard
------------------------------------------------
The header claimed a cross-device mv would "FAIL LOUDLY and leave the live pem
alone". GNU mv does the opposite: across filesystems it copies, so it opens and
truncates the DESTINATION and only then discovers it cannot finish - measured
as a 204800-byte partial live.pem, sentinel gone, before mv reported ENOSPC.
cert_publish() now compares stat -Lc %d of the staging and certs dirs before
writing anything, mirroring the st_dev check the Python half already had, and
the comment says what mv actually does. Latent today (both dirs share a device
on all five hosts) but both are env-overridable.
openssl is present - correct the premise, make the check mandatory
------------------------------------------------------------------
Both halves justified a fail-open with "the image does not necessarily install
the openssl CLI". It does: openssl 3.5.6 in the running container, pulled in by
ca-certificates which certbot needs, and generate_self_signed_cert() already
shells to `openssl req` with check=True during setup. The 'unavailable' branch
never fired, so the pairing check has always run - and that, not the stated
reasoning, is what made the fail-open harmless. Structural validation alone is
weak: a bundle of EMPTY pem blocks passes every structural rule and is caught
only by openssl. The check is now mandatory in both halves and a missing binary
is a loud refusal. No `cryptography` fallback: the app runs on
/usr/local/bin/python3 (3.12) where it is not importable - it belongs to
Debian's /usr/bin/python3 - and reaching for that would be a second unverified
premise.
Smaller items
-------------
* cert_bundle_valid() read the file six times; a concurrent swap between two of
them made openssl x509 and openssl pkey judge different files and log a bogus
"private key does not match the certificate" into the monitored error log. It
now reads one snapshot and feeds openssl from it on stdin.
* except OSError -> except (OSError, UnicodeDecodeError): a BINARY-corrupt live
pem made backup_existing_pem() raise out of publish_pem_bundle() entirely, so
the republish that would have healed the host was the one thing that could
not run. The shell half recovers fine.
* stat -c %a on a symlinked live pem reports the LINK's 0777 and produced a
world-writable private key in the crt directory; now stat -Lc.
* The staging reaper's '*.??????' glob matched mktemp names but not the Python
side's '<name>.<random>.tmp', so those leaked forever. Matches both now.
* renew-certificates.sh and sync-certificates.sh exited 0 even when every
domain failed to publish, so "0 updated, 12 failed" looked identical to a
clean run to cron, to host-renew-certificates.sh (which branches on it) and
to monitoring - a host could silently stop publishing renewals until the
certificates expired. They now exit 1 if any domain failed, still after
publishing the ones that worked.
Test-quality
------------
Four TestCertPublishLibrary tests passed with cert-publish-lib.sh DELETED -
they asserted only rc != 0, and `command not found` is 127. All four now assert
the rejection REASON via assert_rejected(), and setUp() fails if the library is
missing. test_missing_openssl_still_rejects... is replaced by
test_empty_pem_blocks_are_rejected, which pins the case that makes the pairing
check necessary.
Also: the staging-containment test used startswith(certs + os.sep), so
cert_staging_dir() returning the crt directory ITSELF - the exact hazard -
still passed; test_successful_renewal_still_publishes built its "renewed" cert
with a no-op .replace() and could not tell a renewal that published nothing;
test_no_temp_file_survives_a_failed_publish failed before the staging dir
existed and asserted [] == []; FIX_ONLY was skipUnless(hasattr(hm,
'publish_pem_bundle')), so renaming that function turned 10 of 17 tests into
skips while the run still printed OK. Each is fixed and each fix is
mutation-proved: the mutation that the old assertion waved through now fails.
File mode is pinned in both suites (it was pinned nowhere), and the rename
failure is injected with a stub mv instead of chmod 0500, which root ignored -
so that test no longer skips itself precisely where it matters.
Verification
------------
IN THE BUILT IMAGE, as root (the acceptance bar):
scripts/test-cert-scripts.py 38 tests, OK, 0 skipped
scripts/test-cert-write-safety.py 22 tests, OK, 0 skipped
scripts/test-config-rollback.py 17 tests, OK (neighbour, unchanged)
Workstation: 38/38 OK for the shell suite; the Python suite needs flask.
Against the pre-fix tree (main): shell 38 failures; python failures=4, errors=1,
skipped=15 - the FIX_ONLY skips are the 14 fix-only tests plus the API guard.
The old python suite against the pre-fix tree measures skipped=10, confirming
the gate's count.
32 mutation checks, all behaving as intended: every fix breaks a test when
reverted, and every rewritten test fails under the mutation its predecessor
passed. py_compile clean, bash -n clean, shellcheck clean, no new pyflakes
warnings (same 4 pre-existing).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 5
|
22dab685d0 |
fix(certs): never truncate a live PEM; validate before publishing
Every path that refreshed a combined certificate did
with open(combined_path, 'w') as combined: # TRUNCATES
subprocess.run(['cat', cert, key], stdout=combined) # rc ignored
where combined_path is the bundle HAProxy is currently serving. The live file
was emptied before any source material had been read and the cat status was
never checked, so an unreadable source, a zero-length privkey, a full disk or a
killed container left a truncated or key-less PEM in place. HAProxy loads
/etc/haproxy/certs as a directory, so one unusable file there fails the whole
ssl bind - HTTPS down for every site on the host, and unlike a bad haproxy.cfg
it is not recoverable by config rollback.
The bundle endpoint compounded it: superseded .pem files were unlinked and
their lineages `certbot delete`d before anything had checked the replacement,
destroying both copies of a working certificate. Recovery there means fresh,
rate-limited ACME orders.
Bundles are now assembled in a staging directory beside the crt directory (never
inside it - HAProxy would try to load a temp file), validated there, and swapped
in with os.replace(). Validation is mandatory structural checks in pure Python
plus a best-effort openssl key/leaf pairing check; a missing openssl warns
loudly and does not silently pass. The previous bundle is copied to
/etc/haproxy/cert-backups first. Superseded certs are moved aside rather than
deleted, and `certbot delete` runs only after HAProxy has reloaded onto the
replacement. The same guarantees are implemented for the cron/renewal shell
path in scripts/cert-publish-lib.sh, which also gates the reload on `haproxy -c`.
Reuses write_config_atomically() from the config-rollback fix (extended with
staging_dir/validate) rather than adding a second atomic writer.
Also heals a zero-byte QUIC cluster-secret file, which previously made
get_or_create_cluster_secret() return '' forever.
Tests: scripts/test-cert-write-safety.py (17) and scripts/test-cert-scripts.py
(32) in the existing stdlib-unittest, stub-binary convention. Both bug classes
reproduce against the pre-fix tree via HAPROXY_MANAGER_DIR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 5
|
233044fb1d |
fix(config): write blocked_ips.map atomically; close review gaps
Follow-up to
|
||
|
shadowdao
|
9d16151120 |
fix(config): take the rollback backup BEFORE writing the new config
generate_config() wrote /etc/haproxy/haproxy.cfg and only then called
reload_haproxy_safely(), which called create_backup(). The "backup" was
therefore a copy of the config that had just been written, so on a validation
failure restore_backup() restored the identical broken bytes: the advertised
rollback was a no-op and a fatal haproxy.cfg stayed on disk, where
start_haproxy() refuses to launch. Same shape as the June 2026 incident where
a missing template produced a fatal config and took an edge down.
Reproduced end to end before the fix (invalid config generated -> "Backups
created successfully" -> "Backups restored successfully" -> haproxy.cfg on
disk still invalid, `haproxy -c` rc=1).
Changes:
* create_backup() is now called by generate_config() BEFORE the first write,
which also covers blocked_ips.map (rewritten early in generate_config) and
coraza-spoe.cfg - both previously written before the backup and, for the
SPOE file, never backed up at all even though `haproxy -c` parses it.
* create_backup() refuses to promote a config HAProxy already rejects, so a
broken file on disk cannot overwrite a known-good backup ("rollback" must
not mean "restore a different broken config"). It returns (ok, status) so
the caller knows whether a rollback target exists.
* promote_current_config_to_backup() records the config as known-good only
after it has validated AND loaded, so a box whose first generation succeeded
has a rollback target immediately, and a config that never loaded is never
promoted.
* restore_backup() returns (restored, message) and distinguishes "no backup
available" from "restored". Every caller now surfaces the difference; a
failed rollback is logged CRITICAL and reported as ROLLBACK FAILED in the
API error message instead of silently looking like a successful recovery.
* reload_haproxy_safely(backup_status=...) no longer takes its own backup - it
runs after the write, where a backup is meaningless. Called without a status
it logs the contract violation rather than overwriting a good backup.
* validate_config_file() separates "config is invalid" from "validator could
not run" so a missing haproxy binary is not read as a bad config.
* Config writes are atomic (temp file + fsync + os.replace, mode preserved);
a truncated haproxy.cfg is as fatal as an invalid one. Removes the dead
temp_config_path variable whose comment claimed this already happened.
* Fast path: if the live config set is already byte-identical to the backup
(the normal case after a successful reload), skip the re-validation and the
copy, so this adds no `haproxy -c` latency to customer-facing API calls.
Tests: scripts/test-config-rollback.py - 17 self-contained stdlib-unittest
tests, no new dependencies (the repo has no Python test framework; the
existing scripts/test-*.sh are curl integration scripts). A stub `haproxy`
binary stands in for the validator. 12 of the 17 fail against the previous
code; every assertion was mutation-proven (9 mutations, each reddening only
the tests that cover it).
No template, QUIC or HTTP/3 changes.
|
||
|
shadowdaoandClaude Opus 4.8
|
2a2b9739fc |
fix(api): bound all subprocess calls + run 2 workers to prevent API stall
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m9s
The management API wedged on whp01 2026-07-07: every panel call to the manager (config regenerate during a WHP site update, SSL, even /health) timed out at 30s while customer sites stayed up. Root cause: all four gunicorn gthread worker threads were permanently blocked in socket reads inside untimed subprocess.run() calls (certbot ACME / socat reloads). A stalled external command holds its worker thread forever; gunicorn --timeout can't rescue it (gthread only kills a worker whose main thread stops heart-beating, and ours kept polling). Stalled calls accumulated until the 4-thread pool was exhausted and the whole API went dark. - Wrap subprocess.run with a default timeout (HAPROXY_MGR_SUBPROCESS_TIMEOUT, 180s) so every external command is bounded and releases its thread on expiry via the existing per-endpoint try/except. Bounding by default covers all ~30 call sites and any future one. - certbot renew keeps an explicit 900s timeout (walks every lineage). - API_WORKERS default 1 -> 2: a single worker made a thread-pool wedge a total outage; a second worker keeps the API answering while one recycles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.8
|
7732e2a2ff |
chore(log): downgrade "no backend name" domain-skip from WARNING to INFO
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 2m12s
generate_config emits "Skipping domain <host> - no backend name" on every run
for domains registered without a proxy backend — most commonly the panel's own
hostname (whpNN.cloud-hosting.io), which lives in the DB only for certificate
management and intentionally has no backend. Logging it at WARNING tripped the
WHP AI log monitor as a recurring error and prompted a bogus "restart
haproxy-manager" remediation. It's expected, benign, and recurs by design.
Log it at INFO instead (consistent with the sibling per-domain "Added ACL for
domain" INFO lines) with a clearer message ("no proxy backend
(cert/management-only)"). Verified against the WHP monitor's ErrorClassifier:
the old WARNING line classified as non_critical (captured); the new INFO line
classifies as None (skipped) — so it no longer shows up in reports, while
remaining visible in container logs for manual routing debugging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 4.8
|
d9cc5311de |
feat(quic): enable HTTP/3 over QUIC on the edge + versioned images
HTTP/3 is config-only — the Debian haproxy package is built +QUIC via the OpenSSL compat shim. Changes: - hap_header.tpl: `limited-quic` (required to enable QUIC binds under the compat layer) + self-healing `cluster-secret` for QUIC token derivation. - hap_listener.tpl: `bind quic4@:443 ... alpn h3` in the shared frontend (so real-IP/rate-limit/IP-block/Coraza rules apply to H3 too) + alt-svc header. - Dockerfile/README: publish/document 443/udp; stamp image.version from VERSION. - CI: tag :latest + :<VERSION> + :<sha> so there's a pinnable rollback target. No 0-RTT (compat-layer limitation). Validated end-to-end on a standalone edge: config parses, UDP/443 binds, alt-svc advertised, real curl --http3 -> HTTP/3. Container must run with `-p 443:443/udp` + host UDP/443 open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.7
|
131284fd0c |
refactor(suspension): serve via /suspended route on default-backend, drop bk_suspended
The previous design used a separate whp-suspended container (nginx:alpine serving a static 503 page) reachable via a dedicated bk_suspended backend. That was over-engineered — haproxy-manager-base already ships a default-app Flask server on :8080 that serves /default-page and /blocked-ip via path-rewrite ACLs. Mirroring that pattern lets the suspension page live in the SAME container, no extra image to build, no extra container to run/health-monitor. Changes: - Add /suspended Flask route on default_app returning 503 + suspended_page.html - Add templates/suspended_page.html (dark-themed 503 page) - hap_listener.tpl: 'http-request set-path /suspended' + 'use_backend default-backend' when host is in suspended_domains.list (same pattern as is_blocked_ip) - Rename env var from HAPROXY_SUSPENSION_BACKEND (a target hostport) to HAPROXY_SUSPENSION_ENABLED (a bool); accepts 1/true/yes/on (case-insensitive) - Remove hap_suspended_backend.tpl and its rendering in generate_config Non-WHP deployments (env var unset) see byte-identical haproxy.cfg as before (verified via jinja2 render diff). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.7
|
c4a9d9d7e6 |
feat(suspension): opt-in routing for suspended hosts via bk_suspended backend
Adds a new env var HAPROXY_SUSPENSION_BACKEND (default unset). When set (e.g. "whp-suspended:80"), generate_config() renders: - A bk_suspended backend pointing at the configured upstream - An ACL `acl is_suspended_domain hdr(host),lower -f /etc/haproxy/suspended_domains.list` + `use_backend bk_suspended if is_suspended_domain` in the frontend, sitting after IP-blocking and before any per-domain routing - An empty /etc/haproxy/suspended_domains.list if missing (haproxy refuses to start with -f pointing at a non-existent file) External tooling (e.g. WHP's site_disable.php) maintains the list via `docker cp` and HUP-reloads the container. Non-WHP deployments (home networks, standalone use) leave the env var unset and see byte-identical haproxy.cfg output. Same opt-in shape as the existing HAPROXY_CORAZA_SPOE_BACKEND integration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.7
|
69bed61697 |
fix(coraza-spoe): collapse args to one line + ensure trailing LF on spoe cfg
Two HAProxy parse errors caught in staging functional test: 1. coraza-spoe.cfg:39 'args': missing fetch method The args directive had backslash line continuations. HAProxy doesn't support those in SPOE configs — args must be one physical line. Collapsed to a single line. 2. coraza-spoe.cfg:50 Missing LF on last line Same trailing-LF issue we hit on haproxy.cfg one commit ago. The Jinja2 template ends with content rather than a newline, and write() doesn't add one. Belt-and-suspenders: explicitly append '\n' before writing if not already there. After this commit HAProxy validates the generated config cleanly. Will verify on staging now (combined SPOE injection + fail-open + active attack-detection tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.7
|
fed3457f29 |
fix(coraza): ensure haproxy.cfg ends with LF when SPOE backend appended
The SPOE backend block from hap_coraza_spoa_backend.tpl was being appended
last to config_parts. The template's render output doesn't end with a
newline (and config_parts is joined with '\n' BETWEEN elements, not after
the last one), so the resulting haproxy.cfg ended on `server coraza-spoa
...` with no trailing LF. HAProxy refuses to parse such files:
[ALERT] config: parsing [/etc/haproxy/haproxy.cfg:288]: Missing LF
on last line, file might have been truncated at position 70.
Match the existing pattern at the previous-last config_parts.append
(line 1850 uses `'\n'.join(config_backends) + '\n'`) and add an explicit
'\n' on the coraza block append.
Caught immediately on staging: HTTP 000 to localhost:80 because HAProxy
never started; gunicorn/management API kept serving on :8000 fine.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 4.7
|
a89cdab886 |
PR 2/3: opt-in SPOE integration for Coraza WAF
Adds the plumbing that lets haproxy-manager talk to the coraza-spoa sidecar
added in PR 1, while keeping the default behavior bit-identical for any
deployment that doesn't set the new env var (the home network / standalone
use cases).
Single gate: HAPROXY_CORAZA_SPOE_BACKEND env var on the haproxy-manager
container. Unset (default) = generate_config() renders zero SPOE-related
output. Set (e.g. "coraza-spoa:9000") = three things happen at config
generation time:
1. hap_listener.tpl injects 5 lines at the end of the frontend block:
filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
http-request send-spoe-group coraza coraza-check
...placed AFTER rate-limit and IP-block guards so we don't waste WAF
calls on requests we were going to drop anyway.
2. A new TCP backend (hap_coraza_spoa_backend.tpl) is appended:
backend coraza-spoa-backend
mode tcp
server coraza-spoa <env-var-target> check ...
3. The SPOE engine config (hap_coraza_spoe_engine.tpl) is rendered and
written to /etc/haproxy/coraza-spoe.cfg, defining the spoe-agent
"coraza" + spoe-message "coraza-check". This sets:
- option set-on-error continue (FAIL-OPEN if SPOA is unreachable)
- timeout processing 100ms (per-request inspection budget)
- app=str(haproxy) (matches sidecar's application name)
Verification (template render only, before staging deploy):
- hap_listener.tpl with no env var: 55 lines, zero SPOE references
- hap_listener.tpl with env var: 62 lines, filter + send-spoe-group present
- Engine cfg + backend block render with correct agent_target substitution
Next: PR 3 wires this into WHP (sidecar deploy via container-manager.sh
extension, server-settings UI for on/off, AI Monitor source for the audit
log). Staging verification of PR 1 + PR 2 together happens after PR 3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 4.7
|
c22d2cd1f4 |
swap werkzeug dev server for gunicorn + accept all HTTP methods on default/blocked pages
Two related fixes for the issues the AI Monitor surfaced on whp01 on 2026-05-12 (haproxy-manager going "healthy but stalled" after long uptime, and noise from POST /blocked-ip returning 405): 1. Production WSGI server. The Flask app was running on werkzeug's built-in dev server (the one that prints "WARNING: This is a development server" on every startup). werkzeug is single-threaded and accumulates worker state over long uptimes; after ~24h on whp01 the health endpoint stops responding while the container still reports "healthy" because Docker's HEALTHCHECK uses an HTTP probe from inside the same werkzeug process that's stalled. Replace with gunicorn (gthread worker class, --max-requests=1000 with jitter so workers recycle periodically). Two gunicorn instances, one per Flask app — port 8000 for the management API, port 8080 for the default/blocked-ip page server. Both lift their app objects from the haproxy_manager module so gunicorn can import them. Required structural change: default_app was created INSIDE the __name__ == '__main__' block at module bottom, where gunicorn could never reach it. Moved to module level. The __main__ block now stays only for `python haproxy_manager.py` local-dev workflow. Container init (init_db, certbot register, generate_config, start_haproxy) extracted into a do_initial_setup() function called from a new scripts/init.py. start-up.sh runs init.py to completion before either gunicorn binds, which keeps HAProxy startup off the WSGI workers' fork paths (no race between workers all trying to start_haproxy() at once). 2. /blocked-ip and / accept ALL methods. HAProxy proxies blocked-IP traffic to default_app preserving the original verb, so a blocked POST request used to hit Flask's GET-only route and get a 405 + the AI Monitor flagged the noise. Adding the full method list lets the 403 page render regardless of verb. Gunicorn settings tunable via env (workers, timeout, max-requests). API gets --timeout 120 because ACME cert issuance can be slow; the default page server stays on the gunicorn default 30s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.7
|
be3bc040df |
feat: clear stale certbot lock files before each ACME run + at startup
certbot uses fasteners (fcntl-based locking) to serialize concurrent invocations. The kernel auto-releases fcntl locks when the holding process exits, but the .certbot.lock FILES persist on disk — and we've seen real cases where subsequent runs report "Another instance of Certbot is already running" even when no certbot process is alive. Observed during the 2026-05-09 bundling rollout when a hung worker held a lock across container-internal Python crashes. When SSL is blocked on a customer site, this is high-impact: the certbot lock can sit stale until somebody manually deletes it. clear_stale_certbot_locks(): - probes each known lock path with fcntl.LOCK_NB - if the lock is unheld → file is stale → delete it - if the lock IS held → leave it alone (real certbot is running) Wired in: - container startup (init block) - /api/ssl single-domain handler - /api/ssl/bundle handler - /api/certificates/renew handler Safe to call repeatedly; never deletes a lock a real process holds, so can never trigger concurrent certbot runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.7
|
873048e837 |
feat(api/ssl/bundle): clean up superseded lineages after issuance
The bundle endpoint correctly issued multi-SAN certs but left old single-SAN .pem files (e.g. <name>-0001.pem) in /etc/haproxy/certs/. HAProxy's `bind ... ssl crt /etc/haproxy/certs` loads everything in the directory and picked the alphabetically-first matching file — typically the older single-SAN one — so the new bundle had no effect on what was served. Repro on peptidesaver.net: bundle covered 4 SANs but HAProxy kept serving peptidesaver.net-0001.pem (single SAN, April-issued). After a successful bundle write, walk SSL_CERTS_DIR and remove any .pem whose CN is in the new bundle's name list (excluding the bundle's own combined file). Drop the matching certbot lineage with `certbot delete --cert-name <X> -n` so `certbot renew` stops touching the dead lineage too. Returns a `cleanup` summary in the API response so callers can log / display what was deleted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.7
|
c931e54d4d |
feat(api): add /api/ssl/bundle for per-site SAN cert issuance
WHP's renewal orchestrator now bundles a site's domains into one cert
covering all SANs, instead of N separate single-domain orders. Single
ACME order = better behavior under Let's Encrypt's 50/hour orders limit
when many domains need attention at once.
Endpoint: POST /api/ssl/bundle
Body: {"primary": "example.com", "sans": ["www.example.com", ...]}
- Uses --cert-name <primary> so the lineage stays stable across renewals
(no -0001/-0002 proliferation seen with the legacy single-domain flow).
- Single combined .pem at /etc/haproxy/certs/<primary>.pem; HAProxy SNI-
matches against the cert's SAN list, so one file serves all included
hostnames.
- Updates the domains table for every SAN in the bundle.
- Hard cap at 100 SANs (LE limit).
Existing /api/ssl single-domain endpoint kept for backwards compat.
The WHP haproxy_manager::bundleSSL() helper falls back to a per-domain
loop if /api/ssl/bundle returns 404, so the WHP side keeps working
during the rolling image upgrade window.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 4.6
|
124a5373d2 |
Fix wildcard SSL cert: find certbot -NNNN dirs and use _wildcard_ filename
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m1s
Add find_certbot_live_dir() helper to locate the most recent certbot live directory for a domain, handling -NNNN suffixed dirs from repeated requests. Fix combined cert filename from *.domain.pem to _wildcard_.domain.pem. Apply the helper across all SSL endpoints (request, renew, verify, download). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.6
|
657cd28344 |
Fix certbot hook script paths and add logging
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m4s
Hook scripts are at /haproxy/scripts/ inside the container (per Dockerfile COPY), not /app/scripts/. Also added logging of certbot stdout/stderr so failures are visible in haproxy-manager.log. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.6
|
91c92dd07e |
Add wildcard domain support with DNS-01 ACME challenge flow
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m17s
Support wildcard domains (*.domain.tld) in HAProxy config generation with exact-match ACLs prioritized over wildcard ACLs. Add DNS-01 challenge endpoints that coordinate with certbot via auth/cleanup hook scripts for wildcard SSL certificate issuance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
shadowdao
|
1fcb25bb88 |
Update SQL logic to update instead of delete and re-add
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m18s
|
||
|
shadowdaoandClaude
|
1d22d789b8 |
Simplify certificate renewal scripts and add certbot cleanup
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 59s
Simplified all certificate renewal scripts to be more straightforward and reliable: - Scripts now just run certbot renew and copy cert+key files to HAProxy format - Removed overly complex retry logic and error handling - Both in-container and host-side scripts work with cron scheduling Added automatic certbot cleanup when domains are removed: - When a domain is deleted via API, certbot certificate is also removed - Prevents renewal errors for domains that no longer exist in HAProxy - Cleans up both HAProxy combined cert and Let's Encrypt certificate Script changes: - renew-certificates.sh: Simplified to 87 lines (from 215) - sync-certificates.sh: Simplified to 79 lines (from 200+) - host-renew-certificates.sh: Simplified to 36 lines (from 40) - All scripts use same pattern: query DB, copy certs, reload HAProxy Python changes: - remove_domain() now calls 'certbot delete' to remove certificates - Prevents orphaned certificates from causing renewal failures 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
shadowdao
|
71f4b9ef05 |
Add CIDR notation support for IP blocking
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 2m1s
- Update map file format to include value (IP/CIDR 1) - Fix HAProxy template to use map_ip() for CIDR support - Update runtime map commands to include value - Document CIDR range blocking in API documentation - Support blocking entire network ranges (e.g., 192.168.1.0/24) This allows blocking compromised ISP ranges and other large-scale attacks. |
||
|
shadowdaoandClaude
|
65248680a5 |
Fix HAProxy 3.0.11 compatibility issues
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m54s
Major syntax and configuration updates for HAProxy 3.0.11: Configuration Fixes: - Remove conflicting stick-table declarations in frontend - Move security tables to separate backend sections - Fix ACL syntax errors (missing_browser_headers → separate ACLs) - Remove unsupported add-var() syntax - Simplify threat scoring to use flags instead of cumulative values Security Table Architecture: - security_blacklist: 24h persistent offender tracking - wp_403_track: WordPress authentication failure monitoring - Separated from main frontend table to avoid conflicts Simplified Threat Detection: - low_threat: Rate abuse, suspicious methods, missing headers - medium_threat: SQL injection, directory traversal, WordPress brute force - high_threat: Bot scanners, admin scans, shell attempts - critical_threat: Blacklisted IPs, auto-blacklist candidates Response System: - Low threat: Warning headers only - Medium threat: Tarpit delays - High threat: Immediate deny (403) - Critical threat: Blacklist and deny Enhanced Compatibility: - Removed HAProxy 2.6-specific syntax - Updated to HAProxy 3.0.11 requirements - Maintained security effectiveness with simpler logic - Added security tables template integration The system maintains comprehensive protection while being compatible with HAProxy 3.0.11's stricter parsing and syntax requirements. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
shadowdaoandClaude
|
e2f350ce95 |
Add comprehensive anti-scan and brute force protection
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 54s
Implement multi-layered security system to protect against exploit scanning and brute force attacks while maintaining legitimate traffic flow. Security Features: - Attack detection for common exploit paths (WordPress, phpMyAdmin, shells) - Malicious user agent filtering (sqlmap, nikto, metasploit, etc.) - SQL injection and directory traversal pattern detection - Progressive rate limiting (50 req/10s, 20 conn/10s, 10 err/10s) - Three-tier response: tarpit → deny → repeat offender blocking - Strict authentication endpoint protection (5 req/10s limit) - Real IP detection through proxy headers (Cloudflare, X-Real-IP) Management Tools: - manage-blocked-ips.sh: Dynamic IP blocking/unblocking - monitor-attacks.sh: Real-time threat monitoring - API endpoints for security stats and temporary blocking - Auto-expiring temporary blocks with cleanup endpoint HAProxy 2.6 Compatibility: - Removed silent-drop (not available in 2.6) - Fixed stick table counter syntax - Using standard tarpit and deny actions 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
shadowdaoandClaude
|
2406d9f995 |
Add 403 status to blocked IP page and reload HAProxy on IP block/unblock
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 51s
- Modified /blocked-ip route to return 403 Forbidden status with HTML page - Added HAProxy reload after adding blocked IP to ensure consistency - Added HAProxy reload after removing blocked IP to ensure consistency - Includes error handling for reload failures without breaking the operation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
shadowdaoandClaude
|
7869b81f27 |
CRITICAL FIX: Migrate HAProxy IP blocking from ACL to map files
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 51s
**Problem Solved:** - HAProxy ACL 64-word limit caused config parsing failures - "too many words, truncating after word 64" error - Complete service outage when >64 IPs were blocked - Error: "no such ACL : 'is_blocked'" broke all traffic routing **Solution: HAProxy Map Files (v1.6+)** - ✅ Unlimited IP addresses (no word limits) - ✅ Runtime updates without config reloads - ✅ Better performance (hash table vs linear search) - ✅ Safer config management with validation & rollback **Technical Implementation:** **Map File Integration:** - `/etc/haproxy/blocked_ips.map` stores all blocked IPs - `http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }` - Runtime updates: `echo "add map #0 IP" | socat stdio /var/run/haproxy.sock` **Safety Features Added:** - `create_backup()` - Automatic config/map backups before changes - `validate_haproxy_config()` - Config validation before applying - `restore_backup()` - Automatic rollback on failures - `reload_haproxy_safely()` - Safe reload with validation pipeline **Runtime Management:** - `update_blocked_ips_map()` - Sync database to map file - `add_ip_to_runtime_map()` - Immediate IP blocking without reload - `remove_ip_from_runtime_map()` - Immediate IP unblocking **New API Endpoints:** - `POST /api/config/reload` - Safe config reload with rollback - `POST /api/blocked-ips/sync` - Sync database to runtime map **Template Changes:** - Replaced ACL method: `acl is_blocked src IP1 IP2...` (64 limit) - With map method: `http-request deny if { src -f blocked_ips.map }` (unlimited) **Backwards Compatibility:** - Existing API endpoints unchanged (GET/POST/DELETE /api/blocked-ips) - Database schema unchanged - Automatic migration on first config generation **Performance Improvements:** - O(1) hash table lookups vs O(n) linear ACL search - No config reloads needed for IP changes - Supports millions of IPs if needed - Memory efficient external file storage **Documentation:** - Complete migration guide in MIGRATION_GUIDE.md - Updated API documentation with new endpoints - Runtime management examples - Troubleshooting guide **Production Safety:** - All changes include automatic backup/restore - Config validation prevents bad deployments - Runtime updates avoid service interruption - Comprehensive error logging and monitoring This fixes the critical production outage caused by ACL word limits while providing a more scalable and performant IP blocking solution. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
shadowdaoandClaude
|
ca37a68255 |
Add IP blocking functionality to HAProxy Manager
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m1s
- Add blocked_ips database table to store blocked IP addresses - Implement API endpoints for IP blocking management: - GET /api/blocked-ips: List all blocked IPs - POST /api/blocked-ips: Block an IP address - DELETE /api/blocked-ips: Unblock an IP address - Update HAProxy configuration generation to include blocked IP ACLs - Create blocked IP page template for denied access - Add comprehensive API documentation for WHP integration - Include test script for IP blocking functionality - Update .gitignore with Python patterns - Add CLAUDE.md for codebase documentation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
shadowdao
|
d4f54aef35 |
Fix HAProxy crash loop and improve startup resilience
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 36s
- Add configuration regeneration before HAProxy startup - Add configuration validation before starting HAProxy - Add automatic configuration regeneration if invalid config detected - Prevent container crashes when HAProxy fails to start - Allow container to continue running even if HAProxy is not available - Add better error handling and logging for startup issues |
||
|
shadowdao
|
fac6cef0db |
Fix HAProxy 2.6 compatibility for default backend
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 37s
- Replace http-response set-body (HAProxy 2.8+) with local server approach - Add separate Flask server on port 8080 to serve default page - Update default backend template to use local server instead of inline HTML - Maintain all customization features via environment variables - Fix JavaScript error handling for domains API response |
||
|
shadowdao
|
27f3f8959b |
Add default backend page for unmatched domains
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 37s
- Add default backend template (hap_default_backend.tpl) - Add customizable default page HTML template (default_page.html) - Modify generate_config() to include default backend for unmatched domains - Add environment variables for customizing default page content: - HAPROXY_DEFAULT_PAGE_TITLE - HAPROXY_DEFAULT_MAIN_MESSAGE - HAPROXY_DEFAULT_SECONDARY_MESSAGE - Update README with documentation and examples - Ensure backward compatibility with existing configurations - Remove email contact link as requested |
||
|
shadowdao
|
ef488a253d |
Add /api/certificates/request endpoint for programmatic certificate requests, update docs and add test script
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 37s
|
||
|
shadowdao
|
7b0b4c0476 |
Major upgrade: API key authentication, certificate renewal/download endpoints, monitoring/alerting scripts, improved logging, and documentation updates. See UPGRADE_SUMMARY.md for details.
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 43s
|
||
|
shadowdao
|
7550df9890 | Fixing reload issue | ||
|
shadowdao
|
8ae1a6b99f |
debug reload
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m9s
|
||
|
shadowdao
|
9de12c72de |
added missing return
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 58s
|
||
|
shadowdao
|
cb58f1d762 |
Switch reload from post to get
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 35s
|
||
|
shadowdao
|
2492eab708 |
Fix missing '/'
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 35s
|