main
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  shadowdaoandClaude Opus 5
|
b072786192 |
fix(domain-removal): stop deleting certificates other live sites are served from
DELETE /api/domain ran, unconditionally, for any ssl_enabled row:
os.remove(ssl_cert_path)
certbot delete --cert-name <domain> --non-interactive
`domains.domain` is UNIQUE. `domains.ssl_cert_path` is not, and nothing
anywhere guarded against two rows naming the same file. Sharing is not an
edge case -- it is the normal shape of the table, because
request_ssl_bundle() deliberately creates it: one SAN certificate is issued
as `--cert-name <primary>`, published once to
/etc/haproxy/certs/<primary>.pem, and then EVERY included name's row is
pointed at that same path ("Mark every name in the bundle as ssl_enabled,
all pointing at the same combined .pem").
Measured read-only on the live SQLite in the haproxy-manager containers on
2026-08-23:
* whp01: 157 domain rows, 150 ssl_enabled, 71 distinct cert paths.
39 of those paths are referenced by MORE THAN ONE row, covering 118 of
the 150 SSL-enabled rows. Worst cases: brain-jar.com.pem and
arclightcourt.com.pem with 10 domains each, hackerpublicradio.org.pem
with 6, anhonesthost.com.pem with 5.
* whp02: 33 rows, 31 ssl_enabled, 15 distinct paths, 11 shared across 27
rows -- including threeworldsoneheart.org.pem, referenced by the apex,
its www, and mail.threeworldsoneheart.org. A production cleanup of that
mail.* row was stopped short precisely because removing it would have
unlinked the PEM the serving site is using.
So removing one domain unlinked a file up to nine other configured domains
were being served from. HAProxy binds the crt directory
(`bind ... ssl crt /etc/haproxy/certs`), so the loss is not noticed until
the next reload or restart, at which point the listener refuses to come up
or those names fall back to the wrong certificate.
`certbot delete` is the worse half. It destroys the lineage's archive, live
symlinks and renewal config; recovery is a fresh, rate-limited ACME order.
The old code passed `--cert-name <domain>`, which is also simply the wrong
lineage for a SAN member: 81 of whp01's 150 SSL-enabled rows have a cert
path whose basename is not their own domain, so for those the call was a
silent no-op -- while for a bundle PRIMARY it deleted the one lineage still
renewing the certificate every other name in the bundle is served with.
The fix refcounts, after the row is deleted so the query answers "who else
still needs this":
* lineage_name_for_cert_path() -- the lineage is the published bundle's
basename minus .pem, the same derivation _quarantine_superseded_certs()
already uses, not the domain being removed.
* domains_referencing_cert_path() / domains_referencing_lineage() -- the
remaining rows that name that file, and that lineage.
* remove_domain() unlinks only when the list is empty, `certbot delete`s
only when the list is empty, logs the retained names explicitly when it
skips, and reports them as certificate_retained_for /
lineage_retained_for in the API response.
ssl_enabled is deliberately not filtered on in the refcount: the two
mistakes are not symmetric. A stale PEM left in the crt directory costs a
few kilobytes; an unlinked live one is HTTPS down for every name it serves.
Cleanup is deferred, not cancelled -- removing the last name on a bundle
still unlinks the file and deletes the lineage.
No row on either production host has ssl_enabled=1 with an empty
ssl_cert_path, so the "no path, no attributable lineage" branch changes
nothing on the current fleet.
Tests (scripts/test-cert-write-safety.py, +9, suite now 31, all offline):
last reference -> file unlinked and lineage deleted; shared file survives
removal of a SAN member AND of the bundle primary, byte-for-byte, with the
edge still starting; shared lineage is not certbot-deleted; the production
mail.* shape; removing every name eventually cleans up; an unrelated
bundle is never collateral damage. Assertions are on os.path.exists, file
contents and the recorded certbot argv, never on which branch ran.
Mutation-tested, all five mutants killed:
1. guard absent entirely (suite run with HAPROXY_MANAGER_DIR pointed at
main) -> 6 failures, incl. "example.com is still configured and still
served from this file".
2. refcount taken before the row is deleted -> 5 failures, incl. "the
last reference is gone - now it may be removed".
3. certbot guard removed, file guard kept -> 3 failures, incl.
[] != ['delete --cert-name example.com --non-interactive'].
4. lineage taken from the domain name instead of the cert path -> 3
failures, incl. 'delete --cert-name example.com' != 'delete
--cert-name www.example.com'.
5. file-unlink guard removed, certbot guard kept -> 4 failures, incl.
"two sites are still served from this bundle".
Other suites unchanged and green: test-config-rollback, test-cert-scripts,
test-stick-table-contract, test-runtime-map-contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 5
|
c8d16b6990 |
fix(ip-blocking): the runtime map fast path has never once run
add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent `add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True whenever socat exited 0. Neither command has ever worked, on any deployment, for the entire life of the feature -- while logging "Added IP x to runtime map" every single time. Two independent defects: * NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map commands are worker commands. Captured verbatim on whp01: $ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli Unknown command: 'add', but maybe one of the following ones is a better match: @!<pid> : send a command to the <pid> process ... $ echo $? 0 socat exits 0 on the rejection, so `result.returncode == 0` was true. Same silence PR #7 fixed on the `show table` path. * `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move on every config regeneration -- `@1 show map` on whp01 reports blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0. Hardcoding any number is wrong; the map is referenced by FILE PATH, which is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0). And a third silence, which is why a response-body check alone is not enough here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer `Unknown map identifier.`. On the add path the reply is byte-for-byte identical to success. Only reading the entry back can tell them apart. IP blocking itself was never broken: update_blocked_ips_map() rewrites /etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it. That path is untouched and stays authoritative. What was broken is the no-reload fast path, plus every report that it had worked. * haproxy_manager.py: both functions send `@1 add|del map /etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with `get map` before returning True. runtime_map_lookup()/runtime_map_keys() are the read-back primitives. `sync_blocked_ips` loses `clear map #0` (which the master socket rejected just as loudly and just as invisibly) and verifies the whole set with one `show map` instead of counting commands that did not visibly complain; it answers 207 + `runtime_map_synced: false` when the runtime map does not match the database. * haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy answers those with nothing on success, so an empty body is the success and ANY non-empty body is a rejection. That is stricter than the marker list on purpose -- markers only recognise rejections someone has already seen, and it catches `'add map' expects three parameters ...`, which matches nothing. HaproxyCliError carries `.responses` so `del map` answering `Key not found.` (the requested end state) is told apart from a real failure without regex. * The four callers capture the boolean instead of discarding it and report `runtime_map_updated` / `runtime_map_failures` in the API response and the operation log. A runtime failure degrades to "enforced on the reload that already happens two lines later" -- never to an unblocked IP, never to a 500. * scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes on the wire (`@1` first, map by path, value `1`), classifies every captured response, and scans the repo's Python string literals and shell/template code lines for `#<id>` map references -- comments may describe the old form, code may not use it. Verified to fail on each defect reintroduced separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the- reply (1). * The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md and the comment in templates/hap_listener.tpl -- where every copy of it additionally omitted the `1`, which `-m int gt 0` needs to match. The only template change is a comment; `haproxy -c` on the live rendered config with it applied is clean (HAProxy 3.0.11, warnings unchanged). Verified on whp01 against the running container (docker cp + SIGHUP, no recreate). Before: both functions returned True and logged success while `@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed add lands with value "1" and the remove takes it out again; the old command form is now classified as a failure; a `#0` map reference returns False via the read-back. End to end through the API, `runtime_map_updated: true`, and /api/blocked-ips/sync -- which used to be a no-op reporting a full sync -- reports 264/264 verified present. The runtime path was isolated from the reload that normally follows it: with NO map-file write and NO reload (same haproxy worker pid throughout), adding 100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not in the is_local ranges) to the runtime map alone flipped a live site from HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path working for the first time. All test IPs were removed afterwards: 0 rows in blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer sites, the panel /health and `haproxy -c` are byte-identical to the baseline taken before the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 5
|
b6a62e7f9f |
fix(security-stats): stop reporting counters the stick tables never stored
/api/security/stats and scripts/show-tarpit-ips.sh reported "Scan Count",
"offense count" and BLOCKED/TARPITTED status parsed from gpc0/gpc1. No stick
table in this repo has ever stored a general-purpose counter -- the `web` table
stores conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s), and
the two brute-force tables store http_req_rate(60s). Every one of those figures
was fabricated, and an operator was making decisions on them.
Three independent silences kept it alive:
* `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
and the loop `continue`d -- so the endpoint always answered
`active_threats: 0` with an empty list. Live on whp01 it also reported
parts[0], the `0x...:` allocation pointer, as the source IP.
* The command was sent to /tmp/haproxy-cli WITHOUT the `@1` worker prefix.
That is the MASTER CLI socket, which answers "Unknown command: 'show' ..."
-- and socat still exits 0, so the `returncode != 0` guard never fired.
`total_tracked_ips` was the line count of that help text (8) while the real
table held 388 entries.
* The shell consumers wrote `gpc0=${gpc0:-0}`, rendering a field that does
not exist as a confident zero.
Report what the tables actually store, rather than adding gpc counters to make
the old semantics real. Adding them would mean editing hap_listener.tpl -- the
one change here with a silent-total-outage failure mode -- to rebuild
enforcement history that the edge access log (shipped 2026.08.8, on the host at
/var/log/haproxy.log) already records per request, with status codes,
termination states and request references the stick table could never hold.
* haproxy_manager.py: STICK_TABLE_FIELD_CONTRACT names what each table
stores. haproxy_cli() sends worker commands with `@1`, falls back to the
bare form for a plain stats socket, and inspects the RESPONSE BODY because
socat's exit status is worthless here. parse_stick_table_entry() reads
name=value / name(window_ms)=value pairs by NAME, never by position.
read_stick_table() RAISES -- naming the field -- when a row is missing a
contract field, instead of defaulting it to 0.
* /api/security/stats returns the four real counters with their windows, the
true `used:` count, and no invented threat_level/blocked/offense_count.
Fewer numbers, all of them real.
* scripts/show-edge-ip-rates.sh replaces the fabricated report; the four
expected fields are declared once as EXPECTED_FIELDS and drive the parser.
show-tarpit-ips.sh becomes a shim that explains why its numbers are gone
and points at where tarpit events actually live.
* monitor-attacks.sh loses fourteen fabricated "threat" categories and a
composite threat score, all permanently zero; its access-log section now
says the log is on the host instead of silently printing nothing.
* haproxy_tarpit_config.txt -- the never-shipped design sketch these counters
were copied from -- gets a NOT IMPLEMENTED banner.
* scripts/test-stick-table-contract.py (offline, 21 tests) holds the
templates' `store` clauses, STICK_TABLE_FIELD_CONTRACT and every consumer
to each other, and asserts each loud-failure path against the real captured
responses. Template and consumers can no longer drift apart quietly.
No template is touched, so haproxy.cfg is unchanged.
Verified on whp01: total_tracked_ips now tracks `used:` exactly (511 vs the
table's 511, was 8 vs 388), and per-IP values match `show table web key <ip>`
field for field. haproxy PIDs unmoved, `haproxy -c` warnings unchanged, five
customer sites HTTP 200.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Claude
|
b2f835a88c |
fix(logging): capture the full User-Agent, drop per-request SPOE log noise
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m19s
Two defects caught by watching the real production access log on whp01 in the
minutes after 2026.08.7 made access logging work for the first time.
1. User-Agent was being truncated to its tail.
`http-request capture req.hdr(User-Agent)` treats the header as a
comma-separated list and returns only the LAST element. Real User-Agent
strings contain commas, so
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
logged as
ua=like Gecko) Chrome/131.0.0.0 Safari/537.36
losing the platform half -- exactly the half needed to tell a spoofed
crawler from a real browser, which is one of the main reasons the field was
added. Switched to req.fhdr(), which returns the full unsplit header value.
2. SPOE was writing one log line per inspected request.
`log global` inside the spoe-agent block emitted
SPOE: [coraza] <GROUP:coraza-req> sid=537 st=0 0/0/0/0/0 32/32 0/0 0/467
for every single request. Measured on whp01: 618 SPOE lines against 669 real
access lines -- ~48% of the log volume, roughly doubling the edge's log
footprint (~400 MB/day extra) to record `st=0` over and over.
It carries nothing incident response needs. The WAF verdict is already in
the access line (status 403 plus the id= UUID, which joins to
/var/log/coraza/audit.log for the rule_id), and per-transaction WAF detail
is written by the SPOA itself to /var/log/coraza/spoa.log. Agent-level
failures still surface through `option set-on-error error` ->
var(txn.coraza.error) and the fail-open path in hap_listener.tpl.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; wp-admin gate,
trusted-proxy gate and xmlrpc rate-limit suites all still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Claude
|
711c670319 |
fix(logging): stop silently discarding every access log line
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m46s
haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.
The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.
Changes:
* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
/var/log/messages or the Graylog forwarder, bound to the bridge IP rather
than 0.0.0.0.
* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
standalone/home deployments on a different bridge subnet can retarget it.
* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
not sufficient for incident response -- it omits %ID entirely (verified
against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
for CF-fronted sites. The new format keeps the first 16 fields byte-identical
to the httplog default (so existing parsers still work) and appends
cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
Adds a User-Agent capture in slot 1 to feed it.
* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
It does not, which made the documented support-correlation workflow
(X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
supported when it could never have worked.
Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".
UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:
<150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
[22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
shadowdao
|
e7d08c3b30 | chore: release 2026.08.6 | ||
|
shadowdao
|
b931baa9a7 |
chore: release 2026.08.5
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m22s
|
||
|
shadowdao
|
992bf49138 |
chore: release 2026.08.4
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m8s
|
||
|
shadowdaoandClaude Opus 5
|
f21ade06d9 |
fix(haproxy): stop the trusted-proxy volume from shadowing Cloudflare/proxy lists
/etc/haproxy is a named volume in deployed containers, so the baked-in
cloudflare_ips.list and trusted_proxies.list COPYed there in the prior
task never actually reached hosts with a pre-existing volume -- the
start-up.sh guard then found them "missing" and created them empty.
With both lists empty, the from_trusted_proxy ACL in hap_listener.tpl
matched nothing, so CF-Connecting-IP / X-Real-IP / X-Forwarded-For got
stripped from every peer, including Cloudflare's own edge. Confirmed
live: image shipped 34/13 lines, running container had 0/0.
Fix: stage both files under /haproxy/defaults (outside the volume) and
apply their ownership rule in start-up.sh instead of a blind
"create if missing":
- cloudflare_ips.list is shipped data -- always refresh it from the
baked default so Cloudflare range updates reach existing hosts.
- trusted_proxies.list is operator data -- seed it from the baked
default only when missing, and never overwrite what an operator
added on the server.
Both branches fall back to creating an empty file if the baked default
is somehow absent, since a missing "-f" target is a fatal HAProxy
config error.
Verified against a volume pre-populated to shadow the image (mimicking
a real host): cloudflare_ips.list repopulates with all 15 IPv4 + 7
IPv6 ranges even after being truncated and restarted; a distinctive
operator entry appended to trusted_proxies.list survives a restart
untouched; haproxy -c still validates cleanly.
Release-worthy fix for a defect from the just-released 2026.08.2 build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
shadowdao
|
871181c345 |
chore: release 2026.08.2
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m11s
|
||
|
shadowdao
|
9d16151120 |
fix(config): take the rollback backup BEFORE writing the new config
generate_config() wrote /etc/haproxy/haproxy.cfg and only then called
reload_haproxy_safely(), which called create_backup(). The "backup" was
therefore a copy of the config that had just been written, so on a validation
failure restore_backup() restored the identical broken bytes: the advertised
rollback was a no-op and a fatal haproxy.cfg stayed on disk, where
start_haproxy() refuses to launch. Same shape as the June 2026 incident where
a missing template produced a fatal config and took an edge down.
Reproduced end to end before the fix (invalid config generated -> "Backups
created successfully" -> "Backups restored successfully" -> haproxy.cfg on
disk still invalid, `haproxy -c` rc=1).
Changes:
* create_backup() is now called by generate_config() BEFORE the first write,
which also covers blocked_ips.map (rewritten early in generate_config) and
coraza-spoe.cfg - both previously written before the backup and, for the
SPOE file, never backed up at all even though `haproxy -c` parses it.
* create_backup() refuses to promote a config HAProxy already rejects, so a
broken file on disk cannot overwrite a known-good backup ("rollback" must
not mean "restore a different broken config"). It returns (ok, status) so
the caller knows whether a rollback target exists.
* promote_current_config_to_backup() records the config as known-good only
after it has validated AND loaded, so a box whose first generation succeeded
has a rollback target immediately, and a config that never loaded is never
promoted.
* restore_backup() returns (restored, message) and distinguishes "no backup
available" from "restored". Every caller now surfaces the difference; a
failed rollback is logged CRITICAL and reported as ROLLBACK FAILED in the
API error message instead of silently looking like a successful recovery.
* reload_haproxy_safely(backup_status=...) no longer takes its own backup - it
runs after the write, where a backup is meaningless. Called without a status
it logs the contract violation rather than overwriting a good backup.
* validate_config_file() separates "config is invalid" from "validator could
not run" so a missing haproxy binary is not read as a bad config.
* Config writes are atomic (temp file + fsync + os.replace, mode preserved);
a truncated haproxy.cfg is as fatal as an invalid one. Removes the dead
temp_config_path variable whose comment claimed this already happened.
* Fast path: if the live config set is already byte-identical to the backup
(the normal case after a successful reload), skip the re-validation and the
copy, so this adds no `haproxy -c` latency to customer-facing API calls.
Tests: scripts/test-config-rollback.py - 17 self-contained stdlib-unittest
tests, no new dependencies (the repo has no Python test framework; the
existing scripts/test-*.sh are curl integration scripts). A stub `haproxy`
binary stands in for the validator. 12 of the 17 fail against the previous
code; every assertion was mutation-proven (9 mutations, each reddening only
the tests that cover it).
No template, QUIC or HTTP/3 changes.
|
||
|
shadowdaoandClaude Opus 4.8
|
2a2b9739fc |
fix(api): bound all subprocess calls + run 2 workers to prevent API stall
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m9s
The management API wedged on whp01 2026-07-07: every panel call to the manager (config regenerate during a WHP site update, SSL, even /health) timed out at 30s while customer sites stayed up. Root cause: all four gunicorn gthread worker threads were permanently blocked in socket reads inside untimed subprocess.run() calls (certbot ACME / socat reloads). A stalled external command holds its worker thread forever; gunicorn --timeout can't rescue it (gthread only kills a worker whose main thread stops heart-beating, and ours kept polling). Stalled calls accumulated until the 4-thread pool was exhausted and the whole API went dark. - Wrap subprocess.run with a default timeout (HAPROXY_MGR_SUBPROCESS_TIMEOUT, 180s) so every external command is bounded and releases its thread on expiry via the existing per-endpoint try/except. Bounding by default covers all ~30 call sites and any future one. - certbot renew keeps an explicit 900s timeout (walks every lineage). - API_WORKERS default 1 -> 2: a single worker made a thread-pool wedge a total outage; a second worker keeps the API answering while one recycles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.8
|
7732e2a2ff |
chore(log): downgrade "no backend name" domain-skip from WARNING to INFO
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 2m12s
generate_config emits "Skipping domain <host> - no backend name" on every run
for domains registered without a proxy backend — most commonly the panel's own
hostname (whpNN.cloud-hosting.io), which lives in the DB only for certificate
management and intentionally has no backend. Logging it at WARNING tripped the
WHP AI log monitor as a recurring error and prompted a bogus "restart
haproxy-manager" remediation. It's expected, benign, and recurs by design.
Log it at INFO instead (consistent with the sibling per-domain "Added ACL for
domain" INFO lines) with a clearer message ("no proxy backend
(cert/management-only)"). Verified against the WHP monitor's ErrorClassifier:
the old WARNING line classified as non_critical (captured); the new INFO line
classifies as None (skipped) — so it no longer shows up in reports, while
remaining visible in container logs for manual routing debugging.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
shadowdaoandClaude Opus 4.8
|
89c74c10cf |
fix(supervisor): restart haproxy in-place if it dies while container lives
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m11s
haproxy runs as a background child of PID 1 (gunicorn) with nothing watching it after init. If the haproxy master dies mid-life (observed 2026-07-01 on whp01: SIGABRT -> exit 134, reaped by gunicorn and logged as "Worker (pid:22) exited"), the container stays "up", Docker's --restart never fires, and haproxy is down until the external host watchdog full-restarts the whole container minutes later (dropping every connection). Add an in-container supervisor loop in start-up.sh (Phase 1.5) that runs scripts/ensure_haproxy.py every HAPROXY_SUPERVISOR_INTERVAL (default 15s). ensure_haproxy.py calls the existing, idempotent start_haproxy() only when haproxy isn't running (psutil guard), reviving it in place within one interval with no container restart. Same entrypoint-supervision pattern shipped for cac-litespeed. Validated locally: killing haproxy -> revived with new PIDs in ~one interval, container stayed healthy, no spurious restarts while healthy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.8
|
1b557b9931 |
feat(waf): wp-login cookie challenge (defeats distributed credential-stuffing)
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m29s
The per-IP throttle can't see distributed attacks (observed 76k–289k UNIQUE IPs hitting wp-login.php, each low-and-slow). But those bots POST straight to wp-login.php without GETting the form (~15:1 POST:GET on attacked sites). So: hand out a `whplc` cookie on GET of the login form (set-var at request time + http-after-response add-header — request fetches don't evaluate in the response phase) and DENY 403 on login POSTs that lack it. Direct-POST bots are dropped at the edge before reaching PHP; real logins are unaffected (WP login already requires loading the page + cookies). Immediate deny, not tarpit, to avoid connection exhaustion under a 300k-POST flood. Honors the whitelist. Validated locally: GET /wp-login.php emits whplc; other paths don't; config OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.8
|
6ced2f8797 |
feat(waf): edge brute-force throttle for wp-login.php
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 2m8s
The generic rate-limits are tuned high for media-heavy sites, so slow credential-stuffing on wp-login.php slips under them. Add a dedicated sc1 stick-table (backend wp_bruteforce, 60s window) that counts POSTs to wp-login.php per real client IP and tarpits once an IP exceeds 30/min. Only login POSTs are counted (browsing + the login form GET + a legit user's few attempts are unaffected); an offending IP can still browse, just not keep hammering login. Honors the existing whitelist (RFC1918 / trusted_ips.list / trusted_ips.map) and the already-resolved CF/proxy real IP. path_end also covers subdirectory WP installs. Stops attacks at the edge before they reach PHP/WordPress, on all edges regardless of Coraza mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
shadowdaoandClaude Opus 4.8
|
d9cc5311de |
feat(quic): enable HTTP/3 over QUIC on the edge + versioned images
HTTP/3 is config-only — the Debian haproxy package is built +QUIC via the OpenSSL compat shim. Changes: - hap_header.tpl: `limited-quic` (required to enable QUIC binds under the compat layer) + self-healing `cluster-secret` for QUIC token derivation. - hap_listener.tpl: `bind quic4@:443 ... alpn h3` in the shared frontend (so real-IP/rate-limit/IP-block/Coraza rules apply to H3 too) + alt-svc header. - Dockerfile/README: publish/document 443/udp; stamp image.version from VERSION. - CI: tag :latest + :<VERSION> + :<sha> so there's a pinnable rollback target. No 0-RTT (compat-layer limitation). Validated end-to-end on a standalone edge: config parses, UDP/443 binds, alt-svc advertised, real curl --http3 -> HTTP/3. Container must run with `-p 443:443/udp` + host UDP/443 open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |