Commit Graph
239 Commits
Author SHA1 Message Date
shadowdaoandClaude Sonnet 5.5 f0a737832c test: scale the cert-scripts subprocess timeout for emulated (non-x86) builds
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 23m53s
With python:3.12-slim now a real multi-arch index, the arm64 leg really runs
the image gate under QEMU (~30x slower). test-cert-scripts.py took 229 s there
and its 200-iteration bash loop exceeded a flat 120 s timeout (CI run 5756),
and the whole arm64 step then hit its 30-minute ceiling (run 5758). x86 keeps
120 s; other machines get 900 s.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-19 20:20:48 -08:00
shadowdaoandClaude Sonnet 5.5 4cb4ad39f8 docs: nbthread comment names the real plumbing (lib-edge-container.sh)
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 33m19s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-19 19:45:04 -08:00
shadowdaoandClaude Sonnet 5.5 8ab6fdcef0 feat: HAPROXY_NBTHREAD per-host thread-count knob (global nbthread, only when set)
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 22m41s
2026-11-19 sdbees: with nbthread 4 a thread spun on a server lock held by a
TLS health-check thread stalled in a page fault under memory pressure ->
watchdog abort of the whole edge. nbthread 1 removes the failure class.

HAPROXY_NBTHREAD (container env, integer 1-64) renders `nbthread N` in the
global section ONLY when set. Unset/invalid = not rendered: every other host's
config is byte-identical to before (verified against main's template, with and
without Coraza). Invalid values are ignored with a warning, never rendered.

Tests: scripts/test-nbthread-knob.py (parse, render, unset byte-identity vs the
template minus the block, no cpu-map/thread-groups), run in the image build; real
haproxy -c of nbthread 1 (+SPOE+QUIC) and of a bad value in
validate-rendered-config.py. VERSION 2026.11.8 (with the arm64 base guard).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-19 19:30:07 -08:00
shadowdaoandClaude Sonnet 5.5 56404f4777 ci: refuse arm64 images that contain non-arm64 binaries (base-arch guard + ELF verification)
The in-house python:3.12-slim mirror tag was still a flattened amd64-only
manifest, so the linux/arm64 build ran on an amd64 base and published x86-64
binaries labelled arm64 (haproxy-manager-base:latest, :2026.11.6-rc).

- Dockerfile: fail the build when dpkg arch != TARGETARCH (no-op when correct)
- scripts/verify-image-arch.py (+ test): ELF e_machine of haproxy, bash, python,
  psutil .so per platform digest
- build-push.yaml: verify amd64 (fatal) and arm64 before assembling; an arm64
  mismatch is excluded from the tags and fails the job

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-19 18:27:39 -08:00
jknapp 3ce1dc8662 Merge pull request 'fix(wp-admin gate): exempt the WordPress install screens (setup-config.php, install.php)' (#21) from fix/wpadmin-gate-install-exempt into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 14m39s
2026-11-20 02:20:38 +00:00
shadowdaoandClaude Sonnet 5.5 3eee8e6023 wp-admin gate: state the real residual risks of exempting the install screens
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 10m50s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-19 16:47:16 -08:00
shadowdaoandClaude Sonnet 5.5 c91187aa85 fix(wp-admin gate): exempt the WordPress install screens (setup-config.php, install.php)
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 16m35s
A fresh WordPress with no wp-config.php redirects every request to
/wp-admin/setup-config.php. The edge wp-admin gate bounced that cookieless
request to wp-login.php?redirect_to=/wp-admin/setup-config.php, which then
tripped CRS 930120 ("config.php" in ARGS:redirect_to) and showed the WAF
block page: manual installers never saw WordPress's setup screen.

New ACL wp_admin_install (path_end, exact /wp-admin/setup-config.php and
/wp-admin/install.php on the normalised path, case-sensitive) added as a
negated exclusion on the redirect rule. upgrade.php stays gated (only
logged-in users are sent there). Variants (.php.php, .phpx, case, path info,
inserted directory, dotdot/encoded escapes) stay gated; encoded separators
still deny.

Tests: the structural suite now pins the new ACL; new
test-wpadmin-gate-behaviour.py runs the real gate rules in the real haproxy
and sends raw requests (run by the image build). VERSION 2026.11.7.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-19 16:34:52 -08:00
jknapp 193d39e109 Merge pull request 'HAProxy 3.2 LTS edge image (default for all new deployments)' (#20) from feat/haproxy-upgrade into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 9m40s
2026-11-17 21:07:33 +00:00
shadowdaoandClaude Sonnet 5.5 aebb68fe9d ci: make tag verification independent of buildx version
buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'`
and prints the whole inspect text, so the verify step rejected every ref as a
"bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh:
the index digest is the sha256 of `inspect --raw`, cross-checked against the
anchored `Digest:` line and a bare --format output when present; platforms are
read from the raw JSON. All other checks (every tag in every registry, platform
match, one digest across registries, any miss fails) are unchanged. Adds an
offline test with a stub emulating 0.28 and 0.38 output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:30:21 -08:00
shadowdaoandClaude Sonnet 5.5 ba8722b0b7 chore: bump VERSION to 2026.11.6 (rc with gunicorn access-log fix + hardened CI)
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 13m10s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:13:46 -08:00
shadowdaoandClaude Sonnet 5.5 fa8fec0b4f fix(logging): keep gunicorn access logs off container stdout
Both gunicorns ran with --access-logfile -, so every :8080 default-page /
blocked-IP / suspended-site hit and every :8000 API call became a docker
stdout line. With the journald log driver (WHP PR #260) that is one journal
entry per request: journald's default rate limit (10000 msgs / 30 s per
service) could suppress the relay's crash-dump lines, and Graylog would be
flooded (the edge access log must stay out of it).

* :8080 default-page gunicorn: no access log.
* :8000 API gunicorn: access log to /var/log/haproxy/api-access.log (mode
  0640, haproxy-logs volume), bounded by scripts/gunicorn-access-rotate.py
  (copy-truncate, 20 MiB x 3, 30 s poll). 401/403 are in the file only.
* gunicorn ERROR logging stays on stderr; supervisor/relay/startup lines
  unchanged on stdout/stderr.
* New build-gate test scripts/test-gunicorn-access-logging.py; docs/logging.md.

VERSION unchanged (2026.11.5): a NEW rc image must be built from this branch
for the change to ship.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:12:53 -08:00
shadowdaoandClaude Sonnet 5.5 68976142f5 ci: harden assemble step (pinned buildx, plain progress, timeout) and verify every tag in every registry
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 21:10:51 -08:00
shadowdaoandClaude Sonnet 5.5 99279d5347 chore: bump VERSION to 2026.11.5 (HAProxy 3.2 rc)
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 18m29s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 20:42:04 -08:00
claudeandClaude Sonnet 5.5 ea2d4994f1 fix(image): exact haproxy pin + relay tenant-alert class + supervisor backoff + rc publishing
I1: ARG HAPROXY_EXACT=3.2.26-1~bpo13+1 is the apt pin; build fails unless dpkg-query equals it exactly (fail closed when the repo is unreachable); comment corrected.
I2: relay no longer exempts every [ALERT]; tenant-driven 'has no server available' / 'Server ... is DOWN|UP' are their own rate-limited stdout class; dump blocks have their own budget.
Minor: keyring must hold exactly one pub key (gnupg purged), stderr log + rotated copies 0640, supervisor extracted to scripts/haproxy-supervisor.sh (integer-validated knobs, crash-loop backoff 2/5/10/30, reset after 60s).
C1: release/* branches and rc-* tags publish only :<VERSION>-rc and :<sha>, never :latest; main unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 17:33:27 -08:00
shadowdaoandClaude Sonnet 5.5 c2e070d172 fix(relay): anchor crash patterns to HAProxy message structure; crash-evidence budget + dump cooldown
Security review: unanchored keywords let echoed client data bypass the limiter and open
dump blocks; bounds re-armed immediately. Patterns are now anchored, blocks open only on
the 'Thread N is about to kill' header, a 1000-line/10-min budget caps exempt lines
(overflow goes through the normal bucket and the suppressed summary) and a 60 s cooldown
follows a bound-closed block. Tests: forged keywords, flood, cooldown + 3 new mutants.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 15:08:46 -08:00
shadowdaoandClaude Sonnet 5.5 5e27870608 fix(relay): never rate-limit crash evidence; collapse reload noise; stdout/stderr priority split
The 3.2 canary showed the relay's 400-line bucket, drained by ~110 reload lines
per reload, dropping a crash dump (journal kept only 'Thread 1 is about to kill').
Crash lines and whole dump blocks now bypass the bucket (bounded block), routine
reload noise is collapsed to one line per reload, the remainder keeps the limit
with a per-minute '[relay] suppressed N lines' summary. Benign lines go to stdout
(journald PRIORITY 6), crash evidence to stderr (PRIORITY 3).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 15:04:14 -08:00
Claude a753d300db feat: HAProxy 3.2 LTS (haproxy.debian.net) + capture HAProxy stderr + 2 s supervisor
DRAFT for review, not merged. Context: HAProxy 3.0.11 (Debian trixie) aborts via
its own watchdog in the DNS resolver (haproxy #3101, fixed in 3.0.13/3.1.11/3.2.6).

* Dockerfile: haproxy from https://haproxy.debian.net trixie-backports-3.2 (committed
  signing key, fingerprint asserted, series pin 3.2.*, optional HAPROXY_EXACT),
  native OpenSSL QUIC asserted (no compat layer), /haproxy/HAPROXY_VERSION.
* start HAProxy as `haproxy -W -db` (spawn_haproxy_master): `daemon` sent the
  master's stderr to /dev/null, losing ALERTs and the ha_panic thread dump.
  stderr -> /var/log/haproxy/haproxy-stderr.log, forwarded to container stderr by
  scripts/haproxy-stderr-relay.py (rate limited, rotated).
* supervisor: pidof probe every 2 s (was python every 15 s), exponential backoff
  on failed restarts.
* tests: relay unit tests; rollback/cert stubs print "Loading success.".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 13:02:33 -08:00
jknapp 46ea498db3 Merge pull request #19: haproxy pin 3.0.11 (any Debian revision) + multi-arch publish [skip ci] 2026-11-12 02:04:32 +00:00
shadowdaoandClaude Sonnet 5.5 d59d7e0ad4 feat(ci): publish haproxy-manager-base multi-arch (amd64+arm64); pin haproxy by upstream version (2026.11.3)
- build-push.yaml: one job on the x86 runner; amd64 and arm64 (QEMU,
  continue-on-error) are pushed by digest to Gitea+GHCR, then
  scripts/assemble-multiarch-index.sh creates :latest/:<version>/:<sha>
  (same template as coraza-spoa). arm64 failure publishes amd64-only; a
  stale arm64 digest is never re-attached. Adds workflow_dispatch.
- Dockerfile: pin the reviewed UPSTREAM version (HAPROXY_UPSTREAM=3.0.11) via
  an apt preference so the newest Debian security revision is taken
  (3.0.11-1+deb13u4 today), and fail the build unless dpkg and `haproxy -v`
  both report that upstream version. The exact +deb13uN pin froze the image
  on the superseded u3 and would have hard-failed once u3 left the archive.
- The existing build gate (unit suites + haproxy -c) is unchanged and runs
  for each platform.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 17:42:22 -08:00
jknapp 3616fa409f Merge pull request #17: coraza-spoa multi-arch (amd64 + arm64) [skip ci] 2026-11-12 01:15:31 +00:00
jknapp 48edc09e8d Merge pull request #18: mirror index copy works on Gitea (pinned buildx) [skip ci] 2026-11-12 01:03:16 +00:00
shadowdaoandClaude Sonnet 5.5 cf8f09612f fix(ci): mirror index copy works on Gitea (pinned buildx, digest assertion via raw bytes) [skip ci]
Run 5156 failed copying python:3.12-slim / golang:1.25 into Gitea:
`PUT .../manifests/<tag>-staging: 404`, leaving a stray single-manifest
<tag>-staging version. Reproduced on real Gitea with scratch packages:
buildx <= 0.20.x (runner distro buildx is 0.15.1) pushes the children of a
copied index to the TAG path; Gitea only records manifests PUT by digest, so
the index PUT is rejected (MANIFEST_BLOB_UNKNOWN -> 404). buildx >= 0.24
pushes children by digest and works; Gitea then preserves the index digest, so
the digest-equality assertion stays.

- workflow: install a sha256-verified pinned buildx (v0.28.0) instead of
  setup-buildx-action (imagetools needs no builder) and pass it as $BUILDX
- mirror-index-copy.sh: BUILDX/MIN_BUILDX guard refuses buildx < 0.24 before
  any registry traffic; digest computed as sha256 of the raw index bytes
  (buildx 0.28 ignores `inspect --format` here)
- test: attestation-children index and old-buildx guard cases

Merge with [skip ci] in the merge title: any push to main triggers the HMB
image build.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 17:01:09 -08:00
jknapp 74fc9ef183 Merge pull request #16: mirror workflow keeps the multi-arch index [skip ci] 2026-11-12 00:46:13 +00:00
shadowdaoandClaude Sonnet 5.5 a476bb5352 coraza-spoa: multi-arch (amd64+arm64) by cross-compiling; push-by-digest + imagetools index
Dockerfile: Go stages pinned to $BUILDPLATFORM, GOOS/GOARCH from TARGETOS/
TARGETARCH (CGO already off); amd64 rootfs/binary byte-identical to before.
Workflow: one trusted x86 job, amd64 + arm64 pushed untagged by digest to both
registries, arm64 continue-on-error, tag assembled by
scripts/assemble-multiarch-index.sh (arm64 only from this run's successful
step; otherwise amd64-only). Offline test + local e2e rehearsal script.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 16:44:24 -08:00
shadowdaoandClaude Sonnet 5.5 905b2894b6 fix(ci): mirror base images as whole multi-arch indexes, with assertions
mirror-base-image.yaml pulled the upstream index digest on an amd64 runner and
re-pushed it, publishing only the amd64 manifest; the python/golang mirrors
had no arm64 variant. Replace pull/tag/push with a registry-to-registry
`imagetools create` of the digest-pinned index via .gitea/scripts/
mirror-index-copy.sh: copy to <tag>-staging, assert linux/amd64+linux/arm64
are listed and the index digest equals the Docker Hub digest, then move the
real tag and re-assert. A failed assertion never touches the real tag.
Adds a local-registry test (.gitea/scripts/test-mirror-index-copy.sh).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 16:40:22 -08:00
jknapp 50c58c4e9e Default splash for unrouted hosts (404+noindex, answered before the WAF) and in-container access log (2026.11.2) (#15)
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m36s
2026-11-08 20:03:23 +00:00
jknapp 1ee5760c1b Merge pull request 'ci: build through the internal package cache; mirror bases by Docker Hub digest (2026.10.7)' (#14) from ci/pkg-cache into main
Build and push coraza-spoa / Build-and-Push (push) Successful in 1m10s
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m45s
2026-10-22 14:20:16 +00:00
shadowdaoandClaude Opus 5.5 dc1cbeb12a ci: build through the internal package cache; mirror bases by Docker Hub digest (2026.10.7)
- Dockerfile: optional ARG PKG_CACHE; the apt RUN (dist-upgrade + pinned
  haproxy) rewrites sources via scripts/pkg-cache-repos.sh apt and restores
  them before the RUN ends. Without PKG_CACHE: upstream Debian, as before.
- coraza-spoa/Dockerfile: same for the build stage's apt RUN; go mod
  download / go build (build + catalog stages) use go-env (GOPROXY via the
  cache with proxy.golang.org fallback; GOSUMDB unchanged) and restore.
  Build context is ./coraza-spoa, so coraza-spoa/scripts/pkg-cache-repos.sh
  and coraza-spoa/keys/pkg-cache/ are byte-identical copies of the root ones.
- build-push.yaml / build-push-coraza.yaml: "Probe package cache" step
  (TLS with the pinned CA), PKG_CACHE build-arg. Bases come from
  repo.anhonesthost.net / gcr.io, not docker.io: registry mirror not used.
- mirror-base-image.yaml: resolve the tag's digest directly at Docker Hub
  (.gitea/scripts/resolve-base-image.sh) and pull docker.io/library/<img>@sha256
  so the mirrored content is exactly the digest Docker Hub reported; same
  destination tags. Values reach run: via env only.
- Canonical copies (byte-identical) of scripts/pkg-cache-repos.sh,
  .gitea/scripts/resolve-base-image.sh and keys/pkg-cache/ from
  cloud-apache-container.
- VERSION 2026.10.7 so the rebuild does not overwrite the 2026.10.6 tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 07:19:54 -07:00
jknapp 4345732dad Merge pull request 'fix(edge): CVE-2026-109618/-109619 multipart mitigations (2026.10.6)' (#13) from fix/cve-2026-109618-multipart-edge into main
Build and push coraza-spoa / Build-and-Push (push) Successful in 54s
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m18s
Merge pull request #13: CVE-2026-109618/-109619 multipart edge mitigations (2026.10.6)
2026-10-21 14:38:27 +00:00
shadowdaoandClaude Opus 5.5 729098a8f9 fix(edge): CVE-2026-109618/-109619 multipart mitigations (2026.10.6)
PHP "RCE via OOM corruption" (GHSA-w8hw-w9fx-j79w) and the session
upload-progress variant (GHSA-6wmf-4rv5-p3rp), interim until patched PHP is
on every customer image.

HAProxy (hap_listener.tpl, every frontend render, WAF on or off):
  deny 400 multipart/form-data with no Content-Length (HTTP/1.1 chunked,
  HTTP/2-3 streamed bodies) -- vector (a), the post_max_size bypass.
  Measured: Chrome 155 (h2 and h3) and Firefox 153 send content-length on
  form submits, FormData fetch and XHR; only a ReadableStream fetch body
  (duplex:'half') has none.

Coraza (coraza-spoa/overrides.conf, 209610-209616):
  209610       ARGS_NAMES/FILES_NAMES with more than 16 '['
  209611-613   part-header budget (4/part, deny 32 below zero)
  209614-615   more than 64 empty-filename file parts
  209616       multipart PHP_SESSION_UPLOAD_PROGRESS field
Ids sit in the engine range 200000-209999 that WHP's break-glass rule
(991000001) already removes, so no WHP release is needed for break-glass
to lift them. Each flips the engine On for itself, like the day-one
promotions, so they deny in every mode the sidecar runs in.

Limitation: Coraza sees only the first 16 KB of a body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:27:42 -07:00
jknapp b6cc3adfaa Merge pull request 'fix(waf): HTTP/3 bodies were stamped complete when truncated (2026.10.5)' (#12) from fix/waf-h3-body-stamp into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 4m49s
2026-10-19 21:22:13 +00:00
shadowdaoandClaude Opus 5.5 77812399d4 chore: VERSION 2026.10.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:59:45 -07:00
shadowdaoandClaude Opus 5.5 2cc6790db1 fix(waf): stamp X-WHP-WAF-Body from the Content-Length header, not req.body_size
req.body_size is "buffered + htx->extra". In HAProxy 3.0 only the H1 and
H2 muxes set htx->extra from Content-Length; the H3 mux never does, so
over HTTP/3 req.body_size == req.body_len and any body cut at one buffer
(~15 KB) was stamped `complete`. Coraza then failed to parse the
truncated JSON (200002) and WHP's 991900020 returned a 403.

Measured on whp01, Oct 6-19 2026: 73,795 H3 `complete` stamps and 0
`partial`; 7 legitimate requests 403'd (6 Divi sync-to-server saves of
265-417 KB, 1 Groundhogg preview of 18 KB). Reproduced in a lab on
haproxy 3.0.11 with QUIC. H1/H2/H3 all reject a Content-Length that
doesn't match the DATA received, so the header is a protocol-independent
body length. No change on H1/H2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:59:29 -07:00
shadowdaoandClaude Opus 5.5 cbbd15b787 feat(waf): log why Coraza acted (wrule/wrules/wscore); VERSION 2026.10.4
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m14s
coraza-spoa v0.7.1 already returns the interrupting rule id
(txn.coraza.ruleid) and, when the request message sends
exportRuleIDs=bool(true), the comma list of every matched CRS attack
rule (txn.coraza.rule_ids) plus the inbound anomaly score. Send the arg
and append all three to the access log so a block's cause (e.g.
920440,949110 score 5) is visible in haproxy.log without opening the
Coraza audit log. Fields are appended last; existing ones unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:20:46 -07:00
shadowdaoandClaude Opus 5.5 519f2d8ade feat(waf): Coraza transaction ID = HAProxy request reference UUID
Send `id=unique-id` in the coraza-req SPOE message so coraza-spoa calls
waf.NewTransactionWithID() with HAProxy's unique-id (the %[uuid()] shown
on 403-waf.html, sent as X-Request-Reference and logged as `id=` in
haproxy.log). A customer-quoted reference now greps directly to
transaction.id in /var/log/coraza/audit.log and match.unique_id in
spoa.log. Previously coraza-spoa invented a random 16-letter A-Z ID
that joined to nothing on the HAProxy side.

Measured against coraza-spoa v0.7.1 source (no version bump needed):
- internal/agent.go HandleSPOE: only `app` is positional (must be the
  first KV, else Panic); the template comment claiming all args are
  parsed positionally was wrong and is corrected.
- internal/application.go HandleRequest: every other arg is matched by
  name; `id` is recognised and passed to NewTransactionWithID. Empty/
  NULL id falls back to the random ID, so this cannot break inspection.
- coraza v3.7.0 places no format/length constraint on the ID.

response_check is false so there is no coraza-res message; the template
and a test now require any future coraza-res to send the same id
(HandleResponse looks the request transaction up by it).

Tests: new CorazaTransactionIdIsTheRequestReference class in
test-waf-body-inspection.py (fails without the change). Full in-image
suite + validate-rendered-config (haproxy 3.0.11 -c) pass; a negative
control with a bogus fetch name is rejected by haproxy -c.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:03:59 -07:00
shadowdaoandClaude Opus 5.5 7ae8a4780c test(api): starvation test asserts no-starvation; latency bounds only under HAPROXY_MGR_STRICT_TIMING
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 4m45s
CI runner measured median 14.97s on 2 probes (load-dependent), failing the
build gate on a timing bound rather than the property under test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:40:22 -07:00
shadowdaoandClaude Opus 5.5 765b2ce00a Merge fix/ssl-renewal-durable into release/2026.10.3
Build and push coraza-spoa / Build-and-Push (push) Successful in 3m9s
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 5m7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:22:52 -07:00
shadowdaoandClaude Opus 5.5 3712860429 Merge feat/waf-body-hardening into release/2026.10.3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:22:46 -07:00
shadowdaoandClaude Opus 5.5 17c9a8faf8 test(ssl): certbot re-issue test skips on ANY certbot import/init failure
The skip guard caught only ImportError; a half-installed or version-skewed
certbot can fail on import with other exceptions (verified with a stub that
raises RuntimeError: previously an ERROR, now a skip). new_lineage() itself
stays outside the guard - it is the assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:26:19 -07:00
shadowdaoandClaude Opus 5.5 957fba29d9 fix(api): refuse non-local peers on :8000; count header deadline from enqueue
Re-review R1: haproxy-manager shares client-net with every tenant
container, so tenant PHP could reach haproxy-manager:8000 directly (the
127.0.0.1 publish only removed the internet path). On the ports in
HAPROXY_MGR_PEER_FILTER_PORTS (default 8000; NOT :8080, fed by HAProxy
over loopback) the overridden accept() closes any peer not in
HAPROXY_MGR_API_ALLOWED_PEERS (default 127.0.0.1,::1,gateway), before a
TConn, slot, poller registration or thread is used. "gateway" is read
from /proc/net/route: measured, host callers via the 127.0.0.1 publish
arrive from the bridge gateway (docker-proxy). Bad entries fail closed
and are logged; refusals are logged at most once a minute as a per-peer
summary.

Header deadline is now max(enqueue + 10s, pickup + 1s)
(HAPROXY_MGR_HEADER_PICKUP_GRACE): 40 reconnecting drippers no longer
starve legit requests (before: every probe >30s; after: median ~9s).

Tests: 21 (was 15): allowlist parsing/matching incl. v4-mapped, fail
closed, /proc/net/route parsing, real-gunicorn filtered vs unfiltered
port with rate-limited logging, listed peer served, 40 reconnecting
drippers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:25:32 -07:00
shadowdaoandClaude Opus 5.5 2da2020305 fix(waf): stop promoting 944110/944120 chain starters (round 3)
Owner ruling (same mode contract as 920440: no accidental enforcement from
bare words). 944110 and 944120 are CRS chains whose first link matches
`runtime|processbuilder` / `getproperty|forclosure|filewriter|...`
anywhere in the request; the range promotion 944100-944300 put
ctl:ruleEngine=On on those starters, so any request containing the word
ran with full anomaly enforcement, in detect_only too.

Promotion is now 944100 and 944130-944300. Live (haproxy-manager-base
:wafbody-r3 + coraza-spoa :wafbody-r3), both modes: `?q=runtime&<SQLi>`
and `?q=getproperty&<SQLi>` 403 -> 200 on detect_only; Log4Shell in the
query (plain and ${lower:j} obfuscated), User-Agent and a JSON body still
403.

test-waf-body-inspection.py: 21 tests (+2: no CRS chain starter
920440/944110/944120 is covered by a promotion; the Log4Shell ids keep
theirs). Red on round 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:10:41 -07:00
shadowdaoandClaude Opus 5.5 234029c249 fix(waf): promote 920440 chain-safely, not on its chain starter (round 2)
Owner decision. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on
920440's chain STARTER (`REQUEST_BASENAME "@rx \.([^.]+)$"`), which matches
every URL with a file extension, so every *.php/*.css request was evaluated
with the engine On: full CRS anomaly enforcement (949110) and 200002, in
detect_only mode too.

Replaced by `SecRule TX:EXTENSION "@within %{tx.restricted_extensions}"
... ctl:ruleEngine=On` (id 992920440, phase 1, after 920440). CRS 4.25.0
sets tx.extension in 920440's first link (REQUEST-920...conf:667, the
coreruleset version coraza-spoa v0.7.1 pins). Live, both modes:
.bak/.sql/.save/.old/.swp probes still 403; benign .php/.css 200; a
942100 SQLi on /x.php and the onclick / "select ... union" admin saves are
no longer 403 in detect_only.

test-waf-body-inspection.py: 19 tests (2 new, repo-checkout only).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:00:22 -07:00
shadowdaoandClaude Opus 5.5 8c7fd93387 fix(ssl): review round 1 - exact --cert-name lineage, retire whole lineage tree, sweep expired pems
- H1: request_ssl_bundle uses live/<primary> exactly (certbot was given
  --cert-name <primary>) instead of find_certbot_live_dir(), which returns the
  newest of <primary>/<primary>-NNNN. After a "not yet due" no-op that
  published a stale -0001 and retired the real <primary> lineage. keep_lineage
  is now the primary; the bundle's own lineage is never a retirement candidate.
- H2: retire_lineage_tree() moves live/<x>, archive/<x> and renewal/<x>.conf
  together to /etc/letsencrypt/retired/<stamp>/{live,archive,renewal}/ (same
  depth, relative symlinks still resolve, restore is three mv's). Leaving
  archive/<x> behind made certbot's new_lineage() refuse the name forever
  (CertStorageError) and leave an empty renewal conf. Test re-issues the name
  through certbot's own storage code after retirement.
- M2: cert_retire_expired_pems moves already-published EXPIRED pems out of
  /etc/haproxy/certs to cert-retired/ (never the last pem, never the default
  self-signed cert) and the scripts reload when anything was retired.
- GET /api/capabilities + `capabilities` in the bundle response, and VERSION is
  copied into the image, so WHP can refuse whole-site re-bundling against an
  older edge.

Tests: test-cert-write-safety.py 47 -> 52, test-cert-scripts.py 44 -> 50; each
fix reverted in a scratch copy turns its suite red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:00:03 -07:00
shadowdaoandClaude Opus 5.5 fd06b30d78 fix(api): reap never-readable conns, bound total header time, harden watchdog
Review round 1 on 0ce144d (FIX-FIRST):

F1: stock gthread never reaps a connection that never sends a byte; it
holds a worker_connections slot forever and at 1000 per worker the run
loop stops polling, wedging the worker permanently (survives the
attacker disconnecting). accept/on_client_socket_readable/
murder_keepalived now track such conns in a FIFO with a deadline
(HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s) and close expired ones on every
loop pass, including the at-capacity branch.

F2: slowloris (1 header byte per <20s) never tripped the per-recv
timeout. A total request-line+headers deadline
(HAPROXY_MGR_HEADER_DEADLINE, 10s) is armed in handle() and cleared in
handle_request(); a 1s monitor loop shutdown(SHUT_RDWR)s expired
sockets. Armed per thread, not at accept, so legit requests queued
behind busy threads are never cut for waiting.

F4: monitor loop wrapped in try/except with logging; wedge threshold
1200s -> 3600s (multi-domain cert requests run 180s per domain).
F5: docstring now says the timeout bounds each whole sendall().
F6: warn when gunicorn != 23.0.0 (worker overrides gthread internals).
API key compared with hmac.compare_digest.

Tests: +3 production-shape (2x4) tests: 2050 idle conns reaped and
/health recovers while still held; 8 drip conns cut within the deadline
with /health responsive; slow-but-complete headers still served.
15 tests, in the Docker build gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:58:28 -07:00
shadowdaoandClaude Opus 5.5 cc0652e0c0 fix(waf): 400 duplicate Content-Type; JSON depth limit 64 (round 1)
- hap_listener.tpl (Coraza block): `http-request deny deny_status 400 if
  { req.fhdr_cnt(content-type) gt 1 }`. With two Content-Type lines Coraza
  evaluates every value (text/xml on either line selects its XML processor,
  which accepts a form body without error) while Apache merges the lines
  and PHP parses the body as a form. Measured with Coraza v3.7.0 + PHP 8.3:
  `application/x-www-form-urlencoded` + `text/xml` put the payload in
  $_POST uninspected. Live: two lines -> 400 over HTTP/1.1 and HTTP/2; one
  comma-joined line (browser XHR double-set) is unaffected.
- coraza-spoa/config.yaml: SecRequestBodyJsonDepthLimit 64 (was 1024 from
  the recommended file). Coraza's JSON flattening is quadratic in depth:
  depth 300 (601 B) = 297 ms, depth 1000 (2 KB) = 3.1 s, past the 100 ms
  fail-open timeout. Over the limit it is a 0.1 ms parse error, which WHP's
  enforce-mode rule denies on a complete body (live: depth 65/300/1000 ->
  403, depth 64 and a depth-9 global-styles body -> 200). Legitimate depth
  measured on WordPress core, Gutenberg, WooCommerce and Elementor JSON:
  max 8.
- test-waf-body-inspection.py: 17 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:21:49 -07:00
shadowdaoandClaude Opus 5.5 e58996d6c9 fix(ssl): retire SAN-overlapping certs, never publish expired lineages, report no-op renewals
brainjar.net served an expired certificate for two days (2026-10-07..09):
the old brain-jar.com bundle (CN brain-jar.com, SANs incl. all eight
brainjar names) stayed in /etc/haproxy/certs and won the SNI match over the
valid brainjar.net bundle, because superseded-cert cleanup matched on CN
only, and renew-certificates.sh re-published the expired lineage every 12h.

- Retirement now matches on ANY shared name (CN or SAN). The overlapping pem
  is moved to cert-retired/ and its lineage's live/ dir and renewal conf to
  live-retired/ and renewal-retired/ (after HAProxy loads the new bundle).
  Nothing is deleted any more: `certbot delete` is gone from this path.
- Shared-PEM safety: a candidate is kept when it is still the only valid
  certificate for a name in the domains table (retirement_blockers()). Expired
  certs never block; names the caller lists in the new optional
  `unreachable_names` body field do not block.
- /api/ssl/bundle passes --keep-until-expiring and reports `renewed`
  (fingerprint before vs after), `certbot_not_due` and the certificate's
  `not_after`, so WHP can stop recording certbot no-ops as renewals.
- Never publish an expired certificate: renew-certificates.sh and
  sync-certificates.sh skip expired lineages (ERROR-logged, counted
  separately), and cert_publish / publish_pem_bundle refuse expired leaves.
- VERSION 2026.10.3.

Tests: test-cert-write-safety.py 31 -> 47, test-cert-scripts.py 38 -> 44,
each new group verified to fail against a reverted fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:05:07 -07:00
shadowdaoandClaude Opus 5.5 4f21bb2f76 feat(waf): make Coraza see request bodies; stamp complete/partial; 2026.10.2
Measured on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0 (local
haproxy-manager + coraza-spoa + echo backend):

- send-spoe-group fired at header time, so req.body held only what arrived
  with the headers. A 933180 payload whose body came in a second send(),
  2s late, or after Expect: 100-continue passed uninspected; 8 parallel
  14 KB payload-first POSTs: 6/40 blocked before, 40/40 after.
- Add `option http-buffer-request` (Coraza block only). Uploads are not
  held: one buffer is inspected, the rest streams (50 MB upload: sha-ok,
  same timing before/after).
- Stamp X-WHP-WAF-Body: complete|partial for Coraza (client copies
  stripped first, stripped again before the backend). Content-Length:
  exact compare; chunked/no-CL: body+headers < 12 KB => complete. WHP's
  generated rules use it to deny body-parse errors (200002/200003) only on
  complete bodies.
- Pin tune.bufsize 16384 (= default) when Coraza is on. Coraza costs
  ~3.3 ms/KB of body and the 100 ms SPOE timeout fails open: at 32 KB a
  payload at byte 0 of a 30 KB base64 body bypassed. SPOP frames > 65535
  are refused by the agent library.
- coraza-spoa/config.yaml: SecRequestBodyLimit 131072 / InMemory 131072 /
  ProcessPartial (Reject 413s even under DetectionOnly).
- Access log gains wafb= (complete|partial) and wafe= (SPOE error =
  request passed uninspected).
- overrides.conf: document (comment only) that the 920440 promotion turns
  the engine On for every URL with a file extension.
- scripts/test-waf-body-inspection.py (16 tests), wired into the build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:49:27 -07:00
shadowdaoandClaude Opus 5.5 0ce144d2fb fix(api): stop silent clients from wedging the gunicorn thread pool
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.

The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).

- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
  * per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
    re-applied on keep-alive reuse; bounds socket I/O, not handling time
  * watchdog: if every pool thread has been in the same request for more
    than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
    all thread stacks and exit so the master respawns the worker
  * client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
  60s) as defence in depth for any future in-process client; /health now
  reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
  HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
  added to the Docker build gate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:13:12 -07:00
jknapp e134b2b234 Merge pull request 'fix(haproxy): set X-CLIENT-IP instead of adding it' (#11) from fix/x-client-ip-set-not-add into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 2m36s
2026-09-18 23:33:05 +00:00
shadowdaoandClaude Sonnet 5 cc2cfd82a2 chore: bump VERSION to 2026.09.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-18 16:32:53 -07:00