Author SHA1 Message Date
shadowdaoandClaude Sonnet 5.5 99279d5347 chore: bump VERSION to 2026.11.5 (HAProxy 3.2 rc)
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 18m29s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 20:42:04 -08:00
claudeandClaude Sonnet 5.5 ea2d4994f1 fix(image): exact haproxy pin + relay tenant-alert class + supervisor backoff + rc publishing
I1: ARG HAPROXY_EXACT=3.2.26-1~bpo13+1 is the apt pin; build fails unless dpkg-query equals it exactly (fail closed when the repo is unreachable); comment corrected.
I2: relay no longer exempts every [ALERT]; tenant-driven 'has no server available' / 'Server ... is DOWN|UP' are their own rate-limited stdout class; dump blocks have their own budget.
Minor: keyring must hold exactly one pub key (gnupg purged), stderr log + rotated copies 0640, supervisor extracted to scripts/haproxy-supervisor.sh (integer-validated knobs, crash-loop backoff 2/5/10/30, reset after 60s).
C1: release/* branches and rc-* tags publish only :<VERSION>-rc and :<sha>, never :latest; main unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 17:33:27 -08:00
shadowdaoandClaude Sonnet 5.5 c2e070d172 fix(relay): anchor crash patterns to HAProxy message structure; crash-evidence budget + dump cooldown
Security review: unanchored keywords let echoed client data bypass the limiter and open
dump blocks; bounds re-armed immediately. Patterns are now anchored, blocks open only on
the 'Thread N is about to kill' header, a 1000-line/10-min budget caps exempt lines
(overflow goes through the normal bucket and the suppressed summary) and a 60 s cooldown
follows a bound-closed block. Tests: forged keywords, flood, cooldown + 3 new mutants.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 15:08:46 -08:00
shadowdaoandClaude Sonnet 5.5 5e27870608 fix(relay): never rate-limit crash evidence; collapse reload noise; stdout/stderr priority split
The 3.2 canary showed the relay's 400-line bucket, drained by ~110 reload lines
per reload, dropping a crash dump (journal kept only 'Thread 1 is about to kill').
Crash lines and whole dump blocks now bypass the bucket (bounded block), routine
reload noise is collapsed to one line per reload, the remainder keeps the limit
with a per-minute '[relay] suppressed N lines' summary. Benign lines go to stdout
(journald PRIORITY 6), crash evidence to stderr (PRIORITY 3).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 15:04:14 -08:00
Claude a753d300db feat: HAProxy 3.2 LTS (haproxy.debian.net) + capture HAProxy stderr + 2 s supervisor
DRAFT for review, not merged. Context: HAProxy 3.0.11 (Debian trixie) aborts via
its own watchdog in the DNS resolver (haproxy #3101, fixed in 3.0.13/3.1.11/3.2.6).

* Dockerfile: haproxy from https://haproxy.debian.net trixie-backports-3.2 (committed
  signing key, fingerprint asserted, series pin 3.2.*, optional HAPROXY_EXACT),
  native OpenSSL QUIC asserted (no compat layer), /haproxy/HAPROXY_VERSION.
* start HAProxy as `haproxy -W -db` (spawn_haproxy_master): `daemon` sent the
  master's stderr to /dev/null, losing ALERTs and the ha_panic thread dump.
  stderr -> /var/log/haproxy/haproxy-stderr.log, forwarded to container stderr by
  scripts/haproxy-stderr-relay.py (rate limited, rotated).
* supervisor: pidof probe every 2 s (was python every 15 s), exponential backoff
  on failed restarts.
* tests: relay unit tests; rollback/cert stubs print "Loading success.".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 13:02:33 -08:00
jknapp 46ea498db3 Merge pull request #19: haproxy pin 3.0.11 (any Debian revision) + multi-arch publish [skip ci] 2026-11-12 02:04:32 +00:00
shadowdaoandClaude Sonnet 5.5 d59d7e0ad4 feat(ci): publish haproxy-manager-base multi-arch (amd64+arm64); pin haproxy by upstream version (2026.11.3)
- build-push.yaml: one job on the x86 runner; amd64 and arm64 (QEMU,
  continue-on-error) are pushed by digest to Gitea+GHCR, then
  scripts/assemble-multiarch-index.sh creates :latest/:<version>/:<sha>
  (same template as coraza-spoa). arm64 failure publishes amd64-only; a
  stale arm64 digest is never re-attached. Adds workflow_dispatch.
- Dockerfile: pin the reviewed UPSTREAM version (HAPROXY_UPSTREAM=3.0.11) via
  an apt preference so the newest Debian security revision is taken
  (3.0.11-1+deb13u4 today), and fail the build unless dpkg and `haproxy -v`
  both report that upstream version. The exact +deb13uN pin froze the image
  on the superseded u3 and would have hard-failed once u3 left the archive.
- The existing build gate (unit suites + haproxy -c) is unchanged and runs
  for each platform.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 17:42:22 -08:00
jknapp 3616fa409f Merge pull request #17: coraza-spoa multi-arch (amd64 + arm64) [skip ci] 2026-11-12 01:15:31 +00:00
jknapp 48edc09e8d Merge pull request #18: mirror index copy works on Gitea (pinned buildx) [skip ci] 2026-11-12 01:03:16 +00:00
shadowdaoandClaude Sonnet 5.5 cf8f09612f fix(ci): mirror index copy works on Gitea (pinned buildx, digest assertion via raw bytes) [skip ci]
Run 5156 failed copying python:3.12-slim / golang:1.25 into Gitea:
`PUT .../manifests/<tag>-staging: 404`, leaving a stray single-manifest
<tag>-staging version. Reproduced on real Gitea with scratch packages:
buildx <= 0.20.x (runner distro buildx is 0.15.1) pushes the children of a
copied index to the TAG path; Gitea only records manifests PUT by digest, so
the index PUT is rejected (MANIFEST_BLOB_UNKNOWN -> 404). buildx >= 0.24
pushes children by digest and works; Gitea then preserves the index digest, so
the digest-equality assertion stays.

- workflow: install a sha256-verified pinned buildx (v0.28.0) instead of
  setup-buildx-action (imagetools needs no builder) and pass it as $BUILDX
- mirror-index-copy.sh: BUILDX/MIN_BUILDX guard refuses buildx < 0.24 before
  any registry traffic; digest computed as sha256 of the raw index bytes
  (buildx 0.28 ignores `inspect --format` here)
- test: attestation-children index and old-buildx guard cases

Merge with [skip ci] in the merge title: any push to main triggers the HMB
image build.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 17:01:09 -08:00
jknapp 74fc9ef183 Merge pull request #16: mirror workflow keeps the multi-arch index [skip ci] 2026-11-12 00:46:13 +00:00
shadowdaoandClaude Sonnet 5.5 a476bb5352 coraza-spoa: multi-arch (amd64+arm64) by cross-compiling; push-by-digest + imagetools index
Dockerfile: Go stages pinned to $BUILDPLATFORM, GOOS/GOARCH from TARGETOS/
TARGETARCH (CGO already off); amd64 rootfs/binary byte-identical to before.
Workflow: one trusted x86 job, amd64 + arm64 pushed untagged by digest to both
registries, arm64 continue-on-error, tag assembled by
scripts/assemble-multiarch-index.sh (arm64 only from this run's successful
step; otherwise amd64-only). Offline test + local e2e rehearsal script.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 16:44:24 -08:00
shadowdaoandClaude Sonnet 5.5 905b2894b6 fix(ci): mirror base images as whole multi-arch indexes, with assertions
mirror-base-image.yaml pulled the upstream index digest on an amd64 runner and
re-pushed it, publishing only the amd64 manifest; the python/golang mirrors
had no arm64 variant. Replace pull/tag/push with a registry-to-registry
`imagetools create` of the digest-pinned index via .gitea/scripts/
mirror-index-copy.sh: copy to <tag>-staging, assert linux/amd64+linux/arm64
are listed and the index digest equals the Docker Hub digest, then move the
real tag and re-assert. A failed assertion never touches the real tag.
Adds a local-registry test (.gitea/scripts/test-mirror-index-copy.sh).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-11 16:40:22 -08:00
jknapp 50c58c4e9e Default splash for unrouted hosts (404+noindex, answered before the WAF) and in-container access log (2026.11.2) (#15)
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m36s
2026-11-08 20:03:23 +00:00
jknapp 1ee5760c1b Merge pull request 'ci: build through the internal package cache; mirror bases by Docker Hub digest (2026.10.7)' (#14) from ci/pkg-cache into main
Build and push coraza-spoa / Build-and-Push (push) Successful in 1m10s
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m45s
2026-10-22 14:20:16 +00:00
shadowdaoandClaude Opus 5.5 dc1cbeb12a ci: build through the internal package cache; mirror bases by Docker Hub digest (2026.10.7)
- Dockerfile: optional ARG PKG_CACHE; the apt RUN (dist-upgrade + pinned
  haproxy) rewrites sources via scripts/pkg-cache-repos.sh apt and restores
  them before the RUN ends. Without PKG_CACHE: upstream Debian, as before.
- coraza-spoa/Dockerfile: same for the build stage's apt RUN; go mod
  download / go build (build + catalog stages) use go-env (GOPROXY via the
  cache with proxy.golang.org fallback; GOSUMDB unchanged) and restore.
  Build context is ./coraza-spoa, so coraza-spoa/scripts/pkg-cache-repos.sh
  and coraza-spoa/keys/pkg-cache/ are byte-identical copies of the root ones.
- build-push.yaml / build-push-coraza.yaml: "Probe package cache" step
  (TLS with the pinned CA), PKG_CACHE build-arg. Bases come from
  repo.anhonesthost.net / gcr.io, not docker.io: registry mirror not used.
- mirror-base-image.yaml: resolve the tag's digest directly at Docker Hub
  (.gitea/scripts/resolve-base-image.sh) and pull docker.io/library/<img>@sha256
  so the mirrored content is exactly the digest Docker Hub reported; same
  destination tags. Values reach run: via env only.
- Canonical copies (byte-identical) of scripts/pkg-cache-repos.sh,
  .gitea/scripts/resolve-base-image.sh and keys/pkg-cache/ from
  cloud-apache-container.
- VERSION 2026.10.7 so the rebuild does not overwrite the 2026.10.6 tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 07:19:54 -07:00
jknapp 4345732dad Merge pull request 'fix(edge): CVE-2026-109618/-109619 multipart mitigations (2026.10.6)' (#13) from fix/cve-2026-109618-multipart-edge into main
Build and push coraza-spoa / Build-and-Push (push) Successful in 54s
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m18s
Merge pull request #13: CVE-2026-109618/-109619 multipart edge mitigations (2026.10.6)
2026-10-21 14:38:27 +00:00
shadowdaoandClaude Opus 5.5 729098a8f9 fix(edge): CVE-2026-109618/-109619 multipart mitigations (2026.10.6)
PHP "RCE via OOM corruption" (GHSA-w8hw-w9fx-j79w) and the session
upload-progress variant (GHSA-6wmf-4rv5-p3rp), interim until patched PHP is
on every customer image.

HAProxy (hap_listener.tpl, every frontend render, WAF on or off):
  deny 400 multipart/form-data with no Content-Length (HTTP/1.1 chunked,
  HTTP/2-3 streamed bodies) -- vector (a), the post_max_size bypass.
  Measured: Chrome 155 (h2 and h3) and Firefox 153 send content-length on
  form submits, FormData fetch and XHR; only a ReadableStream fetch body
  (duplex:'half') has none.

Coraza (coraza-spoa/overrides.conf, 209610-209616):
  209610       ARGS_NAMES/FILES_NAMES with more than 16 '['
  209611-613   part-header budget (4/part, deny 32 below zero)
  209614-615   more than 64 empty-filename file parts
  209616       multipart PHP_SESSION_UPLOAD_PROGRESS field
Ids sit in the engine range 200000-209999 that WHP's break-glass rule
(991000001) already removes, so no WHP release is needed for break-glass
to lift them. Each flips the engine On for itself, like the day-one
promotions, so they deny in every mode the sidecar runs in.

Limitation: Coraza sees only the first 16 KB of a body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:27:42 -07:00
jknapp b6cc3adfaa Merge pull request 'fix(waf): HTTP/3 bodies were stamped complete when truncated (2026.10.5)' (#12) from fix/waf-h3-body-stamp into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 4m49s
2026-10-19 21:22:13 +00:00
shadowdaoandClaude Opus 5.5 77812399d4 chore: VERSION 2026.10.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:59:45 -07:00
shadowdaoandClaude Opus 5.5 2cc6790db1 fix(waf): stamp X-WHP-WAF-Body from the Content-Length header, not req.body_size
req.body_size is "buffered + htx->extra". In HAProxy 3.0 only the H1 and
H2 muxes set htx->extra from Content-Length; the H3 mux never does, so
over HTTP/3 req.body_size == req.body_len and any body cut at one buffer
(~15 KB) was stamped `complete`. Coraza then failed to parse the
truncated JSON (200002) and WHP's 991900020 returned a 403.

Measured on whp01, Oct 6-19 2026: 73,795 H3 `complete` stamps and 0
`partial`; 7 legitimate requests 403'd (6 Divi sync-to-server saves of
265-417 KB, 1 Groundhogg preview of 18 KB). Reproduced in a lab on
haproxy 3.0.11 with QUIC. H1/H2/H3 all reject a Content-Length that
doesn't match the DATA received, so the header is a protocol-independent
body length. No change on H1/H2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:59:29 -07:00
shadowdaoandClaude Opus 5.5 cbbd15b787 feat(waf): log why Coraza acted (wrule/wrules/wscore); VERSION 2026.10.4
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 3m14s
coraza-spoa v0.7.1 already returns the interrupting rule id
(txn.coraza.ruleid) and, when the request message sends
exportRuleIDs=bool(true), the comma list of every matched CRS attack
rule (txn.coraza.rule_ids) plus the inbound anomaly score. Send the arg
and append all three to the access log so a block's cause (e.g.
920440,949110 score 5) is visible in haproxy.log without opening the
Coraza audit log. Fields are appended last; existing ones unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:20:46 -07:00
shadowdaoandClaude Opus 5.5 519f2d8ade feat(waf): Coraza transaction ID = HAProxy request reference UUID
Send `id=unique-id` in the coraza-req SPOE message so coraza-spoa calls
waf.NewTransactionWithID() with HAProxy's unique-id (the %[uuid()] shown
on 403-waf.html, sent as X-Request-Reference and logged as `id=` in
haproxy.log). A customer-quoted reference now greps directly to
transaction.id in /var/log/coraza/audit.log and match.unique_id in
spoa.log. Previously coraza-spoa invented a random 16-letter A-Z ID
that joined to nothing on the HAProxy side.

Measured against coraza-spoa v0.7.1 source (no version bump needed):
- internal/agent.go HandleSPOE: only `app` is positional (must be the
  first KV, else Panic); the template comment claiming all args are
  parsed positionally was wrong and is corrected.
- internal/application.go HandleRequest: every other arg is matched by
  name; `id` is recognised and passed to NewTransactionWithID. Empty/
  NULL id falls back to the random ID, so this cannot break inspection.
- coraza v3.7.0 places no format/length constraint on the ID.

response_check is false so there is no coraza-res message; the template
and a test now require any future coraza-res to send the same id
(HandleResponse looks the request transaction up by it).

Tests: new CorazaTransactionIdIsTheRequestReference class in
test-waf-body-inspection.py (fails without the change). Full in-image
suite + validate-rendered-config (haproxy 3.0.11 -c) pass; a negative
control with a bogus fetch name is rejected by haproxy -c.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:03:59 -07:00
shadowdaoandClaude Opus 5.5 7ae8a4780c test(api): starvation test asserts no-starvation; latency bounds only under HAPROXY_MGR_STRICT_TIMING
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 4m45s
CI runner measured median 14.97s on 2 probes (load-dependent), failing the
build gate on a timing bound rather than the property under test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:40:22 -07:00
shadowdaoandClaude Opus 5.5 765b2ce00a Merge fix/ssl-renewal-durable into release/2026.10.3
Build and push coraza-spoa / Build-and-Push (push) Successful in 3m9s
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 5m7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:22:52 -07:00
shadowdaoandClaude Opus 5.5 3712860429 Merge feat/waf-body-hardening into release/2026.10.3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:22:46 -07:00
shadowdaoandClaude Opus 5.5 17c9a8faf8 test(ssl): certbot re-issue test skips on ANY certbot import/init failure
The skip guard caught only ImportError; a half-installed or version-skewed
certbot can fail on import with other exceptions (verified with a stub that
raises RuntimeError: previously an ERROR, now a skip). new_lineage() itself
stays outside the guard - it is the assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:26:19 -07:00
shadowdaoandClaude Opus 5.5 957fba29d9 fix(api): refuse non-local peers on :8000; count header deadline from enqueue
Re-review R1: haproxy-manager shares client-net with every tenant
container, so tenant PHP could reach haproxy-manager:8000 directly (the
127.0.0.1 publish only removed the internet path). On the ports in
HAPROXY_MGR_PEER_FILTER_PORTS (default 8000; NOT :8080, fed by HAProxy
over loopback) the overridden accept() closes any peer not in
HAPROXY_MGR_API_ALLOWED_PEERS (default 127.0.0.1,::1,gateway), before a
TConn, slot, poller registration or thread is used. "gateway" is read
from /proc/net/route: measured, host callers via the 127.0.0.1 publish
arrive from the bridge gateway (docker-proxy). Bad entries fail closed
and are logged; refusals are logged at most once a minute as a per-peer
summary.

Header deadline is now max(enqueue + 10s, pickup + 1s)
(HAPROXY_MGR_HEADER_PICKUP_GRACE): 40 reconnecting drippers no longer
starve legit requests (before: every probe >30s; after: median ~9s).

Tests: 21 (was 15): allowlist parsing/matching incl. v4-mapped, fail
closed, /proc/net/route parsing, real-gunicorn filtered vs unfiltered
port with rate-limited logging, listed peer served, 40 reconnecting
drippers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:25:32 -07:00
shadowdaoandClaude Opus 5.5 2da2020305 fix(waf): stop promoting 944110/944120 chain starters (round 3)
Owner ruling (same mode contract as 920440: no accidental enforcement from
bare words). 944110 and 944120 are CRS chains whose first link matches
`runtime|processbuilder` / `getproperty|forclosure|filewriter|...`
anywhere in the request; the range promotion 944100-944300 put
ctl:ruleEngine=On on those starters, so any request containing the word
ran with full anomaly enforcement, in detect_only too.

Promotion is now 944100 and 944130-944300. Live (haproxy-manager-base
:wafbody-r3 + coraza-spoa :wafbody-r3), both modes: `?q=runtime&<SQLi>`
and `?q=getproperty&<SQLi>` 403 -> 200 on detect_only; Log4Shell in the
query (plain and ${lower:j} obfuscated), User-Agent and a JSON body still
403.

test-waf-body-inspection.py: 21 tests (+2: no CRS chain starter
920440/944110/944120 is covered by a promotion; the Log4Shell ids keep
theirs). Red on round 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:10:41 -07:00
shadowdaoandClaude Opus 5.5 234029c249 fix(waf): promote 920440 chain-safely, not on its chain starter (round 2)
Owner decision. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on
920440's chain STARTER (`REQUEST_BASENAME "@rx \.([^.]+)$"`), which matches
every URL with a file extension, so every *.php/*.css request was evaluated
with the engine On: full CRS anomaly enforcement (949110) and 200002, in
detect_only mode too.

Replaced by `SecRule TX:EXTENSION "@within %{tx.restricted_extensions}"
... ctl:ruleEngine=On` (id 992920440, phase 1, after 920440). CRS 4.25.0
sets tx.extension in 920440's first link (REQUEST-920...conf:667, the
coreruleset version coraza-spoa v0.7.1 pins). Live, both modes:
.bak/.sql/.save/.old/.swp probes still 403; benign .php/.css 200; a
942100 SQLi on /x.php and the onclick / "select ... union" admin saves are
no longer 403 in detect_only.

test-waf-body-inspection.py: 19 tests (2 new, repo-checkout only).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:00:22 -07:00
shadowdaoandClaude Opus 5.5 8c7fd93387 fix(ssl): review round 1 - exact --cert-name lineage, retire whole lineage tree, sweep expired pems
- H1: request_ssl_bundle uses live/<primary> exactly (certbot was given
  --cert-name <primary>) instead of find_certbot_live_dir(), which returns the
  newest of <primary>/<primary>-NNNN. After a "not yet due" no-op that
  published a stale -0001 and retired the real <primary> lineage. keep_lineage
  is now the primary; the bundle's own lineage is never a retirement candidate.
- H2: retire_lineage_tree() moves live/<x>, archive/<x> and renewal/<x>.conf
  together to /etc/letsencrypt/retired/<stamp>/{live,archive,renewal}/ (same
  depth, relative symlinks still resolve, restore is three mv's). Leaving
  archive/<x> behind made certbot's new_lineage() refuse the name forever
  (CertStorageError) and leave an empty renewal conf. Test re-issues the name
  through certbot's own storage code after retirement.
- M2: cert_retire_expired_pems moves already-published EXPIRED pems out of
  /etc/haproxy/certs to cert-retired/ (never the last pem, never the default
  self-signed cert) and the scripts reload when anything was retired.
- GET /api/capabilities + `capabilities` in the bundle response, and VERSION is
  copied into the image, so WHP can refuse whole-site re-bundling against an
  older edge.

Tests: test-cert-write-safety.py 47 -> 52, test-cert-scripts.py 44 -> 50; each
fix reverted in a scratch copy turns its suite red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:00:03 -07:00
shadowdaoandClaude Opus 5.5 fd06b30d78 fix(api): reap never-readable conns, bound total header time, harden watchdog
Review round 1 on 0ce144d (FIX-FIRST):

F1: stock gthread never reaps a connection that never sends a byte; it
holds a worker_connections slot forever and at 1000 per worker the run
loop stops polling, wedging the worker permanently (survives the
attacker disconnecting). accept/on_client_socket_readable/
murder_keepalived now track such conns in a FIFO with a deadline
(HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s) and close expired ones on every
loop pass, including the at-capacity branch.

F2: slowloris (1 header byte per <20s) never tripped the per-recv
timeout. A total request-line+headers deadline
(HAPROXY_MGR_HEADER_DEADLINE, 10s) is armed in handle() and cleared in
handle_request(); a 1s monitor loop shutdown(SHUT_RDWR)s expired
sockets. Armed per thread, not at accept, so legit requests queued
behind busy threads are never cut for waiting.

F4: monitor loop wrapped in try/except with logging; wedge threshold
1200s -> 3600s (multi-domain cert requests run 180s per domain).
F5: docstring now says the timeout bounds each whole sendall().
F6: warn when gunicorn != 23.0.0 (worker overrides gthread internals).
API key compared with hmac.compare_digest.

Tests: +3 production-shape (2x4) tests: 2050 idle conns reaped and
/health recovers while still held; 8 drip conns cut within the deadline
with /health responsive; slow-but-complete headers still served.
15 tests, in the Docker build gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:58:28 -07:00
shadowdaoandClaude Opus 5.5 cc0652e0c0 fix(waf): 400 duplicate Content-Type; JSON depth limit 64 (round 1)
- hap_listener.tpl (Coraza block): `http-request deny deny_status 400 if
  { req.fhdr_cnt(content-type) gt 1 }`. With two Content-Type lines Coraza
  evaluates every value (text/xml on either line selects its XML processor,
  which accepts a form body without error) while Apache merges the lines
  and PHP parses the body as a form. Measured with Coraza v3.7.0 + PHP 8.3:
  `application/x-www-form-urlencoded` + `text/xml` put the payload in
  $_POST uninspected. Live: two lines -> 400 over HTTP/1.1 and HTTP/2; one
  comma-joined line (browser XHR double-set) is unaffected.
- coraza-spoa/config.yaml: SecRequestBodyJsonDepthLimit 64 (was 1024 from
  the recommended file). Coraza's JSON flattening is quadratic in depth:
  depth 300 (601 B) = 297 ms, depth 1000 (2 KB) = 3.1 s, past the 100 ms
  fail-open timeout. Over the limit it is a 0.1 ms parse error, which WHP's
  enforce-mode rule denies on a complete body (live: depth 65/300/1000 ->
  403, depth 64 and a depth-9 global-styles body -> 200). Legitimate depth
  measured on WordPress core, Gutenberg, WooCommerce and Elementor JSON:
  max 8.
- test-waf-body-inspection.py: 17 tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:21:49 -07:00
shadowdaoandClaude Opus 5.5 e58996d6c9 fix(ssl): retire SAN-overlapping certs, never publish expired lineages, report no-op renewals
brainjar.net served an expired certificate for two days (2026-10-07..09):
the old brain-jar.com bundle (CN brain-jar.com, SANs incl. all eight
brainjar names) stayed in /etc/haproxy/certs and won the SNI match over the
valid brainjar.net bundle, because superseded-cert cleanup matched on CN
only, and renew-certificates.sh re-published the expired lineage every 12h.

- Retirement now matches on ANY shared name (CN or SAN). The overlapping pem
  is moved to cert-retired/ and its lineage's live/ dir and renewal conf to
  live-retired/ and renewal-retired/ (after HAProxy loads the new bundle).
  Nothing is deleted any more: `certbot delete` is gone from this path.
- Shared-PEM safety: a candidate is kept when it is still the only valid
  certificate for a name in the domains table (retirement_blockers()). Expired
  certs never block; names the caller lists in the new optional
  `unreachable_names` body field do not block.
- /api/ssl/bundle passes --keep-until-expiring and reports `renewed`
  (fingerprint before vs after), `certbot_not_due` and the certificate's
  `not_after`, so WHP can stop recording certbot no-ops as renewals.
- Never publish an expired certificate: renew-certificates.sh and
  sync-certificates.sh skip expired lineages (ERROR-logged, counted
  separately), and cert_publish / publish_pem_bundle refuse expired leaves.
- VERSION 2026.10.3.

Tests: test-cert-write-safety.py 31 -> 47, test-cert-scripts.py 38 -> 44,
each new group verified to fail against a reverted fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:05:07 -07:00
shadowdaoandClaude Opus 5.5 4f21bb2f76 feat(waf): make Coraza see request bodies; stamp complete/partial; 2026.10.2
Measured on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0 (local
haproxy-manager + coraza-spoa + echo backend):

- send-spoe-group fired at header time, so req.body held only what arrived
  with the headers. A 933180 payload whose body came in a second send(),
  2s late, or after Expect: 100-continue passed uninspected; 8 parallel
  14 KB payload-first POSTs: 6/40 blocked before, 40/40 after.
- Add `option http-buffer-request` (Coraza block only). Uploads are not
  held: one buffer is inspected, the rest streams (50 MB upload: sha-ok,
  same timing before/after).
- Stamp X-WHP-WAF-Body: complete|partial for Coraza (client copies
  stripped first, stripped again before the backend). Content-Length:
  exact compare; chunked/no-CL: body+headers < 12 KB => complete. WHP's
  generated rules use it to deny body-parse errors (200002/200003) only on
  complete bodies.
- Pin tune.bufsize 16384 (= default) when Coraza is on. Coraza costs
  ~3.3 ms/KB of body and the 100 ms SPOE timeout fails open: at 32 KB a
  payload at byte 0 of a 30 KB base64 body bypassed. SPOP frames > 65535
  are refused by the agent library.
- coraza-spoa/config.yaml: SecRequestBodyLimit 131072 / InMemory 131072 /
  ProcessPartial (Reject 413s even under DetectionOnly).
- Access log gains wafb= (complete|partial) and wafe= (SPOE error =
  request passed uninspected).
- overrides.conf: document (comment only) that the 920440 promotion turns
  the engine On for every URL with a file extension.
- scripts/test-waf-body-inspection.py (16 tests), wired into the build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:49:27 -07:00
shadowdaoandClaude Opus 5.5 0ce144d2fb fix(api): stop silent clients from wedging the gunicorn thread pool
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.

The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).

- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
  * per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
    re-applied on keep-alive reuse; bounds socket I/O, not handling time
  * watchdog: if every pool thread has been in the same request for more
    than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
    all thread stacks and exit so the master respawns the worker
  * client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
  60s) as defence in depth for any future in-process client; /health now
  reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
  HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
  added to the Docker build gate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:13:12 -07:00
jknapp e134b2b234 Merge pull request 'fix(haproxy): set X-CLIENT-IP instead of adding it' (#11) from fix/x-client-ip-set-not-add into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 2m36s
2026-09-18 23:33:05 +00:00
shadowdaoandClaude Sonnet 5 cc2cfd82a2 chore: bump VERSION to 2026.09.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-18 16:32:53 -07:00
shadowdaoandClaude Sonnet 5 ab3ad42625 fix(haproxy): set X-CLIENT-IP instead of adding it
add-header let a client-supplied X-CLIENT-IP survive next to ours, so a
backend reading the first occurrence got the attacker's value. set-header
replaces any inbound copy. Refs #10.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-18 16:32:49 -07:00
jknapp 43e522104b Merge pull request 'fix(domain-removal): stop deleting certificates other live sites are served from' (#9) from fix/remove-domain-shared-cert-refcount into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m22s
2026-08-23 14:52:02 +00:00
jknapp a00431854d Merge pull request 'fix(ip-blocking): the runtime map fast path has never once run' (#8) from fix/runtime-map-worker-prefix-and-path into main
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 2m1s
Reviewed-on: #8
2026-08-22 18:20:06 +00:00
75 changed files with 7762 additions and 302 deletions

No files matched your search

+94
View File
@@ -0,0 +1,94 @@
#!/bin/bash
## mirror-index-copy.sh SRC DST — CI only (not copied into images).
## SRC registry/path@sha256:<index digest> (digest-pinned upstream reference)
## DST registry/path:<tag> (the mirror tag consumers FROM)
##
## Registry-to-registry copy of the WHOLE upstream image index (every
## platform), then assertions. No local `docker pull`, so nothing is flattened
## to the runner's platform (the old pull/tag/push step re-published only the
## amd64 manifest, leaving the mirror without an arm64 variant).
##
## Ordering (protects a previously multi-arch DST tag):
## 1. copy SRC -> DST_REPO:<tag>-staging (real tag untouched)
## 2. assert staging: index lists every REQUIRED platform and its digest
## equals the digest in SRC
## 3. only then point DST at that digest (registry-side, no data transfer)
## 4. re-assert on DST
## A failure at 1/2 exits non-zero before the real tag is touched. A failure
## at 4 cannot be rolled back by this script (the tag already moved) but it is
## loud and the same digest was already proven at step 2.
##
## Gitea compatibility (measured against repo.anhonesthost.net, 2026-11-11):
## * Gitea's registry accepts whole indexes (OCI index, Docker manifest list,
## attestation children, 14-16 children) and preserves the index digest
## byte-for-byte, so the digest-equality assertion below holds there too.
## * It rejects an index PUT whose child manifests are not already stored as
## by-DIGEST versions (404 MANIFEST_BLOB_UNKNOWN). buildx <= 0.20.x pushes the
## children of a copied index to the TAG path (works on registry:2, not on
## Gitea): the copy dies with `PUT .../manifests/<tag>: 404` and leaves a
## stray single-manifest version named <tag>. buildx >= 0.24 pushes children by
## digest and works. The distro buildx on the runner is 0.15.1, so the
## workflow installs a pinned newer one and this script refuses to run on an
## older one (MIN_BUILDX) instead of leaving a stray version behind.
##
## Env: REQUIRED_PLATFORMS (default "linux/amd64 linux/arm64").
## BUILDX (default "docker buildx"): the buildx command; may be the path of
## a pinned docker-buildx binary.
## IMAGETOOLS (default "$BUILDX imagetools").
## MIN_BUILDX (default 0.24.0; empty disables the version guard).
## (buildx talks plain HTTP to a localhost registry, so the test needs no flags.)
set -euo pipefail
SRC="${1:?usage: mirror-index-copy.sh SRC@sha256:DIGEST DST_REPO:TAG}"
DST="${2:?usage: mirror-index-copy.sh SRC@sha256:DIGEST DST_REPO:TAG}"
REQUIRED_PLATFORMS="${REQUIRED_PLATFORMS:-linux/amd64 linux/arm64}"
BUILDX="${BUILDX:-docker buildx}"
MIN_BUILDX="${MIN_BUILDX-0.24.0}"
read -r -a BX <<<"$BUILDX"
read -r -a IMGTOOLS <<<"${IMAGETOOLS:-${BUILDX} imagetools}"
fail() { echo "::error::mirror-index-copy: $*" >&2; exit 1; }
# Version guard: old buildx corrupts the copy on Gitea (see header).
if [ -n "$MIN_BUILDX" ]; then
have_bx=$("${BX[@]}" version 2>/dev/null | awk 'NR==1{print $2}'); have_bx="${have_bx#v}"
[[ "$have_bx" =~ ^[0-9]+\.[0-9]+\.[0-9]+ ]] || fail "cannot determine buildx version from '${BUILDX} version'"
[ "$(printf '%s\n%s\n' "$MIN_BUILDX" "${have_bx%%[-+]*}" | sort -V | head -1)" = "$MIN_BUILDX" ] \
|| fail "buildx ${have_bx} is too old (need >= ${MIN_BUILDX}): older buildx pushes index children by tag, which Gitea rejects"
fi
# Validate shapes (registry host may carry :port, e.g. localhost:5055).
[[ "$SRC" =~ ^[a-z0-9][a-z0-9._:/-]*@sha256:[0-9a-f]{64}$ ]] || fail "bad source reference '${SRC}'"
[[ "$DST" =~ ^[a-z0-9][a-z0-9._:/-]*:[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || fail "bad destination '${DST}'"
src_digest="${SRC#*@}"
dst_repo="${DST%:*}"; dst_tag="${DST##*:}"
STAGING="${dst_repo}:${dst_tag}-staging"
# assert_ref REF: REF must be an index listing every required platform, and
# its manifest digest must equal the upstream index digest.
assert_ref() {
local ref="$1" raw have want got
raw=$("${IMGTOOLS[@]}" inspect "$ref" --raw) || fail "cannot inspect ${ref}"
have=$(jq -r '[.manifests[]? | select(.platform.os != "unknown") | "\(.platform.os)/\(.platform.architecture)"] | unique | join(" ")' <<<"$raw")
[ -n "$have" ] || fail "${ref} is not a multi-platform index (single-platform manifest?); platforms: none"
for want in $REQUIRED_PLATFORMS; do
case " $have " in *" $want "*) ;; *) fail "${ref} is missing ${want} (has: ${have})" ;; esac
done
# Digest = sha256 of the exact index bytes the registry serves (the same
# thing a client pins by). Not `--format {{.Manifest.Digest}}`: buildx 0.28
# ignores --format here and prints the human summary instead.
got="sha256:$(printf '%s' "$raw" | sha256sum | cut -d' ' -f1)"
[ "$got" = "$src_digest" ] || fail "${ref} digest ${got} != upstream ${src_digest}"
echo "ok: ${ref} platforms=[${have}] digest=${got}"
}
echo "::group::copy index ${SRC} -> ${STAGING}"
"${IMGTOOLS[@]}" create -t "$STAGING" "$SRC"
echo "::endgroup::"
assert_ref "$STAGING"
echo "::group::promote staging -> ${DST}"
"${IMGTOOLS[@]}" create -t "$DST" "${dst_repo}@${src_digest}"
echo "::endgroup::"
assert_ref "$DST"
echo "mirror ok: ${DST} == ${src_digest}"
+35
View File
@@ -0,0 +1,35 @@
#!/bin/bash
## resolve-base-image.sh IMAGE:TAG — CI only (not copied into images).
## Prints IMAGE@sha256:<digest> for a Docker Hub tag, looked up DIRECTLY at
## registry-1.docker.io over HTTPS (never via the pkg-cache mirror), so the
## build can say `FROM IMAGE@sha256:...` and the mirror can only serve
## content-addressed blobs that BuildKit verifies. Exits non-zero (prints
## nothing on stdout) if the lookup fails; the workflow then skips the mirror
## and pulls the tag directly from Docker Hub. HEAD requests do not count
## against Docker Hub pull rate limits.
## Canonical copy: cloud-apache-container/.gitea/scripts/resolve-base-image.sh
## (docs/pkg-cache.md there); other repos carry identical copies.
set -euo pipefail
ref="${1:?usage: resolve-base-image.sh IMAGE:TAG}"
name_re='^[a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)?$'
tag_re='^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$'
name="${ref%:*}"; tag="${ref##*:}"
[ "$name" != "$ref" ] || { echo "resolve-base-image: '$ref' has no :tag" >&2; exit 2; }
[[ "$name" =~ $name_re ]] || { echo "resolve-base-image: bad image name '$name'" >&2; exit 2; }
[[ "$tag" =~ $tag_re ]] || { echo "resolve-base-image: bad tag '$tag'" >&2; exit 2; }
repo="$name"; [[ "$repo" == */* ]] || repo="library/$repo" # official images
curl_opts=(-fsS --proto '=https' --retry 3 --retry-all-errors --max-time 20)
token=$(curl "${curl_opts[@]}" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" |
python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')
[ -n "$token" ] || { echo "resolve-base-image: no Docker Hub token for $repo" >&2; exit 1; }
hdrs=$(curl "${curl_opts[@]}" -I \
-H "Authorization: Bearer ${token}" \
-H 'Accept: application/vnd.oci.image.index.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
"https://registry-1.docker.io/v2/${repo}/manifests/${tag}")
digest=$(printf '%s\n' "$hdrs" | tr -d '\r' | awk 'tolower($1)=="docker-content-digest:" {print $2}' | tail -n1)
[[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "resolve-base-image: no valid digest for $ref (got '${digest}')" >&2; exit 1; }
printf '%s@%s\n' "$name" "$digest"
+76
View File
@@ -0,0 +1,76 @@
#!/bin/bash
## Test for mirror-index-copy.sh. CI/dev only; needs docker + buildx + jq and a
## THROWAWAY registry on REG (default localhost:5055):
## docker run -d --name mirror-test-reg -p 127.0.0.1:5055:5000 registry:2
## (remove it afterwards). Never point REG at a real registry. Builds only
## FROM scratch images, so it needs no network beyond the registry.
## Set MIRROR_TEST_UPSTREAM=docker.io/library/python@sha256:... to also run the
## copy against a real multi-arch upstream digest (anonymous read-only pull).
set -uo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
REG="${REG:-localhost:5055}"
case "$REG" in localhost:*|127.0.0.1:*) ;; *) echo "REG must be a localhost throwaway registry"; exit 2;; esac
curl -fsS -m5 "http://${REG}/v2/" >/dev/null || { echo "no registry on ${REG}"; exit 2; }
run="t$$"; work=$(mktemp -d); trap 'rm -rf "$work"' EXIT
pass=0; failn=0
ok() { echo "PASS: $1"; pass=$((pass+1)); }
bad() { echo "FAIL: $1"; failn=$((failn+1)); }
digest() { docker buildx imagetools inspect "$1" --format '{{.Manifest.Digest}}'; }
mkimg() { # mkimg REPO:TAG PLATFORM -> tiny FROM-scratch image
printf 'FROM scratch\nLABEL t=%s\n' "$2-$run" >"$work/Dockerfile"
docker buildx build -q --platform "$2" --provenance=false --sbom=false \
--output "type=image,name=${1},push=true" "$work" >/dev/null || { echo "build $1 failed"; exit 2; }
}
mkimg "${REG}/${run}/amd64:v" linux/amd64
mkimg "${REG}/${run}/arm64:v" linux/arm64
A=$(digest "${REG}/${run}/amd64:v"); B=$(digest "${REG}/${run}/arm64:v")
# synthetic two-platform upstream index, and a single-platform one
docker buildx imagetools create -t "${REG}/${run}/multi:v" "${REG}/${run}/amd64@${A}" "${REG}/${run}/arm64@${B}" >/dev/null 2>&1 || exit 2
MULTI=$(digest "${REG}/${run}/multi:v"); SINGLE="$A"
# 1. two-platform source: pass, digest preserved, tag lands on source digest
if "$here/mirror-index-copy.sh" "${REG}/${run}/multi@${MULTI}" "${REG}/${run}/dst:1" >"$work/o1" 2>&1 \
&& [ "$(digest "${REG}/${run}/dst:1")" = "$MULTI" ]; then ok "multi-arch copy passes and digest round-trips ($MULTI)"; else bad "multi-arch copy"; cat "$work/o1"; fi
# 2. single-platform source: must FAIL loudly and must not create the real tag
if "$here/mirror-index-copy.sh" "${REG}/${run}/amd64@${SINGLE}" "${REG}/${run}/dst2:1" >"$work/o2" 2>&1; then bad "single-platform source was accepted"; else
grep -q 'is not a multi-platform index\|missing linux/arm64' "$work/o2" && ok "single-platform source rejected: $(grep -m1 '::error::' "$work/o2")" || { bad "failed for the wrong reason"; cat "$work/o2"; }
if docker buildx imagetools inspect "${REG}/${run}/dst2:1" >/dev/null 2>&1; then bad "real tag created despite failure"; else ok "real tag not created on failure"; fi
fi
# 3. existing multi-arch tag must survive a failing (single-platform) refresh
before=$(digest "${REG}/${run}/dst:1")
if "$here/mirror-index-copy.sh" "${REG}/${run}/amd64@${SINGLE}" "${REG}/${run}/dst:1" >"$work/o3" 2>&1; then bad "downgrade accepted"; else
[ "$(digest "${REG}/${run}/dst:1")" = "$before" ] && ok "previously multi-arch :tag untouched after failed refresh" || bad ":tag moved to single-platform"; fi
# 4. malformed references are refused before any registry traffic
"$here/mirror-index-copy.sh" "${REG}/x:tag" "${REG}/${run}/d:1" >/dev/null 2>&1 && bad "tag-only source accepted" || ok "non-digest source refused"
"$here/mirror-index-copy.sh" "${REG}/x@sha256:$(printf 'a%.0s' {1..64})" 'bad dst' >/dev/null 2>&1 && bad "bad dst accepted" || ok "bad destination refused"
# 6. index WITH buildx attestation children (unknown/unknown), like Docker Hub
# official images: copy passes, digest preserved, attestations not counted as platforms
printf 'FROM scratch\nLABEL t=att-%s\n' "$run" >"$work/Dockerfile"
if docker buildx build -q --platform linux/amd64,linux/arm64 --provenance=mode=min --sbom=false \
--output "type=image,name=${REG}/${run}/att:v,push=true" "$work" >/dev/null 2>&1; then
ATT=$(digest "${REG}/${run}/att:v")
if "$here/mirror-index-copy.sh" "${REG}/${run}/att@${ATT}" "${REG}/${run}/attdst:1" >"$work/o6" 2>&1 \
&& [ "$(digest "${REG}/${run}/attdst:1")" = "$ATT" ]; then ok "index with attestation children copied, digest preserved ($ATT)"; else bad "attestation-index copy"; cat "$work/o6"; fi
else bad "could not build attestation index"; fi
# 7. buildx version guard: a too-old buildx is refused BEFORE any registry traffic
# (old buildx leaves a stray single-manifest tag on Gitea)
printf '#!/bin/sh\necho "github.com/docker/buildx v0.15.1 deadbeef"\n' >"$work/old-buildx"; chmod +x "$work/old-buildx"
if BUILDX="$work/old-buildx" "$here/mirror-index-copy.sh" "${REG}/${run}/multi@${MULTI}" "${REG}/${run}/olddst:1" >"$work/o7" 2>&1; then bad "old buildx accepted"; else
grep -q 'too old' "$work/o7" && ok "old buildx refused: $(grep -m1 '::error::' "$work/o7")" || { bad "old-buildx failed for the wrong reason"; cat "$work/o7"; }
docker buildx imagetools inspect "${REG}/${run}/olddst:1-staging" >/dev/null 2>&1 && bad "old buildx touched the registry" || ok "no staging tag created by refused run"
fi
MIN_BUILDX= BUILDX="$work/old-buildx" IMAGETOOLS="docker buildx imagetools" "$here/mirror-index-copy.sh" "${REG}/${run}/multi@${MULTI}" "${REG}/${run}/olddst2:1" >/dev/null 2>&1 \
&& ok "MIN_BUILDX= disables the guard" || bad "guard not disabled by empty MIN_BUILDX"
# 5. optional: real upstream digest
if [ -n "${MIRROR_TEST_UPSTREAM:-}" ]; then
u="${MIRROR_TEST_UPSTREAM}"
if "$here/mirror-index-copy.sh" "$u" "${REG}/${run}/up:1" >"$work/o5" 2>&1 && [ "$(digest "${REG}/${run}/up:1")" = "${u#*@}" ]; then ok "upstream index copied with identical digest ${u#*@}"; grep '^ok:' "$work/o5"; else bad "upstream copy"; tail -5 "$work/o5"; fi
fi
echo "passed=$pass failed=$failn"; [ "$failn" -eq 0 ]
+65 -8
View File
@@ -11,18 +11,45 @@ on:
- main - main
paths: paths:
- 'coraza-spoa/**' - 'coraza-spoa/**'
- 'scripts/assemble-multiarch-index.sh'
- '.gitea/workflows/build-push-coraza.yaml' - '.gitea/workflows/build-push-coraza.yaml'
workflow_dispatch: workflow_dispatch:
# Multi-arch (linux/amd64 + linux/arm64) in ONE job on the trusted x86 runner,
# no QEMU: coraza-spoa/Dockerfile cross-compiles the Go binary
# (--platform=$BUILDPLATFORM + GOOS/GOARCH). Each platform is pushed BY DIGEST
# (untagged) to BOTH registries, then scripts/assemble-multiarch-index.sh
# creates the :latest index from those digests. The arm64 step is
# continue-on-error: if it fails, :latest is published amd64-only exactly as it
# was before multi-arch, and a previous run's arm64 is never re-attached (the
# arm64 digest is read only from THIS run's step output). Registry write
# credentials exist only on this x86 job. Template for the other WHP image
# repos: docs/arm64/IMAGE-PIPELINE-PLAN.md sections 4.1 / 4.4.
jobs: jobs:
Build-and-Push: Build-and-Push:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 30
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v4
- name: Set up QEMU # Internal package cache (CT130 pkg-cache; canonical docs: docs/pkg-cache.md
uses: docker/setup-qemu-action@v3 # in cloud-apache-container). Used ONLY if it answers over TLS with our
# pinned CA; otherwise (down, unreachable, bad/expired cert) this job builds
# from the upstream repositories / default GOPROXY exactly as before. The
# build stages' base (golang:1.25) comes from the in-house registry
# (repo.anhonesthost.net, weekly digest-pinned mirror of Docker Hub -
# mirror-base-image.yaml) and the runtime base from gcr.io, not docker.io,
# so the pkg-cache registry mirror is never used here: repos + Go modules.
# The CA lives under coraza-spoa/ because that is this build's context.
- name: Probe package cache
id: pkgcache
run: |
url=""
if curl -fsS --max-time 5 --cacert coraza-spoa/keys/pkg-cache/ca.crt -o /dev/null https://192.168.1.40/; then url="https://192.168.1.40"; fi
echo "pkg-cache: repos=${url:-<unreachable: upstream>} registry-mirror=<not used: no docker.io base (base from repo.anhonesthost.net)>"
[ -n "$url" ] || echo "::warning::pkg-cache unreachable - building from upstream repositories"
echo "url=${url}" >> "$GITHUB_OUTPUT"
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: https://github.com/docker/setup-buildx-action@v3 uses: https://github.com/docker/setup-buildx-action@v3
@@ -34,7 +61,7 @@ jobs:
username: ${{ secrets.CI_USER }} username: ${{ secrets.CI_USER }}
password: ${{ secrets.CI_TOKEN }} password: ${{ secrets.CI_TOKEN }}
# Mirror to GitHub Container Registry — see build-push.yaml for the # Mirror to GitHub Container Registry - see build-push.yaml for the
# secret/username convention. # secret/username convention.
- name: Login to GHCR - name: Login to GHCR
uses: docker/login-action@v3 uses: docker/login-action@v3
@@ -43,12 +70,42 @@ jobs:
username: shadowdao username: shadowdao
password: ${{ secrets.GHCR_TOKEN }} password: ${{ secrets.GHCR_TOKEN }}
- name: Build Image # amd64: required. Failure here fails the job and publishes nothing, as
# before. Pushed untagged by digest to both registries (same digest in
# each); the tag is only created by the assemble step below.
- name: Build amd64 (push by digest)
id: amd64
uses: docker/build-push-action@v6 uses: docker/build-push-action@v6
with: with:
context: ./coraza-spoa context: ./coraza-spoa
platforms: linux/amd64 platforms: linux/amd64
push: true build-args: |
tags: | PKG_CACHE=${{ steps.pkgcache.outputs.url }}
repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa,ghcr.io/shadowdao/coraza-spoa",push-by-digest=true,name-canonical=true,push=true
ghcr.io/shadowdao/coraza-spoa:latest
# arm64: best effort (cross-compiled, no emulation). Must never block or
# alter the amd64 tag.
- name: Build arm64 (push by digest)
id: arm64
continue-on-error: true
timeout-minutes: 15
uses: docker/build-push-action@v6
with:
context: ./coraza-spoa
platforms: linux/arm64
build-args: |
PKG_CACHE=${{ steps.pkgcache.outputs.url }}
outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa,ghcr.io/shadowdao/coraza-spoa",push-by-digest=true,name-canonical=true,push=true
- name: Assemble :latest index
env:
AMD64_DIGEST: ${{ steps.amd64.outputs.digest }}
ARM64_DIGEST: ${{ steps.arm64.outputs.digest }}
ARM64_OUTCOME: ${{ steps.arm64.outcome }}
run: |
scripts/assemble-multiarch-index.sh \
--amd64 "$AMD64_DIGEST" \
--arm64 "$ARM64_DIGEST" --arm64-outcome "$ARM64_OUTCOME" \
--repo repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa \
--repo ghcr.io/shadowdao/coraza-spoa \
--tag latest
+105 -12
View File
@@ -1,19 +1,59 @@
name: HAProxy Manager Build and Push name: HAProxy Manager Build and Push
run-name: ${{ gitea.actor }} pushed a change to main run-name: ${{ gitea.actor }} pushed ${{ gitea.ref_name }}
on: on:
push: push:
branches: branches:
- main - main
# Release candidates: a push to release/* builds the SAME image but publishes
# ONLY :<VERSION>-rc and :<sha>, never :latest and never the plain :<VERSION>
# (see the "Resolve tags" step). Main keeps its behaviour.
- 'release/**'
tags:
- 'rc-*'
# Manual publish (e.g. after a [skip ci] merge, or to re-publish after a
# failed arm64 leg).
workflow_dispatch:
# Multi-arch (linux/amd64 + linux/arm64) in ONE job on the trusted x86 runner.
# Each platform is pushed BY DIGEST (untagged) to BOTH registries, then
# scripts/assemble-multiarch-index.sh creates :latest, :<version> and :<sha>
# from those digests (same template as build-push-coraza.yaml; see
# docs/arm64/IMAGE-PIPELINE-PLAN.md sections 4.1 / 4.4 in the WHP repo).
# * amd64 is required and built exactly as before; the Dockerfile build gate
# (unit suites + `haproxy -c`) runs in it. Failure = nothing published.
# * arm64 is continue-on-error and runs the SAME Dockerfile gate under QEMU
# (the gate is RUN steps, so it executes on the arm64 binary). If arm64
# fails or times out the tags are published amd64-only, exactly as before
# multi-arch, and a previous run's arm64 is never re-attached (the digest is
# read only from THIS run's step output).
jobs: jobs:
Build-and-Push: Build-and-Push:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 60
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v4
# Internal package cache (CT130 pkg-cache; canonical docs: docs/pkg-cache.md
# in cloud-apache-container). Used ONLY if it answers over TLS with our
# pinned CA; otherwise (down, unreachable, bad/expired cert) this job builds
# from the upstream repositories exactly as before. The base image comes
# from the in-house registry (repo.anhonesthost.net, weekly digest-pinned
# mirror of Docker Hub - mirror-base-image.yaml), not docker.io, so the
# pkg-cache registry mirror is never used here: repos only.
- name: Probe package cache
id: pkgcache
run: |
url=""
if curl -fsS --max-time 5 --cacert keys/pkg-cache/ca.crt -o /dev/null https://192.168.1.40/; then url="https://192.168.1.40"; fi
echo "pkg-cache: repos=${url:-<unreachable: upstream>} registry-mirror=<not used: no docker.io base (base from repo.anhonesthost.net)>"
[ -n "$url" ] || echo "::warning::pkg-cache unreachable - building from upstream repositories"
echo "url=${url}" >> "$GITHUB_OUTPUT"
- name: Set up QEMU - name: Set up QEMU
uses: docker/setup-qemu-action@v3 uses: docker/setup-qemu-action@v3
with:
platforms: arm64
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: https://github.com/docker/setup-buildx-action@v3 uses: https://github.com/docker/setup-buildx-action@v3
@@ -43,19 +83,72 @@ jobs:
id: ver id: ver
run: echo "version=$(cat VERSION)" >> "$GITHUB_OUTPUT" run: echo "version=$(cat VERSION)" >> "$GITHUB_OUTPUT"
- name: Build Image # Publishing strategy. main: :latest + :<version> + :<sha> (unchanged).
# release/* branch or rc-* tag: :<version>-rc + :<sha> ONLY. A release candidate
# is deployed per host by explicit tag (durable per-host pin in WHP); :latest is
# moved to it by a later main publish only after the staged rollout succeeded.
# Any other ref (e.g. a manual workflow_dispatch on a feature branch) is refused
# rather than guessed at.
- name: Resolve tags
id: tags
env:
REF: ${{ gitea.ref }}
VERSION: ${{ steps.ver.outputs.version }}
SHA: ${{ gitea.sha }}
run: |
case "$REF" in
refs/heads/main)
tags="latest ${VERSION} ${SHA}" ;;
refs/heads/release/*|refs/tags/rc-*)
tags="${VERSION}-rc ${SHA}" ;;
*)
echo "::error::refusing to publish from ref ${REF} (only main, release/* and rc-* tags)"; exit 1 ;;
esac
echo "tags=${tags}" >> "$GITHUB_OUTPUT"
echo "publishing tags: ${tags}"
# amd64: required. Pushed untagged by digest to both registries (same
# digest in each); tags are only created by the assemble step below.
- name: Build amd64 (push by digest)
id: amd64
uses: docker/build-push-action@v6 uses: docker/build-push-action@v6
with: with:
platforms: linux/amd64 platforms: linux/amd64
push: true
build-args: | build-args: |
VERSION=${{ steps.ver.outputs.version }} VERSION=${{ steps.ver.outputs.version }}
# Three tags per registry: :latest (moving), :<version> (human-readable PKG_CACHE=${{ steps.pkgcache.outputs.url }}
# release), :<sha> (immutable, guaranteed-unique rollback target). outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base,ghcr.io/shadowdao/haproxy-manager-base",push-by-digest=true,name-canonical=true,push=true
tags: |
repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest # arm64: best effort (QEMU, apt-based image). Must never block or alter
repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ steps.ver.outputs.version }} # the amd64 tags.
repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ gitea.sha }} - name: Build arm64 (push by digest)
ghcr.io/shadowdao/haproxy-manager-base:latest id: arm64
ghcr.io/shadowdao/haproxy-manager-base:${{ steps.ver.outputs.version }} continue-on-error: true
ghcr.io/shadowdao/haproxy-manager-base:${{ gitea.sha }} timeout-minutes: 30
uses: docker/build-push-action@v6
with:
platforms: linux/arm64
build-args: |
VERSION=${{ steps.ver.outputs.version }}
PKG_CACHE=${{ steps.pkgcache.outputs.url }}
outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base,ghcr.io/shadowdao/haproxy-manager-base",push-by-digest=true,name-canonical=true,push=true
# main: three tags per registry: :latest (moving), :<version> (human-readable
# release), :<sha> (immutable, guaranteed-unique rollback target).
# release/* or rc-*: :<version>-rc and :<sha> only (see "Resolve tags").
- name: Assemble tag indexes
env:
AMD64_DIGEST: ${{ steps.amd64.outputs.digest }}
ARM64_DIGEST: ${{ steps.arm64.outputs.digest }}
ARM64_OUTCOME: ${{ steps.arm64.outcome }}
VERSION: ${{ steps.ver.outputs.version }}
TAGS: ${{ steps.tags.outputs.tags }}
run: |
tag_args=()
for t in $TAGS; do tag_args+=(--tag "$t"); done
scripts/assemble-multiarch-index.sh \
--amd64 "$AMD64_DIGEST" \
--arm64 "$ARM64_DIGEST" --arm64-outcome "$ARM64_OUTCOME" \
--repo repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base \
--repo ghcr.io/shadowdao/haproxy-manager-base \
"${tag_args[@]}"
+71 -27
View File
@@ -1,8 +1,8 @@
name: Mirror base images name: Mirror base images
run-name: weekly base-image mirror run-name: weekly base-image mirror
# Pulls each declared base image from upstream and re-pushes to the in-house # Copies each declared base image (the whole multi-arch INDEX, not one
# registry, so any of our images that FROM these don't depend on docker.io's # platform) from upstream to the in-house registry, so any of our images that FROM these don't depend on docker.io's
# Cloudflare R2 blob storage being reachable. The 2026-05-12 Cloudflare # Cloudflare R2 blob storage being reachable. The 2026-05-12 Cloudflare
# incident motivated this for python:3.12-slim and again for golang:1.25 # incident motivated this for python:3.12-slim and again for golang:1.25
# when the coraza-spoa build hit the same blob-fetch failure. # when the coraza-spoa build hit the same blob-fetch failure.
@@ -26,11 +26,58 @@ jobs:
# others from refreshing. # others from refreshing.
fail-fast: false fail-fast: false
matrix: matrix:
# hub: the Docker Hub IMAGE:TAG whose digest is resolved DIRECTLY at
# registry-1.docker.io (resolve-base-image.sh expects no docker.io/
# prefix; official images resolve as library/<image>).
image: image:
- { src: 'docker.io/library/python:3.12-slim', dst_path: 'cloud-hosting-platform/python', tag: '3.12-slim' } - { hub: 'python:3.12-slim', dst_path: 'cloud-hosting-platform/python', tag: '3.12-slim' }
- { src: 'docker.io/library/golang:1.25', dst_path: 'cloud-hosting-platform/golang', tag: '1.25' } - { hub: 'golang:1.25', dst_path: 'cloud-hosting-platform/golang', tag: '1.25' }
steps: steps:
# Only for .gitea/scripts/resolve-base-image.sh.
- name: Checkout
uses: actions/checkout@v4
# Supply chain: ask Docker Hub itself (over HTTPS, never via a mirror) which
# digest the tag points at right now, then pull exactly that digest. What
# lands in the in-house registry is therefore content-addressed to the
# digest Docker Hub reported - not whatever a `docker pull TAG` on the
# runner happened to resolve through any configured registry mirror.
# If the lookup fails the job FAILS and last week's mirror stays in place.
- name: Resolve ${{ matrix.image.hub }} digest (Docker Hub, direct)
id: base
env:
HUB_TAG: ${{ matrix.image.hub }}
run: |
set -euo pipefail
ref=$(.gitea/scripts/resolve-base-image.sh "$HUB_TAG")
[[ "$ref" =~ ^[a-z0-9][a-z0-9._/-]*@sha256:[0-9a-f]{64}$ ]] || { echo "::error::bad digest reference for ${HUB_TAG}"; exit 1; }
echo "base-image: ${HUB_TAG} -> ${ref} (digest from registry-1.docker.io)"
echo "ref=${ref}" >> "$GITHUB_OUTPUT"
# imagetools needs a NEW buildx. The runner's distro buildx is 0.15.1, which
# pushes the children of a copied index to the TAG path: Gitea answers the
# index PUT with 404 and leaves a stray single-manifest <tag>-staging
# version (run 5156). buildx >= 0.24 pushes children by digest and works
# (measured on repo.anhonesthost.net). Pin one, verified by sha256
# (linux-amd64 binary of the v0.28.0 release); mirror-index-copy.sh also
# refuses to run on a buildx older than 0.24.
- name: Install pinned buildx
env:
BUILDX_VERSION: v0.28.0
BUILDX_SHA256: 696bc104bac3bb708eff1af3f8bbc09fda0fd88f5757c1f9b404a35117889224
run: |
set -euo pipefail
dir="${RUNNER_TEMP:-/tmp}/pinned-buildx"; mkdir -p "$dir"
curl -fsSL --retry 3 --max-time 120 -o "$dir/docker-buildx" \
"https://github.com/docker/buildx/releases/download/${BUILDX_VERSION}/buildx-${BUILDX_VERSION}.linux-amd64"
echo "${BUILDX_SHA256} $dir/docker-buildx" | sha256sum -c -
chmod +x "$dir/docker-buildx"
"$dir/docker-buildx" version
echo "BUILDX=$dir/docker-buildx" >> "$GITHUB_ENV"
# The registry token is written ONLY by this trusted x86 runner step
# (unchanged from before; no new secret exposure).
- name: Login to in-house registry - name: Login to in-house registry
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
@@ -38,28 +85,25 @@ jobs:
username: ${{ secrets.CI_USER }} username: ${{ secrets.CI_USER }}
password: ${{ secrets.CI_TOKEN }} password: ${{ secrets.CI_TOKEN }}
- name: Pull, retag, push ${{ matrix.image.src }} # Registry-to-registry copy of the WHOLE upstream index (all platforms,
# incl. linux/arm64; $BUILDX = the pinned buildx from the step above). The old pull/tag/push re-published only the runner's
# amd64 manifest. mirror-index-copy.sh stages to <tag>-staging, asserts
# amd64+arm64 present and index digest == the Docker-Hub-reported digest,
# and only then moves the real tag, so a bad copy never replaces a good
# multi-arch mirror. Logic + tests: .gitea/scripts/mirror-index-copy.sh,
# .gitea/scripts/test-mirror-index-copy.sh.
- name: Copy index by digest, assert platforms + digest ${{ matrix.image.hub }}
# Values reach the script via env, never via ${{ }} inside run:.
env:
REF: ${{ steps.base.outputs.ref }}
DST_PATH: ${{ matrix.image.dst_path }}
DST_TAG: ${{ matrix.image.tag }}
run: | run: |
set -euo pipefail set -euo pipefail
SRC="${{ matrix.image.src }}" [[ "$REF" =~ ^[a-z0-9][a-z0-9._/-]*@sha256:[0-9a-f]{64}$ ]] || { echo "::error::bad digest reference '${REF}'"; exit 1; }
DST="repo.anhonesthost.net/${{ matrix.image.dst_path }}:${{ matrix.image.tag }}" [[ "$DST_PATH" =~ ^[a-z0-9][a-z0-9._/-]*$ && "$DST_TAG" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || { echo "::error::bad destination"; exit 1; }
name="${REF%@*}"; digest="${REF#*@}"
echo "::group::Pulling ${SRC}" [[ "$name" == */* ]] || name="library/${name}" # official images
docker pull "${SRC}" .gitea/scripts/mirror-index-copy.sh \
echo "::endgroup::" "docker.io/${name}@${digest}" \
"repo.anhonesthost.net/${DST_PATH}:${DST_TAG}"
# Capture the upstream digest so the workflow log shows what we
# actually pushed. Helps diagnose "did the mirror really update"
# questions later.
SRC_DIGEST=$(docker image inspect "${SRC}" -f '{{index .RepoDigests 0}}')
echo "upstream digest: ${SRC_DIGEST}"
docker tag "${SRC}" "${DST}"
echo "::group::Pushing ${DST}"
docker push "${DST}"
echo "::endgroup::"
# Sanity: the in-house tag should now resolve to the same content.
DST_DIGEST=$(docker image inspect "${DST}" -f '{{index .RepoDigests 0}}')
echo "mirror digest: ${DST_DIGEST}"
+15 -4
View File
@@ -10,6 +10,12 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line. - **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path. - **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path. - **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
- **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine.
- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
- **Multi-arch index assembly (coraza-spoa)**: `python3 scripts/test-assemble-multiarch-index.py` - offline (NOT run in the image build; it also reads `.gitea/workflows/build-push-coraza.yaml`); `scripts/assemble-multiarch-index.sh` turns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome is `success` (failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing. `scripts/e2e-coraza-multiarch.sh` rehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch").
- **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash".
- **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`.
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
### Reading stick tables (and why it is easy to get silently wrong) ### Reading stick tables (and why it is easy to get silently wrong)
@@ -25,8 +31,10 @@ columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
by NAME, and treat a missing field as an ERROR — never default it to `0`. The by NAME, and treat a missing field as an ERROR — never default it to `0`. The
`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`, `web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
`http_err_rate`; it holds **no history and no counter of past blocks**. What was `http_err_rate`; it holds **no history and no counter of past blocks**. What was
actually denied/tarpitted is in the edge access log on the **host** at actually denied/tarpitted is in the edge access log, since 2026.11.2 INSIDE the
`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table. container at `/var/log/haproxy/access.log` (in-container receiver, volume
`haproxy-logs`; README "Access Log"), not in any stick table. (2026.08.8-2026.11.1
sent it to host rsyslog on 172.18.0.1:514, which had no listener on the fleet.)
This is written down because `/api/security/stats` and `show-tarpit-ips.sh` This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
@@ -166,6 +174,9 @@ re-reads it on reload, so a failed runtime command must degrade to
- Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI) - Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
- Stats page on port 8404 (localhost only inside container) - Stats page on port 8404 (localhost only inside container)
- Management interface on port 8000 should be firewall-protected in production - Management interface on port 8000 should be firewall-protected in production
- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s` - Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/` - Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
- HAProxy is version 3.0.11 - HAProxy is the 3.2 LTS series from haproxy.debian.net (`trixie-backports-3.2`, native OpenSSL 3.5 QUIC); exact build in `/haproxy/HAPROXY_VERSION`. Debian trixie's 3.0.11 had the DNS-resolver watchdog abort (haproxy #3101)
- HAProxy runs as `haproxy -W -db` (foreground master, started by `haproxy_manager.spawn_haproxy_master()`), NOT daemonised: `daemon` re-points the master's stderr at /dev/null and loses ALERTs and the ha_panic/watchdog thread dump. Its stderr is appended to `/var/log/haproxy/haproxy-stderr.log` (haproxy-logs volume) and `scripts/haproxy-stderr-relay.py` re-emits it on the container stderr (`haproxy-stderr: ` prefix). The in-container supervisor probes with `pidof` every `HAPROXY_SUPERVISOR_INTERVAL` (default 2 s) and only runs python when HAProxy is gone; Crash evidence (dump blocks opened only by the anchored `Thread N is about to kill` header, `[ALERT] (PID) :`, `Current worker (N) exited with code N`; ANCHORED patterns, so keywords echoed from URLs/Host/server names never match) bypasses the bucket within a budget (1000 lines / 10 min, 60 s cooldown after a bound closes a block) and goes to STDERR (journald PRIORITY 3); routine reload noise is collapsed into one `[relay] reload/start ok: N routine lines collapsed` line per reload and everything benign goes to STDOUT (PRIORITY 6; docker's journald driver does not parse `<N>` prefixes), with a per-minute `[relay] suppressed N lines` summary. `python3 scripts/test-haproxy-stderr-relay.py` (offline) covers the relay
+89 -20
View File
@@ -23,27 +23,87 @@ LABEL org.opencontainers.image.title="haproxy-manager-base" \
org.opencontainers.image.version="${VERSION}" \ org.opencontainers.image.version="${VERSION}" \
org.opencontainers.image.licenses="MIT" org.opencontainers.image.licenses="MIT"
# haproxy is PINNED. It was previously unpinned, so the binary could move under # haproxy comes from the UPSTREAM-maintained apt repo https://haproxy.debian.net
# us at Debian's timing — an upstream release that rejected our config would have # (Vincent Bernat's builds, the Debian maintainer's own), suite
# broken an unrelated commit's build, or worse, shipped an edge that refuses to # `trixie-backports-<series>`, NOT from Debian trixie's archive.
# start (see the `haproxy -c` gate below for why that matters: a config HAProxy
# rejects leaves the container Up with 80/443 unbound and /health still 200).
# #
# Pinning does NOT make the gate redundant, and the gate does NOT make pinning # WHY (2026-11-16 forensics): Debian trixie ships 3.0.11 (+ security revisions).
# unnecessary — they compose. Pinned means the version moves deliberately; the # That build aborts via HAProxy's own watchdog inside the DNS resolver
# gate then answers immediately whether the new binary still accepts fleet config. # (resolv_process_responses / snr_resolution_cb, haproxy issue #3101) - whole
# It also makes the image reproducible, which it previously was not. # edge down 15-52 s per abort on sdbees/whp01. The fix ("dns: bind the
# nameserver sockets to the initiating thread", "resolvers: make
# process_resolvers() single-threaded") is in 3.0.13+, 3.1.11+ and 3.2.6+;
# Debian trixie will not take it. See docs/go-port/haproxy-edge-upgrade-and-logging.md
# in the WHP repo for the version evidence and the rollout plan.
# #
# To move it: bump the version here, rebuild, and let the gate verify. If Debian # Series: 3.2 (LTS, supported to 2030-Q2). Its trixie package is built with
# security-updates the package (e.g. -1+deb13u4) the build FAILS until this pin is # USE_QUIC=1 against the system OpenSSL 3.5.x NATIVE QUIC API (depends on
# updated — that failure is the point, not a bug. Check availability with: # libssl3t64 >= 3.5.1), so no USE_QUIC_OPENSSL_COMPAT layer. (3.0.x upstream
# apt-cache policy haproxy # has no trixie packages and would need a source build.)
ARG HAPROXY_VERSION=3.0.11-1+deb13u3 #
RUN apt update -y && apt dist-upgrade -y && apt install socat "haproxy=${HAPROXY_VERSION}" cron certbot curl jq net-tools -y && apt-mark hold haproxy && apt clean && rm -rf /var/lib/apt/lists/* # The repo keeps MANY patch versions of a series (27 as of 2026-11: 3.2.10 ..
# 3.2.26, all flagged `NotAutomatic: yes`), so "latest of the series" would let any
# rebuild silently take an untested 3.2.27+. The pin is therefore the EXACT Debian
# package version, HAPROXY_EXACT (default below), and the build:
# * pins apt to it (`Pin: version <full Debian version>`; NOTE a bare
# `3.2.26` does NOT match: apt compares the FULL version "3.2.26-1~bpo13+1",
# and with a non-matching pin apt falls back to Debian's own 3.0.11 package);
# * FAILS unless `dpkg-query` reports EXACTLY HAPROXY_EXACT, and `haproxy -v`
# reports the series (so an unreachable repo, which would leave Debian's
# 3.0.11, fails closed, and a 3.2 -> 3.3/3.4 jump can never arrive by accident);
# * records the resolved version in /haproxy/HAPROXY_VERSION; the `haproxy -c`
# gate below proves the binary accepts the fleet config.
# Taking a new patch is a DELIBERATE edit of the default below (or
# --build-arg HAPROXY_EXACT=3.2.27-1~bpo13+1), tested before it merges; CI never
# passes it, so a rebuild is reproducible.
# The repo signing key is COMMITTED (keys/haproxy-debian-net/, Vincent Bernat,
# fingerprint AEF2 3487 66F3 71C6 89A7 3600 95A4 2FE8 3535 25F9) and used with
# signed-by, never fetched at build time. The fingerprint is asserted below.
ARG HAPROXY_UPSTREAM=3.2
ARG HAPROXY_EXACT=3.2.26-1~bpo13+1
# OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40
# (CT130 pkg-cache, https only; CI sets it only after probing the cache with the
# pinned CA). Unset = the upstream Debian repositories, exactly as before. The
# apt sources are restored before this RUN ends (restore FAILS the build if any
# reference to the cache is left), so no shipped layer carries the cache address.
# NOTE: CT130 does not proxy haproxy.debian.net (see the design doc); that one
# repo is always fetched directly.
# Canonical script + docs: cloud-apache-container (scripts/pkg-cache-repos.sh,
# docs/pkg-cache.md there); this repo carries byte-identical copies.
ARG PKG_CACHE=
COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/
COPY ./keys/haproxy-debian-net/haproxy-archive-keyring.gpg /etc/apt/keyrings/haproxy-archive-keyring.gpg
RUN /usr/local/sbin/pkg-cache-repos.sh apt && \
apt update -y && apt dist-upgrade -y && \
apt install -y --no-install-recommends gnupg ca-certificates && \
keys="$(gpg --show-keys --with-colons /etc/apt/keyrings/haproxy-archive-keyring.gpg)" && \
[ "$(printf '%s\n' "${keys}" | grep -c '^pub:')" = 1 ] && \
[ "$(printf '%s\n' "${keys}" | awk -F: '/^fpr:/{print $10; exit}')" = 'AEF2348766F371C689A7360095A42FE8353525F9' ] && \
apt purge -y gnupg && \
echo "deb [signed-by=/etc/apt/keyrings/haproxy-archive-keyring.gpg] https://haproxy.debian.net trixie-backports-${HAPROXY_UPSTREAM} main" > /etc/apt/sources.list.d/haproxy-debian-net.list && \
apt update -y && \
pin="${HAPROXY_EXACT}" && \
printf 'Package: haproxy\nPin: version %s\nPin-Priority: 1001\n' "${pin}" > /etc/apt/preferences.d/haproxy-upstream-pin && \
apt install socat haproxy cron certbot curl jq net-tools -y && apt-mark hold haproxy && \
rm -f /etc/apt/preferences.d/haproxy-upstream-pin && \
deb_ver="$(dpkg-query -W -f='${Version}' haproxy)" && \
[ "${deb_ver}" = "${HAPROXY_EXACT}" ] || { echo "FATAL: installed haproxy '${deb_ver}' is not the pinned '${HAPROXY_EXACT}' (repo unreachable, or the pin did not match)" >&2; exit 1; } && \
case "${deb_ver}" in "${HAPROXY_UPSTREAM}".*) ;; *) echo "FATAL: installed haproxy ${deb_ver} is not series ${HAPROXY_UPSTREAM}" >&2; exit 1;; esac && \
haproxy -v | head -n1 | grep -F "version ${HAPROXY_UPSTREAM}." && \
haproxy -vv | grep -E 'USE_QUIC=1|\+QUIC ' >/dev/null && \
if haproxy -vv | grep -q 'USE_QUIC_OPENSSL_COMPAT=1'; then echo "FATAL: QUIC compat layer build, expected native OpenSSL QUIC" >&2; exit 1; fi && \
mkdir -p /haproxy && { haproxy -v | head -n1; echo "package ${deb_ver}"; } > /haproxy/HAPROXY_VERSION && \
echo "haproxy OK: series ${HAPROXY_UPSTREAM}, Debian package ${deb_ver} (exact pin)" && \
/usr/local/sbin/pkg-cache-repos.sh restore && \
apt clean && rm -rf /var/lib/apt/lists/*
WORKDIR /haproxy WORKDIR /haproxy
COPY ./templates /haproxy/templates COPY ./templates /haproxy/templates
COPY requirements.txt /haproxy/ COPY requirements.txt /haproxy/
COPY haproxy_manager.py /haproxy/ COPY haproxy_manager.py /haproxy/
COPY haproxy_gunicorn_worker.py /haproxy/
# Read by GET /api/capabilities (WHP gates whole-site re-bundling on it).
COPY VERSION /haproxy/VERSION
COPY scripts /haproxy/scripts COPY scripts /haproxy/scripts
COPY trusted_ips.list /etc/haproxy/trusted_ips.list COPY trusted_ips.list /etc/haproxy/trusted_ips.list
COPY trusted_ips.map /etc/haproxy/trusted_ips.map COPY trusted_ips.map /etc/haproxy/trusted_ips.map
@@ -79,8 +139,8 @@ RUN pip install -r requirements.txt
# * it cannot be skipped, and it protects local `docker build` too; # * it cannot be skipped, and it protects local `docker build` too;
# * no workflow restructuring (build-push-action builds and pushes in one # * no workflow restructuring (build-push-action builds and pushes in one
# step, so gating in CI would mean splitting build from push); # step, so gating in CI would mean splitting build from push);
# * it validates against the EXACT haproxy binary in this image. Line 26 # * it validates against the EXACT haproxy binary in this image. The
# installs haproxy unpinned, so that binary moves between builds - this # installed haproxy can move between builds (Debian revisions) - this
# turns "the new haproxy rejects our config" from a silent production # turns "the new haproxy rejects our config" from a silent production
# risk into a build failure. # risk into a build failure.
# #
@@ -89,12 +149,18 @@ RUN pip install -r requirements.txt
RUN python3 /haproxy/scripts/test-wpadmin-gate.py \ RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
&& python3 /haproxy/scripts/test-trusted-proxy-gate.py \ && python3 /haproxy/scripts/test-trusted-proxy-gate.py \
&& python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \ && python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \
&& python3 /haproxy/scripts/test-default-splash.py \
&& python3 /haproxy/scripts/test-access-log-receiver.py \
&& python3 /haproxy/scripts/test-haproxy-stderr-relay.py \
&& python3 /haproxy/scripts/test-haproxy-supervisor.py \
&& python3 /haproxy/scripts/test-config-rollback.py \ && python3 /haproxy/scripts/test-config-rollback.py \
&& python3 /haproxy/scripts/test-cert-write-safety.py \ && python3 /haproxy/scripts/test-cert-write-safety.py \
&& python3 /haproxy/scripts/test-cert-scripts.py \ && python3 /haproxy/scripts/test-cert-scripts.py \
&& python3 /haproxy/scripts/test-waf-body-inspection.py \
&& python3 /haproxy/scripts/test-gunicorn-socket-timeouts.py \
&& python3 /haproxy/scripts/validate-rendered-config.py && python3 /haproxy/scripts/validate-rendered-config.py
# Create log directories # Create log directories
RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log RUN mkdir -p /var/log/haproxy && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
RUN chmod 755 /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log RUN chmod 755 /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
# Set up cron for certificate renewal with proper permissions and environment # Set up cron for certificate renewal with proper permissions and environment
RUN mkdir -p /var/spool/cron/crontabs && \ RUN mkdir -p /var/spool/cron/crontabs && \
@@ -105,7 +171,10 @@ RUN mkdir -p /var/spool/cron/crontabs && \
# 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container # 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
# must still be run with `-p 443:443/udp` for the UDP listener to be reachable. # must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
EXPOSE 80 443 443/udp 8000 EXPOSE 80 443 443/udp 8000
# Add health check # Health check: management API (:8000, short timeout) AND HAProxy (:80).
# See scripts/healthcheck.sh. NOTE: WHP's lib-edge-container.sh passes its own
# --health-cmd, which REPLACES this one on deployed hosts; it must be pointed
# at /haproxy/scripts/healthcheck.sh for the fleet to get the :8000 probe.
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \ HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1 CMD /haproxy/scripts/healthcheck.sh
CMD ["/haproxy/scripts/start-up.sh"] CMD ["/haproxy/scripts/start-up.sh"]
+130
View File
@@ -379,9 +379,139 @@ tail -f /var/log/haproxy-manager-errors.log
| `HAPROXY_DEFAULT_PAGE_TITLE` | Title for the default page | Site Not Configured | | `HAPROXY_DEFAULT_PAGE_TITLE` | Title for the default page | Site Not Configured |
| `HAPROXY_DEFAULT_MAIN_MESSAGE` | Main message on the default page | This domain has not been configured yet. Please contact your system administrator to set up this website. | | `HAPROXY_DEFAULT_MAIN_MESSAGE` | Main message on the default page | This domain has not been configured yet. Please contact your system administrator to set up this website. |
| `HAPROXY_DEFAULT_SECONDARY_MESSAGE` | Secondary message on the default page | If you believe this is an error, please check the domain name and try again. | | `HAPROXY_DEFAULT_SECONDARY_MESSAGE` | Secondary message on the default page | If you believe this is an error, please check the domain name and try again. |
| `HAPROXY_DEFAULT_SPLASH` | `off` (or `0`/`false`/`no`) disables the default splash for unrouted hosts and restores the default-backend page below | on |
## Default Splash (unrouted hosts)
Since 2026.11.1 (404-everywhere + robots since 2026.11.2), any request whose Host is **not routed to a site** -- the
server's own hostname (registered cert-only, no backend), a bare IP, a stale
or unknown name, a request with no Host at all -- is answered by HAProxy
itself with the static page `errors/default-splash.html` (`http-request
return ... file`): `404` on EVERY path (including `/`) so search engines never
index the server's IP or hostname as a site, with `cache-control: no-store`,
`x-robots-tag: noindex, nofollow, noarchive` and `x-content-type-options:
nosniff`; `/robots.txt` is `200 text/plain` `User-agent: *` / `Disallow: /`
(`errors/robots-unrouted.txt`). Monitors pointed at the bare server hostname
on 80/443 must accept 404 (WHP's own probes -- healthcheck.sh, the watchdog --
accept any status). The page
names no version, echoes nothing from the request (it is loaded with `file`,
not `lf-file`), and links nowhere -- in particular not to the control panel.
**Why.** `use_backend default-backend` does not short-circuit HAProxy's
`http-request` rules, so the old catch-all was inspected by Coraza with the
full server-wide rule set like customer traffic. A browser on
`https://<server-hostname>/` also sends the control panel's cookies (cookies
are not port-scoped, so `:8443` cookies go to `:443`), and anything Coraza
matched there showed the WAF block page for a host with no site behind it.
The splash rules sit before the SPOE call, so unrouted requests never reach
the WAF; there is nothing behind them to protect.
**What still applies / is kept out:** rate limits, the IP block (blocked IPs
still get the blocked-ip page), the wp-* gates, and the multipart guard run
first; `/.well-known/acme-challenge/` still reaches certbot (the server
hostname's own certificate is issued through it); hosts in
`suspended_domains.list` still get the suspended page. Configured sites,
wildcards and their WAF inspection are unchanged. Plain HTTP on `:80` gets
the splash too, not a redirect: an unknown name usually has no valid
certificate, so a redirect would trade a clean page for a TLS warning.
Over TLS the presented certificate is whatever SNI selects -- the
hostname's own cert if one is issued, otherwise the default bundle.
**How "routed" is decided.** `generate_config()` writes
`/etc/haproxy/routed_hosts.list` (exact names) and
`/etc/haproxy/routed_wildcards.list` (`.base` suffixes) from the same domain
rows, in the same call, as the routing ACLs, and both are in the rollback set
next to `haproxy.cfg`. The splash ACLs compare the Host port-stripped and
lowercased, a superset of the routing ACLs (which compare the raw Host), so a
host that routes is never splashed. Never hand-edit the lists; they are
rewritten on every regeneration (also when the splash is off, so turning it
on never meets a missing `-f` file).
**Deploying to an existing host.** The page ships at
`/haproxy/errors/default-splash.html` and the template under
`/haproxy/templates/` -- both in the image, OUTSIDE the `/etc/haproxy` named
volume, so the volume cannot shadow them. The lists live in the volume but are
generated, not copied. So an image bump plus a recreate of the container is
the whole deployment: on start `generate_config()` renders the new listener
and writes both lists, then HAProxy loads them. Nothing to copy into the
volume, no WHP-side change. Verify after the recreate:
```bash
docker exec haproxy-manager grep -c default-splash.html /etc/haproxy/haproxy.cfg # 2
docker exec haproxy-manager cat /etc/haproxy/routed_hosts.list # every site
curl -skI https://<server-hostname>/ # 404 + x-robots-tag noindex
curl -sk https://<server-hostname>/robots.txt # User-agent: * / Disallow: /
curl -sk -o /dev/null -w '%{http_code}\n' https://<a-customer-site>/ # unchanged
```
Rollback: recreate on the previous image tag, or set
`HAPROXY_DEFAULT_SPLASH=off` and regenerate.
Covered by `scripts/test-default-splash.py` (template order/conditions, the
page, list generation) and the `splash-off` scenario of
`scripts/validate-rendered-config.py`.
## Access Log (in-container, since 2026.11.2)
HAProxy logs (`log /log.sock ... format rfc5424`, `hap_header.tpl`) to a UNIX
datagram socket inside its chroot (`/var/lib/haproxy/log.sock`).
`scripts/haproxy-log-receiver.py`, started and supervised by `start-up.sh` (a
restart loop: back within ~2 s), writes one line per request to
`/var/log/haproxy/access.log`, rotated at 40 MB with 4 gzip generations
(worst case 200 MB; `HAPROXY_ACCESS_LOG_MAX_BYTES`, `HAPROXY_ACCESS_LOG_BACKUPS`).
No network listener anywhere. Fields: client/real IP (`cip=`), Host, status,
termination state, request id, UA, SNI, HTTP version, WAF body stamp and the
Coraza `wrule=/wrules=/wscore=` fields (unchanged `log-format`).
* **Never blocks HAProxy.** Sends are non-blocking: a dead or stalled receiver,
or a full disk, only drops log lines (counted as `DroppedLogs` in
`echo "@1 show info" | socat stdio /tmp/haproxy-cli`). Lab: with the
receiver SIGSTOPped, 2500/2500 requests still answered 200 in ~1 s;
1025 lines queued, 1475 dropped and counted.
* **Not in `docker logs`.** The receiver prints no log lines (WHP's AI log
monitor reads `docker logs haproxy-manager`; a per-request firehose there
would storm it). Only its start line and rate-limited errors go to stderr.
* **Persistence.** Mount a named volume at `/var/log/haproxy` (WHP:
`-v haproxy-logs:/var/log/haproxy`) or the log resets on every recreate.
Give the container `--sysctl net.unix.max_dgram_qlen=1024` (a netns starts
at 10) for burst headroom.
* `HAPROXY_SYSLOG_TARGET=<ip>:<port>` still overrides the target for a remote
collector. The old default `172.18.0.1:514` (host rsyslog) is gone.
Reading it:
```bash
docker exec haproxy-manager tail -f /var/log/haproxy/access.log
docker exec haproxy-manager grep ' 403 ' /var/log/haproxy/access.log | tail
docker exec haproxy-manager sh -c 'zcat /var/log/haproxy/access.log.*.gz' | grep <request-id>
docker cp haproxy-manager:/var/log/haproxy/ ./edge-logs/ # or the volume's _data dir on the host
```
Rollout: image bump + recreate with the volume and sysctl (WHP's
`lib-edge-container.sh` adds both). The receiver, its script and the template
are in the image, outside the `/etc/haproxy` named volume, so nothing can be
shadowed; `haproxy.cfg` is regenerated on start with the new `log` line.
## HAProxy stderr (panic dumps, [ALERT]s) - since 2026.11.4
HAProxy is started as `haproxy -W -db` so the master keeps a real stderr (with
the `daemon` keyword HAProxy points it at /dev/null, which discarded the
`[ALERT]` lines, `Current worker (N) exited with code 134` and the whole
watchdog/`ha_panic` thread dump). It is appended to
`/var/log/haproxy/haproxy-stderr.log` (the `haproxy-logs` volume, so it
survives a recreate) and `scripts/haproxy-stderr-relay.py` re-emits every line
on the container's stderr prefixed `haproxy-stderr: ` (rate-limited, file
rotated at 20 MiB x 3). `docker logs haproxy-manager` and whatever docker
logging driver the host configured therefore carry the crash text; the
per-request access log stays out of it. The in-container supervisor probes
with `pidof` every `HAPROXY_SUPERVISOR_INTERVAL` seconds (default 2, was 15) and
backs off to `HAPROXY_SUPERVISOR_MAX_BACKOFF` (30) only when a restart attempt
fails. Measured (3.2.26, empty config): crash to listeners back = 1.2-2.0 s.
## Default Backend Configuration ## Default Backend Configuration
With the default splash on (above), unrouted hosts no longer reach this page; it still serves the blocked-IP and suspended pages, and serves unrouted hosts again when `HAPROXY_DEFAULT_SPLASH=off`.
When a domain is accessed that hasn't been configured in HAProxy, the system will serve a default page instead of showing an error. This default page: When a domain is accessed that hasn't been configured in HAProxy, the system will serve a default page instead of showing an error. This default page:
- Informs visitors that the site is not configured - Informs visitors that the site is not configured
+1 -1
View File
@@ -1 +1 @@
2026.08.11 2026.11.5
+40 -5
View File
@@ -16,25 +16,60 @@ ARG CORAZA_SPOA_VERSION=v0.7.1
# then for this image's golang:1.25), so both are mirrored at # then for this image's golang:1.25), so both are mirrored at
# repo.anhonesthost.net via the .gitea/workflows/mirror-base-image.yaml # repo.anhonesthost.net via the .gitea/workflows/mirror-base-image.yaml
# weekly job. # weekly job.
FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS build #
# Multi-arch (linux/amd64 + linux/arm64) WITHOUT QEMU: both Go stages are pinned
# to the BUILD host's platform (--platform=$BUILDPLATFORM, so the amd64-only
# golang mirror is enough) and the binary is CROSS-COMPILED for the target via
# GOOS/GOARCH. CGO is off (it already was), so cross-compiling is exact. Only
# the final distroless stage is per target platform (gcr.io publishes arm64).
FROM --platform=$BUILDPLATFORM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS build
ARG CORAZA_SPOA_VERSION ARG CORAZA_SPOA_VERSION
ARG TARGETOS
ARG TARGETARCH
# OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40
# (CT130 pkg-cache, https only; CI sets it only after probing the cache with the
# pinned CA). Unset = upstream Debian repositories and the default GOPROXY,
# exactly as before. Go modules stay verified against go.sum / sum.golang.org
# (GOSUMDB unchanged). Every RUN that rewrites ends with `restore`, which FAILS
# the build if any reference to the cache is left. Only build stages use it;
# nothing from here reaches the distroless runtime image except the binaries.
# scripts/ + keys/pkg-cache/ in this directory are byte-identical copies of the
# repo-root ones (this image's build context is ./coraza-spoa); canonical
# source: cloud-apache-container (scripts/pkg-cache-repos.sh, docs/pkg-cache.md).
ARG PKG_CACHE=
COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/
WORKDIR /src WORKDIR /src
RUN apt-get update \ RUN /usr/local/sbin/pkg-cache-repos.sh apt \
&& apt-get update \
&& apt-get install -y --no-install-recommends git \ && apt-get install -y --no-install-recommends git \
&& /usr/local/sbin/pkg-cache-repos.sh restore \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# The source itself is cloned from GitHub over HTTPS (not proxied).
RUN git clone --depth 1 --branch "${CORAZA_SPOA_VERSION}" \ RUN git clone --depth 1 --branch "${CORAZA_SPOA_VERSION}" \
https://github.com/corazawaf/coraza-spoa.git . \ https://github.com/corazawaf/coraza-spoa.git . \
&& eval "$(/usr/local/sbin/pkg-cache-repos.sh go-env)" \
&& go mod download \ && go mod download \
&& CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa . && CGO_ENABLED=0 GOOS="${TARGETOS:-linux}" GOARCH="${TARGETARCH:-amd64}" go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa . \
&& /usr/local/sbin/pkg-cache-repos.sh restore
# Catalog extractor: walks the bundled CRS at build time and emits # Catalog extractor: walks the bundled CRS at build time and emits
# rules-catalog.json so WHP's UI can render rule metadata without parsing # rules-catalog.json so WHP's UI can render rule metadata without parsing
# .conf files at runtime. Uses the SAME coraza-coreruleset version pin as # .conf files at runtime. Uses the SAME coraza-coreruleset version pin as
# the coraza-spoa binary above (drift between the two would mislabel rules). # the coraza-spoa binary above (drift between the two would mislabel rules).
FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog # Runs at build time, so it is built and RUN for the build platform (no
# GOOS/GOARCH); its JSON output is architecture-neutral and is copied into
# every target platform's image.
FROM --platform=$BUILDPLATFORM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog
# Same optional package cache as the build stage (module downloads only).
ARG PKG_CACHE=
COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/
WORKDIR /src WORKDIR /src
COPY catalog-extractor/ . COPY catalog-extractor/ .
RUN go build -trimpath -o /out/catalog-extractor . \ RUN eval "$(/usr/local/sbin/pkg-cache-repos.sh go-env)" \
&& go build -trimpath -o /out/catalog-extractor . \
&& /usr/local/sbin/pkg-cache-repos.sh restore \
&& /out/catalog-extractor > /out/rules-catalog.json && /out/catalog-extractor > /out/rules-catalog.json
# Distroless runtime: no shell, no package manager, no /tmp by default — # Distroless runtime: no shell, no package manager, no /tmp by default —
+30
View File
@@ -76,3 +76,33 @@ When a legitimate request triggers a blocked rule, the audit log shows the rule
2. **Drop from the enforce list**: remove the rule's ID range from the `ctl:ruleEngine=On` overrides; it falls back to detect-only. 2. **Drop from the enforce list**: remove the rule's ID range from the `ctl:ruleEngine=On` overrides; it falls back to detect-only.
After tuning, push the change — CI rebuilds, then `recreate coraza-spoa` on each host to apply. After tuning, push the change — CI rebuilds, then `recreate coraza-spoa` on each host to apply.
## Multi-arch (linux/amd64 + linux/arm64)
The image is built for both platforms **without QEMU**: the two Go stages are
`FROM --platform=$BUILDPLATFORM` (the in-house `golang:1.25` mirror is amd64-only,
which is enough) and the binary is cross-compiled with
`CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH`. The catalog-extractor runs at
build time on the build platform; its JSON output is architecture-neutral. Only the
final distroless stage is per target platform. The amd64 image content is
byte-identical to the pre-multi-arch build (measured: same rootfs tree, same
`/coraza-spoa` sha256).
CI (`.gitea/workflows/build-push-coraza.yaml`, one job on the trusted x86 runner):
amd64 and arm64 are each pushed **by digest, untagged, to both registries**
(Gitea + GHCR; the same digest lands in both), then
`scripts/assemble-multiarch-index.sh` creates `:latest` in each registry with
`docker buildx imagetools create` from `<repo>@<amd64>` and `<repo>@<arm64>`.
The arm64 step is `continue-on-error`; when it fails the tag is published
amd64-only, as before, and the arm64 digest is only ever taken from the current
run's step output (a previous build's arm64 is never re-attached).
Attestations: buildx provenance is kept per platform. A by-digest push with
attestations is a small per-platform index (image + `attestation-manifest`);
`imagetools create` over two of them yields one index with all four entries
(amd64, its attestation, arm64, its attestation), each attestation still pointing
at its own platform manifest; `imagetools inspect --format '{{json .Provenance}}'`
shows both. Nothing flattens or drops them.
Local rehearsal (no real registry is touched): `scripts/e2e-coraza-multiarch.sh`
(usage in its header).
+31
View File
@@ -26,6 +26,37 @@ applications:
# CRS-bundled defaults: recommended Coraza settings + CRS setup + # CRS-bundled defaults: recommended Coraza settings + CRS setup +
# the rule pack itself (~16 MB of rules embedded in the binary). # the rule pack itself (~16 MB of rules embedded in the binary).
Include @coraza.conf-recommended Include @coraza.conf-recommended
# Request-body limits, set explicitly instead of inheriting the
# recommended file's 12.5 MB / Reject. HAProxy only ever hands us one
# buffer of body (tune.bufsize 16384 in haproxy-manager's
# hap_header.tpl), and the SPOP library refuses frames over 65535
# bytes, so no body we receive can exceed 64 KB. The invariant is:
# tune.bufsize (16 KB) < SPOP frame cap (65535) < SecRequestBodyLimit
# Breaking it is NOT harmless: SecRequestBodyLimitAction Reject
# interrupts with 413 even under DetectionOnly (Coraza
# setAndReturnBodyLimitInterruption ignores the engine mode), which the
# edge turns into a 403 on every large upload. ProcessPartial is the
# belt to that brace. InMemoryLimit == BodyLimit keeps the body buffer
# off disk (this image is distroless). Pinned by
# haproxy-manager-base/scripts/test-waf-body-inspection.py.
SecRequestBodyLimit 131072
SecRequestBodyInMemoryLimit 131072
SecRequestBodyLimitAction ProcessPartial
# JSON nesting limit (the recommended file sets 1024). Coraza's JSON
# flattener is quadratic in nesting: a 601-byte `[[[...]]]` of depth 300
# cost 297 ms and depth 1000 (2 KB) 3.1 s of CPU -- past the 100 ms SPOE
# timeout, so the request passed UNINSPECTED (fail-open) while the WAF
# burned CPU. Over the limit the body fails to parse (REQBODY_ERROR,
# 0.1 ms), which WHP's enforce-mode body-integrity rule denies on a
# complete body. Legitimate depth, measured on the JSON shipped in
# WordPress 6.x core, Gutenberg, WooCommerce and Elementor (theme.json,
# styles/*.json, block.json, checkout schemas): max 8, so a global-styles
# save is ~9; 64 leaves ~7x headroom for Divi/Elementor-style module
# trees that could not be sampled. Depth 64 itself costs ~21 ms.
SecRequestBodyJsonDepthLimit 64
Include @crs-setup.conf.example Include @crs-setup.conf.example
# Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default. # Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.
+32
View File
@@ -0,0 +1,32 @@
Trust material for the OPT-IN internal package cache (scripts/pkg-cache-repos.sh,
docs/pkg-cache.md). Copied into images at /usr/local/share/pkg-cache/; nothing
here is added to the system trust store.
ca.crt "pkg-cache internal CA (CT130)", EC P-256, valid to 2036-10-18.
NAME-CONSTRAINED to IP 192.168.1.40 and DNS
pkg-cache.internal.cloud-hosting.io / pkg-cache.geekhome.local.
SHA-256 fingerprint BB:5B:9A:08:05:ED:2F:58:83:92:D0:99:9D:11:0A:30:14:88:68:F0:F0:88:65:5E:94:0B:75:EB:C2:A2:BD:F3
(private key lives only on CT130: /etc/pkgcache-ca/ca.key)
RPM-GPG-KEY-EPEL-10 Fedora (epel10) 7D8D 15CB FC4E 6268 8591 FB26 33D9 8517 E37E D158
fetched 2026-10-21 over HTTPS from dl.fedoraproject.org and
mirror.math.princeton.edu (identical); fingerprint matches
https://fedoraproject.org/security/
RPM-GPG-KEY-remi2024 Remi's RPM repository 2024 CF1D F005 7CE8 5DFF 5B2F 2A37 C2FD 3B2C 2A09 48E4
fetched 2026-10-21 over HTTPS from rpms.remirepo.net; byte-identical
to the key shipped in remi-release-10.2-1.el10.remi. Not found on
keys.openpgp.org / keyserver.ubuntu.com - confirm out of band.
Remi rotates keys (remi2025, ...): when remi-release starts being
signed by a new key, `release-install remi` fails loudly - add the
new key here (and to pkg-cache-repos.sh).
RPM-GPG-KEY-EPEL-9 Fedora (epel9) FF8A D134 4597 106E CE81 3B91 8A38 72BF 3228 467C
fetched 2026-10-22 over HTTPS from dl.fedoraproject.org and
mirror.math.princeton.edu (identical); used by `release-install epel`
on EL9 images (cloud-memcached-container, cloud-haproxy-container).
RPM-GPG-KEY-nodesource Nodesource Operations <operations@nodesource.com>
242B 8138 31AF 0956 2B6C 46F7 6B88 DA4E 3AF2 8A14 (rsa4096, 2023-12-22)
fetched 2026-10-22 over HTTPS from
https://rpm.nodesource.com/gpgkey/ns-operations-public.key (the key
NodeSource's own setup_N.x scripts reference); verified by dnf
gpgcheck against the nodejs RPMs. Not on keys.openpgp.org /
keyserver.ubuntu.com - confirm out of band. Used by
`pkg-cache-repos.sh nodesource MAJOR` (cloud-node-container).
@@ -0,0 +1,29 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGV4X6kBEAC3eQxgiWuo08uc3mHo4ELux++uqTnYz/tJzEf9Ou3h36WnhumA
Nvs+Ts5h8PBx879Y9/aIX1Z20p1kf6tBCinZnEJu59n+TAAsph0+XQlV1l5YkleK
Z2ff/Fg65k8QcLXWaIGykA/FaKznRiSurGuD6tRGhJw7DawEwBJr8QZSkRUpnH1L
URW97Q/iKrRPiE5VEayE0y8eAL28jIIiFvR+4oJMzvCsRRB/2wYZ2MlJOW91hcYf
mbUoXKOBD5UzsJylu7kj25K/ge8rEJ7KicOOwcdYddxsU3DxGSSfwF8AMagENcm2
XROeXknjm84A8sNlUkFZBJwfuc7eRTiZGJrnQQVYLrkKj8Mxpq9Ts7hU51TqAWNI
uvGDlJdYNE3D2RMqjMEsZ8ej08Thrib6xslu4NzTBkt+6QNnXL4E3hEgYtoyio60
GswSz2ulogKg7X4JrNdJYE8/qNowyF3hoVgj5TG1/wQRq+5HlMMOLjgGu9wzLUix
fnVfEUnzaofbrUf4/GabCaeY8xRe4tFQrvzigQ4g+kgwKKnfAeqBmPov0yljkw9z
BYJWR5zvaw0ffg9Ing00KUSaXBXA5jSlgk1603Y+LefY1SlXsTyqohiRvGH6FI77
HNMo72DwoJfFcYjncZUzKgXWJECR4nhVsdj6pKoOjcQ4aSuyVxtsR86ASQARAQAB
tChGZWRvcmEgKGVwZWwxMCkgPGVwZWxAZmVkb3JhcHJvamVjdC5vcmc+iQJOBBMB
CAA4FiEEfY0Vy/xOYmiFkfsmM9mFF+N+0VgFAmV4X6kCGw8FCwkIBwIGFQoJCAsC
BBYCAwECHgECF4AACgkQM9mFF+N+0Vhv/A/+PlhPLSctGRCUEahE+cN4764Acc3p
l40ZYzXRhqR0/Tc1/cSDjlA3qVTc8SPohi5OJXwCyr9EiMqKoyoDN097euqbYpyp
yN/Pj0lBjsXwcpdDtZ21WGeQU0Khb04N68bMtJbDaxeBciTvDDQravZuPPh0m4Rg
Z6myEoa6Aa6EK0hI1Qwi1qIWeRiuEkVT671IaKVETBW5XiUpNBXDAB/L+6DzUF9u
scBzfsUDiPO6NrpYDtV3jwq22y6gWluIct/Ka8brwPbqK2sBfFzrHboRhfqlTGjs
7F9qUGwIQZn/A8iozXZYQ0+JG1bhQyvjA8eN1GOcRpT+O7H7JXN49o6IG2As4+iK
F04+qjqAu2sVfpD8mzM2VubFNllcKKiyCzRYHhSbObRCPzsudDL9GPiXeGGaCuWg
sDkiA1MESvf2tLETAGBs/TziO4GwmXUtlKbRiq1FYm90mVq9mBxPZ/Idn+yZusNB
0O5SXIbI8lYZw5n4XTK4b+byHRBYsOTHiTsGvjTF2Y7oSwW2CVUmL6RZ23mI4qoY
1p5kzRS+GjT1acnTei/FTsOlIKCsjfeHx7uxCkX6xpAD8P3UtLQqfsgH0CL4vSZt
TGO6L1InQlp4ZG3OYIomTKbD3/R0wod3U3dTqdulQMXL895u6OLTY3spY2m2MO2k
p9Dfd2pKuxK9Mys=
=mhQZ
-----END PGP PUBLIC KEY BLOCK-----
@@ -0,0 +1,29 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGE3mOsBEACsU+XwJWDJVkItBaugXhXIIkb9oe+7aadELuVo0kBmc3HXt/Yp
CJW9hHEiGZ6z2jwgPqyJjZhCvcAWvgzKcvqE+9i0NItV1rzfxrBe2BtUtZmVcuE6
2b+SPfxQ2Hr8llaawRjt8BCFX/ZzM4/1Qk+EzlfTcEcpkMf6wdO7kD6ulBk/tbsW
DHX2lNcxszTf+XP9HXHWJlA2xBfP+Dk4gl4DnO2Y1xR0OSywE/QtvEbN5cY94ieu
n7CBy29AleMhmbnx9pw3NyxcFIAsEZHJoU4ZW9ulAJ/ogttSyAWeacW7eJGW31/Z
39cS+I4KXJgeGRI20RmpqfH0tuT+X5Da59YpjYxkbhSK3HYBVnNPhoJFUc2j5iKy
XLgkapu1xRnEJhw05kr4LCbud0NTvfecqSqa+59kuVc+zWmfTnGTYc0PXZ6Oa3rK
44UOmE6eAT5zd/ToleDO0VesN+EO7CXfRsm7HWGpABF5wNK3vIEF2uRr2VJMvgqS
9eNwhJyOzoca4xFSwCkc6dACGGkV+CqhufdFBhmcAsUotSxe3zmrBjqA0B/nxIvH
DVgOAMnVCe+Lmv8T0mFgqZSJdIUdKjnOLu/GRFhjDKIak4jeMBMTYpVnU+HhMHLq
uDiZkNEvEEGhBQmZuI8J55F/a6UURnxUwT3piyi3Pmr2IFD7ahBxPzOBCQARAQAB
tCdGZWRvcmEgKGVwZWw5KSA8ZXBlbEBmZWRvcmFwcm9qZWN0Lm9yZz6JAk4EEwEI
ADgWIQT/itE0RZcQbs6BO5GKOHK/MihGfAUCYTeY6wIbDwULCQgHAgYVCgkICwIE
FgIDAQIeAQIXgAAKCRCKOHK/MihGfFX/EACBPWv20+ttYu1A5WvtHJPzwbj0U4yF
3zTQpBglQ2UfkRpYdipTlT3Ih6j5h2VmgRPtINCc/ZE28adrWpBoeFIS2YAKOCLC
nZYtHl2nCoLq1U7FSttUGsZ/t8uGCBgnugTfnIYcmlP1jKKA6RJAclK89evDQX5n
R9ZD+Cq3CBMlttvSTCht0qQVlwycedH8iWyYgP/mF0W35BIn7NuuZwWhgR00n/VG
4nbKPOzTWbsP45awcmivdrS74P6mL84WfkghipdmcoyVb1B8ZP4Y/Ke0RXOnLhNe
CfrXXvuW+Pvg2RTfwRDtehGQPAgXbmLmz2ZkV69RGIr54HJv84NDbqZovRTMr7gL
9k3ciCzXCiYQgM8yAyGHV0KEhFSQ1HV7gMnt9UmxbxBE2pGU7vu3CwjYga5DpwU7
w5wu1TmM5KgZtZvuWOTDnqDLf0cKoIbW8FeeCOn24elcj32bnQDuF9DPey1mqcvT
/yEo/Ushyz6CVYxN8DGgcy2M9JOsnmjDx02h6qgWGWDuKgb9jZrvRedpAQCeemEd
fhEs6ihqVxRFl16HxC4EVijybhAL76SsM2nbtIqW1apBQJQpXWtQwwdvgTVpdEtE
r4ArVJYX5LrswnWEQMOelugUG6S3ZjMfcyOa/O0364iY73vyVgaYK+2XtT2usMux
VL469Kj5m13T6w==
=Mjs/
-----END PGP PUBLIC KEY BLOCK-----
@@ -0,0 +1,52 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGWFsqMBEAC4YlSuNpHjOiQVTC8StolVx3J8E9SC6G4wmVWbytMD1Zivfanj
3LKDqdy7vpbViXA9whpyGQgQYdOlPKaNHWw4GSeGTnP1FbvaK1NJ9MnAzUysh49R
cqoktYeIQbqgOgZHw5/gVH5bYObf1KJQ2LY/je9WTlW3QMRe1TaW0F0AZnJMBIOG
JllE33QIpxDnOQHnvdUGq+HscUOTrW4Q3AdzwsQxU4GX82d9qFFr8HjAmUjlUKyh
PgvMpb8VDvR7NE6et+wWh6BMYILo3VJFP7/gTXGiy3WWs5GW/PNXIXw5/f2w1B+e
Dh5tlfGG4q/4P8mbv7C6RPHoF2Lq+W69HLqSq/l42WUL41tHlFTsluCIKpeeMxAu
olxx3Z5momHMMyh7rbZtQDqFMTAqxNjT1NinKkWyj7KP5WE9hVosyd3UMR4UYDMm
Zb6LI4N7Hht887+CY7grxTJ9cGQhMFfvLyvlf/FWQY4S6G7sEaUV71N9StvXBNG/
iNoIMzhwjBmrmvT/UGPkLSi0SIBOxeGw2rwngnYikIJxuTQ4V40lm8o6eMFca7rW
BQvAEncV8pvi/3FqrUBZ/U53s7NO24hBDoaQPzFbVCw0GHHf9GD7KoQb5c9ONoCZ
Qu2YWo7ajPLzl/h2pJ2Kn29Ywev9MpuuPTGvifiKilcBVLYP3ZJg4ZfRBwARAQAB
tDFOb2Rlc291cmNlIE9wZXJhdGlvbnMgPG9wZXJhdGlvbnNAbm9kZXNvdXJjZS5j
b20+iQJOBBMBCgA4FiEEJCuBODGvCVYrbEb3a4jaTjryihQFAmWFsqMCGwMFCwkI
BwIGFQoJCAsCBBYCAwECHgECF4AACgkQa4jaTjryihTD+hAAtTU155ke7xmnT0ph
odbGjm3mOGdkdaDH+xmWtPtufiPncgHMCYh4aIDRloog8rFE+pP3qfwJmUKxthwD
drrLAzfUwOrGxPnYH3MWwg66RVVoiiU+Hh42gJTez+P/K0VMQZXH0XTZD96TiwT/
S0+WvQPuJ9pda2L4cjV+YP2+m0ERUG9QPj0Shgw+/PWCGWKD6J9DRToiwfwCTOvc
DNolF1EcRlfPMTsqI2Bdc+TQywqU1Py1MUDSAPXxdK6nW2oM7eC5l3XXefcQ6JOS
AuNdEDIsaf+H76j6/wSGsuIcJjUo9wmKjEV1iJYwT6jLbjaVQxXgEPHBSb+kzanx
FWyrYP6TZpeY2gT4bBMsOKd7ay1I3/nIHwe6x8Yzkqzz+7YVhNpMbybH72NTxDWQ
UmiboJs9r7j1lkHlBjxU9RLF5laYSeCIXD62KQVoP5ao1LSPKsCXFqcdpUgLYtRv
CXxdriiHp4m9eGzbS7RW1TLm6tMOKbI4LzOtMm4DoZzGD2FTfsUtn6ALx9FSgvl/
VMlzqZes+rclcukACtutvtB8e0yRhuzPUN56ctLEZfI1XzQFoN8Vdj6HWtiXWRiJ
EohuufE2cEebmX90NRsbh4KAFXNEQROlMeMf7ufuSsk7QXLTutxmWSQvWwyS+P5/
ZqCapIDlRCJFZX4N9a1IXo0OgCC5Ag0EZYWyowEQAKz7sA6WBKFVcmzcyr6y9TI5
RemnzotRUdqXj8tD70xvV1nBEf+BfTw3hsiV019ZboPBae+cRaD4plQbNb0du/+Q
UFq4ja055ZEld01RqE49L562pWvA5cwSmjs9+ZMI1ja0HbTb2K4razpzX3be+Zul
TbH20X5FHTz+Jyfmw+G3Y9okw3OIzuYo16Lv/Qs8VoBmj01PItzq8dX5wJiIlVxU
8wKHKy1kiKGCgyPnvqcXCGen0EoUS9wsWK1QHfryscYGVF9y1TBQvJa9HIpWPrIZ
slSiVS4tQhGHfLDVlaDsYFrH9SIUkvW7peSEiPE1TKQjvbMS3CcUZRaMcGfgyDPe
gXwnNwZbJ2/cisXbdAJaGT49V1Y+Sc1oKSGSG4FaGUNhS3p3cB7qlX2zS/sR2ael
90fXR6nhjvhMAS98i7663pCUv9q0Z/NXNZwdXjbSF4W8gq3PVt7hCnJrngsGSkGu
Qz0dc227/sAjh9CxYm8kQqUuQtiPVfKwkcUUT+O7jr2/6eAXZbTYVZbvaLEGMrKA
bdWKpZMMOKAI45f+ZAcEmJG4D68N0jzVrLCXMuuLq/yqqyEmxDjSOH0qVutl8jxg
vfxPxKYX8nnAzJZJHGHmV/ihjfClSdYePKExHuFVVhvc4BxE1XfPUeCZG4JpB9/B
gNCiDPsg4JIw6GO5LS2xABEBAAGJAjYEGAEKACAWIQQkK4E4Ma8JVitsRvdriNpO
OvKKFAUCZYWyowIbDAAKCRBriNpOOvKKFOTKD/40WN6ZRfP6hVLAX26DdyN9Bqzg
D+0hJilmHL1YFeLt9U/YPhOZJxPaDGrMfldC0FSXKKr7H5C7FQErl936108wGohz
53KGbNid0crNeaWrQuWk3QjaMF62Yc+NVB8b5LoEk0TEt9KPO6kveYDtW6Ygkw9i
5aaS/AC+8GChOH6lWhLHcO91T7+Hi5qRez7qyme/AlHUTV9/jE6JgFJ7thCCAqBQ
AfX3mGFlTwSTMHDeYq81MMl9UhYgoFJtx8PuqmSV7Eb29ccJofz3J+b5OJvpmLSZ
Hu22Csa8k2V/1ojeL0DjQHD06V75F7BerxLR/rbJ64GkIiOdm4ntXC4a335gjQfM
5tI6LtGsIpr+Le4C068eDj7YpzS0ql3g6o4A1TbtapLccgEj4iw95H4KKh0Ro5r0
0s6LoSLu9uva3LiLVn6xXvpw3iv8FnhTr5PEiPnorXvxcCnUrB3Rkk6Wh1SKjpxa
u6KiFzH0CXR1AgUdWSC0rAmvEdFjXgaoQd+Z2eJFHEAsg91URd9NtdeDsyMwdbHh
iBZPGX5ddPjJA7nwKbkibCv7xiodd9ph8NmGLqD/2SDU1rqvPuCDMqLuxc89Gvhs
3CeLTSJK9Uzb3oNdZLxL8CECip1ZkowVfTL+jY23phS9+Z0Db/yZqot3ZZJAAG8g
oSVkzT/Mwp9zFVQu5w==
=drMP
-----END PGP PUBLIC KEY BLOCK-----
@@ -0,0 +1,52 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGWdEG0BEADV9c8aSIITQmoWKN0ac4c8AEhZcdzpWSRPwHIS8syVRw/gj30W
Mn1OoxqFn2UECh2jjqYaQFECXNrM2/lzO4GhvcuoPn6p0rXFMvYvwtiOJ+rmDvTe
UOwRxkZhYehpvsRPFoVhlYJh3U1BM0UHm2iE/TIqVusY9FHQuCMZVl0BHk81YGDa
qE3cyu+9zOs8z1bYlsYBgLIbtb4+loQTRexFTfZkUcCWgAaNyur8Wy7Msdm5LOqN
ZZu6bNKBOv0Cr+BOwGNoFRbi4IM+5H+0eI1n7V5if4CuXfdYHp+Pm+PQvftm/nqB
jRLKqioRKzHjk7xNMsi5lM+SU4WyfU77pr3RXuJR6fkymLVAeu5sy2U6ySj0SXIO
/I3oPp4/zJ8/e94jG5iDqh92b5IWyXSE+EGw4kkpUvZLrK8vyyEfk8h9Gf8qA5uL
yGfw0vnSVjYb+RdnfZFVdPaIEmeY8tnobS87duj8JESZiSWo5sU4filkciW5e+MJ
w9wohgl+Cqm2J5Ljqus6DRm4ptKbZD8QZRnmGcSk6O2VopBKt2XLat4eIJLr/EiN
0CJ1EH/oJmEeZbYtUjxjsGsbPXicwDH6ZIeIMSaiDgqRxPm1LTGVLjSHAk+k0e5l
ulEmaoA7t/p3U2peGc7ONvozXiyNHZdGiRkqFC1mA9/JpeWw4KFH6JSgmQARAQAB
tEtSZW1pJ3MgUlBNIHJlcG9zaXRvcnkgMjAyNCAoaHR0cHM6Ly9ycG1zLnJlbWly
ZXBvLm5ldC8pIDxyZW1pQHJlbWlyZXBvLm5ldD6JAlEEEwEIADsWIQTPHfAFfOhd
/1svKjfC/TssKglI5AUCZZ0QbQIbAwULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIX
gAAKCRDC/TssKglI5Na7EACDTGjTcuTyudfuu/JC41BSurclCcwZyPsGaClyc0qx
Tk8Rs4rbyVw1u494fiFG/yEVl7LGGja84UJOlMdw82KgVUmr4fGLaBBFWm+KnZ4W
4vvFcCIxUshENOaDrkhby9F5P379gVOK9I3n9l4Q2xt+sWUNwE74CNfCTUeYMIm/
/AHbb1DTM9FuIT6FssgvILMTlksiyrw8U8bj+QUzPZrFCpvv+PnFA7ZDMEyCf2Tz
/zawxoP3a3b+3/g61U9SVIkfToonnorqh9lkQYhsdKTFHW6eESqe7zcqMBZv/sSc
3kFzl22KczSdU8eYxdujv9BmlW2SOrQsnuMYxvoD1LzhwSB/nOapNrWU+djhJCof
ueZ+rBW4QWysmG0dEp5Wpz85TstAnQ+j5aCOMSAkiAVsrJ0Ff3JXakIiL7rEtccS
dhgBT06LJtJy1NGIC0Sotn45n6dtfi6IuI8LnELCKeHxhdFSJcAPB8y95u0uHymy
sqMfzjq1VcYtzC5II6Yd5Gtb2+QzyQwORslwXhUdXwZKUc5i6X5xFMSxU523fH9n
i+kQlUTPkOUr66v4fmYDj+ZlJQSFoPylcHCke/vOo/rH5nj+YMPa6LdohsdzHKpd
DkKrk+Udj5aDXtXGIFYwoNKO4BnCsz/ImZfB4CQ+ekJsY1pFill3+MRTM6IvSA+P
7LkCDQRlnRBtARAAq8k4FM9euMMsXz0buxxyr0ew0ecWNhZGy24a6DXORVTEJbgI
T3HpV/C2PWkwgEBswYzREm+en61jhWr5unCiytO2klT+K2WNcIUNW+z2ciL/JNaJ
OcKwjkiY/A+KtDStzllCQ/XnpgAXGvFZjMjhHrNGQoAwNnPE/cMtYwcuVdW3Ipm3
cOOtzxRmfnhP8YUEyHxdos/7AZowaiLIUkaL5RUaygGrfPGyUXPl1aXS1oiOm37W
RiKCGUxCHSlpBqeh1WQPOaps8y+ME9jMzF9SLvqiXlKPllJIx9i8tHUOnX6Y2U2x
5sXsCEiA+zNVMU3YQ04++Yx2Rp6sDhNEnAbTCqqequqIzaXHZWwszAeG6pj1G90m
8PwDeHy641ySgDMwVPpnH5rHWmmT96XvF9QNg07OBtW9/phryzwAuHogrBY4DyFD
pUxDkTv9TxfBaS/s1uJxEGknIgnKzQo79mRy206waGKRA2AV36ORSfU12sK8pl3r
N95iIDh/OIuMl2STaXilhwb1hRpGFvwRlq3Hvvan4SY1aO219Q+VxMGXeqC5F/rm
Anh7mRVMP2qu0rj7L8xJ/OD/bPfuWL72/PvviBr/jdRM9I+Vj8dPGjhUwhK1dpEG
Q6yvoVXLHXucVHfIVR+ALLBmK8oHXNIFeIXhiUbgmRAilPHRz6Rf5pgbrC0AEQEA
AYkCNgQYAQgAIBYhBM8d8AV86F3/Wy8qN8L9OywqCUjkBQJlnRBtAhsMAAoJEML9
OywqCUjkQ2kP/iwqvt1sX85KM4OSE9ftqemwnvh19NizMKHMYGgikMAvy2CndEOI
wzKVeE1iK9/ivZQQIbD5l0G9hzVZjIxbgu9dAcxM+jdmaPyUj7yn6AX2yeGAeHyA
qzNKZaTLJPieRzdXk/uKPVyWUuzbR0vwR7n8N5N0pe/FEfO9O0uX5iz2kf5+b10c
WqvzLmO+czuyCjTHnjhKfE73iIktYE3MnM8qFHPiKwm+jeTkUnmInoR3qIKR/SCV
u2vk/q5uvpNaUSPSDKji6wh3/hbNbPocLSGIjugfWNm/4LhlSHPNnRJ2dIvSxeiI
qtSWxhULEyHGBPMLDriVumCV7Rb/byJwiMbmCIpxB5XLtbgt7nIqWQTko+6ci71S
4rDXolM3bytYbpqq4b/Xc1bhBuTX3omwrd0kxmGqad8eia6O6iBsP7GAl4QkySm4
MfKQfohjXTnT60wqEzQwLGI4NKN9bHRSsGINHX5p+Kz9HLf87fTY4uglbFxg0/II
Y3nko5KBw/gfe0WpwZY0sWjK20cXJn3xLQjoG8OjUNUCUYTRsiAvn4AiDRt++qtl
bvJjJqOU1k4GrGoCVpEUiOc9Cw1YZREcFe8H2at3pGnxJ1DFlYA916kNAxYANp2s
r3N3agwevgIzP78hynxjm6a5H2w0vkg189pZn2RR9zX9gmrvlqg1o0P1
=meaU
-----END PGP PUBLIC KEY BLOCK-----
+15
View File
@@ -0,0 +1,15 @@
-----BEGIN CERTIFICATE-----
MIICPzCCAeagAwIBAgIUJI0QJZ+PCrSQ/nKZFENwwcp8Zr0wCgYIKoZIzj0EAwIw
TDEmMCQGA1UEAwwdcGtnLWNhY2hlIGludGVybmFsIENBIChDVDEzMCkxIjAgBgNV
BAoMGWNsb3VkLWhvc3RpbmcuaW8gaW50ZXJuYWwwHhcNMjYxMDIxMTkzNjIyWhcN
MzYxMDE4MTkzNjIyWjBMMSYwJAYDVQQDDB1wa2ctY2FjaGUgaW50ZXJuYWwgQ0Eg
KENUMTMwKTEiMCAGA1UECgwZY2xvdWQtaG9zdGluZy5pbyBpbnRlcm5hbDBZMBMG
ByqGSM49AgEGCCqGSM49AwEHA0IABBJh75sC5vSzTnkYl0CYYEA/TKUHEXhbtanV
OVNW4NlD3za5+bu5YuKjpbFAva4TCZqWsaS9IMd/JfsaFbf5XVejgaUwgaIwEgYD
VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFP7UCTf/
v7KU1KBrs1vqa71/TZl0MF0GA1UdHgEB/wRTMFGgTzAKhwjAqAEo/////zAlgiNw
a2ctY2FjaGUuaW50ZXJuYWwuY2xvdWQtaG9zdGluZy5pbzAaghhwa2ctY2FjaGUu
Z2Vla2hvbWUubG9jYWwwCgYIKoZIzj0EAwIDRwAwRAIgOod4wPHu0kAzYAw8KeGg
cIrFMwQ4ATKVTkDioiSFOqICIEhY+sGj/YZ4oBMst3b/AAz9zMEJAXpnCkSTI6tM
YAIl
-----END CERTIFICATE-----
+125 -2
View File
@@ -47,7 +47,17 @@ SecRuleUpdateActionById 933170-933200 "ctl:ruleEngine=On"
# vulnerable, but blocking at the edge keeps logs clean and protects # vulnerable, but blocking at the edge keeps logs clean and protects
# any future Java workloads. # any future Java workloads.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On" # 944110 and 944120 are NOT promoted (changed 2026-10-09, 2026.10.2). Both are
# CHAINS whose first link matches bare words anywhere in the request --
# `runtime|processbuilder` and `getproperty|forclosure|filewriter|...` -- and
# Coraza runs an appended ctl:ruleEngine=On on that first link even when the
# chain then fails. Measured (Coraza v3.7.0 + CRS 4.25.0): in detect_only,
# `?q=runtime&id=1' or '1'='1` was 403 while the SQLi alone was 200. Same
# chain-starter defect as 920440. They still add to the anomaly score.
# Log4Shell (944150/944151/944152, ...) stays promoted. WHP mirrors this split
# in coraza_baked_promotion_ranges() (web-files/libs/coraza_blocking_set.php).
SecRuleUpdateActionById 944100 "ctl:ruleEngine=On"
SecRuleUpdateActionById 944130-944300 "ctl:ruleEngine=On"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# 920440 — URL file extension restricted by policy # 920440 — URL file extension restricted by policy
@@ -59,7 +69,26 @@ SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
# single GCP-hosted scanner. Zero false positives observed; standard # single GCP-hosted scanner. Zero false positives observed; standard
# WP/WooCommerce/Divi/HPR URLs do not end in these extensions. # WP/WooCommerce/Divi/HPR URLs do not end in these extensions.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
SecRuleUpdateActionById 920440 "ctl:ruleEngine=On" # HOW IT IS PROMOTED (changed 2026-10-09, haproxy-manager-base 2026.10.2).
# This used to be `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"`, which
# was WRONG: 920440 is a CHAIN whose first link is
# `REQUEST_BASENAME "@rx \.([^.]+)$"` -- it matches EVERY URL with a file
# extension -- and Coraza runs actions appended to a chain starter when that
# first link matches, even though the chain then fails. Measured (Coraza
# v3.7.0 + CRS 4.25.0, in-process and live): every *.php / *.css / ...
# request ran with the engine On, so the whole CRS anomaly score (949110) and
# 200002 enforced there -- in detect_only mode too, since SecRuleEngine
# DetectionOnly cannot override a runtime ctl.
#
# Now: 920440 itself is untouched (it still sets tx.extension in its first
# link and adds the anomaly score in its second), and this separate phase-1
# rule, which runs after it, flips the engine only when the extension really
# is in tx.restricted_extensions -- the original intent. Removing 920440
# (break-glass, a per-host exception) also removes tx.extension, so this rule
# goes quiet with it. Id 992920440 is outside WHP's generated range
# (990000000-991999999).
SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \
"id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# 930130 — Restricted File Access Attempt # 930130 — Restricted File Access Attempt
@@ -76,6 +105,100 @@ SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
SecRuleUpdateActionById 930130 "ctl:ruleEngine=On" SecRuleUpdateActionById 930130 "ctl:ruleEngine=On"
# ---------------------------------------------------------------------------
# 209610-209616 -- PHP multipart "RCE via OOM corruption" mitigations
# CVE-2026-109618 (GHSA-w8hw-w9fx-j79w) and CVE-2026-109619
# (GHSA-6wmf-4rv5-p3rp). Interim edge mitigation until patched PHP is on every
# customer image; remove once it is. haproxy-manager-base 2026.10.6.
#
# The advisory's request-body vectors and what covers each:
# (a) multipart with no usable Content-Length (chunked / H2-H3 without
# content-length) streams past post_max_size -> denied 400 by HAProxy
# itself (templates/hap_listener.tpl), before any of this runs.
# (b) unlimited headers per multipart part -> 209611-209613 (budget)
# (c) empty file controls (do not count toward max_file_uploads)
# + deeply nested field names (a[][][]...) -> 209614-209615, 209610
# -619: session upload progress during multipart -> 209616
#
# WHY THESE IDS. 200000-209999 is the Coraza engine range, and WHP's site
# break-glass rule (991000001) removes that WHOLE range for a site in
# emergency mode (coraza_blocking_set_ranges(), WHP_WAF_ENGINE_RANGE_LO..HI in
# web-files/libs/coraza_blocking_set.php). Rules that deny on their own MUST be
# inside the derived blocking set or break-glass leaves them armed; inside the
# engine range they are covered without a WHP release. The engine itself uses
# only 200000-200006 (coraza.conf-recommended); CRS uses none of 2xxxxx. WHP
# also treats the range as staff-only for self-service exceptions.
#
# They switch the engine On for themselves (ctl:ruleEngine=On), exactly like
# the day-one promotions above, so they deny in every mode the sidecar runs in.
#
# LIMITATION, read before trusting them: HAProxy hands Coraza only the first
# buffer of the body (tune.bufsize 16 KB). A request that puts its abusive
# parts after ~16 KB of padding, or whose single part has a >16 KB header
# block, is not seen here. Vector (a) is the one enforced on the full request.
# Coraza's multipart parser (Go mime/multipart) also canonicalises part
# header names and stops at the first part it cannot read.
# ---------------------------------------------------------------------------
# (c) Field-name nesting. Counts '[' in every ARGS / FILES name (query,
# urlencoded and multipart names are all raw, PHP-shaped here; Coraza's JSON
# processor names args json.a.b and never produces '['). WordPress-ecosystem
# form names nest a few levels (menu-item-settings[12][title],
# variable_regular_price[0], input_3[]); PHP's own max_input_nesting_level
# is 64, so 16 is far above real forms and far below PHP's ceiling.
SecRule ARGS_NAMES|FILES_NAMES "@rx (?:[^\[]*\[){17}" \
"id:209610,phase:2,deny,status:403,log,t:none,\
msg:'Parameter name nested more than 16 levels deep (CVE-2026-109618 mitigation)',\
logdata:'%{MATCHED_VAR_NAME}',tag:'whp/cve-2026-109618',ctl:ruleEngine=On"
# (b) Part-header budget. A legitimate part carries 1-2 headers
# (Content-Disposition, usually Content-Type; a few clients add
# Content-Transfer-Encoding). Budget = 4 per part - 1 per part header, and
# the request is denied once it is 32 below zero: a single part needs 37+
# headers, and parts averaging 4 or fewer headers never trip it however many
# there are. Parts are counted by what they BECAME (each part is exactly one
# ARGS_POST or FILES entry), not by Content-Disposition lines, because a part
# can repeat Content-Disposition as easily as any other header.
# @unconditionalMatch runs setvar once PER VALUE (measured), and runs on
# empty values too, so empty fields and empty files are counted.
# No REQBODY_PROCESSOR chain on 209611: a chained link's setvar would run once
# per request, not once per part. On a non-multipart body it only RAISES the
# budget, and MULTIPART_PART_HEADERS (the only thing that lowers it) is empty.
SecRule ARGS_POST|FILES "@unconditionalMatch" \
"id:209611,phase:2,pass,nolog,t:none,setvar:tx.whp_cve109618_hdr_budget=+4"
SecRule MULTIPART_PART_HEADERS "@unconditionalMatch" \
"id:209612,phase:2,pass,nolog,t:none,setvar:tx.whp_cve109618_hdr_budget=-1"
SecRule TX:whp_cve109618_hdr_budget "@lt -32" \
"id:209613,phase:2,deny,status:403,log,t:none,\
msg:'Multipart part(s) with an excessive number of headers (CVE-2026-109618 mitigation)',\
logdata:'budget %{tx.whp_cve109618_hdr_budget}',tag:'whp/cve-2026-109618',ctl:ruleEngine=On"
# (c) Empty file controls: parts with an EMPTY filename (`filename=""` or
# `filename=`), which PHP does not count toward max_file_uploads. Coraza turns
# such a part into an ARGS_POST field (no FILES entry), so they are counted
# from the raw part headers. A browser sends one per untouched <input
# type=file>; legitimate forms have a handful. Denied above 64.
SecRule MULTIPART_PART_HEADERS "@rx (?i)^content-disposition:.*;\s*filename\s*=\s*(?:\"\"|;|$)" \
"id:209614,phase:2,pass,nolog,t:none,setvar:tx.whp_cve109618_empty_files=+1"
SecRule TX:whp_cve109618_empty_files "@gt 64" \
"id:209615,phase:2,deny,status:403,log,t:none,\
msg:'Multipart body with more than 64 empty file parts (CVE-2026-109618 mitigation)',\
logdata:'%{tx.whp_cve109618_empty_files} empty file parts',tag:'whp/cve-2026-109618',ctl:ruleEngine=On"
# -619: session upload progress. The engine flip sits on the LAST chain link
# on purpose: Coraza runs a chain starter's ctl as soon as the first link
# matches, even if the chain then fails (the 920440/944110 defect above).
# PHP only tracks upload progress for a
# multipart field named session.upload_progress.name (default
# PHP_SESSION_UPLOAD_PROGRESS, compared case-sensitively) that precedes the
# file parts, so that is exactly what is denied. WordPress core, WooCommerce,
# Divi, Gravity Forms and CF7 do not use the feature.
SecRule ARGS_POST_NAMES "@streq PHP_SESSION_UPLOAD_PROGRESS" \
"id:209616,phase:2,deny,status:403,log,t:none,chain,\
msg:'Multipart session upload-progress field (CVE-2026-109619 mitigation)',\
tag:'whp/cve-2026-109619'"
SecRule REQBODY_PROCESSOR "@streq MULTIPART" "t:none,ctl:ruleEngine=On"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Rule families intentionally kept at DETECT-ONLY for v1 — high FP rate # Rule families intentionally kept at DETECT-ONLY for v1 — high FP rate
# on customer mix. Promote individually after observation: # on customer mix. Promote individually after observation:
+296
View File
@@ -0,0 +1,296 @@
#!/bin/bash
## pkg-cache-repos.sh — BUILD-TIME ONLY, OPT-IN. Points dnf/apt at the internal
## package cache (nginx proxy_cache, CT130 "pkg-cache" on the Proxmox cluster)
## when PKG_CACHE is set; without it the repo files are left alone and builds
## use the upstream repositories as before.
##
## PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh rpm # AlmaLinux/EPEL/Remi/NodeSource
## PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh apt # Ubuntu, Debian, LiteSpeed
## pkg-cache-repos.sh release-install epel [remi] # GPG-verified release RPMs (EL9/EL10)
## pkg-cache-repos.sh nodesource MAJOR # NodeSource repo file + pinned key
## eval "$(pkg-cache-repos.sh go-env)" # GOPROXY via the cache (go.sum/sumdb verified)
## pkg-cache-repos.sh restore # undo everything above
##
## CANONICAL copy: cloud-apache-container/scripts/pkg-cache-repos.sh (docs/pkg-cache.md
## there). Other repos carry byte-identical copies - change it there first.
##
## Every RUN that rewrites must `restore` before it ends: it puts rewritten files
## back, deletes files this script created, deletes any NEW repo/apt file that
## still references the cache, then FAILS if any reference is left — so no shipped
## layer carries the cache's LAN address (images run on hosts that can't reach it).
##
## Why: on 2026-10-21 (PHP security release) builds failed on Remi mirrors lagging
## hours behind origin, Remi's origin timing out over HTTPS, and an EPEL mirror
## with a broken certificate. The cache fetches from the ORIGINS over HTTPS (with
## origin/mirror fallbacks and stale-on-error), keeps metadata ~3 minutes and
## immutable .rpm/.deb files long-term.
##
## Integrity (security review 2026-10-21):
## * PKG_CACHE must be https://. The cache's certificate is issued by the
## name-constrained "pkg-cache internal CA" (keys/pkg-cache/ca.crt, valid only
## for 192.168.1.40 / pkg-cache.*). That CA is trusted ONLY by the rewritten
## repos (dnf sslcacert= bundle, apt per-host CaInfo) and only during the RUN —
## never added to the system trust store.
## * gpgcheck=1 is forced on every rewritten repo; repo_gpgcheck=1 on Remi and
## AlmaLinux, which sign repomd.xml (measured: repomd.xml.asc 200). EPEL
## publishes no repomd signature (404), so its metadata relies on HTTPS end
## to end (client->cache->dl.fedoraproject.org/mirrors, all verified).
## * release-install downloads epel-release / remi-release (from the cache, or
## from the HTTPS origin without PKG_CACHE), checks the package name, imports
## the signing keys committed in keys/pkg-cache/ and requires
## `rpm -K` "signatures OK" before installing with localpkg_gpgcheck=1.
## * The php.net version gates (EXPECT_PHP_MIN / EXPECT_PHP_VERSION) still fail
## a build that gets old PHP (replay of old-but-signed metadata).
##
## rpm: each rewritten baseurl keeps the upstream origin as a SECOND url, so if
## the cache dies mid-build dnf fails over to the origin by itself.
## apt: apt has no per-source failover; CI only sets PKG_CACHE after probing the
## cache (with the CA), and a build without PKG_CACHE uses upstream sources.
set -euo pipefail
MODE="${1:?usage: pkg-cache-repos.sh rpm|apt|release-install NAME...|nodesource MAJOR|go-env|restore}"
shift || true
STATE=/var/lib/pkg-cache-repos # saved originals + manifest + created files
SHARE="${PKG_CACHE_SHARE:-/usr/local/share/pkg-cache}" # ca.crt + RPM-GPG-KEY-*
C="${PKG_CACHE:-}"
C="${C%/}"
die() { echo "FATAL(pkg-cache-repos): $*" >&2; exit 1; }
if [ -n "$C" ]; then
case "$C" in
https://*) ;;
*) die "PKG_CACHE must be an https:// URL (plain http is refused), got '$C'" ;;
esac
[[ "$C" =~ ^https://[A-Za-z0-9.-]+(:[0-9]+)?(/[A-Za-z0-9._/-]*)?$ ]] || die "PKG_CACHE has unexpected characters: '$C'"
HOSTPORT="${C#https://}"; HOSTPORT="${HOSTPORT%%/*}"
[ -s "$SHARE/ca.crt" ] || die "PKG_CACHE set but $SHARE/ca.crt (pkg-cache CA) is missing"
fi
# System CA bundle (EL vs Debian/Ubuntu path); the pkg-cache CA is appended to a
# copy in the state dir, never to the system store.
sys_bundle() {
local b
for b in /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt; do
[ -s "$b" ] && { echo "$b"; return 0; }
done
die "no system CA bundle found"
}
make_bundle() { cat "$(sys_bundle)" "$SHARE/ca.crt" > "$STATE/ca-bundle.pem"; }
# Files that may hold repo configuration; restore re-checks all of them.
repo_files() {
find /etc/yum.repos.d /etc/dnf /etc/apt -type f 2>/dev/null | sort || true
}
start_session() {
[ -d "$STATE" ] && return 0
mkdir -p "$STATE/orig"
repo_files > "$STATE/manifest" # what existed before this RUN touched anything
: > "$STATE/created"
printf '%s\n' "$HOSTPORT" > "$STATE/host"
}
save() { # keep the first-seen (pre-rewrite) copy
[ -e "$STATE/orig$1" ] && return 0
mkdir -p "$STATE/orig$(dirname "$1")"
cp -p "$1" "$STATE/orig$1"
}
created() { printf '%s\n' "$1" >> "$STATE/created"; }
if [ "$MODE" = restore ]; then
[ -d "$STATE" ] || exit 0
host=$(cat "$STATE/host")
# mapfile from a process substitution, not `cmd | while read`: no pipeline,
# so nothing for pipefail to misreport (scripts/tests/lsphp-info-probe.test.sh).
mapfile -t saved < <(cd "$STATE/orig" && find . -type f)
for rel in "${saved[@]}"; do
cat "$STATE/orig/$rel" > "/${rel#./}"
done
while read -r f; do [ -n "$f" ] && rm -f -- "$f"; done < "$STATE/created"
# A repo/apt file that appeared during this RUN and still names the cache
# (e.g. written by a package from the rewritten repos) is removed too.
mapfile -t current < <(repo_files)
for f in "${current[@]}"; do
if ! grep -qxF -- "$f" "$STATE/manifest" && grep -qF -- "$host" "$f"; then
echo "pkg-cache-repos: removing new file that references the cache: $f"
rm -f -- "$f"
fi
done
left=$(repo_files | xargs -r grep -lF -- "$host" 2>/dev/null || true)
rm -rf "$STATE"
[ -z "$left" ] || die "cache address $host still present after restore in: $left"
echo "pkg-cache-repos: original repository files restored (no reference to $host left)"
exit 0
fi
if [ "$MODE" = release-install ]; then
[ $# -gt 0 ] || die "release-install needs: epel and/or remi"
tmp=$(mktemp -d)
curl_opts=(-fsS --proto '=https' --retry 5 --retry-all-errors --max-time 120)
if [ -n "$C" ]; then
start_session
make_bundle
curl_opts+=(--cacert "$STATE/ca-bundle.pem")
fi
files=()
el=$(rpm -E '%{?rhel}' 2>/dev/null || true)
case "$el" in 9|10) ;; *) die "release-install supports EL9/EL10 only (rpm -E %rhel = '$el')" ;; esac
for name in "$@"; do
case "$name" in
epel) pkg=epel-release key=RPM-GPG-KEY-EPEL-$el
path=/epel/epel-release-latest-$el.noarch.rpm
origin=https://dl.fedoraproject.org/pub/epel/epel-release-latest-$el.noarch.rpm ;;
remi) pkg=remi-release key=RPM-GPG-KEY-remi2024
path=/remi/enterprise/remi-release-$el.rpm
origin=https://rpms.remirepo.net/enterprise/remi-release-$el.rpm ;;
*) die "unknown release package '$name'" ;;
esac
url="$origin"; [ -n "$C" ] && url="$C$path"
f="$tmp/$pkg.rpm"
curl "${curl_opts[@]}" -o "$f" "$url" || die "download failed: $url"
rpm --import "$SHARE/$key" || die "cannot import $SHARE/$key"
got=$(rpm -qp --nosignature --qf '%{NAME}' "$f" 2>/dev/null || true)
[ "$got" = "$pkg" ] || die "$url is '$got', expected $pkg"
chk=$(rpm -K "$f" 2>&1 || true)
[[ "$chk" == *": digests signatures OK" ]] || die "signature check failed for $url: $chk"
echo "pkg-cache-repos: $pkg verified ($chk) from $url"
files+=("$f")
done
dnf install -y --setopt=localpkg_gpgcheck=1 "${files[@]}"
rm -rf "$tmp"
exit 0
fi
if [ -z "$C" ] && [ "$MODE" != nodesource ] && [ "$MODE" != go-env ]; then
echo "pkg-cache-repos($MODE): PKG_CACHE not set - using upstream repositories"
exit 0
fi
rewrite_rpm() {
local f="$1"
grep -q -E '^#? ?baseurl=https?://(repo\.almalinux\.org|download\.example|rpms\.remirepo\.net|rpm\.nodesource\.com)/' "$f" || return 0
save "$f"
# Per [section]: a (commented or live) baseurl on a known origin becomes
# "baseurl=<cache>/<prefix>/<path> <origin>/<path>"; that section's
# mirrorlist/metalink is commented out (dnf prefers them over baseurl) and it
# gets sslcacert=<system bundle + pkg-cache CA>, gpgcheck=1 and, for repos
# that sign repomd.xml, repo_gpgcheck=1. Other sections are untouched.
awk -v C="$C" -v BUNDLE="$STATE/ca-bundle.pem" '
function flush( i) {
for (i = 1; i <= n; i++) {
if (hit && buf[i] ~ /^(mirrorlist|metalink)=/) buf[i] = "#" buf[i]
if (hit && buf[i] ~ /^(sslcacert|gpgcheck|repo_gpgcheck)[ \t]*=/) continue
print buf[i]
}
if (hit) {
print "sslcacert=" BUNDLE
print "gpgcheck=1"
if (signed) print "repo_gpgcheck=1"
}
n = 0; hit = 0; signed = 0
}
function rw(line, re, prefix, origin, sig, p) {
if (match(line, re)) {
p = substr(line, RSTART + RLENGTH); sub(/[ \t].*$/, "", p)
hit = 1; if (sig) signed = 1
return "baseurl=" C "/" prefix "/" p " " origin "/" p
}
return line
}
/^\[/ { flush() }
{
l = $0
if (l ~ /^#? ?baseurl=/) {
l = rw(l, "^#? ?baseurl=https?://repo\\.almalinux\\.org/almalinux/", "almalinux", "https://repo.almalinux.org/almalinux", 1)
l = rw(l, "^#? ?baseurl=https?://download\\.example/pub/epel/", "epel", "https://dl.fedoraproject.org/pub/epel", 0)
l = rw(l, "^#? ?baseurl=https?://rpms\\.remirepo\\.net/", "remi", "https://rpms.remirepo.net", 1)
# NodeSource signs its RPMs but not repomd.xml (404): gpgcheck only.
l = rw(l, "^#? ?baseurl=https?://rpm\\.nodesource\\.com/", "nodesource", "https://rpm.nodesource.com", 0)
}
buf[++n] = l
}
END { flush() }
' "$f" > "$f.pkgcache" && cat "$f.pkgcache" > "$f" && rm -f "$f.pkgcache"
}
rewrite_apt() {
local f="$1"
grep -q -E 'https?://([a-z]{2}\.)?(archive|security)\.ubuntu\.com/|https?://rpms\.litespeedtech\.com/|https?://(deb|security)\.debian\.org/' "$f" || return 0
save "$f"
# Debian: debian-security first, and "debian" only as a whole path segment,
# so .../debian-security is never rewritten as /debian/.
sed -i -E \
-e "s%https?://(archive|[a-z]{2}\.archive)\.ubuntu\.com/ubuntu/?%$C/ubuntu/%g" \
-e "s%https?://security\.ubuntu\.com/ubuntu/?%$C/ubuntu-security/%g" \
-e "s%https?://rpms\.litespeedtech\.com/%$C/litespeed/%g" \
-e "s%https?://(deb|security)\.debian\.org/debian-security/?([[:space:]]|\$)%$C/debian-security/\\2%g" \
-e "s%https?://security\.debian\.org/?([[:space:]]|\$)%$C/debian-security/\\1%g" \
-e "s%https?://deb\.debian\.org/debian/?([[:space:]]|\$)%$C/debian/\\1%g" \
"$f"
}
if [ "$MODE" = go-env ]; then
# Prints shell exports for `eval`. Without PKG_CACHE: nothing (Go defaults).
# GOPROXY "a|b" falls through on ANY error, so a dead cache costs one
# failed request, not the build. GOSUMDB stays the default sum.golang.org,
# reached directly: modules are still verified against go.sum + the sumdb.
[ -n "$C" ] || { echo "pkg-cache-repos(go-env): PKG_CACHE not set - default GOPROXY" >&2; exit 0; }
start_session
make_bundle
printf "export GOPROXY='%s/goproxy|https://proxy.golang.org|direct' SSL_CERT_FILE='%s'\n" "$C" "$STATE/ca-bundle.pem"
echo "pkg-cache-repos(go-env): GOPROXY via $C/goproxy (fallback proxy.golang.org)" >&2
exit 0
fi
if [ "$MODE" = nodesource ]; then
# Replaces `curl https://rpm.nodesource.com/setup_N.x | bash` (unverified
# remote script): writes the same nodesource-nodejs repo (origin URL, so the
# shipped image keeps a normal repo file) with gpgcheck=1 and the signing key
# committed in keys/pkg-cache/; with PKG_CACHE it is then rewritten like any
# other repo and `restore` puts the origin version back.
major="${1:?usage: pkg-cache-repos.sh nodesource MAJOR}"
[[ "$major" =~ ^[0-9]{1,3}$ ]] || die "bad Node major '$major'"
key="$SHARE/RPM-GPG-KEY-nodesource"
[ -s "$key" ] || die "missing $key"
rpm --import "$key" || die "cannot import $key"
rm -f /etc/yum.repos.d/nodesource*.repo
f=/etc/yum.repos.d/nodesource-nodejs.repo
printf '%s\n' "[nodesource-nodejs]" \
"name=Node.js Packages for Linux RPM based distros - \$basearch" \
"baseurl=https://rpm.nodesource.com/pub_${major}.x/nodistro/nodejs/\$basearch" \
"priority=9" "enabled=1" "gpgcheck=1" \
"gpgkey=file://$key" "module_hotfixes=1" > "$f"
echo "pkg-cache-repos: wrote $f (Node ${major}.x, key $(basename "$key"))"
if [ -n "$C" ]; then
start_session
make_bundle
rewrite_rpm "$f"
grep -h '^baseurl=' "$f" | sed 's/^/ /'
fi
exit 0
fi
case "$MODE" in
rpm)
start_session
make_bundle
for f in /etc/yum.repos.d/*.repo; do [ -f "$f" ] && rewrite_rpm "$f"; done
echo "pkg-cache-repos(rpm): repos now via $C:"
grep -h '^baseurl=' /etc/yum.repos.d/*.repo | grep -F "$C/" | sort -u | sed 's/^/ /'
;;
apt)
start_session
conf=/etc/apt/apt.conf.d/99pkg-cache
# Trust the pkg-cache CA for the cache host ONLY (apt per-host option).
printf 'Acquire::https::%s::CaInfo "%s";\n' "${HOSTPORT%%:*}" "$SHARE/ca.crt" > "$conf"
created "$conf"
for f in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
[ -f "$f" ] && rewrite_apt "$f"
done
echo "pkg-cache-repos(apt): sources now via $C:"
# (|| true: Debian 13 images have no /etc/apt/sources.list; grep's exit 2
# for the missing file must not fail the build under pipefail)
{ grep -h -E '^(deb|URIs:)' /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null || true; } | grep -F "$C/" | sort -u | sed 's/^/ /' || true
;;
*) die "unknown mode '$MODE'" ;;
esac
+34
View File
@@ -0,0 +1,34 @@
<!DOCTYPE html>
<!--
Default splash: served by HAProxy itself (`http-request return ... file`,
hap_listener.tpl) for any request whose Host is not routed to a site --
the server's own hostname, a bare IP, an unconfigured name. Static on
purpose: no version, no hostname echo, no customer data, no links (in
particular none into the control panel). Loaded with `file`, not
`lf-file`, so `%` is literal here and nothing from the request is
expanded into it. Must stay well under tune.bufsize (16 KB) or HAProxy
rejects the config.
-->
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="robots" content="noindex, nofollow">
<title>Web Hosting Platform</title>
<style>
:root{color-scheme:light dark;--bg:#f4f5f7;--card:#fff;--fg:#1d2330;--muted:#5b6475;--line:#dfe3ea}
@media (prefers-color-scheme:dark){:root{--bg:#0f1217;--card:#181c23;--fg:#e6e9ef;--muted:#9aa3b2;--line:#2a303a}}
html,body{margin:0;height:100%}
body{display:flex;align-items:center;justify-content:center;background:var(--bg);color:var(--fg);font:16px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;padding:16px;box-sizing:border-box}
main{background:var(--card);border:1px solid var(--line);border-radius:10px;padding:32px 28px;max-width:480px;width:100%;text-align:center}
h1{font-size:1.25rem;margin:0 0 .5rem}
p{margin:0;color:var(--muted)}
</style>
</head>
<body>
<main>
<h1>Web Hosting Platform</h1>
<p>This server is operated by a web hosting platform. There is no website configured at this address.</p>
</main>
</body>
</html>
+2
View File
@@ -0,0 +1,2 @@
User-agent: *
Disallow: /
+502
View File
@@ -0,0 +1,502 @@
"""gunicorn gthread worker that cannot be wedged by silent or slow clients.
Incident 2026-10-09 (sdbees), and very likely the 2026-07-07 whp01 one too
-------------------------------------------------------------------------
Both :8000 API workers had 7 of 8 pool threads parked in the kernel in
``tcp_recvmsg -> sk_wait_data``: a blocking socket read with no timeout.
/health and /api/blocked-ips hung for 40s+, the container stayed "healthy".
haproxy_manager.py makes NO in-process outbound network reads (no requests,
urllib, http.client, raw socket or ssl client anywhere; every ACME/HAProxy
interaction is a subprocess, already bounded since 2026.07.3). The reads that
blocked were gunicorn's own reads of the CLIENT socket:
gunicorn/workers/gthread.py TConn.init(): self.sock.setblocking(True)
Once the main loop sees a connection become readable it hands it to a pool
thread and turns the socket fully blocking -- timeout None -- before parsing
the request. Any peer that sends a few bytes and then waits (a TLS ClientHello
sent to this plain-HTTP port, a port scanner, a half-written request) pins
that pool thread forever. :8000 was published on 0.0.0.0 by the deploy
(`-p 8000:8000`, which bypasses firewalld) and internet scanners were measured
connecting to it. 4 such connections exhaust a worker's pool.
gunicorn's --timeout does not catch this: for gthread it only checks that the
worker's MAIN thread keeps heart-beating, and the main thread keeps polling
happily while every pool thread is stuck. --max-requests does not help either:
a worker whose threads never finish a request never reaches the limit.
socket.setdefaulttimeout() does not help: setblocking(True) explicitly resets
the timeout to None, overriding any default.
Stock gthread has two more exhaustion paths on the same port (review F1/F2):
* A connection that NEVER sends a byte never becomes readable, so it is never
handed to a thread and never gets any deadline -- but it holds one of the
worker's ``worker_connections`` (1000) slots. At 1000 the run loop stops
polling altogether and only waits on futures; none exist, so even the
attacker's FINs are never seen and the worker is wedged PERMANENTLY, still
heart-beating, with no request in flight for a watchdog to notice.
* Slowloris: one header byte every <timeout seconds never trips a per-recv
timeout, so a socket timeout alone does not bound header-read time.
What this worker changes
------------------------
1. CLIENT_TIMEOUT (HAPROXY_MGR_CLIENT_TIMEOUT, default 20s): every client
socket gets an explicit socket timeout at the exact point gthread makes it
blocking (re-applied on every keep-alive reuse). Python applies it to each
recv() call and to each WHOLE sendall() call -- gunicorn writes a response
chunk with one sendall, so any single chunk that takes longer than this to
transmit is cut off. Responses here are KB-sized JSON/PEMs to localhost, so
that never happens in practice. It does NOT bound how long the app takes to
handle a request (a long certbot run is a subprocess with its own timeout
and touches the client socket only once it is done).
2. FIRST_BYTE_TIMEOUT (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, default 10s): an
accepted connection that has not become readable within this is closed by
the main loop, freeing its slot (review F1).
3. HEADER_DEADLINE (HAPROXY_MGR_HEADER_DEADLINE, default 10s): total time a
client may take to deliver the request line + headers, counted from when
the connection was ENQUEUED for a thread, with a floor of
HAPROXY_MGR_HEADER_PICKUP_GRACE (1s) after a thread picks it up:
deadline = max(enqueue + 10s, pickup + 1s). Disarmed in handle_request()
(which gunicorn calls only once the parser has returned a complete
request). A 1s monitor loop shuts down (SHUT_RDWR, not close -- no fd-reuse
race with the blocked recv) the socket of any connection past it; the
blocked recv returns EOF and the thread is freed (review F2). A legit
request queued behind busy threads (certbot runs) already has its headers
in the kernel buffer and parses in milliseconds inside the 1s grace; a
slow client that sat in the queue gets only the grace, so reconnecting
drippers cannot starve the pool by queueing (re-review). Bodies are only
read by routes behind require_api_key, so the unauthenticated surface is
exactly this phase.
4. Wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS, default 3600s; 0 disables). If
EVERY pool thread has been inside the same request for longer than that the
worker logs every thread's Python stack to the error log and exits; the
gunicorn master respawns it in about a second, without touching HAProxy or
the edge. 3600s because a multi-domain /api/certificates/request runs one
certbot (up to 180s) per domain; with (1)-(3) closing the known client-side
sources, this is only a backstop for blocking sources not yet found.
5. Peer filter (re-review R1). haproxy-manager shares the client-net bridge
with every tenant container, so tenant PHP could reach haproxy-manager:8000
directly; the 127.0.0.1 publish only removes the internet path. On the
ports in HAPROXY_MGR_PEER_FILTER_PORTS (default "8000" -- NOT the :8080
default-page server, which HAProxy feeds over loopback) accept() closes
any connection whose peer is not in HAPROXY_MGR_API_ALLOWED_PEERS (default
"127.0.0.1,::1,gateway"; IPs/CIDRs; "gateway" = this container's default
gateway, which is where host callers arrive from through docker-proxy).
Refused before a thread or connection slot is used; logged at most once a
minute as a per-peer summary. Effective only once the edge publishes
127.0.0.1:8000 -- with a 0.0.0.0 publish a tenant can hairpin through the
host and arrive from the gateway too.
6. pool_status() exposes busy/total threads and the oldest in-flight age so
/health can report saturation.
This overrides gthread internals (enqueue_req, accept,
on_client_socket_readable, murder_keepalived, handle, handle_request) as they
are in gunicorn 23.0.0, which requirements.txt pins. A warning is logged if a
different version is loaded; scripts/test-gunicorn-socket-timeouts.py runs
real gunicorn in the Docker build gate and fails on behavioural drift.
"""
import errno
import ipaddress
import logging
import os
import selectors
import socket
import sys
import threading
import time
import traceback
from collections import deque
from functools import partial
import gunicorn
from gunicorn.workers.gthread import TConn, ThreadWorker
def _env_float(name, default):
try:
return float(os.environ.get(name, default))
except (TypeError, ValueError):
return float(default)
CLIENT_TIMEOUT = _env_float('HAPROXY_MGR_CLIENT_TIMEOUT', 20)
FIRST_BYTE_TIMEOUT = _env_float('HAPROXY_MGR_FIRST_BYTE_TIMEOUT', 10)
HEADER_DEADLINE = _env_float('HAPROXY_MGR_HEADER_DEADLINE', 10)
WEDGE_SECONDS = _env_float('HAPROXY_MGR_WEDGE_SECONDS', 3600)
WATCHDOG_INTERVAL = _env_float('HAPROXY_MGR_WATCHDOG_INTERVAL', 15)
MONITOR_TICK = 1.0
# Once a thread picks a queued connection up it gets at least this long to
# finish its headers, even if its enqueue-based deadline has already passed.
HEADER_PICKUP_GRACE = _env_float('HAPROXY_MGR_HEADER_PICKUP_GRACE', 1)
# Peer filter (review R1). Listener ports it applies to (comma-separated;
# empty disables) and the peers allowed through. "gateway" means this
# container's default gateway on its bridge network -- the address host-side
# callers arrive from via the 127.0.0.1:8000 publish (docker-proxy) and via a
# direct host->container-IP connect. Tenant containers on the same bridge
# (client-net) arrive from their OWN addresses and are refused.
PEER_FILTER_PORTS = os.environ.get('HAPROXY_MGR_PEER_FILTER_PORTS', '8000')
ALLOWED_PEERS_SPEC = os.environ.get('HAPROXY_MGR_API_ALLOWED_PEERS', '127.0.0.1,::1,gateway')
PEER_REJECT_LOG_INTERVAL = 60.0
def _default_gateways(route_file='/proc/net/route'):
"""IPv4 default gateway(s) from the kernel routing table."""
gws = []
try:
with open(route_file) as fh:
next(fh, None)
for line in fh:
f = line.split()
if len(f) >= 4 and f[1] == '00000000' and int(f[3], 16) & 0x2:
gws.append(str(ipaddress.IPv4Address(int(f[2], 16).to_bytes(4, 'little'))))
except (OSError, ValueError):
pass
return gws
def parse_allowed_peers(spec, gateways=None):
"""Return (networks, problems). Tokens: IPs, CIDRs, or the word
'gateway'. Unparseable tokens are reported, never silently widened."""
nets, problems = [], []
for tok in (t.strip() for t in (spec or '').split(',')):
if not tok:
continue
if tok.lower() == 'gateway':
gws = _default_gateways() if gateways is None else gateways
if not gws:
problems.append('gateway: no IPv4 default route found')
for g in gws:
nets.append(ipaddress.ip_network(g + '/32'))
continue
try:
nets.append(ipaddress.ip_network(tok, strict=False))
except ValueError:
problems.append('unparseable entry %r' % tok)
return nets, problems
def peer_allowed(addr, nets):
try:
ip = ipaddress.ip_address(addr.split('%', 1)[0])
except (ValueError, AttributeError):
return False
if ip.version == 6 and ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped
return any(ip.version == n.version and ip in n for n in nets)
TESTED_GUNICORN_VERSION = '23.0.0'
# Exit status for a watchdog-killed worker. Must NOT be 3 (WORKER_BOOT_ERROR)
# or 4 (APP_LOAD_ERROR): on those the gunicorn master halts the whole server
# instead of respawning the worker.
WEDGED_EXIT_CODE = 1
_current_worker = None
def pool_status():
"""Busy/total threads of the worker serving the caller, or None when the
app is not running under this worker class (e.g. tests, dev server)."""
w = _current_worker
if w is None:
return None
return w.pool_status()
class _QuietClientTimeout(logging.Filter):
"""gthread logs any OSError without an errno it recognises as
log.exception("Socket error processing request.") -- a full traceback.
A client timeout is now an expected, bounded event (scanners hit this port
constantly), so reduce it to one line instead of flooding the error log."""
def filter(self, record):
exc = record.exc_info[1] if record.exc_info else None
if isinstance(exc, TimeoutError) and \
str(record.msg).startswith('Socket error processing request'):
record.msg = ('Closed client connection idle for more than %ss '
'(HAPROXY_MGR_CLIENT_TIMEOUT)' % CLIENT_TIMEOUT)
record.args = ()
record.exc_info = None
record.exc_text = None
record.levelno = logging.INFO
record.levelname = 'INFO'
return True
class _InFlight:
__slots__ = ('conn', 'start', 'header_deadline')
def __init__(self, conn, start, header_deadline):
self.conn = conn
self.start = start
self.header_deadline = header_deadline
class TimeoutThreadWorker(ThreadWorker):
"""gthread worker with client-socket timeouts, a first-byte deadline, a
total header deadline and a wedge watchdog. See module docstring."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._inflight = {} # thread ident -> _InFlight
self._inflight_lock = threading.Lock()
self._pending = deque() # accepted, not yet readable (FIFO by deadline)
self._filter_ports = set()
for tok in PEER_FILTER_PORTS.split(','):
tok = tok.strip()
if tok.isdigit():
self._filter_ports.add(int(tok))
self._allowed_nets = []
self._rejected = {} # peer ip -> count since last log line
self._rejected_lock = threading.Lock()
self._last_reject_log = 0.0
# -- (1) client socket timeout ------------------------------------------
def enqueue_req(self, conn):
# Same three steps as ThreadWorker.enqueue_req, with the timeout set
# right after conn.init() -- init() is what calls setblocking(True)
# (== settimeout(None)), and it runs again on every keep-alive reuse,
# so the timeout must be re-applied here each time, not at accept().
conn.init()
if CLIENT_TIMEOUT > 0:
conn.sock.settimeout(CLIENT_TIMEOUT)
conn.whp_enqueued = time.monotonic() # header deadline counts from here
fs = self.tpool.submit(self.handle, conn)
self._wrap_future(fs, conn)
# -- (2) first-byte deadline for never-readable connections ---------------
def accept(self, server, listener):
# ThreadWorker.accept (23.0.0) plus: a deadline, and the conn is
# tracked in self._pending so murder_keepalived() can reap it.
try:
sock, client = listener.accept()
if self._filter_ports and self._server_port(server) in self._filter_ports \
and not peer_allowed(client[0] if isinstance(client, tuple) else '',
self._allowed_nets):
# Refused before it costs a TConn, a connection slot, a poller
# registration or a thread.
sock.close()
self._note_rejected(client[0] if isinstance(client, tuple) else str(client))
return
conn = TConn(self.cfg, sock, client, server)
conn.timeout = time.time() + FIRST_BYTE_TIMEOUT
self.nr_conns += 1
with self._lock:
self._pending.append(conn)
self.poller.register(conn.sock, selectors.EVENT_READ,
partial(self.on_client_socket_readable, conn))
except OSError as e:
if e.errno not in (errno.EAGAIN, errno.ECONNABORTED,
errno.EWOULDBLOCK):
raise
@staticmethod
def _server_port(server):
try:
return int(server[1])
except (TypeError, IndexError, ValueError):
return None
def _note_rejected(self, peer):
with self._rejected_lock:
self._rejected[peer] = self._rejected.get(peer, 0) + 1
if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
self._flush_rejected()
def _flush_rejected(self):
with self._rejected_lock:
if not self._rejected:
return
items, self._rejected = self._rejected, {}
self._last_reject_log = time.monotonic()
total = sum(items.values())
top = sorted(items.items(), key=lambda kv: -kv[1])[:5]
self.log.warning(
'Refused %d connection(s) to the management API from %d non-allowed peer(s) '
'(HAPROXY_MGR_API_ALLOWED_PEERS=%s): %s%s', total, len(items), ALLOWED_PEERS_SPEC,
', '.join('%s x%d' % kv for kv in top), ' ...' if len(items) > 5 else '')
def on_client_socket_readable(self, conn, client):
# ThreadWorker version only knows about self._keep; a first request
# comes off self._pending instead.
with self._lock:
self.poller.unregister(client)
queue = self._keep if conn.initialized else self._pending
try:
queue.remove(conn)
except ValueError:
# already reaped (race with murder_keepalived)
return
self.enqueue_req(conn)
def murder_keepalived(self):
# Runs on every pass of ThreadWorker.run() -- including the branch
# taken when nr_conns >= worker_connections, where the poller is not
# serviced. Reaping here is what breaks that permanent deadlock.
super().murder_keepalived()
if FIRST_BYTE_TIMEOUT <= 0:
return
now = time.time()
while True:
with self._lock:
if not self._pending or self._pending[0].timeout > now:
break
conn = self._pending.popleft()
try:
self.poller.unregister(conn.sock)
except (OSError, KeyError, ValueError):
pass
self.nr_conns -= 1
conn.close()
# -- (3) header deadline + busy tracking ----------------------------------
def handle(self, conn):
ident = threading.get_ident()
now = time.monotonic()
deadline = None
if HEADER_DEADLINE > 0:
# max(enqueue + D, pickup + grace): a legit request that waited in
# the queue behind busy threads already has its headers in the
# kernel buffer and parses in milliseconds; a slow client that
# waited gets only the grace, not a fresh full deadline (review:
# 40 reconnecting drippers otherwise starved legit requests).
enq = getattr(conn, 'whp_enqueued', now)
deadline = max(enq + HEADER_DEADLINE, now + HEADER_PICKUP_GRACE)
entry = _InFlight(conn, now, deadline)
with self._inflight_lock:
self._inflight[ident] = entry
try:
return super().handle(conn)
finally:
with self._inflight_lock:
self._inflight.pop(ident, None)
def handle_request(self, req, conn):
# The parser has produced a complete request line + headers.
entry = self._inflight.get(threading.get_ident())
if entry is not None:
entry.header_deadline = None
return super().handle_request(req, conn)
def _sweep_header_deadlines(self):
now = time.monotonic()
with self._inflight_lock:
late = [e for e in self._inflight.values()
if e.header_deadline is not None and e.header_deadline < now]
for e in late:
e.header_deadline = None # one shutdown per connection
try:
# shutdown, not close: wakes the blocked recv() with EOF
# without releasing the fd while another thread is in it.
e.conn.sock.shutdown(socket.SHUT_RDWR)
except OSError:
pass
if late:
self.log.info('Cut %d client connection(s) still sending request headers '
'after %ss (HAPROXY_MGR_HEADER_DEADLINE)', len(late), HEADER_DEADLINE)
def pool_status(self):
now = time.monotonic()
with self._inflight_lock:
starts = [e.start for e in self._inflight.values()]
return {
'pid': os.getpid(),
'threads': self.cfg.threads,
'busy': len(starts),
'oldest_busy_seconds': round(now - min(starts), 1) if starts else 0.0,
'client_timeout_seconds': CLIENT_TIMEOUT,
}
# -- (4) watchdog ---------------------------------------------------------
def _wedged_for(self):
"""Seconds the pool has been completely wedged (every thread busy in
its current request), measured from the YOUNGEST in-flight request.
0 when at least one thread is free."""
with self._inflight_lock:
starts = [e.start for e in self._inflight.values()]
if len(starts) < self.cfg.threads:
return 0.0
return time.monotonic() - max(starts)
def _dump_stacks(self):
frames = sys._current_frames()
names = {t.ident: t.name for t in threading.enumerate()}
out = []
for ident, frame in frames.items():
out.append('--- thread %s (%s)\n%s' % (
ident, names.get(ident, '?'), ''.join(traceback.format_stack(frame))))
return '\n'.join(out)
def _check_wedged(self):
wedged = self._wedged_for()
if WEDGE_SECONDS > 0 and wedged > WEDGE_SECONDS:
try:
self.log.critical(
'WEDGED WORKER (pid %s): all %s threads busy in the same request '
'for %.0fs > HAPROXY_MGR_WEDGE_SECONDS=%s. Exiting so the master '
'respawns it. Thread stacks:\n%s',
os.getpid(), self.cfg.threads, wedged, WEDGE_SECONDS,
self._dump_stacks())
for h in logging.getLogger('gunicorn.error').handlers:
try:
h.flush()
except Exception:
pass
finally:
# os._exit, not SIGTERM: a graceful exit joins the (stuck,
# non-daemon) pool threads at interpreter shutdown and would
# hang until the master's --timeout escalates to SIGKILL.
os._exit(WEDGED_EXIT_CODE)
def _monitor(self):
"""One daemon thread: header-deadline sweep every second, wedge check
every WATCHDOG_INTERVAL. Never allowed to die silently."""
last_wedge_check = time.monotonic()
while self.alive:
time.sleep(MONITOR_TICK)
try:
if HEADER_DEADLINE > 0:
self._sweep_header_deadlines()
if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
self._flush_rejected()
if time.monotonic() - last_wedge_check >= WATCHDOG_INTERVAL:
last_wedge_check = time.monotonic()
self._check_wedged()
except Exception:
try:
self.log.exception('haproxy_gunicorn_worker monitor error (continuing)')
except Exception:
pass
def init_process(self):
global _current_worker
_current_worker = self
try:
self.log.error_log.addFilter(_QuietClientTimeout())
except AttributeError:
pass
if gunicorn.__version__ != TESTED_GUNICORN_VERSION:
self.log.warning(
'haproxy_gunicorn_worker overrides gthread internals as of gunicorn %s; '
'running %s -- re-run scripts/test-gunicorn-socket-timeouts.py',
TESTED_GUNICORN_VERSION, gunicorn.__version__)
if self._filter_ports:
self._allowed_nets, problems = parse_allowed_peers(ALLOWED_PEERS_SPEC)
for prob in problems:
self.log.error('HAPROXY_MGR_API_ALLOWED_PEERS: %s -- that entry allows nothing '
'(fail closed)', prob)
self.log.info('Management API peer filter on port(s) %s: allowing %s',
','.join(str(p) for p in sorted(self._filter_ports)),
', '.join(str(n) for n in self._allowed_nets) or 'NOTHING')
threading.Thread(target=self._monitor, name='worker-monitor',
daemon=True).start()
super().init_process() # runs the event loop; does not return
+649 -131
View File
File diff suppressed because it is too large. Load diff
Binary file not shown.
+32
View File
@@ -0,0 +1,32 @@
Trust material for the OPT-IN internal package cache (scripts/pkg-cache-repos.sh,
docs/pkg-cache.md). Copied into images at /usr/local/share/pkg-cache/; nothing
here is added to the system trust store.
ca.crt "pkg-cache internal CA (CT130)", EC P-256, valid to 2036-10-18.
NAME-CONSTRAINED to IP 192.168.1.40 and DNS
pkg-cache.internal.cloud-hosting.io / pkg-cache.geekhome.local.
SHA-256 fingerprint BB:5B:9A:08:05:ED:2F:58:83:92:D0:99:9D:11:0A:30:14:88:68:F0:F0:88:65:5E:94:0B:75:EB:C2:A2:BD:F3
(private key lives only on CT130: /etc/pkgcache-ca/ca.key)
RPM-GPG-KEY-EPEL-10 Fedora (epel10) 7D8D 15CB FC4E 6268 8591 FB26 33D9 8517 E37E D158
fetched 2026-10-21 over HTTPS from dl.fedoraproject.org and
mirror.math.princeton.edu (identical); fingerprint matches
https://fedoraproject.org/security/
RPM-GPG-KEY-remi2024 Remi's RPM repository 2024 CF1D F005 7CE8 5DFF 5B2F 2A37 C2FD 3B2C 2A09 48E4
fetched 2026-10-21 over HTTPS from rpms.remirepo.net; byte-identical
to the key shipped in remi-release-10.2-1.el10.remi. Not found on
keys.openpgp.org / keyserver.ubuntu.com - confirm out of band.
Remi rotates keys (remi2025, ...): when remi-release starts being
signed by a new key, `release-install remi` fails loudly - add the
new key here (and to pkg-cache-repos.sh).
RPM-GPG-KEY-EPEL-9 Fedora (epel9) FF8A D134 4597 106E CE81 3B91 8A38 72BF 3228 467C
fetched 2026-10-22 over HTTPS from dl.fedoraproject.org and
mirror.math.princeton.edu (identical); used by `release-install epel`
on EL9 images (cloud-memcached-container, cloud-haproxy-container).
RPM-GPG-KEY-nodesource Nodesource Operations <operations@nodesource.com>
242B 8138 31AF 0956 2B6C 46F7 6B88 DA4E 3AF2 8A14 (rsa4096, 2023-12-22)
fetched 2026-10-22 over HTTPS from
https://rpm.nodesource.com/gpgkey/ns-operations-public.key (the key
NodeSource's own setup_N.x scripts reference); verified by dnf
gpgcheck against the nodejs RPMs. Not on keys.openpgp.org /
keyserver.ubuntu.com - confirm out of band. Used by
`pkg-cache-repos.sh nodesource MAJOR` (cloud-node-container).
+29
View File
@@ -0,0 +1,29 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGV4X6kBEAC3eQxgiWuo08uc3mHo4ELux++uqTnYz/tJzEf9Ou3h36WnhumA
Nvs+Ts5h8PBx879Y9/aIX1Z20p1kf6tBCinZnEJu59n+TAAsph0+XQlV1l5YkleK
Z2ff/Fg65k8QcLXWaIGykA/FaKznRiSurGuD6tRGhJw7DawEwBJr8QZSkRUpnH1L
URW97Q/iKrRPiE5VEayE0y8eAL28jIIiFvR+4oJMzvCsRRB/2wYZ2MlJOW91hcYf
mbUoXKOBD5UzsJylu7kj25K/ge8rEJ7KicOOwcdYddxsU3DxGSSfwF8AMagENcm2
XROeXknjm84A8sNlUkFZBJwfuc7eRTiZGJrnQQVYLrkKj8Mxpq9Ts7hU51TqAWNI
uvGDlJdYNE3D2RMqjMEsZ8ej08Thrib6xslu4NzTBkt+6QNnXL4E3hEgYtoyio60
GswSz2ulogKg7X4JrNdJYE8/qNowyF3hoVgj5TG1/wQRq+5HlMMOLjgGu9wzLUix
fnVfEUnzaofbrUf4/GabCaeY8xRe4tFQrvzigQ4g+kgwKKnfAeqBmPov0yljkw9z
BYJWR5zvaw0ffg9Ing00KUSaXBXA5jSlgk1603Y+LefY1SlXsTyqohiRvGH6FI77
HNMo72DwoJfFcYjncZUzKgXWJECR4nhVsdj6pKoOjcQ4aSuyVxtsR86ASQARAQAB
tChGZWRvcmEgKGVwZWwxMCkgPGVwZWxAZmVkb3JhcHJvamVjdC5vcmc+iQJOBBMB
CAA4FiEEfY0Vy/xOYmiFkfsmM9mFF+N+0VgFAmV4X6kCGw8FCwkIBwIGFQoJCAsC
BBYCAwECHgECF4AACgkQM9mFF+N+0Vhv/A/+PlhPLSctGRCUEahE+cN4764Acc3p
l40ZYzXRhqR0/Tc1/cSDjlA3qVTc8SPohi5OJXwCyr9EiMqKoyoDN097euqbYpyp
yN/Pj0lBjsXwcpdDtZ21WGeQU0Khb04N68bMtJbDaxeBciTvDDQravZuPPh0m4Rg
Z6myEoa6Aa6EK0hI1Qwi1qIWeRiuEkVT671IaKVETBW5XiUpNBXDAB/L+6DzUF9u
scBzfsUDiPO6NrpYDtV3jwq22y6gWluIct/Ka8brwPbqK2sBfFzrHboRhfqlTGjs
7F9qUGwIQZn/A8iozXZYQ0+JG1bhQyvjA8eN1GOcRpT+O7H7JXN49o6IG2As4+iK
F04+qjqAu2sVfpD8mzM2VubFNllcKKiyCzRYHhSbObRCPzsudDL9GPiXeGGaCuWg
sDkiA1MESvf2tLETAGBs/TziO4GwmXUtlKbRiq1FYm90mVq9mBxPZ/Idn+yZusNB
0O5SXIbI8lYZw5n4XTK4b+byHRBYsOTHiTsGvjTF2Y7oSwW2CVUmL6RZ23mI4qoY
1p5kzRS+GjT1acnTei/FTsOlIKCsjfeHx7uxCkX6xpAD8P3UtLQqfsgH0CL4vSZt
TGO6L1InQlp4ZG3OYIomTKbD3/R0wod3U3dTqdulQMXL895u6OLTY3spY2m2MO2k
p9Dfd2pKuxK9Mys=
=mhQZ
-----END PGP PUBLIC KEY BLOCK-----
+29
View File
@@ -0,0 +1,29 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGE3mOsBEACsU+XwJWDJVkItBaugXhXIIkb9oe+7aadELuVo0kBmc3HXt/Yp
CJW9hHEiGZ6z2jwgPqyJjZhCvcAWvgzKcvqE+9i0NItV1rzfxrBe2BtUtZmVcuE6
2b+SPfxQ2Hr8llaawRjt8BCFX/ZzM4/1Qk+EzlfTcEcpkMf6wdO7kD6ulBk/tbsW
DHX2lNcxszTf+XP9HXHWJlA2xBfP+Dk4gl4DnO2Y1xR0OSywE/QtvEbN5cY94ieu
n7CBy29AleMhmbnx9pw3NyxcFIAsEZHJoU4ZW9ulAJ/ogttSyAWeacW7eJGW31/Z
39cS+I4KXJgeGRI20RmpqfH0tuT+X5Da59YpjYxkbhSK3HYBVnNPhoJFUc2j5iKy
XLgkapu1xRnEJhw05kr4LCbud0NTvfecqSqa+59kuVc+zWmfTnGTYc0PXZ6Oa3rK
44UOmE6eAT5zd/ToleDO0VesN+EO7CXfRsm7HWGpABF5wNK3vIEF2uRr2VJMvgqS
9eNwhJyOzoca4xFSwCkc6dACGGkV+CqhufdFBhmcAsUotSxe3zmrBjqA0B/nxIvH
DVgOAMnVCe+Lmv8T0mFgqZSJdIUdKjnOLu/GRFhjDKIak4jeMBMTYpVnU+HhMHLq
uDiZkNEvEEGhBQmZuI8J55F/a6UURnxUwT3piyi3Pmr2IFD7ahBxPzOBCQARAQAB
tCdGZWRvcmEgKGVwZWw5KSA8ZXBlbEBmZWRvcmFwcm9qZWN0Lm9yZz6JAk4EEwEI
ADgWIQT/itE0RZcQbs6BO5GKOHK/MihGfAUCYTeY6wIbDwULCQgHAgYVCgkICwIE
FgIDAQIeAQIXgAAKCRCKOHK/MihGfFX/EACBPWv20+ttYu1A5WvtHJPzwbj0U4yF
3zTQpBglQ2UfkRpYdipTlT3Ih6j5h2VmgRPtINCc/ZE28adrWpBoeFIS2YAKOCLC
nZYtHl2nCoLq1U7FSttUGsZ/t8uGCBgnugTfnIYcmlP1jKKA6RJAclK89evDQX5n
R9ZD+Cq3CBMlttvSTCht0qQVlwycedH8iWyYgP/mF0W35BIn7NuuZwWhgR00n/VG
4nbKPOzTWbsP45awcmivdrS74P6mL84WfkghipdmcoyVb1B8ZP4Y/Ke0RXOnLhNe
CfrXXvuW+Pvg2RTfwRDtehGQPAgXbmLmz2ZkV69RGIr54HJv84NDbqZovRTMr7gL
9k3ciCzXCiYQgM8yAyGHV0KEhFSQ1HV7gMnt9UmxbxBE2pGU7vu3CwjYga5DpwU7
w5wu1TmM5KgZtZvuWOTDnqDLf0cKoIbW8FeeCOn24elcj32bnQDuF9DPey1mqcvT
/yEo/Ushyz6CVYxN8DGgcy2M9JOsnmjDx02h6qgWGWDuKgb9jZrvRedpAQCeemEd
fhEs6ihqVxRFl16HxC4EVijybhAL76SsM2nbtIqW1apBQJQpXWtQwwdvgTVpdEtE
r4ArVJYX5LrswnWEQMOelugUG6S3ZjMfcyOa/O0364iY73vyVgaYK+2XtT2usMux
VL469Kj5m13T6w==
=Mjs/
-----END PGP PUBLIC KEY BLOCK-----
+52
View File
@@ -0,0 +1,52 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGWFsqMBEAC4YlSuNpHjOiQVTC8StolVx3J8E9SC6G4wmVWbytMD1Zivfanj
3LKDqdy7vpbViXA9whpyGQgQYdOlPKaNHWw4GSeGTnP1FbvaK1NJ9MnAzUysh49R
cqoktYeIQbqgOgZHw5/gVH5bYObf1KJQ2LY/je9WTlW3QMRe1TaW0F0AZnJMBIOG
JllE33QIpxDnOQHnvdUGq+HscUOTrW4Q3AdzwsQxU4GX82d9qFFr8HjAmUjlUKyh
PgvMpb8VDvR7NE6et+wWh6BMYILo3VJFP7/gTXGiy3WWs5GW/PNXIXw5/f2w1B+e
Dh5tlfGG4q/4P8mbv7C6RPHoF2Lq+W69HLqSq/l42WUL41tHlFTsluCIKpeeMxAu
olxx3Z5momHMMyh7rbZtQDqFMTAqxNjT1NinKkWyj7KP5WE9hVosyd3UMR4UYDMm
Zb6LI4N7Hht887+CY7grxTJ9cGQhMFfvLyvlf/FWQY4S6G7sEaUV71N9StvXBNG/
iNoIMzhwjBmrmvT/UGPkLSi0SIBOxeGw2rwngnYikIJxuTQ4V40lm8o6eMFca7rW
BQvAEncV8pvi/3FqrUBZ/U53s7NO24hBDoaQPzFbVCw0GHHf9GD7KoQb5c9ONoCZ
Qu2YWo7ajPLzl/h2pJ2Kn29Ywev9MpuuPTGvifiKilcBVLYP3ZJg4ZfRBwARAQAB
tDFOb2Rlc291cmNlIE9wZXJhdGlvbnMgPG9wZXJhdGlvbnNAbm9kZXNvdXJjZS5j
b20+iQJOBBMBCgA4FiEEJCuBODGvCVYrbEb3a4jaTjryihQFAmWFsqMCGwMFCwkI
BwIGFQoJCAsCBBYCAwECHgECF4AACgkQa4jaTjryihTD+hAAtTU155ke7xmnT0ph
odbGjm3mOGdkdaDH+xmWtPtufiPncgHMCYh4aIDRloog8rFE+pP3qfwJmUKxthwD
drrLAzfUwOrGxPnYH3MWwg66RVVoiiU+Hh42gJTez+P/K0VMQZXH0XTZD96TiwT/
S0+WvQPuJ9pda2L4cjV+YP2+m0ERUG9QPj0Shgw+/PWCGWKD6J9DRToiwfwCTOvc
DNolF1EcRlfPMTsqI2Bdc+TQywqU1Py1MUDSAPXxdK6nW2oM7eC5l3XXefcQ6JOS
AuNdEDIsaf+H76j6/wSGsuIcJjUo9wmKjEV1iJYwT6jLbjaVQxXgEPHBSb+kzanx
FWyrYP6TZpeY2gT4bBMsOKd7ay1I3/nIHwe6x8Yzkqzz+7YVhNpMbybH72NTxDWQ
UmiboJs9r7j1lkHlBjxU9RLF5laYSeCIXD62KQVoP5ao1LSPKsCXFqcdpUgLYtRv
CXxdriiHp4m9eGzbS7RW1TLm6tMOKbI4LzOtMm4DoZzGD2FTfsUtn6ALx9FSgvl/
VMlzqZes+rclcukACtutvtB8e0yRhuzPUN56ctLEZfI1XzQFoN8Vdj6HWtiXWRiJ
EohuufE2cEebmX90NRsbh4KAFXNEQROlMeMf7ufuSsk7QXLTutxmWSQvWwyS+P5/
ZqCapIDlRCJFZX4N9a1IXo0OgCC5Ag0EZYWyowEQAKz7sA6WBKFVcmzcyr6y9TI5
RemnzotRUdqXj8tD70xvV1nBEf+BfTw3hsiV019ZboPBae+cRaD4plQbNb0du/+Q
UFq4ja055ZEld01RqE49L562pWvA5cwSmjs9+ZMI1ja0HbTb2K4razpzX3be+Zul
TbH20X5FHTz+Jyfmw+G3Y9okw3OIzuYo16Lv/Qs8VoBmj01PItzq8dX5wJiIlVxU
8wKHKy1kiKGCgyPnvqcXCGen0EoUS9wsWK1QHfryscYGVF9y1TBQvJa9HIpWPrIZ
slSiVS4tQhGHfLDVlaDsYFrH9SIUkvW7peSEiPE1TKQjvbMS3CcUZRaMcGfgyDPe
gXwnNwZbJ2/cisXbdAJaGT49V1Y+Sc1oKSGSG4FaGUNhS3p3cB7qlX2zS/sR2ael
90fXR6nhjvhMAS98i7663pCUv9q0Z/NXNZwdXjbSF4W8gq3PVt7hCnJrngsGSkGu
Qz0dc227/sAjh9CxYm8kQqUuQtiPVfKwkcUUT+O7jr2/6eAXZbTYVZbvaLEGMrKA
bdWKpZMMOKAI45f+ZAcEmJG4D68N0jzVrLCXMuuLq/yqqyEmxDjSOH0qVutl8jxg
vfxPxKYX8nnAzJZJHGHmV/ihjfClSdYePKExHuFVVhvc4BxE1XfPUeCZG4JpB9/B
gNCiDPsg4JIw6GO5LS2xABEBAAGJAjYEGAEKACAWIQQkK4E4Ma8JVitsRvdriNpO
OvKKFAUCZYWyowIbDAAKCRBriNpOOvKKFOTKD/40WN6ZRfP6hVLAX26DdyN9Bqzg
D+0hJilmHL1YFeLt9U/YPhOZJxPaDGrMfldC0FSXKKr7H5C7FQErl936108wGohz
53KGbNid0crNeaWrQuWk3QjaMF62Yc+NVB8b5LoEk0TEt9KPO6kveYDtW6Ygkw9i
5aaS/AC+8GChOH6lWhLHcO91T7+Hi5qRez7qyme/AlHUTV9/jE6JgFJ7thCCAqBQ
AfX3mGFlTwSTMHDeYq81MMl9UhYgoFJtx8PuqmSV7Eb29ccJofz3J+b5OJvpmLSZ
Hu22Csa8k2V/1ojeL0DjQHD06V75F7BerxLR/rbJ64GkIiOdm4ntXC4a335gjQfM
5tI6LtGsIpr+Le4C068eDj7YpzS0ql3g6o4A1TbtapLccgEj4iw95H4KKh0Ro5r0
0s6LoSLu9uva3LiLVn6xXvpw3iv8FnhTr5PEiPnorXvxcCnUrB3Rkk6Wh1SKjpxa
u6KiFzH0CXR1AgUdWSC0rAmvEdFjXgaoQd+Z2eJFHEAsg91URd9NtdeDsyMwdbHh
iBZPGX5ddPjJA7nwKbkibCv7xiodd9ph8NmGLqD/2SDU1rqvPuCDMqLuxc89Gvhs
3CeLTSJK9Uzb3oNdZLxL8CECip1ZkowVfTL+jY23phS9+Z0Db/yZqot3ZZJAAG8g
oSVkzT/Mwp9zFVQu5w==
=drMP
-----END PGP PUBLIC KEY BLOCK-----
+52
View File
@@ -0,0 +1,52 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGWdEG0BEADV9c8aSIITQmoWKN0ac4c8AEhZcdzpWSRPwHIS8syVRw/gj30W
Mn1OoxqFn2UECh2jjqYaQFECXNrM2/lzO4GhvcuoPn6p0rXFMvYvwtiOJ+rmDvTe
UOwRxkZhYehpvsRPFoVhlYJh3U1BM0UHm2iE/TIqVusY9FHQuCMZVl0BHk81YGDa
qE3cyu+9zOs8z1bYlsYBgLIbtb4+loQTRexFTfZkUcCWgAaNyur8Wy7Msdm5LOqN
ZZu6bNKBOv0Cr+BOwGNoFRbi4IM+5H+0eI1n7V5if4CuXfdYHp+Pm+PQvftm/nqB
jRLKqioRKzHjk7xNMsi5lM+SU4WyfU77pr3RXuJR6fkymLVAeu5sy2U6ySj0SXIO
/I3oPp4/zJ8/e94jG5iDqh92b5IWyXSE+EGw4kkpUvZLrK8vyyEfk8h9Gf8qA5uL
yGfw0vnSVjYb+RdnfZFVdPaIEmeY8tnobS87duj8JESZiSWo5sU4filkciW5e+MJ
w9wohgl+Cqm2J5Ljqus6DRm4ptKbZD8QZRnmGcSk6O2VopBKt2XLat4eIJLr/EiN
0CJ1EH/oJmEeZbYtUjxjsGsbPXicwDH6ZIeIMSaiDgqRxPm1LTGVLjSHAk+k0e5l
ulEmaoA7t/p3U2peGc7ONvozXiyNHZdGiRkqFC1mA9/JpeWw4KFH6JSgmQARAQAB
tEtSZW1pJ3MgUlBNIHJlcG9zaXRvcnkgMjAyNCAoaHR0cHM6Ly9ycG1zLnJlbWly
ZXBvLm5ldC8pIDxyZW1pQHJlbWlyZXBvLm5ldD6JAlEEEwEIADsWIQTPHfAFfOhd
/1svKjfC/TssKglI5AUCZZ0QbQIbAwULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIX
gAAKCRDC/TssKglI5Na7EACDTGjTcuTyudfuu/JC41BSurclCcwZyPsGaClyc0qx
Tk8Rs4rbyVw1u494fiFG/yEVl7LGGja84UJOlMdw82KgVUmr4fGLaBBFWm+KnZ4W
4vvFcCIxUshENOaDrkhby9F5P379gVOK9I3n9l4Q2xt+sWUNwE74CNfCTUeYMIm/
/AHbb1DTM9FuIT6FssgvILMTlksiyrw8U8bj+QUzPZrFCpvv+PnFA7ZDMEyCf2Tz
/zawxoP3a3b+3/g61U9SVIkfToonnorqh9lkQYhsdKTFHW6eESqe7zcqMBZv/sSc
3kFzl22KczSdU8eYxdujv9BmlW2SOrQsnuMYxvoD1LzhwSB/nOapNrWU+djhJCof
ueZ+rBW4QWysmG0dEp5Wpz85TstAnQ+j5aCOMSAkiAVsrJ0Ff3JXakIiL7rEtccS
dhgBT06LJtJy1NGIC0Sotn45n6dtfi6IuI8LnELCKeHxhdFSJcAPB8y95u0uHymy
sqMfzjq1VcYtzC5II6Yd5Gtb2+QzyQwORslwXhUdXwZKUc5i6X5xFMSxU523fH9n
i+kQlUTPkOUr66v4fmYDj+ZlJQSFoPylcHCke/vOo/rH5nj+YMPa6LdohsdzHKpd
DkKrk+Udj5aDXtXGIFYwoNKO4BnCsz/ImZfB4CQ+ekJsY1pFill3+MRTM6IvSA+P
7LkCDQRlnRBtARAAq8k4FM9euMMsXz0buxxyr0ew0ecWNhZGy24a6DXORVTEJbgI
T3HpV/C2PWkwgEBswYzREm+en61jhWr5unCiytO2klT+K2WNcIUNW+z2ciL/JNaJ
OcKwjkiY/A+KtDStzllCQ/XnpgAXGvFZjMjhHrNGQoAwNnPE/cMtYwcuVdW3Ipm3
cOOtzxRmfnhP8YUEyHxdos/7AZowaiLIUkaL5RUaygGrfPGyUXPl1aXS1oiOm37W
RiKCGUxCHSlpBqeh1WQPOaps8y+ME9jMzF9SLvqiXlKPllJIx9i8tHUOnX6Y2U2x
5sXsCEiA+zNVMU3YQ04++Yx2Rp6sDhNEnAbTCqqequqIzaXHZWwszAeG6pj1G90m
8PwDeHy641ySgDMwVPpnH5rHWmmT96XvF9QNg07OBtW9/phryzwAuHogrBY4DyFD
pUxDkTv9TxfBaS/s1uJxEGknIgnKzQo79mRy206waGKRA2AV36ORSfU12sK8pl3r
N95iIDh/OIuMl2STaXilhwb1hRpGFvwRlq3Hvvan4SY1aO219Q+VxMGXeqC5F/rm
Anh7mRVMP2qu0rj7L8xJ/OD/bPfuWL72/PvviBr/jdRM9I+Vj8dPGjhUwhK1dpEG
Q6yvoVXLHXucVHfIVR+ALLBmK8oHXNIFeIXhiUbgmRAilPHRz6Rf5pgbrC0AEQEA
AYkCNgQYAQgAIBYhBM8d8AV86F3/Wy8qN8L9OywqCUjkBQJlnRBtAhsMAAoJEML9
OywqCUjkQ2kP/iwqvt1sX85KM4OSE9ftqemwnvh19NizMKHMYGgikMAvy2CndEOI
wzKVeE1iK9/ivZQQIbD5l0G9hzVZjIxbgu9dAcxM+jdmaPyUj7yn6AX2yeGAeHyA
qzNKZaTLJPieRzdXk/uKPVyWUuzbR0vwR7n8N5N0pe/FEfO9O0uX5iz2kf5+b10c
WqvzLmO+czuyCjTHnjhKfE73iIktYE3MnM8qFHPiKwm+jeTkUnmInoR3qIKR/SCV
u2vk/q5uvpNaUSPSDKji6wh3/hbNbPocLSGIjugfWNm/4LhlSHPNnRJ2dIvSxeiI
qtSWxhULEyHGBPMLDriVumCV7Rb/byJwiMbmCIpxB5XLtbgt7nIqWQTko+6ci71S
4rDXolM3bytYbpqq4b/Xc1bhBuTX3omwrd0kxmGqad8eia6O6iBsP7GAl4QkySm4
MfKQfohjXTnT60wqEzQwLGI4NKN9bHRSsGINHX5p+Kz9HLf87fTY4uglbFxg0/II
Y3nko5KBw/gfe0WpwZY0sWjK20cXJn3xLQjoG8OjUNUCUYTRsiAvn4AiDRt++qtl
bvJjJqOU1k4GrGoCVpEUiOc9Cw1YZREcFe8H2at3pGnxJ1DFlYA916kNAxYANp2s
r3N3agwevgIzP78hynxjm6a5H2w0vkg189pZn2RR9zX9gmrvlqg1o0P1
=meaU
-----END PGP PUBLIC KEY BLOCK-----
+15
View File
@@ -0,0 +1,15 @@
-----BEGIN CERTIFICATE-----
MIICPzCCAeagAwIBAgIUJI0QJZ+PCrSQ/nKZFENwwcp8Zr0wCgYIKoZIzj0EAwIw
TDEmMCQGA1UEAwwdcGtnLWNhY2hlIGludGVybmFsIENBIChDVDEzMCkxIjAgBgNV
BAoMGWNsb3VkLWhvc3RpbmcuaW8gaW50ZXJuYWwwHhcNMjYxMDIxMTkzNjIyWhcN
MzYxMDE4MTkzNjIyWjBMMSYwJAYDVQQDDB1wa2ctY2FjaGUgaW50ZXJuYWwgQ0Eg
KENUMTMwKTEiMCAGA1UECgwZY2xvdWQtaG9zdGluZy5pbyBpbnRlcm5hbDBZMBMG
ByqGSM49AgEGCCqGSM49AwEHA0IABBJh75sC5vSzTnkYl0CYYEA/TKUHEXhbtanV
OVNW4NlD3za5+bu5YuKjpbFAva4TCZqWsaS9IMd/JfsaFbf5XVejgaUwgaIwEgYD
VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFP7UCTf/
v7KU1KBrs1vqa71/TZl0MF0GA1UdHgEB/wRTMFGgTzAKhwjAqAEo/////zAlgiNw
a2ctY2FjaGUuaW50ZXJuYWwuY2xvdWQtaG9zdGluZy5pbzAaghhwa2ctY2FjaGUu
Z2Vla2hvbWUubG9jYWwwCgYIKoZIzj0EAwIDRwAwRAIgOod4wPHu0kAzYAw8KeGg
cIrFMwQ4ATKVTkDioiSFOqICIEhY+sGj/YZ4oBMst3b/AAz9zMEJAXpnCkSTI6tM
YAIl
-----END CERTIFICATE-----
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env bash
# Assemble the multi-arch index tag(s) from per-platform images that were
# already pushed BY DIGEST (no tag). Used by the trusted x86 publish job
# (see .gitea/workflows/build-push-coraza.yaml; template for the other WHP
# image repos, docs/arm64/IMAGE-PIPELINE-PLAN.md section 4.1/4.4).
#
# assemble-multiarch-index.sh \
# --amd64 sha256:<digest> \
# --arm64 sha256:<digest> --arm64-outcome success|failure|cancelled|skipped \
# --repo <registry/path> [--repo <other registry/path> ...] \
# --tag <tag> [--tag <tag> ...] [--builder <buildx builder>]
#
# Rules (all enforced here so the workflow stays thin and these are testable):
# * amd64 is mandatory. A missing/malformed amd64 digest, or an amd64 digest
# that does not inspect as linux/amd64, aborts with exit 1 and publishes
# NOTHING (the same "build failed, no tag" outcome as before multi-arch).
# * arm64 is attached ONLY when its step outcome is exactly "success" AND the
# digest is well-formed AND it inspects as linux/arm64. Otherwise the tag is
# published amd64-only (warning, exit 0). The arm64 digest must come from
# THIS run's step output - the workflow never reads it from a registry tag,
# so a previous build's arm64 can never be re-attached to a newer amd64.
# * every --tag is created in every --repo from <repo>@<digest> sources in
# that same repo (both digests were pushed to every --repo by the build).
#
# Env: IMAGETOOLS overrides the command (default: "docker buildx imagetools");
# the test suite points it at a stub.
set -euo pipefail
IMAGETOOLS=${IMAGETOOLS:-docker buildx imagetools}
amd64=""; arm64=""; outcome="skipped"; builder=""
repos=(); tags=()
die() { echo "assemble-multiarch-index: $*" >&2; exit 1; }
warn() { echo "::warning::assemble-multiarch-index: $*"; }
while [ $# -gt 0 ]; do
case "$1" in
--amd64) amd64=${2-}; shift 2 ;;
--arm64) arm64=${2-}; shift 2 ;;
--arm64-outcome) outcome=${2-}; shift 2 ;;
--repo) repos+=("${2-}"); shift 2 ;;
--tag) tags+=("${2-}"); shift 2 ;;
--builder) builder=${2-}; shift 2 ;;
*) die "unknown argument: $1" ;;
esac
done
re='^sha256:[0-9a-f]{64}$'
[ ${#repos[@]} -gt 0 ] || die "need at least one --repo"
[ ${#tags[@]} -gt 0 ] || die "need at least one --tag"
[[ $amd64 =~ $re ]] || die "amd64 digest missing or malformed ('$amd64'); refusing to publish"
bflag=(); [ -n "$builder" ] && bflag=(--builder "$builder")
# arch_of <ref>: prints the image config's architecture ("" on any failure).
arch_of() {
# shellcheck disable=SC2086
$IMAGETOOLS inspect "${bflag[@]}" "$1" --format '{{json .Image}}' 2>/dev/null \
| python3 -I -c 'import json,sys; print(json.load(sys.stdin).get("architecture",""))' 2>/dev/null || true
}
for r in "${repos[@]}"; do
[ "$(arch_of "$r@$amd64")" = amd64 ] || die "$r@$amd64 does not inspect as linux/amd64; refusing to publish"
done
use_arm64=0
if [ "$outcome" != success ]; then
warn "arm64 step outcome is '$outcome': publishing amd64-only (no arm64 digest is carried over)"
elif ! [[ $arm64 =~ $re ]]; then
warn "arm64 step reported success but its digest is missing/malformed: publishing amd64-only"
elif [ "$arm64" = "$amd64" ]; then
warn "arm64 digest equals the amd64 digest: publishing amd64-only"
else
use_arm64=1
for r in "${repos[@]}"; do
if [ "$(arch_of "$r@$arm64")" != arm64 ]; then
warn "$r@$arm64 does not inspect as linux/arm64: publishing amd64-only"
use_arm64=0; break
fi
done
fi
for r in "${repos[@]}"; do
targs=(); for t in "${tags[@]}"; do targs+=(-t "$r:$t"); done
srcs=("$r@$amd64"); [ "$use_arm64" = 1 ] && srcs+=("$r@$arm64")
echo "assemble-multiarch-index: $r tags=${tags[*]} sources=${#srcs[@]} ($([ "$use_arm64" = 1 ] && echo 'amd64+arm64' || echo 'amd64-only'))"
# shellcheck disable=SC2086
$IMAGETOOLS create "${bflag[@]}" "${targs[@]}" "${srcs[@]}"
done
echo "platforms=$([ "$use_arm64" = 1 ] && echo linux/amd64,linux/arm64 || echo linux/amd64)"
+81
View File
@@ -203,6 +203,82 @@ cert_bundle_valid() {
return 0 return 0
} }
# cert_is_expired CERT_FILE
#
# Returns 0 only when openssl positively reports that the first certificate in
# CERT_FILE has expired. Anything else (not expired, unreadable, no openssl)
# returns 1; unreadable material is rejected with a better message by
# cert_bundle_valid().
#
# Why: HAProxy loads the crt directory wholesale and serves the FIRST file that
# claims an SNI name. An expired bundle in there is never useful and can only
# shadow a valid one - on 2026-10-07 renew-certificates.sh kept re-publishing
# the expired brain-jar.com lineage every 12 hours, and HAProxy served it for
# brainjar.net instead of the valid brainjar.net bundle next to it.
cert_is_expired() {
local file="$1" out
[ -f "$file" ] || return 1
command -v openssl >/dev/null 2>&1 || return 1
out="$(openssl x509 -checkend 0 -noout -in "$file" 2>/dev/null </dev/null)"
[ $? -eq 1 ] && [[ "$out" == *"will expire"* ]]
}
# cert_not_after CERT_FILE - the notAfter date string, or empty.
cert_not_after() {
openssl x509 -enddate -noout -in "$1" 2>/dev/null </dev/null | sed 's/^notAfter=//'
}
cert_retired_dir() {
if [ -n "${CERT_RETIRED_DIR:-}" ]; then
echo "$CERT_RETIRED_DIR"
else
echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-retired"
fi
}
# cert_retire_expired_pems
#
# Move every EXPIRED *.pem out of $SSL_CERTS_DIR into cert-retired/ (a sibling,
# never inside the crt dir). Sets CERT_RETIRED_EXPIRED to the number moved.
#
# Not publishing expired lineages stops the re-publish loop, but a pem that was
# published while valid and has expired since stays in the crt dir and keeps
# shadowing - that was exactly brain-jar.com.pem (review M2). An expired
# certificate is never a valid certificate for anyone, so it can always go: the
# only-valid-cert safety rule cannot be violated by moving it. Two exceptions:
# * default_self_signed_cert.pem - HAProxy's fallback, regenerated separately;
# * the LAST .pem in the directory - a crt directory with no certificate at
# all fails the whole :443 bind, which is worse than serving an expired one.
# Nothing is deleted; reverse with mv.
cert_retire_expired_pems() {
local dir="${SSL_CERTS_DIR:-/etc/haproxy/certs}" retired f base dest remaining
CERT_RETIRED_EXPIRED=0
retired="$(cert_retired_dir)"
for f in "$dir"/*.pem; do
[ -f "$f" ] || continue
base="$(basename "$f")"
[ "$base" = "default_self_signed_cert.pem" ] && continue
cert_is_expired "$f" || continue
remaining="$(find "$dir" -maxdepth 1 -type f -name '*.pem' | wc -l)"
if [ "$remaining" -le 1 ]; then
log_error "Expired certificate $f is the only certificate left in $dir - NOT retiring it (an empty crt directory fails the :443 bind)"
continue
fi
if ! mkdir -p "$retired"; then
log_error "cannot create $retired - expired $f left in place"
continue
fi
dest="$retired/$base"
[ -e "$dest" ] && dest="$dest.$(date -u +%Y%m%dT%H%M%SZ)"
if mv -- "$f" "$dest"; then
log_error "Retired EXPIRED certificate $base (notAfter $(cert_not_after "$dest")) from $dir to $dest - it could only shadow valid certificates"
CERT_RETIRED_EXPIRED=$((CERT_RETIRED_EXPIRED + 1))
else
log_error "failed to move expired $f to $dest"
fi
done
}
# cert_publish CERT_FILE KEY_FILE DEST_FILE # cert_publish CERT_FILE KEY_FILE DEST_FILE
# #
# Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a # Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
@@ -227,6 +303,11 @@ cert_publish() {
log_error "cert_publish: private key $key_file is missing or empty" log_error "cert_publish: private key $key_file is missing or empty"
return 1 return 1
fi fi
# (a2) never publish an expired certificate (see cert_is_expired).
if cert_is_expired "$cert_file"; then
log_error "cert_publish: certificate $cert_file expired ($(cert_not_after "$cert_file")) - refusing to publish $dest_file"
return 1
fi
# (b) assemble in the staging dir - NOT in the certs dir, which HAProxy # (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
# scans wholesale. # scans wholesale.
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
# LOCAL end-to-end rehearsal of the build-push-coraza.yaml multi-arch logic
# against THROWAWAY registries. Never touches repo.anhonesthost.net / ghcr.io.
#
# docker run -d --name e2ereg1 -p 127.0.0.1:5077:5000 registry:2
# docker run -d --name e2ereg2 -p 127.0.0.1:5078:5000 registry:2
# REG1=127.0.0.1:5077 REG2=127.0.0.1:5078 scripts/e2e-coraza-multiarch.sh
# (then: docker rm -f e2ereg1 e2ereg2; docker buildx rm e2ebuilder)
#
# If the Docker daemon is not on this machine (remote/VM daemon), publish the
# registries on an address both sides reach (e.g. -p 172.17.0.1:5077:5000) and
# set REG1/REG2 to it; plain-HTTP is enabled for them in the builder config.
# Scenario 1: both platforms build, pushed by digest to TWO registries,
# index assembled; checks both platforms + attestations present.
# Scenario 2: arm64 forced to fail (bad upstream tag) while a STALE arm64
# digest from scenario 1 is still known: the tag must become
# amd64-only and must not contain the stale arm64 digest.
set -euo pipefail
cd "$(dirname "$0")/.."
REG1=${REG1:?set REG1}; REG2=${REG2:?set REG2}
B=${BUILDER:-e2ebuilder}
R1=$REG1/cloud-hosting-platform/coraza-spoa
R2=$REG2/shadowdao/coraza-spoa
if ! docker buildx inspect "$B" >/dev/null 2>&1; then
cfg=$(mktemp)
printf '[registry."%s"]\n http=true\n insecure=true\n[registry."%s"]\n http=true\n insecure=true\n' "$REG1" "$REG2" > "$cfg"
docker buildx create --name "$B" --driver docker-container --driver-opt network=host --buildkitd-config "$cfg" --bootstrap >/dev/null
fi
OUT="type=image,\"name=$R1,$R2\",push-by-digest=true,name-canonical=true,push=true"
build() { # build <arch> [extra args]; prints digest
local arch=$1; shift; local meta; meta=$(mktemp)
docker buildx build --builder "$B" --platform "linux/$arch" --output "$OUT" --metadata-file "$meta" "$@" ./coraza-spoa >&2
python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["containerimage.digest"])' "$meta"
}
inspect() { docker buildx imagetools inspect --builder "$B" "$@"; }
echo "=== scenario 1: both platforms ==="
AMD=$(build amd64); ARM=$(build arm64)
echo "amd64=$AMD arm64=$ARM"
IMAGETOOLS="docker buildx imagetools" scripts/assemble-multiarch-index.sh --builder "$B" \
--amd64 "$AMD" --arm64 "$ARM" --arm64-outcome success --repo "$R1" --repo "$R2" --tag latest
for r in "$R1" "$R2"; do echo "--- $r:latest"; inspect "$r:latest" | grep -E 'Digest:|Platform:'; done
echo "=== scenario 2: forced arm64 failure, stale arm64 digest ($ARM) still in hand ==="
if NEWARM=$(build arm64 --build-arg CORAZA_SPOA_VERSION=v0.0.0-does-not-exist 2>/dev/null); then echo "BUG: forced failure succeeded"; exit 1; fi
echo "arm64 step failed as forced"
scripts/assemble-multiarch-index.sh --builder "$B" \
--amd64 "$AMD" --arm64 "$ARM" --arm64-outcome failure --repo "$R1" --repo "$R2" --tag latest
for r in "$R1" "$R2"; do
echo "--- $r:latest"; inspect "$r:latest" | grep -E 'Digest:|Platform:'
if inspect "$r:latest" | grep -q 'linux/arm64'; then echo "FAIL: arm64 still attached"; exit 1; fi
if inspect "$r:latest" --raw | grep -q "$ARM"; then echo "FAIL: stale arm64 digest present"; exit 1; fi
done
echo "E2E OK"
+12
View File
@@ -34,6 +34,18 @@ def main():
haproxy_manager.logger.warning( haproxy_manager.logger.warning(
"[haproxy-supervisor] haproxy process not found — attempting in-place restart" "[haproxy-supervisor] haproxy process not found — attempting in-place restart"
) )
# Breadcrumb for the monitoring pipeline: the last thing HAProxy itself said
# before it vanished (the relay forwards the same text; this ties it to the
# supervisor event even if the relay is behind).
try:
with open(haproxy_manager._stderr_log_path(), 'rb') as fh:
fh.seek(0, 2)
fh.seek(max(0, fh.tell() - 1500))
tail = fh.read().decode('utf-8', 'replace').strip().splitlines()[-6:]
for line in tail:
haproxy_manager.logger.warning("[haproxy-supervisor] last haproxy stderr: %s", line)
except OSError:
pass
# start_haproxy() validates the config (and regenerates it if invalid) # start_haproxy() validates the config (and regenerates it if invalid)
# before launching, and swallows its own errors, so it will not raise here. # before launching, and swallows its own errors, so it will not raise here.
haproxy_manager.start_haproxy() haproxy_manager.start_haproxy()
+264
View File
@@ -0,0 +1,264 @@
#!/usr/bin/env python3
"""In-container receiver for the HAProxy access log.
HAProxy sends its access log (`log <socket> ... format rfc5424`, hap_header.tpl)
as datagrams to a UNIX socket inside this container. This process is the only
reader: it writes one line per datagram to a size-capped, rotating file in the
haproxy-logs volume (default /var/log/haproxy/access.log).
Why it looks the way it does
----------------------------
* UNIX DATAGRAM socket, no network listener. Nothing is exposed on the host
or on client-net (the old target was the host's rsyslog on the docker
bridge, 172.18.0.1:514, which had no listener on the fleet).
* HAProxy never waits for us. Its log sends are non-blocking: a dead receiver
(ECONNREFUSED/ENOENT) or a full socket buffer (EAGAIN) drops that line and
HAProxy carries on. So a crash, a stall or a full disk here can only lose
log lines, never slow or fail a request. That is the intended direction.
* Bounded on disk: the live file is rotated at MAX_BYTES and BACKUPS gzip
generations are kept, so the worst case (incompressible data) is
(1 + BACKUPS) * MAX_BYTES -- 5 * 40 MB = 200 MB by default. Compression runs
in a thread so a rotation does not stall reading.
* A full disk (ENOSPC) or any write error drops lines and is reported at most
once a minute; the file is reopened on the next attempt.
* Lines go to the FILE ONLY, never stdout: `docker logs haproxy-manager` is a
source of WHP's AI log monitor, and a per-request firehose there would storm
it (see the WHP note on audit/firehose sources). Our own status messages are
rare and go to stderr.
* Control characters inside a datagram are replaced so one datagram is always
exactly one line (a crafted header cannot forge extra log lines).
Supervised by start-up.sh (restart loop), started before HAProxy.
Environment
-----------
HAPROXY_LOG_SOCKET socket path (default /var/lib/haproxy/log.sock)
HAPROXY_ACCESS_LOG file path (default /var/log/haproxy/access.log)
HAPROXY_ACCESS_LOG_MAX_BYTES rotate size (default 41943040 = 40 MiB)
HAPROXY_ACCESS_LOG_BACKUPS gzip generations kept (default 4)
"""
import gzip
import os
import shutil
import socket
import sys
import threading
import time
# HAProxy sends at most `len` (2048) bytes per line; leave headroom.
MAX_DATAGRAM = 65535
FLUSH_INTERVAL = 1.0
ERROR_REPORT_INTERVAL = 60.0
# Kernel receive buffer to absorb bursts while a write is slow.
RCVBUF = 4 * 1024 * 1024
_CTRL = {c: ' ' for c in range(0x20) if c != 0x09}
_CTRL[0x7f] = ' '
def _env_int(name, default, minimum):
try:
value = int(os.environ.get(name, default))
except ValueError:
value = default
return max(minimum, value)
def clean_line(datagram):
"""One datagram -> one log line (str, no trailing newline).
Drops the RFC 5424 `<PRI>` prefix (constant for our single facility) and
replaces any control character so the datagram cannot span lines.
"""
text = datagram.decode('utf-8', 'replace').rstrip('\r\n\x00')
if text.startswith('<'):
end = text.find('>', 1, 6)
if end > 0 and text[1:end].isdigit():
text = text[end + 1:]
return text.translate(_CTRL)
class RotatingWriter:
"""Size-capped writer: live file + `backups` gzip generations."""
def __init__(self, path, max_bytes, backups, report=None):
self.path = path
self.max_bytes = max_bytes
self.backups = backups
self.report = report or (lambda msg: None)
self.fh = None
self.size = 0
self.dirty = False
self.dropped = 0
self._compressor = None
self._last_error = 0.0
# -- file handling --------------------------------------------------
def _open(self):
os.makedirs(os.path.dirname(self.path) or '.', exist_ok=True)
self.fh = open(self.path, 'a', encoding='utf-8', buffering=65536)
os.chmod(self.path, 0o640)
self.size = self.fh.tell()
def _error(self, what, exc):
self.dropped += 1
try:
if self.fh:
self.fh.close()
except OSError:
pass
self.fh = None
now = time.monotonic()
if now - self._last_error >= ERROR_REPORT_INTERVAL:
self._last_error = now
self.report(f'{what} failed ({exc}); dropping access-log lines '
f'(dropped so far: {self.dropped})')
def _gen(self, n):
return f'{self.path}.{n}.gz'
def _compress(self, src, dst):
tmp = dst + '.tmp'
try:
with open(src, 'rb') as fin, gzip.open(tmp, 'wb', compresslevel=6) as fout:
shutil.copyfileobj(fin, fout, 1024 * 1024)
os.replace(tmp, dst)
os.chmod(dst, 0o640)
except OSError as exc:
self.report(f'compressing {src} failed ({exc}); removing it to keep the size cap')
for leftover in (tmp,):
try:
os.unlink(leftover)
except OSError:
pass
finally:
try:
os.unlink(src)
except OSError:
pass
def rotate(self):
if self._compressor is not None:
self._compressor.join()
self._compressor = None
if self.fh:
try:
self.fh.close()
except OSError:
pass
self.fh = None
if self.backups <= 0:
try:
os.unlink(self.path)
except OSError:
pass
else:
try:
os.unlink(self._gen(self.backups))
except OSError:
pass
for n in range(self.backups - 1, 0, -1):
if os.path.exists(self._gen(n)):
os.replace(self._gen(n), self._gen(n + 1))
staged = f'{self.path}.1'
if os.path.exists(self.path):
os.replace(self.path, staged)
self._compressor = threading.Thread(
target=self._compress, args=(staged, self._gen(1)), daemon=True)
self._compressor.start()
self.size = 0
# -- public ---------------------------------------------------------
def write(self, line):
data = line + '\n'
try:
if self.fh is None:
self._open()
if self.size and self.size + len(data) > self.max_bytes:
self.rotate()
self._open()
self.fh.write(data)
self.size += len(data.encode('utf-8'))
self.dirty = True
except OSError as exc:
self._error('writing ' + self.path, exc)
def flush(self):
if self.fh and self.dirty:
try:
self.fh.flush()
self.dirty = False
except OSError as exc:
self._error('flushing ' + self.path, exc)
def close(self):
self.flush()
if self._compressor is not None:
self._compressor.join()
if self.fh:
try:
self.fh.close()
except OSError:
pass
def bind_socket(path):
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
try:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, RCVBUF)
except OSError:
pass
os.makedirs(os.path.dirname(path) or '.', exist_ok=True)
try:
os.unlink(path) # stale socket from a previous (crashed) receiver
except FileNotFoundError:
pass
sock.bind(path)
# HAProxy's workers run as user haproxy (hap_header.tpl `user haproxy`).
os.chmod(path, 0o666)
sock.settimeout(FLUSH_INTERVAL)
return sock
def serve(sock, writer, stop=None):
last_flush = time.monotonic()
while stop is None or not stop.is_set():
try:
datagram = sock.recv(MAX_DATAGRAM)
except socket.timeout:
datagram = None
except OSError:
if stop is not None and stop.is_set():
break
raise
if datagram:
writer.write(clean_line(datagram))
now = time.monotonic()
if now - last_flush >= FLUSH_INTERVAL:
writer.flush()
last_flush = now
def main():
sock_path = os.environ.get('HAPROXY_LOG_SOCKET', '/var/lib/haproxy/log.sock')
log_path = os.environ.get('HAPROXY_ACCESS_LOG', '/var/log/haproxy/access.log')
max_bytes = _env_int('HAPROXY_ACCESS_LOG_MAX_BYTES', 40 * 1024 * 1024, 64 * 1024)
backups = _env_int('HAPROXY_ACCESS_LOG_BACKUPS', 4, 0)
def report(msg):
sys.stderr.write(f'haproxy-log-receiver: {msg}\n')
sys.stderr.flush()
writer = RotatingWriter(log_path, max_bytes, backups, report)
sock = bind_socket(sock_path)
report(f'listening on {sock_path} -> {log_path} '
f'(rotate at {max_bytes} bytes, keep {backups} gz)')
try:
serve(sock, writer)
finally:
writer.close()
if __name__ == '__main__':
main()
+363
View File
@@ -0,0 +1,363 @@
#!/usr/bin/env python3
"""Forward HAProxy's own stderr to the container's stderr, and bound the file.
Why
---
HAProxy's master stderr carries what the access log never will: `[ALERT]`
lines, "Current worker (N) exited with code 134", and the complete
ha_panic()/watchdog thread dump of a crash. haproxy_manager.spawn_haproxy_master()
launches `haproxy -W -db` with that stderr appended to HAPROXY_STDERR_LOG (in the
haproxy-logs volume, so it survives a container recreate). This process tails the
file and re-emits each line on ITS stderr, prefixed, which is the container's
stderr: `docker logs`, and through the docker logging driver the host journal
and Graylog.
Properties
----------
* Rare by nature: HAProxy only writes here on start/reload/server state
changes/crashes. The per-request access log is NOT here (it goes to the
receiver socket), so this does not feed the WHP AI monitor a firehose.
* Crash evidence bypasses the rate limit: lines that match the ANCHORED CRASH_RE
("[ALERT] (PID) :", "Current worker (N) exited with code N", "Thread N is
about to kill the process", "Thread N : id=0x..."; a keyword inside a URL/Host/
server name never matches) and every line of a dump block (from a panic/watchdog header to
the master's "All workers exited" line, bounded by
DUMP_MAX_LINES / DUMP_MAX_SECONDS, then a DUMP_COOLDOWN before another block
can open; blocks open only on the anchored "Thread N is about to kill" header)
are forwarded in full even when the bucket is empty, within a total budget of
CRASH_BUDGET lines per CRASH_WINDOW (after that they take the normal limiter
and are counted in the suppressed summary). They go to STDERR.
* Tenant-driven state lines (TENANT_RE: "backend 'b' has no server available!",
"Server b/s is DOWN|UP ...") are NOT crash evidence even though HAProxy prints
the first at [ALERT]: every tenant container stop/crash/deploy makes them.
Own rate-limited STDOUT class (own bucket TENANT_BURST/TENANT_PER_MIN, own
"[relay] suppressed N tenant-driven ..." summary). They also reach the WHP AI
monitor through `docker logs haproxy-manager`; rate-limiting them is intended.
Dump blocks have their own budget (DUMP_BUDGET) that nothing else can spend.
* Routine reload noise (ROUTINE_RE: "Proxy X stopped (cumulated conns ...)",
config "parsing [...]" warnings, "Former worker (N) exited with code 0",
"Initializing new worker", "Loading success", ...) never touches the bucket:
it is counted and collapsed into ONE "[relay] reload/start ok: N routine
lines collapsed" line per reload (plus one on the minute tick for the late
"Proxy stopped" lines). The full text stays in the stderr file.
* Everything else is rate-limited (token bucket, HAPROXY_STDERR_RELAY_BURST
lines burst, HAPROXY_STDERR_RELAY_PER_MIN refill) with a per-minute
"[relay] suppressed N lines" summary.
* Priority: the docker journald driver maps stdout -> PRIORITY 6 (info) and
stderr -> PRIORITY 3 (err) and does NOT parse "<N>" prefixes. So benign
lines (routine, rate-limited remainder, relay summaries) are written to
STDOUT and crash evidence to STDERR: Graylog/journal can alert on priority.
* Bounded file: copy-truncate rotation at HAPROXY_STDERR_LOG_MAX_BYTES
(default 20 MiB), HAPROXY_STDERR_LOG_BACKUPS generations (default 3).
Truncating is safe because HAProxy's fd is O_APPEND.
* Never blocks or affects HAProxy: it only reads a file.
Environment
-----------
HAPROXY_STDERR_LOG, HAPROXY_STDERR_RELAY_BURST (400), HAPROXY_STDERR_RELAY_PER_MIN (120),
HAPROXY_STDERR_LOG_MAX_BYTES (20971520), HAPROXY_STDERR_LOG_BACKUPS (3),
HAPROXY_STDERR_RELAY_POLL (0.5 s)
"""
import os
import re
import shutil
import sys
import time
PREFIX = 'haproxy-stderr: '
MAX_LINE = 2000 # longer lines are truncated (hostile / runaway output)
MAX_PENDING = 65536 # an unterminated line is force-emitted past this size
DUMP_MAX_LINES = 500 # a dump block never bypasses the limiter longer than this
DUMP_MAX_SECONDS = 30.0
SUMMARY_EVERY = 60.0
# Every pattern is ANCHORED to HAProxy's own message structure (captured from
# real 3.2 crashes): a keyword inside a request path / Host header / server
# name that HAProxy echoes into a warning can never match, because the line
# prefix "[ALERT] (PID) :" is written by HAProxy itself.
# Crash evidence: forwarded (to stderr) without the bucket, within CRASH_BUDGET.
CRASH_RE = re.compile(
r'^(?:'
r'\[ALERT\]\s+\(\d+\)\s*:\s*(?:Current worker \(\d+\) exited with code|exit-on-failure:|A worker process unexpectedly died)'
r'|\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*(?:Current|Former) worker \(\d+\) exited with code (?!0\b)\d+'
r'|\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*A worker process unexpectedly died'
r'|\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*All workers exited\b'
r'|Thread \d+ is about to kill the process'
r'|\s*\*?\s*Thread \d+ : id=0x[0-9a-f]+ '
r')')
# Tenant-driven state lines. WHP gives every tenant server `check`, so every
# tenant container stop/crash/OOM/deploy and every host-wide container restart
# makes HAProxy print these at runtime (at ALERT/WARNING level). They are NOT crash
# evidence: own RATE-LIMITED class on STDOUT with their own bucket and summary, so
# they can neither reach priority 3 nor spend the general bucket / crash budgets.
TENANT_RE = re.compile(
r'^\[(?:ALERT|WARNING|NOTICE)\]\s+\(\d+\)\s*:\s*(?:'
r"backend '[^']*' has no server available!"
r'|Server \S+ (?:is (?:DOWN|UP|going DOWN|stopping|NOLB)\b|was DOWN\b)'
r'|Health check for (?:backup )?server \S+ '
r')')
# A dump block opens ONLY on the watchdog/panic header, and runs until DUMP_END_RE.
DUMP_START_RE = re.compile(r'^Thread \d+ is about to kill the process')
DUMP_END_RE = re.compile(r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*All workers exited\b')
# A new start/reload cycle also closes a dump block.
CYCLE_RE = re.compile(r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*(?:Reloading HAProxy|Initializing new worker|Loading success)')
# Routine per-start/reload noise: counted and collapsed, never rate-limited.
ROUTINE_RE = re.compile(
r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*(?:'
r'Proxy \S+ stopped \(cumulated conns'
r'|config\s*:\s*parsing \[[^\]]*\]\s*:'
r'|Former worker \(\d+\) exited with code 0\b'
r'|Reloading HAProxy|Initializing new worker|Loading success'
r'|path to executable is|haproxy version is'
r"|config: Can't load stats-file|'option forwardfor' ignored)")
LOADED_RE = re.compile(r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*Loading success')
# Total exemption budget: even real-looking evidence cannot flood without bound.
CRASH_BUDGET = 1000 # exempt lines per window (a 4-thread dump is ~70)
CRASH_WINDOW = 600.0 # seconds
DUMP_BUDGET = 1000 # dump-block lines (header + body) per window: a SEPARATE budget that
# ordinary evidence lines (worker exits, alerts) can never spend
TENANT_BURST = 100 # tenant-driven state lines: own bucket (burst / refill per minute)
TENANT_PER_MIN = 30
DUMP_COOLDOWN = 60.0 # after a BOUND closes a block, no new block opens for this long
def _env_num(name, default, cast=int):
try:
v = cast(os.environ.get(name, default))
return v if v > 0 else default
except (TypeError, ValueError):
return default
class Relay:
def __init__(self, path, out, burst, per_min, max_bytes, backups, clock=time.monotonic, err=None):
self.path = path
self.out = out # benign lines (docker journald: PRIORITY 6)
self.err = err if err is not None else out # crash lines (PRIORITY 3)
self.burst, self.per_min = burst, per_min
self.max_bytes, self.backups = max_bytes, backups
self.clock = clock
self.tokens = float(burst)
self.last = clock()
self.suppressed = 0
self.routine = 0 # collapsed routine lines since the last summary
self.summary_at = clock()
self.in_dump = False
self.dump_lines = 0
self.dump_since = 0.0
self.cooldown_until = 0.0
self.budget_left = CRASH_BUDGET
self.window_start = clock()
self.dump_budget_left = DUMP_BUDGET
self.dump_window_start = clock()
self.tenant_tokens = float(TENANT_BURST)
self.tenant_last = clock()
self.tenant_suppressed = 0
self.offset = None # None = first pass: start at EOF (do not replay history)
self.pending = ''
self._last_stream = None
def _take(self):
now = self.clock()
self.tokens = min(self.burst, self.tokens + (now - self.last) * self.per_min / 60.0)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
def _take_tenant(self):
now = self.clock()
self.tenant_tokens = min(TENANT_BURST, self.tenant_tokens + (now - self.tenant_last) * TENANT_PER_MIN / 60.0)
self.tenant_last = now
if self.tenant_tokens >= 1:
self.tenant_tokens -= 1
return True
return False
def _write(self, stream, text):
# keep the two streams in emission order: flush the other one first
if self._last_stream is not None and self._last_stream is not stream:
self._last_stream.flush()
stream.write(PREFIX + text + '\n')
self._last_stream = stream
def _flush_all(self):
for st in {id(self.out): self.out, id(self.err): self.err}.values():
try:
st.flush()
except Exception:
pass
def _summaries(self, force=False):
"""Per-minute summary of what the limiter / collapse withheld."""
now = self.clock()
if not force and now - self.summary_at < SUMMARY_EVERY:
return
self.summary_at = now
if self.routine:
self._write(self.out, '[relay] %d routine reload/start lines collapsed (full text in the stderr file)'
% self.routine)
self.routine = 0
if self.tenant_suppressed:
self._write(self.out, '[relay] suppressed %d tenant-driven server/backend state lines '
'(rate limit; full text in the stderr file)' % self.tenant_suppressed)
self.tenant_suppressed = 0
if self.suppressed:
self._write(self.out, '[relay] suppressed %d lines (rate limit; full text in the stderr file)'
% self.suppressed)
self.suppressed = 0
def _budget(self, now, kind='crash'):
"""Take one unit of a budget; False once the window's budget is spent.
kind 'crash': ordinary evidence lines; kind 'dump': dump-block lines (own budget)."""
if kind == 'dump':
if now - self.dump_window_start >= CRASH_WINDOW:
self.dump_window_start, self.dump_budget_left = now, DUMP_BUDGET
if self.dump_budget_left > 0:
self.dump_budget_left -= 1
return True
return False
if now - self.window_start >= CRASH_WINDOW:
self.window_start, self.budget_left = now, CRASH_BUDGET
if self.budget_left > 0:
self.budget_left -= 1
return True
return False
def _end_dump(self, now, why=None):
self.in_dump = False
if why: # closed by a bound, not by the master's end line: no re-arming for a while
self.cooldown_until = now + DUMP_COOLDOWN
self._write(self.err, '[relay] dump block closed: %s; rate limit applies again' % why)
def _emit(self, line):
now = self.clock()
if self.in_dump:
if (self.dump_lines >= DUMP_MAX_LINES) or (now - self.dump_since > DUMP_MAX_SECONDS):
self._end_dump(now, 'bound reached (%d lines / %.0fs)' % (self.dump_lines, now - self.dump_since))
elif CYCLE_RE.match(line):
self._end_dump(now)
if self.in_dump or CRASH_RE.match(line):
is_dump = self.in_dump or bool(DUMP_START_RE.match(line))
kind = 'dump' if is_dump else 'crash'
if self._budget(now, kind):
# crash evidence: bypasses the bucket, the collapse and the summary
if not self.in_dump and DUMP_START_RE.match(line) and now >= self.cooldown_until:
self.in_dump, self.dump_lines, self.dump_since = True, 0, now
if self.in_dump:
self.dump_lines += 1
self._write(self.err, line)
if self.in_dump and DUMP_END_RE.match(line):
self._end_dump(now)
return
if self.in_dump: # budget spent mid-block: the normal limiter takes over
self._end_dump(now, 'crash-evidence budget (%d lines/%.0fs) spent' % (CRASH_BUDGET, CRASH_WINDOW))
if ROUTINE_RE.match(line):
self.routine += 1
if LOADED_RE.match(line): # one summary per start/reload
self._write(self.out, '[relay] reload/start ok: %d routine lines collapsed (full text in the stderr file)'
% self.routine)
self.routine = 0
self._summaries()
return
if TENANT_RE.match(line):
if self._take_tenant():
self._write(self.out, line)
else:
self.tenant_suppressed += 1
self._summaries()
return
if self._take():
self._write(self.out, line)
else:
self.suppressed += 1
self._summaries()
@staticmethod
def _clean(line):
line = ''.join(c if c >= ' ' or c == '\t' else ' ' for c in line).rstrip()
if len(line) > MAX_LINE:
line = line[:MAX_LINE] + ' [relay: line truncated, %d chars]' % len(line)
return line
def rotate_if_needed(self):
try:
if os.path.getsize(self.path) <= self.max_bytes:
return
for i in range(self.backups, 1, -1):
src = '%s.%d' % (self.path, i - 1)
if os.path.exists(src):
os.replace(src, '%s.%d' % (self.path, i))
shutil.copyfile(self.path, self.path + '.1')
os.chmod(self.path + '.1', 0o640)
with open(self.path, 'r+b') as fh:
fh.truncate(0)
self.offset = 0
except OSError as exc:
sys.stderr.write('haproxy-stderr-relay: rotate failed: %s\n' % exc)
def poll_once(self):
try:
size = os.path.getsize(self.path)
except OSError:
if self.offset is None:
self.offset = 0 # file not created yet: whatever appears is new
return
if self.offset is None:
self.offset = size
return
if size < self.offset: # truncated by someone else
self.offset = 0
if size == self.offset:
return
with open(self.path, 'rb') as fh:
fh.seek(self.offset)
chunk = fh.read(1 << 20)
self.offset += len(chunk)
text = self.pending + chunk.decode('utf-8', 'replace')
lines = text.split('\n')
self.pending = lines.pop() # incomplete trailing line stays buffered
if len(self.pending) > MAX_PENDING: # hostile / runaway unterminated line
lines.append(self.pending)
self.pending = ''
for line in lines:
line = self._clean(line)
if line:
self._emit(line)
self._summaries()
self._flush_all()
def tick(self):
"""Idle-time housekeeping: emit the per-minute summaries even when no line arrives."""
self._summaries()
self._flush_all()
def main():
path = os.environ.get('HAPROXY_STDERR_LOG', '/var/log/haproxy/haproxy-stderr.log')
relay = Relay(path, sys.stdout,
_env_num('HAPROXY_STDERR_RELAY_BURST', 400),
_env_num('HAPROXY_STDERR_RELAY_PER_MIN', 120),
_env_num('HAPROXY_STDERR_LOG_MAX_BYTES', 20 * 1024 * 1024),
_env_num('HAPROXY_STDERR_LOG_BACKUPS', 3),
err=sys.stderr)
poll = _env_num('HAPROXY_STDERR_RELAY_POLL', 0.5, float)
sys.stdout.write('haproxy-stderr-relay: following %s\n' % path)
sys.stdout.flush()
ticks = 0
while True:
try:
relay.poll_once()
relay.tick()
ticks += 1
if ticks % 120 == 0:
relay.rotate_if_needed()
except Exception as exc: # never die: the supervisor loop restarts us anyway
sys.stderr.write('haproxy-stderr-relay: %r\n' % exc)
time.sleep(poll)
if __name__ == '__main__':
main()
+92
View File
@@ -0,0 +1,92 @@
#!/usr/bin/env bash
# In-container haproxy supervisor (started in the background by start-up.sh).
#
# haproxy runs as a background child of PID 1 (gunicorn) with NOTHING watching
# it after init. If the haproxy master dies mid-life (e.g. SIGABRT -> exit 134,
# segfault), the container stays "up" (gunicorn is PID 1), Docker's --restart
# policy never fires, and haproxy is down until the external host watchdog
# full-restarts the whole container minutes later (dropping every connection).
# This loop revives haproxy in place within one interval.
#
# The liveness probe is `pidof` (a few ms), NOT python: ensure_haproxy.py imports
# the 4000-line manager (Flask, Jinja, ...) which costs ~0.5 s of CPU per run and
# was affordable at 15 s but not at 2 s. python only runs when haproxy is gone.
# 2026-11 evidence: every watchdog abort took the whole edge down for 15-52 s,
# most of it detection latency + start; the default is now 2 s (was 15).
#
# Two backoffs:
# * a restart attempt that FAILS (config HAProxy rejects): the loop doubles its
# delay up to HAPROXY_SUPERVISOR_MAX_BACKOFF (default 30 s) instead of
# re-rendering/re-validating the same broken config every 2 s;
# * haproxy that starts fine but DIES again within HAPROXY_SUPERVISOR_QUICK_DEATH
# seconds (default 10) of that start (a crash loop): extra wait before the
# next restart, stepping through HAPROXY_SUPERVISOR_BACKOFF_STEPS (default
# "2 5 10 30", each capped at MAX_BACKOFF). A run that survives
# HAPROXY_SUPERVISOR_STABLE seconds (default 60) resets it. The first death
# after a long run is still revived within one interval.
#
# All knobs are validated as positive INTEGERS (seconds); anything else (e.g. 0.5,
# "abc") falls back to the default instead of killing this subshell under set -e.
set -eo pipefail
_posint() { # _posint VALUE DEFAULT
case "$1" in
''|*[!0-9]*) echo "$2" ;;
*) if [ "$1" -ge 1 ]; then echo "$1"; else echo "$2"; fi ;;
esac
}
INTERVAL="$(_posint "${HAPROXY_SUPERVISOR_INTERVAL:-}" 2)"
MAX_BACKOFF="$(_posint "${HAPROXY_SUPERVISOR_MAX_BACKOFF:-}" 30)"
QUICK_DEATH="$(_posint "${HAPROXY_SUPERVISOR_QUICK_DEATH:-}" 10)"
STABLE="$(_posint "${HAPROXY_SUPERVISOR_STABLE:-}" 60)"
STEPS=""
for s in ${HAPROXY_SUPERVISOR_BACKOFF_STEPS:-2 5 10 30}; do
v="$(_posint "$s" 0)"
[ "$v" -ge 1 ] && STEPS="${STEPS} ${v}"
done
[ -n "${STEPS}" ] || STEPS=" 2 5 10 30"
# step N (1-based, past the end = last), capped at MAX_BACKOFF
_step() {
local n="$1" i=0 v=0
for s in ${STEPS}; do
i=$((i + 1)); v="$s"
[ "${i}" -ge "${n}" ] && break
done
[ "${v}" -gt "${MAX_BACKOFF}" ] && v="${MAX_BACKOFF}"
echo "${v}"
}
delay="${INTERVAL}"
last_start=-1 # SECONDS at our last successful (re)start; -1 = none yet
quick=0 # consecutive deaths within QUICK_DEATH of a start
while true; do
sleep "${delay}"
if pidof -q haproxy; then
delay="${INTERVAL}"
if [ "${last_start}" -ge 0 ] && [ $((SECONDS - last_start)) -ge "${STABLE}" ]; then
quick=0; last_start=-1
fi
continue
fi
if [ "${last_start}" -ge 0 ]; then
up=$((SECONDS - last_start))
if [ "${up}" -lt "${QUICK_DEATH}" ]; then
quick=$((quick + 1))
wait_s="$(_step "${quick}")"
echo "haproxy-supervisor: haproxy died ${up}s after start (quick death #${quick}); waiting ${wait_s}s before restart" >&2
sleep "${wait_s}"
pidof -q haproxy && continue
elif [ "${up}" -ge "${STABLE}" ]; then
quick=0
fi
fi
if python /haproxy/scripts/ensure_haproxy.py 2>&1; then
delay="${INTERVAL}"
last_start="${SECONDS}"
else
delay=$(( delay * 2 ))
[ "${delay}" -gt "${MAX_BACKOFF}" ] && delay="${MAX_BACKOFF}"
fi
done
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Container liveness probe: BOTH planes must answer quickly.
#
# 1. Management API (:8000) -- GET /health with a short --max-time. On
# 2026-10-09 (sdbees) the API workers' thread pools were wedged for long
# enough that /health hung 40s+, while the container reported "healthy"
# because the deployed --health-cmd only probed :80. A hung API is a dead
# control plane (every WHP site update times out), so it must turn the
# container unhealthy.
# 2. HAProxy data plane (:80) -- any HTTP answer at all (the default page,
# a redirect, a 4xx/5xx) proves HAProxy accepted and answered. Only a
# connect/timeout failure fails it, which is why there is no -f here.
#
# 127.0.0.1, not localhost: the API binds IPv4 only, and resolving localhost
# to ::1 first costs a failed connect on every probe.
#
# WHP's lib-edge-container.sh overrides the image HEALTHCHECK with its own
# --health-cmd; point that at this script so the fleet gets the same probe.
set -u
API_TIMEOUT="${HAPROXY_MGR_HEALTH_API_TIMEOUT:-4}"
EDGE_TIMEOUT="${HAPROXY_MGR_HEALTH_EDGE_TIMEOUT:-5}"
if ! curl -sf -o /dev/null --max-time "$API_TIMEOUT" http://127.0.0.1:8000/health; then
echo "management API :8000 /health failed or took longer than ${API_TIMEOUT}s"
exit 1
fi
if ! curl -s -o /dev/null --max-time "$EDGE_TIMEOUT" http://127.0.0.1/; then
echo "HAProxy :80 did not answer within ${EDGE_TIMEOUT}s"
exit 1
fi
exit 0
+7 -7
View File
@@ -20,13 +20,13 @@
# against. # against.
# #
# Usage: monitor-attacks.sh [live] # Usage: monitor-attacks.sh [live]
# Env: LOG_FILE=<path> access log to read (default /var/log/haproxy.log) # Env: LOG_FILE=<path> access log to read (default /var/log/haproxy/access.log)
# LOG_LINES=<n> how many trailing log lines to scan (default 500) # LOG_LINES=<n> how many trailing log lines to scan (default 500)
set -uo pipefail set -uo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
LOG_FILE="${LOG_FILE:-/var/log/haproxy.log}" LOG_FILE="${LOG_FILE:-/var/log/haproxy/access.log}"
LOG_LINES="${LOG_LINES:-500}" LOG_LINES="${LOG_LINES:-500}"
# --- Section 1: current rates (real stick-table data) ----------------------- # --- Section 1: current rates (real stick-table data) -----------------------
@@ -47,12 +47,12 @@ show_recent_blocks() {
syslog on the DOCKER HOST, and the file lives on the host, not in here. syslog on the DOCKER HOST, and the file lives on the host, not in here.
Read it from the host instead: Read it from the host instead:
grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20 # tarpit / deny grep -aE ' (PT|PR)--' /var/log/haproxy/access.log | tail -20 # tarpit / deny
grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20 # rate-limited / blocked grep -aE ' (429|403) ' /var/log/haproxy/access.log | tail -20 # rate-limited / blocked
grep -a 'cip=<IP>' /var/log/haproxy.log | tail -50 # one client IP grep -a 'cip=<IP>' /var/log/haproxy/access.log | tail -50 # one client IP
grep -a 'id=<uuid>' /var/log/haproxy.log # one request reference grep -a 'id=<uuid>' /var/log/haproxy/access.log # one request reference
# (the UUID on the block page) # (the UUID on the block page)
tail -f /var/log/haproxy.log | grep -aE ' (PT|PR)--' # live tail -f /var/log/haproxy/access.log | grep -aE ' (PT|PR)--' # live
Or point this script at a copy: LOG_FILE=/path/to/haproxy.log $0 Or point this script at a copy: LOG_FILE=/path/to/haproxy.log $0
MSG MSG
+296
View File
@@ -0,0 +1,296 @@
#!/bin/bash
## pkg-cache-repos.sh — BUILD-TIME ONLY, OPT-IN. Points dnf/apt at the internal
## package cache (nginx proxy_cache, CT130 "pkg-cache" on the Proxmox cluster)
## when PKG_CACHE is set; without it the repo files are left alone and builds
## use the upstream repositories as before.
##
## PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh rpm # AlmaLinux/EPEL/Remi/NodeSource
## PKG_CACHE=https://192.168.1.40 pkg-cache-repos.sh apt # Ubuntu, Debian, LiteSpeed
## pkg-cache-repos.sh release-install epel [remi] # GPG-verified release RPMs (EL9/EL10)
## pkg-cache-repos.sh nodesource MAJOR # NodeSource repo file + pinned key
## eval "$(pkg-cache-repos.sh go-env)" # GOPROXY via the cache (go.sum/sumdb verified)
## pkg-cache-repos.sh restore # undo everything above
##
## CANONICAL copy: cloud-apache-container/scripts/pkg-cache-repos.sh (docs/pkg-cache.md
## there). Other repos carry byte-identical copies - change it there first.
##
## Every RUN that rewrites must `restore` before it ends: it puts rewritten files
## back, deletes files this script created, deletes any NEW repo/apt file that
## still references the cache, then FAILS if any reference is left — so no shipped
## layer carries the cache's LAN address (images run on hosts that can't reach it).
##
## Why: on 2026-10-21 (PHP security release) builds failed on Remi mirrors lagging
## hours behind origin, Remi's origin timing out over HTTPS, and an EPEL mirror
## with a broken certificate. The cache fetches from the ORIGINS over HTTPS (with
## origin/mirror fallbacks and stale-on-error), keeps metadata ~3 minutes and
## immutable .rpm/.deb files long-term.
##
## Integrity (security review 2026-10-21):
## * PKG_CACHE must be https://. The cache's certificate is issued by the
## name-constrained "pkg-cache internal CA" (keys/pkg-cache/ca.crt, valid only
## for 192.168.1.40 / pkg-cache.*). That CA is trusted ONLY by the rewritten
## repos (dnf sslcacert= bundle, apt per-host CaInfo) and only during the RUN —
## never added to the system trust store.
## * gpgcheck=1 is forced on every rewritten repo; repo_gpgcheck=1 on Remi and
## AlmaLinux, which sign repomd.xml (measured: repomd.xml.asc 200). EPEL
## publishes no repomd signature (404), so its metadata relies on HTTPS end
## to end (client->cache->dl.fedoraproject.org/mirrors, all verified).
## * release-install downloads epel-release / remi-release (from the cache, or
## from the HTTPS origin without PKG_CACHE), checks the package name, imports
## the signing keys committed in keys/pkg-cache/ and requires
## `rpm -K` "signatures OK" before installing with localpkg_gpgcheck=1.
## * The php.net version gates (EXPECT_PHP_MIN / EXPECT_PHP_VERSION) still fail
## a build that gets old PHP (replay of old-but-signed metadata).
##
## rpm: each rewritten baseurl keeps the upstream origin as a SECOND url, so if
## the cache dies mid-build dnf fails over to the origin by itself.
## apt: apt has no per-source failover; CI only sets PKG_CACHE after probing the
## cache (with the CA), and a build without PKG_CACHE uses upstream sources.
set -euo pipefail
MODE="${1:?usage: pkg-cache-repos.sh rpm|apt|release-install NAME...|nodesource MAJOR|go-env|restore}"
shift || true
STATE=/var/lib/pkg-cache-repos # saved originals + manifest + created files
SHARE="${PKG_CACHE_SHARE:-/usr/local/share/pkg-cache}" # ca.crt + RPM-GPG-KEY-*
C="${PKG_CACHE:-}"
C="${C%/}"
die() { echo "FATAL(pkg-cache-repos): $*" >&2; exit 1; }
if [ -n "$C" ]; then
case "$C" in
https://*) ;;
*) die "PKG_CACHE must be an https:// URL (plain http is refused), got '$C'" ;;
esac
[[ "$C" =~ ^https://[A-Za-z0-9.-]+(:[0-9]+)?(/[A-Za-z0-9._/-]*)?$ ]] || die "PKG_CACHE has unexpected characters: '$C'"
HOSTPORT="${C#https://}"; HOSTPORT="${HOSTPORT%%/*}"
[ -s "$SHARE/ca.crt" ] || die "PKG_CACHE set but $SHARE/ca.crt (pkg-cache CA) is missing"
fi
# System CA bundle (EL vs Debian/Ubuntu path); the pkg-cache CA is appended to a
# copy in the state dir, never to the system store.
sys_bundle() {
local b
for b in /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt; do
[ -s "$b" ] && { echo "$b"; return 0; }
done
die "no system CA bundle found"
}
make_bundle() { cat "$(sys_bundle)" "$SHARE/ca.crt" > "$STATE/ca-bundle.pem"; }
# Files that may hold repo configuration; restore re-checks all of them.
repo_files() {
find /etc/yum.repos.d /etc/dnf /etc/apt -type f 2>/dev/null | sort || true
}
start_session() {
[ -d "$STATE" ] && return 0
mkdir -p "$STATE/orig"
repo_files > "$STATE/manifest" # what existed before this RUN touched anything
: > "$STATE/created"
printf '%s\n' "$HOSTPORT" > "$STATE/host"
}
save() { # keep the first-seen (pre-rewrite) copy
[ -e "$STATE/orig$1" ] && return 0
mkdir -p "$STATE/orig$(dirname "$1")"
cp -p "$1" "$STATE/orig$1"
}
created() { printf '%s\n' "$1" >> "$STATE/created"; }
if [ "$MODE" = restore ]; then
[ -d "$STATE" ] || exit 0
host=$(cat "$STATE/host")
# mapfile from a process substitution, not `cmd | while read`: no pipeline,
# so nothing for pipefail to misreport (scripts/tests/lsphp-info-probe.test.sh).
mapfile -t saved < <(cd "$STATE/orig" && find . -type f)
for rel in "${saved[@]}"; do
cat "$STATE/orig/$rel" > "/${rel#./}"
done
while read -r f; do [ -n "$f" ] && rm -f -- "$f"; done < "$STATE/created"
# A repo/apt file that appeared during this RUN and still names the cache
# (e.g. written by a package from the rewritten repos) is removed too.
mapfile -t current < <(repo_files)
for f in "${current[@]}"; do
if ! grep -qxF -- "$f" "$STATE/manifest" && grep -qF -- "$host" "$f"; then
echo "pkg-cache-repos: removing new file that references the cache: $f"
rm -f -- "$f"
fi
done
left=$(repo_files | xargs -r grep -lF -- "$host" 2>/dev/null || true)
rm -rf "$STATE"
[ -z "$left" ] || die "cache address $host still present after restore in: $left"
echo "pkg-cache-repos: original repository files restored (no reference to $host left)"
exit 0
fi
if [ "$MODE" = release-install ]; then
[ $# -gt 0 ] || die "release-install needs: epel and/or remi"
tmp=$(mktemp -d)
curl_opts=(-fsS --proto '=https' --retry 5 --retry-all-errors --max-time 120)
if [ -n "$C" ]; then
start_session
make_bundle
curl_opts+=(--cacert "$STATE/ca-bundle.pem")
fi
files=()
el=$(rpm -E '%{?rhel}' 2>/dev/null || true)
case "$el" in 9|10) ;; *) die "release-install supports EL9/EL10 only (rpm -E %rhel = '$el')" ;; esac
for name in "$@"; do
case "$name" in
epel) pkg=epel-release key=RPM-GPG-KEY-EPEL-$el
path=/epel/epel-release-latest-$el.noarch.rpm
origin=https://dl.fedoraproject.org/pub/epel/epel-release-latest-$el.noarch.rpm ;;
remi) pkg=remi-release key=RPM-GPG-KEY-remi2024
path=/remi/enterprise/remi-release-$el.rpm
origin=https://rpms.remirepo.net/enterprise/remi-release-$el.rpm ;;
*) die "unknown release package '$name'" ;;
esac
url="$origin"; [ -n "$C" ] && url="$C$path"
f="$tmp/$pkg.rpm"
curl "${curl_opts[@]}" -o "$f" "$url" || die "download failed: $url"
rpm --import "$SHARE/$key" || die "cannot import $SHARE/$key"
got=$(rpm -qp --nosignature --qf '%{NAME}' "$f" 2>/dev/null || true)
[ "$got" = "$pkg" ] || die "$url is '$got', expected $pkg"
chk=$(rpm -K "$f" 2>&1 || true)
[[ "$chk" == *": digests signatures OK" ]] || die "signature check failed for $url: $chk"
echo "pkg-cache-repos: $pkg verified ($chk) from $url"
files+=("$f")
done
dnf install -y --setopt=localpkg_gpgcheck=1 "${files[@]}"
rm -rf "$tmp"
exit 0
fi
if [ -z "$C" ] && [ "$MODE" != nodesource ] && [ "$MODE" != go-env ]; then
echo "pkg-cache-repos($MODE): PKG_CACHE not set - using upstream repositories"
exit 0
fi
rewrite_rpm() {
local f="$1"
grep -q -E '^#? ?baseurl=https?://(repo\.almalinux\.org|download\.example|rpms\.remirepo\.net|rpm\.nodesource\.com)/' "$f" || return 0
save "$f"
# Per [section]: a (commented or live) baseurl on a known origin becomes
# "baseurl=<cache>/<prefix>/<path> <origin>/<path>"; that section's
# mirrorlist/metalink is commented out (dnf prefers them over baseurl) and it
# gets sslcacert=<system bundle + pkg-cache CA>, gpgcheck=1 and, for repos
# that sign repomd.xml, repo_gpgcheck=1. Other sections are untouched.
awk -v C="$C" -v BUNDLE="$STATE/ca-bundle.pem" '
function flush( i) {
for (i = 1; i <= n; i++) {
if (hit && buf[i] ~ /^(mirrorlist|metalink)=/) buf[i] = "#" buf[i]
if (hit && buf[i] ~ /^(sslcacert|gpgcheck|repo_gpgcheck)[ \t]*=/) continue
print buf[i]
}
if (hit) {
print "sslcacert=" BUNDLE
print "gpgcheck=1"
if (signed) print "repo_gpgcheck=1"
}
n = 0; hit = 0; signed = 0
}
function rw(line, re, prefix, origin, sig, p) {
if (match(line, re)) {
p = substr(line, RSTART + RLENGTH); sub(/[ \t].*$/, "", p)
hit = 1; if (sig) signed = 1
return "baseurl=" C "/" prefix "/" p " " origin "/" p
}
return line
}
/^\[/ { flush() }
{
l = $0
if (l ~ /^#? ?baseurl=/) {
l = rw(l, "^#? ?baseurl=https?://repo\\.almalinux\\.org/almalinux/", "almalinux", "https://repo.almalinux.org/almalinux", 1)
l = rw(l, "^#? ?baseurl=https?://download\\.example/pub/epel/", "epel", "https://dl.fedoraproject.org/pub/epel", 0)
l = rw(l, "^#? ?baseurl=https?://rpms\\.remirepo\\.net/", "remi", "https://rpms.remirepo.net", 1)
# NodeSource signs its RPMs but not repomd.xml (404): gpgcheck only.
l = rw(l, "^#? ?baseurl=https?://rpm\\.nodesource\\.com/", "nodesource", "https://rpm.nodesource.com", 0)
}
buf[++n] = l
}
END { flush() }
' "$f" > "$f.pkgcache" && cat "$f.pkgcache" > "$f" && rm -f "$f.pkgcache"
}
rewrite_apt() {
local f="$1"
grep -q -E 'https?://([a-z]{2}\.)?(archive|security)\.ubuntu\.com/|https?://rpms\.litespeedtech\.com/|https?://(deb|security)\.debian\.org/' "$f" || return 0
save "$f"
# Debian: debian-security first, and "debian" only as a whole path segment,
# so .../debian-security is never rewritten as /debian/.
sed -i -E \
-e "s%https?://(archive|[a-z]{2}\.archive)\.ubuntu\.com/ubuntu/?%$C/ubuntu/%g" \
-e "s%https?://security\.ubuntu\.com/ubuntu/?%$C/ubuntu-security/%g" \
-e "s%https?://rpms\.litespeedtech\.com/%$C/litespeed/%g" \
-e "s%https?://(deb|security)\.debian\.org/debian-security/?([[:space:]]|\$)%$C/debian-security/\\2%g" \
-e "s%https?://security\.debian\.org/?([[:space:]]|\$)%$C/debian-security/\\1%g" \
-e "s%https?://deb\.debian\.org/debian/?([[:space:]]|\$)%$C/debian/\\1%g" \
"$f"
}
if [ "$MODE" = go-env ]; then
# Prints shell exports for `eval`. Without PKG_CACHE: nothing (Go defaults).
# GOPROXY "a|b" falls through on ANY error, so a dead cache costs one
# failed request, not the build. GOSUMDB stays the default sum.golang.org,
# reached directly: modules are still verified against go.sum + the sumdb.
[ -n "$C" ] || { echo "pkg-cache-repos(go-env): PKG_CACHE not set - default GOPROXY" >&2; exit 0; }
start_session
make_bundle
printf "export GOPROXY='%s/goproxy|https://proxy.golang.org|direct' SSL_CERT_FILE='%s'\n" "$C" "$STATE/ca-bundle.pem"
echo "pkg-cache-repos(go-env): GOPROXY via $C/goproxy (fallback proxy.golang.org)" >&2
exit 0
fi
if [ "$MODE" = nodesource ]; then
# Replaces `curl https://rpm.nodesource.com/setup_N.x | bash` (unverified
# remote script): writes the same nodesource-nodejs repo (origin URL, so the
# shipped image keeps a normal repo file) with gpgcheck=1 and the signing key
# committed in keys/pkg-cache/; with PKG_CACHE it is then rewritten like any
# other repo and `restore` puts the origin version back.
major="${1:?usage: pkg-cache-repos.sh nodesource MAJOR}"
[[ "$major" =~ ^[0-9]{1,3}$ ]] || die "bad Node major '$major'"
key="$SHARE/RPM-GPG-KEY-nodesource"
[ -s "$key" ] || die "missing $key"
rpm --import "$key" || die "cannot import $key"
rm -f /etc/yum.repos.d/nodesource*.repo
f=/etc/yum.repos.d/nodesource-nodejs.repo
printf '%s\n' "[nodesource-nodejs]" \
"name=Node.js Packages for Linux RPM based distros - \$basearch" \
"baseurl=https://rpm.nodesource.com/pub_${major}.x/nodistro/nodejs/\$basearch" \
"priority=9" "enabled=1" "gpgcheck=1" \
"gpgkey=file://$key" "module_hotfixes=1" > "$f"
echo "pkg-cache-repos: wrote $f (Node ${major}.x, key $(basename "$key"))"
if [ -n "$C" ]; then
start_session
make_bundle
rewrite_rpm "$f"
grep -h '^baseurl=' "$f" | sed 's/^/ /'
fi
exit 0
fi
case "$MODE" in
rpm)
start_session
make_bundle
for f in /etc/yum.repos.d/*.repo; do [ -f "$f" ] && rewrite_rpm "$f"; done
echo "pkg-cache-repos(rpm): repos now via $C:"
grep -h '^baseurl=' /etc/yum.repos.d/*.repo | grep -F "$C/" | sort -u | sed 's/^/ /'
;;
apt)
start_session
conf=/etc/apt/apt.conf.d/99pkg-cache
# Trust the pkg-cache CA for the cache host ONLY (apt per-host option).
printf 'Acquire::https::%s::CaInfo "%s";\n' "${HOSTPORT%%:*}" "$SHARE/ca.crt" > "$conf"
created "$conf"
for f in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
[ -f "$f" ] && rewrite_apt "$f"
done
echo "pkg-cache-repos(apt): sources now via $C:"
# (|| true: Debian 13 images have no /etc/apt/sources.list; grep's exit 2
# for the missing file must not fail the build under pipefail)
{ grep -h -E '^(deb|URIs:)' /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null || true; } | grep -F "$C/" | sort -u | sed 's/^/ /' || true
;;
*) die "unknown mode '$MODE'" ;;
esac
+17 -3
View File
@@ -65,13 +65,22 @@ fi
# Copy certificates for each domain # Copy certificates for each domain
UPDATED=0 UPDATED=0
FAILED=0 FAILED=0
SKIPPED_EXPIRED=0
while read -r domain; do while read -r domain; do
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem" CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem" KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem" COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
# Never publish an expired lineage. It cannot help any client and,
# because HAProxy serves the first file claiming an SNI name, it can
# shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
# Counted separately from FAILED: a dead lineage that keeps expiring is
# reported on every run (ERROR log) without making every run "fail".
log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
# Assemble in a staging dir and rename into place. NEVER redirect into # Assemble in a staging dir and rename into place. NEVER redirect into
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -89,10 +98,15 @@ while read -r domain; do
fi fi
done <<< "$DOMAINS" done <<< "$DOMAINS"
log_info "Certificate update completed: $UPDATED updated, $FAILED failed" # Already-published pems that have expired since are moved out too (they
# can only shadow; see cert_retire_expired_pems).
cert_retire_expired_pems
RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
log_info "Certificate update completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
# Reload HAProxy if any certificates were updated # Reload HAProxy if any certificates were updated
if [ $UPDATED -gt 0 ]; then if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
# Never reload onto unvalidated material: a reload that fails to load the # Never reload onto unvalidated material: a reload that fails to load the
# certs directory drops HTTPS for every site on this host. # certs directory drops HTTPS for every site on this host.
if ! haproxy_config_ok; then if ! haproxy_config_ok; then
+5 -5
View File
@@ -12,9 +12,9 @@
# #
# The stick table also keeps NO history, so nothing in it can identify who was # The stick table also keeps NO history, so nothing in it can identify who was
# tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST # tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST
# at /var/log/haproxy.log (it does not exist inside this container): # at /var/log/haproxy/access.log (inside this container since 2026.11.2):
# grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20 # tarpit / deny # grep -aE ' (PT|PR)--' /var/log/haproxy/access.log | tail -20 # tarpit / deny
# grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20 # rate-limit / block # grep -aE ' (429|403) ' /var/log/haproxy/access.log | tail -20 # rate-limit / block
# #
# What IS knowable from the stick table — the current per-IP rates — is printed # What IS knowable from the stick table — the current per-IP rates — is printed
# by show-edge-ip-rates.sh, which this shim now runs. # by show-edge-ip-rates.sh, which this shim now runs.
@@ -25,8 +25,8 @@ cat >&2 <<'NOTE'
NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits. NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits.
The HAProxy stick table stores no history and no gpc0/gpc1 counters, so The HAProxy stick table stores no history and no gpc0/gpc1 counters, so
the old "Scan Count"/"BLOCKED" columns were fabricated numbers. the old "Scan Count"/"BLOCKED" columns were fabricated numbers.
Actual tarpit/deny events are in /var/log/haproxy.log ON THE HOST: Actual tarpit/deny events are in /var/log/haproxy/access.log (in this container since 2026.11.2):
grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20 grep -aE ' (PT|PR)--' /var/log/haproxy/access.log | tail -20
Running show-edge-ip-rates.sh instead (current rates, real values): Running show-edge-ip-rates.sh instead (current rates, real values):
NOTE NOTE
+54 -18
View File
@@ -55,29 +55,50 @@ if [ ! -f /etc/haproxy/wpadmin_gate_exempt.list ]; then
fi fi
fi fi
# HAProxy access log receiver (see scripts/haproxy-log-receiver.py and the
# `log` line in hap_header.tpl). HAProxy sends datagrams to a UNIX socket
# inside its chroot; this process writes them to a size-capped, rotating
# /var/log/haproxy/access.log (WHP mounts the haproxy-logs volume there).
# Started BEFORE init.py so haproxy's first lines land. Supervised by a
# restart loop: if it dies it is back within 2s, and while it is down HAProxy
# just drops log lines (non-blocking sends), it never blocks on them.
# stdout is discarded on purpose -- it never prints log lines, and
# `docker logs` must not carry a per-request firehose (WHP's AI monitor reads
# it); its rare status/error lines go to stderr.
mkdir -p /var/log/haproxy /var/lib/haproxy
(
while true; do
python /haproxy/scripts/haproxy-log-receiver.py >/dev/null || true
sleep 2
done
) &
cron & cron &
# HAProxy stderr relay (see scripts/haproxy-stderr-relay.py and
# haproxy_manager.spawn_haproxy_master()). HAProxy's master stderr (ALERTs,
# "exited with code 134", the watchdog/ha_panic thread dump) is appended to
# /var/log/haproxy/haproxy-stderr.log; this loop re-emits it on the container's
# stderr so `docker logs` / the docker logging driver / Graylog get it. Started
# BEFORE init.py so the first start's lines are followed. Restarted within 2 s
# if it ever dies; it only reads a file, so it can never affect HAProxy.
(
while true; do
python /haproxy/scripts/haproxy-stderr-relay.py || true
sleep 2
done
) &
# Phase 1: container init # Phase 1: container init
python /haproxy/scripts/init.py python /haproxy/scripts/init.py
# Phase 1.5: in-container haproxy supervisor. # Phase 1.5: in-container haproxy supervisor (see scripts/haproxy-supervisor.sh:
# haproxy runs as a background child of PID 1 (gunicorn) with NOTHING watching # revives a dead haproxy master within HAPROXY_SUPERVISOR_INTERVAL seconds, with
# it after init. If the haproxy master dies mid-life (e.g. SIGABRT -> exit 134, # backoff on failed restarts and on crash loops; knobs validated as integers).
# segfault), the container stays "up" (gunicorn is PID 1), Docker's --restart /haproxy/scripts/haproxy-supervisor.sh &
# policy never fires, and haproxy is down until the external host watchdog
# full-restarts the whole container minutes later (dropping every connection).
# This loop revives haproxy in place within one interval. ensure_haproxy.py is
# idempotent — a cheap no-op whenever haproxy is already running.
HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
(
while true; do
sleep "${HAPROXY_SUPERVISOR_INTERVAL}"
python /haproxy/scripts/ensure_haproxy.py 2>&1 || true
done
) &
# Phase 2: WSGI servers # Phase 2: WSGI servers
# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT # Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
# (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default # (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
# 1000 — worker recycle frequency). # 1000 — worker recycle frequency).
# #
@@ -91,12 +112,27 @@ API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}" MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}" MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
# Worker class (incident 2026-10-09): stock gthread makes every client socket
# fully blocking with NO timeout, so any peer that sends a few bytes and goes
# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. Stock gthread
# also never reaps connections that send nothing (2x1000 of them wedge it
# permanently) and has no total header-read bound (slowloris). This subclass
# adds: a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, 20s), a
# first-byte deadline (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s), a total
# request-line+headers deadline (HAPROXY_MGR_HEADER_DEADLINE, 10s), and a
# watchdog that exits a worker whose threads have ALL been stuck for
# HAPROXY_MGR_WEDGE_SECONDS (3600) so the master respawns it.
# See haproxy_gunicorn_worker.py. --timeout below does NOT cover any of these:
# for gthread it only watches the main thread's heartbeat.
WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
# Default page server on :8080. Stays in the background. # Default page server on :8080. Stays in the background.
# --threads 4 lets one worker handle bursts of blocked-IP/default-page hits # --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
# without forking. --max-requests recycles the worker to bound memory drift. # without forking. --max-requests recycles the worker to bound memory drift.
gunicorn \ gunicorn \
--bind 0.0.0.0:8080 \ --bind 0.0.0.0:8080 \
--workers 1 --threads 4 --worker-class gthread \ --workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
--max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \ --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
--timeout 30 \ --timeout 30 \
--access-logfile - --error-logfile - --log-level info \ --access-logfile - --error-logfile - --log-level info \
@@ -108,7 +144,7 @@ gunicorn \
# up). Longer --timeout because cert issuance hits ACME and can take a while. # up). Longer --timeout because cert issuance hits ACME and can take a while.
exec gunicorn \ exec gunicorn \
--bind 0.0.0.0:8000 \ --bind 0.0.0.0:8000 \
--workers "${API_WORKERS}" --threads 4 --worker-class gthread \ --workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
--max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \ --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
--timeout "${API_TIMEOUT}" \ --timeout "${API_TIMEOUT}" \
--access-logfile - --error-logfile - --log-level info \ --access-logfile - --error-logfile - --log-level info \
+17 -3
View File
@@ -47,13 +47,22 @@ fi
# Copy certificates for each domain # Copy certificates for each domain
UPDATED=0 UPDATED=0
FAILED=0 FAILED=0
SKIPPED_EXPIRED=0
while read -r domain; do while read -r domain; do
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem" CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem" KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem" COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
# Never publish an expired lineage. It cannot help any client and,
# because HAProxy serves the first file claiming an SNI name, it can
# shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
# Counted separately from FAILED: a dead lineage that keeps expiring is
# reported on every run (ERROR log) without making every run "fail".
log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
# Assemble in a staging dir and rename into place. NEVER redirect into # Assemble in a staging dir and rename into place. NEVER redirect into
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and # $COMBINED_FILE: the shell truncates the live pem before cat runs, and
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there # HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -71,10 +80,15 @@ while read -r domain; do
fi fi
done <<< "$DOMAINS" done <<< "$DOMAINS"
log_info "Certificate sync completed: $UPDATED updated, $FAILED failed" # Already-published pems that have expired since are moved out too (they
# can only shadow; see cert_retire_expired_pems).
cert_retire_expired_pems
RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
log_info "Certificate sync completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
# Reload HAProxy if any certificates were updated # Reload HAProxy if any certificates were updated
if [ $UPDATED -gt 0 ]; then if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
# Never reload onto unvalidated material: a reload that fails to load the # Never reload onto unvalidated material: a reload that fails to load the
# certs directory drops HTTPS for every site on this host. # certs directory drops HTTPS for every site on this host.
if ! haproxy_config_ok; then if ! haproxy_config_ok; then
+178
View File
@@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Tests for scripts/haproxy-log-receiver.py (in-container access log).
Pins: one datagram == one line (PRI stripped, control chars neutralised),
the size cap (live file + N gzip generations, oldest dropped), a write error
(full disk / unwritable dir) drops lines without raising, and an end-to-end
datagram path through a real UNIX socket.
Running
-------
python3 scripts/test-access-log-receiver.py
"""
import gzip
import importlib.util
import os
import shutil
import socket
import tempfile
import threading
import time
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
_spec = importlib.util.spec_from_file_location(
'haproxy_log_receiver', os.path.join(HERE, 'haproxy-log-receiver.py'))
rx = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(rx)
class CleanLine(unittest.TestCase):
def test_strips_pri_and_trailing_newline(self):
self.assertEqual(rx.clean_line(b'<150>1 2026-11-08T00:00:00Z h haproxy 7 - - GET /\n'),
'1 2026-11-08T00:00:00Z h haproxy 7 - - GET /')
def test_embedded_newlines_cannot_forge_lines(self):
out = rx.clean_line(b'<150>1 x ua=evil\r\nFAKE 200 line\x1b[2J')
self.assertNotIn('\n', out)
self.assertNotIn('\r', out)
self.assertNotIn('\x1b', out)
self.assertIn('FAKE 200 line', out)
def test_invalid_utf8_does_not_raise(self):
self.assertIn('�', rx.clean_line(b'<150>1 \xff\xfe'))
def test_non_pri_angle_bracket_kept(self):
self.assertEqual(rx.clean_line(b'<html> x'), '<html> x')
class Rotation(unittest.TestCase):
def setUp(self):
self.dir = tempfile.mkdtemp(prefix='hlr-test-')
self.path = os.path.join(self.dir, 'access.log')
def tearDown(self):
shutil.rmtree(self.dir, ignore_errors=True)
def _files(self):
return sorted(os.listdir(self.dir))
def test_size_cap_and_generations(self):
max_bytes, backups = 4096, 3
w = rx.RotatingWriter(self.path, max_bytes, backups)
line = 'x' * 99 # 100 bytes with newline
for i in range(1000): # ~100 KB -> ~24 rotations
w.write(f'{i:05d}' + line[5:])
w.close()
files = self._files()
self.assertEqual(files, ['access.log', 'access.log.1.gz',
'access.log.2.gz', 'access.log.3.gz'], files)
self.assertLessEqual(os.path.getsize(self.path), max_bytes)
total = sum(os.path.getsize(os.path.join(self.dir, f)) for f in files)
self.assertLessEqual(total, (1 + backups) * max_bytes)
# newest generation holds the lines written just before the live file
with gzip.open(os.path.join(self.dir, 'access.log.1.gz'), 'rt') as fh:
gen1 = fh.read().splitlines()
with open(self.path) as fh:
live = fh.read().splitlines()
self.assertEqual(int(gen1[-1][:5]) + 1, int(live[0][:5]))
self.assertEqual(live[-1][:5], '00999')
for l in gen1:
self.assertEqual(len(l), 99)
def test_zero_backups_keeps_only_live_file(self):
w = rx.RotatingWriter(self.path, 1024, 0)
for _ in range(100):
w.write('y' * 63)
w.close()
self.assertEqual(self._files(), ['access.log'])
self.assertLessEqual(os.path.getsize(self.path), 1024)
def test_reopens_existing_file_and_counts_its_size(self):
with open(self.path, 'w') as fh:
fh.write('z' * 1000 + '\n')
w = rx.RotatingWriter(self.path, 1500, 2)
w.write('a' * 600) # 1001 + 601 > 1500 -> rotate first
w.close()
self.assertIn('access.log.1.gz', self._files())
with open(self.path) as fh:
self.assertEqual(fh.read(), 'a' * 600 + '\n')
def test_write_error_drops_and_never_raises(self):
blocker = os.path.join(self.dir, 'not-a-dir')
with open(blocker, 'w') as fh:
fh.write('')
msgs = []
w = rx.RotatingWriter(os.path.join(blocker, 'access.log'), 4096, 2, msgs.append)
for _ in range(50):
w.write('line')
w.flush()
w.close()
self.assertEqual(w.dropped, 50)
self.assertEqual(len(msgs), 1, msgs) # rate-limited report
class EndToEnd(unittest.TestCase):
def test_datagrams_become_lines(self):
d = tempfile.mkdtemp(prefix='hlr-e2e-')
try:
sock_path = os.path.join(d, 'log.sock')
log_path = os.path.join(d, 'logs', 'access.log')
sock = rx.bind_socket(sock_path)
self.assertEqual(os.stat(sock_path).st_mode & 0o777, 0o666)
w = rx.RotatingWriter(log_path, 1 << 20, 2)
stop = threading.Event()
t = threading.Thread(target=rx.serve, args=(sock, w, stop), daemon=True)
t.start()
cl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
cl.setblocking(False)
for i in range(20):
msg = f'<150>1 t h haproxy 1 - - req {i} host=a.example cip=192.0.2.1'.encode()
# The default net.unix.max_dgram_qlen is 10 per netns: a full
# queue is EAGAIN (HAProxy drops the line); here, retry.
for _ in range(100):
try:
cl.sendto(msg, sock_path)
break
except BlockingIOError:
time.sleep(0.01)
deadline = time.time() + 5
lines = []
while time.time() < deadline:
time.sleep(0.2)
w.flush()
if os.path.exists(log_path):
with open(log_path) as fh:
lines = fh.read().splitlines()
if len(lines) == 20:
break
stop.set()
t.join(3)
sock.close()
w.close()
self.assertEqual(len(lines), 20, lines)
self.assertTrue(lines[0].startswith('1 t h haproxy 1 - - req 0 host=a.example'))
finally:
shutil.rmtree(d, ignore_errors=True)
def test_sender_to_dead_receiver_fails_fast_nonblocking(self):
"""What HAProxy sees when the receiver is down: an immediate error,
never a block (HAProxy then just counts a dropped log line)."""
d = tempfile.mkdtemp(prefix='hlr-dead-')
try:
cl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
cl.setblocking(False)
start = time.monotonic()
with self.assertRaises(OSError):
cl.sendto(b'x', os.path.join(d, 'log.sock'))
self.assertLess(time.monotonic() - start, 0.5)
finally:
shutil.rmtree(d, ignore_errors=True)
if __name__ == '__main__':
unittest.main(verbosity=2)
+164
View File
@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""Offline tests for scripts/assemble-multiarch-index.sh (multi-arch index assembly).
The script is the publish-side half of the multi-arch pipeline: it receives the
amd64 and (optional) arm64 digests that the build steps pushed BY DIGEST and
turns them into the tag(s). A stub stands in for `docker buildx imagetools`
(env IMAGETOOLS), records every `create` call and answers `inspect` from a
table, so no registry or Docker is needed.
Held invariants:
- arm64 failure / cancel / skip -> amd64-only create, exit 0 (tag still published)
- a STALE arm64 digest is never attached unless this run's arm64 outcome is success
- bad/missing amd64 digest, or amd64 that is not linux/amd64 -> exit 1, NO create
- an arm64 digest that does not inspect as arm64 is dropped (amd64-only)
- every tag x every repo is created; sources are <repo>@<digest> in that repo
Running: python3 scripts/test-assemble-multiarch-index.py
"""
import json
import os
import stat
import subprocess
import sys
import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
SCRIPT = os.path.join(HERE, 'assemble-multiarch-index.sh')
A = 'sha256:' + 'a' * 64
B = 'sha256:' + 'b' * 64
C = 'sha256:' + 'c' * 64
STUB = r'''#!/usr/bin/env python3
import json, os, sys
a = sys.argv[1:]
log = os.environ['STUB_LOG']
table = json.loads(os.environ['STUB_ARCH'])
if a[0] == 'inspect':
ref = [x for x in a[1:] if '@sha256:' in x][0]
arch = table.get(ref.split('@', 1)[1])
if arch is None:
sys.exit(1)
print(json.dumps({'architecture': arch, 'os': 'linux'}))
elif a[0] == 'create':
open(log, 'a').write(json.dumps(a[1:]) + '\n')
else:
sys.exit(2)
'''
class AssembleTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.stub = os.path.join(self.tmp.name, 'stub')
with open(self.stub, 'w') as f:
f.write(STUB)
os.chmod(self.stub, os.stat(self.stub).st_mode | stat.S_IEXEC)
self.log = os.path.join(self.tmp.name, 'creates.log')
def tearDown(self):
self.tmp.cleanup()
def run_script(self, args, arch=None):
env = dict(os.environ, IMAGETOOLS=self.stub, STUB_LOG=self.log,
STUB_ARCH=json.dumps(arch if arch is not None else {A: 'amd64', B: 'arm64'}))
p = subprocess.run(['bash', SCRIPT] + args, env=env, capture_output=True, text=True)
creates = []
if os.path.exists(self.log):
creates = [json.loads(l) for l in open(self.log)]
return p, creates
BASE = ['--repo', 'r.example/o/img', '--tag', 'latest']
def test_both_platforms(self):
p, c = self.run_script(['--amd64', A, '--arm64', B, '--arm64-outcome', 'success'] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A, 'r.example/o/img@' + B]])
self.assertIn('platforms=linux/amd64,linux/arm64', p.stdout)
def test_arm64_failure_is_amd64_only_even_with_a_digest(self):
# A stale digest (e.g. left over from an earlier run) must not be attached.
for outcome in ('failure', 'cancelled', 'skipped', ''):
if os.path.exists(self.log):
os.remove(self.log)
p, c = self.run_script(['--amd64', A, '--arm64', B, '--arm64-outcome', outcome] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A]], outcome)
self.assertIn('::warning::', p.stdout)
self.assertIn('platforms=linux/amd64\n', p.stdout)
def test_no_arm64_args_is_amd64_only(self):
p, c = self.run_script(['--amd64', A] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A]])
def test_success_with_empty_or_malformed_arm64_digest(self):
for bad in ('', 'sha256:short', 'latest'):
if os.path.exists(self.log):
os.remove(self.log)
p, c = self.run_script(['--amd64', A, '--arm64', bad, '--arm64-outcome', 'success'] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(len(c[0]), 3, bad) # -t, ref, one source
def test_arm64_digest_that_is_not_arm64_is_dropped(self):
p, c = self.run_script(['--amd64', A, '--arm64', C, '--arm64-outcome', 'success'] + self.BASE,
arch={A: 'amd64', C: 'amd64'})
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A]])
def test_arm64_equal_to_amd64_is_dropped(self):
p, c = self.run_script(['--amd64', A, '--arm64', A, '--arm64-outcome', 'success'] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(len(c[0]), 3)
def test_amd64_problems_publish_nothing(self):
for amd in ('', 'nope', B): # B inspects as arm64
p, c = self.run_script(['--amd64', amd, '--arm64', B, '--arm64-outcome', 'success'] + self.BASE)
self.assertNotEqual(p.returncode, 0, amd)
self.assertEqual(c, [], amd)
p, c = self.run_script(['--amd64', C, '--arm64-outcome', 'failure'] + self.BASE, arch={})
self.assertNotEqual(p.returncode, 0) # not inspectable
self.assertEqual(c, [])
def test_all_tags_in_all_repos(self):
p, c = self.run_script(['--amd64', A, '--arm64', B, '--arm64-outcome', 'success',
'--repo', 'gitea.example/o/img', '--repo', 'ghcr.io/o/img',
'--tag', 'latest', '--tag', '1.2.3', '--tag', 'abc123'])
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(len(c), 2)
self.assertEqual(c[0], ['-t', 'gitea.example/o/img:latest', '-t', 'gitea.example/o/img:1.2.3',
'-t', 'gitea.example/o/img:abc123',
'gitea.example/o/img@' + A, 'gitea.example/o/img@' + B])
self.assertEqual(c[1][-2:], ['ghcr.io/o/img@' + A, 'ghcr.io/o/img@' + B])
def test_requires_repo_and_tag_and_rejects_unknown_args(self):
for args in (['--amd64', A, '--tag', 't'], ['--amd64', A, '--repo', 'r'], ['--bogus', 'x']):
p, c = self.run_script(args)
self.assertNotEqual(p.returncode, 0)
self.assertEqual(c, [])
class WorkflowWiring(unittest.TestCase):
"""The workflow must keep the properties the script relies on."""
def setUp(self):
self.wf = open(os.path.join(HERE, '..', '.gitea', 'workflows', 'build-push-coraza.yaml')).read()
def test_arm64_step_is_non_fatal_and_digest_comes_from_step_output(self):
self.assertIn('continue-on-error: true', self.wf)
self.assertIn('steps.arm64.outputs.digest', self.wf)
self.assertIn('steps.arm64.outcome', self.wf)
self.assertIn('assemble-multiarch-index.sh', self.wf)
def test_both_builds_push_by_digest_untagged(self):
self.assertEqual(self.wf.count('push-by-digest=true'), 2)
self.assertNotIn('tags:', self.wf.replace('# tags:', ''))
def test_no_tag_is_pushed_by_a_build_step(self):
# tags exist only via imagetools create (the assemble script)
self.assertNotIn('ghcr.io/shadowdao/coraza-spoa:latest\n', self.wf.split('assemble-multiarch-index.sh')[0])
if __name__ == '__main__':
unittest.main(verbosity=2)
+121
View File
@@ -73,6 +73,15 @@ LIB = os.path.join(SCRIPTS_DIR, 'cert-publish-lib.sh')
BROKEN_TOKEN = '__BROKEN__' BROKEN_TOKEN = '__BROKEN__'
DOMAIN = 'test.example.com' DOMAIN = 'test.example.com'
# Real SAN certificates (see scripts/test-fixtures/san-certs/). expired_old is
# the brain-jar.com shape: a multi-SAN certificate that expired in 2020.
SAN_FIXTURES = os.path.join(SCRIPTS_DIR, 'test-fixtures', 'san-certs')
def san_fixture(name):
with open(os.path.join(SAN_FIXTURES, name)) as fh:
return fh.read()
# --- test key material ------------------------------------------------------- # --- test key material -------------------------------------------------------
# openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \ # openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \
# -subj /CN=test.example.com # -subj /CN=test.example.com
@@ -618,6 +627,72 @@ class CertScriptBehaviour:
self.assertFalse(self.reload_attempted()) self.assertFalse(self.reload_attempted())
# -- never publish an expired certificate (brainjar.net, 2026-10-07) ----
def test_expired_lineage_is_skipped_and_reported(self):
"""renew-certificates.sh used to re-publish EVERY live/ lineage every 12h,
expired or not; the expired brain-jar.com bundle then shadowed the valid
brainjar.net bundle for the names they shared."""
dead = os.path.join(self.le_live, 'brain-jar.com')
os.makedirs(dead)
write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
result = self.run_it()
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertFalse(
os.path.exists(os.path.join(self.certs_dir, 'brain-jar.com.pem')),
'an expired lineage was published into the crt directory')
# The healthy lineage next to it is still published.
self.assertEqual(read(self.live_pem), NEW_BUNDLE)
self.assertIn('Skipping EXPIRED certificate lineage brain-jar.com', self.logs())
self.assertIn('1 updated, 0 failed, 1 expired lineage(s) skipped', self.logs())
self.assertIn('brain-jar.com', read(self.error_log),
'the skip must land in the monitored error log')
def test_expired_lineage_does_not_replace_an_existing_pem(self):
dead = os.path.join(self.le_live, 'brain-jar.com')
os.makedirs(dead)
write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
existing = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'), OTHER_BUNDLE)
self.run_it()
self.assertEqual(read(existing), OTHER_BUNDLE,
'an expired lineage overwrote the file in the crt dir')
def test_already_published_expired_pem_is_retired(self):
"""Review M2: skipping publication is not enough - an expired pem that
is ALREADY in the crt dir (brain-jar.com.pem) keeps shadowing."""
expired = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'),
san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
result = self.run_it()
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertFalse(os.path.exists(expired), 'expired pem still in the crt dir')
retired = os.path.join(self.haproxy_dir, 'cert-retired', 'brain-jar.com.pem')
self.assertTrue(os.path.exists(retired), 'expired pem was not moved to cert-retired/ (never delete)')
self.assertIn('Retired EXPIRED certificate brain-jar.com.pem', self.logs())
self.assertIn('1 expired pem(s) retired', self.logs())
self.assertTrue(self.reload_attempted())
self.assertEqual(read(self.live_pem), NEW_BUNDLE, 'valid neighbours untouched')
self.assert_certs_dir_is_clean()
def test_retiring_an_expired_pem_alone_triggers_a_reload(self):
self.seed_previous_bundle(NEW_BUNDLE)
shutil.rmtree(self.domain_dir)
other = os.path.join(self.le_live, 'other.example')
os.makedirs(other)
write(os.path.join(other, 'fullchain.pem'), san_fixture('expired_old.crt'))
write(os.path.join(other, 'privkey.pem'), san_fixture('expired_old.key'))
write(os.path.join(self.certs_dir, 'other.example.pem'),
san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
result = self.run_it()
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertIn('0 updated', self.logs())
self.assertTrue(self.reload_attempted(), 'HAProxy must reload to drop the retired pem')
class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture): class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture):
SCRIPT = 'renew-certificates.sh' SCRIPT = 'renew-certificates.sh'
@@ -638,6 +713,52 @@ class TestCertPublishLibrary(CertScriptFixture):
path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE) path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0) self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0)
def test_cert_is_expired_reports_both_directions(self):
expired = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
valid = write(os.path.join(self.tmp, 'valid.crt'), san_fixture('new_bundle.crt'))
self.assertEqual(0, self.call('cert_is_expired "$1"', expired).returncode,
'an expired certificate was not recognised as expired')
self.assertEqual(1, self.call('cert_is_expired "$1"', valid).returncode,
'a valid certificate was reported as expired')
self.assertEqual(1, self.call('cert_is_expired "$1"',
os.path.join(self.tmp, 'missing')).returncode)
def test_expired_sweep_keeps_the_last_pem_and_the_default_cert(self):
exp = san_fixture('expired_old.crt') + san_fixture('expired_old.key')
only = write(os.path.join(self.certs_dir, 'only.pem'), exp)
r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
self.assertIn('n=0', r.stdout)
self.assertTrue(os.path.exists(only), 'emptied the crt dir: that fails the whole :443 bind')
dflt = write(os.path.join(self.certs_dir, 'default_self_signed_cert.pem'), exp)
r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
self.assertIn('n=1', r.stdout)
self.assertTrue(os.path.exists(dflt), 'the default self-signed cert is not ours to retire')
self.assertFalse(os.path.exists(only))
def test_expired_sweep_never_touches_a_valid_pem(self):
valid = write(os.path.join(self.certs_dir, 'a.pem'), NEW_BUNDLE)
write(os.path.join(self.certs_dir, 'b.pem'), OTHER_BUNDLE)
r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
self.assertIn('n=0', r.stdout)
self.assertEqual(read(valid), NEW_BUNDLE)
def test_cert_publish_refuses_an_expired_certificate(self):
cert = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
key = write(os.path.join(self.tmp, 'expired.key'), san_fixture('expired_old.key'))
dest = os.path.join(self.certs_dir, 'brain-jar.com.pem')
result = self.call('cert_publish "$1" "$2" "$3"', cert, key, dest)
self.assert_rejected(result, because='expired')
self.assertFalse(os.path.exists(dest), 'expired certificate was published')
# And the same material with a valid certificate IS publishable, so the
# refusal above is about expiry, not about the fixture.
good_cert = write(os.path.join(self.tmp, 'good.crt'), san_fixture('new_bundle.crt'))
good_key = write(os.path.join(self.tmp, 'good.key'), san_fixture('new_bundle.key'))
ok = self.call('cert_publish "$1" "$2" "$3"', good_cert, good_key, dest)
self.assertEqual(0, ok.returncode, ok.stdout + ok.stderr)
self.assertTrue(os.path.exists(dest))
def test_empty_and_missing_bundles_rejected(self): def test_empty_and_missing_bundles_rejected(self):
empty = write(os.path.join(self.tmp, 'empty.pem'), '') empty = write(os.path.join(self.tmp, 'empty.pem'), '')
self.assert_rejected(self.call('cert_bundle_valid "$1"', empty), self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),
+396 -7
View File
@@ -57,6 +57,7 @@ import subprocess
import sys import sys
import tempfile import tempfile
import textwrap import textwrap
import time
import unittest import unittest
BROKEN_TOKEN = '__BROKEN__' BROKEN_TOKEN = '__BROKEN__'
@@ -74,6 +75,9 @@ sys.path.insert(0, MODULE_DIR)
# The module opens /var/log/haproxy-manager.log at import time via # The module opens /var/log/haproxy-manager.log at import time via
# logging.FileHandler. Redirect that one call so the suite runs unprivileged. # logging.FileHandler. Redirect that one call so the suite runs unprivileged.
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-') _LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
# spawn_haproxy_master() appends the master's stderr here.
os.environ.setdefault('HAPROXY_STDERR_LOG', os.path.join(_LOG_DIR, 'haproxy-stderr.log'))
os.environ.setdefault('HAPROXY_START_WAIT', '3')
_real_file_handler = logging.FileHandler _real_file_handler = logging.FileHandler
logging.FileHandler = ( logging.FileHandler = (
lambda fn, *a, **kw: _real_file_handler( lambda fn, *a, **kw: _real_file_handler(
@@ -224,8 +228,9 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
# haproxy -c -f FILE -> reject FILE containing {BROKEN_TOKEN} # haproxy -c -f FILE -> reject FILE containing {BROKEN_TOKEN}
# reject any unusable file in $TEST_CERTS_DIR # reject any unusable file in $TEST_CERTS_DIR
cfg="" cfg=""
start=0
while [ $# -gt 0 ]; do while [ $# -gt 0 ]; do
case "$1" in -f) cfg="$2"; shift ;; esac case "$1" in -f) cfg="$2"; shift ;; -W) start=1 ;; esac
shift shift
done done
if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then
@@ -249,11 +254,16 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
fi fi
done done
fi fi
# real `haproxy -W -db` prints this once the config is loaded
[ "$start" = 1 ] && echo "[NOTICE] (1) : Loading success." >&2
exit 0 exit 0
""") """)
# certbot stub: succeeds, announces a renewal, and records every invocation so # certbot stub: succeeds, announces a renewal, and records every invocation so
# tests can assert that `certbot delete` did or did not run. # tests can assert that `certbot delete` did or did not run.
# TEST_CERTBOT_INSTALL_SRC/_DIR: on certonly, copy fullchain.pem + privkey.pem
# from SRC into DIR (a real issuance changes the lineage's certificate).
# TEST_CERTBOT_NOT_DUE: on certonly, print certbot's "not yet due" wording.
FAKE_CERTBOT = textwrap.dedent("""\ FAKE_CERTBOT = textwrap.dedent("""\
#!/bin/sh #!/bin/sh
if [ -n "$TEST_CERTBOT_LOG" ]; then if [ -n "$TEST_CERTBOT_LOG" ]; then
@@ -262,6 +272,14 @@ FAKE_CERTBOT = textwrap.dedent("""\
case "$1" in case "$1" in
renew) echo "Congratulations, all renewals succeeded" ;; renew) echo "Congratulations, all renewals succeeded" ;;
delete) [ -n "$TEST_CERTBOT_DELETE_FAILS" ] && exit 1 ;; delete) [ -n "$TEST_CERTBOT_DELETE_FAILS" ] && exit 1 ;;
certonly)
if [ -n "$TEST_CERTBOT_INSTALL_SRC" ]; then
mkdir -p "$TEST_CERTBOT_INSTALL_DIR"
cp "$TEST_CERTBOT_INSTALL_SRC/fullchain.pem" "$TEST_CERTBOT_INSTALL_SRC/privkey.pem" "$TEST_CERTBOT_INSTALL_DIR/"
fi
if [ -n "$TEST_CERTBOT_NOT_DUE" ]; then
echo "Certificate not yet due for renewal; no action taken."
fi ;;
esac esac
exit 0 exit 0
""") """)
@@ -331,6 +349,7 @@ class CertPublishTestCase(unittest.TestCase):
'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'), 'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'),
'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'), 'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'),
'SSL_CERTS_DIR': self.certs, 'SSL_CERTS_DIR': self.certs,
'LETSENCRYPT_DIR': os.path.join(self.tmp, 'letsencrypt'),
'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'), 'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'),
'API_KEY': None, 'API_KEY': None,
} }
@@ -574,7 +593,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
@FIX_ONLY @FIX_ONLY
@NEEDS_OPENSSL @NEEDS_OPENSSL
def test_lineage_is_deleted_only_after_haproxy_loads_the_bundle(self): def test_lineage_is_untouched_when_haproxy_does_not_load_the_bundle(self):
"""A reload failure must leave the lineage intact and the file recoverable.""" """A reload failure must leave the lineage intact and the file recoverable."""
old_path = self._setup_supersede(broken=False) old_path = self._setup_supersede(broken=False)
# Make generate_config() produce a config the validator rejects, so the # Make generate_config() produce a config the validator rejects, so the
@@ -591,7 +610,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
[], self.certbot_deletes(), [], self.certbot_deletes(),
'the lineage must not be deleted when HAProxy did not reload') 'the lineage must not be deleted when HAProxy did not reload')
self.assertFalse(os.path.exists(old_path)) self.assertFalse(os.path.exists(old_path))
quarantined = os.path.join(hm.cert_backup_dir(), quarantined = os.path.join(hm.cert_retired_dir(),
os.path.basename(old_path)) os.path.basename(old_path))
self.assertTrue(os.path.exists(quarantined), self.assertTrue(os.path.exists(quarantined),
'the superseded file must be recoverable by hand') 'the superseded file must be recoverable by hand')
@@ -599,8 +618,9 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
@FIX_ONLY @FIX_ONLY
@NEEDS_OPENSSL @NEEDS_OPENSSL
def test_successful_bundle_still_supersedes_and_deletes(self): def test_successful_bundle_still_supersedes_without_deleting(self):
"""The cleanup must still do its job on the happy path.""" """The cleanup must still do its job on the happy path - by moving, never
by `certbot delete` (irreversible; recovery is a rate-limited ACME order)."""
old_path = self._setup_supersede(broken=False) old_path = self._setup_supersede(broken=False)
resp = self.client.post('/api/ssl/bundle', json={ resp = self.client.post('/api/ssl/bundle', json={
@@ -612,12 +632,381 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
self.assertFalse(os.path.exists(old_path), self.assertFalse(os.path.exists(old_path),
'the superseded file must leave the crt directory or ' 'the superseded file must leave the crt directory or '
'it keeps shadowing the new bundle') 'it keeps shadowing the new bundle')
self.assertEqual(['delete --cert-name www.test.example.com -n'], self.assertTrue(os.path.exists(os.path.join(
self.certbot_deletes()) hm.cert_retired_dir(), os.path.basename(old_path))),
'the superseded file must be moved aside, not destroyed')
self.assertEqual([], self.certbot_deletes(),
'retirement must never `certbot delete` anything')
self.assertTrue(self.edge_would_start()) self.assertTrue(self.edge_would_start())
self.assert_only_final_pems_in_certs_dir() self.assert_only_final_pems_in_certs_dir()
# --- SAN-overlap retirement (brainjar.net, 2026-10-07) -----------------------
SAN_FIXTURES = os.path.join(MODULE_DIR, 'scripts', 'test-fixtures', 'san-certs')
def san(name):
with open(os.path.join(SAN_FIXTURES, name)) as fh:
return fh.read()
def make_certbot_lineage(le_dir, lineage, cert, key):
"""archive/<x>/{cert,chain,fullchain,privkey}1.pem + live/<x>/*.pem
symlinks (../../archive/<x>/...1.pem) + renewal/<x>.conf, like certbot."""
archive = os.path.join(le_dir, 'archive', lineage)
live = os.path.join(le_dir, 'live', lineage)
os.makedirs(archive, exist_ok=True)
os.makedirs(live, exist_ok=True)
for kind, body in (('cert', cert), ('chain', cert), ('fullchain', cert), ('privkey', key)):
with open(os.path.join(archive, f'{kind}1.pem'), 'w') as fh:
fh.write(body)
link = os.path.join(live, f'{kind}.pem')
if os.path.lexists(link):
os.remove(link)
os.symlink(f'../../archive/{lineage}/{kind}1.pem', link)
os.makedirs(os.path.join(le_dir, 'renewal'), exist_ok=True)
with open(os.path.join(le_dir, 'renewal', f'{lineage}.conf'), 'w') as fh:
fh.write(f'archive_dir = {archive}\ncert = {live}/cert.pem\n')
return live
def retired_path(le_dir, part, name):
hits = sorted(__import__('glob').glob(os.path.join(le_dir, 'retired', '*', part, name)))
return hits[-1] if hits else os.path.join(le_dir, 'retired', '<none>', part, name)
FIX_ONLY_RETIRE = unittest.skipIf(
TESTING_FOREIGN_TREE and not hasattr(hm, 'retirement_blockers'),
'HAPROXY_MANAGER_DIR points at a tree without SAN-overlap retirement')
class TestRetirementRules(unittest.TestCase):
"""retirement_blockers() is the shared-PEM safety rule, tested directly."""
NOW = 2_000_000_000
def cert(self, names, days_left, fp):
return {'names': set(names), 'fingerprint': fp,
'not_after': self.NOW + days_left * 86400, 'cn': sorted(names)[0]}
@FIX_ONLY_RETIRE
def test_expired_candidate_never_blocks(self):
old = self.cert(['brain-jar.com', 'brainjar.net'], -2, 'A')
self.assertEqual([], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'brain-jar.com', 'brainjar.net'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_valid_candidate_that_is_last_cert_for_a_served_name_is_kept(self):
old = self.cert(['shared.example', 'brainjar.net', 'other.example'], 40, 'A')
self.assertEqual(['other.example'], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'brainjar.net', 'other.example'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_unreachable_names_do_not_block(self):
old = self.cert(['dead.example', 'brainjar.net'], 40, 'A')
self.assertEqual([], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'brainjar.net', 'dead.example'},
['dead.example'], now=self.NOW))
@FIX_ONLY_RETIRE
def test_name_covered_by_another_valid_cert_does_not_block(self):
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
other = self.cert(['other.example'], 60, 'B')
self.assertEqual([], hm.retirement_blockers(
old, [other], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_cover_by_an_expired_or_identical_cert_does_not_count(self):
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
expired = self.cert(['other.example'], -1, 'B')
same = self.cert(['other.example'], 60, 'A')
self.assertEqual(['other.example'], hm.retirement_blockers(
old, [expired, same], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_unknown_served_set_keeps_every_valid_name(self):
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
self.assertEqual(['other.example'], hm.retirement_blockers(
old, [], {'brainjar.net'}, None, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_wildcard_coverage(self):
self.assertTrue(hm.cert_name_covers('*.example.com', 'a.example.com'))
self.assertFalse(hm.cert_name_covers('*.example.com', 'a.b.example.com'))
self.assertFalse(hm.cert_name_covers('*.example.com', 'example.com'))
old = self.cert(['*.example.com', 'brainjar.net'], 40, 'A')
self.assertEqual(['shop.example.com'], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'shop.example.com'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_read_cert_info_reads_sans_and_expiry(self):
tmp = tempfile.mkdtemp(prefix='cert-info-')
self.addCleanup(shutil.rmtree, tmp, True)
path = os.path.join(tmp, 'c.pem')
with open(path, 'w') as fh:
fh.write(san('expired_old.crt'))
info = hm.read_cert_info(path)
self.assertEqual({'brain-jar.com', 'www.brain-jar.com', 'brainjar.net',
'www.brainjar.net'}, info['names'])
self.assertEqual('brain-jar.com', info['cn'])
self.assertEqual(1585699200, info['not_after']) # 2020-04-01T00:00:00Z
self.assertTrue(info['fingerprint'])
class TestSanOverlapRetirement(CertPublishTestCase):
"""The new bundle must not stay shadowed by an older cert sharing a SAN."""
def setUp(self):
super().setUp()
self.le = os.path.join(self.tmp, 'letsencrypt')
self.new_src = os.path.join(self.tmp, 'issued', 'brainjar.net')
self.write(os.path.join(self.new_src, 'fullchain.pem'), san('new_bundle.crt'))
self.write(os.path.join(self.new_src, 'privkey.pem'), san('new_bundle.key'))
# certbot "issues" brainjar.net into live/ when called.
os.environ['TEST_CERTBOT_INSTALL_SRC'] = self.new_src
os.environ['TEST_CERTBOT_INSTALL_DIR'] = os.path.join(self.le_live, 'brainjar.net')
for name in ('brainjar.net', 'www.brainjar.net'):
self.add_domain(name, name.replace('.', '_'))
def old_lineage(self, lineage, fixture, publish=True):
"""A lineage laid out the way certbot does it: real files in
archive/<x>/*1.pem, relative symlinks in live/<x>/, a renewal conf."""
live = make_certbot_lineage(self.le, lineage, san(f'{fixture}.crt'),
san(f'{fixture}.key'))
conf = os.path.join(self.le, 'renewal', f'{lineage}.conf')
if not publish:
return live, conf, None
pem = self.write(os.path.join(self.certs, f'{lineage}.pem'),
san(f'{fixture}.crt') + san(f'{fixture}.key'))
return live, conf, pem
def bundle(self, **extra):
body = {'primary': 'brainjar.net', 'sans': ['www.brainjar.net']}
body.update(extra)
resp = self.client.post('/api/ssl/bundle', json=body)
self.assertEqual(200, resp.status_code, resp.get_data(as_text=True))
return resp.get_json()
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_expired_overlapping_cert_with_a_different_cn_is_retired(self):
"""The incident: CN brain-jar.com, SANs include brainjar.net."""
self.add_domain('brain-jar.com', 'bjc')
self.add_domain('www.brain-jar.com', 'wbjc')
live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
data = self.bundle(unreachable_names=['brain-jar.com', 'www.brain-jar.com'])
self.assertFalse(os.path.exists(pem),
'the expired SAN-overlapping pem is still shadowing the bundle')
retired = [e['file'] for e in data['cleanup']['removed']]
self.assertEqual(['brain-jar.com.pem'], retired)
self.assertTrue(os.path.exists(os.path.join(hm.cert_retired_dir(), 'brain-jar.com.pem')))
# Lineage retired, not deleted: renew-certificates.sh can no longer
# re-publish it and certbot can no longer try to renew it.
self.assertFalse(os.path.exists(live))
self.assertFalse(os.path.exists(conf))
# live, archive AND renewal leave together (review H2), into one tree.
self.assertFalse(os.path.exists(os.path.join(self.le, 'archive', 'brain-jar.com')))
self.assertTrue(os.path.isdir(retired_path(self.le, 'live', 'brain-jar.com')))
self.assertTrue(os.path.isdir(retired_path(self.le, 'archive', 'brain-jar.com')))
self.assertTrue(os.path.exists(retired_path(self.le, 'renewal', 'brain-jar.com.conf')))
# Relative symlinks still resolve in the retired tree (restore = mv back).
self.assertIn('BEGIN CERTIFICATE', self.read(
os.path.join(retired_path(self.le, 'live', 'brain-jar.com'), 'cert.pem')))
self.assertEqual([], self.certbot_deletes())
self.assertTrue(os.path.exists(os.path.join(self.certs, 'brainjar.net.pem')))
self.assertTrue(self.edge_would_start())
self.assert_only_final_pems_in_certs_dir()
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_expired_overlap_is_retired_even_without_the_unreachable_hint(self):
self.add_domain('brain-jar.com', 'bjc')
live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
self.bundle()
self.assertFalse(os.path.exists(pem))
self.assertFalse(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_shared_pem_still_serving_another_site_is_kept(self):
"""The shared-PEM hazard: never retire the only valid cert for a served name."""
self.add_domain('other-site.example', 'other_site')
live, conf, pem = self.old_lineage('shared-old.example', 'shared_old')
data = self.bundle()
self.assertTrue(os.path.exists(pem),
'retired the only valid certificate for other-site.example')
self.assertTrue(os.path.exists(live))
self.assertTrue(os.path.exists(conf))
kept = {e['file']: e['blockers'] for e in data['cleanup']['kept']}
self.assertEqual({'shared-old.example.pem': ['other-site.example']}, kept)
self.assertEqual(['other-site.example'],
data['cleanup']['lineages']['kept'][0]['blockers'])
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_valid_overlap_whose_other_names_are_unreachable_is_retired(self):
self.add_domain('dead.example', 'dead')
live, conf, pem = self.old_lineage('dead.example', 'dead_old')
self.bundle(unreachable_names=['dead.example'])
self.assertFalse(os.path.exists(pem))
self.assertFalse(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_valid_overlap_still_served_without_the_hint_is_kept(self):
"""Same files as above, but nobody said dead.example is gone: keep it."""
self.add_domain('dead.example', 'dead')
live, conf, pem = self.old_lineage('dead.example', 'dead_old')
self.bundle()
self.assertTrue(os.path.exists(pem))
self.assertTrue(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_non_overlapping_certificates_are_untouched(self):
live, conf, pem = self.old_lineage('unrelated.example', 'unrelated')
self.bundle()
self.assertTrue(os.path.exists(pem), 'retired a cert sharing no name with the bundle')
self.assertTrue(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_response_distinguishes_issuance_from_not_yet_due(self):
issued = self.bundle()
self.assertTrue(issued['renewed'], 'a fresh issuance was not reported as one')
self.assertEqual(4922899200, issued['not_after']) # 2126-01-01T00:00:00Z
# Lineage now exists with the same names; certbot keeps it.
del os.environ['TEST_CERTBOT_INSTALL_SRC']
os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
noop = self.bundle()
self.assertFalse(noop['renewed'], 'a certbot no-op was reported as a renewal')
self.assertTrue(noop['certbot_not_due'])
self.assertEqual(4922899200, noop['not_after'])
self.assertIn('--keep-until-expiring', self.certbot_invocations()[-1])
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_not_due_noop_never_retires_its_own_cert_name_lineage(self):
"""Review H1: find_certbot_live_dir() returns the NEWEST of brainjar.net
and brainjar.net-0001. After a certbot no-op that was the stale -0001:
it got published as brainjar.net.pem and the real brainjar.net lineage
was retired as "superseded"."""
del os.environ['TEST_CERTBOT_INSTALL_SRC']
os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
real = make_certbot_lineage(self.le, 'brainjar.net', san('new_bundle.crt'), san('new_bundle.key'))
stale = make_certbot_lineage(self.le, 'brainjar.net-0001', san('dead_old.crt'), san('dead_old.key'))
later = time.time() + 60
os.utime(os.path.join(self.le, 'archive', 'brainjar.net-0001', 'fullchain1.pem'), (later, later))
# Production behaviour of find_certbot_live_dir(): newest wins.
hm.find_certbot_live_dir = lambda base: stale if base == 'brainjar.net' else None
data = self.bundle()
self.assertTrue(os.path.isdir(real), 'the --cert-name lineage itself was retired')
self.assertTrue(os.path.exists(os.path.join(self.le, 'renewal', 'brainjar.net.conf')))
self.assertNotIn('brainjar.net', [e['lineage'] for e in data['cleanup']['lineages']['retired']])
published = self.read(os.path.join(self.certs, 'brainjar.net.pem'))
self.assertIn(san('new_bundle.crt').strip().splitlines()[1], published,
'published the stale -0001 certificate instead of live/brainjar.net')
self.assertFalse(data['renewed'])
@FIX_ONLY_RETIRE
def test_bundle_reports_capabilities(self):
data = self.bundle()
self.assertIn('san_overlap_retirement', data.get('capabilities', []))
caps = self.client.get('/api/capabilities').get_json()
self.assertIn('unreachable_names', caps['capabilities'])
class TestRetiredLineageCanBeReissued(CertPublishTestCase):
"""Review H2: retirement must leave certbot able to issue the name again."""
def setUp(self):
super().setUp()
self.le = hm.LETSENCRYPT_DIR
@FIX_ONLY_RETIRE
def test_retirement_removes_every_part_certbot_checks(self):
make_certbot_lineage(self.le, 'brain-jar.com', san('expired_old.crt'), san('expired_old.key'))
out = hm.retire_lineage_tree('brain-jar.com')
# certbot new_lineage() refuses if any of these exist for the name.
for part in (('live', 'brain-jar.com'), ('archive', 'brain-jar.com'),
('renewal', 'brain-jar.com.conf')):
self.assertFalse(os.path.lexists(os.path.join(self.le, *part)), part)
self.assertTrue(os.path.lexists(os.path.join(out['retired_to'], *part)), part)
self.assertIn('BEGIN CERTIFICATE',
self.read(os.path.join(out['retired_to'], 'live', 'brain-jar.com', 'fullchain.pem')))
@FIX_ONLY_RETIRE
def test_second_retirement_of_the_same_name_does_not_collide(self):
make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
a = hm.retire_lineage_tree('x.example', stamp='S')
make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
b = hm.retire_lineage_tree('x.example', stamp='S')
self.assertNotEqual(a['retired_to'], b['retired_to'])
self.assertTrue(os.path.isdir(os.path.join(a['retired_to'], 'archive', 'x.example')))
@FIX_ONLY_RETIRE
def test_certbot_can_issue_the_name_again_after_retirement(self):
"""Against certbot's own storage code (skipped where certbot is not
importable, e.g. the image's app python)."""
# Import AND initialisation are both guarded: a half-installed or
# version-skewed certbot can fail with far more than ImportError (a
# missing transitive dependency, a pkg_resources/entry-point error, a
# moved internal). Any of those means "cannot run this check here",
# not "retirement is broken". new_lineage() itself is NOT inside the
# guard - that is the assertion.
try:
import types
from certbot._internal import storage
real_rv = storage.relevant_values
new_lineage = storage.RenewableCert.new_lineage
cfg = types.SimpleNamespace(
config_dir=self.le, renewal_configs_dir=os.path.join(self.le, 'renewal'),
default_archive_dir=os.path.join(self.le, 'archive'),
live_dir=os.path.join(self.le, 'live'))
except Exception as e: # noqa: BLE001 - deliberately broad, see above
self.skipTest(f'certbot unusable from this python ({type(e).__name__}: {e})')
storage.relevant_values = lambda c: {'server': 'https://acme.invalid/directory'}
self.addCleanup(setattr, storage, 'relevant_values', real_rv)
cert, key = san('new_bundle.crt').encode(), san('new_bundle.key').encode()
first = new_lineage('brain-jar.com', cert, key, cert, cfg)
self.assertEqual('brain-jar.com', first.lineagename)
hm.retire_lineage_tree('brain-jar.com')
again = new_lineage('brain-jar.com', cert, key, cert, cfg)
self.assertEqual('brain-jar.com', again.lineagename,
'certbot could not re-create the lineage under the same name')
self.assertTrue(os.path.getsize(os.path.join(self.le, 'renewal', 'brain-jar.com.conf')) > 0)
class TestExpiredCertificateIsNeverPublished(CertPublishTestCase):
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_publisher_refuses_an_expired_leaf(self):
src = os.path.join(self.tmp, 'expired')
cert = self.write(os.path.join(src, 'fullchain.pem'), san('expired_old.crt'))
key = self.write(os.path.join(src, 'privkey.pem'), san('expired_old.key'))
live = self.publish_live_bundle('brain-jar.com')
with self.assertRaises(hm.CertificatePublishError) as ctx:
hm.publish_pem_bundle(live, [cert, key])
self.assertIn('expired', str(ctx.exception))
self.assertEqual(PREVIOUS_BUNDLE, self.read(live))
class TestBundleValidation(CertPublishTestCase): class TestBundleValidation(CertPublishTestCase):
"""What may and may not be published.""" """What may and may not be published."""
+9 -1
View File
@@ -56,6 +56,10 @@ sys.path.insert(0, MODULE_DIR)
# The module opens /var/log/haproxy-manager.log at import time via # The module opens /var/log/haproxy-manager.log at import time via
# logging.FileHandler. Redirect that one call so the suite runs unprivileged. # logging.FileHandler. Redirect that one call so the suite runs unprivileged.
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-') _LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
# spawn_haproxy_master() appends the master's stderr here (default is the
# haproxy-logs volume path, not writable unprivileged).
os.environ.setdefault('HAPROXY_STDERR_LOG', os.path.join(_LOG_DIR, 'haproxy-stderr.log'))
os.environ.setdefault('HAPROXY_START_WAIT', '3')
_real_file_handler = logging.FileHandler _real_file_handler = logging.FileHandler
logging.FileHandler = ( logging.FileHandler = (
lambda fn, *a, **kw: _real_file_handler( lambda fn, *a, **kw: _real_file_handler(
@@ -80,14 +84,18 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
# haproxy -c -f FILE -> exit 1 if FILE contains {BROKEN_TOKEN}, else 0 # haproxy -c -f FILE -> exit 1 if FILE contains {BROKEN_TOKEN}, else 0
# haproxy -W -S ... -f FILE (start) -> same validation, then exit 0 # haproxy -W -S ... -f FILE (start) -> same validation, then exit 0
cfg="" cfg=""
start=0
while [ $# -gt 0 ]; do while [ $# -gt 0 ]; do
case "$1" in -f) cfg="$2"; shift ;; esac case "$1" in -f) cfg="$2"; shift ;; -W) start=1 ;; esac
shift shift
done done
if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then
echo "[ALERT] parsing [$cfg:1] : unknown keyword '{BROKEN_TOKEN}'" >&2 echo "[ALERT] parsing [$cfg:1] : unknown keyword '{BROKEN_TOKEN}'" >&2
exit 1 exit 1
fi fi
# A real `haproxy -W -db` prints this once the config is loaded;
# spawn_haproxy_master() waits for it.
[ "$start" = 1 ] && echo "[NOTICE] (1) : Loading success." >&2
exit 0 exit 0
""") """)
+304
View File
@@ -0,0 +1,304 @@
#!/usr/bin/env python3
"""Regression tests for the default splash (unrouted hosts answered pre-WAF).
Why this file exists
--------------------
Visiting a WHP server's own hostname (https://whpNN.<domain>/ on 80/443, not
the panel port), its bare IP, or any name with no site behind it used to fall
through to `default_backend default-backend` -- but only AFTER every
http-request rule had run, Coraza's send-spoe-group included. `use_backend`
does not short-circuit http-request processing. So the catch-all was WAF
inspected with the server-wide rule set, and a browser on the server hostname
also carries the control panel's cookies (cookies are not port-scoped), which
produced the WAF block page for a host that has nothing behind it to protect.
The splash answers those requests with `http-request return` BEFORE the SPOE
call. Its safety property is that it must NEVER answer a host that routes to
a site. That is held by:
- routed_hosts.list / routed_wildcards.list being written from the same
rows, in the same generate_config() call, as the routing ACLs
- the lists being port-stripped + lowercased (a superset of the raw-Host
routing ACLs)
- the lists being part of the rollback set
and it must keep ACME http-01, the blocked-IP page and the suspended page.
Running
-------
python3 scripts/test-default-splash.py
"""
import logging
import os
import re
import sqlite3
import sys
import tempfile
import unittest
MODULE_DIR = os.path.abspath(
os.environ.get('HAPROXY_MANAGER_DIR',
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
)
os.chdir(MODULE_DIR)
sys.path.insert(0, MODULE_DIR)
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
_real_file_handler = logging.FileHandler
logging.FileHandler = (
lambda filename, *a, **kw: _real_file_handler(
os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
)
try:
import haproxy_manager as hm # noqa: E402
finally:
logging.FileHandler = _real_file_handler
logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
SPLASH_FILE = os.path.join(MODULE_DIR, 'errors', 'default-splash.html')
def render_listener(**kw):
args = dict(crt_path='/etc/haproxy/certs', suspension_enabled=False,
coraza_spoe_backend='127.0.0.1:9000', default_splash=True)
args.update(kw)
return hm.template_env.get_template('hap_listener.tpl').render(**args)
def return_rules(cfg):
return [l.strip() for l in cfg.splitlines()
if l.strip().startswith('http-request return')
and ('default-splash.html' in l or 'robots-unrouted.txt' in l)]
class Template(unittest.TestCase):
def test_off_renders_nothing(self):
cfg = render_listener(default_splash=False)
self.assertNotIn('default-splash', cfg)
self.assertNotIn('splash_routed_host', cfg)
def test_404_everywhere_robots_txt_200(self):
"""Josh 2026-11-08: unrouted hosts are 404 on EVERY path (incl. "/")
so search engines never index the server IP/hostname; /robots.txt is
the one 200 and disallows everything."""
rules = return_rules(render_listener())
self.assertEqual(len(rules), 2, rules)
robots, splash = rules
self.assertTrue(robots.startswith('http-request return status 200 content-type "text/plain'))
self.assertIn(' file /haproxy/errors/robots-unrouted.txt ', robots)
self.assertTrue(robots.endswith(' splash_robots'))
self.assertTrue(splash.startswith('http-request return status 404 content-type "text/html'))
self.assertTrue(splash.endswith(' !splash_robots'))
self.assertNotIn('splash_root', render_listener())
self.assertRegex(render_listener(), r'acl splash_robots path /robots\.txt\n')
def test_noindex_and_no_store_headers_on_both(self):
for rule in return_rules(render_listener()):
self.assertIn('hdr x-robots-tag "noindex, nofollow, noarchive"', rule)
self.assertIn('hdr cache-control "no-store"', rule)
self.assertIn('hdr x-content-type-options "nosniff"', rule)
def test_static_file_not_lf_file(self):
"""`file` (verbatim), never `lf-file`: nothing from the request may be
expanded into the page (Host echo, unique-id, ...)."""
for rule in return_rules(render_listener()):
self.assertRegex(rule, r' file /haproxy/errors/(default-splash\.html|robots-unrouted\.txt) ')
self.assertNotIn('lf-file', rule)
def test_conditions_keep_acme_blocked_and_suspended_out(self):
for susp in (False, True):
for rule in return_rules(render_listener(suspension_enabled=susp)):
cond = rule.split(' if ', 1)[1].split()
self.assertIn('!splash_routed_host', cond)
self.assertIn('!splash_acme', cond)
self.assertIn('!is_blocked_ip', cond)
if susp:
self.assertIn('!is_suspended_domain', cond)
else:
self.assertNotIn('!is_suspended_domain', cond)
def test_acme_acl_is_the_challenge_prefix(self):
self.assertRegex(render_listener(),
r'acl splash_acme path_beg /\.well-known/acme-challenge/\n')
def test_routed_acls_strip_port_and_lowercase(self):
cfg = render_listener()
self.assertIn('acl splash_routed_host req.hdr(host),field(1,:),lower '
'-f /etc/haproxy/routed_hosts.list', cfg)
self.assertIn('acl splash_routed_host req.hdr(host),field(1,:),lower '
'-m end -f /etc/haproxy/routed_wildcards.list', cfg)
def test_splash_runs_before_waf_and_after_ip_block_and_suspension(self):
for susp in (False, True):
cfg = render_listener(suspension_enabled=susp)
first_return = cfg.index('file /haproxy/errors/robots-unrouted.txt')
self.assertLess(first_return, cfg.index('http-request send-spoe-group coraza'))
self.assertLess(first_return, cfg.index('filter spoe engine coraza'))
self.assertGreater(first_return, cfg.index('acl is_blocked_ip'))
self.assertGreater(first_return, cfg.index('http-request track-sc0'))
if susp:
self.assertGreater(first_return, cfg.index('acl is_suspended_domain'))
class Page(unittest.TestCase):
def setUp(self):
with open(SPLASH_FILE, 'rb') as fh:
self.raw = fh.read()
self.text = self.raw.decode('utf-8')
def test_small_enough_for_one_buffer(self):
# http-request return payload must fit tune.bufsize (16384) with headers.
self.assertLess(len(self.raw), 8192)
def test_no_links_scripts_or_external_resources(self):
low = self.text.lower()
for needle in ('href', 'src=', '<script', '<form', '<a ', 'http://', 'https://',
'8443', '@import', 'url('):
self.assertNotIn(needle, low, needle)
def test_no_version_or_template_expansions(self):
self.assertNotRegex(self.text, r'20\d\d\.\d+\.\d+')
self.assertNotIn('{{', self.text)
self.assertNotIn('%[', self.text)
class Robots(unittest.TestCase):
def test_robots_disallows_everything(self):
with open(os.path.join(MODULE_DIR, 'errors', 'robots-unrouted.txt')) as fh:
self.assertEqual(fh.read(), 'User-agent: *\nDisallow: /\n')
class RoutedLists(unittest.TestCase):
def test_render_sorted_dedup_lower(self):
body = hm.render_routed_list(['B.example.com', 'a.example.com', 'b.example.com ', ''])
lines = [l for l in body.splitlines() if not l.startswith('#')]
self.assertEqual(lines, ['a.example.com', 'b.example.com'])
def test_name_with_whitespace_is_dropped_not_split(self):
body = hm.render_routed_list(['ok.example.com', 'bad\nevil.example.com', 'a b'])
lines = [l for l in body.splitlines() if not l.startswith('#')]
self.assertEqual(lines, ['ok.example.com'])
def test_lists_are_in_the_rollback_set(self):
pairs = dict(hm._config_backup_pairs())
self.assertIn(hm.ROUTED_HOSTS_PATH, pairs)
self.assertIn(hm.ROUTED_WILDCARDS_PATH, pairs)
class GenerateConfigWritesLists(unittest.TestCase):
"""Run the real generate_config() with paths redirected and reload stubbed."""
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix='splash-test-')
self.saved = {k: getattr(hm, k) for k in (
'DB_FILE', 'HAPROXY_CONFIG_PATH', 'HAPROXY_BACKUP_PATH',
'CLUSTER_SECRET_PATH', 'HAPROXY_SOCKET_PATH', 'SSL_CERTS_DIR',
'BLOCKED_IPS_MAP_PATH', 'BLOCKED_IPS_MAP_BACKUP_PATH',
'CORAZA_SPOE_CONFIG_PATH', 'CORAZA_SPOE_BACKUP_PATH',
'ROUTED_HOSTS_PATH', 'ROUTED_HOSTS_BACKUP_PATH',
'ROUTED_WILDCARDS_PATH', 'ROUTED_WILDCARDS_BACKUP_PATH',
'reload_haproxy_safely', 'create_backup')}
j = lambda n: os.path.join(self.tmp, n) # noqa: E731
hm.DB_FILE = j('db.sqlite')
hm.HAPROXY_CONFIG_PATH = j('haproxy.cfg')
hm.HAPROXY_BACKUP_PATH = j('haproxy.cfg.backup')
hm.CLUSTER_SECRET_PATH = j('cluster-secret')
hm.HAPROXY_SOCKET_PATH = j('haproxy.sock')
hm.SSL_CERTS_DIR = j('certs')
hm.BLOCKED_IPS_MAP_PATH = j('blocked_ips.map')
hm.BLOCKED_IPS_MAP_BACKUP_PATH = j('blocked_ips.map.backup')
hm.CORAZA_SPOE_CONFIG_PATH = j('coraza-spoe.cfg')
hm.CORAZA_SPOE_BACKUP_PATH = j('coraza-spoe.cfg.backup')
hm.ROUTED_HOSTS_PATH = j('routed_hosts.list')
hm.ROUTED_HOSTS_BACKUP_PATH = j('routed_hosts.list.backup')
hm.ROUTED_WILDCARDS_PATH = j('routed_wildcards.list')
hm.ROUTED_WILDCARDS_BACKUP_PATH = j('routed_wildcards.list.backup')
hm.reload_haproxy_safely = lambda *a, **kw: (True, 'stubbed')
hm.create_backup = lambda *a, **kw: (True, 'unavailable')
self.env = {k: os.environ.pop(k, None) for k in (
'HAPROXY_DEFAULT_SPLASH', 'HAPROXY_SUSPENSION_ENABLED',
'HAPROXY_CORAZA_SPOE_BACKEND')}
hm.init_db()
with sqlite3.connect(hm.DB_FILE) as conn:
cur = conn.cursor()
def add(domain, backend, wildcard=0):
cur.execute('INSERT INTO domains (domain, ssl_enabled, ssl_cert_path, '
'template_override, is_wildcard) VALUES (?, 1, NULL, NULL, ?)',
(domain, wildcard))
did = cur.lastrowid
if backend:
cur.execute('INSERT INTO backends (name, domain_id, settings) VALUES (?, ?, NULL)',
(backend, did))
cur.execute('INSERT INTO backend_servers (backend_id, server_name, '
'server_address, server_port, server_options) VALUES (?, ?, ?, ?, ?)',
(cur.lastrowid, 'web1', '10.0.0.10', 80, 'check'))
add('Site.Example.com', 'site')
add('*.wild.example.com', 'wild', wildcard=1)
add('whp01.example.net', None) # server hostname: cert-only
conn.commit()
def tearDown(self):
for k, v in self.saved.items():
setattr(hm, k, v)
for k, v in self.env.items():
os.environ.pop(k, None)
if v is not None:
os.environ[k] = v
import shutil
shutil.rmtree(self.tmp, ignore_errors=True)
def _lines(self, path):
with open(path) as fh:
return [l for l in fh.read().splitlines() if l and not l.startswith('#')]
def test_lists_written_from_routed_rows_only(self):
hm.generate_config()
self.assertEqual(self._lines(hm.ROUTED_HOSTS_PATH), ['site.example.com'])
self.assertEqual(self._lines(hm.ROUTED_WILDCARDS_PATH), ['.wild.example.com'])
with open(hm.HAPROXY_CONFIG_PATH) as fh:
cfg = fh.read()
# every exact routing ACL name is in the list (superset property)
for name in re.findall(r'acl \S+-acl hdr\(host\) -i (\S+)', cfg):
self.assertIn(name.lower(), self._lines(hm.ROUTED_HOSTS_PATH))
for base in re.findall(r'acl \S+-acl hdr_end\(host\) -i (\S+)', cfg):
self.assertIn(base.lower(), self._lines(hm.ROUTED_WILDCARDS_PATH))
self.assertIn('default-splash.html', cfg)
def test_default_log_target_is_the_in_container_socket(self):
"""Access log goes to the receiver's UNIX socket (chroot-relative),
never the old host bridge target 172.18.0.1:514."""
os.environ.pop('HAPROXY_SYSLOG_TARGET', None)
hm.generate_config()
with open(hm.HAPROXY_CONFIG_PATH) as fh:
cfg = fh.read()
self.assertRegex(cfg, r'\n log /log\.sock len 2048 format rfc5424 local2 info\n')
self.assertIn('chroot /var/lib/haproxy', cfg)
log_lines = [l for l in cfg.splitlines() if l.strip().startswith('log ')
and 'global' not in l]
self.assertEqual(len(log_lines), 1, log_lines)
self.assertNotIn('172.18.0.1', log_lines[0])
def test_log_target_env_override(self):
os.environ['HAPROXY_SYSLOG_TARGET'] = '192.0.2.5:514'
try:
hm.generate_config()
finally:
os.environ.pop('HAPROXY_SYSLOG_TARGET', None)
with open(hm.HAPROXY_CONFIG_PATH) as fh:
self.assertIn('log 192.0.2.5:514 len 2048', fh.read())
def test_lists_written_even_with_splash_off(self):
os.environ['HAPROXY_DEFAULT_SPLASH'] = 'off'
hm.generate_config()
self.assertTrue(os.path.exists(hm.ROUTED_HOSTS_PATH))
self.assertTrue(os.path.exists(hm.ROUTED_WILDCARDS_PATH))
with open(hm.HAPROXY_CONFIG_PATH) as fh:
self.assertNotIn('default-splash.html', fh.read())
if __name__ == '__main__':
unittest.main(verbosity=2)
@@ -0,0 +1,18 @@
-----BEGIN CERTIFICATE-----
MIIC9DCCAdygAwIBAgICEAMwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMZGVh
ZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
EwYDVQQDDAxkZWFkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCumL3T27a3D6fZbIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnO
abvnQNxEVqyYtTJOn9taitVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BU
SpW96rK40KVftzzDEUnH+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0
seOIdyP/n3T2cX1wsCa/F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9b
PBd2cBaBwTe/3TICu0RnvvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2
wzG4zlKU31fdLvQOzMGH1W5H91sBAgMBAAGjSDBGMCUGA1UdEQQeMByCDGRlYWQu
ZXhhbXBsZYIMYnJhaW5qYXIubmV0MB0GA1UdDgQWBBSPDJiv0Z2YpuloexOfvpIQ
UrWufTANBgkqhkiG9w0BAQsFAAOCAQEAJwBtN3Qw41XvYuSa8Mjn869QOTKEgkaR
EuNQ8WXVBOTRJdxaLHDunGO4ExjHHswCWzXkefYOoIClAU9qLnk9M667BSNxOcz0
VcG/DUMd0tO8X4Zm0zOMjt4VC7whgj7MdYtYVg2+Jhb4sMSx6OqmCUAD4X6ULSTn
lZK2holEC/H3bKt8Q6cecgeubgWojP1bWmWK7jQL5NRIzxQ0KzFxgAf3t2Z05nX8
loABkBiR7vDthkQseUW6RQMayDyzhVqORLt+gZ9D5T3hT1M7VaDc4Twr7UbWPFcR
GuQ7RwKplPIIbPe7+c91pm+agdSDjnV5HmKcAMCwf7bWGQ9/Xc0BFg==
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCumL3T27a3D6fZ
bIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnOabvnQNxEVqyYtTJOn9ta
itVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BUSpW96rK40KVftzzDEUnH
+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0seOIdyP/n3T2cX1wsCa/
F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9bPBd2cBaBwTe/3TICu0Rn
vvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2wzG4zlKU31fdLvQOzMGH
1W5H91sBAgMBAAECggEADUvYBFLn6gPWSZj55nxOqR+2Lmtwh04omgbu2fpWhPTE
ujgKd/23srnnMZXnR6qZQQIPGwJiOPnJuRjiu/PMZSj8j54bbwfA6XmbH9SCIzm0
la50UGBSkw0T50jBI76vICMm6PBkf0rT1oKN9VOd/qlkJoCUiFcGCh177Fbt07oY
R3iAKu7/+yEANX0iEcxbO330TPW3oI0ah5OryfKVoNvxpF8NUfV9pWY66APkOYMJ
zsYvIMvzV/KIgfrtyxJjSjgd7Ud8uictYG1BrZBXpIuGxuxyKUJ7OXOvMMVNnEME
2uzk/aL75UjpurJJU0LCywdUOGwRqS2q4PYQiTeDLQKBgQDjdywH7RODoYeH05Rd
sa2MkNx0Day1i3A4IBds55mzn8jxpT55HlqvHWhNA30kEw01H8+6sB7iMX04xQtS
u3N2opR1VlCvMJhGyRUYUtfu4NnAi/HSu7mCOFGAwaLclI7BAI2or2UrclrlUhGb
CYIBKfEr6Tn6b6AffWaOruCbtQKBgQDEf70R922fnJK+//QuiYoaESwJd3/k10yJ
nH0ZAgbnFcNLT9YVFxHvlt4RHGIotaBjfN+FKMn045jeekgeVdX+uF8+7FwEazAH
zOfxswOeKd8EOooqCD7n7cW3Ma6ev8oWmXwJsz2coRW7Z5zCGQWSzCEDB4yFU9Mt
0z6aN9SJnQKBgQCvBZSIqL+6KgD53zArA7sRDKnEVn1M13TKis7bij02/DyGyuoO
EfAfDBakI4oXxt0Dwm15LWkmDIRYuS0kRH3si96FhpvgoRiaZsDnjV7/PF3wjP3O
c4mjDCiSVOw2o8xP71gQspy62GveRc6sJ52zvY0Hh5aH/3oKoNfX8/F5fQKBgALQ
vbnDLMvrp2KbMjl0eQkUk+fy0V9LAmq6VkSEderbsfejRjL5KKWhKL9gA7wHliu2
v1EeG36pwCf34o6ktjM3Gw5AvrTFgGOsOOqKoROkD/D0kcQ2aAhUrM+KI7FcmB2o
x1EwQ0K6QCB2RJW3XBAVYqezdPz4DX+LlksJX781AoGBAISGOer3s4+oRjO/2VuT
hnMQrET3A0GTol8cgYL2N9wfx7jy5bhSfiJIX/dkh5IrPGYoBdWM87gcUgNQ8AZ7
il5dVeG80oc7KFU/QltZJu9VS32vwXnNsnrXupQ5Ee0ztTLciTMWU5jxvB0ZJHtK
StgYepr4IPDUaObqrPx4Jvfm
-----END PRIVATE KEY-----
@@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDGjCCAgKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwGDEWMBQGA1UEAwwNYnJh
aW4tamFyLmNvbTAeFw0yMDAxMDEwMDAwMDBaFw0yMDA0MDEwMDAwMDBaMBgxFjAU
BgNVBAMMDWJyYWluLWphci5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC9SJUMsK5HhVEATu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9s
i5KzgYp0K2z899wQAKuLzZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ
7jax2QWFMQVhyy6M5QTCHUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyN
N8eolukhvFxQNU3E7oo1jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSy
UEc+Gje6dJLZ9CdJtSQSNGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8
FHC7atJOe5EoaekaAXw5Tq+yB+ODAgMBAAGjbjBsMEsGA1UdEQREMEKCDWJyYWlu
LWphci5jb22CEXd3dy5icmFpbi1qYXIuY29tggxicmFpbmphci5uZXSCEHd3dy5i
cmFpbmphci5uZXQwHQYDVR0OBBYEFAWqbBh2PDUvqQzZoyh960cpuhcOMA0GCSqG
SIb3DQEBCwUAA4IBAQCaovtrXl56YcFlXWMrW/XZBe2/fXI14860qyiR5rb07lS6
CbKOsnRyQdxZ3cdfRwTN1UMmz5IPOZ6fhdo3IPaS2Wae1OdeupRaRWr1SAJtpVZ3
WWugDaumsWV5eWVxVTf93yDEkMERWtTGMviCKBeeHjzHIFiD/EjZznE87V5DYwuf
guOd1F4wXCMRFu8q7sBdPXjQ8rZMlQT+dQXzMST9Ee5PXiInmCAA5nT8vMInAwd2
rP6FxuHJB75ekYzs57CJ82BKKboOCESjUbm54wmVV5pBL305/EjxGf+VovNlRktg
t1znolntIIofE25UCbVNPdrtuDkZgKT3Rv8tCdTP
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC9SJUMsK5HhVEA
Tu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9si5KzgYp0K2z899wQAKuL
zZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ7jax2QWFMQVhyy6M5QTC
HUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyNN8eolukhvFxQNU3E7oo1
jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSyUEc+Gje6dJLZ9CdJtSQS
NGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8FHC7atJOe5EoaekaAXw5
Tq+yB+ODAgMBAAECggEABAeOCL71eYH4jKRMOBbXu98483u1CxnTEa8klnw9ZVES
JctJjY96B/EMwv+g/V+gxhgAkiK9VWfg6bEplOVDWgR9fRBoP57mVkOv3QPn5fcR
xlt9LQQZ+pULQRZoUAFEkVEkV1sfIBJmDLMufOnbAHCKa9+hVlrlCqu0lJgM8put
2bkpSAqwroHVfmD0SmKFAZggBhFwXFJzwmUf3WL+N25f4RZpJsX17FOJ216mlhQ5
edje6dcDpmko2VQtGJXYr8Ge02ZYeRyKGcTsgUZlarB4xUZq1pcnUUSy0Ez7LVbk
tKjPcUpZLlnAA/a2pDgHmwN1+fv/X+Wwv/vZaQMBTQKBgQDuZFyfDRSHCssHtQ4+
Ezxg4WEjwXkq58WxL0zTtLTcquZfQZ1VXgLwAylVwver0I4/UgCfW/AYY+2eLrba
FhuYOosd/3rsUQqf9y+TkSkhe1uVS2/1zjzC1G/5coleLddjzd+YBpEtVxfxDRbA
rVePY3GxkJiTrjAg5OETk7GYpwKBgQDLQ6bPhaocJPQmDi3hKyW8zPJ8wAJNyI0c
RWt9gjeidjmMvwCKtJbBDcV5V0oQ5SB2M5E49mT58B62totAD7FS18c4a4133haV
a1pLBX0U524eQgDzMJ6YA6N9bHdXOH4FZd+Pw90bHetbT1HcKpOdXv4SF/XITju5
Ro5xiymdxQKBgBB4fq2+iZyK8lwl9kXzrkhDSiLeSzDbspz7V6OI6LYWWLRcJDKx
vCI5FbIHNnV4g1t8nu5ju8LzpIc5UwjeXim2YcZA0I1jTAihW3gmn2Dg1QMLo0Co
bypG4zWo5DTNlgQfRUOLsJwhWFYfQWMuls4hYMmT4/bQIiMjrSldGvghAoGALpaB
7DyZobvCjdpgH6TREHX0+izUGlnmJd2vZmPrNnxccmrwx9neV789TLN6EWdCynTF
S0YGHNcxYvVx8yvlm7RDQCY+lYO8r2CcoGdQpPLqZVsBdrvlDbh/x2aN79rN35Ta
6cudqPJxmEsU2x6TGNXF7AW72r47qBqI2XNpBXUCgYAFQdHRRxa2E5rgV0WkL7PF
vXok3isAzpnKAsb5oxPZbhIdLo5EPUGCslsTE1Na8NBdUSoyjhgkMELEDx/ceGL7
IIlNrc/4j71+IahljzaX/D5b5qaW+RnoTVAlbMRR7L/ecbkoovhQR9d37QdoiD/t
B9si92+XB4HYDAjXFxTK3w==
-----END PRIVATE KEY-----
@@ -0,0 +1,18 @@
-----BEGIN CERTIFICATE-----
MIIC+DCCAeCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMYnJh
aW5qYXIubmV0MCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
EwYDVQQDDAxicmFpbmphci5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCyhL5YcFhPEekc/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdb
l4twhlkcyvyaEbcHIgLEgz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgU
QcAAvdAyP77jDF6lX11oCjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zz
XRVp4l2WyqEUJmKxcpppAnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU
0v78QfI+ueCw261UWZ+iXwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQ
dFOfKjH0QGInyByji9eAaXlmvCKtAgMBAAGjTDBKMCkGA1UdEQQiMCCCDGJyYWlu
amFyLm5ldIIQd3d3LmJyYWluamFyLm5ldDAdBgNVHQ4EFgQUdDxfZKX96j61mW7O
NiQpN4T+sZ4wDQYJKoZIhvcNAQELBQADggEBAJE1lP62caByZT7GkKtJLfOj84Ih
MeP751w2gdU8ZfiO8GxXXiR3ZjYzbJgsEi9LiNT9jqHeBf3pv/mJLjPVZ+U557ws
Gq3V5A3uKlRKGcejXlHaWnExhWqLyIaLTeLKstbJyiB/ja4yeTL2TYcOvvlu6weh
ERqwx0p6r7LSQ3nRfTdjEhNq3FVUmqQ8neBP/vXLp2hKr2ACZndGyQOiv0oZWzud
part6RMd8gRNq1Tctrl0YqRYoLRIddBrnTtUtac4FLqxLSBNc03/I9ZgF56Yf+Sl
M1ikmyDJ2ZYJFU6v42Og4/8o3fpq0ttP6pKPi8fInGaFQ+KXO1NWFf1ogD0=
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCyhL5YcFhPEekc
/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdbl4twhlkcyvyaEbcHIgLE
gz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgUQcAAvdAyP77jDF6lX11o
CjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zzXRVp4l2WyqEUJmKxcppp
AnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU0v78QfI+ueCw261UWZ+i
XwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQdFOfKjH0QGInyByji9eA
aXlmvCKtAgMBAAECggEAEQi8lvjFXGojmco3H7PVD7SZftLOFidVh9Fo8/JAe+2Z
LQM/hIjZ1i7sNverbQXMzAQMfXESS2Jx7Clv0mZ0aRgE0hq7wXrabyHfFUPOtqDa
6cPHsz8YvwAOmcQ6W3fBPYI6pzSfdV03rv1GGeCC7ETI3mIupwgUbANDuiDcVY1q
+VuK/hDzPJ5KslDHUNR42MgWacIftq99fV9jN0zQQw1EcyO8QwpXzTSEbdyiAkT+
n/tR9He2MYhgRhjyWPMaxmY6XQpv1h5ASq1yT1kI2oNTaRT3xvd40sLkbfwpZnxi
HnqzCkPptEpWeCLrme+r71+Hxu/+Bph7tPtcuK7jgQKBgQD8XCoPRiWbgCl6kM2G
E3GeM6fftTieTIs7sFVte7wutIKlsec4vizc94ipGaoT7yGEG0T8g9+5hAMbebc2
zzhxvJMB2s18Z/He+Ujv1L1AhBBNVY3zCAxfdq8/n5MGXppEdVRo9QOQndoBfuAV
Qw2Vp4bcy728n4T7sBugW0Ad7QKBgQC1F+utuiF8XI2yx0mX0+Fbty6o+odyM0eb
G/QQDlcGpY0K7V3WAFMESNfytig8OhidzRfWn8lNoqhd80SJfCuiYjiGJ8tjBzcp
+Vqrd/KAxJySJ3NJ2MwyAp7W3l3HAEZLGVYVCankoa4O3qR8YqaIZwc2nk5+8KnG
nBYFlSF/wQKBgQCqaRYh0E2O7wGTeXq/ohukJ9+hqf88rhWLTeyyDM8rfsuGVlb8
MzeaUzEed20pHt60FbIFDowh+yoaAJwez9mevCdNJlKCQrLKze1Zvn+5tDRJ9I7x
YSrxv/Rhc0FWGMX1moFQirtIib9wKRvqX6gpucHREWfjmpizWN2e8F+glQKBgAOp
/jy/nVnUzZEmcxi9Urv2Ut1cZE0tOG+s+x0wDRa3Vsk30EQ6MXPOvX/Nq7a8Pi9D
LQ5MoVgOyHVOLL7TYlagwiAEwhq41KbmwqzhemickcZrQevmlQsV/ny+dwtFuCZg
1tsKHQPhOZJXcuCJZnEwFHdIcLIk0GqSUee/ArSBAoGALOD+Opk+0j4ObNcqtrpP
UcEYqhOuAscbxLoUv53wnOvc7XzMG8JCK1y5LthB2Xanr1JSlheMRrvGxC+WlDmc
5v0Nz71cq5QQ1p8AXnosLRjuy/+edMvGIxxwFMEJBO2ra2Jizpdwd34bjpQ42+CQ
6pVwFQX0oet8SI5xWUWx7NU=
-----END PRIVATE KEY-----
@@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDGjCCAgKgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwHTEbMBkGA1UEAwwSc2hh
cmVkLW9sZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAw
WjAdMRswGQYDVQQDDBJzaGFyZWQtb2xkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQCBg7iYuvjhIG7nlFeja5Lyv2t2fdtNvxzUib7ygejC
+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpjo5/ym2KFZPwiSZoqKUugi9lCoKuL
2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMHYFwURjARRAku3a/GJUHQR0C572mW
mt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64fGeYSOaYOHEOp/VqIRlzsy7eGQh3
xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2JbqFxvijCLnBz5WsrS/S/VWOELNEfGr
84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLfWuSgp5hRAgMBAAGjYjBgMD8GA1Ud
EQQ4MDaCEnNoYXJlZC1vbGQuZXhhbXBsZYIMYnJhaW5qYXIubmV0ghJvdGhlci1z
aXRlLmV4YW1wbGUwHQYDVR0OBBYEFP+AdZ9MkBmFjAQTgLnWZxbJtnvkMA0GCSqG
SIb3DQEBCwUAA4IBAQA32Xl/1IrOyESd2UQcWp/fuHvkivLgamJrheKvKP0Y0vMV
atRu0D0GhqMOr79OT60H/H1FFW7bWQ+UPcZpnuXec40/PcrHxwaSoiLoIUVWhB4t
BOhtTDReX29gEjnB7OqBCJu4ub3HnpFTflUZd7n5A4EcetBsowc5EZ9UAtfvkgGD
nK9ONvNVGurk/V8xgtMJ877ZK1fn+UtgvIWp/F1Gc/taLhe+oCZOPkOqvvqUnS8q
QyIhqmMWRUkBP+jwRA4TP4pj7OH38VY/Ae408Xe4Y97GF7NbOjAL0bkX/bHkLnSG
WzoGO0TK7pCemeFQ+Ghic8Yl0eDynoP7rZIhCKS0
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCBg7iYuvjhIG7n
lFeja5Lyv2t2fdtNvxzUib7ygejC+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpj
o5/ym2KFZPwiSZoqKUugi9lCoKuL2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMH
YFwURjARRAku3a/GJUHQR0C572mWmt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64
fGeYSOaYOHEOp/VqIRlzsy7eGQh3xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2Jbq
FxvijCLnBz5WsrS/S/VWOELNEfGr84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLf
WuSgp5hRAgMBAAECggEACB1tvTwciKvp5wliGF45feDv5c5q+vXE462h0OYXjyfl
1sNuMuoLjhYyFvdcQO2rlTBjB8Ald1hXoeidsVYlF9XvqmJRjjS6MAazMpEzTnba
66V6U1j5gOQ6pi3SjfBJGFYo8L+cy/NqUIBWz0+6hp/BsjW3uW2aMGer2Y/zs2Oi
s0kUNqqUWQpGpEDxbDnUAzto4OOcTOO4A5NB5Oe2ifzFiXCWtTNqFIC0ae9tiqGy
m0sggF2hzmlH1WC54rbv7xujYT6tOAEi1OKp/Cpj8XVLbjaqju5avRl1t2nDRQri
9vnOI9P4bnJKbIYzwJVyy8B1M5iK+W2K7wUQ4ZMetQKBgQC2zwtywQ4jUduLWf4f
6HLUIvmcHdJI0YBHS1+4fKc7tsakRo/0FHxNdUlBhqkYTyp57h6/R1T2gG70Kla1
MarGIgyPIlUXsRojlUcNyKXqSVs+kejftIClb+VDXWrpKtGtyxuESsMjaJBsBBAN
hg3yUx2NL2ct4nnlpl1iq3pH9wKBgQC1XkrjntuzjVYlFeg2ih5va3PUD2qmH9Ej
leNXTGhzZL1eQWz4GaRCgzJH8q41pXBxcXQ8hMXZCWdW5GHBYfVqjTRKMUPdI6FI
ypwMbbTBfQfw1kz5VHfLH3romxikNQXeorZkIZqv7/aLPsxLY78XdW8nSDIsT9ZP
mh7It+Tf9wKBgHT7VSYf6D4t2cH42qfshd6+auC+LJywg9gboatbCWEq+z6QcPoO
SeT/Utt9oqONtW7FNCPPL812LopMtaPJnrCfhU3q+GzqxgFTO17UsWeVfbAyisTs
qw4y5pEPLKT846TFPlyPwwPbm9+jUep1iGMzUMvSM8KnFL4ikcvcU6nNAoGBAJQI
MVHV0HadzokYd19EnpDMgE6+goqy/21vIo6XY82DlPjAp+NccnCDSt/kmQJaVQuS
0Y+vwxhhj4IhiBRiEwBz3hEMYtq1ASg7juCqDlW66mdNUqhSpDAbMqN2K8ejK3ZL
EllqACrHzlBa4j5KXKcWB7JlP+ZhB4gTQDsNtXaDAoGBAJCktIdyM3b6UtM5OofB
lzn6/2uT5u46ECBRybk/J+Y2gfcS2cFzkH179SUB4WgfR0h21FlLmjns5ZvHVk0p
/U4aY3/J42JU6lCYrsyV5BBH+O0stSmrUG/4JdKRe6Mq/pB321RsWD06LVJrBDpi
6NUPW03tNLc+Qe+cnQu0c7y+
-----END PRIVATE KEY-----
@@ -0,0 +1,18 @@
-----BEGIN CERTIFICATE-----
MIIC8zCCAdugAwIBAgICEAQwDQYJKoZIhvcNAQELBQAwHDEaMBgGA1UEAwwRdW5y
ZWxhdGVkLmV4YW1wbGUwHhcNMjAwMTAxMDAwMDAwWhcNMjAwNDAxMDAwMDAwWjAc
MRowGAYDVQQDDBF1bnJlbGF0ZWQuZXhhbXBsZTCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBALEpfqdRPBSrbzixAxD2WhsO1AOTUd/Gguf3zjDZ3TUmWJKb
OhlFthtU6PJVhMdAqOk+hiz9Coo/UngWjykyNMOj1YIpq3M4/P8wBTNxkJd/z2x0
iAQBKvWlB6hxf9c1kenu/ea1iMIThA/uudtX3g8k39Asj/37+o070cdBJExWGnbG
lz103FCO6kBzlJPGIVQwzKH6OKthTzi7Uxy5Fxc/ymS9gc43F0tyY6892Wyl8Y62
a3r1PpZoWR0GDFxPIyCS7NP44T15DSL/2RBi4YnWXOC8QMvPnHbSKjoQ6GLQtGkO
1NO4Uy3XbwDFanYXv6Wxv0oUmZMuY7CRTOMdxCMCAwEAAaM/MD0wHAYDVR0RBBUw
E4IRdW5yZWxhdGVkLmV4YW1wbGUwHQYDVR0OBBYEFBh0L/5U5f94TylibBCgS9pD
/VXkMA0GCSqGSIb3DQEBCwUAA4IBAQB5MLqNjTqIdyztjJsKPdYUkA27NpfdqqY6
Yprv5N7U0slsGUqFIDrSuJ1mMMwTpt02IAFyCOWMUtze+/0PqM9k9TqxJH2cTODp
0ET34Qh8NTo0B534EQUWhny+EzgNAzWK02wbpgm88+Mp/t//sHzhWzOm7uk1dlkq
+5ty/8OveWwAtqAWlDtcsRhkEjzynZKUCAF1extwUSuapFSm20mJ4ivt7Ff5MkTI
fq8P+3XAxWTaBW8DzBIzYd1mwFHK2pcklLMRRTW3hukPxEcTivBgJdwsHMskM3UO
/ejy5EHem5ohCMbwMybvUqLS4xdoqXV36Sj2I1zv0IjgO0wgdK1r
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCxKX6nUTwUq284
sQMQ9lobDtQDk1HfxoLn984w2d01JliSmzoZRbYbVOjyVYTHQKjpPoYs/QqKP1J4
Fo8pMjTDo9WCKatzOPz/MAUzcZCXf89sdIgEASr1pQeocX/XNZHp7v3mtYjCE4QP
7rnbV94PJN/QLI/9+/qNO9HHQSRMVhp2xpc9dNxQjupAc5STxiFUMMyh+jirYU84
u1McuRcXP8pkvYHONxdLcmOvPdlspfGOtmt69T6WaFkdBgxcTyMgkuzT+OE9eQ0i
/9kQYuGJ1lzgvEDLz5x20io6EOhi0LRpDtTTuFMt128AxWp2F7+lsb9KFJmTLmOw
kUzjHcQjAgMBAAECggEAJjBKo+iH6fC2beMCRTLFY5XDr+2etX0GFNi+8prT6zQT
3a5mTLJROf0Ipcw3bfgLMRQQOjxlfD2AGnDKdVfkNdPoH0G6OoErUBra1CcDwxSC
8t7/vtU9nGvxhabJputeQ64HvyacjzFDw2dzWsvLCzqooCgq9dXOQYmGg8wP9gAx
pxfKnd84h41C3pXERdufi6nJHPjZPRW+dKsQ1NMLmtgO5YQnTt3LfVOkZaRa7Izq
ZFWZCBfdsNHHuXCYj3FtqYrqSxELuZAxJ62hxpizrfQdKixguGTObHhC4muIo9fR
eIMCvrAzrs4Fs8nM/K/IPyDU9wPwINK6Z+sxJkhM+QKBgQDaL3Yt3T/LE6bTYyb0
EjVcXoybnuAjFQAenCTIsBRJv21Yv2ddqjq6rOTD64hFuFWPMEOi1TJxIPsFFDMw
NeAwF41fiSrtU8awXs0hTKRQeHjElSBckumOl1DrHu2oKuwWXLniViD62vMsWmpZ
89KT2rgFQuMPvsD3pps8l6W2HwKBgQDP3eTwdVJTdRYlZonqd0+LHG7h715TT1OE
QJOdN9vK5g/9GOCOGHoUxB3IxJVmvz57PrfWaD2rSZ6JMs4TcAY5a0PbJEpgqoJY
kIm/EDnSrFxu2L8DNsAzuThfzXrwWc7PzajYJctYtEfvxCyfo/GdZahg2xaEU0Le
dp+7fxtJfQKBgGVdLWtLYarH+3Yzl1JG71F1f5+A5SKHTCHsas5tt083xuY3Lh9+
yYAWeMVotEunwuAeDR4mLCQQrW4Dn3BjZz8QghaDVFUj2afwsx4oTCIyhfDWU02O
10DunaJOIR1Cq6IL/DPhzbqdY+IUM1qNAyS7tUiEdZbICUBEv9Qbo9oDAoGAORnY
cSGZaumoGsAPNMRvqmQ2JYSO4c3RwGvI4RBeJV+LlthFn2Mj7h5tnCA8dkwkytBQ
yeo6vpnrnf7B018bppxLuKBAekt8oP2biUvYxoGopp6N4kEhDVRdIxwcIHhbMoRp
VG8VbBmiBUBbjcMIdVdeqLSme9UvclMdz/C7uzUCgYEAzD2dvoJSu9zfo3fACw1X
te53Rvr6Kq6DTw0SS4UrAnxlhX9TwD9gymIrR9rxnC+UPwfMwWMdpYqY+Da4sF1G
uwd+nyZ4iFNq+lbLii9rmJ58h03+uo4lUXWTrAik6qOF1ztIfYKWFrOyBQmxjWwo
HnUinudalENFJYZRwSUAtXo=
-----END PRIVATE KEY-----
+631
View File
@@ -0,0 +1,631 @@
#!/usr/bin/env python3
"""Regression tests: a silent peer must not be able to wedge the API's threads.
Why this file exists
--------------------
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool threads
blocked in the kernel in tcp_recvmsg -> sk_wait_data, /health hung 40s+, and the
container stayed "healthy". The app makes no in-process outbound socket reads;
the blocked reads were gunicorn gthread reading the CLIENT socket, which it
makes fully blocking with no timeout (TConn.init -> setblocking(True)).
These tests run REAL gunicorn processes (no mocks) against tiny WSGI apps and
pin:
* the failure mode itself, on the stock gthread worker (control);
* that haproxy_gunicorn_worker.TimeoutThreadWorker frees the thread when a
client stalls before / during the request line, mid-body, or on a reused
keep-alive connection;
* that a legitimately SLOW request (handling time > client timeout) is not
cut off -- the timeout bounds socket I/O, not request duration;
* the watchdog: a worker whose every thread is stuck is replaced;
* review round 1, at production shape (2 workers x 4 threads): 2050
connections that never send a byte are reaped (F1, stock gthread wedges
permanently), and slowloris drip connections are cut by the total header
deadline while /health stays responsive (F2);
* the timeout is logged as one line, not a traceback per scanner;
* haproxy_manager's socket.setdefaulttimeout: an in-process client call to a
stalled upstream (accepts, never replies) times out instead of hanging;
* why that default alone is not enough (setblocking(True) clears it).
Running
-------
python3 scripts/test-gunicorn-socket-timeouts.py
stdlib unittest, like the other scripts/test-*.py; needs only the app's own
requirements (gunicorn, Flask...). Runs in the Docker build gate. ~70s.
"""
import os
import socket
import subprocess
import sys
import tempfile
import textwrap
import threading
import time
import unittest
import urllib.request
REPO_DIR = os.path.abspath(
os.environ.get('HAPROXY_MANAGER_DIR',
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')))
try:
import gunicorn # noqa: F401
except ImportError:
sys.stderr.write('SKIP: gunicorn not installed (pip install -r requirements.txt)\n')
raise SystemExit(77)
WSGI_APP = textwrap.dedent('''\
import os, time
def app(environ, start_response):
path = environ.get('PATH_INFO', '/')
if path == '/sleep':
time.sleep(float(environ.get('QUERY_STRING') or 3))
elif path == '/hang':
time.sleep(3600)
elif path == '/body':
n = int(environ.get('CONTENT_LENGTH') or 0)
environ['wsgi.input'].read(n)
body = ('pid=%d' % os.getpid()).encode()
start_response('200 OK', [('Content-Type', 'text/plain'),
('Content-Length', str(len(body)))])
return [body]
''')
def free_port():
s = socket.socket()
s.bind(('127.0.0.1', 0))
port = s.getsockname()[1]
s.close()
return port
def http_get(port, path='/', timeout=5):
with urllib.request.urlopen('http://127.0.0.1:%d%s' % (port, path),
timeout=timeout) as r:
return r.read().decode()
class Gunicorn:
"""A real gunicorn master on a free port, with its output captured."""
def __init__(self, worker_class, threads=1, env=None, workers=1, extra_ports=()):
self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
fh.write(WSGI_APP)
self.port = free_port()
self.log_path = os.path.join(self.tmp, 'gunicorn.log')
e = dict(os.environ)
e.update(env or {})
self.log = open(self.log_path, 'w')
self.proc = subprocess.Popen(
[sys.executable, '-m', 'gunicorn',
'--bind', '127.0.0.1:%d' % self.port,
*sum((['--bind', '127.0.0.1:%d' % p] for p in extra_ports), []),
'--workers', str(workers), '--threads', str(threads),
'--worker-class', worker_class,
'--timeout', '120', '--graceful-timeout', '2',
'--log-level', 'info', '--error-logfile', '-',
'--pythonpath', '%s,%s' % (REPO_DIR, self.tmp),
'tapp:app'],
stdout=self.log, stderr=subprocess.STDOUT, env=e)
deadline = time.time() + 15
while time.time() < deadline:
try:
http_get(self.port, '/', timeout=1)
return
except Exception:
time.sleep(0.2)
self.stop()
raise RuntimeError('gunicorn did not come up:\n' + self.output())
def output(self):
self.log.flush()
with open(self.log_path) as fh:
return fh.read()
def stop(self):
if self.proc.poll() is None:
self.proc.terminate()
try:
self.proc.wait(5)
except subprocess.TimeoutExpired:
self.proc.kill()
self.proc.wait()
self.log.close()
def stall_client(port, payload=b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03'):
"""Connect and send a few bytes that are not a complete HTTP request (by
default the start of a TLS ClientHello -- what an HTTPS scanner sends to
this plain-HTTP port), then go silent and keep the connection open."""
s = socket.create_connection(('127.0.0.1', port))
if payload:
s.sendall(payload)
return s
def wait_closed_by_server(sock, within):
"""True if the server closed (EOF/RST) the connection within `within` s."""
sock.settimeout(within)
try:
while True:
data = sock.recv(4096)
if not data:
return True
except (ConnectionResetError, BrokenPipeError):
return True
except socket.timeout:
return False
STOCK = 'gthread'
FIXED = 'haproxy_gunicorn_worker.TimeoutThreadWorker'
FAST = {'HAPROXY_MGR_CLIENT_TIMEOUT': '1'}
class StockGthreadControl(unittest.TestCase):
"""Proves the failure mode is real, so the fixed-worker tests mean something."""
def test_stock_gthread_is_wedged_by_one_silent_client(self):
g = Gunicorn(STOCK, threads=1)
self.addCleanup(g.stop)
s = stall_client(g.port)
self.addCleanup(s.close)
time.sleep(0.3)
with self.assertRaises(Exception, msg='stock gthread should be wedged'):
http_get(g.port, '/', timeout=3)
class TimeoutWorkerTests(unittest.TestCase):
def start(self, threads=1, env=None):
g = Gunicorn(FIXED, threads=threads, env=env or FAST)
self.addCleanup(g.stop)
return g
def test_silent_client_is_dropped_and_thread_freed(self):
g = self.start()
s = stall_client(g.port)
self.addCleanup(s.close)
t0 = time.time()
# Only one thread: this request can only be served once the stalled
# connection has released it.
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
self.assertLess(time.time() - t0, 4)
self.assertTrue(wait_closed_by_server(s, 3),
'server should have closed the stalled connection')
def test_connect_and_send_nothing_does_not_hold_a_thread(self):
# Never readable -> never handed to a pool thread (main loop only).
g = self.start()
s = stall_client(g.port, payload=b'')
self.addCleanup(s.close)
self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
def test_half_written_headers(self):
g = self.start()
s = stall_client(g.port, payload=b'GET /health HTTP/1.1\r\nHost: x\r\n')
self.addCleanup(s.close)
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
self.assertTrue(wait_closed_by_server(s, 3))
def test_body_shorter_than_content_length(self):
g = self.start()
s = stall_client(g.port, payload=(
b'POST /body HTTP/1.1\r\nHost: x\r\nContent-Length: 100\r\n\r\n0123456789'))
self.addCleanup(s.close)
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
def test_timeout_reapplied_on_keepalive_reuse(self):
g = self.start()
s = socket.create_connection(('127.0.0.1', g.port))
self.addCleanup(s.close)
s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
s.settimeout(3)
self.assertIn(b'200 OK', s.recv(4096))
# Same connection, now stall mid-request. gthread calls conn.init()
# (setblocking(True)) again for the reused connection.
s.sendall(b'GET / HTT')
time.sleep(0.3)
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
self.assertTrue(wait_closed_by_server(s, 3))
def test_slow_legit_request_is_not_cut_off(self):
# Handling takes 2.5s, client timeout is 1s: must still succeed.
g = self.start()
self.assertTrue(http_get(g.port, '/sleep?2.5', timeout=10).startswith('pid='))
def test_timeout_logged_as_one_line_not_traceback(self):
g = self.start()
s = stall_client(g.port)
self.addCleanup(s.close)
http_get(g.port, '/', timeout=5)
time.sleep(0.3)
out = g.output()
self.assertIn('Closed client connection idle', out)
self.assertNotIn('Traceback', out)
def test_watchdog_replaces_fully_wedged_worker(self):
g = self.start(env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
'HAPROXY_MGR_WEDGE_SECONDS': '2',
'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.5'})
pid_before = http_get(g.port, '/')
# Wedge the only thread inside the app (not a socket read).
hung = threading.Thread(target=lambda: self._swallow(g.port), daemon=True)
hung.start()
time.sleep(0.5)
deadline = time.time() + 12
pid_after = None
while time.time() < deadline:
try:
pid_after = http_get(g.port, '/', timeout=2)
if pid_after != pid_before:
break
except Exception:
pass
time.sleep(0.3)
self.assertIsNotNone(pid_after)
self.assertNotEqual(pid_before, pid_after, 'worker was not replaced')
out = g.output()
self.assertIn('WEDGED WORKER', out)
self.assertIn('tapp.py', out) # the stuck thread's stack was dumped
self.assertIsNone(g.proc.poll(), 'master must keep running')
def test_watchdog_leaves_partially_busy_pool_alone(self):
# 2 threads, one hung forever: the pool still serves, so no restart.
g = self.start(threads=2, env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
'HAPROXY_MGR_WEDGE_SECONDS': '1',
'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.3'})
pid_before = http_get(g.port, '/')
threading.Thread(target=lambda: self._swallow(g.port), daemon=True).start()
time.sleep(3)
self.assertEqual(pid_before, http_get(g.port, '/', timeout=3))
self.assertNotIn('WEDGED WORKER', g.output())
@staticmethod
def _swallow(port):
try:
http_get(port, '/hang', timeout=60)
except Exception:
pass
def raise_nofile(n):
try:
import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
if soft < n:
resource.setrlimit(resource.RLIMIT_NOFILE, (min(n, hard), hard))
return resource.getrlimit(resource.RLIMIT_NOFILE)[0]
except (ImportError, ValueError, OSError):
return None
class ProductionShapeExhaustion(unittest.TestCase):
"""Review round 1 (F1/F2), at the production shape: 2 workers x 4 threads,
default worker_connections=1000."""
def start(self, env):
g = Gunicorn(FIXED, threads=4, workers=2, env=env)
self.addCleanup(g.stop)
return g
def test_f1_2050_idle_conns_reaped_and_health_recovers(self):
# Stock gthread: 2x1000 never-readable conns fill every slot, the run
# loop stops polling, and the worker stays dead even after the
# attacker closes everything. Here they must be reaped after
# FIRST_BYTE_TIMEOUT -- even while the attacker still holds them open.
if (raise_nofile(4500) or 0) < 4300:
self.skipTest('RLIMIT_NOFILE too low for 2050 client sockets')
g = self.start({'HAPROXY_MGR_FIRST_BYTE_TIMEOUT': '3'})
socks = []
self.addCleanup(lambda: [x.close() for x in socks])
for _ in range(2050):
s = socket.socket()
s.settimeout(2)
s.connect(('127.0.0.1', g.port))
socks.append(s)
# Attacker keeps every socket open. Within FIRST_BYTE_TIMEOUT plus the
# time to drain the backlog, the API must answer again.
deadline = time.time() + 20
ok = None
while time.time() < deadline:
try:
ok = http_get(g.port, '/', timeout=3)
break
except Exception:
time.sleep(0.5)
self.assertIsNotNone(ok, '/health never recovered while idle conns were held')
held = time.time()
# The reaped sockets were closed by the server.
closed = sum(1 for x in socks[:1500] if wait_closed_by_server(x, 0.01))
self.assertGreater(closed, 1000, 'server did not close idle conns (%d)' % closed)
for x in socks:
x.close()
socks.clear()
time.sleep(1)
for _ in range(5):
self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
self.assertLess(held, time.time())
def test_f2_drip_connections_cut_and_health_responsive_throughout(self):
# 8 connections (= every thread at production shape) each drip one
# header byte per second -- far under the 20s per-recv timeout, so only
# the total HEADER_DEADLINE can cut them.
g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '3'}) # CLIENT_TIMEOUT stays 20
stop = threading.Event()
cut_after = []
def drip():
s = socket.create_connection(('127.0.0.1', g.port))
t0 = time.time()
req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 1000
try:
for i in range(len(req)):
if stop.is_set():
return
s.send(req[i:i + 1])
time.sleep(1.0)
s.settimeout(0.01)
try:
if s.recv(1) == b'':
raise ConnectionResetError
except socket.timeout:
pass
except OSError:
cut_after.append(time.time() - t0)
finally:
s.close()
drippers = [threading.Thread(target=drip, daemon=True) for _ in range(8)]
for t in drippers:
t.start()
self.addCleanup(stop.set)
time.sleep(0.5)
worst = 0.0
t_end = time.time() + 12
while time.time() < t_end:
t0 = time.time()
self.assertTrue(http_get(g.port, '/', timeout=6).startswith('pid='))
worst = max(worst, time.time() - t0)
time.sleep(0.5)
stop.set()
self.assertEqual(len(cut_after), 8, 'not every drip connection was cut: %r' % cut_after)
# The deadline is armed when a THREAD picks the connection up, not at
# accept: a request queued behind busy threads (e.g. legit certbot
# runs) must not be cut for waiting. So a drip conn that was queued
# (the 2 workers rarely split 8 conns 4/4) is cut at up to about
# 2 x deadline; one that got a thread at once within ~deadline + 1s.
self.assertLessEqual(min(cut_after), 3 + 2, cut_after)
self.assertTrue(all(c < 2 * 3 + 2 for c in cut_after), cut_after)
self.assertLess(worst, 5.0)
self.assertIn('HAPROXY_MGR_HEADER_DEADLINE', g.output())
def test_f2_slow_but_complete_headers_within_deadline_are_served(self):
g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '4'})
s = socket.create_connection(('127.0.0.1', g.port))
self.addCleanup(s.close)
for part in (b'GET / HTTP/1.1\r\n', b'Host: x\r\n', b'\r\n'):
s.sendall(part)
time.sleep(1)
s.settimeout(5)
self.assertIn(b'200 OK', s.recv(4096))
def refused(port, wait=2.0):
"""True if the server closes a fresh connection without answering a
complete request (peer filter)."""
s = socket.create_connection(('127.0.0.1', port), timeout=wait)
try:
try:
s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
except OSError:
return True
s.settimeout(wait)
try:
return s.recv(4096) == b''
except (ConnectionResetError, BrokenPipeError):
return True
except socket.timeout:
return False
finally:
s.close()
class PeerFilter(unittest.TestCase):
"""Re-review R1: tenant containers share client-net with haproxy-manager,
so the API port must refuse peers that are not loopback / the host path."""
@classmethod
def setUpClass(cls):
sys.path.insert(0, REPO_DIR)
import haproxy_gunicorn_worker as w
cls.w = w
def test_allowlist_parsing_and_matching(self):
nets, problems = self.w.parse_allowed_peers('127.0.0.1,::1,gateway', gateways=['172.18.0.1'])
self.assertEqual(problems, [])
ok = self.w.peer_allowed
self.assertTrue(ok('127.0.0.1', nets))
self.assertTrue(ok('::1', nets))
self.assertTrue(ok('::ffff:127.0.0.1', nets))
self.assertTrue(ok('172.18.0.1', nets)) # docker-proxy / host
self.assertFalse(ok('172.18.0.7', nets)) # a tenant container
self.assertFalse(ok('127.0.0.2', nets))
self.assertFalse(ok('', nets))
self.assertFalse(ok('garbage', nets))
def test_bad_entries_fail_closed_and_are_reported(self):
nets, problems = self.w.parse_allowed_peers('nonsense,gateway', gateways=[])
self.assertEqual(nets, [])
self.assertEqual(len(problems), 2, problems)
nets, _ = self.w.parse_allowed_peers('10.0.0.0/8')
self.assertTrue(self.w.peer_allowed('10.1.2.3', nets))
def test_default_gateway_parsed_from_route_table(self):
fn = os.path.join(tempfile.mkdtemp(), 'route')
with open(fn, 'w') as fh:
fh.write('Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\n'
'eth0\t00000000\t010012AC\t0003\t0\t0\t0\t00000000\n'
'eth0\t000012AC\t00000000\t0001\t0\t0\t0\t0000FFFF\n')
self.assertEqual(self.w._default_gateways(fn), ['172.18.0.1'])
def test_filtered_port_refuses_unlisted_peer_other_port_unaffected(self):
other = free_port()
g = Gunicorn(FIXED, threads=1, extra_ports=(other,), env={
'HAPROXY_MGR_PEER_FILTER_PORTS': str(other),
'HAPROXY_MGR_API_ALLOWED_PEERS': '10.255.255.1'})
self.addCleanup(g.stop)
# g.port (unfiltered) works; `other` refuses 127.0.0.1 (not listed).
self.assertTrue(http_get(g.port, '/').startswith('pid='))
for _ in range(5):
self.assertTrue(refused(other), 'filtered port answered an unlisted peer')
self.assertTrue(http_get(g.port, '/').startswith('pid='))
out = g.output()
self.assertIn('Refused 1 connection(s) to the management API', out) # first one, immediately
self.assertEqual(out.count('Refused '), 1, 'reject logging is not rate-limited')
def test_filtered_port_serves_listed_peer(self):
port = free_port()
g = Gunicorn(FIXED, threads=1, extra_ports=(port,), env={
'HAPROXY_MGR_PEER_FILTER_PORTS': str(port),
'HAPROXY_MGR_API_ALLOWED_PEERS': '127.0.0.1,::1'})
self.addCleanup(g.stop)
self.assertTrue(http_get(port, '/').startswith('pid='))
self.assertNotIn('Refused', g.output())
class QueuedDripStarvation(unittest.TestCase):
"""Re-review: with the deadline armed at thread pickup, 40 reconnecting
drippers queued behind each other starved legit requests (every probe
>30s). Armed as max(enqueue + D, pickup + 1s) they must not."""
def test_40_reconnecting_drippers_do_not_starve_legit_requests(self):
g = Gunicorn(FIXED, threads=4, workers=2, env={}) # production defaults
self.addCleanup(g.stop)
stop = threading.Event()
self.addCleanup(stop.set)
def drip():
while not stop.is_set():
try:
s = socket.create_connection(('127.0.0.1', g.port))
except OSError:
time.sleep(0.2)
continue
try:
req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 5000
for i in range(len(req)):
if stop.is_set():
break
s.send(req[i:i + 1])
time.sleep(1.0)
except OSError:
pass
finally:
s.close()
for _ in range(40):
threading.Thread(target=drip, daemon=True).start()
time.sleep(2)
lat = []
t_end = time.time() + 25
while time.time() < t_end:
t0 = time.time()
try:
http_get(g.port, '/', timeout=30)
lat.append(time.time() - t0)
except Exception:
lat.append(99.0)
time.sleep(1)
stop.set()
lat.sort()
median = lat[len(lat) // 2]
sys.stderr.write('\n 40 drippers: probes=%d median=%.2fs worst=%.2fs\n'
% (len(lat), median, lat[-1]))
# Armed at pickup, every probe here exceeded 30s (total starvation).
# With max(enqueue + D, pickup + 1s) queued drippers get ~1s of thread
# time, so legit requests keep getting through. They still queue: the
# equilibrium wait is ~(drippers/threads) x hold time, about 9s at 40
# drippers on 8 threads -- measured, see the round-2 report. The
# remaining fix for that is not admitting partial requests to a thread
# at all; the peer filter makes it a local-only scenario.
# The property under test is "no starvation" (every probe answered
# inside 30s). The latency bounds depend on host load: a shared CI
# runner measured median 14.97s on 2 probes, so they are only enforced
# when HAPROXY_MGR_STRICT_TIMING=1 (quiet local box).
self.assertNotIn(99.0, lat, 'a legit request was starved for 30s+')
if os.environ.get('HAPROXY_MGR_STRICT_TIMING') == '1':
self.assertLess(median, 12.0, lat)
self.assertLess(lat[-1], 20.0, lat)
class StalledUpstreamDefaultTimeout(unittest.TestCase):
"""haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
client call to an upstream that accepts and never replies must time out."""
def test_in_process_client_to_stalled_upstream_times_out(self):
srv = socket.socket()
srv.bind(('127.0.0.1', 0))
srv.listen(5)
self.addCleanup(srv.close)
port = srv.getsockname()[1]
accepted = []
threading.Thread(target=lambda: accepted.append(srv.accept()),
daemon=True).start()
# Import the real module in a child (it changes process-global socket
# state and opens its log files at import), then make a client call
# with NO explicit timeout -- what a careless future handler would do.
probe = textwrap.dedent('''\
import logging, os, sys, tempfile, time, urllib.request
d = tempfile.mkdtemp()
real = logging.FileHandler
logging.FileHandler = lambda fn, *a, **k: real(os.path.join(d, os.path.basename(fn)), *a, **k)
os.chdir(sys.argv[1]); sys.path.insert(0, sys.argv[1])
import haproxy_manager
logging.FileHandler = real
import socket
print('default', socket.getdefaulttimeout())
t0 = time.time()
try:
urllib.request.urlopen('http://127.0.0.1:%s/' % sys.argv[2]).read()
print('result returned')
except Exception as e:
print('result', type(e).__name__, round(time.time() - t0, 1))
''')
env = dict(os.environ, HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT='1.5')
r = subprocess.run([sys.executable, '-c', probe, REPO_DIR, str(port)],
capture_output=True, text=True, timeout=30, env=env)
if 'No module named' in r.stderr and 'flask' in r.stderr.lower():
self.skipTest('Flask not installed')
self.assertIn('default 1.5', r.stdout, r.stderr)
line = [l for l in r.stdout.splitlines() if l.startswith('result')][0]
self.assertTrue(line.startswith('result TimeoutError '), line)
elapsed = float(line.split()[-1])
self.assertLess(elapsed, 5)
self.assertTrue(accepted, 'upstream never saw the connection')
accepted[0][0].close()
def test_default_timeout_alone_does_not_cover_gthread_client_sockets(self):
# Documents why the custom worker exists: gthread's setblocking(True)
# resets any timeout -- including the process default -- to None.
old = socket.getdefaulttimeout()
socket.setdefaulttimeout(60)
try:
a, b = socket.socketpair()
finally:
socket.setdefaulttimeout(old)
self.addCleanup(a.close)
self.addCleanup(b.close)
self.assertEqual(a.gettimeout(), 60)
a.setblocking(True)
self.assertIsNone(a.gettimeout())
if __name__ == '__main__':
unittest.main(verbosity=2)
+614
View File
@@ -0,0 +1,614 @@
#!/usr/bin/env python3
"""Offline tests for scripts/haproxy-stderr-relay.py (runs in the image build)."""
import importlib.util
import io
import os
import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
spec = importlib.util.spec_from_file_location('relay', os.path.join(HERE, 'haproxy-stderr-relay.py'))
relay = importlib.util.module_from_spec(spec)
spec.loader.exec_module(relay)
class Clock:
def __init__(self):
self.t = 0.0
def __call__(self):
return self.t
class RelayTests(unittest.TestCase):
def setUp(self):
self.dir = tempfile.mkdtemp(prefix='hap-stderr-relay-')
self.path = os.path.join(self.dir, 'haproxy-stderr.log')
open(self.path, 'w').close()
self.out = io.StringIO()
self.clock = Clock()
def make(self, burst=1000, per_min=120, max_bytes=10**6, backups=3):
return relay.Relay(self.path, self.out, burst, per_min, max_bytes, backups, clock=self.clock)
def append(self, text):
with open(self.path, 'a') as fh:
fh.write(text)
def test_history_is_not_replayed_and_new_lines_are_prefixed(self):
self.append('old line from a previous container\n')
r = self.make()
r.poll_once()
self.assertEqual(self.out.getvalue(), '')
self.append('[ALERT] (10) : Current worker (67) exited with code 134 (Aborted)\n')
r.poll_once()
self.assertEqual(self.out.getvalue(),
'haproxy-stderr: [ALERT] (10) : Current worker (67) exited with code 134 (Aborted)\n')
def test_partial_line_is_held_until_complete(self):
r = self.make()
r.poll_once()
self.append('Thread 1 is abo')
r.poll_once()
self.assertEqual(self.out.getvalue(), '')
self.append('ut to kill the process.\n')
r.poll_once()
self.assertIn('Thread 1 is about to kill the process.', self.out.getvalue())
def test_control_characters_cannot_forge_lines(self):
r = self.make()
r.poll_once()
self.append('a\x1b[31mb\r\x00c\n')
r.poll_once()
self.assertEqual(self.out.getvalue().count('\n'), 1)
self.assertNotIn('\x1b', self.out.getvalue())
def test_rate_limit_passes_a_full_dump_then_summarises(self):
r = self.make(burst=200, per_min=60)
r.poll_once()
self.append(''.join('dump line %d\n' % i for i in range(150)))
r.poll_once()
self.assertEqual(self.out.getvalue().count('\n'), 150)
self.out.truncate(0), self.out.seek(0)
self.append(''.join('flap %d\n' % i for i in range(500)))
r.poll_once()
self.assertLessEqual(self.out.getvalue().count('\n'), 52) # ~50 tokens left
self.clock.t += 600 # bucket refills
self.out.truncate(0), self.out.seek(0)
self.append('after storm\n')
r.poll_once()
r.tick()
self.assertIn('[relay] suppressed ', self.out.getvalue())
self.assertIn('after storm', self.out.getvalue())
def test_copytruncate_rotation_bounds_the_file_and_keeps_following(self):
r = self.make(max_bytes=100, backups=2)
r.poll_once()
self.append('x' * 150 + '\n')
r.poll_once()
r.rotate_if_needed()
self.assertEqual(os.path.getsize(self.path), 0)
self.assertTrue(os.path.exists(self.path + '.1'))
self.append('after rotate\n')
r.poll_once()
self.assertIn('after rotate', self.out.getvalue())
def test_missing_file_is_not_an_error(self):
os.remove(self.path)
r = self.make()
r.poll_once()
self.append('late\n')
r.poll_once()
self.assertIn('late', self.out.getvalue())
# --- captured scenario: 4 quick reloads (bucket drained) then a forced worker panic ---------------
def reload_block(pid, n_proxies=100):
"""What HAProxy 3.2 writes on stderr for ONE reload (~110 lines on a small config)."""
L = ['[NOTICE] (%d) : Reloading HAProxy' % pid,
'[NOTICE] (%d) : Initializing new worker (%d)' % (pid, pid + 100),
'[NOTICE] (%d) : haproxy version is 3.2.26-1~bpo13+1' % (pid + 100),
'[NOTICE] (%d) : path to executable is /usr/sbin/haproxy' % (pid + 100)]
for i in range(6):
L.append("[WARNING] (%d) : config : parsing [/etc/haproxy/haproxy.cfg:%d] : a 'http-request' rule "
"placed after a 'use_backend' rule will still be processed before." % (pid + 100, 800 + i))
L.append("[WARNING] (%d) : config: Can't load stats-file '/var/lib/haproxy/stats.dat': cannot open file." % pid)
L.append('[NOTICE] (%d) : Loading success.' % pid)
for i in range(n_proxies):
L.append('[WARNING] (%d) : Proxy be_%d stopped (cumulated conns: FE: 0, BE: 12).' % (pid + 50, i))
L.append('[WARNING] (%d) : Former worker (%d) exited with code 0 (Exit)' % (pid, pid + 50))
return L
DUMP = [
'Thread 1 is about to kill the process (pid 6631).',
'* Thread 1 : id=0x7f60efa77400 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
' 1/1 loops=94 ctxsw=234 stuck=0 prof=0 harmless=0 isolated=0',
' call trace(15):',
' | 0x55c557cd3582 <00 00 00 e8 ae 03 dc ff]: ha_dump_backtrace+0x82/0x43d > main-0x7d0',
' Thread 2 : id=0x7f60ef5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
' lock_hist: R:TASK_WQ U:TASK_WQ',
' Thread 3 : id=0x7f60ed5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
' Thread 4 : id=0x7f60ecdfe6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
'',
"Hint: when reporting this bug to developers, please check if a core file was",
" produced, open it with 'gdb', issue 't a a bt full', check that the",
'[ALERT] (6328) : Current worker (6631) exited with code 134 (Aborted)',
'[WARNING] (6328) : A worker process unexpectedly died and this can only be explained by a bug in haproxy or its dependencies.',
'Please check that you are running an up to date and maintained version of haproxy and open a bug report.',
'HAProxy version 3.2.26-1~bpo13+1 2026/10/14 - https://haproxy.org/',
'Known bugs: http://www.haproxy.org/bugs/bugs-3.2.26.html',
'[ALERT] (6328) : exit-on-failure: killing every processes with SIGTERM',
'[WARNING] (6328) : All workers exited. Exiting... (134)',
]
KEYS = ('about to kill', 'Thread 1 :', 'Thread 2 :', 'Thread 3 :', 'Thread 4 :', 'Current worker',
'exited with code 134', 'exit-on-failure', 'All workers exited', 'call trace', 'ha_dump_backtrace',
'bug in haproxy', 'Please check that you are running', 'produced, open it with')
def scenario(mod, burst=400, drained=True):
"""4 quick reloads (>400 lines: the bucket drains) then the panic dump. Returns facts."""
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = mod.Relay('/nonexistent', out, burst, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
lines = []
for k in range(4):
lines += reload_block(100 * (k + 1))
n_before = len(lines)
lines += DUMP
for l in lines:
if l.strip(): # poll_once drops blank lines before _emit
r._emit(l)
t[0] += 0.003
r.tick()
return {'out': out.getvalue(), 'err': err.getvalue(), 'reload_lines': n_before, 'bucket': r.tokens, 'r': r}
class CapturedScenarioTests(unittest.TestCase):
def test_drained_bucket_then_panic_forwards_every_crash_line(self):
f = scenario(relay)
self.assertGreater(f['reload_lines'], 400) # would drain a 400-line bucket
e = f['err']
for k in KEYS:
self.assertIn(k, e, k)
# the whole dump, line for line (blank lines are dropped by design)
got = [l[len(relay.PREFIX):] for l in e.split('\n') if l]
self.assertEqual(got, [d.rstrip() for d in DUMP if d.strip()])
def test_dump_is_forwarded_in_full_even_with_an_empty_bucket(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for l in ('benign %d' % i for i in range(10)):
r._emit(l)
self.assertEqual(out.getvalue(), '')
for l in DUMP:
if l.strip():
r._emit(l)
self.assertEqual(err.getvalue().count('\n'), len([d for d in DUMP if d.strip()]))
self.assertEqual(r.suppressed, 10)
def test_reload_noise_is_collapsed_not_drained(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for k in range(10):
for l in reload_block(100 * (k + 1)):
r._emit(l)
self.assertEqual(r.tokens, 400.0) # bucket untouched by 1000+ noise lines
self.assertEqual(r.suppressed, 0)
self.assertEqual(err.getvalue(), '')
summ = [l for l in out.getvalue().split('\n') if 'reload/start ok' in l]
self.assertEqual(len(summ), 10) # exactly one per reload
self.assertNotIn('Proxy be_', out.getvalue())
self.assertNotIn('parsing [', out.getvalue())
self.assertLess(out.getvalue().count('\n'), 15)
def test_suppressed_lines_are_summarised_per_minute(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 5, 60, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(105):
r._emit('flapping server state %d' % i)
self.assertNotIn('[relay] suppressed', out.getvalue())
t[0] += 61
r.tick()
self.assertRegex(out.getvalue(), r'\[relay\] suppressed 100 lines')
self.assertEqual(r.suppressed, 0)
self.assertEqual(err.getvalue(), '')
def test_benign_lines_go_to_stdout_and_crash_lines_to_stderr(self):
# docker's journald driver: stdout -> PRIORITY 6, stderr -> PRIORITY 3, "<N>" is NOT parsed
f = scenario(relay)
self.assertIn('[relay] reload/start ok', f['out'])
self.assertNotIn('Thread', f['out'])
self.assertNotIn('Current worker', f['out'])
self.assertNotIn('[ALERT]', f['out'])
self.assertNotIn('reload/start ok', f['err'])
for text in (f['out'], f['err']):
for l in text.split('\n'):
self.assertFalse(l.startswith('<') or l.startswith(relay.PREFIX + '<'), l)
def test_clean_worker_exit_is_not_crash_evidence(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
r._emit('[WARNING] (1) : Former worker (5) exited with code 0 (Exit)')
r._emit('[WARNING] (1) : Former worker (6) exited with code 143 (Terminated)')
self.assertEqual(err.getvalue().count('\n'), 1)
self.assertIn('code 143', err.getvalue())
def test_crash_classes_never_limited(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
evidence = ['[ALERT] (6328) : Current worker (6631) exited with code 134 (Aborted)',
'[ALERT] (6328) : exit-on-failure: killing every processes with SIGTERM',
'[WARNING] (6328) : Current worker (9) exited with code 139 (Segmentation fault)',
'[WARNING] (6328) : A worker process unexpectedly died and this can only be explained by a bug in haproxy',
'[WARNING] (6328) : All workers exited. Exiting... (134)',
' Thread 2 : id=0x7f60ef5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
'* Thread 1 : id=0x7f60ef5ff6c0 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0']
for l in evidence:
r._emit(l)
self.assertEqual(err.getvalue().count('\n'), len(evidence))
FORGED = [
'GET /FATAL/SIGSEGV/segfault HTTP/1.1',
'[WARNING] (6328) : Server be_x/ [ALERT] FATAL watchdog is DOWN, reason: Layer4 timeout',
'[WARNING] (6328) : Server be/Current worker (1) exited with code 134 is DOWN',
'[NOTICE] (6328) : proxy example.com/Thread 1 is about to kill the process (pid 1)',
'Host: Current worker (1) exited with code 134 (Aborted)',
'Host: [ALERT] (1) : forged alert',
'x-forwarded-for: Thread 1 is about to kill the process (pid 1).',
'user-agent: * Thread 1 : id=0x7f60ef5ff6c0 act=1',
'ha_panic watchdog FATAL SIGABRT',
'backend ALERT_Current worker_exited with code 134',
'[WARNING] (6328) : Server Former worker (7) exited with code 134 is DOWN',
]
def test_forged_keywords_stay_rate_limited(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for l in self.FORGED * 50:
r._emit(l)
self.assertEqual(err.getvalue(), '')
self.assertFalse(r.in_dump)
self.assertEqual(r.suppressed, len(self.FORGED) * 50)
self.assertEqual(r.budget_left, relay.CRASH_BUDGET) # none of it touched the crash budget
# a forged header in the middle of normal traffic cannot open a block either
r._emit('proxy: Thread 1 is about to kill the process (pid 1).')
r._emit('free text that would be a dump line')
self.assertFalse(r.in_dump)
self.assertEqual(err.getvalue(), '')
def test_forged_lines_cannot_be_collapsed_away_or_open_blocks(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
r._emit("GET /Loading success HTTP/1.1")
self.assertEqual(r.routine, 0)
self.assertIn('Loading success', out.getvalue())
def test_flood_of_dump_headers_is_capped_by_the_window_budget(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
for i in range(20000):
r._emit('Thread 1 is about to kill the process (pid %d).' % i)
r._emit(' Thread 2 : id=0x7f60ef5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0')
r._emit('[WARNING] (1) : All workers exited. Exiting... (134)')
n = err.getvalue().count('\n')
self.assertLessEqual(n, relay.CRASH_BUDGET + relay.DUMP_BUDGET + 5)
self.assertGreater(r.suppressed, 50000) # the rest is counted, not lost silently
t[0] += relay.CRASH_WINDOW + 1 # a new window: real evidence flows again
before = err.getvalue().count('\n')
r._emit('[ALERT] (1) : Current worker (2) exited with code 134 (Aborted)')
self.assertEqual(err.getvalue().count('\n'), before + 1)
def test_bound_closed_block_cannot_rearm_during_cooldown(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(relay.DUMP_MAX_LINES + 5):
r._emit('never closed %d' % i)
self.assertFalse(r.in_dump)
r._emit('Thread 1 is about to kill the process (pid 2).') # immediate new header
self.assertFalse(r.in_dump)
r._emit('call trace line after the second header')
self.assertNotIn('call trace line after', err.getvalue())
t[0] += relay.DUMP_COOLDOWN + 1 # cooldown over: a real dump opens again
r._emit('Thread 1 is about to kill the process (pid 3).')
self.assertTrue(r.in_dump)
def test_hostile_and_very_long_lines(self):
r = self.make_relay()
r.poll_once()
self.append('A' * 500000 + '\n')
self.append('\x1b]0;evil\x07 [ALERT] forged\x00\r\n')
r.poll_once()
outs = self.out.getvalue().split('\n')
self.assertTrue(all(len(l) < relay.MAX_LINE + 200 for l in outs))
self.assertNotIn('\x1b', self.out.getvalue())
self.assertNotIn('\x00', self.out.getvalue())
self.assertEqual(self.out.getvalue().count('\n'), 2)
def test_unterminated_flood_is_bounded_in_memory(self):
r = self.make_relay()
r.poll_once()
for _ in range(3):
self.append('B' * 400000) # never a newline
r.poll_once()
self.assertLessEqual(len(r.pending), relay.MAX_PENDING)
def test_unclosed_dump_block_cannot_disable_the_limiter(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(relay.DUMP_MAX_LINES + 500):
r._emit('never closed %d' % i)
self.assertFalse(r.in_dump)
self.assertLessEqual(err.getvalue().count('\n'), relay.DUMP_MAX_LINES + 3)
self.assertGreaterEqual(r.suppressed, 400)
def test_dump_block_is_closed_by_time_bound(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
t[0] += relay.DUMP_MAX_SECONDS + 1
r._emit('later benign line')
self.assertFalse(r.in_dump)
self.assertNotIn('later benign', err.getvalue())
# --- review finding I2: tenant-triggered ALERT lines must not spend crash evidence --------
TENANT_LINES = [
"[ALERT] (10) : backend 'be_tenant1' has no server available!",
"[WARNING] (10) : Server be_tenant1/s1 is DOWN, reason: Layer4 connection problem, info: \"Connection refused\", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.",
"[WARNING] (10) : Server be_tenant1/s1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.",
"[NOTICE] (10) : Server be_tenant1/s1 is going DOWN for maintenance.",
]
def test_tenant_alert_flood_then_panic_still_forwards_the_full_dump_at_stderr(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(1200):
r._emit(self.TENANT_LINES[0])
t[0] += 0.001
t[0] += 5.0
for l in DUMP:
if l.strip():
r._emit(l)
got = [l[len(relay.PREFIX):] for l in err.getvalue().split('\n') if l]
self.assertEqual(got, [d.rstrip() for d in DUMP if d.strip()])
self.assertIn('exited with code 134', err.getvalue())
self.assertNotIn('exited with code 134', out.getvalue())
self.assertNotIn('has no server available', err.getvalue())
def test_tenant_lines_are_their_own_rate_limited_stdout_class(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(1200):
r._emit(self.TENANT_LINES[i % len(self.TENANT_LINES)])
self.assertEqual(err.getvalue(), '') # never PRIORITY 3
self.assertLessEqual(out.getvalue().count('\n'), relay.TENANT_BURST + 2)
self.assertGreater(out.getvalue().count('\n'), 0) # but not silently dropped
self.assertEqual(r.tokens, 400.0) # general bucket untouched
self.assertEqual(r.budget_left, relay.CRASH_BUDGET) # crash budget untouched
self.assertEqual(r.dump_budget_left, relay.DUMP_BUDGET)
self.assertGreater(r.tenant_suppressed, 1000)
t[0] += relay.SUMMARY_EVERY + 1
r.tick()
self.assertRegex(out.getvalue(), r'\[relay\] suppressed \d+ tenant-driven server/backend state lines')
self.assertEqual(r.tenant_suppressed, 0)
def test_other_alerts_are_not_crash_evidence(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
r._emit('[ALERT] (12) : config : unable to load')
r._emit("[ALERT] (12) : backend 'x' has no server available!")
self.assertEqual(err.getvalue(), '')
self.assertEqual(out.getvalue().count("\n"), 2) # both reach stdout (rate-limited classes), neither stderr
def test_dump_block_has_a_budget_ordinary_evidence_cannot_spend(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
for i in range(relay.CRASH_BUDGET + 500): # e.g. a worker crash loop
r._emit('[ALERT] (10) : Current worker (%d) exited with code 134 (Aborted)' % i)
self.assertEqual(r.budget_left, 0)
before = err.getvalue().count('\n')
for l in DUMP:
if l.strip():
r._emit(l)
self.assertEqual(err.getvalue().count('\n') - before, len([d for d in DUMP if d.strip()]))
def test_rotated_backup_is_mode_0640(self):
r = relay.Relay(self.path, io.StringIO(), 1000, 120, 10, 3, clock=lambda: 0.0)
os.chmod(self.path, 0o644)
self.append('x' * 100 + '\n')
r.rotate_if_needed()
self.assertEqual(os.stat(self.path + '.1').st_mode & 0o777, 0o640)
# shared helpers for the file-driven tests
def setUp(self):
self.dir = tempfile.mkdtemp(prefix='hap-stderr-relay-')
self.path = os.path.join(self.dir, 'haproxy-stderr.log')
open(self.path, 'w').close()
self.out = io.StringIO()
def make_relay(self):
return relay.Relay(self.path, self.out, 1000, 120, 10 ** 6, 3, clock=lambda: 0.0)
def append(self, text):
with open(self.path, 'a') as fh:
fh.write(text)
# --- mutants: the same scenario must FAIL when a guarantee is removed (private copies) ------------
def _scenario_ok(mod):
f = scenario(mod)
got = [l[len(mod.PREFIX):] for l in f['err'].split('\n') if l]
crash_ok = all(k in f['err'] for k in KEYS) and got == [d.rstrip() for d in DUMP if d.strip()]
collapse_ok = f['out'].count('reload/start ok') == 4 and 'Proxy be_' not in f['out'] and f['bucket'] > 300
prio_ok = 'Thread' not in f['out'] and 'reload/start ok' not in f['err']
return crash_ok, collapse_ok, prio_ok
class MutantTests(unittest.TestCase):
SRC = os.path.join(HERE, 'haproxy-stderr-relay.py')
def load_mutant(self, old, new):
with open(self.SRC) as fh:
src = fh.read()
self.assertIn(old, src, 'mutation target vanished: update the test')
mutated = src.replace(old, new, 1)
self.assertNotEqual(mutated, src)
d = tempfile.mkdtemp(prefix='hap-relay-mutant-') # private copy, the real file is untouched
path = os.path.join(d, 'mutant.py')
with open(path, 'w') as fh:
fh.write(mutated)
sp = importlib.util.spec_from_file_location('mutant', path)
m = importlib.util.module_from_spec(sp)
sp.loader.exec_module(m)
return m
def test_unmutated_copy_passes_everything(self):
self.assertEqual(_scenario_ok(relay), (True, True, True))
def test_mutant_limiter_applied_to_crash_lines(self):
m = self.load_mutant('if self.in_dump or CRASH_RE.match(line):', 'if False:')
crash_ok, _, _ = _scenario_ok(m)
self.assertFalse(crash_ok)
def test_mutant_dump_block_not_tracked(self):
m = self.load_mutant('if not self.in_dump and DUMP_START_RE.match(line) and now >= self.cooldown_until:', 'if False:')
crash_ok, _, _ = _scenario_ok(m)
self.assertFalse(crash_ok)
def test_mutant_collapse_disabled(self):
m = self.load_mutant('if ROUTINE_RE.match(line):', 'if False:')
_, collapse_ok, _ = _scenario_ok(m)
self.assertFalse(collapse_ok)
def test_mutant_crash_lines_on_stdout(self):
m = self.load_mutant('self._write(self.err, line)', 'self._write(self.out, line)')
_, _, prio_ok = _scenario_ok(m)
self.assertFalse(prio_ok)
def test_mutant_dump_bound_removed(self):
m = self.load_mutant('DUMP_MAX_LINES = 500 ', 'DUMP_MAX_LINES = 10 ** 9 ')
err = io.StringIO()
r = m.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(m.DUMP_MAX_LINES if False else 800):
r._emit('never closed %d' % i)
self.assertTrue(r.in_dump) # real module closes at 500: see test_unclosed_dump_block...
def _forged_run(self, mod):
err = io.StringIO()
r = mod.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for l in CapturedScenarioTests.FORGED * 20:
r._emit(l)
return err.getvalue()
def test_forged_run_clean_on_real_module(self):
self.assertEqual(self._forged_run(relay), '')
def test_mutant_anchors_removed(self):
m = self.load_mutant('if self.in_dump or CRASH_RE.match(line):',
"if self.in_dump or re.search(r'ALERT|FATAL|Current worker|about to kill|Thread \\d+ : id=', line):")
self.assertNotEqual(self._forged_run(m), '')
def _flood(self, mod):
err, t = io.StringIO(), [0.0]
r = mod.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
for i in range(5000):
r._emit('Thread 1 is about to kill the process (pid %d).' % i)
r._emit('[WARNING] (1) : All workers exited. Exiting... (134)')
return err.getvalue().count('\n')
def test_flood_capped_on_real_module(self):
self.assertLessEqual(self._flood(relay), relay.CRASH_BUDGET + relay.DUMP_BUDGET + 5)
def test_mutant_budget_removed(self):
m = self.load_mutant('if self._budget(now, kind):', 'if True:')
self.assertGreater(self._flood(m), relay.CRASH_BUDGET + relay.DUMP_BUDGET + 5)
def _rearm(self, mod):
r = mod.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=io.StringIO())
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(relay.DUMP_MAX_LINES + 5):
r._emit('x %d' % i)
r._emit('Thread 1 is about to kill the process (pid 2).')
return r.in_dump
def test_mutant_cooldown_removed(self):
self.assertFalse(self._rearm(relay))
m = self.load_mutant(' and now >= self.cooldown_until:', ':')
self.assertTrue(self._rearm(m))
def test_mutant_summary_removed(self):
m = self.load_mutant("self._write(self.out, '[relay] suppressed %d lines", "self._noop = ('[relay] suppressed %d lines")
out, t = io.StringIO(), [0.0]
r = m.Relay('/nonexistent', out, 5, 60, 1 << 30, 3, clock=lambda: t[0], err=io.StringIO())
for i in range(105):
r._emit('x %d' % i)
t[0] += 61
r.tick()
self.assertNotIn('[relay] suppressed', out.getvalue())
def _tenant_then_panic_ok(self, mod):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = mod.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(1200):
r._emit("[ALERT] (10) : backend 'be_tenant1' has no server available!")
t[0] += 0.001
t[0] += 5.0
for l in DUMP:
if l.strip():
r._emit(l)
got = [l[len(mod.PREFIX):] for l in err.getvalue().split('\n') if l]
return got == [d.rstrip() for d in DUMP if d.strip()] and 'no server available' not in err.getvalue()
def test_tenant_then_panic_clean_on_real_module(self):
self.assertTrue(self._tenant_then_panic_ok(relay))
def test_mutant_wide_alert_exemption_restored(self):
m = self.load_mutant("\\[ALERT\\]\\s+\\(\\d+\\)\\s*:\\s*(?:Current worker", "\\[ALERT\\]\\s+\\(\\d+\\)\\s*:\\s*(?:.|Current worker")
self.assertFalse(self._tenant_then_panic_ok(m))
def test_mutant_tenant_class_removed(self):
m = self.load_mutant('if TENANT_RE.match(line):', 'if False:')
self.assertFalse(self._tenant_then_panic_ok(m) and self._tenant_stdout_ok(m))
def _tenant_stdout_ok(self, mod):
out, err = io.StringIO(), io.StringIO()
r = mod.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
for i in range(1200):
r._emit("[WARNING] (10) : Server be_t/s1 is DOWN, reason: Layer4 timeout")
return r.tokens == 400.0 and out.getvalue().count('\n') <= mod.TENANT_BURST + 2
def test_mutant_shared_budget(self):
m = self.load_mutant("'dump' if is_dump else 'crash'", "'crash'")
out, err = io.StringIO(), io.StringIO()
r = m.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for i in range(m.CRASH_BUDGET + 500):
r._emit('[ALERT] (10) : Current worker (%d) exited with code 134 (Aborted)' % i)
before = err.getvalue().count('\n')
for l in DUMP:
if l.strip():
r._emit(l)
self.assertNotEqual(err.getvalue().count('\n') - before, len([d for d in DUMP if d.strip()]))
if __name__ == '__main__':
unittest.main()
+97
View File
@@ -0,0 +1,97 @@
#!/usr/bin/env python3
"""Offline tests for scripts/haproxy-supervisor.sh (pidof / python are stubbed on PATH)."""
import os
import stat
import subprocess
import tempfile
import time
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
SCRIPT = os.path.join(HERE, 'haproxy-supervisor.sh')
PIDOF = """#!/bin/sh
# "haproxy" is alive iff the start stamp exists and is younger than $LIFE seconds
[ -f "$STATE/started" ] || exit 1
start=$(cat "$STATE/started")
now=$(date +%s.%N)
awk -v s="$start" -v n="$now" -v l="$LIFE" 'BEGIN{exit !(n - s < l)}'
"""
PYTHON = """#!/bin/sh
# stand-in for ensure_haproxy.py: stamp the start, record it
date +%s.%N > "$STATE/started"
date +%s.%N >> "$STATE/starts"
exit ${ENSURE_RC:-0}
"""
class SupervisorTests(unittest.TestCase):
def setUp(self):
self.d = tempfile.mkdtemp(prefix='hap-sup-')
self.state = os.path.join(self.d, 'state')
os.mkdir(self.state)
for name, body in (('pidof', PIDOF), ('python', PYTHON)):
p = os.path.join(self.d, name)
with open(p, 'w') as fh:
fh.write(body)
os.chmod(p, os.stat(p).st_mode | stat.S_IXUSR)
def run_sup(self, seconds, **env):
e = dict(os.environ, PATH=self.d + ':' + os.environ['PATH'], STATE=self.state)
e.setdefault('LIFE', '0')
e.update(env)
try:
subprocess.run(['bash', SCRIPT], env=e, timeout=seconds, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return 'exited'
except subprocess.TimeoutExpired:
return 'running' # still looping when the clock ran out = alive
def starts(self):
try:
with open(os.path.join(self.state, 'starts')) as fh:
return [float(x) for x in fh.read().split()]
except OSError:
return []
def test_non_integer_interval_does_not_kill_the_supervisor(self):
self.assertEqual(self.run_sup(8, HAPROXY_SUPERVISOR_INTERVAL='0.5', ENSURE_RC='1'), 'running')
self.assertGreaterEqual(len(self.starts()), 2) # fell back to the default interval and kept trying
def test_garbage_interval_and_max_fall_back(self):
self.assertEqual(self.run_sup(4, HAPROXY_SUPERVISOR_INTERVAL='abc', HAPROXY_SUPERVISOR_MAX_BACKOFF='x'), 'running')
self.assertGreaterEqual(len(self.starts()), 1)
def test_crash_loop_backs_off_2_5_10_capped(self):
# haproxy dies immediately after every successful start (LIFE=0) -> quick deaths
self.run_sup(14, HAPROXY_SUPERVISOR_INTERVAL='1', HAPROXY_SUPERVISOR_BACKOFF_STEPS='1 2 3',
HAPROXY_SUPERVISOR_QUICK_DEATH='5', HAPROXY_SUPERVISOR_STABLE='8',
HAPROXY_SUPERVISOR_MAX_BACKOFF='3')
s = self.starts()
gaps = [round(b - a, 1) for a, b in zip(s, s[1:])]
self.assertGreaterEqual(len(s), 4, gaps)
# gap = interval(1) + backoff step: 2, 3, 4, then capped at 1+3 = 4
self.assertLess(gaps[0], gaps[1] + 0.3, gaps)
self.assertGreaterEqual(gaps[1], 3.0 - 0.3, gaps)
self.assertGreaterEqual(gaps[2], 4.0 - 0.3, gaps)
self.assertLessEqual(max(gaps), 4.0 + 0.8, gaps)
def test_stable_haproxy_resets_the_backoff(self):
# alive for 2s after each start; QUICK_DEATH=1 so a 2s life is NOT quick: no backoff growth
self.run_sup(9, HAPROXY_SUPERVISOR_INTERVAL='1', LIFE='2', HAPROXY_SUPERVISOR_BACKOFF_STEPS='3 3 3',
HAPROXY_SUPERVISOR_QUICK_DEATH='1', HAPROXY_SUPERVISOR_STABLE='8',
HAPROXY_SUPERVISOR_MAX_BACKOFF='3')
s = self.starts()
gaps = [b - a for a, b in zip(s, s[1:])]
self.assertGreaterEqual(len(s), 3, gaps)
self.assertLess(max(gaps), 3.6, gaps) # never the 3 s backoff on top
def test_running_haproxy_is_left_alone(self):
with open(os.path.join(self.state, 'started'), 'w') as fh:
fh.write(str(time.time()))
self.assertEqual(self.run_sup(3, HAPROXY_SUPERVISOR_INTERVAL='1', LIFE='1000'), 'running')
self.assertEqual(self.starts(), [])
if __name__ == '__main__':
unittest.main()
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env python3
"""Contract tests for the CVE-2026-109618 / -109619 edge mitigations.
* templates/hap_listener.tpl denies multipart/form-data without a
Content-Length (400), with AND without the Coraza SPOE block rendered.
* coraza-spoa/overrides.conf carries the body rules 209610-209616.
What these rules DO is measured elsewhere: the Coraza verdicts by WHP's
scripts/tests/coraza-body-parse-harness (cve109618.go, run with
HMB_OVERRIDES pointing at this checkout's overrides.conf), the HAProxy verdict
by live curl against a host. This file pins the invariants that make those
measurements stay true:
* the deny is outside the `coraza_spoe_backend` conditional;
* it keys on the Content-Length HEADER count, never req.body_size (see the
H3 note in hap_listener.tpl);
* every 2096xx id is inside 200000-209999 -- WHP's break-glass rule
(991000001) removes exactly that range, so an id outside it would stay
armed for a site in emergency mode;
* no ctl:ruleEngine=On sits on a chain STARTER (Coraza runs it as soon as
the first link matches -- the 920440 defect).
Running
-------
python3 scripts/test-multipart-cve-mitigation.py
"""
import logging
import os
import re
import sys
import tempfile
import unittest
MODULE_DIR = os.path.abspath(
os.environ.get('HAPROXY_MANAGER_DIR',
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
)
os.chdir(MODULE_DIR)
sys.path.insert(0, MODULE_DIR)
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
_real_file_handler = logging.FileHandler
logging.FileHandler = (
lambda filename, *a, **kw: _real_file_handler(
os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
)
import haproxy_manager # noqa: E402
CORAZA_OVERRIDES = os.path.join(MODULE_DIR, 'coraza-spoa', 'overrides.conf')
ENGINE_LO, ENGINE_HI = 200000, 209999 # WHP_WAF_ENGINE_RANGE_LO..HI
CVE_IDS = range(209610, 209617)
def render(coraza):
kw = dict(crt_path='/etc/haproxy/certs', suspension_enabled=True,
coraza_spoe_backend='coraza-spoa:9000' if coraza else None,
cluster_secret=None, syslog_target='172.18.0.1:514')
return haproxy_manager.template_env.get_template('hap_listener.tpl').render(**kw)
def active_lines(text):
return [l.strip() for l in text.splitlines()
if l.strip() and not l.strip().startswith('#')]
DENY_RE = re.compile(
r'^http-request deny deny_status 400 .*\bif is_multipart_form '
r'!\{ req\.hdr_cnt\(content-length\) gt 0 \}$')
class ListenerDeny(unittest.TestCase):
def check(self, coraza):
lines = active_lines(render(coraza))
acl = [i for i, l in enumerate(lines) if l.startswith('acl is_multipart_form ')]
deny = [i for i, l in enumerate(lines) if DENY_RE.match(l)]
self.assertEqual(len(acl), 1, 'is_multipart_form acl missing or duplicated')
self.assertEqual(len(deny), 1, 'multipart no-Content-Length deny missing or duplicated')
self.assertEqual(lines[acl[0]],
'acl is_multipart_form req.fhdr(content-type) -m sub -i multipart/form-data')
self.assertLess(acl[0], deny[0])
self.assertNotIn('body_size', lines[deny[0]])
# after the real-IP resolution, before any backend selection rule
real_ip = max(i for i, l in enumerate(lines) if 'set-var(txn.real_ip)' in l)
self.assertLess(real_ip, deny[0])
if coraza:
spoe = [i for i, l in enumerate(lines) if l.startswith('http-request send-spoe-group')]
self.assertTrue(spoe, 'coraza render has no send-spoe-group -- test is blind')
self.assertLess(deny[0], spoe[0], 'deny must run before the WAF is consulted')
def test_with_coraza(self):
self.check(True)
def test_without_coraza(self):
self.check(False)
@unittest.skipUnless(os.path.exists(CORAZA_OVERRIDES),
'coraza-spoa/overrides.conf is not shipped in this image')
class OverrideRules(unittest.TestCase):
def setUp(self):
with open(CORAZA_OVERRIDES) as fh:
text = fh.read()
# Join backslash continuations so one directive is one string.
joined = re.sub(r'\\\n\s*', '', text)
self.rules = [l.strip() for l in joined.splitlines()
if l.strip().startswith('SecRule') or l.strip().startswith('SecAction')]
def ids(self):
out = []
for r in self.rules:
m = re.search(r'\bid:(\d+)', r)
if m:
out.append(int(m.group(1)))
return out
def test_every_cve_id_present_once(self):
ids = self.ids()
for rid in CVE_IDS:
self.assertEqual(ids.count(rid), 1, f'{rid} missing or duplicated')
def test_every_id_inside_the_break_glass_engine_range(self):
# every rule that belongs to the mitigation, found by what it DOES
# (its tag or its tx variables), not by its id
cve = [int(re.search(r'\bid:(\d+)', r).group(1)) for r in self.rules
if re.search(r'\bid:\d+', r)
and ('whp/cve-2026-1096' in r or 'whp_cve109618' in r)]
self.assertGreaterEqual(len(cve), 7, 'CVE rules not found -- the test is blind')
self.assertTrue(cve, 'no CVE ids parsed -- the test is blind')
for rid in cve:
self.assertTrue(ENGINE_LO <= rid <= ENGINE_HI, rid)
def test_deny_rules_flip_the_engine_for_themselves(self):
denies = [r for r in self.rules if re.search(r'\bid:2096\d\d\b', r) and ',deny,' in r]
self.assertEqual(sorted(int(re.search(r'id:(\d+)', r).group(1)) for r in denies),
[209610, 209613, 209615, 209616])
for i, r in enumerate(self.rules):
if r not in denies:
continue
if ',chain' in r:
self.assertNotIn('ctl:ruleEngine', r, 'engine flip on a chain starter')
self.assertIn('ctl:ruleEngine=On', self.rules[i + 1])
else:
self.assertIn('ctl:ruleEngine=On', r)
def test_no_chain_starter_anywhere_in_the_block_flips_the_engine(self):
for r in self.rules:
if re.search(r'\bid:2096\d\d\b', r) and ',chain' in r:
self.assertNotIn('ctl:ruleEngine', r)
if __name__ == '__main__':
unittest.main(verbosity=2)
+395
View File
@@ -0,0 +1,395 @@
#!/usr/bin/env python3
"""Regression tests for WAF request-body inspection (Coraza via SPOE).
Why this file exists
--------------------
Measured 2026-10-09 on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0
(report: waf-body-hardening-report.md):
* Without `option http-buffer-request`, send-spoe-group fires at header
time and `req.body` holds only what happened to arrive with the headers.
A 14-byte `content=$a($b)` POST (CRS 933180, an ENFORCED rule) passed
uninspected when its body was written in a second send(); a body sent 2s
late, or after `Expect: 100-continue`, was never inspected at all.
* Coraza+CRS costs ~3.3 ms per KB of body. The SPOE `timeout processing`
is 100 ms and FAILS OPEN, so raising tune.bufsize to 32 KB turned a
payload at byte 0 of a 30 KB base64 body into a bypass. bufsize is a
security limit, not a performance knob.
* coraza-spoa's SPOP library refuses frames > 65535 bytes, and Coraza's
SecRequestBodyLimitAction Reject 413s even under DetectionOnly, so the
three limits must stay ordered.
* The X-WHP-WAF-Body stamp drives enforcement of body-parse errors
(200002/200003) in WHP's generated Coraza rules. A client must never be
able to set it, and a backend must never see it.
These tests pin the rendered config. That HAProxy ACCEPTS it is
validate-rendered-config.py's job; that it BEHAVES is the live measurement.
Running
-------
python3 scripts/test-waf-body-inspection.py
"""
import logging
import os
import re
import sys
import tempfile
import unittest
MODULE_DIR = os.path.abspath(
os.environ.get('HAPROXY_MANAGER_DIR',
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
)
os.chdir(MODULE_DIR)
sys.path.insert(0, MODULE_DIR)
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
_real_file_handler = logging.FileHandler
logging.FileHandler = (
lambda filename, *a, **kw: _real_file_handler(
os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
)
import haproxy_manager # noqa: E402
SPOA_FRAME_CAP = 65535 # dropmorepackets/haproxy-go v0.0.8 maxFrameSize
CORAZA_CONFIG = os.path.join(MODULE_DIR, 'coraza-spoa', 'config.yaml')
def render(template, coraza):
kw = dict(crt_path='/etc/haproxy/certs', suspension_enabled=False,
coraza_spoe_backend='coraza-spoa:9000' if coraza else None,
cluster_secret=None, syslog_target='172.18.0.1:514')
return haproxy_manager.template_env.get_template(template).render(**kw)
def active_lines(text):
"""Config lines with comments and blanks stripped (comments mention the
directives by name, so a raw substring check would pass on prose)."""
out = []
for line in text.splitlines():
s = line.strip()
if s and not s.startswith('#'):
out.append(s)
return out
def index_of(lines, pattern):
for i, line in enumerate(lines):
if re.search(pattern, line):
return i
return -1
class ListenerWithCoraza(unittest.TestCase):
def setUp(self):
self.lines = active_lines(render('hap_listener.tpl', coraza=True))
def test_body_is_buffered_before_inspection(self):
self.assertIn('option http-buffer-request', self.lines)
def test_stamp_order_strip_set_inspect_strip(self):
strip1 = index_of(self.lines, r'^http-request del-header X-WHP-WAF-Body$')
size = index_of(self.lines, r'^http-request set-var\(txn\.waf_body_size\) req\.hdr_val\(content-length\) if \{ req\.hdr_cnt\(content-length\) eq 1 \}$')
complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if \{ var\(txn\.waf_body_size\) -m found \} !\{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
hdrs_len = index_of(self.lines, r'^http-request set-var\(txn\.waf_hdrs_len\) req\.hdrs,length if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \}$')
nocl_complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_hdrs_len\) -m found \} \{ req\.body_len,add\(txn\.waf_hdrs_len\) lt (\d+) \}$')
nocl_partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \} !\{ var\(txn\.waf_body\) -m found \}$')
stamp = index_of(self.lines, r'^http-request set-header X-WHP-WAF-Body %\[var\(txn\.waf_body\)\] if \{ var\(txn\.waf_body\) -m found \}$')
spoe = index_of(self.lines, r'^http-request send-spoe-group coraza coraza-req$')
deny = index_of(self.lines, r'^http-request return status 403 .*var\(txn\.coraza\.action\) -m str deny')
dup_ct = index_of(self.lines, r'^http-request deny deny_status 400 if \{ req\.fhdr_cnt\(content-type\) gt 1 \}$')
self.assertGreaterEqual(dup_ct, 0, 'duplicate Content-Type must be rejected at the edge')
self.assertLess(dup_ct, spoe, 'and before Coraza is asked')
for name, idx in (('client-copy strip', strip1), ('size var', size), ('complete', complete),
('partial', partial), ('no-CL header length', hdrs_len),
('no-CL complete', nocl_complete),
('no-CL partial', nocl_partial), ('stamp', stamp),
('send-spoe-group', spoe), ('deny', deny)):
self.assertGreaterEqual(idx, 0, f'{name} rule missing from the rendered listener')
# Client copies stripped BEFORE we stamp, stamp BEFORE the SPOE call.
self.assertLess(strip1, size)
self.assertLess(size, complete)
# The no-CL `partial` fallback relies on "-m found" of an earlier
# `complete`: order is load-bearing.
self.assertLess(size, hdrs_len)
self.assertLess(hdrs_len, nocl_complete)
self.assertLess(nocl_complete, nocl_partial)
self.assertLess(max(complete, partial, nocl_partial), stamp)
self.assertLess(stamp, spoe)
# Stripped again AFTER the SPOE call so no backend ever sees it.
strip2 = [i for i, l in enumerate(self.lines)
if l == 'http-request del-header X-WHP-WAF-Body' and i > spoe]
self.assertTrue(strip2, 'X-WHP-WAF-Body must be deleted again after send-spoe-group')
self.assertLess(strip2[0], deny)
def test_no_content_length_complete_cannot_have_filled_the_buffer(self):
"""Without a Content-Length, `complete` is inferred from "the buffer
cannot have been full". body + headers thresholds must sit well below
bufsize - maxrewrite (1024), or a truncated chunked body would be
stamped complete and a parse error on it would 403."""
line = [l for l in self.lines if 'str(complete) if { var(txn.waf_hdrs_len)' in l][0]
total_lt = int(re.search(r'add\(txn\.waf_hdrs_len\) lt (\d+)', line).group(1))
header = active_lines(render('hap_header.tpl', coraza=True))
bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
self.assertLessEqual(total_lt, (bufsize - 1024) * 0.85)
def test_stamp_never_derived_from_req_body_size(self):
"""req.body_size is "buffered + htx->extra" and HAProxy 3.0's H3 mux
never sets htx->extra, so over HTTP/3 it equals req.body_len and a
body truncated at one buffer reads as `complete` (measured on whp01
2026-10-17: six Divi sync-to-server saves of 252-417 KB 403'd by
WHP 991900020; lab-reproduced on haproxy 3.0.11 +QUIC). The
advertised length must come from the Content-Length header."""
uses = [l for l in self.lines if 'req.body_size' in l]
self.assertEqual(uses, [], 'req.body_size must not feed the X-WHP-WAF-Body stamp')
size = [l for l in self.lines if l.startswith('http-request set-var(txn.waf_body_size) ')]
self.assertEqual(len(size), 1)
self.assertIn('req.hdr_val(content-length)', size[0])
def test_complete_and_partial_are_mutually_exclusive_conditions(self):
"""With a Content-Length, `complete` and `partial` must be the same
condition, negated -- an inverted or loosened `complete` would enforce
parse errors on truncated bodies (403 on large legitimate posts)."""
c = [l for l in self.lines if 'str(complete) if { var(txn.waf_body_size)' in l]
p = [l for l in self.lines if 'str(partial) if { var(txn.waf_body_size)' in l]
self.assertEqual((len(c), len(p)), (1, 1))
c, p = c[0], p[0]
self.assertEqual(c.split(' if ', 1)[1].replace('!{', '{'),
p.split(' if ', 1)[1].replace('!{', '{'))
self.assertIn('!{ req.body_len', p)
self.assertNotIn('!{', c)
def test_stamp_never_set_from_client_input(self):
"""The only set-header for the stamp is from our own var."""
sets = [l for l in self.lines if re.search(r'set-header\s+X-WHP-WAF-Body', l, re.I)]
self.assertEqual(len(sets), 1)
self.assertIn('%[var(txn.waf_body)]', sets[0])
self.assertFalse([l for l in self.lines if re.search(r'add-header\s+X-WHP-WAF-Body', l, re.I)])
def test_access_log_carries_coverage_fields(self):
lf = [l for l in self.lines if l.startswith('log-format ')]
self.assertEqual(len(lf), 1)
self.assertIn('wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)]', lf[0])
def test_access_log_carries_waf_verdict_fields(self):
"""wrule/wrules/wscore are appended LAST so any consumer anchored on
the earlier fields keeps working."""
lf = [l for l in self.lines if l.startswith('log-format ')]
self.assertEqual(len(lf), 1)
self.assertTrue(lf[0].endswith(
'wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] '
'wscore=%[var(txn.coraza.anomaly_score)]"'))
def test_fail_open_preserved(self):
"""Nothing may deny on var(txn.coraza.error): an SPOA outage must not
503 customer sites."""
self.assertFalse([l for l in self.lines if 'txn.coraza.error' in l and
re.search(r'\b(deny|reject|return|silent-drop|tarpit)\b', l)])
class ListenerWithoutCoraza(unittest.TestCase):
def test_waf_off_config_has_no_body_buffering(self):
lines = active_lines(render('hap_listener.tpl', coraza=False))
self.assertNotIn('option http-buffer-request', lines)
self.assertFalse([l for l in lines if 'X-WHP-WAF-Body' in l])
self.assertFalse([l for l in lines if 'send-spoe-group' in l])
def test_waf_off_header_keeps_default_bufsize(self):
lines = active_lines(render('hap_header.tpl', coraza=False))
self.assertFalse([l for l in lines if l.startswith('tune.bufsize')])
class BufsizeCeilings(unittest.TestCase):
def setUp(self):
lines = active_lines(render('hap_header.tpl', coraza=True))
found = [l for l in lines if l.startswith('tune.bufsize')]
self.assertEqual(len(found), 1, 'tune.bufsize must be pinned when Coraza is on')
self.bufsize = int(found[0].split()[1])
spoe = active_lines(haproxy_manager.template_env.get_template(
'hap_coraza_spoe_engine.tpl').render())
m = [re.match(r'timeout\s+processing\s+(\d+)ms$', l) for l in spoe]
m = [x for x in m if x]
self.assertEqual(len(m), 1)
self.processing_ms = int(m[0].group(1))
def test_bufsize_fits_the_spoe_time_budget(self):
"""~3.3 ms/KB measured; 31 KB took 104 ms against a 100 ms timeout and
FAILED OPEN. Keep the worst case at about half the budget."""
worst_ms = (self.bufsize / 1024.0) * 3.3
self.assertLessEqual(worst_ms, self.processing_ms * 0.6,
f'tune.bufsize {self.bufsize} => ~{worst_ms:.0f} ms of Coraza time '
f'against a {self.processing_ms} ms fail-open timeout')
def test_bufsize_below_spop_frame_cap(self):
self.assertLess(self.bufsize, SPOA_FRAME_CAP)
def test_body_arg_is_the_full_buffered_body(self):
"""The message must keep sending req.body unsliced: the `complete`
stamp compares req.body_len to Content-Length, so a slice would make
a truncated body look complete to Coraza."""
spoe = haproxy_manager.template_env.get_template('hap_coraza_spoe_engine.tpl').render()
args = [l.strip() for l in spoe.splitlines() if l.strip().startswith('args ')]
self.assertEqual(len(args), 1)
self.assertTrue(args[0].endswith(' body=req.body'))
class CorazaTransactionIdIsTheRequestReference(unittest.TestCase):
"""Coraza's transaction ID must be HAProxy's unique-id, so the reference a
blocked customer quotes (403-waf.html / X-Request-Reference / `id=` in
haproxy.log) is the `unique_id` in /var/log/coraza/audit.log and
spoa.log. Without `id=` coraza-spoa v0.7.1 invents a random 16-letter ID
(internal/application.go HandleRequest) that joins to nothing."""
def setUp(self):
spoe = haproxy_manager.template_env.get_template(
'hap_coraza_spoe_engine.tpl').render()
lines = active_lines(spoe)
self.messages = {}
current = None
for line in lines:
m = re.match(r'^spoe-message\s+(\S+)$', line)
if m:
current = m.group(1)
continue
if line.startswith('spoe-') or line.startswith('['):
current = None
elif current and line.startswith('args '):
self.messages[current] = line.split()[1:]
def args_of(self, message):
return dict(a.split('=', 1) for a in self.messages[message])
def test_request_message_sends_unique_id(self):
self.assertIn('coraza-req', self.messages)
self.assertEqual(self.args_of('coraza-req').get('id'), 'unique-id')
def test_request_message_exports_rule_ids(self):
"""Without exportRuleIDs coraza-spoa never sets txn.coraza.rule_ids
and the access log's wrules= is always '-'."""
self.assertEqual(self.args_of('coraza-req').get('exportRuleIDs'), 'bool(true)')
def test_app_is_still_the_first_arg(self):
"""agent.go HandleSPOE Panics unless the FIRST kv entry is `app`."""
for name, args in self.messages.items():
self.assertTrue(args[0].startswith('app='),
f'{name}: first arg must be app=, got {args[0]}')
def test_every_response_message_carries_the_same_id(self):
"""HandleResponse finds the request transaction by `id`; a coraza-res
without it errors ("response id is empty") on every response."""
for name in self.messages:
if name != 'coraza-req':
self.assertEqual(self.args_of(name).get('id'), 'unique-id', name)
def test_unique_id_is_the_header_reference(self):
"""The fetch is only meaningful while unique-id-format is the UUID that
is also emitted as X-Request-Reference and on the 403 page."""
header = active_lines(render('hap_header.tpl', coraza=True))
self.assertIn('unique-id-format %[uuid()]', [re.sub(r'\s+', ' ', l) for l in header])
self.assertIn('unique-id-header X-Request-Reference', [re.sub(r'\s+', ' ', l) for l in header])
listener = render('hap_listener.tpl', coraza=True)
self.assertIn('x-request-reference "%[unique-id]"', listener)
CORAZA_OVERRIDES = os.path.join(MODULE_DIR, 'coraza-spoa', 'overrides.conf')
@unittest.skipUnless(os.path.exists(CORAZA_OVERRIDES),
'coraza-spoa/overrides.conf is not shipped in this image; '
'this class runs from a repo checkout')
class BakedPromotions(unittest.TestCase):
"""920440 is a CHAIN whose first link matches every URL with a file
extension. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on that
first link, so every *.php / *.css request was evaluated with the engine
On (full CRS anomaly enforcement, in detect_only mode too). Measured
in-process and live, 2026-10-09; fixed by keying the flip on the chain's
real condition."""
def setUp(self):
with open(CORAZA_OVERRIDES) as fh:
self.lines = active_lines(fh.read())
def test_920440_is_not_promoted_on_its_chain_starter(self):
self.assertFalse([l for l in self.lines
if re.match(r'SecRuleUpdateActionById\s+(\S*-)?920440\b', l)],
'920440 promoted via SecRuleUpdateActionById again')
def test_no_chain_starter_is_promoted(self):
"""Chain-starter rules in the promoted ranges (CRS 4.25.0): 920440,
944110, 944120. None may be covered by a SecRuleUpdateActionById."""
chain_starters = (920440, 944110, 944120)
promoted = []
for l in self.lines:
m = re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l)
if m:
lo = int(m.group(1))
promoted.append((lo, int(m.group(2) or lo)))
self.assertTrue(promoted, 'no promotions parsed -- the test is blind')
for rid in chain_starters:
self.assertFalse([r for r in promoted if r[0] <= rid <= r[1]],
f'{rid} is a CRS chain starter and must not be promoted')
def test_log4shell_range_still_promoted(self):
for rid in (944100, 944130, 944150, 944151, 944152, 944300):
self.assertTrue([1 for l in self.lines
if (m := re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l))
and int(m.group(1)) <= rid <= int(m.group(2) or m.group(1))],
f'{rid} lost its promotion')
def test_920440_flip_keys_on_the_restricted_extension(self):
i = self.lines.index('SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \\')
self.assertEqual(self.lines[i + 1],
'"id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"')
@unittest.skipUnless(os.path.exists(CORAZA_CONFIG),
'coraza-spoa/config.yaml is not shipped in this image; '
'this class runs from a repo checkout')
class CorazaBodyLimits(unittest.TestCase):
def setUp(self):
with open(CORAZA_CONFIG) as fh:
self.cfg = active_lines(fh.read())
header = active_lines(render('hap_header.tpl', coraza=True))
self.bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
def directive(self, name):
vals = [l.split(None, 1)[1] for l in self.cfg if l.split()[0] == name]
self.assertEqual(len(vals), 1, f'{name} must be set exactly once in config.yaml')
return vals[0]
def test_limit_strictly_above_anything_haproxy_can_send(self):
limit = int(self.directive('SecRequestBodyLimit'))
self.assertGreater(limit, SPOA_FRAME_CAP)
self.assertGreater(limit, self.bufsize)
def test_over_limit_never_rejects(self):
"""Reject interrupts even under DetectionOnly -> 403 on uploads."""
self.assertEqual(self.directive('SecRequestBodyLimitAction'), 'ProcessPartial')
def test_body_buffer_stays_in_memory(self):
self.assertGreaterEqual(int(self.directive('SecRequestBodyInMemoryLimit')),
int(self.directive('SecRequestBodyLimit')))
def test_json_depth_limit_pinned(self):
"""Quadratic JSON flattening: depth 300 = 297 ms, past the fail-open
timeout. Legitimate WordPress JSON measured at depth <= 9."""
depth = int(self.directive('SecRequestBodyJsonDepthLimit'))
self.assertGreaterEqual(depth, 32, 'too tight for real block/theme payloads')
self.assertLessEqual(depth, 64, 'depth 100 already costs ~40 ms')
def test_limits_are_set_after_the_recommended_file(self):
inc = self.cfg.index('Include @coraza.conf-recommended')
for name in ('SecRequestBodyLimit ', 'SecRequestBodyJsonDepthLimit '):
lim = [i for i, l in enumerate(self.cfg) if l.startswith(name)][0]
self.assertLess(inc, lim, f'the recommended file would override {name.strip()}')
if __name__ == '__main__':
unittest.main(verbosity=2)
+13 -1
View File
@@ -128,6 +128,9 @@ STUB_FILES = {
# `lf-file` on the Coraza deny rule; loaded at parse time. Present in the # `lf-file` on the Coraza deny rule; loaded at parse time. Present in the
# image (COPY errors /haproxy/errors), stubbed for anything else. # image (COPY errors /haproxy/errors), stubbed for anything else.
'/haproxy/errors/403-waf.html': '<html><body>blocked %[unique-id]</body></html>\n', '/haproxy/errors/403-waf.html': '<html><body>blocked %[unique-id]</body></html>\n',
# `file` on the default-splash return rules; loaded at parse time too.
'/haproxy/errors/default-splash.html': '<html><body>splash</body></html>\n',
'/haproxy/errors/robots-unrouted.txt': 'User-agent: *\nDisallow: /\n',
} }
# (suspension_enabled, coraza_spoe_backend) combinations to render + validate. # (suspension_enabled, coraza_spoe_backend) combinations to render + validate.
@@ -137,6 +140,12 @@ SCENARIOS = (
'HAPROXY_SUSPENSION_ENABLED': 'true', 'HAPROXY_SUSPENSION_ENABLED': 'true',
'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000', 'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
}), }),
# The default splash is ON unless switched off; render the off shape too.
('splash-off', {
'HAPROXY_SUSPENSION_ENABLED': 'true',
'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
'HAPROXY_DEFAULT_SPLASH': 'off',
}),
) )
@@ -298,7 +307,8 @@ def render(scenario_name, env_overrides, workdir):
seed_database(os.path.join(scenario_dir, 'haproxy_config.db')) seed_database(os.path.join(scenario_dir, 'haproxy_config.db'))
saved_env = {} saved_env = {}
for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND'): for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND',
'HAPROXY_DEFAULT_SPLASH'):
saved_env[key] = os.environ.pop(key, None) saved_env[key] = os.environ.pop(key, None)
os.environ.update(env_overrides) os.environ.update(env_overrides)
@@ -330,6 +340,8 @@ def assert_scenario_branches(scenario_name, config_path, env_overrides):
'HAPROXY_SUSPENSION_ENABLED' in env_overrides), 'HAPROXY_SUSPENSION_ENABLED' in env_overrides),
'coraza': ('filter spoe engine coraza', 'coraza': ('filter spoe engine coraza',
'HAPROXY_CORAZA_SPOE_BACKEND' in env_overrides), 'HAPROXY_CORAZA_SPOE_BACKEND' in env_overrides),
'splash': ('file /haproxy/errors/default-splash.html',
env_overrides.get('HAPROXY_DEFAULT_SPLASH') != 'off'),
} }
for label, (needle, should_be_present) in expected.items(): for label, (needle, should_be_present) in expected.items():
present = needle in text present = needle in text
+2 -2
View File
@@ -4,7 +4,7 @@ backend {{ name }}-backend
option forwardfor option forwardfor
# Pass the real client IP to backend (from proxy headers or direct connection) # Pass the real client IP to backend (from proxy headers or direct connection)
# This is crucial for container-level logging and security tools # This is crucial for container-level logging and security tools
http-request add-header X-CLIENT-IP %[var(txn.real_ip)] http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
http-request set-header X-Real-IP %[var(txn.real_ip)] http-request set-header X-Real-IP %[var(txn.real_ip)]
http-request set-header X-Forwarded-For %[var(txn.real_ip)] http-request set-header X-Forwarded-For %[var(txn.real_ip)]
http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -29,7 +29,7 @@ backend {{ name }}-sse-backend
option forwardfor option forwardfor
# Pass the real client IP to backend (from proxy headers or direct connection) # Pass the real client IP to backend (from proxy headers or direct connection)
http-request add-header X-CLIENT-IP %[var(txn.real_ip)] http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
http-request set-header X-Real-IP %[var(txn.real_ip)] http-request set-header X-Real-IP %[var(txn.real_ip)]
http-request set-header X-Forwarded-For %[var(txn.real_ip)] http-request set-header X-Forwarded-For %[var(txn.real_ip)]
http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc }
+1 -1
View File
@@ -4,7 +4,7 @@ backend {{ name }}-backend
option httpchk option httpchk
# Pass the real client IP to backend (from proxy headers or direct connection) # Pass the real client IP to backend (from proxy headers or direct connection)
# This is crucial for container-level logging and security tools # This is crucial for container-level logging and security tools
http-request add-header X-CLIENT-IP %[var(txn.real_ip)] http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
http-request set-header X-Real-IP %[var(txn.real_ip)] http-request set-header X-Real-IP %[var(txn.real_ip)]
http-request set-header X-Forwarded-For %[var(txn.real_ip)] http-request set-header X-Forwarded-For %[var(txn.real_ip)]
{% if ssl_enabled %}http-request set-header X-Forwarded-Proto https if { ssl_fc }{% endif %} {% if ssl_enabled %}http-request set-header X-Forwarded-Proto https if { ssl_fc }{% endif %}
+2 -2
View File
@@ -14,7 +14,7 @@ backend {{ name }}-backend
timeout tunnel 6h timeout tunnel 6h
timeout http-keep-alive 6h timeout http-keep-alive 6h
option forwardfor option forwardfor
http-request add-header X-CLIENT-IP %[var(txn.real_ip)] http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
http-request set-header X-Real-IP %[var(txn.real_ip)] http-request set-header X-Real-IP %[var(txn.real_ip)]
http-request set-header X-Forwarded-For %[var(txn.real_ip)] http-request set-header X-Forwarded-For %[var(txn.real_ip)]
http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -31,7 +31,7 @@ backend {{ name }}-sse-backend
timeout tunnel 6h timeout tunnel 6h
timeout http-keep-alive 6h timeout http-keep-alive 6h
option forwardfor option forwardfor
http-request add-header X-CLIENT-IP %[var(txn.real_ip)] http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
http-request set-header X-Real-IP %[var(txn.real_ip)] http-request set-header X-Real-IP %[var(txn.real_ip)]
http-request set-header X-Forwarded-For %[var(txn.real_ip)] http-request set-header X-Forwarded-For %[var(txn.real_ip)]
http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc }
+2 -2
View File
@@ -7,7 +7,7 @@ backend {{ name }}-backend
timeout tunnel 6h timeout tunnel 6h
timeout http-keep-alive 6h timeout http-keep-alive 6h
option forwardfor option forwardfor
http-request add-header X-CLIENT-IP %[var(txn.real_ip)] http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
http-request set-header X-Real-IP %[var(txn.real_ip)] http-request set-header X-Real-IP %[var(txn.real_ip)]
http-request set-header X-Forwarded-For %[var(txn.real_ip)] http-request set-header X-Forwarded-For %[var(txn.real_ip)]
http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc }
@@ -24,7 +24,7 @@ backend {{ name }}-sse-backend
timeout tunnel 6h timeout tunnel 6h
timeout http-keep-alive 6h timeout http-keep-alive 6h
option forwardfor option forwardfor
http-request add-header X-CLIENT-IP %[var(txn.real_ip)] http-request set-header X-CLIENT-IP %[var(txn.real_ip)]
http-request set-header X-Real-IP %[var(txn.real_ip)] http-request set-header X-Real-IP %[var(txn.real_ip)]
http-request set-header X-Forwarded-For %[var(txn.real_ip)] http-request set-header X-Forwarded-For %[var(txn.real_ip)]
http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto https if { ssl_fc }
+39 -7
View File
@@ -10,9 +10,9 @@
# config.yaml. # config.yaml.
# #
# Reference: this config follows the shape from coraza-spoa's upstream # Reference: this config follows the shape from coraza-spoa's upstream
# example/haproxy/coraza.cfg (v0.7.1). Arg names + ordering are required by # example/haproxy/coraza.cfg (v0.7.1). Arg NAMES are a contract with
# Coraza-SPOA exactly as specified — DO NOT reorder or rename without # coraza-spoa — DO NOT rename without coordinating with the agent. Ordering:
# coordinating with the agent. # only `app` is positional (it must be FIRST); the rest are matched by name.
[coraza] [coraza]
@@ -56,9 +56,41 @@ spoe-agent coraza
# Per-request inspection message. No `event` directive — fires only when # Per-request inspection message. No `event` directive — fires only when
# explicitly invoked from haproxy.cfg via `http-request send-spoe-group`. # explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.
# Arg order/names are mandatory: Coraza-SPOA parses positionally and renames #
# break the agent. `app=str(haproxy)` is the literal application name from # How coraza-spoa v0.7.1 reads these args (measured from source, not inferred):
# coraza-spoa's config.yaml `applications:` block. # * `app` MUST be the first arg. internal/agent.go HandleSPOE reads exactly
# one KV entry and Panics if its name is not "app".
# * Every other arg is matched BY NAME in a switch (internal/application.go
# HandleRequest); order is irrelevant and unknown names are ignored with a
# debug log. So a typo'd name silently drops that input — renames still
# break inspection, they just don't error.
# `app=str(haproxy)` is the literal application name from coraza-spoa's
# config.yaml `applications:` block.
#
# id=unique-id makes Coraza's transaction ID the SAME UUID HAProxy generates
# via `unique-id-format %[uuid()]` (hap_header.tpl) — the value shown on the
# 403-waf.html page, sent as X-Request-Reference, and logged as `id=` in
# /var/log/haproxy.log. coraza-spoa passes it to waf.NewTransactionWithID(),
# so it becomes `transaction.id` in /var/log/coraza/audit.log and
# `match.unique_id` in /var/log/coraza/spoa.log: a customer-quoted reference
# greps straight to the rule that fired. Without it coraza-spoa invents a
# random 16-letter A-Z ID that appears nowhere on the HAProxy side. If the
# fetch ever came back empty the agent falls back to that random ID, so this
# cannot break inspection.
#
# exportRuleIDs=bool(true) makes coraza-spoa (application.go exportWAFMetrics)
# set var(txn.coraza.rule_ids) = comma list of every matched CRS attack rule
# (910000-959999 + 190000-199999; control rules such as our 99xxxxxxx host
# exceptions and 2000xx body rules are NOT listed), alongside rules_hit and
# anomaly_score which it sets regardless. var(txn.coraza.ruleid) is set on
# every interruption (agent.go) -- for anomaly blocks that is 949110, so
# rule_ids is what actually says WHY. All three are logged in the access
# log (hap_listener.tpl log-format: wrule= wrules= wscore=).
#
# Response inspection (a `coraza-res` message) is NOT configured:
# config.yaml has response_check: false. If it is ever enabled, coraza-res
# MUST also send `id=unique-id` -- HandleResponse
# looks the request transaction up by that ID and errors without it.
# #
# src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the # src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
# top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src, # top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
@@ -66,7 +98,7 @@ spoe-agent coraza
# when no proxy headers are present. This is what shows up as `client_ip` # when no proxy headers are present. This is what shows up as `client_ip`
# in /var/log/coraza/audit.log and the rule manager's view-matches panel. # in /var/log/coraza/audit.log and the rule manager's view-matches panel.
spoe-message coraza-req spoe-message coraza-req
args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body args app=str(haproxy) id=unique-id exportRuleIDs=bool(true) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
# Group binding for send-spoe-group invocation in the frontend. One group, # Group binding for send-spoe-group invocation in the frontend. One group,
# one message; could add more in the future (e.g. coraza-res for response # one message; could add more in the future (e.g. coraza-res for response
+58 -32
View File
@@ -2,42 +2,33 @@
# Global settings # Global settings
#--------------------------------------------------------------------- #---------------------------------------------------------------------
global global
# ACCESS LOG DESTINATION. # ACCESS LOG DESTINATION: a UNIX datagram socket INSIDE this container.
# #
# This used to be `log 127.0.0.1 local2`, which was a silent black hole: # scripts/haproxy-log-receiver.py (supervised by start-up.sh) reads it and
# 127.0.0.1 is the CONTAINER's own loopback, nothing has ever listened on # writes a size-capped, rotating /var/log/haproxy/access.log in the
# udp/514 in the container netns, and there is no /dev/log in the image. # haproxy-logs volume. No network listener anywhere.
# Every access log line -- ~1.5M/day across the whole edge -- was written
# to a socket with no receiver and dropped. Nothing errored, nothing
# warned, and `haproxy -c` was perfectly happy. The cost only shows up
# during an incident: per-IP 429s, tarpits, wp-admin gate redirects,
# WAF 403s and `silent-drop`s left no record anywhere, so the edge could
# not be asked what it had rejected. Only aggregate stick-table counters
# survived.
# #
# Now points at the DOCKER BRIDGE GATEWAY, where the host's rsyslog has an # History: `log 127.0.0.1 local2` was a black hole (nothing listened in
# imudp listener bound (installed idempotently by WHP's # the container netns); its successor, the docker bridge gateway
# setup-haproxy-logrotate.sh, which also writes the logrotate stanza). # 172.18.0.1:514 with a host rsyslog imudp listener, had no listener on
# The host writes local2 to /var/log/haproxy.log and stops it there, so it # the fleet either (probable cause, not measured: rsyslog binds that
# does not also flood /var/log/messages or the Graylog forwarder. # address once at boot, before docker has created the bridge, and does
# not retry). Either way per-IP
# 429s, tarpits, gate redirects and WAF 403s left no record.
# #
# WHY NOT `log stdout format raw local0`: it is INCOMPATIBLE with the # WHY NOT `log stdout`: silently emits NOTHING with `daemon` (below) on
# `daemon` keyword below, and incompatible SILENTLY. Verified on the # the pinned 3.0.11 (`haproxy -c` still passes), and dropping `daemon`
# pinned 3.0.11 binary: with `daemon` set, a `log stdout` config serves # breaks the synchronous `haproxy -W` launch sites in haproxy_manager.py.
# traffic normally and emits ZERO log lines, and `haproxy -c` returns 0 # Also, `docker logs haproxy-manager` feeds WHP's AI log monitor, which
# with no error and no warning -- so the CI config gate # must not get a per-request firehose.
# (scripts/validate-rendered-config.py) cannot catch it either. Making it
# work means dropping `daemon` / adding -db so haproxy stays in the
# foreground, which in turn breaks the three synchronous
# `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites in
# haproxy_manager.py (they would block until the 180s timeout and then be
# killed). That is a change to the exact code path whose failure mode is
# "container Up, ports 80/443 never bound, every site down, /health still
# 200". Not worth it for a logging change.
# #
# UDP means a dead listener degrades to dropped log lines, never to a # The path is relative to the `chroot` below: HAProxy resolves it after
# stalled or failing request path -- the correct failure direction for an # chrooting, so `/log.sock` here is /var/lib/haproxy/log.sock on disk,
# edge fronting ~60 customer sites. # which is where the receiver binds (HAPROXY_LOG_SOCKET).
#
# Sends are non-blocking: a dead receiver or a full socket buffer drops
# log lines, never stalls a request. HAPROXY_SYSLOG_TARGET still
# overrides the target (`<ip>:<port>` for a remote collector).
# #
# len 2048 accommodates the enriched log-format in hap_listener.tpl # len 2048 accommodates the enriched log-format in hap_listener.tpl
# (URL + User-Agent + UUID); the default 1024 would truncate long ones. # (URL + User-Agent + UUID); the default 1024 would truncate long ones.
@@ -106,6 +97,41 @@ global
# Stats persistence for zero-downtime reloads # Stats persistence for zero-downtime reloads
stats-file /var/lib/haproxy/stats.dat stats-file /var/lib/haproxy/stats.dat
{%- if coraza_spoe_backend %}
# WAF BODY INSPECTION LIMIT. Only emitted when Coraza is wired in, so a
# WAF-off / standalone config is unchanged byte-for-byte. 16384 is also
# HAProxy's default: this line changes no behaviour or memory, it pins
# the value so nobody raises it without reading this.
#
# The request body Coraza sees is whatever `req.body` holds when
# send-spoe-group fires, and with `option http-buffer-request`
# (hap_listener.tpl) that is "the whole body, or as much as fits in ONE
# buffer". So bufsize IS the inspection limit: a body up to roughly
# bufsize - maxrewrite (1024) - the request headers (~15 KB) is inspected
# whole and stamped `complete`; anything larger is forwarded intact, its
# first ~15 KB inspected, and stamped `partial`. Large uploads are never
# rejected or held for it.
#
# DO NOT RAISE THIS WITHOUT RE-MEASURING -- three ceilings, all measured:
# * TIME. Coraza+CRS 4.25 costs ~3.3 ms per KB of body (linear; 31 KB
# base64 = 104 ms, 15 KB = ~50 ms on a 32-core box). The SPOE
# `timeout processing` is 100 ms (hap_coraza_spoe_engine.tpl) and a
# timeout FAILS OPEN: with tune.bufsize 32768 a payload at byte 0 of a
# 30 KB base64 multipart body passed 933180 uninspected. A larger
# buffer is a bypass, not more coverage, unless the timeout (and CPU
# budget) grows with it.
# * FRAME. coraza-spoa's SPOP library (dropmorepackets/haproxy-go
# v0.0.8) refuses a HELLO whose max-frame-size exceeds 65535; HAProxy
# derives its offer from bufsize. bufsize >= 64 KB = every SPOE
# handshake fails = WAF silently off for all traffic.
# * MEMORY. maxconn 4000 x 2 buffers x bufsize, inside the 1 GB edge
# container.
# Coraza's SecRequestBodyLimit (coraza-spoa/config.yaml, 131072) must stay
# above anything this can deliver. Pinned by
# scripts/test-waf-body-inspection.py.
tune.bufsize 16384
{%- endif %}
#--------------------------------------------------------------------- #---------------------------------------------------------------------
# DNS resolver for Docker container name resolution # DNS resolver for Docker container name resolution
+157 -1
View File
@@ -63,7 +63,18 @@ frontend web
# host=/ua= identify the vhost and client; %ST/%B/%tsc give status, # host=/ua= identify the vhost and client; %ST/%B/%tsc give status,
# bytes and the termination state that distinguishes a rate-limit # bytes and the termination state that distinguishes a rate-limit
# deny (PR--) from a tarpit (PT--) from a normal close. # deny (PR--) from a tarpit (PT--) from a normal close.
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major]" # wafb=/wafe= WAF body coverage: wafb is complete|partial|- (did Coraza
# get the whole request body, see the Coraza block below);
# wafe is var(txn.coraza.error), set when the SPOE call failed or
# timed out and the request went through UNINSPECTED (fail-open).
# Both are "-" when the WAF is off or the request had no body.
# wrule=/wrules=/wscore= WHY Coraza acted, without opening audit.log:
# wrule is the interrupting rule (949110 for anomaly blocks),
# wrules every matched CRS attack rule (e.g. 920440,949110),
# wscore the inbound anomaly score. Set by coraza-spoa via the
# exportRuleIDs SPOE arg (hap_coraza_spoe_engine.tpl). All "-"
# when the WAF is off; wrule is "-" unless Coraza interrupted.
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major] wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)] wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] wscore=%[var(txn.coraza.anomaly_score)]"
# --- URI normalisation (MUST be the first path-touching block here) --- # --- URI normalisation (MUST be the first path-touching block here) ---
# Every path-based control in this frontend (the ACME health-check bypass, # Every path-based control in this frontend (the ACME health-check bypass,
@@ -605,13 +616,158 @@ frontend web
http-request set-path /suspended if is_suspended_domain http-request set-path /suspended if is_suspended_domain
use_backend default-backend if is_suspended_domain use_backend default-backend if is_suspended_domain
{%- endif %} {%- endif %}
{%- if default_splash %}
# Default splash for traffic that is NOT routed to any site: the server's
# own hostname (registered cert-only, no backend), a bare IP, a stale or
# mistyped name. HAProxy answers it ITSELF with a static page and the
# request goes no further -- in particular it never reaches the Coraza
# SPOE call below.
#
# Why: `use_backend default-backend` does not short-circuit anything --
# HAProxy runs EVERY http-request rule (Coraza included) before it picks
# a backend. So the catch-all used to be WAF-inspected with the full
# server-wide rule set like customer traffic, and a browser visiting the
# server hostname also carries the control panel's cookies (cookies are
# not port-scoped: whp01:8443 cookies are sent to whp01:443). Anything
# Coraza matched there showed the WAF block page for a host that has no
# site behind it to protect. A fixed static reply has nothing to protect.
#
# "Routed" is decided from two lists generate_config() writes from the
# SAME domain rows it renders the routing ACLs from (exact names, and
# ".base" suffixes for wildcards), port stripped and lowercased. They
# are a SUPERSET of what routes -- the routing ACLs below compare the raw
# Host (port included) -- so a host that routes is never splashed; the
# worst a mismatch can do is send an unrouted request through the WAF to
# default-backend, which is the pre-splash behaviour. Both lists are part
# of the rollback set (_config_backup_pairs) so they cannot drift from
# the haproxy.cfg that is actually loaded.
#
# Kept OUT of the splash: ACME http-01 (the server hostname's own cert is
# issued through it), blocked IPs (blocked-ip page) and suspended hosts
# (suspended page). Rate limits, the IP block and the wp-* gates above
# still apply. EVERY path is 404 with the same page, plus
# X-Robots-Tag noindex and /robots.txt "Disallow: /", so search engines
# never index the server's IP or hostname as a site.
# Disable with HAPROXY_DEFAULT_SPLASH=off (back to default-backend).
acl splash_routed_host req.hdr(host),field(1,:),lower -f /etc/haproxy/routed_hosts.list
acl splash_routed_host req.hdr(host),field(1,:),lower -m end -f /etc/haproxy/routed_wildcards.list
acl splash_acme path_beg /.well-known/acme-challenge/
acl splash_robots path /robots.txt
http-request return status 200 content-type "text/plain; charset=utf-8" hdr cache-control "no-store" hdr x-robots-tag "noindex, nofollow, noarchive" hdr x-content-type-options "nosniff" file /haproxy/errors/robots-unrouted.txt if !splash_routed_host !splash_acme !is_blocked_ip{% if suspension_enabled %} !is_suspended_domain{% endif %} splash_robots
http-request return status 404 content-type "text/html; charset=utf-8" hdr cache-control "no-store" hdr x-robots-tag "noindex, nofollow, noarchive" hdr x-content-type-options "nosniff" file /haproxy/errors/default-splash.html if !splash_routed_host !splash_acme !is_blocked_ip{% if suspension_enabled %} !is_suspended_domain{% endif %} !splash_robots
{%- endif %}
# CVE-2026-109618 (GHSA-w8hw-w9fx-j79w, PHP "RCE via OOM corruption"),
# vector (a): a multipart/form-data body with no declared length is not
# bounded by post_max_size -- PHP's multipart parser keeps reading it. A
# body WITH a Content-Length is checked against post_max_size before it is
# parsed. So a multipart request must carry exactly what every browser and
# HTTP library already sends for a form: a Content-Length.
#
# req.hdr_cnt(content-length) is the header as the client sent it, on
# every protocol: HTTP/1.1 chunked has none (and the H1 mux DROPS a
# Content-Length sent alongside Transfer-Encoding, so smuggling one in
# beside chunked does not pass either); HTTP/2 and HTTP/3 have none when
# the client streams the body. H1/H2/H3 all check a declared
# content-length against the DATA received, so it cannot lie about the
# length (same reasoning as the WAF body stamp below).
# Browsers send content-length for form posts and FormData uploads on
# h1/h2/h3; the only body a browser sends without one is a fetch() with a
# ReadableStream body (Chrome, h2/h3 only, `duplex: 'half'`), which is not
# what form uploads use.
#
# Not WAF-gated and not exempt for trusted IPs: this is a protocol
# guarantee the vulnerable parser needs, independent of Coraza and its
# mode. req.fhdr (not req.hdr) so a comma inside the value cannot split it.
# The rest of the advisory (part-header floods, empty file controls, deep
# field names) needs the body and lives in coraza-spoa/overrides.conf
# (209610-209616). Remove once patched PHP is on every customer image.
acl is_multipart_form req.fhdr(content-type) -m sub -i multipart/form-data
http-request deny deny_status 400 hdr x-whp-edge-block "multipart-without-content-length" if is_multipart_form !{ req.hdr_cnt(content-length) gt 0 }
{%- if coraza_spoe_backend %} {%- if coraza_spoe_backend %}
# Coraza WAF inspection via SPOE. Runs AFTER rate-limit and IP-block # Coraza WAF inspection via SPOE. Runs AFTER rate-limit and IP-block
# guards (no point asking the WAF about requests we're already dropping) # guards (no point asking the WAF about requests we're already dropping)
# and AFTER the real-client-IP resolution (so Coraza sees the right src). # and AFTER the real-client-IP resolution (so Coraza sees the right src).
filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
# --- Make Coraza actually see the request body -------------------------
# send-spoe-group sends `req.body` = whatever body bytes are in the buffer
# AT THAT MOMENT. Without buffering that is usually NOTHING: measured on
# haproxy 3.0.11 + coraza-spoa v0.7.1, a 14-byte `content=$a($b)` POST
# whose body was written in a second send() right after the headers
# passed 933180 (an enforced rule) uninspected, while the identical bytes
# in one send() were blocked. A body sent 2s late, or after an
# `Expect: 100-continue`, was never inspected at all.
#
# `option http-buffer-request` makes HAProxy wait, before evaluating the
# http-request rules, until the WHOLE body is in or the buffer is full,
# bounded by `timeout http-request` (30s, hap_header.tpl). It answers
# `Expect: 100-continue` itself so the client actually sends the body.
# It does NOT hold uploads: once one buffer (tune.bufsize, 16 KB) is full
# the request is forwarded and the rest streams through as before -- a
# 50 MB upload is not stored at the edge.
option http-buffer-request
# Exactly one Content-Type, or 400. With two Content-Type lines Coraza and
# the backend disagree about what the body IS: Coraza evaluates every value
# (`text/xml` on either line selects its XML processor, which accepts a
# form body without error), while Apache merges the lines with ", " and
# PHP cuts the media type at the first ',' / ';'. Measured (Coraza v3.7.0
# + PHP 8.3): `application/x-www-form-urlencoded` + `text/xml` puts the
# payload in $_POST with nothing inspected. req.fhdr_cnt counts header
# LINES, so a single comma-joined value (which browsers produce when a
# script calls setRequestHeader twice) is not affected here; that form is
# parsed consistently enough by both sides (see the report's matrix).
# Browsers, curl, PHP-curl and the common HTTP libraries send one line.
http-request deny deny_status 400 if { req.fhdr_cnt(content-type) gt 1 }
# Tell Coraza whether it got the whole body. Coraza cannot tell on its
# own (REQUEST_BODY_LENGTH is only filled by the urlencoded/raw
# processors), and the difference matters: a TRUNCATED JSON body, or a
# multipart body cut inside a boundary/part header, fails to parse by
# construction, so the body-parse-error rules (200002/200003) may only
# deny a COMPLETE body. WHP's generated Coraza rules
# (regenerate-coraza-overrides.sh) key on this header:
# complete -> 200002/200003 deny on multipart/JSON (enforce mode)
# partial -> 200002/200003 removed for this request (every mode)
#
# With a Content-Length: complete iff req.body_len (buffered) equals the
# Content-Length HEADER value (req.hdr_val). NOT req.body_size: that is
# "buffered + htx->extra", and in HAProxy 3.0 only the H1 and H2 muxes
# set htx->extra from Content-Length -- the H3 mux never does (h3.c), so
# over HTTP/3 req.body_size == req.body_len and every body cut at one
# buffer was stamped `complete`. Measured: whp01 logged 73,795 H3
# `complete` and 0 `partial` (Oct 6-19 2026); a 265,778-byte Divi save
# over H3 reached Coraza as its first ~15 KB, failed to parse and was
# 403'd by WHP's 991900020. H1/H2/H3 all validate Content-Length against
# the DATA actually received (h3.c rejects a mismatch), so the header is
# a protocol-independent "how long the body really is".
# Without one (HTTP/1.1 chunked, HTTP/2-3 without content-length) HAProxy
# 3.0 has no "end of message" fetch and req.body_size is just what is
# buffered. But http-buffer-request only releases the request early when
# the body ENDED or the buffer FILLED (a stalled body gets a 408 instead),
# and body + headers well below one buffer (12 KB of the ~15 KB usable)
# cannot have filled it. So that is complete, anything else partial.
# Without this branch an attacker only had to add
# `Transfer-Encoding: chunked` to escape enforcement (measured).
#
# Client-supplied copies are deleted FIRST so it cannot be forged, and it
# is deleted again after the SPOE call so backends never see it.
# txn.waf_body is also logged (`wafb=` in the log-format above) so the
# complete/partial mix is measurable per host.
http-request del-header X-WHP-WAF-Body
http-request set-var(txn.waf_body_size) req.hdr_val(content-length) if { req.hdr_cnt(content-length) eq 1 }
http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_body_size) -m found } { req.body_len,sub(txn.waf_body_size) eq 0 }
http-request set-var(txn.waf_body) str(partial) if { var(txn.waf_body_size) -m found } !{ req.body_len,sub(txn.waf_body_size) eq 0 }
http-request set-var(txn.waf_hdrs_len) req.hdrs,length if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 }
http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_hdrs_len) -m found } { req.body_len,add(txn.waf_hdrs_len) lt 12288 }
http-request set-var(txn.waf_body) str(partial) if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 } !{ var(txn.waf_body) -m found }
http-request set-header X-WHP-WAF-Body %[var(txn.waf_body)] if { var(txn.waf_body) -m found }
http-request send-spoe-group coraza coraza-req http-request send-spoe-group coraza coraza-req
http-request del-header X-WHP-WAF-Body
# Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to # Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to
# "deny" / "drop" / "redirect" when a rule with the corresponding # "deny" / "drop" / "redirect" when a rule with the corresponding