HAProxy 3.2 LTS becomes the default edge image (Josh decided 2026-11-17: roll 3.2 to all hosts and use it for all future deployments).
Contents (9 commits, all reviewed by Opus; running in production as the rc6 image 2026.11.6-rc (index sha256:a0cc66d1844935266ce5a5ecacfb714deace485e74fbb066eb2bb9dfbbc450e8) on whp01, whp02, whp03, sdbees and TrueSelfCA since 2026-11-16/17 with no aborts):
HAProxy 3.2.26 from haproxy.debian.net (exact version pin 3.2.26-1~bpo13+1, fingerprint-asserted key, build fails closed)
stderr relay (crash evidence never starved; tenant-driven lines rate-limited) and a supervisor with backoff
gunicorn access logs no longer on container stdout (API access log to a bounded file in the logs volume)
hardened CI (pinned buildx, timeouts, publish verification independent of buildx version; release/* and rc-* refs publish only -rc tags and never latest)
Every production host is pinned by digest (whp edge image set, WHP >= 2026.11.57), so this merge does not change any running edge; it changes what NEW installs and unpinned hosts get (:latest).
Known first-run item: the hardened 'Verify published tags' CI step (commit aebb68f) has only been exercised by unit tests with a stub buildx; a red result on the first main run with correct tags would be that step, not the image.
HAProxy 3.2 LTS becomes the default edge image (Josh decided 2026-11-17: roll 3.2 to all hosts and use it for all future deployments).
Contents (9 commits, all reviewed by Opus; running in production as the rc6 image `2026.11.6-rc` (index sha256:a0cc66d1844935266ce5a5ecacfb714deace485e74fbb066eb2bb9dfbbc450e8) on whp01, whp02, whp03, sdbees and TrueSelfCA since 2026-11-16/17 with no aborts):
- HAProxy 3.2.26 from haproxy.debian.net (exact version pin `3.2.26-1~bpo13+1`, fingerprint-asserted key, build fails closed)
- stderr relay (crash evidence never starved; tenant-driven lines rate-limited) and a supervisor with backoff
- gunicorn access logs no longer on container stdout (API access log to a bounded file in the logs volume)
- hardened CI (pinned buildx, timeouts, publish verification independent of buildx version; release/* and rc-* refs publish only `-rc` tags and never `latest`)
Every production host is pinned by digest (`whp edge image set`, WHP >= 2026.11.57), so this merge does not change any running edge; it changes what NEW installs and unpinned hosts get (`:latest`).
Known first-run item: the hardened 'Verify published tags' CI step (commit aebb68f) has only been exercised by unit tests with a stub buildx; a red result on the first main run with correct tags would be that step, not the image.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
DRAFT for review, not merged. Context: HAProxy 3.0.11 (Debian trixie) aborts via
its own watchdog in the DNS resolver (haproxy #3101, fixed in 3.0.13/3.1.11/3.2.6).
* Dockerfile: haproxy from https://haproxy.debian.net trixie-backports-3.2 (committed
signing key, fingerprint asserted, series pin 3.2.*, optional HAPROXY_EXACT),
native OpenSSL QUIC asserted (no compat layer), /haproxy/HAPROXY_VERSION.
* start HAProxy as `haproxy -W -db` (spawn_haproxy_master): `daemon` sent the
master's stderr to /dev/null, losing ALERTs and the ha_panic thread dump.
stderr -> /var/log/haproxy/haproxy-stderr.log, forwarded to container stderr by
scripts/haproxy-stderr-relay.py (rate limited, rotated).
* supervisor: pidof probe every 2 s (was python every 15 s), exponential backoff
on failed restarts.
* tests: relay unit tests; rollback/cert stubs print "Loading success.".
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
The 3.2 canary showed the relay's 400-line bucket, drained by ~110 reload lines
per reload, dropping a crash dump (journal kept only 'Thread 1 is about to kill').
Crash lines and whole dump blocks now bypass the bucket (bounded block), routine
reload noise is collapsed to one line per reload, the remainder keeps the limit
with a per-minute '[relay] suppressed N lines' summary. Benign lines go to stdout
(journald PRIORITY 6), crash evidence to stderr (PRIORITY 3).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
Security review: unanchored keywords let echoed client data bypass the limiter and open
dump blocks; bounds re-armed immediately. Patterns are now anchored, blocks open only on
the 'Thread N is about to kill' header, a 1000-line/10-min budget caps exempt lines
(overflow goes through the normal bucket and the suppressed summary) and a 60 s cooldown
follows a bound-closed block. Tests: forged keywords, flood, cooldown + 3 new mutants.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
I1: ARG HAPROXY_EXACT=3.2.26-1~bpo13+1 is the apt pin; build fails unless dpkg-query equals it exactly (fail closed when the repo is unreachable); comment corrected.
I2: relay no longer exempts every [ALERT]; tenant-driven 'has no server available' / 'Server ... is DOWN|UP' are their own rate-limited stdout class; dump blocks have their own budget.
Minor: keyring must hold exactly one pub key (gnupg purged), stderr log + rotated copies 0640, supervisor extracted to scripts/haproxy-supervisor.sh (integer-validated knobs, crash-loop backoff 2/5/10/30, reset after 60s).
C1: release/* branches and rc-* tags publish only :<VERSION>-rc and :<sha>, never :latest; main unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
Both gunicorns ran with --access-logfile -, so every :8080 default-page /
blocked-IP / suspended-site hit and every :8000 API call became a docker
stdout line. With the journald log driver (WHP PR #260) that is one journal
entry per request: journald's default rate limit (10000 msgs / 30 s per
service) could suppress the relay's crash-dump lines, and Graylog would be
flooded (the edge access log must stay out of it).
* :8080 default-page gunicorn: no access log.
* :8000 API gunicorn: access log to /var/log/haproxy/api-access.log (mode
0640, haproxy-logs volume), bounded by scripts/gunicorn-access-rotate.py
(copy-truncate, 20 MiB x 3, 30 s poll). 401/403 are in the file only.
* gunicorn ERROR logging stays on stderr; supervisor/relay/startup lines
unchanged on stdout/stderr.
* New build-gate test scripts/test-gunicorn-access-logging.py; docs/logging.md.
VERSION unchanged (2026.11.5): a NEW rc image must be built from this branch
for the change to ship.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'`
and prints the whole inspect text, so the verify step rejected every ref as a
"bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh:
the index digest is the sha256 of `inspect --raw`, cross-checked against the
anchored `Digest:` line and a bare --format output when present; platforms are
read from the raw JSON. All other checks (every tag in every registry, platform
match, one digest across registries, any miss fails) are unchanged. Adds an
offline test with a stub emulating 0.28 and 0.38 output.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
jknapp
merged commit 193d39e109 into main2026-11-17 21:07:34 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
HAProxy 3.2 LTS becomes the default edge image (Josh decided 2026-11-17: roll 3.2 to all hosts and use it for all future deployments).
Contents (9 commits, all reviewed by Opus; running in production as the rc6 image
2026.11.6-rc(index sha256:a0cc66d1844935266ce5a5ecacfb714deace485e74fbb066eb2bb9dfbbc450e8) on whp01, whp02, whp03, sdbees and TrueSelfCA since 2026-11-16/17 with no aborts):3.2.26-1~bpo13+1, fingerprint-asserted key, build fails closed)-rctags and neverlatest)Every production host is pinned by digest (
whp edge image set, WHP >= 2026.11.57), so this merge does not change any running edge; it changes what NEW installs and unpinned hosts get (:latest).Known first-run item: the hardened 'Verify published tags' CI step (commit
aebb68f) has only been exercised by unit tests with a stub buildx; a red result on the first main run with correct tags would be that step, not the image.🤖 Generated with Claude Code
https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'` and prints the whole inspect text, so the verify step rejected every ref as a "bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh: the index digest is the sha256 of `inspect --raw`, cross-checked against the anchored `Digest:` line and a bare --format output when present; platforms are read from the raw JSON. All other checks (every tag in every registry, platform match, one digest across registries, any miss fails) are unchanged. Adds an offline test with a stub emulating 0.28 and 0.38 output. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6