HAProxy 3.2 LTS edge image (default for all new deployments) #20

Merged
jknapp merged 9 commits from feat/haproxy-upgrade into main 2026-11-17 21:07:34 +00:00
9 Commits
Author SHA1 Message Date
shadowdaoandClaude Sonnet 5.5 aebb68fe9d ci: make tag verification independent of buildx version
buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'`
and prints the whole inspect text, so the verify step rejected every ref as a
"bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh:
the index digest is the sha256 of `inspect --raw`, cross-checked against the
anchored `Digest:` line and a bare --format output when present; platforms are
read from the raw JSON. All other checks (every tag in every registry, platform
match, one digest across registries, any miss fails) are unchanged. Adds an
offline test with a stub emulating 0.28 and 0.38 output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:30:21 -08:00
shadowdaoandClaude Sonnet 5.5 ba8722b0b7 chore: bump VERSION to 2026.11.6 (rc with gunicorn access-log fix + hardened CI)
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 13m10s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:13:46 -08:00
shadowdaoandClaude Sonnet 5.5 fa8fec0b4f fix(logging): keep gunicorn access logs off container stdout
Both gunicorns ran with --access-logfile -, so every :8080 default-page /
blocked-IP / suspended-site hit and every :8000 API call became a docker
stdout line. With the journald log driver (WHP PR #260) that is one journal
entry per request: journald's default rate limit (10000 msgs / 30 s per
service) could suppress the relay's crash-dump lines, and Graylog would be
flooded (the edge access log must stay out of it).

* :8080 default-page gunicorn: no access log.
* :8000 API gunicorn: access log to /var/log/haproxy/api-access.log (mode
  0640, haproxy-logs volume), bounded by scripts/gunicorn-access-rotate.py
  (copy-truncate, 20 MiB x 3, 30 s poll). 401/403 are in the file only.
* gunicorn ERROR logging stays on stderr; supervisor/relay/startup lines
  unchanged on stdout/stderr.
* New build-gate test scripts/test-gunicorn-access-logging.py; docs/logging.md.

VERSION unchanged (2026.11.5): a NEW rc image must be built from this branch
for the change to ship.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:12:53 -08:00
shadowdaoandClaude Sonnet 5.5 68976142f5 ci: harden assemble step (pinned buildx, plain progress, timeout) and verify every tag in every registry
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 21:10:51 -08:00
shadowdaoandClaude Sonnet 5.5 99279d5347 chore: bump VERSION to 2026.11.5 (HAProxy 3.2 rc)
HAProxy Manager Build and Push / Build-and-Push (push) Failing after 18m29s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 20:42:04 -08:00
claudeandClaude Sonnet 5.5 ea2d4994f1 fix(image): exact haproxy pin + relay tenant-alert class + supervisor backoff + rc publishing
I1: ARG HAPROXY_EXACT=3.2.26-1~bpo13+1 is the apt pin; build fails unless dpkg-query equals it exactly (fail closed when the repo is unreachable); comment corrected.
I2: relay no longer exempts every [ALERT]; tenant-driven 'has no server available' / 'Server ... is DOWN|UP' are their own rate-limited stdout class; dump blocks have their own budget.
Minor: keyring must hold exactly one pub key (gnupg purged), stderr log + rotated copies 0640, supervisor extracted to scripts/haproxy-supervisor.sh (integer-validated knobs, crash-loop backoff 2/5/10/30, reset after 60s).
C1: release/* branches and rc-* tags publish only :<VERSION>-rc and :<sha>, never :latest; main unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 17:33:27 -08:00
shadowdaoandClaude Sonnet 5.5 c2e070d172 fix(relay): anchor crash patterns to HAProxy message structure; crash-evidence budget + dump cooldown
Security review: unanchored keywords let echoed client data bypass the limiter and open
dump blocks; bounds re-armed immediately. Patterns are now anchored, blocks open only on
the 'Thread N is about to kill' header, a 1000-line/10-min budget caps exempt lines
(overflow goes through the normal bucket and the suppressed summary) and a 60 s cooldown
follows a bound-closed block. Tests: forged keywords, flood, cooldown + 3 new mutants.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 15:08:46 -08:00
shadowdaoandClaude Sonnet 5.5 5e27870608 fix(relay): never rate-limit crash evidence; collapse reload noise; stdout/stderr priority split
The 3.2 canary showed the relay's 400-line bucket, drained by ~110 reload lines
per reload, dropping a crash dump (journal kept only 'Thread 1 is about to kill').
Crash lines and whole dump blocks now bypass the bucket (bounded block), routine
reload noise is collapsed to one line per reload, the remainder keeps the limit
with a per-minute '[relay] suppressed N lines' summary. Benign lines go to stdout
(journald PRIORITY 6), crash evidence to stderr (PRIORITY 3).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 15:04:14 -08:00
Claude a753d300db feat: HAProxy 3.2 LTS (haproxy.debian.net) + capture HAProxy stderr + 2 s supervisor
DRAFT for review, not merged. Context: HAProxy 3.0.11 (Debian trixie) aborts via
its own watchdog in the DNS resolver (haproxy #3101, fixed in 3.0.13/3.1.11/3.2.6).

* Dockerfile: haproxy from https://haproxy.debian.net trixie-backports-3.2 (committed
  signing key, fingerprint asserted, series pin 3.2.*, optional HAPROXY_EXACT),
  native OpenSSL QUIC asserted (no compat layer), /haproxy/HAPROXY_VERSION.
* start HAProxy as `haproxy -W -db` (spawn_haproxy_master): `daemon` sent the
  master's stderr to /dev/null, losing ALERTs and the ha_panic thread dump.
  stderr -> /var/log/haproxy/haproxy-stderr.log, forwarded to container stderr by
  scripts/haproxy-stderr-relay.py (rate limited, rotated).
* supervisor: pidof probe every 2 s (was python every 15 s), exponential backoff
  on failed restarts.
* tests: relay unit tests; rollback/cert stubs print "Loading success.".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 13:02:33 -08:00