From aebb68fe9d1f27d052e7d395e483b735a88845bd Mon Sep 17 00:00:00 2001 From: Josh Knapp Date: Mon, 16 Nov 2026 22:30:21 -0800 Subject: [PATCH] ci: make tag verification independent of buildx version buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'` and prints the whole inspect text, so the verify step rejected every ref as a "bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh: the index digest is the sha256 of `inspect --raw`, cross-checked against the anchored `Digest:` line and a bare --format output when present; platforms are read from the raw JSON. All other checks (every tag in every registry, platform match, one digest across registries, any miss fails) are unchanged. Adds an offline test with a stub emulating 0.28 and 0.38 output. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6 --- .gitea/workflows/build-push.yaml | 33 ++--- CLAUDE.md | 1 + scripts/test-verify-published-tags.py | 193 ++++++++++++++++++++++++++ scripts/verify-published-tags.sh | 97 +++++++++++++ 4 files changed, 301 insertions(+), 23 deletions(-) create mode 100755 scripts/test-verify-published-tags.py create mode 100755 scripts/verify-published-tags.sh diff --git a/.gitea/workflows/build-push.yaml b/.gitea/workflows/build-push.yaml index cdfde84..407307d 100644 --- a/.gitea/workflows/build-push.yaml +++ b/.gitea/workflows/build-push.yaml @@ -199,26 +199,13 @@ jobs: PLATFORMS: ${{ steps.assemble.outputs.platforms }} run: | set -euo pipefail - repos="repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base ghcr.io/shadowdao/haproxy-manager-base" - want=$(printf '%s' "$PLATFORMS" | tr ',' '\n' | sort | tr '\n' ' ') - ref_digest=""; ref_name=""; fail=0 - for r in $repos; do - for t in $TAGS; do - ref="${r}:${t}" - if ! d=$(timeout 90 $BUILDX imagetools inspect "$ref" --format '{{.Manifest.Digest}}' 2>&1); then - echo "::error::MISSING or unreadable: ${ref}: ${d}"; fail=1; continue - fi - [[ "$d" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "::error::bad digest for ${ref}: ${d}"; fail=1; continue; } - got=$(timeout 90 $BUILDX imagetools inspect "$ref" --format '{{range .Manifest.Manifests}}{{if ne .Platform.OS "unknown"}}{{.Platform.OS}}/{{.Platform.Architecture}}{{"\n"}}{{end}}{{end}}' | sort | tr '\n' ' ') - if [ "$got" != "$want" ]; then - echo "::error::platform mismatch for ${ref}: have [${got}] want [${want}]"; fail=1 - fi - echo "ok ${ref} ${d} [${got}]" - if [ -z "$ref_digest" ]; then ref_digest="$d"; ref_name="$ref" - elif [ "$d" != "$ref_digest" ]; then - echo "::error::digest mismatch: ${ref}=${d} vs ${ref_name}=${ref_digest}"; fail=1 - fi - done - done - [ "$fail" = 0 ] || { echo "::error::tag verification FAILED"; exit 1; } - echo "verified: ${TAGS} in both registries, digest ${ref_digest}, platforms ${PLATFORMS}" + # Version-independent digest logic lives in the script (tested by + # scripts/test-verify-published-tags.py); buildx 0.28 ignores --format. + export IMAGETOOLS="${BUILDX} imagetools" + tag_args=() + for t in $TAGS; do tag_args+=(--tag "$t"); done + scripts/verify-published-tags.sh \ + --platforms "$PLATFORMS" \ + --repo repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base \ + --repo ghcr.io/shadowdao/haproxy-manager-base \ + "${tag_args[@]}" diff --git a/CLAUDE.md b/CLAUDE.md index c76740a..2da050c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,6 +14,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co - **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine. - **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired//` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`. - **Multi-arch index assembly (coraza-spoa)**: `python3 scripts/test-assemble-multiarch-index.py` - offline (NOT run in the image build; it also reads `.gitea/workflows/build-push-coraza.yaml`); `scripts/assemble-multiarch-index.sh` turns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome is `success` (failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing. `scripts/e2e-coraza-multiarch.sh` rehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch"). +- **Publish verify gate**: `python3 scripts/test-verify-published-tags.py` - offline (needs `jq`; NOT run in the image build); `scripts/verify-published-tags.sh` is the last build-push.yaml step: every tag in every registry must resolve, carry the assembled platforms, and share ONE index digest. The digest is the sha256 of `imagetools inspect --raw` cross-checked against the text `Digest:` line / bare `--format` output (buildx 0.28 ignores `--format` and prints full text; CI run 5381). - **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash". - **gunicorn access logging**: `python3 scripts/test-gunicorn-access-logging.py` - real gunicorn; :8080 has no access log, :8000 access goes to `/var/log/haproxy/api-access.log` (rotated by `scripts/gunicorn-access-rotate.py`), nothing per-request on stdout (journald rate-limit/Graylog flood), gunicorn errors stay on stderr. See docs/logging.md. Run after touching start-up.sh's gunicorn lines. - **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`. diff --git a/scripts/test-verify-published-tags.py b/scripts/test-verify-published-tags.py new file mode 100755 index 0000000..e189e3d --- /dev/null +++ b/scripts/test-verify-published-tags.py @@ -0,0 +1,193 @@ +#!/usr/bin/env python3 +"""Offline tests for scripts/verify-published-tags.sh (final publish gate). + +CI run 5381 failed because buildx v0.28.0 ignores `--format '{{.Manifest.Digest}}'` +and prints the whole inspect text, while buildx 0.38 prints a bare digest. A stub +`imagetools` emulates BOTH output styles (plus `--raw`), so no registry or Docker +is needed. Held invariants: + - the 0.28-style (full text) and 0.38-style (bare digest) outputs both verify + - a reported digest that disagrees with the sha256 of the raw manifest fails + - a missing tag, a platform mismatch, or different digests across registries / + tags fail the job + - build-push.yaml's verify step uses the script, and release/* + rc-* tags + resolve to ONLY -rc and (never latest) + +Running: python3 scripts/test-verify-published-tags.py (needs jq) +""" +import hashlib +import json +import os +import re +import stat +import subprocess +import tempfile +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +SCRIPT = os.path.join(HERE, 'verify-published-tags.sh') +WORKFLOW = os.path.join(ROOT, '.gitea', 'workflows', 'build-push.yaml') + +STUB = r'''#!/usr/bin/env python3 +import hashlib, json, os, sys +a = sys.argv[1:] +mode = os.environ['STUB_MODE'] # v028 | v038 +table = json.load(open(os.environ['STUB_TABLE'])) # ref -> {"raw": str, "lie": optional digest} +assert a[0] == 'inspect' +ref = [x for x in a[1:] if not x.startswith('-') and not x.startswith('{{')][0] +ent = table.get(ref) +if ent is None: + sys.stderr.write('ERROR: %s: not found\n' % ref); sys.exit(1) +raw = ent['raw'].encode() + (b'\n' if ent.get('nl') else b'') +digest = ent.get('lie') or 'sha256:' + hashlib.sha256(ent['raw'].encode()).hexdigest() +if '--raw' in a: + sys.stdout.buffer.write(raw); sys.exit(0) +if '--format' in a: + if mode == 'v038': + print(digest) + else: # 0.28: --format ignored, whole text printed + print('Name: ' + ref); print('MediaType: application/vnd.oci.image.index.v1+json') + print('Digest: ' + digest) + sys.exit(0) +print('Name: ' + ref) +print('MediaType: application/vnd.oci.image.index.v1+json') +print('Digest: ' + digest) +print(' ') +print('Manifests: ') +for m in json.loads(ent['raw'])['manifests']: + print(' Name: %s@%s' % (ref, m['digest'])) + print(' Digest: sha256:' + 'f' * 64) # indented: must never be mistaken for the index digest + print(' Platform: %s/%s' % (m['platform']['os'], m['platform']['architecture'])) +''' + + +def index(archs, salt=''): + ms = [{'mediaType': 'application/vnd.oci.image.manifest.v1+json', + 'digest': 'sha256:' + hashlib.sha256((x + salt).encode()).hexdigest(), 'size': 1, + 'platform': {'architecture': x, 'os': 'linux'}} for x in archs] + ms.append({'mediaType': 'application/vnd.oci.image.manifest.v1+json', + 'digest': 'sha256:' + '7' * 64, 'size': 1, + 'platform': {'architecture': 'unknown', 'os': 'unknown'}, + 'annotations': {'vnd.docker.reference.type': 'attestation-manifest'}}) + return json.dumps({'schemaVersion': 2, 'mediaType': 'application/vnd.oci.image.index.v1+json', + 'manifests': ms}, separators=(',', ':')) + + +REPOS = ['reg.one/a/img', 'reg.two/b/img'] +TAGS = ['1.2-rc', 'abc123'] + + +class VerifyTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.stub = os.path.join(self.tmp.name, 'stub') + open(self.stub, 'w').write(STUB) + os.chmod(self.stub, os.stat(self.stub).st_mode | stat.S_IEXEC) + self.table = os.path.join(self.tmp.name, 'table.json') + + def tearDown(self): + self.tmp.cleanup() + + def table_for(self, raw, **over): + t = {'%s:%s' % (r, g): {'raw': raw} for r in REPOS for g in TAGS} + for k, v in over.items(): + ref = k.replace('__', ':').replace('_', '/', 0) + t[ref] = v + return t + + def run_script(self, table, mode, platforms='linux/amd64,linux/arm64', tags=TAGS): + json.dump(table, open(self.table, 'w')) + args = [SCRIPT, '--platforms', platforms] + for r in REPOS: + args += ['--repo', r] + for t in tags: + args += ['--tag', t] + env = dict(os.environ, IMAGETOOLS=self.stub, STUB_MODE=mode, STUB_TABLE=self.table) + return subprocess.run(args, env=env, capture_output=True, text=True, timeout=60) + + def test_buildx_028_full_text_output_verifies(self): + r = self.run_script(self.table_for(index(['amd64', 'arm64'])), 'v028') + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn('verified:', r.stdout) + self.assertNotIn('bad digest', r.stdout) + + def test_buildx_038_bare_digest_output_verifies(self): + r = self.run_script(self.table_for(index(['amd64', 'arm64'])), 'v038') + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + + def test_trailing_newline_on_raw_output_tolerated(self): + raw = index(['amd64', 'arm64']) + t = {'%s:%s' % (r, g): {'raw': raw, 'nl': True} for r in REPOS for g in TAGS} + self.assertEqual(self.run_script(t, 'v028').returncode, 0) + + def test_amd64_only_publish_matches_platforms(self): + raw = index(['amd64']) + self.assertEqual(self.run_script(self.table_for(raw), 'v028', 'linux/amd64').returncode, 0) + r = self.run_script(self.table_for(raw), 'v028', 'linux/amd64,linux/arm64') + self.assertEqual(r.returncode, 1) + self.assertIn('platform mismatch', r.stdout) + + def test_missing_tag_fails(self): + t = self.table_for(index(['amd64', 'arm64'])) + del t['reg.two/b/img:abc123'] + r = self.run_script(t, 'v038') + self.assertEqual(r.returncode, 1) + self.assertIn('MISSING', r.stdout) + + def test_different_digest_across_registries_fails(self): + t = self.table_for(index(['amd64', 'arm64'])) + t['reg.two/b/img:1.2-rc'] = {'raw': index(['amd64', 'arm64'], salt='other')} + for mode in ('v028', 'v038'): + r = self.run_script(t, mode) + self.assertEqual(r.returncode, 1, mode) + self.assertIn('digest mismatch', r.stdout) + + def test_reported_digest_disagreeing_with_raw_manifest_fails(self): + t = self.table_for(index(['amd64', 'arm64'])) + t['reg.one/a/img:1.2-rc']['lie'] = 'sha256:' + '0' * 64 + for mode in ('v028', 'v038'): + r = self.run_script(t, mode) + self.assertEqual(r.returncode, 1, mode) + self.assertIn('cross-check FAILED', r.stdout) + + def test_garbage_manifest_fails(self): + t = self.table_for('not json') + self.assertEqual(self.run_script(t, 'v038').returncode, 1) + + +class WorkflowTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + import yaml + cls.src = open(WORKFLOW).read() + cls.doc = yaml.safe_load(cls.src) + cls.steps = cls.doc['jobs']['Build-and-Push']['steps'] + + def step(self, name): + return [s for s in self.steps if s.get('name') == name][0] + + def test_verify_step_uses_script_not_format_digest(self): + run = self.step('Verify published tags in every registry')['run'] + self.assertIn('scripts/verify-published-tags.sh', run) + self.assertNotIn('{{.Manifest.Digest}}', run) + + def test_run_blocks_are_valid_bash(self): + for s in self.steps: + if 'run' in s: + p = subprocess.run(['bash', '-n'], input=s['run'], text=True, capture_output=True) + self.assertEqual(p.returncode, 0, '%s: %s' % (s.get('name'), p.stderr)) + + def test_release_and_rc_never_publish_latest(self): + run = self.step('Resolve tags')['run'] + m = re.search(r'refs/heads/release/\*\|refs/tags/rc-\*\)\s*\n\s*tags="([^"]*)"', run) + self.assertIsNotNone(m) + self.assertEqual(m.group(1), '${VERSION}-rc ${SHA}') + main = re.search(r'refs/heads/main\)\s*\n\s*tags="([^"]*)"', run) + self.assertIsNotNone(main) + self.assertEqual(main.group(1), 'latest ${VERSION} ${SHA}') + # latest appears in no other tag assignment + self.assertEqual(len(re.findall(r'tags="[^"]*latest', run)), 1) + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/verify-published-tags.sh b/scripts/verify-published-tags.sh new file mode 100755 index 0000000..3ee6905 --- /dev/null +++ b/scripts/verify-published-tags.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# Final publish gate: every : must resolve, carry exactly the +# platforms the assemble step reported, and have ONE index digest across all +# tags and all registries. Any miss or mismatch exits 1. +# +# verify-published-tags.sh --platforms linux/amd64,linux/arm64 \ +# --repo [--repo ...] --tag [--tag ...] +# +# The index digest is computed in a buildx-VERSION-INDEPENDENT way. CI run 5381 +# showed buildx v0.28.0 ignores `--format '{{.Manifest.Digest}}'` and prints the +# whole inspect text, while 0.38 prints a bare digest. So the digest comes from: +# 1. sha256 of the raw manifest bytes (`inspect --raw`) - always, authoritative +# (a content digest IS the sha256 of the manifest bytes); +# 2. the strictly anchored `Digest:` line of the default text output, if present; +# 3. the `--format '{{.Manifest.Digest}}'` output, ONLY if it is a bare digest +# (0.28's full-text output is ignored here, not an error). +# Every source that is available must agree, else the ref fails. +# Platforms are read from the raw JSON (also version independent). +# +# Env: IMAGETOOLS overrides the command (default "docker buildx imagetools"); +# the CI step sets it to "$BUILDX imagetools"; the test suite uses a stub. +set -uo pipefail + +IMAGETOOLS=${IMAGETOOLS:-docker buildx imagetools} +platforms=""; repos=(); tags=() +while [ $# -gt 0 ]; do + case "$1" in + --platforms) platforms=${2-}; shift 2 ;; + --repo) repos+=("${2-}"); shift 2 ;; + --tag) tags+=("${2-}"); shift 2 ;; + *) echo "verify-published-tags: unknown argument: $1" >&2; exit 2 ;; + esac +done +[ -n "$platforms" ] && [ ${#repos[@]} -gt 0 ] && [ ${#tags[@]} -gt 0 ] \ + || { echo "::error::verify-published-tags: --platforms, --repo and --tag are required"; exit 2; } +command -v jq >/dev/null || { echo "::error::verify-published-tags: jq is required"; exit 2; } + +want=$(printf '%s' "$platforms" | tr ',' '\n' | sort | tr '\n' ' ') +work=$(mktemp -d); trap 'rm -rf "$work"' EXIT +digest_re='^sha256:[0-9a-f]{64}$' +ref_digest=""; ref_name=""; fail=0 + +for r in "${repos[@]}"; do + for t in "${tags[@]}"; do + ref="${r}:${t}" + raw="$work/raw" + if ! timeout 90 $IMAGETOOLS inspect --raw "$ref" >"$raw" 2>"$work/err" || [ ! -s "$raw" ]; then + echo "::error::MISSING or unreadable: ${ref}: $(head -c 300 "$work/err")"; fail=1; continue + fi + # Raw bytes as served, and with trailing newlines stripped (some buildx + # versions append one when printing); the content digest is one of the two. + h1="sha256:$(sha256sum <"$raw" | cut -d' ' -f1)" + h2="sha256:$(printf '%s' "$(cat "$raw")" | sha256sum | cut -d' ' -f1)" + + # Source 2: strictly anchored top-level "Digest:" line of the text output. + textd="" + if timeout 90 $IMAGETOOLS inspect "$ref" >"$work/text" 2>/dev/null; then + mapfile -t lines < <(grep -E '^Digest:[[:space:]]+sha256:[0-9a-f]{64}[[:space:]]*$' "$work/text" | sed -E 's/^Digest:[[:space:]]+//; s/[[:space:]]+$//') + [ ${#lines[@]} -eq 1 ] && textd=${lines[0]} + fi + # Source 3: bare --format output, only when it really is a bare digest. + fmtd="" + if out=$(timeout 90 $IMAGETOOLS inspect "$ref" --format '{{.Manifest.Digest}}' 2>/dev/null); then + out=$(printf '%s' "$out" | tr -d '[:space:]') + [[ "$out" =~ $digest_re ]] && fmtd=$out + fi + + d="" + for cand in "$textd" "$fmtd"; do + [ -n "$cand" ] || continue + if [ "$cand" != "$h1" ] && [ "$cand" != "$h2" ]; then + echo "::error::digest cross-check FAILED for ${ref}: reported ${cand}, raw manifest hashes to ${h1}"; fail=1; d="!"; break + fi + d=$cand + done + [ "$d" = "!" ] && continue + [ -n "$d" ] || d=$h2 + [[ "$d" =~ $digest_re ]] || { echo "::error::bad digest for ${ref}: ${d}"; fail=1; continue; } + if [ -z "$textd$fmtd" ]; then + echo "::warning::no reported digest for ${ref}; using sha256 of the raw manifest" + fi + + if ! got=$(jq -r '.manifests[]? | select(.platform.os != "unknown") | "\(.platform.os)/\(.platform.architecture)"' "$raw" 2>/dev/null | sort | tr '\n' ' '); then + echo "::error::unparseable manifest for ${ref}"; fail=1; continue + fi + if [ "$got" != "$want" ]; then + echo "::error::platform mismatch for ${ref}: have [${got}] want [${want}]"; fail=1 + fi + echo "ok ${ref} ${d} [${got}]" + if [ -z "$ref_digest" ]; then ref_digest="$d"; ref_name="$ref" + elif [ "$d" != "$ref_digest" ]; then + echo "::error::digest mismatch: ${ref}=${d} vs ${ref_name}=${ref_digest}"; fail=1 + fi + done +done +[ "$fail" = 0 ] || { echo "::error::tag verification FAILED"; exit 1; } +echo "verified: ${tags[*]} in ${#repos[@]} registries, digest ${ref_digest}, platforms ${platforms}"