ci: arm64 policy - serialise legs, one retry, never drop existing arm64, ELF check #49

Merged
jknapp merged 2 commits from fix/arm64-policy into trunk 2026-11-20 05:13:13 +00:00
Owner

Stacked on #48 (arm64-attach); merge after it (this branch contains its commit and uses scripts/verify-image-arch.py).

Policy fixes for the normal build (cause of the 2026-11-19 arm64 loss)

  1. Concurrency: max-parallel: 2 on the cac and cac-fpm matrices. shared-httpd is a single job (no matrix). Trade-off: max-parallel: 1 is safest (matches "one leg alone succeeds") but a full 5-version family takes ~5x a leg (10-20 min each) before the security fix is fully published; 2 halves the load seen on 11-19. Josh can pick by editing one number. Whether Gitea's act_runner honours strategy.max-parallel could not be verified here (unknowable without a run); if it is ignored, this is a no-op and the retry/guard still protect.
  2. One retry of a failed arm64 leg (cac, cac-fpm, shared-httpd): Wait before arm64 retry (90 s) + Retry arm64 (push by digest) (id arm64_retry, same inputs, timeout, continue-on-error). Assemble takes the first attempt's digest, else the retry's; outcome is success if either succeeded.
  3. "Would drop arm64" guard in assemble-multiarch-index.sh --guard-existing-arm64: if the result would be amd64-only and the current tag has arm64 (via index-platforms.sh; missing tag ok, unreadable = fail closed) the step FAILS and publishes nothing; the error names the amd64 digest and remedies. New dispatch input allow_amd64_only (string, default false, passed via env) lets a security rebuild publish amd64-only anyway; restore arm64 afterwards with arm64-attach. Note: push-triggered runs have no input, so the guard is always on there. Trade-off vs "keep the old arm64 attached": rejected because it would pair a stale arm64 PHP with a newer amd64.
  4. ELF check before publish (--verify-bin, cac: httpd+php, cac-fpm: php-fpm+php, shared-httpd: httpd): amd64 must be x86-64 (else abort, nothing published); arm64 must be AArch64 (else treated as a failed arm64 leg, then the guard applies).

lsphp-source-emergency.yaml is unchanged (it already has its own guard).

Tests

Shell-Checks reproduced locally: bash -n, shellcheck -S warning on the full list (incl. new test), all scripts/tests/*.test.sh (new assemble-guard.test.sh: 25 checks), multiarch-workflow.test.py (40 tests, new retry/max-parallel/guard wiring tests). lsphp-abi-gate skip-fails (no pahole), as known. No workflow was dispatched.

🤖 Generated with Claude Code

https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6

Review follow-up

Job timeouts raised (130/80) so the retry fits; verify step bounded + retried + cleans up pulled images; digest only from a successful first attempt. Not changed: N1 (index-platforms 404 match), N2.

Stacked on #48 (arm64-attach); merge after it (this branch contains its commit and uses `scripts/verify-image-arch.py`). ## Policy fixes for the normal build (cause of the 2026-11-19 arm64 loss) 1. **Concurrency**: `max-parallel: 2` on the cac and cac-fpm matrices. shared-httpd is a single job (no matrix). Trade-off: `max-parallel: 1` is safest (matches "one leg alone succeeds") but a full 5-version family takes ~5x a leg (10-20 min each) before the security fix is fully published; 2 halves the load seen on 11-19. Josh can pick by editing one number. Whether Gitea's act_runner honours `strategy.max-parallel` could not be verified here (unknowable without a run); if it is ignored, this is a no-op and the retry/guard still protect. 2. **One retry** of a failed arm64 leg (cac, cac-fpm, shared-httpd): `Wait before arm64 retry` (90 s) + `Retry arm64 (push by digest)` (id `arm64_retry`, same inputs, timeout, continue-on-error). Assemble takes the first attempt's digest, else the retry's; outcome is success if either succeeded. 3. **"Would drop arm64" guard** in `assemble-multiarch-index.sh --guard-existing-arm64`: if the result would be amd64-only and the current tag has arm64 (via `index-platforms.sh`; missing tag ok, unreadable = fail closed) the step FAILS and publishes nothing; the error names the amd64 digest and remedies. New dispatch input `allow_amd64_only` (string, default false, passed via env) lets a security rebuild publish amd64-only anyway; restore arm64 afterwards with `arm64-attach`. Note: push-triggered runs have no input, so the guard is always on there. Trade-off vs "keep the old arm64 attached": rejected because it would pair a stale arm64 PHP with a newer amd64. 4. **ELF check before publish** (`--verify-bin`, cac: httpd+php, cac-fpm: php-fpm+php, shared-httpd: httpd): amd64 must be x86-64 (else abort, nothing published); arm64 must be AArch64 (else treated as a failed arm64 leg, then the guard applies). lsphp-source-emergency.yaml is unchanged (it already has its own guard). ## Tests Shell-Checks reproduced locally: bash -n, shellcheck -S warning on the full list (incl. new test), all `scripts/tests/*.test.sh` (new `assemble-guard.test.sh`: 25 checks), `multiarch-workflow.test.py` (40 tests, new retry/max-parallel/guard wiring tests). `lsphp-abi-gate` skip-fails (no pahole), as known. No workflow was dispatched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6 ### Review follow-up Job timeouts raised (130/80) so the retry fits; verify step bounded + retried + cleans up pulled images; digest only from a successful first attempt. Not changed: N1 (index-platforms 404 match), N2.
jknapp added 2 commits 2026-11-20 05:13:08 +00:00
- max-parallel: 2 on the cac / cac-fpm matrices (8 concurrent QEMU legs failed on 2026-11-19)
- ONE retry of a failed arm64 leg (90 s pause) in cac, cac-fpm, shared-httpd
- assemble-multiarch-index.sh: --guard-existing-arm64 (fail instead of replacing a multi-arch
  tag by amd64-only; dispatch input allow_amd64_only overrides) and --verify-bin (ELF machine
  type of amd64 and arm64 before any tag is written)
- tests: assemble-guard.test.sh, wiring tests; docs/arm64-publish-policy.md

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
- job timeouts 130 (cac, cac-fpm) / 80 (shared-httpd) so the retry fits; test pins the budget
- verify-image-arch.py: bounded docker calls, pull retried 3x, exit 4 'could not pull' (not an
  arch verdict), pulled image removed afterwards
- ARM64_DIGEST only taken from the first attempt when it succeeded
- comments/docs: max-parallel is per-job and unverified; guard also fires on skipped/cancelled

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
jknapp force-pushed fix/arm64-policy from 8644d8e38c to 344c5e8d0d 2026-11-20 05:13:08 +00:00 Compare
jknapp merged commit 2ad7249d9f into trunk 2026-11-20 05:13:13 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cloud-hosting-platform/cloud-apache-container#49