ci: build through the internal package cache; pin almalinux base by digest #1

Merged
jknapp merged 11 commits from ci/pkg-cache into main 2026-10-22 14:52:20 +00:00
11 Commits
Author SHA1 Message Date
shadowdaoandClaude Opus 5.5 517a091539 ci: build through the internal package cache; pin almalinux base by digest
- BASE_IMAGE: CI resolves almalinux:10-minimal to almalinux@sha256:<digest>
  directly at registry-1.docker.io; the pkg-cache registry mirror is only
  used for a digest-pinned base. Both builds (smoke + push) get the same
  BASE_IMAGE / PKG_CACHE build-args.
- PKG_CACHE: the package RUN rewrites the AlmaLinux repos (pkg-cache-repos.sh
  rpm) before microdnf, installs epel-release + remi-release via
  `release-install epel remi` (signature-checked against keys/pkg-cache/,
  replacing microdnf epel-release + `dnf install https://...remi-release`),
  rewrites the new repo files, and restores everything before the RUN ends.
  Without PKG_CACHE the build uses the upstream repositories as before.
- scripts/pkg-cache-repos.sh, .gitea/scripts/resolve-base-image.sh and
  keys/pkg-cache/ are byte-identical copies from cloud-apache-container.

freshclam / SaneSecurity rsync build-time downloads are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 07:47:03 -07:00
shadowdaoandClaude Opus 5.5 fc1b7e2d3c Move PHP from Remi 8.4 to 8.5 (security update)
The php:remi-8.4 stream has a known exploit. Enable php:remi-8.5
instead (still unpinned to a patch version). Package list is unchanged
and resolves on 8.5 (php-json is built in; json module present).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 05:59:33 -07:00
shadowdaoandClaude Opus 5.5 36d4949d34 extract: own-account links judged re-rooted, normalized, for the trusted username only (N2-1 review 2)
- strip-escaping-links.py: the /home[N]/<user>/ exemption ran on the RAW
  target (`/home/u/../../etc/shadow` was exempt) and on usernames taken
  from the archive's own top-dir names (an archive could name itself
  cpmove-root). Now only the username WHP passes (extract.sh <username> =
  IMPORT_USERNAME, validated) is trusted; the target must be normalized; and
  it is RE-ROOTED at the link's <top>/homedir in the extract and kept only
  if that resolves (realpath strict) inside it. No trusted name: nothing
  exempt.
- scan-symlinks.php: same trusted-name rule; a non-normalized absolute
  target is DANGEROUS outright (never prefix-compared).
- tests: /home/u/../../etc/shadow, /home/u/./x, re-rooted dangling,
  re-rooted through an escaping link, another user's /home, an
  archive-named user (cpmove-root) via extract.sh, no/invalid trusted name.
  8 checks fail on 72b8fa9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 10:51:17 -07:00
shadowdaoandClaude Opus 5.5 72b8fa9d08 extract: post-extract realpath check of every symlink; classifier refuses link chains (N2-1 review)
The lexical relative-target check missed chains: `a -> b/../..` with
`b -> ../..`, or `l1 -> l2/../../..`, each resolve inside the archive
lexically but above the extract through the other link.

- scripts/lib/strip-escaping-links.py (new, the authority): after tar,
  lstat-walk the WHOLE extract (never following links), realpath(strict)
  every symlink, and remove + record (stripped_escaping_symlink) any that
  resolves outside the extract or to it, dangles or loops. Repeats to a
  fixed point; a walk error fails the extract. Own-account absolute
  /home[N]/<user>/ targets are kept (they cannot resolve in the sandbox and
  resolve only inside the customer's site container on WHP).
- scan-symlinks.php: two passes; a relative link whose resolution passes
  through another symlink entry of the archive is DANGEROUS.
- fixtures/tests: cpmove-chain (both shapes) and a hand-built tree for the
  post-extract pass (chains, /etc, dangling .bak sidecar, escaping .bak,
  loop, link to the top). 3 chain checks fail on a226b49.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 10:47:04 -07:00
shadowdaoandClaude Opus 5.5 a226b496a2 scan-symlinks: relative targets escaping the archive or the homedir are DANGEROUS (N2-1)
A relative symlink was skipped as safe (`if ($target[0] !== '/') continue`),
so `cpmove-u/sds -> ../../../../root/.my.cnf` survived extraction and WHP
parsed the host file as the subdomain list. Resolve relative targets
lexically from the link's own directory: reaching or climbing above the
archive root, or a link under <top>/homedir/ resolving outside that homedir,
is DANGEROUS (stripped by extract.sh). Same rule as WHP's
CpanelBackupImporter::relativeSymlinkEscape().

The clean fixture's about-shortcut (../public_html/about.txt resolved
outside homedir/) now points at public_html/about.txt as its comment meant.
New cpmove-relesc fixture + checks (5 fail on 4a43d26).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 10:42:15 -07:00
shadowdaoandClaude Opus 5.5 4a43d26de7 extract: mail-tree policy covers ANY top-level dir, not just cpmove-* (I-1)
Review I-1: WHP's CpanelArchivePaths::selectTopDir() takes the first
non-hidden top dir of any name (cpmove-<u>, cPanel's user download
backup-<date>_<u>/, acct/ ...), but the link classifier, the post-extract
verify and scan-files' --skip-mail only matched cpmove-*. Measured by the
reviewer: an acct/ archive kept a ../../etc symlink in its Maildir, and
ClamAV would quarantine messages OUT of a backup-* Maildir (=> Verified
with mail missing).

- mail-tree-links.py: MAIL = ^([^/]+)/homedir/mail; verify walks every
  top-level dir, and fails on a top-level or homedir symlink instead of
  walking through it. Covering every top dir is a superset of whichever
  one WHP picks, so multi-top-dir archives need no special case.
- scan-files.php: --skip-mail patterns use [^/]+ for the top dir.
- M-7: entrypoint refuses IMPORT_INCLUDE_MAIL=true as uid 0 (the verify's
  os.access check is meaningless for root; WHP runs --user 999:999).
- M-2: the mail-tree refusals tell the operator the same backup may
  import with "Import without email".
- M-4: run-tests.sh checks the env/marker contract against a WHP checkout
  when WHP_REPO is set.
- Fixtures cpmove-acct / cpmove-backupdir (the mail fixture under acct/
  and backup-10.20.2026_10-00-00_testuser/). 13 checks fail on d4a63fb.
  Docker: backup-* Maildir EICAR not quarantined, public_html one is;
  --user 0 with the opt-in => status failed.

The default (no-opt-in) mail exclude keeps its historical cpmove-* scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:59:41 -07:00
shadowdaoandClaude Opus 5.5 d4a63fba0a extract: verify always runs, failures remove the extract, walk fails closed
Security review of 52b1e65:
1. (HIGH) tarfile's stream could enumerate differently from GNU tar. Now
   extract.sh captures tar's exit code (`|| TAR_RC=$?`), ALWAYS runs the
   post-extract verify on whatever tar wrote, and any failure (tar,
   verify, either pre-extract scan) removes $DEST before exiting non-zero.
   The classifier also refuses (exit 3) on any stream exception and when
   its member count differs from GNU tar's own escape-quoted listing.
   Measured: a lone zero block mid-archive stops BOTH readers at the same
   member (no -i is used), so this is a guard, not a known differential.
2. (MEDIUM) os.walk swallowed unreadable directories. verify now passes
   onerror (recorded => fail) and refuses any directory that is not
   readable+searchable. 52b1e65 exited 0 on the unreadable fixture.
3. (MEDIUM) a refused tree stayed in the bind-mounted output. fail_extract
   opens up 0000 dirs (find -P -type d, never follows links), then
   rm -rf -- "$DEST".

Fixtures: cpmove-bad-{unreadable,tarfail,truncated}; a fake tar that lists
one extra member proves the count check. 14 checks fail on 52b1e65.
Docker: unreadable / tarfail runs end with status failed, failed_stage
extract and no extract-work left in /host/sanitized/<id>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:18:34 -07:00
shadowdaoandClaude Opus 5.5 52b1e656c3 extract: refuse parser-differential archives; post-extract link check is the authority
Security review of 4788ca4 (inline python link classifier):
1. Names like cpmove-x/./homedir/mail/..., homedir//mail, homedir/../homedir
   or /cpmove-x/... were matched differently by the classifier than by tar,
   so a mail-tree symlink could escape the strip (measured: the dot, dslash
   and abs fixtures extracted with exit 0 on 4788ca4).
2. A newline in a member name split one --exclude-from pattern in two.

Moved the policy to scripts/lib/mail-tree-links.py:
- scan (pre-extract, an optimisation): REFUSES the archive (exit 3) on any
  control character in a member name or link target, or any non-canonical
  member name / hard-link target (absolute, '..', or normpath() differing
  after one leading './'). Symlink targets are not canonical-checked:
  ordinary cPanel links (../public_html) are legitimate outside the mail
  tree, and every symlink inside it is stripped anyway.
- verify (post-extract, the authority): lstat-walks each mail tree; exit 4
  on any symlink, or any regular file whose st_nlink exceeds its names
  inside that same mail tree (a link leaving it). In-tree Dovecot hard-link
  copies (maildir_copy_with_hardlinks) are kept; a blanket nlink>1 refusal
  would drop real mail.
- Fixtures: cpmove-bad-{dot,dslash,dotdot,abs,newline}; verify tests for
  symlinks and hard links into/out of/between mail trees surviving the
  pre-filter. 14 of the new checks fail on 4788ca4.

Only in IMPORT_INCLUDE_MAIL=true mode; the default path is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:13:26 -07:00
shadowdaoandClaude Opus 5.5 4788ca4957 extract: strip every symlink and boundary-crossing hard link in the mail tree
Security review of 8217397: with IMPORT_INCLUDE_MAIL=true only
DANGEROUS-classified (absolute system-path) symlinks were stripped, so a
relative link (../../../../etc/shadow, ../other-domain/box) or a hard link
into/out of the mail tree survived into the extract.

- From the archive's member list (python tarfile, exact names/types):
  exclude EVERY symlink under cpmove-*/homedir/mail, and every hard link
  that crosses the mail-tree boundary (either direction), spans two
  accounts' mail trees, or has ".." in its target. Recorded as
  stripped_mail_link actions (deduped against the DANGEROUS strips).
- Hard links with name AND target inside the same mail tree are KEPT:
  Dovecot copies messages between folders as hard links
  (maildir_copy_with_hardlinks, default yes), so they are real mail.
- Belt: after extract, any symlink left under the mail tree is removed and
  logged.
- Fixtures: relative file/folder symlinks, an in-tree Dovecot-style hard
  link, hard links crossing in and out (cpmove-mail-xlink.tar.gz). 6 new
  checks fail on 8217397's extract.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:09:10 -07:00
shadowdaoandClaude Opus 5.5 9232114af4 tests: mail-tree fixture + host-side run-tests.sh
cpmove-mail.tar.gz adds two Maildirs (one empty) and an ALFA-style
symlink inside the mail tree. tests/run-tests.sh runs extract.sh and
scan-files.php from the checkout with a stub clamscan: mail skipped by
default, extracted with IMPORT_INCLUDE_MAIL=true, the in-tree symlink
still stripped, --skip-mail excludes/never quarantines the mail tree.
12 of 23 checks fail on the previous scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:04:44 -07:00
shadowdaoandClaude Opus 5.5 8217397154 extract: opt-in mail tree (IMPORT_INCLUDE_MAIL=true) + mail_included marker
WHP's cPanel mailbox import needs cpmove-*/homedir/mail, which extract.sh
always excluded: in sanitization-container mode every mailbox was queued
with no Maildir and "verified" empty (B5, whp02 canary 2026-10-20).

- IMPORT_INCLUDE_MAIL=true (exactly "true"; default off) keeps the mail
  tree in the extract. Default behaviour is unchanged. The DANGEROUS
  symlink strips still apply inside the mail tree (scan-symlinks.php lists
  every archive entry).
- scan-files.php --skip-mail (passed by entrypoint.sh with the opt-in):
  clamscan --exclude-dir for the mail tree, the file count skips it, and a
  hit inside it is never quarantined. Logged with the reason: mail is
  copied verbatim into the customer's own mailbox and verified against the
  source, and multi-GB maildirs would dominate the run.
- report.json gains "mail_included" (true only with the opt-in) so the
  panel can tell this image from an older one, which never writes the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:04:44 -07:00