cloud-hosting-platform/cpanel-importer
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ci: build through the internal package cache; pin almalinux base by digest #1

Merged
jknapp merged 11 commits from ci/pkg-cache into main 2026-10-22 14:52:20 +00:00
Conversation 0 Commits 11 Files Changed 20
+1721 -49
jknapp commented 2026-10-22 14:22:36 +00:00
Owner
Copy link
Copy source

Update 2026-10-22 (Josh: "go with option b, merge both"): rebased onto fix/include-mail-tree (the PHP 8.5 + mail-tree/extract-hardening work, 11 commits incl. fc1b7e2 "Move PHP from Remi 8.4 to 8.5", which is what the live :latest was built from). Merging this PR brings those commits to main unmodified. Conflict resolved in the Remi step: keeps php:remi-8.5, uses release-install epel remi.
Re-tested on the rebased head 517a091: local fallback build (no PKG_CACHE, BASE_IMAGE=almalinux@sha256:7127d3a9...) EXIT 0, epel/remi release RPMs digests signatures OK, php -v = 8.5.12, no cache reference / state dir in the image; bash tests/run-tests.sh ALL PASS.

Wires this image build to the internal package cache (CT130 pkg-cache), same pattern as cloud-apache-container / cloud-node-container.

What changes

  • Dockerfile
    • ARG BASE_IMAGE=almalinux:10-minimal + FROM ${BASE_IMAGE} (local builds without args unchanged).
    • ARG PKG_CACHE=; copies scripts/pkg-cache-repos.sh → /usr/local/sbin/, keys/pkg-cache/ → /usr/local/share/pkg-cache/.
    • First RUN: pkg-cache-repos.sh rpm → microdnf install dnf shadow-utils → pkg-cache-repos.sh release-install epel remi (GPG-verified release RPMs; replaces microdnf epel-release + dnf install https://rpms.remirepo.net/enterprise/remi-release-10.rpm) → pkg-cache-repos.sh rpm again (new EPEL/Remi repo files) → unchanged module/install steps → pkg-cache-repos.sh restore before dnf clean all. Without PKG_CACHE the script is a no-op for rpm/restore.
  • Workflow (job name, triggers, tags unchanged): digest resolve step (Docker Hub direct), cache probe (pkg-cache: repos=… registry-mirror=…, mirror only for a pinned base), two buildx variants (mirror + ca=). Both builds (smoke load: true, no-cache: true and push with cache-from) get identical BASE_IMAGE / PKG_CACHE build-args.
  • scripts/pkg-cache-repos.sh, .gitea/scripts/resolve-base-image.sh, keys/pkg-cache/: byte-identical copies of cloud-apache-container's (sha256 111f85c2…, 8a3af5c4…). .gitea is not copied into the image. Note COPY scripts/ /scripts/ also ships pkg-cache-repos.sh in /scripts/ (harmless; the CI bash -n loop covers it and passes).

Side effects (measured, local fallback vs unmodified main)

  • epel-release now comes from dl.fedoraproject.org (10-8.el10_2) instead of AlmaLinux extras (10-6.el10); it Recommends dnf-command(config-manager), so dnf-plugins-core + 6 python3/dbus deps are added (image 608 MB → 613 MB). Otherwise identical package set; PHP 8.4.27, ClamAV 1.4.6.

microdnf findings (CT990)

  • microdnf honours the rewritten repo files: with repo.almalinux.org/mirrors.almalinux.org blackholed (--add-host …:127.0.0.1), microdnf install dnf shadow-utils succeeded via https://192.168.1.40 using sslcacert=.
  • microdnf enforces repo_gpgcheck=1: with a wrong gpgkey it fails repomd.xml GPG signature verification error: Signing key not found.
  • microdnf on this base does not enforce per-package gpgcheck=1 (installed with a wrong key and no AlmaLinux key in the rpmdb) — same behaviour upstream without the cache (measured); dnf does enforce it. Through the cache, the two microdnf packages are still covered by the signed repomd.xml (repo_gpgcheck forced by the script) → checksummed metadata, and every later install uses dnf.

Tests (measured)

  • Unmodified main builds locally.
  • Local fallback (cache unreachable, no PKG_CACHE): BASE_IMAGE=almalinux@sha256:7127d3a9… → OK, FROM docker.io/library/almalinux@sha256:7127d3a958a0ba8bb673d7b4cd378c11052404ce1897940c32f1dd944448f7ab; epel-release verified (… digests signatures OK) from https://dl.fedoraproject.org/…, remi-release verified … from https://rpms.remirepo.net/…; CI smoke (echo ok), php -l on all scripts, bash -n on all scripts pass.
  • CT990 cache build (PKG_CACHE=https://192.168.1.40 + digest) → OK: AlmaLinux/EPEL/Remi baseurls via https://192.168.1.40/… with origin fallback, epel-release verified … from https://192.168.1.40/epel/epel-release-latest-10.noarch.rpm, remi-release verified … from https://192.168.1.40/remi/enterprise/remi-release-10.rpm, pkg-cache-repos: original repository files restored (no reference to 192.168.1.40 left).

Not changed

  • Build-time freshclam and SaneSecurity rsync downloads (not proxied).

🤖 Generated with Claude Code

**Update 2026-10-22 (Josh: "go with option b, merge both"):** rebased onto `fix/include-mail-tree` (the PHP 8.5 + mail-tree/extract-hardening work, 11 commits incl. fc1b7e2 "Move PHP from Remi 8.4 to 8.5", which is what the live `:latest` was built from). Merging this PR brings those commits to main unmodified. Conflict resolved in the Remi step: keeps `php:remi-8.5`, uses `release-install epel remi`. Re-tested on the rebased head 517a091: local fallback build (no PKG_CACHE, `BASE_IMAGE=almalinux@sha256:7127d3a9...`) EXIT 0, epel/remi release RPMs `digests signatures OK`, `php -v` = **8.5.12**, no cache reference / state dir in the image; `bash tests/run-tests.sh` ALL PASS. Wires this image build to the internal package cache (CT130 pkg-cache), same pattern as cloud-apache-container / cloud-node-container. ## What changes - **Dockerfile** - `ARG BASE_IMAGE=almalinux:10-minimal` + `FROM ${BASE_IMAGE}` (local builds without args unchanged). - `ARG PKG_CACHE=`; copies `scripts/pkg-cache-repos.sh` → `/usr/local/sbin/`, `keys/pkg-cache/` → `/usr/local/share/pkg-cache/`. - First RUN: `pkg-cache-repos.sh rpm` → `microdnf install dnf shadow-utils` → `pkg-cache-repos.sh release-install epel remi` (GPG-verified release RPMs; replaces microdnf `epel-release` + `dnf install https://rpms.remirepo.net/enterprise/remi-release-10.rpm`) → `pkg-cache-repos.sh rpm` again (new EPEL/Remi repo files) → unchanged module/install steps → `pkg-cache-repos.sh restore` before `dnf clean all`. Without `PKG_CACHE` the script is a no-op for rpm/restore. - **Workflow** (job name, triggers, tags unchanged): digest resolve step (Docker Hub direct), cache probe (`pkg-cache: repos=… registry-mirror=…`, mirror only for a pinned base), two buildx variants (mirror + `ca=`). **Both** builds (smoke `load: true, no-cache: true` and push with `cache-from`) get identical `BASE_IMAGE` / `PKG_CACHE` build-args. - `scripts/pkg-cache-repos.sh`, `.gitea/scripts/resolve-base-image.sh`, `keys/pkg-cache/`: byte-identical copies of cloud-apache-container's (sha256 111f85c2…, 8a3af5c4…). `.gitea` is not copied into the image. Note `COPY scripts/ /scripts/` also ships `pkg-cache-repos.sh` in `/scripts/` (harmless; the CI `bash -n` loop covers it and passes). ## Side effects (measured, local fallback vs unmodified main) - `epel-release` now comes from dl.fedoraproject.org (10-8.el10_2) instead of AlmaLinux extras (10-6.el10); it Recommends `dnf-command(config-manager)`, so `dnf-plugins-core` + 6 python3/dbus deps are added (image 608 MB → 613 MB). Otherwise identical package set; PHP 8.4.27, ClamAV 1.4.6. ## microdnf findings (CT990) - microdnf honours the rewritten repo files: with `repo.almalinux.org`/`mirrors.almalinux.org` blackholed (`--add-host …:127.0.0.1`), `microdnf install dnf shadow-utils` succeeded via `https://192.168.1.40` using `sslcacert=`. - microdnf **enforces `repo_gpgcheck=1`**: with a wrong `gpgkey` it fails `repomd.xml GPG signature verification error: Signing key not found`. - microdnf on this base does **not** enforce per-package `gpgcheck=1` (installed with a wrong key and no AlmaLinux key in the rpmdb) — **same behaviour upstream without the cache** (measured); `dnf` does enforce it. Through the cache, the two microdnf packages are still covered by the signed repomd.xml (repo_gpgcheck forced by the script) → checksummed metadata, and every later install uses `dnf`. ## Tests (measured) - Unmodified `main` builds locally. - Local fallback (cache unreachable, no PKG_CACHE): `BASE_IMAGE=almalinux@sha256:7127d3a9…` → OK, `FROM docker.io/library/almalinux@sha256:7127d3a958a0ba8bb673d7b4cd378c11052404ce1897940c32f1dd944448f7ab`; `epel-release verified (… digests signatures OK) from https://dl.fedoraproject.org/…`, `remi-release verified … from https://rpms.remirepo.net/…`; CI smoke (`echo ok`), `php -l` on all scripts, `bash -n` on all scripts pass. - CT990 cache build (`PKG_CACHE=https://192.168.1.40` + digest) → OK: AlmaLinux/EPEL/Remi baseurls via `https://192.168.1.40/…` with origin fallback, `epel-release verified … from https://192.168.1.40/epel/epel-release-latest-10.noarch.rpm`, `remi-release verified … from https://192.168.1.40/remi/enterprise/remi-release-10.rpm`, `pkg-cache-repos: original repository files restored (no reference to 192.168.1.40 left)`. ## Not changed - Build-time `freshclam` and SaneSecurity rsync downloads (not proxied). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
jknapp added 11 commits 2026-10-22 14:52:06 +00:00
extract: opt-in mail tree (IMPORT_INCLUDE_MAIL=true) + mail_included marker 8217397154 
WHP's cPanel mailbox import needs cpmove-*/homedir/mail, which extract.sh
always excluded: in sanitization-container mode every mailbox was queued
with no Maildir and "verified" empty (B5, whp02 canary 2026-10-20).

- IMPORT_INCLUDE_MAIL=true (exactly "true"; default off) keeps the mail
  tree in the extract. Default behaviour is unchanged. The DANGEROUS
  symlink strips still apply inside the mail tree (scan-symlinks.php lists
  every archive entry).
- scan-files.php --skip-mail (passed by entrypoint.sh with the opt-in):
  clamscan --exclude-dir for the mail tree, the file count skips it, and a
  hit inside it is never quarantined. Logged with the reason: mail is
  copied verbatim into the customer's own mailbox and verified against the
  source, and multi-GB maildirs would dominate the run.
- report.json gains "mail_included" (true only with the opt-in) so the
  panel can tell this image from an older one, which never writes the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests: mail-tree fixture + host-side run-tests.sh 9232114af4 
cpmove-mail.tar.gz adds two Maildirs (one empty) and an ALFA-style
symlink inside the mail tree. tests/run-tests.sh runs extract.sh and
scan-files.php from the checkout with a stub clamscan: mail skipped by
default, extracted with IMPORT_INCLUDE_MAIL=true, the in-tree symlink
still stripped, --skip-mail excludes/never quarantines the mail tree.
12 of 23 checks fail on the previous scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extract: strip every symlink and boundary-crossing hard link in the mail tree 4788ca4957 
Security review of 8217397: with IMPORT_INCLUDE_MAIL=true only
DANGEROUS-classified (absolute system-path) symlinks were stripped, so a
relative link (../../../../etc/shadow, ../other-domain/box) or a hard link
into/out of the mail tree survived into the extract.

- From the archive's member list (python tarfile, exact names/types):
  exclude EVERY symlink under cpmove-*/homedir/mail, and every hard link
  that crosses the mail-tree boundary (either direction), spans two
  accounts' mail trees, or has ".." in its target. Recorded as
  stripped_mail_link actions (deduped against the DANGEROUS strips).
- Hard links with name AND target inside the same mail tree are KEPT:
  Dovecot copies messages between folders as hard links
  (maildir_copy_with_hardlinks, default yes), so they are real mail.
- Belt: after extract, any symlink left under the mail tree is removed and
  logged.
- Fixtures: relative file/folder symlinks, an in-tree Dovecot-style hard
  link, hard links crossing in and out (cpmove-mail-xlink.tar.gz). 6 new
  checks fail on 8217397's extract.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extract: refuse parser-differential archives; post-extract link check is the authority 52b1e656c3 
Security review of 4788ca4 (inline python link classifier):
1. Names like cpmove-x/./homedir/mail/..., homedir//mail, homedir/../homedir
   or /cpmove-x/... were matched differently by the classifier than by tar,
   so a mail-tree symlink could escape the strip (measured: the dot, dslash
   and abs fixtures extracted with exit 0 on 4788ca4).
2. A newline in a member name split one --exclude-from pattern in two.

Moved the policy to scripts/lib/mail-tree-links.py:
- scan (pre-extract, an optimisation): REFUSES the archive (exit 3) on any
  control character in a member name or link target, or any non-canonical
  member name / hard-link target (absolute, '..', or normpath() differing
  after one leading './'). Symlink targets are not canonical-checked:
  ordinary cPanel links (../public_html) are legitimate outside the mail
  tree, and every symlink inside it is stripped anyway.
- verify (post-extract, the authority): lstat-walks each mail tree; exit 4
  on any symlink, or any regular file whose st_nlink exceeds its names
  inside that same mail tree (a link leaving it). In-tree Dovecot hard-link
  copies (maildir_copy_with_hardlinks) are kept; a blanket nlink>1 refusal
  would drop real mail.
- Fixtures: cpmove-bad-{dot,dslash,dotdot,abs,newline}; verify tests for
  symlinks and hard links into/out of/between mail trees surviving the
  pre-filter. 14 of the new checks fail on 4788ca4.

Only in IMPORT_INCLUDE_MAIL=true mode; the default path is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extract: verify always runs, failures remove the extract, walk fails closed d4a63fba0a 
Security review of 52b1e65:
1. (HIGH) tarfile's stream could enumerate differently from GNU tar. Now
   extract.sh captures tar's exit code (`|| TAR_RC=$?`), ALWAYS runs the
   post-extract verify on whatever tar wrote, and any failure (tar,
   verify, either pre-extract scan) removes $DEST before exiting non-zero.
   The classifier also refuses (exit 3) on any stream exception and when
   its member count differs from GNU tar's own escape-quoted listing.
   Measured: a lone zero block mid-archive stops BOTH readers at the same
   member (no -i is used), so this is a guard, not a known differential.
2. (MEDIUM) os.walk swallowed unreadable directories. verify now passes
   onerror (recorded => fail) and refuses any directory that is not
   readable+searchable. 52b1e65 exited 0 on the unreadable fixture.
3. (MEDIUM) a refused tree stayed in the bind-mounted output. fail_extract
   opens up 0000 dirs (find -P -type d, never follows links), then
   rm -rf -- "$DEST".

Fixtures: cpmove-bad-{unreadable,tarfail,truncated}; a fake tar that lists
one extra member proves the count check. 14 checks fail on 52b1e65.
Docker: unreadable / tarfail runs end with status failed, failed_stage
extract and no extract-work left in /host/sanitized/<id>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extract: mail-tree policy covers ANY top-level dir, not just cpmove-* (I-1) 4a43d26de7 
Review I-1: WHP's CpanelArchivePaths::selectTopDir() takes the first
non-hidden top dir of any name (cpmove-<u>, cPanel's user download
backup-<date>_<u>/, acct/ ...), but the link classifier, the post-extract
verify and scan-files' --skip-mail only matched cpmove-*. Measured by the
reviewer: an acct/ archive kept a ../../etc symlink in its Maildir, and
ClamAV would quarantine messages OUT of a backup-* Maildir (=> Verified
with mail missing).

- mail-tree-links.py: MAIL = ^([^/]+)/homedir/mail; verify walks every
  top-level dir, and fails on a top-level or homedir symlink instead of
  walking through it. Covering every top dir is a superset of whichever
  one WHP picks, so multi-top-dir archives need no special case.
- scan-files.php: --skip-mail patterns use [^/]+ for the top dir.
- M-7: entrypoint refuses IMPORT_INCLUDE_MAIL=true as uid 0 (the verify's
  os.access check is meaningless for root; WHP runs --user 999:999).
- M-2: the mail-tree refusals tell the operator the same backup may
  import with "Import without email".
- M-4: run-tests.sh checks the env/marker contract against a WHP checkout
  when WHP_REPO is set.
- Fixtures cpmove-acct / cpmove-backupdir (the mail fixture under acct/
  and backup-10.20.2026_10-00-00_testuser/). 13 checks fail on d4a63fb.
  Docker: backup-* Maildir EICAR not quarantined, public_html one is;
  --user 0 with the opt-in => status failed.

The default (no-opt-in) mail exclude keeps its historical cpmove-* scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scan-symlinks: relative targets escaping the archive or the homedir are DANGEROUS (N2-1) a226b496a2 
A relative symlink was skipped as safe (`if ($target[0] !== '/') continue`),
so `cpmove-u/sds -> ../../../../root/.my.cnf` survived extraction and WHP
parsed the host file as the subdomain list. Resolve relative targets
lexically from the link's own directory: reaching or climbing above the
archive root, or a link under <top>/homedir/ resolving outside that homedir,
is DANGEROUS (stripped by extract.sh). Same rule as WHP's
CpanelBackupImporter::relativeSymlinkEscape().

The clean fixture's about-shortcut (../public_html/about.txt resolved
outside homedir/) now points at public_html/about.txt as its comment meant.
New cpmove-relesc fixture + checks (5 fail on 4a43d26).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extract: post-extract realpath check of every symlink; classifier refuses link chains (N2-1 review) 72b8fa9d08 
The lexical relative-target check missed chains: `a -> b/../..` with
`b -> ../..`, or `l1 -> l2/../../..`, each resolve inside the archive
lexically but above the extract through the other link.

- scripts/lib/strip-escaping-links.py (new, the authority): after tar,
  lstat-walk the WHOLE extract (never following links), realpath(strict)
  every symlink, and remove + record (stripped_escaping_symlink) any that
  resolves outside the extract or to it, dangles or loops. Repeats to a
  fixed point; a walk error fails the extract. Own-account absolute
  /home[N]/<user>/ targets are kept (they cannot resolve in the sandbox and
  resolve only inside the customer's site container on WHP).
- scan-symlinks.php: two passes; a relative link whose resolution passes
  through another symlink entry of the archive is DANGEROUS.
- fixtures/tests: cpmove-chain (both shapes) and a hand-built tree for the
  post-extract pass (chains, /etc, dangling .bak sidecar, escaping .bak,
  loop, link to the top). 3 chain checks fail on a226b49.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extract: own-account links judged re-rooted, normalized, for the trusted username only (N2-1 review 2) 36d4949d34 
- strip-escaping-links.py: the /home[N]/<user>/ exemption ran on the RAW
  target (`/home/u/../../etc/shadow` was exempt) and on usernames taken
  from the archive's own top-dir names (an archive could name itself
  cpmove-root). Now only the username WHP passes (extract.sh <username> =
  IMPORT_USERNAME, validated) is trusted; the target must be normalized; and
  it is RE-ROOTED at the link's <top>/homedir in the extract and kept only
  if that resolves (realpath strict) inside it. No trusted name: nothing
  exempt.
- scan-symlinks.php: same trusted-name rule; a non-normalized absolute
  target is DANGEROUS outright (never prefix-compared).
- tests: /home/u/../../etc/shadow, /home/u/./x, re-rooted dangling,
  re-rooted through an escaping link, another user's /home, an
  archive-named user (cpmove-root) via extract.sh, no/invalid trusted name.
  8 checks fail on 72b8fa9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move PHP from Remi 8.4 to 8.5 (security update) fc1b7e2d3c 
The php:remi-8.4 stream has a known exploit. Enable php:remi-8.5
instead (still unpinned to a patch version). Package list is unchanged
and resolves on 8.5 (php-json is built in; json module present).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ci: build through the internal package cache; pin almalinux base by digest 517a091539 
- BASE_IMAGE: CI resolves almalinux:10-minimal to almalinux@sha256:<digest>
  directly at registry-1.docker.io; the pkg-cache registry mirror is only
  used for a digest-pinned base. Both builds (smoke + push) get the same
  BASE_IMAGE / PKG_CACHE build-args.
- PKG_CACHE: the package RUN rewrites the AlmaLinux repos (pkg-cache-repos.sh
  rpm) before microdnf, installs epel-release + remi-release via
  `release-install epel remi` (signature-checked against keys/pkg-cache/,
  replacing microdnf epel-release + `dnf install https://...remi-release`),
  rewrites the new repo files, and restores everything before the RUN ends.
  Without PKG_CACHE the build uses the upstream repositories as before.
- scripts/pkg-cache-repos.sh, .gitea/scripts/resolve-base-image.sh and
  keys/pkg-cache/ are byte-identical copies from cloud-apache-container.

freshclam / SaneSecurity rsync build-time downloads are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp force-pushed ci/pkg-cache from 931d63799b to 517a091539 2026-10-22 14:52:06 +00:00 Compare
jknapp changed title from [DO NOT MERGE - rebase onto PHP 8.5 first] ci: build through the internal package cache; pin almalinux base by digest to ci: build through the internal package cache; pin almalinux base by digest 2026-10-22 14:52:12 +00:00
jknapp merged commit 228eb94ad5 into main 2026-10-22 14:52:20 +00:00
jknapp deleted branch ci/pkg-cache 2026-10-22 14:52:21 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
hub-agent Bot jknapp renovate-bot (Renovate Bot) whmcs-sam whp-cicd-user whp-deploy
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cloud-hosting-platform/cpanel-importer#1