diff --git a/docs/pkg-cache.md b/docs/pkg-cache.md index b32b772..0b2a368 100644 --- a/docs/pkg-cache.md +++ b/docs/pkg-cache.md @@ -21,6 +21,10 @@ On that day a build without the cache spent **1031 s** in its first dnf step. Th | `https://192.168.1.40/ubuntu/` | `https://archive.ubuntu.com/ubuntu`, `https://us.archive.ubuntu.com/ubuntu` | | `https://192.168.1.40/ubuntu-security/` | `https://security.ubuntu.com/ubuntu`, `https://archive.ubuntu.com/ubuntu` | | `https://192.168.1.40/litespeed/` | `https://rpms.litespeedtech.com`, then `http://rpms.litespeedtech.com` | +| `https://192.168.1.40/nodesource/` | `https://rpm.nodesource.com` (origin only; RPMs signed, repomd not) — added 2026-10-22 | +| `https://192.168.1.40/debian/` | `https://deb.debian.org/debian` (CDN origin; princeton was 6 h behind on trixie-updates) — added 2026-10-22 | +| `https://192.168.1.40/debian-security/` | `https://security.debian.org/debian-security`, `https://deb.debian.org/debian-security` — added 2026-10-22 | +| `https://192.168.1.40/goproxy/` | `https://proxy.golang.org` (Go module proxy; `/sumdb/` is 404 there, so `go` checks sum.golang.org directly) — added 2026-10-22 | | `192.168.1.40:5000` (TLS, pkg-cache CA) | Docker Hub pull-through cache (`registry-1.docker.io`) | Plain `http://192.168.1.40/` answers **403** to LAN clients. @@ -32,6 +36,7 @@ Cache rules: - apt `dists/`, except `by-hash/` - `*-latest*.rpm` and `remi-release-N.rpm` - `RPM-GPG-KEY*` + - Go proxy `/@v/list` and `/@latest` - directory listings - **Packages** (`*.rpm`, `*.deb`, hash-named repodata, `by-hash/`) are cached for 30 days and evicted after 60 days unused. Total size is capped at 110 GB, with 15 GB kept free. - On a connect error, timeout or 5xx, the request falls through to the next tier. Only the **last** tier may answer from a stale copy, so the order is primary, then fallback, then stale. @@ -65,6 +70,18 @@ docker buildx build --build-arg PKG_CACHE=https://192.168.1.40 --build-arg PHPVE - Requires `rpm -K` to report `digests signatures OK`. - Installs with `--setopt=localpkg_gpgcheck=1`. +**`nodesource MAJOR`** (added 2026-10-22) replaces `curl https://rpm.nodesource.com/setup_N.x | bash`: +- writes `/etc/yum.repos.d/nodesource-nodejs.repo` with the ORIGIN baseurl, `gpgcheck=1` and `gpgkey=file:///usr/local/share/pkg-cache/RPM-GPG-KEY-nodesource` (key committed in `keys/pkg-cache/`); +- with `PKG_CACHE` it is then rewritten like the other repos; `restore` puts the origin version back, so the shipped image keeps a normal NodeSource repo. + +**`go-env`** (added 2026-10-22): `eval "$(pkg-cache-repos.sh go-env)"` inside the RUN that runs `go mod download`/`go build`. Prints `GOPROXY=/goproxy|https://proxy.golang.org|direct` (`|` = fall through on ANY error) and `SSL_CERT_FILE=/ca-bundle.pem`; nothing without `PKG_CACHE`. Modules stay verified by `go.sum` + sum.golang.org. End the RUN with `restore` (removes the bundle). + +**EL9**: `release-install epel|remi` picks the EL major from `rpm -E %rhel` (9 or 10); `RPM-GPG-KEY-EPEL-9` is committed. + +**Debian** (`apt` mode): `deb.debian.org/debian`, `deb.debian.org/debian-security` and `security.debian.org` (classic and deb822 `.sources`) are rewritten to `/debian/` and `/debian-security/`. + +**Not proxied, on purpose:** PyPI and the npm registry. Our pip/npm installs are not hash-locked (`pip install -r` without `--require-hashes`, `npm install -g pm2@latest`), so the client would trust whatever the cache served — a compromised cache would mean code execution in shipped images. Proxy them only after the installs are hash-pinned. GitHub-release downloads are likewise left direct. + **`restore`**: - Puts rewritten files back. - Deletes files the script created. @@ -75,9 +92,31 @@ Every RUN that rewrites ends with `restore`, so no shipped layer carries the LAN `PKG_CACHE` must start with `https://`. `http://` is refused. -`Dockerfile.litespeed` and `Dockerfile.lsphp` are **not wired yet**. Two things are needed first: -- the multi-stage `lsphp-apt-upgrade.sh` flow needs a `PKG_CACHE` ARG in each stage; -- its madison check must accept the cache URL. It currently requires `*rpms.litespeedtech.com*`, and via the cache madison shows `https://192.168.1.40/litespeed/debian`. +`Dockerfile.litespeed` and `Dockerfile.lsphp` are wired since 2026-10-22 (normal apt path; the php.net EMERGENCY stage `lsphp-src` stays on upstream). `lsphp-apt-upgrade.sh`'s madison gate accepts `/litespeed/` only when `PKG_CACHE` is set. + +## Repositories wired (2026-10-22) + +The script and `keys/pkg-cache/` are **canonical here**; every other repo carries byte-identical copies (`sha256sum scripts/pkg-cache-repos.sh` to compare). Change it here first, then sync. Same for `.gitea/scripts/resolve-base-image.sh`. + +| Repo | Image(s) | Bases | Through the cache | Base digest-pinned | +|---|---|---|---|---| +| cloud-apache-container | cac, cac-fpm, shared-httpd (almalinux/10-base); shared-ols, cac-litespeed, cac-lsphp (litespeedtech/openlitespeed) | Docker Hub | AlmaLinux/EPEL/Remi (rpm); Ubuntu + LiteSpeed (apt) | yes, all 6 jobs | +| cloud-node-container | cnoc:node18..26 | almalinux/10-base | AlmaLinux/EPEL + NodeSource (`nodesource N`) | yes | +| cloud-memcached-container | cmc | almalinux/9-base | AlmaLinux 9 + EPEL 9 | yes | +| cloud-haproxy-container | chc | almalinux/9-base | AlmaLinux 9 + EPEL 9 (inotify-tools GitHub tarball direct) | yes | +| whp-default-container | whp-default | httpd:alpine | nothing to install (mirror only) | yes | +| versitygw-container | versitygw | ubuntu:latest | Ubuntu (versitygw .deb from GitHub direct) | yes | +| haproxy-manager-base | haproxy-manager-base, coraza-spoa | repo.anhonesthost.net python:3.12-slim / golang:1.25 (not Docker Hub: no mirror); gcr.io distroless | Debian (apt) + Go modules (`go-env`); pip direct | n/a; `mirror-base-image.yaml` now copies Docker Hub images by the digest resolved directly from Docker Hub | +| cpanel-importer | cpanel-importer | almalinux:10-minimal | AlmaLinux/EPEL/Remi — **PR #1 open, not merged** (pending rebase onto the PHP 8.5 work) | in the PR | + +Not image builds (nothing to wire): kb-anhonesthost, anhonesthost-blog, dao-mail-signup, renovate-runner. ols-mail-client builds images in CI only for scanning/smoke tests (never pushed), its bases are already digest-pinned, and they install no OS packages. + +## Adding a new upstream + +1. On CT130: back up, then add the prefix to `UPSTREAMS` in `/usr/local/sbin/pkgcache-gen-nginx` (origin first; add a mirror tier only if you measured it is as fresh as the origin). If its mutable metadata isn't covered by `META_RE`, extend it there AND in `/usr/local/sbin/pkgcache-refresh`. Add the prefix to the `location = /` text in `/etc/nginx/pkgcache/server-common.conf`. +2. `pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx` (`pct exec` has no /usr/local/sbin in PATH: use full paths). Verify old and new prefixes with `curl --cacert keys/pkg-cache/ca.crt https://192.168.1.40//...` (look at `X-Cache-Status`/`X-Pkgcache-Tier`). CT130 has no egress filter, so no firewall change is needed for new origins. +3. Only proxy upstreams whose content the CLIENT verifies end to end (GPG-signed packages/metadata, Go sumdb, content digests). Commit any signing key under `keys/pkg-cache/` with its fingerprint and provenance in the README. +4. Teach `scripts/pkg-cache-repos.sh` the rewrite (keep the origin as a fallback where the client supports it), test through the cache from a runner-network host, then sync the copies. ## Trust model @@ -174,8 +213,7 @@ Hub, direct)"**: - Log lines to look for: `base-image: almalinux/10-base:latest -> almalinux/10-base@sha256:...` and `pkg-cache: repos=https://192.168.1.40 registry-mirror=192.168.1.40:5000`. -Applied to: Build-and-Push, Build-FPM-Images, Build-Shared-httpd, Build-Shared-OLS -(`litespeedtech/openlitespeed:${OLS_VERSION}-lsphp${PHPVER}`) here, and cloud-node-container. +Applied to every job here (Build-and-Push, FPM, Shared-httpd, Shared-OLS, LiteSpeed, LSPHP) and to every other repo with a Docker Hub base (see "Repositories wired"). The script is canonical here (`.gitea/scripts/resolve-base-image.sh`); other repos carry byte-identical copies.