Without `PKG_CACHE`, builds use the upstream repositories. The one change from before is that the EPEL and Remi release RPMs are now **GPG-verified** against `keys/pkg-cache/` before they are installed.
Why it exists: during the 2026-10-21 PHP security release, builds failed because:
- Remi mirrors lagged hours behind origin;
- Remi's origin reset or timed out HTTPS under load;
- an EPEL mirror had a broken certificate.
On that day a build without the cache spent **1031 s** in its first dnf step. Through the cache the same step took **40 s cold and 21 s warm**.
- **Packages** (`*.rpm`, `*.deb`, hash-named repodata, `by-hash/`) are cached for 30 days and evicted after 60 days unused. Total size is capped at 110 GB, with 15 GB kept free.
- On a connect error, timeout or 5xx, the request falls through to the next tier. Only the **last** tier may answer from a stale copy, so the order is primary, then fallback, then stale.
-`proxy_cache_lock` is on.
- Responses carry `X-Cache-Status` and `X-Pkgcache-Tier`.
`scripts/pkg-cache-repos.sh` (installed as `/usr/local/sbin/`) and `keys/pkg-cache/` (installed as `/usr/local/share/pkg-cache/`) are wired into `Dockerfile`, `Dockerfile.fpm`, `Dockerfile.shared-httpd` and `Dockerfile.shared-ols`. The script has four modes.
**`rpm`** handles AlmaLinux, EPEL and Remi:
- Each `baseurl` becomes `<cache>/<prefix>/... <origin>/...`. dnf fails over to the origin if the cache dies.
- That section's mirrorlist/metalink is commented out.
- The section gets `sslcacert=` pointing at a bundle of the system CAs plus the pkg-cache CA. The bundle lives in the state dir and is deleted by `restore`.
- It also gets `gpgcheck=1`.
- AlmaLinux and Remi also get `repo_gpgcheck=1`, because both sign `repomd.xml` (measured 2026-10-21: `repomd.xml.asc` returns 200). EPEL publishes no repomd signature (404), so its metadata integrity rests on HTTPS end to end.
**`apt`** handles Ubuntu archive/security and LiteSpeed:
- Their URIs are rewritten to the cache.
-`/etc/apt/apt.conf.d/99pkg-cache` sets `Acquire::https::192.168.1.40::CaInfo`, so the CA is trusted for the cache host only.
- apt verifies the signed `InRelease`/`Release.gpg` as always. It has no failover, so CI must probe the cache first.
**`release-install epel [remi]`**:
- Downloads the release RPM over HTTPS, from the cache or from the origin.
- Checks the package name.
- Imports the key committed in `keys/pkg-cache/`.
- Requires `rpm -K` to report `digests signatures OK`.
- writes `/etc/yum.repos.d/nodesource-nodejs.repo` with the ORIGIN baseurl, `gpgcheck=1` and `gpgkey=file:///usr/local/share/pkg-cache/RPM-GPG-KEY-nodesource` (key committed in `keys/pkg-cache/`);
- with `PKG_CACHE` it is then rewritten like the other repos; `restore` puts the origin version back, so the shipped image keeps a normal NodeSource repo.
**`go-env`** (added 2026-10-22): `eval "$(pkg-cache-repos.sh go-env)"` inside the RUN that runs `go mod download`/`go build`. Prints `GOPROXY=<cache>/goproxy|https://proxy.golang.org|direct` (`|` = fall through on ANY error) and `SSL_CERT_FILE=<state>/ca-bundle.pem`; nothing without `PKG_CACHE`. Modules stay verified by `go.sum` + sum.golang.org. End the RUN with `restore` (removes the bundle).
**EL9**: `release-install epel|remi` picks the EL major from `rpm -E %rhel` (9 or 10); `RPM-GPG-KEY-EPEL-9` is committed.
**Debian** (`apt` mode): `deb.debian.org/debian`, `deb.debian.org/debian-security` and `security.debian.org` (classic and deb822 `.sources`) are rewritten to `<cache>/debian/` and `<cache>/debian-security/`.
**Not proxied, on purpose:** PyPI and the npm registry. Our pip/npm installs are not hash-locked (`pip install -r` without `--require-hashes`, `npm install -g pm2@latest`), so the client would trust whatever the cache served — a compromised cache would mean code execution in shipped images. Proxy them only after the installs are hash-pinned. GitHub-release downloads are likewise left direct.
Every RUN that rewrites ends with `restore`, so no shipped layer carries the LAN address. This was verified on the built images: 0 references, no state dir, no apt drop-in.
`Dockerfile.litespeed` and `Dockerfile.lsphp` are wired since 2026-10-22 (normal apt path; the php.net EMERGENCY stage `lsphp-src` stays on upstream). `lsphp-apt-upgrade.sh`'s madison gate accepts `<PKG_CACHE>/litespeed/` only when `PKG_CACHE` is set.
## Repositories wired (2026-10-22)
The script and `keys/pkg-cache/` are **canonical here**; every other repo carries byte-identical copies (`sha256sum scripts/pkg-cache-repos.sh` to compare). Change it here first, then sync. Same for `.gitea/scripts/resolve-base-image.sh`.
| Repo | Image(s) | Bases | Through the cache | Base digest-pinned |
| haproxy-manager-base | haproxy-manager-base, coraza-spoa | repo.anhonesthost.net python:3.12-slim / golang:1.25 (not Docker Hub: no mirror); gcr.io distroless | Debian (apt) + Go modules (`go-env`); pip direct | n/a; `mirror-base-image.yaml` now copies Docker Hub images by the digest resolved directly from Docker Hub |
| cpanel-importer | cpanel-importer | almalinux:10-minimal | AlmaLinux/EPEL/Remi — **PR #1 open, not merged** (pending rebase onto the PHP 8.5 work) | in the PR |
Not image builds (nothing to wire): kb-anhonesthost, anhonesthost-blog, dao-mail-signup, renovate-runner. ols-mail-client builds images in CI only for scanning/smoke tests (never pushed), its bases are already digest-pinned, and they install no OS packages.
## Adding a new upstream
1. On CT130: back up, then add the prefix to `UPSTREAMS` in `/usr/local/sbin/pkgcache-gen-nginx` (origin first; add a mirror tier only if you measured it is as fresh as the origin). If its mutable metadata isn't covered by `META_RE`, extend it there AND in `/usr/local/sbin/pkgcache-refresh`. Add the prefix to the `location = /` text in `/etc/nginx/pkgcache/server-common.conf`.
2.`pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx` (`pct exec` has no /usr/local/sbin in PATH: use full paths). Verify old and new prefixes with `curl --cacert keys/pkg-cache/ca.crt https://192.168.1.40/<prefix>/...` (look at `X-Cache-Status`/`X-Pkgcache-Tier`). CT130 has no egress filter, so no firewall change is needed for new origins.
3. Only proxy upstreams whose content the CLIENT verifies end to end (GPG-signed packages/metadata, Go sumdb, content digests). Commit any signing key under `keys/pkg-cache/` with its fingerprint and provenance in the README.
4. Teach `scripts/pkg-cache-repos.sh` the rewrite (keep the origin as a fallback where the client supports it), test through the cache from a runner-network host, then sync the copies.
`keys/pkg-cache/ca.crt` is the *pkg-cache internal CA*:
- EC P-256, valid to 2036.
- **Name-constrained** to IP `192.168.1.40` and DNS `pkg-cache.internal.cloud-hosting.io` / `pkg-cache.geekhome.local`. Even its private key, which exists only on CT130 at `/etc/pkgcache-ca/ca.key`, cannot mint a certificate that clients would accept for any other host.
- It is trusted only by the rewritten repos, the apt per-host option and the BuildKit/dockerd registry config. It is never added to a system trust store.
The server certificate `/etc/pkgcache-ca/server.crt`**expires 2029-10-20**. `pkgcache-stats` prints the expiry. Renew it with the same `openssl x509 -req ... -extfile srv.cnf` steps in `/etc/pkgcache-ca/`.
Remaining risks:
- **Replay of old but validly signed metadata.** Signatures stop tampering, not freshness. The cache's own plain-http tiers for Remi and LiteSpeed are the leg where an internet MITM could replay. The php.net gates (`EXPECT_PHP_MIN`, `EXPECT_LSPHP_PKG`/`EXPECT_PHP_VERSION`) fail an image build that gets old PHP. **Josh to decide** whether to drop those http tiers: you gain replay resistance and lose availability when Remi's HTTPS is down, as it was all day on 2026-10-21.
- **Proposed, not implemented:** the same kind of minimum-version gate for other security-relevant packages:
- CI resolves the newest `openssl`/`openssl-libs` and `httpd` from AlmaLinux BaseOS/AppStream, and `ImageMagick` from EPEL, **directly from the origin over HTTPS** (e.g. `dnf repoquery --repofrompath=o,https://repo.almalinux.org/almalinux/10/BaseOS/x86_64/os/ --repo=o --latest-limit=1 --qf '%{evr}' openssl-libs`).
- It passes the results as `EXPECT_OPENSSL_MIN` / `EXPECT_HTTPD_MIN` / `EXPECT_IMAGEMAGICK_MIN`.
- The Dockerfile asserts `rpm -q --qf '%{EVR}'` is not older (`rpm --eval "%{lua: print(rpm.vercmp('A','B'))}"`).
Applied to every job here (Build-and-Push, FPM, Shared-httpd, Shared-OLS, LiteSpeed, LSPHP) and to every other repo with a Docker Hub base (see "Repositories wired").
- `/etc/nginx/conf.d/pkgcache-http.conf` holds the cache zone, maps and log format.
- `/etc/nginx/pkgcache/{server-common,meta,pkg}.conf` hold the shared settings.
- `/etc/nginx/sites-available/pkgcache` is **generated** by `/usr/local/sbin/pkgcache-gen-nginx`. Edit the `UPSTREAMS` table, then run `pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx`.
- Listeners: :443 TLS for the LAN, `127.0.0.1:80` for local tools, and :80 for the LAN answers 403.
- **Registry.** Config is `/etc/docker/registry/config.yml` (Debian `docker-registry` 2.8.3, the same code as `registry:2`, with TLS). Service `docker-registry`.
- **TLS.** Everything is in `/etc/pkgcache-ca/`: `ca.key` (0600), `ca.crt`, `server.key`, `server-chain.crt`, and the `ca.cnf`/`srv.cnf` openssl configs.
- **Data.** nginx uses `/srv/cache/nginx`, the registry uses `/srv/cache/registry`. Both live on the 150 GB mount point `mp0` (`backup=0`).
- **Security-release day.** `pkgcache-refresh` re-fetches every cached metadata entry right away. Variants:
- `pkgcache-refresh remi` refreshes one prefix;
- `--list` lists the entries;
- `--purge` deletes them instead.
A refresh bypasses the cache from localhost only and replaces each entry in place. If upstream is down, the old copy stays. Metadata is at most 3 minutes old anyway.
- **Monitoring.** `pkgcache-stats` shows:
- disk usage;
- hit ratio;
- fallback-tier use;
- the STALE count;
- upstream errors;
- certificate expiry.
Logs are `/var/log/nginx/pkgcache.access.log` (fields `cache=`, `tier=`, `up=`) and `journalctl -u docker-registry`. Suggested Uptime Kuma checks are `https://192.168.1.40/` and `https://192.168.1.40:5000/v2/`, with certificate checking against the CA.
- **Docker Hub rate limits.** Anonymous pulls are limited per egress IP, and all clients share it. Add `username`/`password` (a Hub access token) under `proxy:` if needed.
