cloud-apache-container/docs/pkg-cache.md

Internal package + image cache (PKG_CACHE)

Opt-in build support for the LAN package cache CT130 pkg-cache at https://192.168.1.40 (Proxmox node pve, storage shared-storage-tn).

Without PKG_CACHE, builds use the upstream repositories. The one change from before is that the EPEL and Remi release RPMs are now GPG-verified against keys/pkg-cache/ before they are installed.

Why it exists: during the 2026-10-21 PHP security release, builds failed because:

• Remi mirrors lagged hours behind origin;
• Remi's origin reset or timed out HTTPS under load;
• an EPEL mirror had a broken certificate.

On that day a build without the cache spent 1031 s in its first dnf step. Through the cache the same step took 40 s cold and 21 s warm.

What the cache serves

URL (https, pkg-cache CA)	Upstream tiers, tried in order (stale copy last)
https://192.168.1.40/remi/	https://rpms.remirepo.net, then http://rpms.remirepo.net (origin only, no mirrors)
https://192.168.1.40/epel/	https://dl.fedoraproject.org/pub/epel, https://mirror.math.princeton.edu/pub/epel, https://mirrors.kernel.org/fedora-epel
https://192.168.1.40/almalinux/	https://repo.almalinux.org/almalinux, https://mirror.math.princeton.edu/pub/almalinux
https://192.168.1.40/ubuntu/	https://archive.ubuntu.com/ubuntu, https://us.archive.ubuntu.com/ubuntu
https://192.168.1.40/ubuntu-security/	https://security.ubuntu.com/ubuntu, https://archive.ubuntu.com/ubuntu
https://192.168.1.40/litespeed/	https://rpms.litespeedtech.com, then http://rpms.litespeedtech.com
https://192.168.1.40/nodesource/	https://rpm.nodesource.com (origin only; RPMs signed, repomd not) — added 2026-10-22
https://192.168.1.40/debian/	https://deb.debian.org/debian (CDN origin; princeton was 6 h behind on trixie-updates) — added 2026-10-22
https://192.168.1.40/debian-security/	https://security.debian.org/debian-security, https://deb.debian.org/debian-security — added 2026-10-22
https://192.168.1.40/goproxy/	https://proxy.golang.org (Go module proxy; /sumdb/ is 404 there, so go checks sum.golang.org directly) — added 2026-10-22
192.168.1.40:5000 (TLS, pkg-cache CA)	Docker Hub pull-through cache (registry-1.docker.io)

Plain http://192.168.1.40/ answers 403 to LAN clients.

Cache rules:

• Metadata has a 3-minute TTL. After that it is revalidated upstream (If-Modified-Since). Metadata means:
  • repomd.xml*
  • repodata files whose names don't start with a hash
  • apt dists/, except by-hash/
  • *-latest*.rpm and remi-release-N.rpm
  • RPM-GPG-KEY*
  • Go proxy /@v/list and /@latest
  • directory listings
• Packages (*.rpm, *.deb, hash-named repodata, by-hash/) are cached for 30 days and evicted after 60 days unused. Total size is capped at 110 GB, with 15 GB kept free.
• On a connect error, timeout or 5xx, the request falls through to the next tier. Only the last tier may answer from a stale copy, so the order is primary, then fallback, then stale.
• proxy_cache_lock is on.
• Responses carry X-Cache-Status and X-Pkgcache-Tier.

Building with the cache

docker buildx build --build-arg PKG_CACHE=https://192.168.1.40 --build-arg PHPVER=85 -f Dockerfile .

scripts/pkg-cache-repos.sh (installed as /usr/local/sbin/) and keys/pkg-cache/ (installed as /usr/local/share/pkg-cache/) are wired into Dockerfile, Dockerfile.fpm, Dockerfile.shared-httpd and Dockerfile.shared-ols. The script has four modes.

rpm handles AlmaLinux, EPEL and Remi:

• Each baseurl becomes <cache>/<prefix>/... <origin>/.... dnf fails over to the origin if the cache dies.
• That section's mirrorlist/metalink is commented out.
• The section gets sslcacert= pointing at a bundle of the system CAs plus the pkg-cache CA. The bundle lives in the state dir and is deleted by restore.
• It also gets gpgcheck=1.
• AlmaLinux and Remi also get repo_gpgcheck=1, because both sign repomd.xml (measured 2026-10-21: repomd.xml.asc returns 200). EPEL publishes no repomd signature (404), so its metadata integrity rests on HTTPS end to end.

apt handles Ubuntu archive/security and LiteSpeed:

• Their URIs are rewritten to the cache.
• /etc/apt/apt.conf.d/99pkg-cache sets Acquire::https::192.168.1.40::CaInfo, so the CA is trusted for the cache host only.
• apt verifies the signed InRelease/Release.gpg as always. It has no failover, so CI must probe the cache first.

release-install epel [remi]:

• Downloads the release RPM over HTTPS, from the cache or from the origin.
• Checks the package name.
• Imports the key committed in keys/pkg-cache/.
• Requires rpm -K to report digests signatures OK.
• Installs with --setopt=localpkg_gpgcheck=1.

nodesource MAJOR (added 2026-10-22) replaces curl https://rpm.nodesource.com/setup_N.x | bash:

• writes /etc/yum.repos.d/nodesource-nodejs.repo with the ORIGIN baseurl, gpgcheck=1 and gpgkey=file:///usr/local/share/pkg-cache/RPM-GPG-KEY-nodesource (key committed in keys/pkg-cache/);
• with PKG_CACHE it is then rewritten like the other repos; restore puts the origin version back, so the shipped image keeps a normal NodeSource repo.

go-env (added 2026-10-22): eval "$(pkg-cache-repos.sh go-env)" inside the RUN that runs go mod download/go build. Prints GOPROXY=<cache>/goproxy|https://proxy.golang.org|direct (| = fall through on ANY error) and SSL_CERT_FILE=<state>/ca-bundle.pem; nothing without PKG_CACHE. Modules stay verified by go.sum + sum.golang.org. End the RUN with restore (removes the bundle).

EL9: release-install epel|remi picks the EL major from rpm -E %rhel (9 or 10); RPM-GPG-KEY-EPEL-9 is committed.

Debian (apt mode): deb.debian.org/debian, deb.debian.org/debian-security and security.debian.org (classic and deb822 .sources) are rewritten to <cache>/debian/ and <cache>/debian-security/.

Not proxied, on purpose: PyPI and the npm registry. Our pip/npm installs are not hash-locked (pip install -r without --require-hashes, npm install -g pm2@latest), so the client would trust whatever the cache served — a compromised cache would mean code execution in shipped images. Proxy them only after the installs are hash-pinned. GitHub-release downloads are likewise left direct.

restore:

• Puts rewritten files back.
• Deletes files the script created.
• Deletes any repo/apt file that appeared during the RUN and still references the cache.
• Then fails the build if any file under /etc/yum.repos.d, /etc/dnf or /etc/apt still mentions the cache.

Every RUN that rewrites ends with restore, so no shipped layer carries the LAN address. This was verified on the built images: 0 references, no state dir, no apt drop-in.

PKG_CACHE must start with https://. http:// is refused.

Dockerfile.litespeed and Dockerfile.lsphp are wired since 2026-10-22 (normal apt path; the php.net EMERGENCY stage lsphp-src stays on upstream). lsphp-apt-upgrade.sh's madison gate accepts <PKG_CACHE>/litespeed/ only when PKG_CACHE is set.

Repositories wired (2026-10-22)

The script and keys/pkg-cache/ are canonical here; every other repo carries byte-identical copies (sha256sum scripts/pkg-cache-repos.sh to compare). Change it here first, then sync. Same for .gitea/scripts/resolve-base-image.sh.

Repo	Image(s)	Bases	Through the cache	Base digest-pinned
cloud-apache-container	cac, cac-fpm, shared-httpd (almalinux/10-base); shared-ols, cac-litespeed, cac-lsphp (litespeedtech/openlitespeed)	Docker Hub	AlmaLinux/EPEL/Remi (rpm); Ubuntu + LiteSpeed (apt)	yes, all 6 jobs
cloud-node-container	cnoc:node18..26	almalinux/10-base	AlmaLinux/EPEL + NodeSource (nodesource N)	yes
cloud-memcached-container	cmc	almalinux/9-base	AlmaLinux 9 + EPEL 9	yes
cloud-haproxy-container	chc	almalinux/9-base	AlmaLinux 9 + EPEL 9 (inotify-tools GitHub tarball direct)	yes
whp-default-container	whp-default	httpd:alpine	nothing to install (mirror only)	yes
versitygw-container	versitygw	ubuntu:latest	Ubuntu (versitygw .deb from GitHub direct)	yes
haproxy-manager-base	haproxy-manager-base, coraza-spoa	repo.anhonesthost.net python:3.12-slim / golang:1.25 (not Docker Hub: no mirror); gcr.io distroless	Debian (apt) + Go modules (go-env); pip direct	n/a; mirror-base-image.yaml now copies Docker Hub images by the digest resolved directly from Docker Hub
cpanel-importer	cpanel-importer	almalinux:10-minimal	AlmaLinux/EPEL/Remi — PR #1 open, not merged (pending rebase onto the PHP 8.5 work)	in the PR

Not image builds (nothing to wire): kb-anhonesthost, anhonesthost-blog, dao-mail-signup, renovate-runner. ols-mail-client builds images in CI only for scanning/smoke tests (never pushed), its bases are already digest-pinned, and they install no OS packages.

Adding a new upstream

1. On CT130: back up, then add the prefix to UPSTREAMS in /usr/local/sbin/pkgcache-gen-nginx (origin first; add a mirror tier only if you measured it is as fresh as the origin). If its mutable metadata isn't covered by META_RE, extend it there AND in /usr/local/sbin/pkgcache-refresh. Add the prefix to the location = / text in /etc/nginx/pkgcache/server-common.conf.
2. pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx (pct exec has no /usr/local/sbin in PATH: use full paths). Verify old and new prefixes with curl --cacert keys/pkg-cache/ca.crt https://192.168.1.40/<prefix>/... (look at X-Cache-Status/X-Pkgcache-Tier). CT130 has no egress filter, so no firewall change is needed for new origins.
3. Only proxy upstreams whose content the CLIENT verifies end to end (GPG-signed packages/metadata, Go sumdb, content digests). Commit any signing key under keys/pkg-cache/ with its fingerprint and provenance in the README.
4. Teach scripts/pkg-cache-repos.sh the rewrite (keep the origin as a fallback where the client supports it), test through the cache from a runner-network host, then sync the copies.

Trust model

keys/pkg-cache/ca.crt is the pkg-cache internal CA:

• EC P-256, valid to 2036.
• Name-constrained to IP 192.168.1.40 and DNS pkg-cache.internal.cloud-hosting.io / pkg-cache.geekhome.local. Even its private key, which exists only on CT130 at /etc/pkgcache-ca/ca.key, cannot mint a certificate that clients would accept for any other host.
• It is trusted only by the rewritten repos, the apt per-host option and the BuildKit/dockerd registry config. It is never added to a system trust store.

The server certificate /etc/pkgcache-ca/server.crt expires 2029-10-20. pkgcache-stats prints the expiry. Renew it with the same openssl x509 -req ... -extfile srv.cnf steps in /etc/pkgcache-ca/.

Remaining risks:

• Replay of old but validly signed metadata. Signatures stop tampering, not freshness. The cache's own plain-http tiers for Remi and LiteSpeed are the leg where an internet MITM could replay. The php.net gates (EXPECT_PHP_MIN, EXPECT_LSPHP_PKG/EXPECT_PHP_VERSION) fail an image build that gets old PHP. Josh to decide whether to drop those http tiers: you gain replay resistance and lose availability when Remi's HTTPS is down, as it was all day on 2026-10-21.
• Proposed, not implemented: the same kind of minimum-version gate for other security-relevant packages:
  • CI resolves the newest openssl/openssl-libs and httpd from AlmaLinux BaseOS/AppStream, and ImageMagick from EPEL, directly from the origin over HTTPS (e.g. dnf repoquery --repofrompath=o,https://repo.almalinux.org/almalinux/10/BaseOS/x86_64/os/ --repo=o --latest-limit=1 --qf '%{evr}' openssl-libs).
  • It passes the results as EXPECT_OPENSSL_MIN / EXPECT_HTTPD_MIN / EXPECT_IMAGEMAGICK_MIN.
  • The Dockerfile asserts rpm -q --qf '%{EVR}' is not older (rpm --eval "%{lua: print(rpm.vercmp('A','B'))}").

CI integration (APPLIED 2026-10-21)

1. Runner network. Ephemeral runners are NATed and iptables FORWARD drops all RFC1918 destinations:

   • eph-pve-* on pve, vmbr2, 10.99.98.0/24;
   • eph-pve3-* on pve3, vmbr1, 10.99.99.0/24.

   One rule per node now sits above those DROPs. It is persistent through a post-up/post-down pair placed before the first DROP line in /etc/network/interfaces.d/vmbr2-ci-isolated (pve) and /etc/network/interfaces.d/vmbr1-ci-isolated (pve3). The Proxmox firewall is disabled on both nodes, so its reloads do not touch these chains. Backups: /root/vmbr{2,1}-ci-isolated.bak-pkgcache-20261021.

   iptables -I FORWARD 1 -i vmbr2 -o vmbr0 -d 192.168.1.40/32 -p tcp -m multiport --dports 443,5000 \
     -m comment --comment pkg-cache-ct130 -j ACCEPT
   

   Verified from throwaway CTs on vmbr2 and vmbr1:

   • :443 and :5000 work with TLS verified;
   • CT130 :22/:80, pve :8006, .193 :22, the router's :53 and 10.10.14.20 stay blocked;
   • internet access is unchanged.

   The LAN runner VM Git-Builders (VM103, 192.168.1.126 on vmbr0) needs no rule. gitea-runner.internal.cloud-hosting.io resolves to 10.10.14.20, which is unroutable from both the LAN and the DC (stale record?).

2. Workflow (.gitea/workflows/build-push.yaml). Jobs Build-and-Push, Build-FPM-Images, Build-Shared-httpd and Build-Shared-OLS run a "Probe package cache" step. It curls https://192.168.1.40/ and https://192.168.1.40:5000/v2/ with --cacert keys/pkg-cache/ca.crt --max-time 5.

   • If both answer: PKG_CACHE=https://192.168.1.40 is passed as a build arg, and buildx is set up with the Docker Hub mirror (buildkitd-config-inline, ca=).
   • If either fails: that output is empty, the build uses upstream repos / Docker Hub, and a ::warning:: is emitted.

   Litespeed and lsphp jobs are unchanged (their Dockerfiles are not wired). Fail-back measured with the workflow's own step code, from a runner-bridge CT:

   cache state	probe result	time
   up	url + mirror set	<1 s
   nginx + registry stopped	both empty, warning	<1 s (connection refused)
   silently dropped	both empty, warning	10 s (2 x 5 s)

   Mid-build cache loss is also covered: dnf has the origin as a 2nd baseurl, and BuildKit falls back from the mirror to Docker Hub (measured with the registry stopped).

3. dockerd clients (plain docker pull). Untested, because it needs a daemon restart. Use TLS, no insecure-registries:

   install -D -m0644 keys/pkg-cache/ca.crt /etc/docker/certs.d/192.168.1.40:5000/ca.crt
   # /etc/docker/daemon.json:  { "registry-mirrors": ["https://192.168.1.40:5000"] }   then restart docker
   

Base-image digest pinning (2026-10-22)

Gap closed (security review): with the BuildKit mirror on, FROM almalinux/10-base was resolved by tag through the cache, so a compromised cache could hand out a different image for that tag.

Now every job that uses the mirror first runs "Resolve base image digest (Docker Hub, direct)":

• .gitea/scripts/resolve-base-image.sh IMAGE:TAG gets an anonymous pull token from auth.docker.io and sends a HEAD /v2/<repo>/manifests/<tag> to registry-1.docker.io over HTTPS (never the mirror; HEADs don't count against the pull rate limit). It prints IMAGE@sha256:<index digest>. Input and output are regex-validated; values reach run: only via env:.
• The job passes BASE_IMAGE=<that> as a build arg; the Dockerfiles say ARG BASE_IMAGE=<tag> / FROM ${BASE_IMAGE}. The mirror can then only serve content-addressed manifests/blobs, which BuildKit verifies against the digest.
• Lookup fails: BASE_IMAGE falls back to the tag, the probe step does not enable the registry mirror (registry-mirror=<not used: Docker Hub direct>), and the tag is pulled straight from Docker Hub, as before the cache existed. The package-repo cache (PKG_CACHE) is unaffected, since its content is GPG-checked.
• Freshness is automatic: the digest is looked up on every run, nothing is hard-coded. Local builds without the arg use the tag.
• Log lines to look for: base-image: almalinux/10-base:latest -> almalinux/10-base@sha256:... and pkg-cache: repos=https://192.168.1.40 registry-mirror=192.168.1.40:5000.

Applied to every job here (Build-and-Push, FPM, Shared-httpd, Shared-OLS, LiteSpeed, LSPHP) and to every other repo with a Docker Hub base (see "Repositories wired"). The script is canonical here (.gitea/scripts/resolve-base-image.sh); other repos carry byte-identical copies.

WHP hosts (scripts/host/pkgcache-host-repos.sh)

pkgcache-host-repos.sh apply https://192.168.1.40 does the following:

• backs up every /etc/yum.repos.d/*.repo to /root/pkgcache-repo-backup-<UTC>/;
• installs the name-constrained CA as a system anchor;
• rewrites the AlmaLinux, EPEL and Remi sections to baseurl=<cache>/... <origin>/...;
• comments out mirrorlist/metalink in those sections;
• sets gpgcheck=1, plus repo_gpgcheck=1 for AlmaLinux and Remi.

revert [DIR] restores byte-identical files (verified on AL9 and AL10) and removes the anchor. status shows the current state.

Why not "cache first, mirrorlist as fallback": measured on AL10, with mirrorlist/metalink left in place dnf ignores baseurl completely (0 cache requests). The origin as second baseurl is the fallback dnf really performs.

Measured on 192.168.1.148 (AlmaLinux 10.2), 2026-10-21:

test	result
baseline dnf makecache --refresh (original mirrorlists)	rc=0, 1209 s
via cache makecache --refresh	rc=0, 22 s (2nd run, non-interactive: 2 s)
via cache dnf download php-cli	rc=0, 1 s
dnf -y --downloadonly update php-cli	"Nothing to do" (already 8.5.12)
cache down (nginx stopped): makecache --refresh / download	rc=0 16 s / rc=0 24 s, served by origin
cache silently dropped (iptables DROP): makecache --refresh / download	rc=0 225 s / rc=0 59 s (dnf's 30 s connect timeout per attempt)

Datacenter hosts (whp01, whp02, sdbees, TrueSelfCA) cannot reach 192.168.1.40 and are not configured. That needs CT130 on NetBird (see the report: setup key / route pending).

Operating the cache (CT130)

• nginx config.

  • /etc/nginx/conf.d/pkgcache-http.conf holds the cache zone, maps and log format.
  • /etc/nginx/pkgcache/{server-common,meta,pkg}.conf hold the shared settings.
  • /etc/nginx/sites-available/pkgcache is generated by /usr/local/sbin/pkgcache-gen-nginx. Edit the UPSTREAMS table, then run pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx.
  • Listeners: :443 TLS for the LAN, 127.0.0.1:80 for local tools, and :80 for the LAN answers 403.

• Registry. Config is /etc/docker/registry/config.yml (Debian docker-registry 2.8.3, the same code as registry:2, with TLS). Service docker-registry.

• TLS. Everything is in /etc/pkgcache-ca/: ca.key (0600), ca.crt, server.key, server-chain.crt, and the ca.cnf/srv.cnf openssl configs.

• Data. nginx uses /srv/cache/nginx, the registry uses /srv/cache/registry. Both live on the 150 GB mount point mp0 (backup=0).

• Security-release day. pkgcache-refresh re-fetches every cached metadata entry right away. Variants:

  • pkgcache-refresh remi refreshes one prefix;
  • --list lists the entries;
  • --purge deletes them instead.

  A refresh bypasses the cache from localhost only and replaces each entry in place. If upstream is down, the old copy stays. Metadata is at most 3 minutes old anyway.

• Monitoring. pkgcache-stats shows:

  • disk usage;
  • hit ratio;
  • fallback-tier use;
  • the STALE count;
  • upstream errors;
  • certificate expiry.

  Logs are /var/log/nginx/pkgcache.access.log (fields cache=, tier=, up=) and journalctl -u docker-registry. Suggested Uptime Kuma checks are https://192.168.1.40/ and https://192.168.1.40:5000/v2/, with certificate checking against the CA.

• Docker Hub rate limits. Anonymous pulls are limited per egress IP, and all clients share it. Add username/password (a Hub access token) under proxy: if needed.