cloud-apache-container/docs/pkg-cache.md

Internal package + image cache (PKG_CACHE)

Opt-in build support for the LAN package cache CT130 pkg-cache at https://192.168.1.40 (Proxmox node pve, storage shared-storage-tn).

Without PKG_CACHE, builds use the upstream repositories. The one change from before is that the EPEL and Remi release RPMs are now GPG-verified against keys/pkg-cache/ before they are installed.

Why it exists: during the 2026-10-21 PHP security release, builds failed because:

• Remi mirrors lagged hours behind origin;
• Remi's origin reset or timed out HTTPS under load;
• an EPEL mirror had a broken certificate.

On that day a build without the cache spent 1031 s in its first dnf step. Through the cache the same step took 40 s cold and 21 s warm.

What the cache serves

URL (https, pkg-cache CA)	Upstream tiers, tried in order (stale copy last)
https://192.168.1.40/remi/	https://rpms.remirepo.net, then http://rpms.remirepo.net (origin only, no mirrors)
https://192.168.1.40/epel/	https://dl.fedoraproject.org/pub/epel, https://mirror.math.princeton.edu/pub/epel, https://mirrors.kernel.org/fedora-epel
https://192.168.1.40/almalinux/	https://repo.almalinux.org/almalinux, https://mirror.math.princeton.edu/pub/almalinux
https://192.168.1.40/ubuntu/	https://archive.ubuntu.com/ubuntu, https://us.archive.ubuntu.com/ubuntu
https://192.168.1.40/ubuntu-security/	https://security.ubuntu.com/ubuntu, https://archive.ubuntu.com/ubuntu
https://192.168.1.40/litespeed/	https://rpms.litespeedtech.com, then http://rpms.litespeedtech.com
192.168.1.40:5000 (TLS, pkg-cache CA)	Docker Hub pull-through cache (registry-1.docker.io)

Plain http://192.168.1.40/ answers 403 to LAN clients.

Cache rules:

• Metadata has a 3-minute TTL. After that it is revalidated upstream (If-Modified-Since). Metadata means:
  • repomd.xml*
  • repodata files whose names don't start with a hash
  • apt dists/, except by-hash/
  • *-latest*.rpm and remi-release-N.rpm
  • RPM-GPG-KEY*
  • directory listings
• Packages (*.rpm, *.deb, hash-named repodata, by-hash/) are cached for 30 days and evicted after 60 days unused. Total size is capped at 110 GB, with 15 GB kept free.
• On a connect error, timeout or 5xx, the request falls through to the next tier. Only the last tier may answer from a stale copy, so the order is primary, then fallback, then stale.
• proxy_cache_lock is on.
• Responses carry X-Cache-Status and X-Pkgcache-Tier.

Building with the cache

docker buildx build --build-arg PKG_CACHE=https://192.168.1.40 --build-arg PHPVER=85 -f Dockerfile .

scripts/pkg-cache-repos.sh (installed as /usr/local/sbin/) and keys/pkg-cache/ (installed as /usr/local/share/pkg-cache/) are wired into Dockerfile, Dockerfile.fpm, Dockerfile.shared-httpd and Dockerfile.shared-ols. The script has four modes.

rpm handles AlmaLinux, EPEL and Remi:

• Each baseurl becomes <cache>/<prefix>/... <origin>/.... dnf fails over to the origin if the cache dies.
• That section's mirrorlist/metalink is commented out.
• The section gets sslcacert= pointing at a bundle of the system CAs plus the pkg-cache CA. The bundle lives in the state dir and is deleted by restore.
• It also gets gpgcheck=1.
• AlmaLinux and Remi also get repo_gpgcheck=1, because both sign repomd.xml (measured 2026-10-21: repomd.xml.asc returns 200). EPEL publishes no repomd signature (404), so its metadata integrity rests on HTTPS end to end.

apt handles Ubuntu archive/security and LiteSpeed:

• Their URIs are rewritten to the cache.
• /etc/apt/apt.conf.d/99pkg-cache sets Acquire::https::192.168.1.40::CaInfo, so the CA is trusted for the cache host only.
• apt verifies the signed InRelease/Release.gpg as always. It has no failover, so CI must probe the cache first.

release-install epel [remi]:

• Downloads the release RPM over HTTPS, from the cache or from the origin.
• Checks the package name.
• Imports the key committed in keys/pkg-cache/.
• Requires rpm -K to report digests signatures OK.
• Installs with --setopt=localpkg_gpgcheck=1.

restore:

• Puts rewritten files back.
• Deletes files the script created.
• Deletes any repo/apt file that appeared during the RUN and still references the cache.
• Then fails the build if any file under /etc/yum.repos.d, /etc/dnf or /etc/apt still mentions the cache.

Every RUN that rewrites ends with restore, so no shipped layer carries the LAN address. This was verified on the built images: 0 references, no state dir, no apt drop-in.

PKG_CACHE must start with https://. http:// is refused.

Dockerfile.litespeed and Dockerfile.lsphp are not wired yet. Two things are needed first:

• the multi-stage lsphp-apt-upgrade.sh flow needs a PKG_CACHE ARG in each stage;
• its madison check must accept the cache URL. It currently requires *rpms.litespeedtech.com*, and via the cache madison shows https://192.168.1.40/litespeed/debian.

Trust model

keys/pkg-cache/ca.crt is the pkg-cache internal CA:

• EC P-256, valid to 2036.
• Name-constrained to IP 192.168.1.40 and DNS pkg-cache.internal.cloud-hosting.io / pkg-cache.geekhome.local. Even its private key, which exists only on CT130 at /etc/pkgcache-ca/ca.key, cannot mint a certificate that clients would accept for any other host.
• It is trusted only by the rewritten repos, the apt per-host option and the BuildKit/dockerd registry config. It is never added to a system trust store.

The server certificate /etc/pkgcache-ca/server.crt expires 2029-10-20. pkgcache-stats prints the expiry. Renew it with the same openssl x509 -req ... -extfile srv.cnf steps in /etc/pkgcache-ca/.

Remaining risks:

• Replay of old but validly signed metadata. Signatures stop tampering, not freshness. The cache's own plain-http tiers for Remi and LiteSpeed are the leg where an internet MITM could replay. The php.net gates (EXPECT_PHP_MIN, EXPECT_LSPHP_PKG/EXPECT_PHP_VERSION) fail an image build that gets old PHP. Josh to decide whether to drop those http tiers: you gain replay resistance and lose availability when Remi's HTTPS is down, as it was all day on 2026-10-21.
• Proposed, not implemented: the same kind of minimum-version gate for other security-relevant packages:
  • CI resolves the newest openssl/openssl-libs and httpd from AlmaLinux BaseOS/AppStream, and ImageMagick from EPEL, directly from the origin over HTTPS (e.g. dnf repoquery --repofrompath=o,https://repo.almalinux.org/almalinux/10/BaseOS/x86_64/os/ --repo=o --latest-limit=1 --qf '%{evr}' openssl-libs).
  • It passes the results as EXPECT_OPENSSL_MIN / EXPECT_HTTPD_MIN / EXPECT_IMAGEMAGICK_MIN.
  • The Dockerfile asserts rpm -q --qf '%{EVR}' is not older (rpm --eval "%{lua: print(rpm.vercmp('A','B'))}").

CI integration plan (NOT applied — needs Josh)

1. Network (blocker). The ephemeral runners are NATed, and iptables FORWARD drops all RFC1918 destinations, including 192.168.1.40:

   • eph-pve-* on pve, vmbr2, 10.99.98.0/24;
   • eph-pve3-* on pve3, vmbr1, 10.99.99.0/24.

   A narrow exception is needed, placed before the DROP lines:

   # pve, /etc/network/interfaces.d/ vmbr2 stanza, BEFORE the 192.168.0.0/16 DROP:
   post-up   iptables -A FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 443,5000 -j ACCEPT
   post-down iptables -D FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 443,5000 -j ACCEPT
   # live: iptables -I FORWARD 1 -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 443,5000 -j ACCEPT
   # pve3: the same with vmbr1
   

   gitea-runner.internal.cloud-hosting.io resolves to 10.10.14.20, which is not on this LAN. Whether it can reach the cache is untested.

2. Probe with the CA, then opt in. Keep ${{ }} out of run:. The probe fails when the cache is down or its certificate is expired or untrusted, and the build then uses upstream.

   - name: Probe package cache
     id: pkgcache
     run: |
       if curl -fsS --max-time 5 --cacert keys/pkg-cache/ca.crt -o /dev/null https://192.168.1.40/ ; then
         echo "url=https://192.168.1.40" >> "$GITHUB_OUTPUT"
       else
         echo "url=" >> "$GITHUB_OUTPUT"; echo "::warning::pkg-cache unreachable - building from upstream"
       fi
   # docker/build-push-action `build-args:` add   PKG_CACHE=${{ steps.pkgcache.outputs.url }}
   

3. Docker Hub mirror for BuildKit, over TLS and without insecure. setup-buildx-action uses the docker-container driver, which ignores dockerd's registry-mirrors. buildx copies the ca file into the builder.

   - uses: docker/setup-buildx-action@v3
     with:
       buildkitd-config-inline: |
         [registry."docker.io"]
           mirrors = ["192.168.1.40:5000"]
         [registry."192.168.1.40:5000"]
           ca = ["keys/pkg-cache/ca.crt"]
   

   The exact equivalent (absolute ca path) was measured on 192.168.1.193. BuildKit falls back to Docker Hub when the mirror is down; this was measured too, with the registry stopped. The relative ca path inside a workflow is untested; use ${{ github.workspace }}/keys/pkg-cache/ca.crt if it doesn't resolve.

4. dockerd clients (plain docker pull). Untested, because it needs a daemon restart. Use TLS, no insecure-registries:

   install -D -m0644 keys/pkg-cache/ca.crt /etc/docker/certs.d/192.168.1.40:5000/ca.crt
   # /etc/docker/daemon.json:  { "registry-mirrors": ["https://192.168.1.40:5000"] }   then restart docker
   

WHP hosts (plan only)

The production hosts (whp01, whp02, sdbees) are in the datacenter and cannot reach 192.168.1.40. Only LAN hosts can, e.g. 192.168.1.148. Options for Josh:

• (a) a second identical cache in the datacenter, with its own name-constrained CA;
• (b) expose this one over NetBird;
• (c) LAN hosts only.

For a host that can reach a cache:

• install the CA at /etc/pki/pkg-cache/ca.crt;
• create a bundle (cat /etc/pki/tls/certs/ca-bundle.crt /etc/pki/pkg-cache/ca.crt > /etc/pki/pkg-cache/bundle.pem);
• in each repo section, keep the origin as the second URL:

[remi-safe]
baseurl=https://<cache>/remi/enterprise/$releasever_major/safe/$basearch/ https://rpms.remirepo.net/enterprise/$releasever_major/safe/$basearch/
#mirrorlist=...
sslcacert=/etc/pki/pkg-cache/bundle.pem
gpgcheck=1
repo_gpgcheck=1      # Remi + AlmaLinux sign repomd; EPEL does not

Hosts have no php.net gate. On a security-release day, run pkgcache-refresh on the cache and check dnf info php-cli against php.net.

Operating the cache (CT130)

• nginx config.

  • /etc/nginx/conf.d/pkgcache-http.conf holds the cache zone, maps and log format.
  • /etc/nginx/pkgcache/{server-common,meta,pkg}.conf hold the shared settings.
  • /etc/nginx/sites-available/pkgcache is generated by /usr/local/sbin/pkgcache-gen-nginx. Edit the UPSTREAMS table, then run pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx.
  • Listeners: :443 TLS for the LAN, 127.0.0.1:80 for local tools, and :80 for the LAN answers 403.

• Registry. Config is /etc/docker/registry/config.yml (Debian docker-registry 2.8.3, the same code as registry:2, with TLS). Service docker-registry.

• TLS. Everything is in /etc/pkgcache-ca/: ca.key (0600), ca.crt, server.key, server-chain.crt, and the ca.cnf/srv.cnf openssl configs.

• Data. nginx uses /srv/cache/nginx, the registry uses /srv/cache/registry. Both live on the 150 GB mount point mp0 (backup=0).

• Security-release day. pkgcache-refresh re-fetches every cached metadata entry right away. Variants:

  • pkgcache-refresh remi refreshes one prefix;
  • --list lists the entries;
  • --purge deletes them instead.

  A refresh bypasses the cache from localhost only and replaces each entry in place. If upstream is down, the old copy stays. Metadata is at most 3 minutes old anyway.

• Monitoring. pkgcache-stats shows:

  • disk usage;
  • hit ratio;
  • fallback-tier use;
  • the STALE count;
  • upstream errors;
  • certificate expiry.

  Logs are /var/log/nginx/pkgcache.access.log (fields cache=, tier=, up=) and journalctl -u docker-registry. Suggested Uptime Kuma checks are https://192.168.1.40/ and https://192.168.1.40:5000/v2/, with certificate checking against the CA.

• Docker Hub rate limits. Anonymous pulls are limited per egress IP, and all clients share it. Add username/password (a Hub access token) under proxy: if needed.