cloud-apache-container/docs/pkg-cache.md

Internal package + image cache (PKG_CACHE)

Opt-in build support for the LAN package cache CT130 pkg-cache = 192.168.1.40 (Proxmox node pve, storage shared-storage-tn). Without PKG_CACHE, builds use the upstream repositories exactly as before.

Why it exists: during the 2026-10-21 PHP security release, builds failed because Remi mirrors lagged hours behind origin, Remi's origin reset or timed out HTTPS under load, and an EPEL mirror had a broken certificate.

What the cache serves

URL on the cache	Upstream tiers, tried in order (stale copy last)
http://192.168.1.40/remi/	https://rpms.remirepo.net, then http://rpms.remirepo.net (origin only, no mirrors)
http://192.168.1.40/epel/	https://dl.fedoraproject.org/pub/epel, https://mirror.math.princeton.edu/pub/epel, https://mirrors.kernel.org/fedora-epel
http://192.168.1.40/almalinux/	https://repo.almalinux.org/almalinux, https://mirror.math.princeton.edu/pub/almalinux
http://192.168.1.40/ubuntu/	https://archive.ubuntu.com/ubuntu, https://us.archive.ubuntu.com/ubuntu
http://192.168.1.40/ubuntu-security/	https://security.ubuntu.com/ubuntu, https://archive.ubuntu.com/ubuntu
http://192.168.1.40/litespeed/	https://rpms.litespeedtech.com, then http://rpms.litespeedtech.com
192.168.1.40:5000 (plain http)	Docker Hub pull-through cache (registry-1.docker.io)

Cache rules:

• Metadata has a 3-minute TTL. After that it is revalidated upstream (If-Modified-Since). Metadata here means repomd.xml*, repodata files whose names don't start with a hash, apt dists/ except by-hash/, *-latest*.rpm, remi-release-N.rpm, RPM-GPG-KEY* and directory listings.
• Packages (*.rpm, *.deb, hash-named repodata, by-hash/) are kept for 30 days, and evicted after 60 days without use. The cache is capped at 110 GB (min_free 15 GB).
• On a connect error, timeout or 5xx, a request falls through to the next tier. Only the last tier answers from a stale copy, so metadata goes primary → fallback → stale. A fresher fallback always wins over stale.
• proxy_cache_lock is on: concurrent misses on one file make a single upstream fetch.
• Each response carries X-Cache-Status (HIT/MISS/REVALIDATED/EXPIRED/STALE/BYPASS) and X-Pkgcache-Tier. On a HIT, the tier header names the location, not the upstream that filled the entry.

Building with the cache

docker buildx build --build-arg PKG_CACHE=http://192.168.1.40 --build-arg PHPVER=85 -f Dockerfile .

scripts/pkg-cache-repos.sh (copied to /usr/local/sbin/) is wired into Dockerfile, Dockerfile.fpm, Dockerfile.shared-httpd and Dockerfile.shared-ols:

• rpm: rewrites the AlmaLinux, EPEL and Remi baseurls to <cache>/<prefix>/... <origin>/..., and comments out that section's mirrorlist/metalink. dnf treats the second URL as a failover, so a dead cache falls back to the origin by itself.
• apt: rewrites the Ubuntu archive/security and LiteSpeed URIs to the cache. apt has no failover, so CI must probe the cache first (see below).
• release-urls: the EPEL and Remi release RPM URLs, from the cache when it is set.
• restore: puts the original repo files back. Every RUN that rewrites ends with restore, so no shipped layer contains the cache address. Images run on datacenter hosts that cannot reach it.

Dockerfile.litespeed and Dockerfile.lsphp are not wired yet. Their multi-stage lsphp-apt-upgrade.sh flow needs two changes first:

• a PKG_CACHE ARG in each stage;
• the madison check accepting the cache URL. It currently requires *rpms.litespeedtech.com*, and through the cache apt-cache madison shows http://192.168.1.40/litespeed/debian.

The PHP gates are unchanged and still apply: EXPECT_PHP_MIN, EXPECT_LSPHP_PKG/EXPECT_PHP_VERSION.

Why plain HTTP on the LAN (trade-off)

• The cache fetches upstream over HTTPS with certificate verification. Remi and LiteSpeed also have a plain-http origin fallback, which mirrors what the Dockerfile already did.
• Client to cache is plain HTTP. dnf still verifies package GPG signatures, and apt verifies the signed InRelease/Release.gpg plus package hashes. A LAN attacker therefore cannot inject packages. On 2026-10-21 apt measurably refused the cached LiteSpeed index when given the wrong key (NO_PUBKEY 011AA62DEDA1F085).
• The residual risk is metadata replay or freeze: an attacker on the LAN path, or on the plain-http fallback leg, serves an old but validly signed repomd.xml. That would let a build install old PHP. Remi/EPEL/AlmaLinux do not enable repo_gpgcheck, so dnf does not check metadata signatures. The php.net version gates cover this for our images: a build fails if it gets PHP older than php.net's current release.
• WHP hosts have no such gate. They should keep the origin as the second baseurl and not rely on the cache for freshness on security-release day. Run pkgcache-refresh on the cache instead.
• Signing keys must not be fetched through the cache. Take them from the base image or directly over HTTPS.
• HTTPS to the cache (internal CA or a Let's Encrypt DNS-01 certificate for pkg-cache.internal.cloud-hosting.io) would close the LAN leg. It needs a CA in every build image or a public DNS name plus certificate. Not done; Josh to decide.

CI integration plan (NOT applied — needs Josh)

1. Network (blocker). The ephemeral runners (eph-pve-* on pve vmbr2 10.99.98.0/24, eph-pve3-* on pve3 vmbr1 10.99.99.0/24) are NATed, and iptables FORWARD drops all RFC1918 destinations, including 192.168.1.40. A narrow exception is needed on each node, placed before the DROP lines in /etc/network/interfaces.d/*:

   # pve  (/etc/network/interfaces.d/ vmbr2 stanza), before the 192.168.0.0/16 DROP:
   post-up   iptables -A FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 80,5000 -j ACCEPT
   post-down iptables -D FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 80,5000 -j ACCEPT
   # live, without ifreload:
   iptables -I FORWARD 1 -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 80,5000 -j ACCEPT
   # pve3: same with vmbr1
   

   gitea-runner.internal.cloud-hosting.io resolves to 10.10.14.20, which is not on this LAN. Whether it can reach 192.168.1.40 is untested.
2. Probe, then opt in (no ${{ }} inside run:, per repo rule):

   - name: Probe package cache
     id: pkgcache
     run: |
       if curl -fsS --max-time 5 -o /dev/null http://192.168.1.40/ ; then
         echo "url=http://192.168.1.40" >> "$GITHUB_OUTPUT"
       else
         echo "url=" >> "$GITHUB_OUTPUT"; echo "::warning::pkg-cache unreachable - building from upstream"
       fi
   # in docker/build-push-action `build-args:` add
   #   PKG_CACHE=${{ steps.pkgcache.outputs.url }}
   

3. Docker Hub mirror for BuildKit. The workflows use docker/setup-buildx-action (docker-container driver). BuildKit ignores dockerd's registry-mirrors, so configure the builder instead:

   - uses: docker/setup-buildx-action@v3
     with:
       buildkitd-config-inline: |
         [registry."docker.io"]
           mirrors = ["192.168.1.40:5000"]
         [registry."192.168.1.40:5000"]
           http = true
   

   This exact config was used for the test builds on 192.168.1.193. Only Docker Hub images go through it; repo.anhonesthost.net/... is unaffected.
4. dockerd clients (plain docker pull, e.g. the build server or test hosts). Untested, because applying it needs a daemon restart. Add to /etc/docker/daemon.json and restart docker:

   { "registry-mirrors": ["http://192.168.1.40:5000"], "insecure-registries": ["192.168.1.40:5000"] }
   

WHP hosts (plan only)

The production hosts (whp01/whp02/sdbees) are in the datacenter and cannot reach 192.168.1.40. Only LAN hosts such as 192.168.1.148 can. Options for Josh:

• (a) a second identical cache CT in the datacenter;
• (b) expose this one over NetBird;
• (c) LAN hosts only.

For a host that can reach a cache, use the same idea as pkg-cache-repos.sh rpm, keeping the origin second so the cache is never a single point of failure:

# /etc/yum.repos.d/remi-safe.repo [remi-safe]   (AL9: enterprise/9)
baseurl=http://<cache>/remi/enterprise/$releasever_major/safe/$basearch/ https://rpms.remirepo.net/enterprise/$releasever_major/safe/$basearch/
#mirrorlist=...

Operating the cache (CT130)

• Config: /etc/nginx/conf.d/pkgcache-http.conf (cache zone, maps, log format) and /etc/nginx/pkgcache/{server-common,meta,pkg}.conf. /etc/nginx/sites-available/pkgcache is generated by /usr/local/sbin/pkgcache-gen-nginx: edit its UPSTREAMS table, then run pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx.
• Registry: /etc/docker/registry/config.yml (Debian docker-registry 2.8.3, the same code as registry:2), service docker-registry.
• Data: nginx in /srv/cache/nginx, registry in /srv/cache/registry (150 GB mount point mp0, backup=0).
• Security-release day: run pkgcache-refresh to re-fetch every cached metadata entry now (or pkgcache-refresh remi for one prefix; --list shows them; --purge deletes them instead). A refresh bypasses the cache from localhost only and replaces each entry in place. If upstream is down, the old copy stays. Without it, metadata is at most 3 minutes old anyway.
• Monitoring: pkgcache-stats shows disk, hit ratio, fallback-tier use, STALE count and upstream errors. Logs are in /var/log/nginx/pkgcache.access.log (with cache=, tier=, up= fields) and journalctl -u docker-registry. Suggested Uptime Kuma checks: http://192.168.1.40/ and http://192.168.1.40:5000/v2/.
• Docker Hub: anonymous pulls are rate-limited per egress IP, and every client now shares this cache's IP. To raise the limit, add username/password (a Hub access token) under proxy:.