feat(build): opt-in PKG_CACHE build arg for the internal package cache
Adds scripts/pkg-cache-repos.sh and wires it into Dockerfile, Dockerfile.fpm, Dockerfile.shared-httpd and Dockerfile.shared-ols. With --build-arg PKG_CACHE=http://192.168.1.40 (CT130 pkg-cache), dnf/apt go through the LAN cache. That cache fetches from the Remi/EPEL/AlmaLinux/ Ubuntu/LiteSpeed origins over HTTPS, with fallback tiers and stale-on-error. Without the arg the script is a no-op and builds are unchanged. - rpm: each baseurl becomes "<cache>/<prefix>/... <origin>/...", so dnf fails over to the origin if the cache dies mid-build - every RUN restores the original repo files before it ends, so no shipped layer carries the LAN address - EPEL/Remi release RPMs come from the cache when it is set - the php.net version gates are untouched Dockerfile.litespeed/.lsphp are not wired yet (see docs/pkg-cache.md). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
shadowdaoandClaude Opus 5.5 committed
1 parent
fa7808897f
commit
b2bec5b3ee
6 files changed
+311
-11
No files matched your search
@@ -0,0 +1,124 @@
|
||||
# Internal package + image cache (`PKG_CACHE`)
|
||||
|
||||
Opt-in build support for the LAN package cache **CT130 `pkg-cache` = `192.168.1.40`**
|
||||
(Proxmox node `pve`, storage `shared-storage-tn`). Without `PKG_CACHE`, builds use
|
||||
the upstream repositories exactly as before.
|
||||
|
||||
Why it exists: during the 2026-10-21 PHP security release, builds failed because
|
||||
Remi mirrors lagged hours behind origin, Remi's origin reset or timed out HTTPS
|
||||
under load, and an EPEL mirror had a broken certificate.
|
||||
|
||||
## What the cache serves
|
||||
|
||||
| URL on the cache | Upstream tiers, tried in order (stale copy last) |
|
||||
|---|---|
|
||||
| `http://192.168.1.40/remi/` | `https://rpms.remirepo.net`, then `http://rpms.remirepo.net` (origin only, no mirrors) |
|
||||
| `http://192.168.1.40/epel/` | `https://dl.fedoraproject.org/pub/epel`, `https://mirror.math.princeton.edu/pub/epel`, `https://mirrors.kernel.org/fedora-epel` |
|
||||
| `http://192.168.1.40/almalinux/` | `https://repo.almalinux.org/almalinux`, `https://mirror.math.princeton.edu/pub/almalinux` |
|
||||
| `http://192.168.1.40/ubuntu/` | `https://archive.ubuntu.com/ubuntu`, `https://us.archive.ubuntu.com/ubuntu` |
|
||||
| `http://192.168.1.40/ubuntu-security/` | `https://security.ubuntu.com/ubuntu`, `https://archive.ubuntu.com/ubuntu` |
|
||||
| `http://192.168.1.40/litespeed/` | `https://rpms.litespeedtech.com`, then `http://rpms.litespeedtech.com` |
|
||||
| `192.168.1.40:5000` (plain http) | Docker Hub pull-through cache (`registry-1.docker.io`) |
|
||||
|
||||
Cache rules:
|
||||
- **Metadata** has a 3-minute TTL. After that it is revalidated upstream (`If-Modified-Since`). Metadata here means `repomd.xml*`, repodata files whose names don't start with a hash, apt `dists/` except `by-hash/`, `*-latest*.rpm`, `remi-release-N.rpm`, `RPM-GPG-KEY*` and directory listings.
|
||||
- **Packages** (`*.rpm`, `*.deb`, hash-named repodata, `by-hash/`) are kept for 30 days, and evicted after 60 days without use. The cache is capped at 110 GB (`min_free` 15 GB).
|
||||
- On a connect error, timeout or 5xx, a request falls through to the next tier. Only the **last** tier answers from a stale copy, so metadata goes primary → fallback → stale. A fresher fallback always wins over stale.
|
||||
- `proxy_cache_lock` is on: concurrent misses on one file make a single upstream fetch.
|
||||
- Each response carries `X-Cache-Status` (`HIT`/`MISS`/`REVALIDATED`/`EXPIRED`/`STALE`/`BYPASS`) and `X-Pkgcache-Tier`. On a HIT, the tier header names the location, not the upstream that filled the entry.
|
||||
|
||||
## Building with the cache
|
||||
|
||||
```bash
|
||||
docker buildx build --build-arg PKG_CACHE=http://192.168.1.40 --build-arg PHPVER=85 -f Dockerfile .
|
||||
```
|
||||
|
||||
`scripts/pkg-cache-repos.sh` (copied to `/usr/local/sbin/`) is wired into
|
||||
`Dockerfile`, `Dockerfile.fpm`, `Dockerfile.shared-httpd` and `Dockerfile.shared-ols`:
|
||||
|
||||
- `rpm`: rewrites the AlmaLinux, EPEL and Remi `baseurl`s to `<cache>/<prefix>/... <origin>/...`, and comments out that section's mirrorlist/metalink. dnf treats the second URL as a failover, so a dead cache falls back to the origin by itself.
|
||||
- `apt`: rewrites the Ubuntu archive/security and LiteSpeed URIs to the cache. apt has no failover, so CI must probe the cache first (see below).
|
||||
- `release-urls`: the EPEL and Remi release RPM URLs, from the cache when it is set.
|
||||
- `restore`: puts the original repo files back. Every RUN that rewrites ends with `restore`, so **no shipped layer contains the cache address**. Images run on datacenter hosts that cannot reach it.
|
||||
|
||||
`Dockerfile.litespeed` and `Dockerfile.lsphp` are **not wired yet**. Their
|
||||
multi-stage `lsphp-apt-upgrade.sh` flow needs two changes first:
|
||||
- a `PKG_CACHE` ARG in each stage;
|
||||
- the madison check accepting the cache URL. It currently requires `*rpms.litespeedtech.com*`, and through the cache `apt-cache madison` shows `http://192.168.1.40/litespeed/debian`.
|
||||
|
||||
The PHP gates are unchanged and still apply: `EXPECT_PHP_MIN`,
|
||||
`EXPECT_LSPHP_PKG`/`EXPECT_PHP_VERSION`.
|
||||
|
||||
## Why plain HTTP on the LAN (trade-off)
|
||||
|
||||
- The cache fetches upstream over **HTTPS with certificate verification**. Remi and LiteSpeed also have a plain-http origin fallback, which mirrors what the Dockerfile already did.
|
||||
- Client to cache is plain HTTP. dnf still verifies **package GPG signatures**, and apt verifies the signed `InRelease`/`Release.gpg` plus package hashes. A LAN attacker therefore cannot inject packages. On 2026-10-21 apt measurably refused the cached LiteSpeed index when given the wrong key (`NO_PUBKEY 011AA62DEDA1F085`).
|
||||
- The residual risk is **metadata replay or freeze**: an attacker on the LAN path, or on the plain-http fallback leg, serves an *old* but validly signed `repomd.xml`. That would let a build install old PHP. Remi/EPEL/AlmaLinux do not enable `repo_gpgcheck`, so dnf does not check metadata signatures. The **php.net version gates** cover this for our images: a build fails if it gets PHP older than php.net's current release.
|
||||
- WHP **hosts** have no such gate. They should keep the origin as the second `baseurl` and not rely on the cache for freshness on security-release day. Run `pkgcache-refresh` on the cache instead.
|
||||
- Signing keys must **not** be fetched through the cache. Take them from the base image or directly over HTTPS.
|
||||
- HTTPS to the cache (internal CA or a Let's Encrypt DNS-01 certificate for `pkg-cache.internal.cloud-hosting.io`) would close the LAN leg. It needs a CA in every build image or a public DNS name plus certificate. Not done; Josh to decide.
|
||||
|
||||
## CI integration plan (NOT applied — needs Josh)
|
||||
|
||||
1. **Network (blocker).** The ephemeral runners (`eph-pve-*` on `pve` vmbr2 10.99.98.0/24, `eph-pve3-*` on `pve3` vmbr1 10.99.99.0/24) are NATed, and `iptables FORWARD` **drops all RFC1918 destinations**, including 192.168.1.40. A narrow exception is needed on each node, placed before the DROP lines in `/etc/network/interfaces.d/*`:
|
||||
```
|
||||
# pve (/etc/network/interfaces.d/ vmbr2 stanza), before the 192.168.0.0/16 DROP:
|
||||
post-up iptables -A FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 80,5000 -j ACCEPT
|
||||
post-down iptables -D FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 80,5000 -j ACCEPT
|
||||
# live, without ifreload:
|
||||
iptables -I FORWARD 1 -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 80,5000 -j ACCEPT
|
||||
# pve3: same with vmbr1
|
||||
```
|
||||
`gitea-runner.internal.cloud-hosting.io` resolves to 10.10.14.20, which is not on this LAN. Whether it can reach 192.168.1.40 is untested.
|
||||
2. **Probe, then opt in** (no `${{ }}` inside `run:`, per repo rule):
|
||||
```yaml
|
||||
- name: Probe package cache
|
||||
id: pkgcache
|
||||
run: |
|
||||
if curl -fsS --max-time 5 -o /dev/null http://192.168.1.40/ ; then
|
||||
echo "url=http://192.168.1.40" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "url=" >> "$GITHUB_OUTPUT"; echo "::warning::pkg-cache unreachable - building from upstream"
|
||||
fi
|
||||
# in docker/build-push-action `build-args:` add
|
||||
# PKG_CACHE=${{ steps.pkgcache.outputs.url }}
|
||||
```
|
||||
3. **Docker Hub mirror for BuildKit.** The workflows use `docker/setup-buildx-action` (docker-container driver). BuildKit **ignores dockerd's `registry-mirrors`**, so configure the builder instead:
|
||||
```yaml
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
buildkitd-config-inline: |
|
||||
[registry."docker.io"]
|
||||
mirrors = ["192.168.1.40:5000"]
|
||||
[registry."192.168.1.40:5000"]
|
||||
http = true
|
||||
```
|
||||
This exact config was used for the test builds on 192.168.1.193. Only Docker Hub images go through it; `repo.anhonesthost.net/...` is unaffected.
|
||||
4. **dockerd clients** (plain `docker pull`, e.g. the build server or test hosts). Untested, because applying it needs a daemon restart. Add to `/etc/docker/daemon.json` and restart docker:
|
||||
```json
|
||||
{ "registry-mirrors": ["http://192.168.1.40:5000"], "insecure-registries": ["192.168.1.40:5000"] }
|
||||
```
|
||||
|
||||
## WHP hosts (plan only)
|
||||
|
||||
The production hosts (whp01/whp02/sdbees) are in the datacenter and **cannot reach 192.168.1.40**. Only LAN hosts such as 192.168.1.148 can. Options for Josh:
|
||||
- (a) a second identical cache CT in the datacenter;
|
||||
- (b) expose this one over NetBird;
|
||||
- (c) LAN hosts only.
|
||||
|
||||
For a host that can reach a cache, use the same idea as `pkg-cache-repos.sh rpm`, keeping the origin second so the cache is never a single point of failure:
|
||||
|
||||
```ini
|
||||
# /etc/yum.repos.d/remi-safe.repo [remi-safe] (AL9: enterprise/9)
|
||||
baseurl=http://<cache>/remi/enterprise/$releasever_major/safe/$basearch/ https://rpms.remirepo.net/enterprise/$releasever_major/safe/$basearch/
|
||||
#mirrorlist=...
|
||||
```
|
||||
|
||||
## Operating the cache (CT130)
|
||||
|
||||
- Config: `/etc/nginx/conf.d/pkgcache-http.conf` (cache zone, maps, log format) and `/etc/nginx/pkgcache/{server-common,meta,pkg}.conf`. `/etc/nginx/sites-available/pkgcache` is **generated** by `/usr/local/sbin/pkgcache-gen-nginx`: edit its `UPSTREAMS` table, then run `pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx`.
|
||||
- Registry: `/etc/docker/registry/config.yml` (Debian `docker-registry` 2.8.3, the same code as `registry:2`), service `docker-registry`.
|
||||
- Data: nginx in `/srv/cache/nginx`, registry in `/srv/cache/registry` (150 GB mount point `mp0`, `backup=0`).
|
||||
- **Security-release day:** run `pkgcache-refresh` to re-fetch every cached metadata entry now (or `pkgcache-refresh remi` for one prefix; `--list` shows them; `--purge` deletes them instead). A refresh bypasses the cache from localhost only and replaces each entry in place. If upstream is down, the old copy stays. Without it, metadata is at most 3 minutes old anyway.
|
||||
- Monitoring: `pkgcache-stats` shows disk, hit ratio, fallback-tier use, STALE count and upstream errors. Logs are in `/var/log/nginx/pkgcache.access.log` (with `cache=`, `tier=`, `up=` fields) and `journalctl -u docker-registry`. Suggested Uptime Kuma checks: `http://192.168.1.40/` and `http://192.168.1.40:5000/v2/`.
|
||||
- Docker Hub: anonymous pulls are rate-limited per egress IP, and every client now shares this cache's IP. To raise the limit, add `username`/`password` (a Hub access token) under `proxy:`.
|
||||
Reference in new issue
Block a user