cloud-apache-container/docs/pkg-cache.md

# Internal package + image cache (`PKG_CACHE`)

Opt-in build support for the LAN package cache **CT130 `pkg-cache` at `https://192.168.1.40`** (Proxmox node `pve`, storage `shared-storage-tn`).

Without `PKG_CACHE`, builds use the upstream repositories. The one change from before is that the EPEL and Remi release RPMs are now **GPG-verified** against `keys/pkg-cache/` before they are installed.

Why it exists: during the 2026-10-21 PHP security release, builds failed because:
- Remi mirrors lagged hours behind origin;
- Remi's origin reset or timed out HTTPS under load;
- an EPEL mirror had a broken certificate.

On that day a build without the cache spent **1031 s** in its first dnf step. Through the cache the same step took **40 s cold and 21 s warm**.

## What the cache serves

| URL (https, pkg-cache CA) | Upstream tiers, tried in order (stale copy last) |
|---|---|
| `https://192.168.1.40/remi/` | `https://rpms.remirepo.net`, then `http://rpms.remirepo.net` (origin only, no mirrors) |
| `https://192.168.1.40/epel/` | `https://dl.fedoraproject.org/pub/epel`, `https://mirror.math.princeton.edu/pub/epel`, `https://mirrors.kernel.org/fedora-epel` |
| `https://192.168.1.40/almalinux/` | `https://repo.almalinux.org/almalinux`, `https://mirror.math.princeton.edu/pub/almalinux` |
| `https://192.168.1.40/ubuntu/` | `https://archive.ubuntu.com/ubuntu`, `https://us.archive.ubuntu.com/ubuntu` |
| `https://192.168.1.40/ubuntu-security/` | `https://security.ubuntu.com/ubuntu`, `https://archive.ubuntu.com/ubuntu` |
| `https://192.168.1.40/litespeed/` | `https://rpms.litespeedtech.com`, then `http://rpms.litespeedtech.com` |
| `192.168.1.40:5000` (TLS, pkg-cache CA) | Docker Hub pull-through cache (`registry-1.docker.io`) |

Plain `http://192.168.1.40/` answers **403** to LAN clients.

Cache rules:
- **Metadata** has a 3-minute TTL. After that it is revalidated upstream (`If-Modified-Since`). Metadata means:
  - `repomd.xml*`
  - repodata files whose names don't start with a hash
  - apt `dists/`, except `by-hash/`
  - `*-latest*.rpm` and `remi-release-N.rpm`
  - `RPM-GPG-KEY*`
  - directory listings
- **Packages** (`*.rpm`, `*.deb`, hash-named repodata, `by-hash/`) are cached for 30 days and evicted after 60 days unused. Total size is capped at 110 GB, with 15 GB kept free.
- On a connect error, timeout or 5xx, the request falls through to the next tier. Only the **last** tier may answer from a stale copy, so the order is primary, then fallback, then stale.
- `proxy_cache_lock` is on.
- Responses carry `X-Cache-Status` and `X-Pkgcache-Tier`.

## Building with the cache

```bash
docker buildx build --build-arg PKG_CACHE=https://192.168.1.40 --build-arg PHPVER=85 -f Dockerfile .
```

`scripts/pkg-cache-repos.sh` (installed as `/usr/local/sbin/`) and `keys/pkg-cache/` (installed as `/usr/local/share/pkg-cache/`) are wired into `Dockerfile`, `Dockerfile.fpm`, `Dockerfile.shared-httpd` and `Dockerfile.shared-ols`. The script has four modes.

**`rpm`** handles AlmaLinux, EPEL and Remi:
- Each `baseurl` becomes `<cache>/<prefix>/... <origin>/...`. dnf fails over to the origin if the cache dies.
- That section's mirrorlist/metalink is commented out.
- The section gets `sslcacert=` pointing at a bundle of the system CAs plus the pkg-cache CA. The bundle lives in the state dir and is deleted by `restore`.
- It also gets `gpgcheck=1`.
- AlmaLinux and Remi also get `repo_gpgcheck=1`, because both sign `repomd.xml` (measured 2026-10-21: `repomd.xml.asc` returns 200). EPEL publishes no repomd signature (404), so its metadata integrity rests on HTTPS end to end.

**`apt`** handles Ubuntu archive/security and LiteSpeed:
- Their URIs are rewritten to the cache.
- `/etc/apt/apt.conf.d/99pkg-cache` sets `Acquire::https::192.168.1.40::CaInfo`, so the CA is trusted for the cache host only.
- apt verifies the signed `InRelease`/`Release.gpg` as always. It has no failover, so CI must probe the cache first.

**`release-install epel [remi]`**:
- Downloads the release RPM over HTTPS, from the cache or from the origin.
- Checks the package name.
- Imports the key committed in `keys/pkg-cache/`.
- Requires `rpm -K` to report `digests signatures OK`.
- Installs with `--setopt=localpkg_gpgcheck=1`.

**`restore`**:
- Puts rewritten files back.
- Deletes files the script created.
- Deletes any repo/apt file that **appeared during the RUN** and still references the cache.
- Then **fails the build** if any file under `/etc/yum.repos.d`, `/etc/dnf` or `/etc/apt` still mentions the cache.

Every RUN that rewrites ends with `restore`, so no shipped layer carries the LAN address. This was verified on the built images: 0 references, no state dir, no apt drop-in.

`PKG_CACHE` must start with `https://`. `http://` is refused.

`Dockerfile.litespeed` and `Dockerfile.lsphp` are **not wired yet**. Two things are needed first:
- the multi-stage `lsphp-apt-upgrade.sh` flow needs a `PKG_CACHE` ARG in each stage;
- its madison check must accept the cache URL. It currently requires `*rpms.litespeedtech.com*`, and via the cache madison shows `https://192.168.1.40/litespeed/debian`.

## Trust model

`keys/pkg-cache/ca.crt` is the *pkg-cache internal CA*:
- EC P-256, valid to 2036.
- **Name-constrained** to IP `192.168.1.40` and DNS `pkg-cache.internal.cloud-hosting.io` / `pkg-cache.geekhome.local`. Even its private key, which exists only on CT130 at `/etc/pkgcache-ca/ca.key`, cannot mint a certificate that clients would accept for any other host.
- It is trusted only by the rewritten repos, the apt per-host option and the BuildKit/dockerd registry config. It is never added to a system trust store.

The server certificate `/etc/pkgcache-ca/server.crt` **expires 2029-10-20**. `pkgcache-stats` prints the expiry. Renew it with the same `openssl x509 -req ... -extfile srv.cnf` steps in `/etc/pkgcache-ca/`.

Remaining risks:
- **Replay of old but validly signed metadata.** Signatures stop tampering, not freshness. The cache's own plain-http tiers for Remi and LiteSpeed are the leg where an internet MITM could replay. The php.net gates (`EXPECT_PHP_MIN`, `EXPECT_LSPHP_PKG`/`EXPECT_PHP_VERSION`) fail an image build that gets old PHP. **Josh to decide** whether to drop those http tiers: you gain replay resistance and lose availability when Remi's HTTPS is down, as it was all day on 2026-10-21.
- **Proposed, not implemented:** the same kind of minimum-version gate for other security-relevant packages:
  - CI resolves the newest `openssl`/`openssl-libs` and `httpd` from AlmaLinux BaseOS/AppStream, and `ImageMagick` from EPEL, **directly from the origin over HTTPS** (e.g. `dnf repoquery --repofrompath=o,https://repo.almalinux.org/almalinux/10/BaseOS/x86_64/os/ --repo=o --latest-limit=1 --qf '%{evr}' openssl-libs`).
  - It passes the results as `EXPECT_OPENSSL_MIN` / `EXPECT_HTTPD_MIN` / `EXPECT_IMAGEMAGICK_MIN`.
  - The Dockerfile asserts `rpm -q --qf '%{EVR}'` is not older (`rpm --eval "%{lua: print(rpm.vercmp('A','B'))}"`).

## CI integration (APPLIED 2026-10-21)

1. **Runner network.** Ephemeral runners are NATed and `iptables FORWARD` drops
   all RFC1918 destinations:
   - `eph-pve-*` on `pve`, vmbr2, 10.99.98.0/24;
   - `eph-pve3-*` on `pve3`, vmbr1, 10.99.99.0/24.

   One rule per node now sits **above** those DROPs. It is persistent through a
   `post-up`/`post-down` pair placed before the first DROP line in
   `/etc/network/interfaces.d/vmbr2-ci-isolated` (pve) and
   `/etc/network/interfaces.d/vmbr1-ci-isolated` (pve3). The Proxmox firewall is
   disabled on both nodes, so its reloads do not touch these chains. Backups:
   `/root/vmbr{2,1}-ci-isolated.bak-pkgcache-20261021`.
   ```
   iptables -I FORWARD 1 -i vmbr2 -o vmbr0 -d 192.168.1.40/32 -p tcp -m multiport --dports 443,5000 \
     -m comment --comment pkg-cache-ct130 -j ACCEPT
   ```
   Verified from throwaway CTs on vmbr2 and vmbr1:
   - :443 and :5000 work with TLS verified;
   - CT130 :22/:80, pve :8006, .193 :22, the router's :53 and 10.10.14.20 stay **blocked**;
   - internet access is unchanged.

   The LAN runner VM *Git-Builders* (VM103, 192.168.1.126 on vmbr0) needs no
   rule. `gitea-runner.internal.cloud-hosting.io` resolves to 10.10.14.20, which
   is unroutable from both the LAN and the DC (stale record?).
2. **Workflow** (`.gitea/workflows/build-push.yaml`). Jobs Build-and-Push,
   Build-FPM-Images, Build-Shared-httpd and Build-Shared-OLS run a
   **"Probe package cache"** step. It curls `https://192.168.1.40/` and
   `https://192.168.1.40:5000/v2/` with `--cacert keys/pkg-cache/ca.crt
   --max-time 5`.
   - If both answer: `PKG_CACHE=https://192.168.1.40` is passed as a build arg,
     and buildx is set up with the Docker Hub mirror (`buildkitd-config-inline`,
     `ca=`).
   - If either fails: that output is empty, the build uses upstream repos /
     Docker Hub, and a `::warning::` is emitted.

   Litespeed and lsphp jobs are unchanged (their Dockerfiles are not wired).
   **Fail-back measured** with the workflow's own step code, from a runner-bridge CT:

   | cache state | probe result | time |
   |---|---|---|
   | up | url + mirror set | <1 s |
   | nginx + registry stopped | both empty, warning | <1 s (connection refused) |
   | silently dropped | both empty, warning | 10 s (2 x 5 s) |

   Mid-build cache loss is also covered: dnf has the origin as a 2nd baseurl, and
   BuildKit falls back from the mirror to Docker Hub (measured with the registry stopped).
3. **dockerd clients** (plain `docker pull`). Untested, because it needs a daemon restart.
   Use TLS, no `insecure-registries`:
   ```
   install -D -m0644 keys/pkg-cache/ca.crt /etc/docker/certs.d/192.168.1.40:5000/ca.crt
   # /etc/docker/daemon.json:  { "registry-mirrors": ["https://192.168.1.40:5000"] }   then restart docker
   ```

## Base-image digest pinning (2026-10-22)

Gap closed (security review): with the BuildKit mirror on, `FROM almalinux/10-base`
was resolved **by tag through the cache**, so a compromised cache could hand out a
different image for that tag.

Now every job that uses the mirror first runs **"Resolve base image digest (Docker
Hub, direct)"**:
- `.gitea/scripts/resolve-base-image.sh IMAGE:TAG` gets an anonymous pull token from
  `auth.docker.io` and sends a `HEAD /v2/<repo>/manifests/<tag>` to
  `registry-1.docker.io` over HTTPS (never the mirror; HEADs don't count against the
  pull rate limit). It prints `IMAGE@sha256:<index digest>`. Input and output are
  regex-validated; values reach `run:` only via `env:`.
- The job passes `BASE_IMAGE=<that>` as a build arg; the Dockerfiles say
  `ARG BASE_IMAGE=<tag>` / `FROM ${BASE_IMAGE}`. The mirror can then only serve
  content-addressed manifests/blobs, which BuildKit verifies against the digest.
- **Lookup fails:** `BASE_IMAGE` falls back to the tag, the probe step does **not**
  enable the registry mirror (`registry-mirror=<not used: Docker Hub direct>`), and
  the tag is pulled straight from Docker Hub, as before the cache existed. The
  package-repo cache (`PKG_CACHE`) is unaffected, since its content is GPG-checked.
- Freshness is automatic: the digest is looked up on every run, nothing is
  hard-coded. Local builds without the arg use the tag.
- Log lines to look for: `base-image: almalinux/10-base:latest -> almalinux/10-base@sha256:...`
  and `pkg-cache: repos=https://192.168.1.40 registry-mirror=192.168.1.40:5000`.

Applied to: Build-and-Push, Build-FPM-Images, Build-Shared-httpd, Build-Shared-OLS
(`litespeedtech/openlitespeed:${OLS_VERSION}-lsphp${PHPVER}`) here, and cloud-node-container.
The script is canonical here (`.gitea/scripts/resolve-base-image.sh`); other repos
carry byte-identical copies.

## WHP hosts (`scripts/host/pkgcache-host-repos.sh`)

`pkgcache-host-repos.sh apply https://192.168.1.40` does the following:
- backs up every `/etc/yum.repos.d/*.repo` to `/root/pkgcache-repo-backup-<UTC>/`;
- installs the name-constrained CA as a system anchor;
- rewrites the AlmaLinux, EPEL and Remi sections to `baseurl=<cache>/... <origin>/...`;
- comments out mirrorlist/metalink in those sections;
- sets `gpgcheck=1`, plus `repo_gpgcheck=1` for AlmaLinux and Remi.

`revert [DIR]` restores byte-identical files (verified on AL9 and AL10) and
removes the anchor. `status` shows the current state.

**Why not "cache first, mirrorlist as fallback":** measured on AL10, with
mirrorlist/metalink left in place dnf **ignores `baseurl` completely** (0 cache
requests). The origin as second baseurl is the fallback dnf really performs.

**Measured on 192.168.1.148** (AlmaLinux 10.2), 2026-10-21:

| test | result |
|---|---|
| baseline `dnf makecache --refresh` (original mirrorlists) | rc=0, **1209 s** |
| via cache `makecache --refresh` | rc=0, **22 s** (2nd run, non-interactive: 2 s) |
| via cache `dnf download php-cli` | rc=0, 1 s |
| `dnf -y --downloadonly update php-cli` | "Nothing to do" (already 8.5.12) |
| cache **down** (nginx stopped): makecache --refresh / download | rc=0 **16 s** / rc=0 24 s, served by origin |
| cache **silently dropped** (iptables DROP): makecache --refresh / download | rc=0 **225 s** / rc=0 59 s (dnf's 30 s connect timeout per attempt) |

**Datacenter hosts** (whp01, whp02, sdbees, TrueSelfCA) cannot reach
192.168.1.40 and are **not configured**. That needs CT130 on NetBird (see the
report: setup key / route pending).

## Operating the cache (CT130)

- **nginx config.**
  - `/etc/nginx/conf.d/pkgcache-http.conf` holds the cache zone, maps and log format.
  - `/etc/nginx/pkgcache/{server-common,meta,pkg}.conf` hold the shared settings.
  - `/etc/nginx/sites-available/pkgcache` is **generated** by `/usr/local/sbin/pkgcache-gen-nginx`. Edit the `UPSTREAMS` table, then run `pkgcache-gen-nginx > /etc/nginx/sites-available/pkgcache && nginx -t && systemctl reload nginx`.
  - Listeners: :443 TLS for the LAN, `127.0.0.1:80` for local tools, and :80 for the LAN answers 403.
- **Registry.** Config is `/etc/docker/registry/config.yml` (Debian `docker-registry` 2.8.3, the same code as `registry:2`, with TLS). Service `docker-registry`.
- **TLS.** Everything is in `/etc/pkgcache-ca/`: `ca.key` (0600), `ca.crt`, `server.key`, `server-chain.crt`, and the `ca.cnf`/`srv.cnf` openssl configs.
- **Data.** nginx uses `/srv/cache/nginx`, the registry uses `/srv/cache/registry`. Both live on the 150 GB mount point `mp0` (`backup=0`).
- **Security-release day.** `pkgcache-refresh` re-fetches every cached metadata entry right away. Variants:
  - `pkgcache-refresh remi` refreshes one prefix;
  - `--list` lists the entries;
  - `--purge` deletes them instead.

  A refresh bypasses the cache from localhost only and replaces each entry in place. If upstream is down, the old copy stays. Metadata is at most 3 minutes old anyway.
- **Monitoring.** `pkgcache-stats` shows:
  - disk usage;
  - hit ratio;
  - fallback-tier use;
  - the STALE count;
  - upstream errors;
  - certificate expiry.

  Logs are `/var/log/nginx/pkgcache.access.log` (fields `cache=`, `tier=`, `up=`) and `journalctl -u docker-registry`. Suggested Uptime Kuma checks are `https://192.168.1.40/` and `https://192.168.1.40:5000/v2/`, with certificate checking against the CA.
- **Docker Hub rate limits.** Anonymous pulls are limited per egress IP, and all clients share it. Add `username`/`password` (a Hub access token) under `proxy:` if needed.