cloud-hosting-platform/cloud-apache-container
6
0 Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

ci: enable PKG_CACHE via a TLS probe that falls back to upstream; host repo tool

Browse Source 
- build-push.yaml: Build-and-Push, Build-FPM-Images, Build-Shared-httpd
  and Build-Shared-OLS probe https://192.168.1.40 and :5000 with the
  pinned pkg-cache CA (5 s timeouts). When the cache answers they pass
  PKG_CACHE and configure the BuildKit Docker Hub mirror (ca=). When it
  doesn't, they build from upstream exactly as before and emit a warning.
  Fallback measured from the runner bridge: refused <1 s, silent drop 10 s.
- scripts/host/pkgcache-host-repos.sh: apply/revert/status for WHP hosts.
  Each rewritten repo uses baseurl=<cache> <origin> as the real fallback;
  mirrorlist/metalink stay commented because measured dnf ignores baseurl
  when they are present. Keeps gpgcheck=1 / repo_gpgcheck=1 and backs up
  every file. Tested on 192.168.1.148, including cache-down and
  silent-drop fallback.
- docs: runner firewall rule (applied), measured fallback results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
shadowdaoandClaude Opus 5.5 committed 2026-10-21 16:46:03 -07:00
1 parent 18f65c6d84
commit 2b9e7f302c
3 files changed
+301 -51
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/pkg-cache.md
+69 -51
View File
@@ -95,72 +95,90 @@ Remaining risks:
- It passes the results as `EXPECT_OPENSSL_MIN` / `EXPECT_HTTPD_MIN` / `EXPECT_IMAGEMAGICK_MIN`.
- The Dockerfile asserts `rpm -q --qf '%{EVR}'` is not older (`rpm --eval "%{lua: print(rpm.vercmp('A','B'))}"`).
## CI integration plan (NOT applied — needs Josh)
## CI integration (APPLIED 2026-10-21)
1. **Network (blocker).** The ephemeral runners are NATed, and `iptables FORWARD` **drops all RFC1918 destinations**, including 192.168.1.40:
1. **Runner network.** Ephemeral runners are NATed and `iptables FORWARD` drops
all RFC1918 destinations:
- `eph-pve-*` on `pve`, vmbr2, 10.99.98.0/24;
- `eph-pve3-*` on `pve3`, vmbr1, 10.99.99.0/24.
A narrow exception is needed, placed before the DROP lines:
One rule per node now sits **above** those DROPs. It is persistent through a
`post-up`/`post-down` pair placed before the first DROP line in
`/etc/network/interfaces.d/vmbr2-ci-isolated` (pve) and
`/etc/network/interfaces.d/vmbr1-ci-isolated` (pve3). The Proxmox firewall is
disabled on both nodes, so its reloads do not touch these chains. Backups:
`/root/vmbr{2,1}-ci-isolated.bak-pkgcache-20261021`.
```
# pve, /etc/network/interfaces.d/ vmbr2 stanza, BEFORE the 192.168.0.0/16 DROP:
post-up iptables -A FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 443,5000 -j ACCEPT
post-down iptables -D FORWARD -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 443,5000 -j ACCEPT
# live: iptables -I FORWARD 1 -i vmbr2 -o vmbr0 -d 192.168.1.40 -p tcp -m multiport --dports 443,5000 -j ACCEPT
# pve3: the same with vmbr1
iptables -I FORWARD 1 -i vmbr2 -o vmbr0 -d 192.168.1.40/32 -p tcp -m multiport --dports 443,5000 \
-m comment --comment pkg-cache-ct130 -j ACCEPT
```
`gitea-runner.internal.cloud-hosting.io` resolves to 10.10.14.20, which is not on this LAN. Whether it can reach the cache is untested.
2. **Probe with the CA, then opt in.** Keep `${{ }}` out of `run:`. The probe fails when the cache is down **or its certificate is expired or untrusted**, and the build then uses upstream.
```yaml
- name: Probe package cache
id: pkgcache
run: |
if curl -fsS --max-time 5 --cacert keys/pkg-cache/ca.crt -o /dev/null https://192.168.1.40/ ; then
echo "url=https://192.168.1.40" >> "$GITHUB_OUTPUT"
else
echo "url=" >> "$GITHUB_OUTPUT"; echo "::warning::pkg-cache unreachable - building from upstream"
fi
# docker/build-push-action `build-args:` add PKG_CACHE=${{ steps.pkgcache.outputs.url }}
```
3. **Docker Hub mirror for BuildKit**, over TLS and without `insecure`. `setup-buildx-action` uses the docker-container driver, which **ignores dockerd's `registry-mirrors`**. buildx copies the `ca` file into the builder.
```yaml
- uses: docker/setup-buildx-action@v3
with:
buildkitd-config-inline: |
[registry."docker.io"]
mirrors = ["192.168.1.40:5000"]
[registry."192.168.1.40:5000"]
ca = ["keys/pkg-cache/ca.crt"]
```
The exact equivalent (absolute `ca` path) was measured on 192.168.1.193. BuildKit falls back to Docker Hub when the mirror is down; this was measured too, with the registry stopped. The relative `ca` path inside a workflow is **untested**; use `${{ github.workspace }}/keys/pkg-cache/ca.crt` if it doesn't resolve.
4. **dockerd clients** (plain `docker pull`). Untested, because it needs a daemon restart. Use TLS, no `insecure-registries`:
Verified from throwaway CTs on vmbr2 and vmbr1:
- :443 and :5000 work with TLS verified;
- CT130 :22/:80, pve :8006, .193 :22, the router's :53 and 10.10.14.20 stay **blocked**;
- internet access is unchanged.
The LAN runner VM *Git-Builders* (VM103, 192.168.1.126 on vmbr0) needs no
rule. `gitea-runner.internal.cloud-hosting.io` resolves to 10.10.14.20, which
is unroutable from both the LAN and the DC (stale record?).
2. **Workflow** (`.gitea/workflows/build-push.yaml`). Jobs Build-and-Push,
Build-FPM-Images, Build-Shared-httpd and Build-Shared-OLS run a
**"Probe package cache"** step. It curls `https://192.168.1.40/` and
`https://192.168.1.40:5000/v2/` with `--cacert keys/pkg-cache/ca.crt
--max-time 5`.
- If both answer: `PKG_CACHE=https://192.168.1.40` is passed as a build arg,
and buildx is set up with the Docker Hub mirror (`buildkitd-config-inline`,
`ca=`).
- If either fails: that output is empty, the build uses upstream repos /
Docker Hub, and a `::warning::` is emitted.
Litespeed and lsphp jobs are unchanged (their Dockerfiles are not wired).
**Fail-back measured** with the workflow's own step code, from a runner-bridge CT:
| cache state | probe result | time |
|---|---|---|
| up | url + mirror set | <1 s |
| nginx + registry stopped | both empty, warning | <1 s (connection refused) |
| silently dropped | both empty, warning | 10 s (2 x 5 s) |
Mid-build cache loss is also covered: dnf has the origin as a 2nd baseurl, and
BuildKit falls back from the mirror to Docker Hub (measured with the registry stopped).
3. **dockerd clients** (plain `docker pull`). Untested, because it needs a daemon restart.
Use TLS, no `insecure-registries`:
```
install -D -m0644 keys/pkg-cache/ca.crt /etc/docker/certs.d/192.168.1.40:5000/ca.crt
# /etc/docker/daemon.json: { "registry-mirrors": ["https://192.168.1.40:5000"] } then restart docker
```
## WHP hosts (plan only)
## WHP hosts (`scripts/host/pkgcache-host-repos.sh`)
The production hosts (whp01, whp02, sdbees) are in the datacenter and **cannot reach 192.168.1.40**. Only LAN hosts can, e.g. 192.168.1.148. Options for Josh:
- (a) a second identical cache in the datacenter, with its own name-constrained CA;
- (b) expose this one over NetBird;
- (c) LAN hosts only.
`pkgcache-host-repos.sh apply https://192.168.1.40` does the following:
- backs up every `/etc/yum.repos.d/*.repo` to `/root/pkgcache-repo-backup-<UTC>/`;
- installs the name-constrained CA as a system anchor;
- rewrites the AlmaLinux, EPEL and Remi sections to `baseurl=<cache>/... <origin>/...`;
- comments out mirrorlist/metalink in those sections;
- sets `gpgcheck=1`, plus `repo_gpgcheck=1` for AlmaLinux and Remi.
For a host that can reach a cache:
- install the CA at `/etc/pki/pkg-cache/ca.crt`;
- create a bundle (`cat /etc/pki/tls/certs/ca-bundle.crt /etc/pki/pkg-cache/ca.crt > /etc/pki/pkg-cache/bundle.pem`);
- in each repo section, keep the origin as the second URL:
`revert [DIR]` restores byte-identical files (verified on AL9 and AL10) and
removes the anchor. `status` shows the current state.
```ini
[remi-safe]
baseurl=https://<cache>/remi/enterprise/$releasever_major/safe/$basearch/ https://rpms.remirepo.net/enterprise/$releasever_major/safe/$basearch/
#mirrorlist=...
sslcacert=/etc/pki/pkg-cache/bundle.pem
gpgcheck=1
repo_gpgcheck=1 # Remi + AlmaLinux sign repomd; EPEL does not
```
**Why not "cache first, mirrorlist as fallback":** measured on AL10, with
mirrorlist/metalink left in place dnf **ignores `baseurl` completely** (0 cache
requests). The origin as second baseurl is the fallback dnf really performs.
Hosts have no php.net gate. On a security-release day, run `pkgcache-refresh` on the cache and check `dnf info php-cli` against php.net.
**Measured on 192.168.1.148** (AlmaLinux 10.2), 2026-10-21:
| test | result |
|---|---|
| baseline `dnf makecache --refresh` (original mirrorlists) | rc=0, **1209 s** |
| via cache `makecache --refresh` | rc=0, **22 s** (2nd run, non-interactive: 2 s) |
| via cache `dnf download php-cli` | rc=0, 1 s |
| `dnf -y --downloadonly update php-cli` | "Nothing to do" (already 8.5.12) |
| cache **down** (nginx stopped): makecache --refresh / download | rc=0 **16 s** / rc=0 24 s, served by origin |
| cache **silently dropped** (iptables DROP): makecache --refresh / download | rc=0 **225 s** / rc=0 59 s (dnf's 30 s connect timeout per attempt) |
**Datacenter hosts** (whp01, whp02, sdbees, TrueSelfCA) cannot reach
192.168.1.40 and are **not configured**. That needs CT130 on NetBird (see the
report: setup key / route pending).
## Operating the cache (CT130)