Read a stopped container instead of claiming there is nothing to read
Secret Scan / scan (push) Successful in 4s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 6s
Build App (Preview) / create-release (pull_request) Successful in 1s
Build App (Preview) / build-macos (pull_request) Successful in 2m55s
Build App (Preview) / build-linux (pull_request) Successful in 4m56s
Build App (Preview) / build-windows (pull_request) Successful in 5m53s
Build App (Preview) / prune-previews (pull_request) Successful in 1s

A project that was merely stopped reported "This project has no container
or snapshot image yet, so there is nothing to compare against the base
image" — with its container sitting right there — and Update stayed
disabled. Start it and the checks passed, which is the tell: the staleness
probe had only two sources, a *running* container via `docker exec` or the
project's snapshot image.

The snapshot is not a checkpoint. `commit_container_snapshot` runs only
before a container is destroyed (a config-change recreate) or inside a
migration, never on stop, so a project in daily use for a year can have no
snapshot at all — and five of the six projects on the box that reported
this had none. Absence of a snapshot was being read as absence of anything
to inspect.

So probe the stopped container directly: commit its writable layer to a
throwaway image, probe that, drop it. A stopped container now also outranks
the snapshot, for the same reason a running one already did — the snapshot
lags it by everything installed since the last commit. `pick_probe_source`
is the whole decision and is unit-tested; the message it used to emit now
describes only the case it is true of, no container and no snapshot.

Two things found on the way, both documented in CLAUDE.md:

`bollard` never hands back the image id from a commit — its `Commit` model
deserialises "ID" while the daemon sends "Id" — so the probe image has to be
tagged, and a tagged image is dangling-proof and therefore invisible to
`sweep_orphaned_snapshots`, `reap_stale_migration_pins` and
`scrub_secrets_from_snapshots` alike. Without a reaper of its own a crashed
probe would leak a multi-gigabyte image that nothing could ever reclaim, so
`reap_probe_images` runs at startup beside `reap_probe_containers`, age-gated
for the same reason that one is: `reference=` is daemon-wide and a second
instance's live probe matches the glob.

It removes by tag, never by image id: a force removal by id untags an image
everywhere, which is how a first draft of the reaper test deleted an
unrelated `alpine:latest`. Names are unique per call rather than stable per
container, because container ids do not survive a recreate and two
overlapping probes would otherwise fight over one tag.

Verified against the container that reported the bug: 13,365 paths and an
apt delta of cmake, ffmpeg, libobs-dev, qt6-base-dev and nine more — the
migration payload the Update flow could not see. 546 unit tests plus three
live-Docker tests pass; no new clippy warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz
This commit is contained in:
2026-09-10 19:24:05 -07:00
co-authored by Claude Opus 5
parent 3aec2998d8
commit 307ea07409
5 changed files with 591 additions and 16 deletions
+33
View File
@@ -436,6 +436,39 @@ security update. Migration is the non-destructive way out; Reset is the destruct
bump: churn on the old base, and it would consume the "you should migrate" signal without
migrating. `get_container_staleness` surfaces it; `migrate_project_to_base` acts on it.
- **A missing lineage label means "unknown, probe instead", never "stale".**
- **The snapshot image is not a checkpoint — never read its absence as "nothing to inspect".**
`commit_container_snapshot` runs only before a container is destroyed (a config-change recreate)
or inside a migration. **Never on stop.** So a project in daily use for a year can legitimately
have no `triple-c-snapshot-{id}:latest` at all, and one that has is stale by everything installed
since. `pick_probe_source` therefore reads a *stopped* container directly — commit its writable
layer to `triple-c-probe-{cid}:latest`, probe that, drop it — and ranks it **above** the snapshot,
for the same reason a running container already outranked it. Assuming a snapshot existed is what
made a stopped, never-recreated project report "no container or snapshot image yet" with its
container sitting right there, and left Update disabled on the projects furthest behind.
- **`bollard` never gives you the image id back from a commit.** Its `Commit` response model
deserialises `"ID"`; the daemon sends `"Id"`, so `commit_container` returns `id: None` every time
(verified: bollard 0.18.1, Engine 29.6). Neither long-standing commit site notices because both
discard the response — but it means any commit you need a *reference* to has to be **tagged**.
- **A tagged leftover is the one orphan no sweep can reach, so the probe image has its own reaper.**
`sweep_orphaned_snapshots` collects `dangling` + `triple-c.managed=true`; `reap_stale_migration_pins`
and `scrub_secrets_from_snapshots` both filter `triple-c-snapshot-*`. A `triple-c-probe-*` image is
tagged and so matches none of them, which would make a crashed probe a permanent multi-gigabyte
leak with no UI to find it. `reap_probe_images` runs at startup beside `reap_probe_containers` and
is **load-bearing, not tidying** — it is also what makes the probe image's unscrubbed writable
layer acceptable. Two rules it earned the hard way:
- **Age-gate it** (`PROBE_REAP_MIN_AGE_SECS`, same as the container reaper). `reference=` is
daemon-wide, so a second copy of the app has live probe images matching the glob.
- **Remove by tag, never by image id.** A `force` removal by id untags an image *everywhere*; a
fixture that tagged `alpine:latest` into this namespace deleted the user's alpine that way.
- **Probe image names are unique per call, and must stay that way.** A stable per-container name was
tried: container ids do not survive a recreate, so most leftovers were stranded permanently, and
two concurrent probes fought over one tag — whichever finished first force-removed the image the
other was still reading, reporting a bogus `probe_error` on a healthy project. `get_container_staleness`
takes no `project_lock` claim (the migration banner needs it to answer *during* a migration), so
uniqueness is what makes overlapping probes safe.
- **An image's `Created` is the image's own, not its tag's.** Tagging an existing image gives you
that image's age; BuildKit stamps `docker build` output with a fixed epoch. Only `docker commit`
stamps *now* — which is what real probe images do, and what any fixture for them must do.
- **`:latest` keeps pointing at the old lineage until the final commit.** That is what makes every
crash before that point self-heal — `start_project_container` just recreates from the old
snapshot. After the container swap, the new container's `triple-c.migration-state=in-progress`