This project has no container or snapshot image yet, so there is nothing to compare against the base image.
— with its container sitting right there — and Update stayed disabled. Starting it made the checks pass, which is the tell: the staleness probe had only two sources, a running container via docker exec, or the project's snapshot image.
Root cause
The snapshot image is not a checkpoint.commit_container_snapshot runs only before a container is destroyed (a config-change recreate) or inside a migration — never on stop. So a project in daily use for a year can legitimately have no snapshot at all, and five of the six projects on the reporting box had none. Absence of a snapshot was being read as absence of anything to inspect.
The base-image update that prompted this only made it visible: the new base left every un-recreated container stale, so the banner started asking a question the probe had never been able to answer.
The fix
manifest_from_stopped_container commits the stopped container's writable layer to a throwaway image, probes it, and drops it.
A stopped container now outranks the snapshot, for the same reason a running one already did — the snapshot lags it by everything installed since the last commit.
pick_probe_source is the whole decision, pure and unit-tested. The old message now describes only the case it is true of: no container and no snapshot.
Two things found on the way
bollard never hands back the image id from a commit. Its Commit response model deserialises "ID"; the daemon sends "Id", so commit_container returns id: None every time. Neither pre-existing commit site notices, because both discard the response. That forces the probe image to be tagged — and a tagged image is not dangling, so it is invisible to sweep_orphaned_snapshots, reap_stale_migration_pins and scrub_secrets_from_snapshots alike. Without a reaper of its own, a crashed probe would leak a multi-gigabyte image that nothing in the tree could ever reclaim. reap_probe_images now runs at startup beside reap_probe_containers, age-gated for the same reason that one is: reference= is daemon-wide and a second instance's live probe matches the glob.
It removes by tag, never by image id. A force removal by id untags an image everywhere — which is how the first draft of the reaper test deleted an unrelated alpine:latest. Names are unique per call rather than stable per container, because container ids do not survive a recreate and two overlapping probes would otherwise fight over one tag (get_container_staleness takes no project_lock claim, by design — the banner has to answer during a migration).
Verification
Against the container that reported the bug: 13,365 paths and an apt delta of cmake, ffmpeg, g++-mingw-w64-x86-64, gdb, libcurl4-openssl-dev, libobs-dev, ninja-build, p7zip-full, python3-yaml, qt6-base-dev, shellcheck, whois, zip — the migration payload the Update flow could not see.
Every new test was mutation-checked: the cleanup, the age gate and the name helper were each broken in turn and the corresponding test confirmed failing. The first draft's leak assertion was vacuous (it asserted on the dangling set, which a tagged image never joins) and now asserts on the probe tags.
Not addressed
find_existing_container(...).unwrap_or(None) still maps a daemon error to "no container", so an unreachable daemon produces the same misleading message. Pre-existing, and fixing it means threading error propagation through that function — deliberately left for its own change.
A project that was merely stopped reported
> This project has no container or snapshot image yet, so there is nothing to compare against the base image.
— with its container sitting right there — and Update stayed disabled. Starting it made the checks pass, which is the tell: the staleness probe had only two sources, a **running** container via `docker exec`, or the project's snapshot image.
## Root cause
**The snapshot image is not a checkpoint.** `commit_container_snapshot` runs only before a container is destroyed (a config-change recreate) or inside a migration — *never on stop*. So a project in daily use for a year can legitimately have no snapshot at all, and five of the six projects on the reporting box had none. Absence of a snapshot was being read as absence of anything to inspect.
The base-image update that prompted this only made it visible: the new base left every un-recreated container stale, so the banner started asking a question the probe had never been able to answer.
## The fix
- `manifest_from_stopped_container` commits the stopped container's writable layer to a throwaway image, probes it, and drops it.
- A stopped container now **outranks** the snapshot, for the same reason a running one already did — the snapshot lags it by everything installed since the last commit.
- `pick_probe_source` is the whole decision, pure and unit-tested. The old message now describes only the case it is true of: no container *and* no snapshot.
## Two things found on the way
**`bollard` never hands back the image id from a commit.** Its `Commit` response model deserialises `"ID"`; the daemon sends `"Id"`, so `commit_container` returns `id: None` every time. Neither pre-existing commit site notices, because both discard the response. That forces the probe image to be **tagged** — and a tagged image is not dangling, so it is invisible to `sweep_orphaned_snapshots`, `reap_stale_migration_pins` and `scrub_secrets_from_snapshots` alike. Without a reaper of its own, a crashed probe would leak a multi-gigabyte image that nothing in the tree could ever reclaim. `reap_probe_images` now runs at startup beside `reap_probe_containers`, age-gated for the same reason that one is: `reference=` is daemon-wide and a second instance's live probe matches the glob.
**It removes by tag, never by image id.** A `force` removal by id untags an image *everywhere* — which is how the first draft of the reaper test deleted an unrelated `alpine:latest`. Names are unique per call rather than stable per container, because container ids do not survive a recreate and two overlapping probes would otherwise fight over one tag (`get_container_staleness` takes no `project_lock` claim, by design — the banner has to answer during a migration).
## Verification
Against the container that reported the bug: **13,365 paths** and an apt delta of `cmake, ffmpeg, g++-mingw-w64-x86-64, gdb, libcurl4-openssl-dev, libobs-dev, ninja-build, p7zip-full, python3-yaml, qt6-base-dev, shellcheck, whois, zip` — the migration payload the Update flow could not see.
- 546 unit tests pass.
- 3 `--ignored` live-Docker tests pass (2 new, 1 pre-existing).
- Clippy: 44 warnings before, 44 after — zero new.
- Every new test was mutation-checked: the cleanup, the age gate and the name helper were each broken in turn and the corresponding test confirmed failing. The first draft's leak assertion was vacuous (it asserted on the dangling set, which a tagged image never joins) and now asserts on the probe tags.
## Not addressed
`find_existing_container(...).unwrap_or(None)` still maps a daemon error to "no container", so an unreachable daemon produces the same misleading message. Pre-existing, and fixing it means threading error propagation through that function — deliberately left for its own change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz
A project that was merely stopped reported "This project has no container
or snapshot image yet, so there is nothing to compare against the base
image" — with its container sitting right there — and Update stayed
disabled. Start it and the checks passed, which is the tell: the staleness
probe had only two sources, a *running* container via `docker exec` or the
project's snapshot image.
The snapshot is not a checkpoint. `commit_container_snapshot` runs only
before a container is destroyed (a config-change recreate) or inside a
migration, never on stop, so a project in daily use for a year can have no
snapshot at all — and five of the six projects on the box that reported
this had none. Absence of a snapshot was being read as absence of anything
to inspect.
So probe the stopped container directly: commit its writable layer to a
throwaway image, probe that, drop it. A stopped container now also outranks
the snapshot, for the same reason a running one already did — the snapshot
lags it by everything installed since the last commit. `pick_probe_source`
is the whole decision and is unit-tested; the message it used to emit now
describes only the case it is true of, no container and no snapshot.
Two things found on the way, both documented in CLAUDE.md:
`bollard` never hands back the image id from a commit — its `Commit` model
deserialises "ID" while the daemon sends "Id" — so the probe image has to be
tagged, and a tagged image is dangling-proof and therefore invisible to
`sweep_orphaned_snapshots`, `reap_stale_migration_pins` and
`scrub_secrets_from_snapshots` alike. Without a reaper of its own a crashed
probe would leak a multi-gigabyte image that nothing could ever reclaim, so
`reap_probe_images` runs at startup beside `reap_probe_containers`, age-gated
for the same reason that one is: `reference=` is daemon-wide and a second
instance's live probe matches the glob.
It removes by tag, never by image id: a force removal by id untags an image
everywhere, which is how a first draft of the reaper test deleted an
unrelated `alpine:latest`. Names are unique per call rather than stable per
container, because container ids do not survive a recreate and two
overlapping probes would otherwise fight over one tag.
Verified against the container that reported the bug: 13,365 paths and an
apt delta of cmake, ffmpeg, libobs-dev, qt6-base-dev and nine more — the
migration payload the Update flow could not see. 546 unit tests plus three
live-Docker tests pass; no new clippy warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz
Six findings, all real. The one that mattered: `getContainerStaleness` is
called from a `useEffect` that fires whenever the container settles, so
merely opening a stopped project's Overview now committed its whole writable
layer — 44 s on a real project, against ~3 s for the snapshot probe it
replaced. Shipping that would have traded one bad banner for a bad page.
A stopped container's writable layer cannot change, so the probe is exactly
cacheable: `STOPPED_MANIFEST_CACHE` keys on the container's `FinishedAt`,
which moves on every stop. Cold 2967 ms, warm 1 ms, measured. A live test
asserts the restart case as well as the hit, because a cache that failed to
invalidate would plan a migration against a filesystem the project no longer
has — verified by breaking the token and watching that assertion fail.
Skipping the probe for projects that are not stale looked like the cheaper
fix and is unsafe: the deltas would be empty while `probeSettled` stayed
true, and the migrate action in the project menu is not gated on the banner,
so the pre-flight would report nothing to copy while the backend was told to
copy nothing. That is the hazard `canMigrate`'s comment already warns about.
Not done, and written down so it is not tried again.
Also from the review:
- A failed commit no longer costs an answer the snapshot could have given.
Before this feature a stopped project read its snapshot directly, so
surfacing this error would have made the banner worse than it was — and
the failure modes are where the fallback earns its keep: a full disk (the
commit allocates the whole layer, the snapshot probe allocates nothing)
and a 409 from a concurrent claim.
- The probe no longer commits while the project is claimed. The collision is
not symmetric: the probe losing is a retryable `probe_error`, but
`start_project_container` removes the old container with a hard `?`, so a
remove that raced a commit would fail the user's Start with an opaque
error. `stopped_probe_policy` reads `project_lock::held` and probes the
snapshot instead, or defers with a message that says so.
- The cleanup-failure warning claimed the next probe of the same container
would reclaim the leftover. Unique names made that false the moment they
landed; it is `reap_probe_images` that collects it.
- The TS binding still called the command read-only, which is how the
auto-refresh got added in the first place.
- CLAUDE.md still documented the stable `triple-c-probe-{cid}:latest` name
this PR removed as unsafe.
548 unit tests, 752 frontend tests, 4 live-Docker tests. Clippy unchanged at
44 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz
jknapp
merged commit b59c6148ff into main2026-09-11 03:54:40 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
A project that was merely stopped reported
— with its container sitting right there — and Update stayed disabled. Starting it made the checks pass, which is the tell: the staleness probe had only two sources, a running container via
docker exec, or the project's snapshot image.Root cause
The snapshot image is not a checkpoint.
commit_container_snapshotruns only before a container is destroyed (a config-change recreate) or inside a migration — never on stop. So a project in daily use for a year can legitimately have no snapshot at all, and five of the six projects on the reporting box had none. Absence of a snapshot was being read as absence of anything to inspect.The base-image update that prompted this only made it visible: the new base left every un-recreated container stale, so the banner started asking a question the probe had never been able to answer.
The fix
manifest_from_stopped_containercommits the stopped container's writable layer to a throwaway image, probes it, and drops it.pick_probe_sourceis the whole decision, pure and unit-tested. The old message now describes only the case it is true of: no container and no snapshot.Two things found on the way
bollardnever hands back the image id from a commit. ItsCommitresponse model deserialises"ID"; the daemon sends"Id", socommit_containerreturnsid: Noneevery time. Neither pre-existing commit site notices, because both discard the response. That forces the probe image to be tagged — and a tagged image is not dangling, so it is invisible tosweep_orphaned_snapshots,reap_stale_migration_pinsandscrub_secrets_from_snapshotsalike. Without a reaper of its own, a crashed probe would leak a multi-gigabyte image that nothing in the tree could ever reclaim.reap_probe_imagesnow runs at startup besidereap_probe_containers, age-gated for the same reason that one is:reference=is daemon-wide and a second instance's live probe matches the glob.It removes by tag, never by image id. A
forceremoval by id untags an image everywhere — which is how the first draft of the reaper test deleted an unrelatedalpine:latest. Names are unique per call rather than stable per container, because container ids do not survive a recreate and two overlapping probes would otherwise fight over one tag (get_container_stalenesstakes noproject_lockclaim, by design — the banner has to answer during a migration).Verification
Against the container that reported the bug: 13,365 paths and an apt delta of
cmake, ffmpeg, g++-mingw-w64-x86-64, gdb, libcurl4-openssl-dev, libobs-dev, ninja-build, p7zip-full, python3-yaml, qt6-base-dev, shellcheck, whois, zip— the migration payload the Update flow could not see.--ignoredlive-Docker tests pass (2 new, 1 pre-existing).Not addressed
find_existing_container(...).unwrap_or(None)still maps a daemon error to "no container", so an unreachable daemon produces the same misleading message. Pre-existing, and fixing it means threading error propagation through that function — deliberately left for its own change.🤖 Generated with Claude Code
https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz
Six findings, all real. The one that mattered: `getContainerStaleness` is called from a `useEffect` that fires whenever the container settles, so merely opening a stopped project's Overview now committed its whole writable layer — 44 s on a real project, against ~3 s for the snapshot probe it replaced. Shipping that would have traded one bad banner for a bad page. A stopped container's writable layer cannot change, so the probe is exactly cacheable: `STOPPED_MANIFEST_CACHE` keys on the container's `FinishedAt`, which moves on every stop. Cold 2967 ms, warm 1 ms, measured. A live test asserts the restart case as well as the hit, because a cache that failed to invalidate would plan a migration against a filesystem the project no longer has — verified by breaking the token and watching that assertion fail. Skipping the probe for projects that are not stale looked like the cheaper fix and is unsafe: the deltas would be empty while `probeSettled` stayed true, and the migrate action in the project menu is not gated on the banner, so the pre-flight would report nothing to copy while the backend was told to copy nothing. That is the hazard `canMigrate`'s comment already warns about. Not done, and written down so it is not tried again. Also from the review: - A failed commit no longer costs an answer the snapshot could have given. Before this feature a stopped project read its snapshot directly, so surfacing this error would have made the banner worse than it was — and the failure modes are where the fallback earns its keep: a full disk (the commit allocates the whole layer, the snapshot probe allocates nothing) and a 409 from a concurrent claim. - The probe no longer commits while the project is claimed. The collision is not symmetric: the probe losing is a retryable `probe_error`, but `start_project_container` removes the old container with a hard `?`, so a remove that raced a commit would fail the user's Start with an opaque error. `stopped_probe_policy` reads `project_lock::held` and probes the snapshot instead, or defers with a message that says so. - The cleanup-failure warning claimed the next probe of the same container would reclaim the leftover. Unique names made that false the moment they landed; it is `reap_probe_images` that collects it. - The TS binding still called the command read-only, which is how the auto-refresh got added in the first place. - CLAUDE.md still documented the stable `triple-c-probe-{cid}:latest` name this PR removed as unsafe. 548 unit tests, 752 frontend tests, 4 live-Docker tests. Clippy unchanged at 44 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz