Compare commits

..
Author SHA1 Message Date
jknapp b59c6148ff Merge pull request 'Read a stopped container instead of claiming there is nothing to read' (#55) from fix/staleness-probe-stopped-container into main
Build App / compute-version (push) Successful in 4s
Secret Scan / scan (push) Successful in 4s
Build App / build-macos (push) Successful in 3m29s
Build App / build-windows (push) Successful in 5m1s
Build App / build-linux (push) Successful in 5m10s
Build App / create-tag (push) Successful in 7s
Build App / sync-to-github (push) Successful in 1m36s
2026-09-11 03:54:40 +00:00
shadowdaoandClaude Opus 5 95a78fe9a3 Take the review: cache the stopped probe, and never let it cost an answer
Secret Scan / scan (push) Successful in 6s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 4s
Build App (Preview) / create-release (pull_request) Successful in 1s
Build App (Preview) / build-macos (pull_request) Successful in 2m58s
Build App (Preview) / build-linux (pull_request) Successful in 4m43s
Build App (Preview) / build-windows (pull_request) Successful in 5m9s
Build App (Preview) / prune-previews (pull_request) Successful in 2s
Six findings, all real. The one that mattered: `getContainerStaleness` is
called from a `useEffect` that fires whenever the container settles, so
merely opening a stopped project's Overview now committed its whole writable
layer — 44 s on a real project, against ~3 s for the snapshot probe it
replaced. Shipping that would have traded one bad banner for a bad page.

A stopped container's writable layer cannot change, so the probe is exactly
cacheable: `STOPPED_MANIFEST_CACHE` keys on the container's `FinishedAt`,
which moves on every stop. Cold 2967 ms, warm 1 ms, measured. A live test
asserts the restart case as well as the hit, because a cache that failed to
invalidate would plan a migration against a filesystem the project no longer
has — verified by breaking the token and watching that assertion fail.

Skipping the probe for projects that are not stale looked like the cheaper
fix and is unsafe: the deltas would be empty while `probeSettled` stayed
true, and the migrate action in the project menu is not gated on the banner,
so the pre-flight would report nothing to copy while the backend was told to
copy nothing. That is the hazard `canMigrate`'s comment already warns about.
Not done, and written down so it is not tried again.

Also from the review:

- A failed commit no longer costs an answer the snapshot could have given.
  Before this feature a stopped project read its snapshot directly, so
  surfacing this error would have made the banner worse than it was — and
  the failure modes are where the fallback earns its keep: a full disk (the
  commit allocates the whole layer, the snapshot probe allocates nothing)
  and a 409 from a concurrent claim.
- The probe no longer commits while the project is claimed. The collision is
  not symmetric: the probe losing is a retryable `probe_error`, but
  `start_project_container` removes the old container with a hard `?`, so a
  remove that raced a commit would fail the user's Start with an opaque
  error. `stopped_probe_policy` reads `project_lock::held` and probes the
  snapshot instead, or defers with a message that says so.
- The cleanup-failure warning claimed the next probe of the same container
  would reclaim the leftover. Unique names made that false the moment they
  landed; it is `reap_probe_images` that collects it.
- The TS binding still called the command read-only, which is how the
  auto-refresh got added in the first place.
- CLAUDE.md still documented the stable `triple-c-probe-{cid}:latest` name
  this PR removed as unsafe.

548 unit tests, 752 frontend tests, 4 live-Docker tests. Clippy unchanged at
44 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz
2026-09-10 20:48:10 -07:00
shadowdaoandClaude Opus 5 307ea07409 Read a stopped container instead of claiming there is nothing to read
Secret Scan / scan (push) Successful in 4s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 6s
Build App (Preview) / create-release (pull_request) Successful in 1s
Build App (Preview) / build-macos (pull_request) Successful in 2m55s
Build App (Preview) / build-linux (pull_request) Successful in 4m56s
Build App (Preview) / build-windows (pull_request) Successful in 5m53s
Build App (Preview) / prune-previews (pull_request) Successful in 1s
A project that was merely stopped reported "This project has no container
or snapshot image yet, so there is nothing to compare against the base
image" — with its container sitting right there — and Update stayed
disabled. Start it and the checks passed, which is the tell: the staleness
probe had only two sources, a *running* container via `docker exec` or the
project's snapshot image.

The snapshot is not a checkpoint. `commit_container_snapshot` runs only
before a container is destroyed (a config-change recreate) or inside a
migration, never on stop, so a project in daily use for a year can have no
snapshot at all — and five of the six projects on the box that reported
this had none. Absence of a snapshot was being read as absence of anything
to inspect.

So probe the stopped container directly: commit its writable layer to a
throwaway image, probe that, drop it. A stopped container now also outranks
the snapshot, for the same reason a running one already did — the snapshot
lags it by everything installed since the last commit. `pick_probe_source`
is the whole decision and is unit-tested; the message it used to emit now
describes only the case it is true of, no container and no snapshot.

Two things found on the way, both documented in CLAUDE.md:

`bollard` never hands back the image id from a commit — its `Commit` model
deserialises "ID" while the daemon sends "Id" — so the probe image has to be
tagged, and a tagged image is dangling-proof and therefore invisible to
`sweep_orphaned_snapshots`, `reap_stale_migration_pins` and
`scrub_secrets_from_snapshots` alike. Without a reaper of its own a crashed
probe would leak a multi-gigabyte image that nothing could ever reclaim, so
`reap_probe_images` runs at startup beside `reap_probe_containers`, age-gated
for the same reason that one is: `reference=` is daemon-wide and a second
instance's live probe matches the glob.

It removes by tag, never by image id: a force removal by id untags an image
everywhere, which is how a first draft of the reaper test deleted an
unrelated `alpine:latest`. Names are unique per call rather than stable per
container, because container ids do not survive a recreate and two
overlapping probes would otherwise fight over one tag.

Verified against the container that reported the bug: 13,365 paths and an
apt delta of cmake, ffmpeg, libobs-dev, qt6-base-dev and nine more — the
migration payload the Update flow could not see. 546 unit tests plus three
live-Docker tests pass; no new clippy warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz
2026-09-10 19:24:05 -07:00
6 changed files with 895 additions and 19 deletions
+57
View File
@@ -456,6 +456,63 @@ security update. Migration is the non-destructive way out; Reset is the destruct
bump: churn on the old base, and it would consume the "you should migrate" signal without
migrating. `get_container_staleness` surfaces it; `migrate_project_to_base` acts on it.
- **A missing lineage label means "unknown, probe instead", never "stale".**
- **The snapshot image is not a checkpoint — never read its absence as "nothing to inspect".**
`commit_container_snapshot` runs only before a container is destroyed (a config-change recreate)
or inside a migration. **Never on stop.** So a project in daily use for a year can legitimately
have no `triple-c-snapshot-{id}:latest` at all, and one that has is stale by everything installed
since. `pick_probe_source` therefore reads a *stopped* container directly — commit its writable
layer to a unique `triple-c-probe-*` image, probe that, drop it — and ranks it **above** the snapshot,
for the same reason a running container already outranked it. Assuming a snapshot existed is what
made a stopped, never-recreated project report "no container or snapshot image yet" with its
container sitting right there, and left Update disabled on the projects furthest behind.
- **`bollard` never gives you the image id back from a commit.** Its `Commit` response model
deserialises `"ID"`; the daemon sends `"Id"`, so `commit_container` returns `id: None` every time
(verified: bollard 0.18.1, Engine 29.6). Neither long-standing commit site notices because both
discard the response — but it means any commit you need a *reference* to has to be **tagged**.
- **A tagged leftover is the one orphan no sweep can reach, so the probe image has its own reaper.**
`sweep_orphaned_snapshots` collects `dangling` + `triple-c.managed=true`; `reap_stale_migration_pins`
and `scrub_secrets_from_snapshots` both filter `triple-c-snapshot-*`. A `triple-c-probe-*` image is
tagged and so matches none of them, which would make a crashed probe a permanent multi-gigabyte
leak with no UI to find it. `reap_probe_images` runs at startup beside `reap_probe_containers` and
is **load-bearing, not tidying** — it is also what makes the probe image's unscrubbed writable
layer acceptable. Two rules it earned the hard way:
- **Age-gate it** (`PROBE_REAP_MIN_AGE_SECS`, same as the container reaper). `reference=` is
daemon-wide, so a second copy of the app has live probe images matching the glob.
- **Remove by tag, never by image id.** A `force` removal by id untags an image *everywhere*; a
fixture that tagged `alpine:latest` into this namespace deleted the user's alpine that way.
- **Probe image names are unique per call, and must stay that way.** A stable per-container name was
tried: container ids do not survive a recreate, so most leftovers were stranded permanently, and
two concurrent probes fought over one tag — whichever finished first force-removed the image the
other was still reading, reporting a bogus `probe_error` on a healthy project. `get_container_staleness`
takes no `project_lock` claim (the migration banner needs it to answer *during* a migration), so
uniqueness is what makes overlapping probes safe.
- **The stopped-container probe is cached per stop, and that is not an optimisation you may drop.**
`getContainerStaleness` is called from a `useEffect` that fires whenever the container settles, so
merely opening a stopped project's Overview probes it. Uncached that is a `docker commit` of the
whole writable layer per visit — measured at 44 s on a real project, against ~3 s for the snapshot
probe it replaced. `STOPPED_MANIFEST_CACHE` is keyed on the container's `FinishedAt`, which is
exact rather than merely plausible: nothing can write to a stopped container's writable layer, and
`FinishedAt` moves on every stop. A live test asserts the restart case, because a cache that
failed to invalidate would plan a migration against a filesystem the project no longer has.
- **Do not "skip the probe when the project is not stale" to save that cost.** It was tried. The
deltas would be empty while `probeSettled` (`!probing && staleness && !probe_error`) stayed *true*,
which leaves the migrate action in the project menu enabled — that action is not gated on the
banner — so the pre-flight would report nothing to copy while the backend was told to copy
nothing. That is the exact hazard `ProjectHome.tsx`'s `canMigrate` comment already warns about.
- **A failed stopped-container probe falls back to the snapshot whenever one exists.** Before this
feature a stopped project read its snapshot directly, so surfacing a commit failure where the
snapshot could have answered would make the banner *worse* than it was — and the failure modes are
exactly the ones where the fallback earns its keep: a full disk (the commit allocates the whole
writable layer; the snapshot probe allocates nothing) and a 409 from a concurrent claim.
- **`get_container_staleness` never commits while the project is claimed.** It takes no
`project_lock` claim itself, deliberately — the banner has to answer *during* a migration — so it
reads `project_lock::held` instead and probes the snapshot rather than the container. The
collision is not symmetric: the probe losing is a retryable `probe_error`, but
`start_project_container` removes the old container with a hard `?`, so a remove that raced a
commit would fail the user's Start with an opaque error.
- **An image's `Created` is the image's own, not its tag's.** Tagging an existing image gives you
that image's age; BuildKit stamps `docker build` output with a fixed epoch. Only `docker commit`
stamps *now* — which is what real probe images do, and what any fixture for them must do.
- **`:latest` keeps pointing at the old lineage until the final commit.** That is what makes every
crash before that point self-heal — `start_project_container` just recreates from the old
snapshot. After the container swap, the new container's `triple-c.migration-state=in-progress`
+213 -11
View File
@@ -92,8 +92,111 @@ fn pick_recorded_lineage(
.or_else(|| from_snapshot.filter(|v| !v.is_empty()))
}
/// Read-only. Runs two filesystem probes (~3 s each) and is therefore meant to
/// be called on demand, not polled.
/// Reported as `probe_error` when there is genuinely nothing to read: no
/// container, stopped or otherwise, and no snapshot image.
///
/// It used to be reported for a *stopped* container too, which was simply
/// untrue — the container was sitting right there — and it disabled Update on
/// exactly the long-lived projects that had never been recreated and so had no
/// snapshot to fall back on.
const NOTHING_TO_PROBE: &str = "This project has no container or snapshot image yet, so there is nothing to compare against the base image.";
/// Where [`get_container_staleness`] reads the project's *current* filesystem
/// from, in descending order of how current the answer is.
#[derive(Debug, PartialEq, Eq)]
enum ProbeSource {
/// `docker exec` into the live container. The only source that includes
/// everything installed since the last commit *in this session*.
RunningContainer,
/// Commit the stopped container's writable layer to a throwaway image and
/// probe that. Exactly as current as the container, which is what makes it
/// preferable to the snapshot — see below.
StoppedContainer,
/// A throwaway container from `triple-c-snapshot-<id>:latest`.
Snapshot,
/// Nothing to read: no container, no snapshot.
Nothing,
}
/// Pick the probe source. `container_running` is `None` when the project has no
/// container at all, `Some(false)` when it has a stopped one.
///
/// **A stopped container outranks the snapshot.** The snapshot image is not a
/// checkpoint — `commit_container_snapshot` runs only before a removal (a
/// config-change recreate) or inside a migration, so a project that has never
/// hit either has *no snapshot at all*, however long it has been in use, and
/// one that has is stale by everything installed since. The container's
/// writable layer is the truth in both cases. This is the same argument
/// [`mig::manifest_from_container`] already makes for the running case; it does
/// not stop applying when the container is stopped.
///
/// Getting this wrong is what made a stopped, never-recreated project report
/// "no container or snapshot image yet" — with its container sitting right
/// there — and left Update disabled on the projects that most needed it.
fn pick_probe_source(container_running: Option<bool>, snapshot_exists: bool) -> ProbeSource {
match (container_running, snapshot_exists) {
(Some(true), _) => ProbeSource::RunningContainer,
(Some(false), _) => ProbeSource::StoppedContainer,
(None, true) => ProbeSource::Snapshot,
(None, false) => ProbeSource::Nothing,
}
}
/// Reported as `probe_error` when another operation owns the project and there
/// is no snapshot image to read instead. Deliberately not a claim about the
/// container: nothing is wrong with it, the answer is simply not safe to take
/// right now. See [`stopped_probe_policy`].
const PROJECT_BUSY: &str = "Another operation is running on this project, so its contents could not be inspected. Try again once it finishes.";
/// What to do about a stopped container, whose probe is the expensive one: it
/// commits the writable layer before it can read anything.
#[derive(Debug, PartialEq, Eq)]
enum StoppedProbe {
/// Commit and probe. The current answer, and the default.
Commit,
/// Probe the snapshot image instead. Less current — it lags the container by
/// everything installed since the last commit — but it allocates nothing and
/// touches nothing, which is what makes it the right answer while another
/// operation owns the container.
SnapshotInstead,
/// Report rather than guess.
Defer,
}
/// Pick what to do about a stopped container.
///
/// **Never commits while the project is claimed.** `get_container_staleness`
/// takes no [`crate::project_lock`] claim of its own, by design, so a commit
/// here can overlap a Recreate or Reset — and the collision is not symmetric.
/// The probe losing is harmless: a surfaced `probe_error` the user retries. The
/// *recreate* losing is not, because `start_project_container` removes the old
/// container with a hard `?`, so a non-404 from a remove that raced this commit
/// fails the whole Start with an opaque "Failed to remove container". Reading
/// the claim costs nothing and takes that failure off the table.
fn stopped_probe_policy(project_is_busy: bool, snapshot_exists: bool) -> StoppedProbe {
match (project_is_busy, snapshot_exists) {
(false, _) => StoppedProbe::Commit,
(true, true) => StoppedProbe::SnapshotInstead,
(true, false) => StoppedProbe::Defer,
}
}
/// Runs two filesystem probes (~3 s each) and is therefore meant to be called
/// on demand, not polled.
///
/// **Not read-only, despite only reporting.** The stopped-container path commits
/// a throwaway image and force-removes it, which makes this a writer of a
/// `triple-c-probe-*` image and puts it in the class of thing
/// [`crate::project_lock`] exists for — and it takes no claim. That is
/// deliberate: this is what the migration banner calls to decide whether to
/// offer an update, including while a migration is in flight, so refusing it
/// under a claim would blank the banner exactly when it has the most to say.
/// The exposure is bounded to a surfaced error — a concurrent Recreate, Reset or
/// migration can remove the container out from under the commit, and the result
/// is a `probe_error` the user can retry, never a damaged container or a
/// mislabelled image. Two overlapping probes cannot collide either, because
/// probe image names are unique per call; see
/// [`crate::docker::container::get_probe_image_name`].
#[tauri::command]
pub async fn get_container_staleness(
project_id: String,
@@ -145,16 +248,62 @@ pub async fn get_container_staleness(
};
// ── Probes ───────────────────────────────────────────────────────────
let running = match &container_id {
Some(id) => docker::is_container_running(id).await.unwrap_or(false),
None => false,
let container_running = match &container_id {
Some(id) => Some(docker::is_container_running(id).await.unwrap_or(false)),
None => None,
};
let from_manifest = if running {
mig::manifest_from_container(container_id.as_ref().unwrap()).await
} else if docker::image_exists(&snapshot_image).await.unwrap_or(false) {
mig::manifest_from_image(&snapshot_image).await
} else {
Err("This project has no container or snapshot image yet, so there is nothing to compare against the base image.".to_string())
let snapshot_exists = docker::image_exists(&snapshot_image).await.unwrap_or(false);
let from_manifest = match (
pick_probe_source(container_running, snapshot_exists),
&container_id,
) {
(ProbeSource::RunningContainer, Some(id)) => mig::manifest_from_container(id).await,
(ProbeSource::StoppedContainer, Some(id)) => {
let busy = crate::project_lock::held(&project_id).is_some();
match stopped_probe_policy(busy, snapshot_exists) {
StoppedProbe::Commit => {
match mig::manifest_from_stopped_container_cached(id).await {
Ok(m) => Ok(m),
// **Never let a failed commit cost an answer the
// snapshot could have given.** Before stopped
// containers were readable at all, a stopped project
// fell straight through to its snapshot, so surfacing
// this error where the snapshot exists would make the
// banner *worse* than it was — and the ways this fails
// are the ones where the fallback matters most: a full
// disk (the commit has to allocate the whole writable
// layer; the snapshot probe allocates nothing) and a
// 409 from an operation that claimed the project after
// the check above.
Err(e) if snapshot_exists => {
log::warn!(
"Probing the stopped container for project {} failed ({}) — \
falling back to its snapshot image, which may lag it",
project_id,
e
);
mig::manifest_from_image(&snapshot_image).await
}
Err(e) => Err(e),
}
}
StoppedProbe::SnapshotInstead => {
log::info!(
"Project {} is claimed by another operation — probing its snapshot image \
rather than committing the container",
project_id
);
mig::manifest_from_image(&snapshot_image).await
}
StoppedProbe::Defer => Err(PROJECT_BUSY.to_string()),
}
}
(ProbeSource::Snapshot, _) => mig::manifest_from_image(&snapshot_image).await,
// `container_running` is `Some` exactly when `container_id` is, so the
// two arms above are the only ones those variants can reach. This arm
// is `ProbeSource::Nothing` — and now *only* that: it used to also
// swallow every stopped container, which is the bug.
(_, _) => Err(NOTHING_TO_PROBE.to_string()),
};
let (from_manifest, base_manifest) = match from_manifest {
@@ -1964,6 +2113,59 @@ mod tests {
assert_eq!(pick_recorded_lineage(some(""), None), None);
}
#[test]
fn a_stopped_container_is_probed_rather_than_reported_missing() {
// The regression: a container that exists but is stopped, with no
// snapshot ever taken, read as "nothing to compare against".
assert_eq!(
pick_probe_source(Some(false), false),
ProbeSource::StoppedContainer
);
}
#[test]
fn the_container_outranks_the_snapshot_whether_or_not_it_is_running() {
// The snapshot lags the container by everything installed since the
// last commit, in both states.
assert_eq!(
pick_probe_source(Some(true), true),
ProbeSource::RunningContainer
);
assert_eq!(
pick_probe_source(Some(false), true),
ProbeSource::StoppedContainer
);
}
#[test]
fn the_snapshot_is_the_fallback_only_once_the_container_is_gone() {
assert_eq!(pick_probe_source(None, true), ProbeSource::Snapshot);
}
#[test]
fn nothing_to_probe_is_reserved_for_no_container_and_no_snapshot() {
// The one case the "no container or snapshot image yet" message may
// still describe.
assert_eq!(pick_probe_source(None, false), ProbeSource::Nothing);
}
#[test]
fn a_stopped_container_is_committed_only_when_nothing_else_owns_the_project() {
assert_eq!(stopped_probe_policy(false, false), StoppedProbe::Commit);
assert_eq!(stopped_probe_policy(false, true), StoppedProbe::Commit);
}
#[test]
fn a_busy_project_falls_back_rather_than_racing_a_recreate() {
// The snapshot lags, but a stale answer beats failing someone's Start.
assert_eq!(
stopped_probe_policy(true, true),
StoppedProbe::SnapshotInstead
);
// Nothing to fall back to: say so instead of committing anyway.
assert_eq!(stopped_probe_policy(true, false), StoppedProbe::Defer);
}
#[test]
fn byte_sizes_read_the_way_a_disk_warning_should() {
assert_eq!(human_bytes(512), "512 B");
+140 -4
View File
@@ -3052,6 +3052,118 @@ fn blanked_secret_env() -> Vec<String> {
.collect()
}
/// Image-name prefix for the throwaway commit a staleness probe of a stopped
/// container makes. The reaper's only handle on a leftover — see
/// [`crate::docker::migration::reap_probe_images`] — so nothing else may use it.
pub const PROBE_IMAGE_PREFIX: &str = "triple-c-probe-";
/// The throwaway image a staleness probe of a **stopped** container commits to.
///
/// **Unique per call**, and both halves of the name earn their place: the
/// container id prefix makes a leftover traceable in `docker images`, and the
/// counter makes two overlapping probes independent.
///
/// An earlier version of this was deliberately *stable* per container, on the
/// theory that the next probe would move the tag off an abandoned image and
/// leave it dangling for [`sweep_orphaned_snapshots`]. That was wrong twice
/// over. A container id does not survive a recreate, so for most leftovers
/// there is no "next probe of the same container" and the image was stranded
/// permanently; and a stable name made two concurrent probes fight over one
/// tag, where whichever finished first force-removed the image the other was
/// still reading and turned a healthy project into a bogus `probe_error`.
/// Uniqueness fixes both, and [`crate::docker::migration::reap_probe_images`]
/// is what collects the leftovers instead.
pub fn get_probe_image_name(container_id: &str) -> String {
use std::sync::atomic::{AtomicU64, Ordering};
static SEQ: AtomicU64 = AtomicU64::new(0);
let short: String = container_id.chars().take(12).collect();
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
format!(
"{}{}-{}-{}:latest",
PROBE_IMAGE_PREFIX,
short,
nanos,
SEQ.fetch_add(1, Ordering::Relaxed)
)
}
/// Commit a **stopped** container's filesystem to a throwaway image, returning
/// its name. The caller owns the image and must remove it.
///
/// This exists so a stopped project can be read at all. `docker exec` needs a
/// running container and the snapshot image is not a checkpoint — see
/// [`crate::commands::migration_commands`]'s probe-source pick — so without
/// this there is no way to see inside a project that is merely stopped.
///
/// ## Why it is tagged at all
///
/// An untagged commit would be tidier: untagged plus the `triple-c.managed=true`
/// that `docker commit` copies off the container is exactly the pair
/// [`sweep_orphaned_snapshots`] already collects, so a leftover would self-heal
/// with no new machinery. **It is not available.** `bollard`'s `Commit` response
/// model deserialises `"ID"` while the daemon sends `"Id"`, so
/// `commit_container` hands back `id: None` every time and there is no
/// reference left to probe. Neither existing commit site notices, because both
/// discard the response. Verified against Engine 29.6, bollard 0.18.1.
///
/// So the image needs a name, a tagged image is not dangling, and the sweep
/// therefore cannot be the safety net. [`crate::docker::migration::reap_probe_images`]
/// is, and [`get_probe_image_name`] carries the rest of that argument.
///
/// ## What is in the image, and what is not
///
/// `pause: false` because nothing is running — pausing a stopped container is
/// an error, the same reason [`recommit_without_secrets`]'s scratch commit
/// passes `false`.
///
/// Secrets are blanked from the env for the same reason
/// [`commit_container_snapshot`] blanks them: the commit bakes the container's
/// full ENV into the image, and "it only lives a few seconds" is not a property
/// this function can promise after a crash.
///
/// **The writable layer is committed unscrubbed, and that is unavoidable here.**
/// [`commit_container_snapshot`] runs [`scrub_writable_layer`] first precisely
/// because a commit stacks a layer and never rewrites one — but that scrub is a
/// `docker exec`, which is exactly what a stopped container cannot serve, and
/// scrubbing is not wanted anyway: the probe's whole job is to report the
/// filesystem as it actually is. What makes it acceptable is that this copies
/// bytes that are *already on this disk* in the container's own writable layer,
/// into an image that is never pushed, never created from, and reaped — so it
/// duplicates data inside one trust domain rather than widening it. That
/// argument depends on the reaping actually happening; treat
/// [`crate::docker::migration::reap_probe_images`] as load-bearing, not tidying.
pub async fn commit_container_for_probe(container_id: &str) -> Result<String, String> {
let docker = get_docker()?;
let image_name = get_probe_image_name(container_id);
let (repo, tag) = image_name
.rsplit_once(':')
.map(|(r, t)| (r.to_string(), t.to_string()))
.expect("get_probe_image_name always emits a tag");
docker
.commit_container(
CommitContainerOptions {
container: container_id.to_string(),
repo,
tag,
pause: false,
..Default::default()
},
Config::<String> {
env: Some(blanked_secret_env()),
..Default::default()
},
)
.await
.map_err(|e| format!("Failed to commit stopped container {}: {}", container_id, e))?;
Ok(image_name)
}
/// Whether `env` (an image's `Config.Env`) holds a non-empty value for any
/// name in [`SECRET_ENV_KEYS`].
fn env_holds_a_secret(env: &[String]) -> bool {
@@ -3518,9 +3630,10 @@ pub async fn remove_snapshot_image(project: &Project) -> Result<(), String> {
remove_image_by_name(&get_snapshot_image_name(project)).await
}
/// Remove a Docker image by name/tag, treating "does not exist" as success.
/// Shared by [`remove_snapshot_image`] and the pending-cleanup retry, which
/// only has the image name (the project record is already gone by then).
/// Remove a Docker image by name, tag or **id**, treating "does not exist" as
/// success. Shared by [`remove_snapshot_image`], the pending-cleanup retry
/// (which only has the image name the project record is already gone by
/// then), and the staleness probe's throwaway commit, which has only an id.
pub async fn remove_image_by_name(image_name: &str) -> Result<(), String> {
let docker = get_docker()?;
@@ -3536,7 +3649,7 @@ pub async fn remove_image_by_name(image_name: &str) -> Result<(), String> {
.await
{
Ok(_) => {
log::info!("Removed snapshot image {}", image_name);
log::info!("Removed image {}", image_name);
Ok(())
}
Err(bollard::errors::Error::DockerResponseServerError {
@@ -4464,6 +4577,29 @@ mod tests {
assert!(env_holds_a_secret(&env));
}
/// The probe image's name must be **unique per call**. A stable name was
/// tried and is wrong twice over: a container id does not survive a
/// recreate, so a crashed probe's leftover would never be reclaimed by "the
/// next probe of the same container"; and two concurrent probes sharing one
/// tag means whichever finishes first force-removes the image the other is
/// still reading. See `commit_container_for_probe` and `reap_probe_images`.
#[test]
fn probe_image_names_are_unique_per_call_and_reapable_by_prefix() {
let id = "75993e6d5e1ab473b029a408c5ff0339";
let a = get_probe_image_name(id);
let b = get_probe_image_name(id);
assert_ne!(a, b, "two probes of one container must not share a tag");
// The prefix is the reaper's only handle on a leftover, so every name
// has to carry it — and it must not be the snapshot namespace, which is
// what a project is rebuilt from.
assert!(a.starts_with(PROBE_IMAGE_PREFIX), "{}", a);
assert!(!a.starts_with("triple-c-snapshot-"), "{}", a);
// Traceable back to its container, which is the point of the prefix.
assert!(a.contains("75993e6d5e1a"), "{}", a);
assert!(a.ends_with(":latest"), "{}", a);
}
#[test]
fn the_scrub_report_only_claims_success_when_nothing_is_left() {
let clean = SnapshotScrubReport {
+461
View File
@@ -886,6 +886,100 @@ pub async fn reap_probe_containers() {
}
}
/// Remove throwaway images left behind by a staleness probe of a stopped
/// container — [`super::container::commit_container_for_probe`]'s commits.
///
/// **Load-bearing, not tidying.** A probe image is *tagged*, because bollard
/// gives no image id back from a commit and there has to be something to probe.
/// Tagged means not dangling, so [`super::container::sweep_orphaned_snapshots`]
/// — which collects every other kind of orphan this app can leave — will never
/// see one. Without this, a probe that dies between its commit and its own
/// cleanup (SIGKILL, a crash, a 409 from a concurrent remove) strands a
/// multi-gigabyte image that **no code path can ever reclaim**, and there is no
/// UI to find it either. That is the one leak in this app with no floor on it,
/// so this runs at startup beside [`reap_probe_containers`].
///
/// Age-gated for exactly the reason that one is: `reference=` is a daemon-wide
/// filter, so a second copy of the app probing a project on the same daemon has
/// images matching this glob, and removing one mid-capture fails that probe with
/// "No such image" — the bogus `probe_error` the staleness work exists to get
/// rid of. In-process state cannot see the other instance, so age is the only
/// brake, and [`PROBE_REAP_MIN_AGE_SECS`] is already the right one: a probe is a
/// `find` over a root filesystem, not a multi-minute job.
///
/// Never fails the caller. Housekeeping, like every other sweep here.
pub async fn reap_probe_images() {
use bollard::image::{ListImagesOptions, RemoveImageOptions};
let docker = match get_docker() {
Ok(d) => d,
Err(e) => {
log::warn!("Could not reap leftover probe images: {}", e);
return;
}
};
let filters = HashMap::from([(
"reference".to_string(),
vec![format!("{}*", super::container::PROBE_IMAGE_PREFIX)],
)]);
let images = match docker
.list_images(Some(ListImagesOptions {
all: false,
filters,
..Default::default()
}))
.await
{
Ok(images) => images,
Err(e) => {
log::warn!("Could not list leftover probe images: {}", e);
return;
}
};
let now = chrono::Utc::now().timestamp();
for image in images {
// Unlike a container summary, an image summary always carries a
// `Created`, so there is no unknown-age case to defend against here.
if now - image.created < PROBE_REAP_MIN_AGE_SECS {
log::info!(
"Leaving probe image {:?} alone — it is younger than {} minutes, so it may belong \
to another Triple-C instance's live probe",
image.repo_tags,
PROBE_REAP_MIN_AGE_SECS / 60
);
continue;
}
// By **tag**, never by image id. A `force` removal by id untags an
// image everywhere, so an id that happens to carry another name loses
// that name too — which is how a test fixture that tagged
// `alpine:latest` into this namespace deleted the user's alpine. A real
// leftover has exactly the one probe tag, so removing the tag removes
// the image; anything else keeps whatever other names it has.
for tag in image
.repo_tags
.iter()
.filter(|t| t.starts_with(super::container::PROBE_IMAGE_PREFIX))
{
log::info!("Removing leftover probe image {}", tag);
if let Err(e) = docker
.remove_image(
tag,
Some(RemoveImageOptions {
force: true,
noprune: false,
}),
None,
)
.await
{
log::warn!("Could not remove leftover probe image {}: {}", tag, e);
}
}
}
}
/// How old a `triple-c.probe=migration` container must be before
/// [`reap_probe_containers`] will force-remove it, in seconds.
///
@@ -993,6 +1087,119 @@ pub async fn manifest_from_container(container_id: &str) -> Result<Manifest, Str
Ok(parse_manifest(&out))
}
/// Cached stopped-container manifests, keyed by container id, each paired with
/// the container's `FinishedAt` at the time it was captured.
///
/// **Sound because a stopped container's writable layer cannot change.** Nothing
/// can write to it while it is not running, so a manifest captured after it
/// stopped stays true until it is started again — and `FinishedAt` moves on
/// every stop, which is what makes the key exact rather than merely plausible.
///
/// This exists because `get_container_staleness` is called from a `useEffect`
/// that fires whenever the container settles, so simply opening a stopped
/// project's Overview probes it. Uncached that meant a `docker commit` of the
/// whole writable layer per visit — measured at 44 s on a real project — where
/// before this feature the same visit cost one throwaway container or nothing at
/// all. A regression like that is not worth the answer it buys.
///
/// Capped, because a `Manifest` of a real container is a few MB: this only has
/// to serve "the project whose page is open", so a handful of entries is the
/// whole working set and the oldest is dropped past that.
static STOPPED_MANIFEST_CACHE: std::sync::Mutex<
Option<Vec<(String, String, Manifest)>>,
> = std::sync::Mutex::new(None);
/// How many stopped-container manifests [`STOPPED_MANIFEST_CACHE`] keeps.
const STOPPED_MANIFEST_CACHE_MAX: usize = 4;
/// `FinishedAt` for a container, the cache's validity token. `None` when it
/// cannot be read, which is never treated as a hit.
async fn container_finished_at(container_id: &str) -> Option<String> {
let docker = get_docker().ok()?;
docker
.inspect_container(container_id, None)
.await
.ok()?
.state?
.finished_at
.filter(|s| !s.is_empty())
}
/// Capture a [`Manifest`] from a **stopped** container, reusing a cached one
/// when the container has not been started since it was taken.
///
/// See [`STOPPED_MANIFEST_CACHE`] for why this is exact and why it is needed.
pub async fn manifest_from_stopped_container_cached(
container_id: &str,
) -> Result<Manifest, String> {
let finished_at = container_finished_at(container_id).await;
if let Some(token) = &finished_at {
let guard = STOPPED_MANIFEST_CACHE.lock();
if let Ok(cache) = guard {
if let Some(entries) = cache.as_ref() {
if let Some((_, _, manifest)) = entries
.iter()
.find(|(id, tok, _)| id == container_id && tok == token)
{
log::debug!(
"Reusing the cached manifest for stopped container {}",
container_id
);
return Ok(manifest.clone());
}
}
}
}
let manifest = manifest_from_stopped_container(container_id).await?;
// Only cacheable if the container's state could be read at all; an unknown
// `FinishedAt` means there is no token that could later be compared.
if let Some(token) = finished_at {
if let Ok(mut cache) = STOPPED_MANIFEST_CACHE.lock() {
let entries = cache.get_or_insert_with(Vec::new);
entries.retain(|(id, _, _)| id != container_id);
entries.push((container_id.to_string(), token, manifest.clone()));
while entries.len() > STOPPED_MANIFEST_CACHE_MAX {
entries.remove(0);
}
}
}
Ok(manifest)
}
/// Capture a [`Manifest`] from a **stopped** container.
///
/// Commits the container's writable layer to a throwaway image, probes that,
/// and removes it. This is as current as [`manifest_from_container`] — it reads
/// the same filesystem — and it is why a stopped project no longer has to fall
/// back to its snapshot image, which may not exist at all and lags the
/// container by everything installed since the last commit when it does.
///
/// The image is removed on every path, including a failed probe. See
/// [`super::container::commit_container_for_probe`] for what a crash in the
/// window between the two costs, and why it is bounded.
pub async fn manifest_from_stopped_container(container_id: &str) -> Result<Manifest, String> {
let image = super::container::commit_container_for_probe(container_id).await?;
let manifest = manifest_from_image(&image)
.await
.map_err(|e| format!("Probe of the stopped container did not complete: {}", e));
if let Err(e) = super::container::remove_image_by_name(&image).await {
log::warn!(
"Could not remove the staleness probe's throwaway image {}: {} — `reap_probe_images` \
collects it at the next app start; the orphan sweep never will, because it is tagged",
image,
e
);
}
manifest
}
/// The image ID (`sha256:…`) of a local image, or `None` if it is not present.
///
/// Deliberately the **ID**, not a repo digest: locally built images and custom
@@ -2146,4 +2353,258 @@ mod tests {
assert!(!pin_is_reapable("pre-migration-handmade", false, ancient, &now));
assert!(!pin_is_reapable("latest", false, ancient, &now));
}
// ── Live Docker ─────────────────────────────────────────────────────────
/// The cache serves a second read of an unchanged stopped container, and —
/// the half that matters — stops serving it the moment the container is
/// started and stopped again. If invalidation were wrong this would report a
/// filesystem the project no longer has, and a migration would be planned
/// against it.
///
/// ```text
/// cargo test -- --ignored --nocapture stopped_manifest_cache
/// ```
#[cfg(unix)]
#[tokio::test]
#[ignore = "needs a Docker daemon; creates, commits and removes a throwaway container"]
async fn the_stopped_manifest_cache_survives_a_reread_but_not_a_restart() {
fn docker_cli(args: &[&str]) -> String {
let out = std::process::Command::new("docker")
.args(args)
.output()
.expect("docker CLI");
assert!(
out.status.success(),
"docker {:?} failed: {}",
args,
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
}
let image = std::env::var("TRIPLE_C_TEST_IMAGE")
.unwrap_or_else(|_| "ghcr.io/shadowdao/triple-c-sandbox:latest".to_string());
let first = format!("/opt/cache-marker-a-{}", std::process::id());
let second = format!("/opt/cache-marker-b-{}", std::process::id());
let id = docker_cli(&[
"run", "-d", "--label", "triple-c.managed=true",
"--entrypoint", "/bin/sh",
&image, "-c", "sleep 600",
]);
let cleanup = || {
let _ = std::process::Command::new("docker")
.args(["rm", "-f", &id])
.output();
};
docker_cli(&["exec", &id, "mkdir", "-p", &first]);
docker_cli(&["stop", "-t", "1", &id]);
let t0 = std::time::Instant::now();
let cold = manifest_from_stopped_container_cached(&id).await;
let cold_ms = t0.elapsed().as_millis();
let t1 = std::time::Instant::now();
let warm = manifest_from_stopped_container_cached(&id).await;
let warm_ms = t1.elapsed().as_millis();
// Restart, change the filesystem, stop again — `FinishedAt` moves.
docker_cli(&["start", &id]);
docker_cli(&["exec", &id, "mkdir", "-p", &second]);
docker_cli(&["stop", "-t", "1", &id]);
let after_restart = manifest_from_stopped_container_cached(&id).await;
cleanup();
let has = |m: &Manifest, p: &str| m.paths.iter().any(|e| e.path == p && e.is_dir());
let cold = cold.expect("cold read");
let warm = warm.expect("warm read");
let after_restart = after_restart.expect("read after restart");
assert!(has(&cold, &first), "cold read missed {}", first);
assert!(has(&warm, &first), "warm read missed {}", first);
println!("cold {} ms, warm {} ms", cold_ms, warm_ms);
assert!(
warm_ms * 5 < cold_ms.max(5),
"the second read cost {} ms against a cold {} ms — it re-committed \
instead of using the cache",
warm_ms,
cold_ms
);
// The restart must have invalidated it: the new directory has to show up.
assert!(
has(&after_restart, &second),
"a restart did not invalidate the cache — {} is missing, so this is \
a stale manifest of a filesystem the container no longer has",
second
);
assert!(has(&after_restart, &first), "the restart lost {}", first);
}
/// The reaper finds a leftover probe image by prefix and — crucially —
/// refuses to remove a young one, because that image may be another
/// Triple-C instance's live probe. Only a real daemon can say whether the
/// `reference=` glob matches the names `get_probe_image_name` produces.
///
/// The fixture is **committed**, not tagged and not built. An image's
/// `Created` is its own, not its tag's, so tagging something already on disk
/// into this namespace yields a fixture the reaper is right to call ancient
/// — and BuildKit stamps a fixed epoch on `docker build` output, so a built
/// one looks ancient too. A commit stamps *now*, verified against Engine
/// 29.6, which is also how real probe images get their age.
///
/// Both of those mistakes were made here first, and one of them deleted an
/// unrelated `alpine:latest` — which is why `reap_probe_images` removes by
/// tag rather than by image id.
///
/// ```text
/// cargo test -- --ignored --nocapture reaper_spares
/// ```
#[cfg(unix)]
#[tokio::test]
#[ignore = "needs a Docker daemon; builds and removes a throwaway image"]
async fn the_reaper_spares_a_probe_image_young_enough_to_be_someone_elses() {
use std::process::Command;
fn docker_out(args: &[&str]) -> std::process::Output {
Command::new("docker").args(args).output().expect("docker CLI")
}
let base = std::env::var("TRIPLE_C_TEST_IMAGE")
.unwrap_or_else(|_| "alpine:latest".to_string());
let name = crate::docker::container::get_probe_image_name("reapertest01234");
// A never-started container is enough to commit from, and leaves the
// daemon's run state alone entirely.
let created = docker_out(&["create", &base, "true"]);
assert!(
created.status.success(),
"could not create the fixture container from {}: {}",
base,
String::from_utf8_lossy(&created.stderr)
);
let cid = String::from_utf8_lossy(&created.stdout).trim().to_string();
let committed = docker_out(&["commit", "--pause=false", &cid, &name]);
let _ = docker_out(&["rm", "-f", &cid]);
assert!(
committed.status.success(),
"could not commit the fixture image: {}",
String::from_utf8_lossy(&committed.stderr)
);
reap_probe_images().await;
let still_there = Command::new("docker")
.args(["image", "inspect", &name])
.output()
.expect("docker image inspect")
.status
.success();
let _ = Command::new("docker").args(["rmi", &name]).output();
assert!(
still_there,
"a probe image committed seconds ago was reaped — that is another \
instance's live probe being broken, see PROBE_REAP_MIN_AGE_SECS"
);
}
/// A *stopped* container is readable, and what comes back is its writable
/// layer rather than the image it was created from. This is the whole point
/// of the function: the base image cannot answer it, and the project may
/// well have no snapshot image at all.
///
/// Also asserts the throwaway commit leaves nothing behind, which no unit
/// test can. It has to assert on the `triple-c-probe-*` tags specifically:
/// the probe image is *tagged*, so a leak never shows up as a dangling
/// image and a dangling-set assertion here would pass either way.
///
/// Ignored because it needs Docker and commits a container; run it with
///
/// ```text
/// cargo test -- --ignored --nocapture stopped_container
/// ```
#[cfg(unix)]
#[tokio::test]
#[ignore = "needs a Docker daemon; creates, commits and removes a throwaway container"]
async fn a_stopped_container_is_read_from_its_writable_layer() {
fn docker_cli(args: &[&str]) -> String {
let out = std::process::Command::new("docker")
.args(args)
.output()
.expect("docker CLI");
assert!(
out.status.success(),
"docker {:?} failed: {}",
args,
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
}
fn probe_images() -> Vec<String> {
let mut ids: Vec<String> = docker_cli(&[
"images", "-q",
"--filter",
&format!("reference={}*", crate::docker::container::PROBE_IMAGE_PREFIX),
])
.lines()
.map(|l| l.trim().to_string())
.filter(|l| !l.is_empty())
.collect();
ids.sort();
ids
}
let image = std::env::var("TRIPLE_C_TEST_IMAGE")
.unwrap_or_else(|_| "ghcr.io/shadowdao/triple-c-sandbox:latest".to_string());
// A marker only the writable layer can carry, under a MANIFEST_ROOTS root.
let marker = format!("/opt/probe-marker-{}", std::process::id());
// Another instance's live probe images are allowed to exist; what must
// hold is that this probe adds none of its own.
let before = probe_images();
let id = docker_cli(&[
"run", "-d", "--label", "triple-c.managed=true",
"--entrypoint", "/bin/sh",
&image, "-c", "sleep 300",
]);
let cleanup = |id: &str| {
let _ = std::process::Command::new("docker")
.args(["rm", "-f", id])
.output();
};
docker_cli(&["exec", &id, "mkdir", "-p", &marker]);
docker_cli(&["stop", "-t", "1", &id]);
let result = manifest_from_stopped_container(&id).await;
cleanup(&id);
let manifest = result.expect("a stopped container must be probeable");
assert!(
manifest.paths.iter().any(|e| e.path == marker && e.is_dir()),
"the probe read the image, not the container's writable layer: {} missing",
marker
);
// Non-empty package sets prove the probe script really ran, rather than
// parsing an empty transcript into an empty-but-Ok manifest.
assert!(
!manifest.apt_manual.is_empty(),
"apt-mark showmanual came back empty, so the probe did not run"
);
assert_eq!(
probe_images(),
before,
"the throwaway probe image was not cleaned up"
);
}
}
+9 -1
View File
@@ -263,12 +263,20 @@ pub fn run() {
// logged warning rather than a failed start.
//
// Ordering matters. Probes are removed first because a probe holds
// an image open and the sweep will not force; pins are untagged
// an image open and the sweep will not force — both the probe
// containers and the probe images, the latter being the one orphan
// the sweep can never reach on its own; pins are untagged
// second so the images they were holding are dangling by the time
// the sweep lists them; the sweep runs last and collects both.
let projects_store_for_cleanup = projects_store_setup.clone();
tauri::async_runtime::spawn(async move {
crate::docker::reap_probe_containers().await;
// Probe *images* too, and for a sharper reason: a probe
// container merely pins an image the sweep then refuses to
// touch, whereas a leftover probe image is tagged and so
// nothing else in this app can ever collect it. See
// `reap_probe_images`.
crate::docker::reap_probe_images().await;
let reaped = crate::docker::reap_stale_migration_pins().await;
if reaped > 0 {
log::info!("Startup housekeeping dropped {} stale rollback pin(s)", reaped);
+15 -3
View File
@@ -350,8 +350,8 @@ export const sweepClaudeTokenSnapshots = () =>
// without deleting its volumes. Reset is the destructive alternative: it wipes
// ~/.claude, the OAuth credential, installed skills and every transcript.
//
// Flow: getContainerStaleness (read-only, ~6s — two filesystem probes, so call
// it on demand rather than polling) → migrateProjectToBase → the project sits
// Flow: getContainerStaleness (~6s — two filesystem probes, so call it on demand
// rather than polling) → migrateProjectToBase → the project sits
// in "awaiting-confirmation" while the user tries it → confirmMigration or
// rollbackMigration.
//
@@ -361,7 +361,19 @@ export const sweepClaudeTokenSnapshots = () =>
//
// Progress arrives on the existing `container-progress` event.
/** Read-only. Runs two container/image filesystem probes; not for polling. */
/**
* Runs two container/image filesystem probes; not for polling.
*
* **Not read-only, despite only reporting.** When the container is *stopped*
* the backend has to commit its writable layer to a throwaway image before it
* can read anything `docker exec` needs a running container so this writes
* (and then removes) an image. The result is cached per stop, so repeat calls
* while the container stays stopped are cheap, but the first one after each stop
* pays for a commit of the whole layer: seconds on a small project, tens of
* seconds on a large one. Do not add a caller that fires more often than "the
* container settled into a new state" without re-reading
* `get_container_staleness`'s doc comment first.
*/
export const getContainerStaleness = (projectId: string) =>
invoke<ContainerStaleness>("get_container_staleness", { projectId });