From 307ea074090f2cd5c1bd61709efaa1a0246eeb84 Mon Sep 17 00:00:00 2001 From: Josh Knapp Date: Thu, 10 Sep 2026 19:24:05 -0700 Subject: [PATCH 1/2] Read a stopped container instead of claiming there is nothing to read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A project that was merely stopped reported "This project has no container or snapshot image yet, so there is nothing to compare against the base image" — with its container sitting right there — and Update stayed disabled. Start it and the checks passed, which is the tell: the staleness probe had only two sources, a *running* container via `docker exec` or the project's snapshot image. The snapshot is not a checkpoint. `commit_container_snapshot` runs only before a container is destroyed (a config-change recreate) or inside a migration, never on stop, so a project in daily use for a year can have no snapshot at all — and five of the six projects on the box that reported this had none. Absence of a snapshot was being read as absence of anything to inspect. So probe the stopped container directly: commit its writable layer to a throwaway image, probe that, drop it. A stopped container now also outranks the snapshot, for the same reason a running one already did — the snapshot lags it by everything installed since the last commit. `pick_probe_source` is the whole decision and is unit-tested; the message it used to emit now describes only the case it is true of, no container and no snapshot. Two things found on the way, both documented in CLAUDE.md: `bollard` never hands back the image id from a commit — its `Commit` model deserialises "ID" while the daemon sends "Id" — so the probe image has to be tagged, and a tagged image is dangling-proof and therefore invisible to `sweep_orphaned_snapshots`, `reap_stale_migration_pins` and `scrub_secrets_from_snapshots` alike. Without a reaper of its own a crashed probe would leak a multi-gigabyte image that nothing could ever reclaim, so `reap_probe_images` runs at startup beside `reap_probe_containers`, age-gated for the same reason that one is: `reference=` is daemon-wide and a second instance's live probe matches the glob. It removes by tag, never by image id: a force removal by id untags an image everywhere, which is how a first draft of the reaper test deleted an unrelated `alpine:latest`. Names are unique per call rather than stable per container, because container ids do not survive a recreate and two overlapping probes would otherwise fight over one tag. Verified against the container that reported the bug: 13,365 paths and an apt delta of cmake, ffmpeg, libobs-dev, qt6-base-dev and nine more — the migration payload the Update flow could not see. 546 unit tests plus three live-Docker tests pass; no new clippy warnings. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz --- CLAUDE.md | 33 ++ .../src/commands/migration_commands.rs | 131 +++++++- app/src-tauri/src/docker/container.rs | 144 ++++++++- app/src-tauri/src/docker/migration.rs | 289 ++++++++++++++++++ app/src-tauri/src/lib.rs | 10 +- 5 files changed, 591 insertions(+), 16 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 837cc43..545f969 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -436,6 +436,39 @@ security update. Migration is the non-destructive way out; Reset is the destruct bump: churn on the old base, and it would consume the "you should migrate" signal without migrating. `get_container_staleness` surfaces it; `migrate_project_to_base` acts on it. - **A missing lineage label means "unknown, probe instead", never "stale".** +- **The snapshot image is not a checkpoint — never read its absence as "nothing to inspect".** + `commit_container_snapshot` runs only before a container is destroyed (a config-change recreate) + or inside a migration. **Never on stop.** So a project in daily use for a year can legitimately + have no `triple-c-snapshot-{id}:latest` at all, and one that has is stale by everything installed + since. `pick_probe_source` therefore reads a *stopped* container directly — commit its writable + layer to `triple-c-probe-{cid}:latest`, probe that, drop it — and ranks it **above** the snapshot, + for the same reason a running container already outranked it. Assuming a snapshot existed is what + made a stopped, never-recreated project report "no container or snapshot image yet" with its + container sitting right there, and left Update disabled on the projects furthest behind. +- **`bollard` never gives you the image id back from a commit.** Its `Commit` response model + deserialises `"ID"`; the daemon sends `"Id"`, so `commit_container` returns `id: None` every time + (verified: bollard 0.18.1, Engine 29.6). Neither long-standing commit site notices because both + discard the response — but it means any commit you need a *reference* to has to be **tagged**. +- **A tagged leftover is the one orphan no sweep can reach, so the probe image has its own reaper.** + `sweep_orphaned_snapshots` collects `dangling` + `triple-c.managed=true`; `reap_stale_migration_pins` + and `scrub_secrets_from_snapshots` both filter `triple-c-snapshot-*`. A `triple-c-probe-*` image is + tagged and so matches none of them, which would make a crashed probe a permanent multi-gigabyte + leak with no UI to find it. `reap_probe_images` runs at startup beside `reap_probe_containers` and + is **load-bearing, not tidying** — it is also what makes the probe image's unscrubbed writable + layer acceptable. Two rules it earned the hard way: + - **Age-gate it** (`PROBE_REAP_MIN_AGE_SECS`, same as the container reaper). `reference=` is + daemon-wide, so a second copy of the app has live probe images matching the glob. + - **Remove by tag, never by image id.** A `force` removal by id untags an image *everywhere*; a + fixture that tagged `alpine:latest` into this namespace deleted the user's alpine that way. +- **Probe image names are unique per call, and must stay that way.** A stable per-container name was + tried: container ids do not survive a recreate, so most leftovers were stranded permanently, and + two concurrent probes fought over one tag — whichever finished first force-removed the image the + other was still reading, reporting a bogus `probe_error` on a healthy project. `get_container_staleness` + takes no `project_lock` claim (the migration banner needs it to answer *during* a migration), so + uniqueness is what makes overlapping probes safe. +- **An image's `Created` is the image's own, not its tag's.** Tagging an existing image gives you + that image's age; BuildKit stamps `docker build` output with a fixed epoch. Only `docker commit` + stamps *now* — which is what real probe images do, and what any fixture for them must do. - **`:latest` keeps pointing at the old lineage until the final commit.** That is what makes every crash before that point self-heal — `start_project_container` just recreates from the old snapshot. After the container swap, the new container's `triple-c.migration-state=in-progress` diff --git a/app/src-tauri/src/commands/migration_commands.rs b/app/src-tauri/src/commands/migration_commands.rs index 6be50da..e97fc31 100644 --- a/app/src-tauri/src/commands/migration_commands.rs +++ b/app/src-tauri/src/commands/migration_commands.rs @@ -92,8 +92,72 @@ fn pick_recorded_lineage( .or_else(|| from_snapshot.filter(|v| !v.is_empty())) } -/// Read-only. Runs two filesystem probes (~3 s each) and is therefore meant to -/// be called on demand, not polled. +/// Reported as `probe_error` when there is genuinely nothing to read: no +/// container, stopped or otherwise, and no snapshot image. +/// +/// It used to be reported for a *stopped* container too, which was simply +/// untrue — the container was sitting right there — and it disabled Update on +/// exactly the long-lived projects that had never been recreated and so had no +/// snapshot to fall back on. +const NOTHING_TO_PROBE: &str = "This project has no container or snapshot image yet, so there is nothing to compare against the base image."; + +/// Where [`get_container_staleness`] reads the project's *current* filesystem +/// from, in descending order of how current the answer is. +#[derive(Debug, PartialEq, Eq)] +enum ProbeSource { + /// `docker exec` into the live container. The only source that includes + /// everything installed since the last commit *in this session*. + RunningContainer, + /// Commit the stopped container's writable layer to a throwaway image and + /// probe that. Exactly as current as the container, which is what makes it + /// preferable to the snapshot — see below. + StoppedContainer, + /// A throwaway container from `triple-c-snapshot-:latest`. + Snapshot, + /// Nothing to read: no container, no snapshot. + Nothing, +} + +/// Pick the probe source. `container_running` is `None` when the project has no +/// container at all, `Some(false)` when it has a stopped one. +/// +/// **A stopped container outranks the snapshot.** The snapshot image is not a +/// checkpoint — `commit_container_snapshot` runs only before a removal (a +/// config-change recreate) or inside a migration, so a project that has never +/// hit either has *no snapshot at all*, however long it has been in use, and +/// one that has is stale by everything installed since. The container's +/// writable layer is the truth in both cases. This is the same argument +/// [`mig::manifest_from_container`] already makes for the running case; it does +/// not stop applying when the container is stopped. +/// +/// Getting this wrong is what made a stopped, never-recreated project report +/// "no container or snapshot image yet" — with its container sitting right +/// there — and left Update disabled on the projects that most needed it. +fn pick_probe_source(container_running: Option, snapshot_exists: bool) -> ProbeSource { + match (container_running, snapshot_exists) { + (Some(true), _) => ProbeSource::RunningContainer, + (Some(false), _) => ProbeSource::StoppedContainer, + (None, true) => ProbeSource::Snapshot, + (None, false) => ProbeSource::Nothing, + } +} + +/// Runs two filesystem probes (~3 s each) and is therefore meant to be called +/// on demand, not polled. +/// +/// **Not read-only, despite only reporting.** The stopped-container path commits +/// a throwaway image and force-removes it, which makes this a writer of a +/// `triple-c-probe-*` image and puts it in the class of thing +/// [`crate::project_lock`] exists for — and it takes no claim. That is +/// deliberate: this is what the migration banner calls to decide whether to +/// offer an update, including while a migration is in flight, so refusing it +/// under a claim would blank the banner exactly when it has the most to say. +/// The exposure is bounded to a surfaced error — a concurrent Recreate, Reset or +/// migration can remove the container out from under the commit, and the result +/// is a `probe_error` the user can retry, never a damaged container or a +/// mislabelled image. Two overlapping probes cannot collide either, because +/// probe image names are unique per call; see +/// [`crate::docker::container::get_probe_image_name`]. #[tauri::command] pub async fn get_container_staleness( project_id: String, @@ -145,16 +209,25 @@ pub async fn get_container_staleness( }; // ── Probes ─────────────────────────────────────────────────────────── - let running = match &container_id { - Some(id) => docker::is_container_running(id).await.unwrap_or(false), - None => false, + let container_running = match &container_id { + Some(id) => Some(docker::is_container_running(id).await.unwrap_or(false)), + None => None, }; - let from_manifest = if running { - mig::manifest_from_container(container_id.as_ref().unwrap()).await - } else if docker::image_exists(&snapshot_image).await.unwrap_or(false) { - mig::manifest_from_image(&snapshot_image).await - } else { - Err("This project has no container or snapshot image yet, so there is nothing to compare against the base image.".to_string()) + let snapshot_exists = docker::image_exists(&snapshot_image).await.unwrap_or(false); + let from_manifest = match ( + pick_probe_source(container_running, snapshot_exists), + &container_id, + ) { + (ProbeSource::RunningContainer, Some(id)) => mig::manifest_from_container(id).await, + (ProbeSource::StoppedContainer, Some(id)) => { + mig::manifest_from_stopped_container(id).await + } + (ProbeSource::Snapshot, _) => mig::manifest_from_image(&snapshot_image).await, + // `container_running` is `Some` exactly when `container_id` is, so the + // two arms above are the only ones those variants can reach. This arm + // is `ProbeSource::Nothing` — and now *only* that: it used to also + // swallow every stopped container, which is the bug. + (_, _) => Err(NOTHING_TO_PROBE.to_string()), }; let (from_manifest, base_manifest) = match from_manifest { @@ -1964,6 +2037,42 @@ mod tests { assert_eq!(pick_recorded_lineage(some(""), None), None); } + #[test] + fn a_stopped_container_is_probed_rather_than_reported_missing() { + // The regression: a container that exists but is stopped, with no + // snapshot ever taken, read as "nothing to compare against". + assert_eq!( + pick_probe_source(Some(false), false), + ProbeSource::StoppedContainer + ); + } + + #[test] + fn the_container_outranks_the_snapshot_whether_or_not_it_is_running() { + // The snapshot lags the container by everything installed since the + // last commit, in both states. + assert_eq!( + pick_probe_source(Some(true), true), + ProbeSource::RunningContainer + ); + assert_eq!( + pick_probe_source(Some(false), true), + ProbeSource::StoppedContainer + ); + } + + #[test] + fn the_snapshot_is_the_fallback_only_once_the_container_is_gone() { + assert_eq!(pick_probe_source(None, true), ProbeSource::Snapshot); + } + + #[test] + fn nothing_to_probe_is_reserved_for_no_container_and_no_snapshot() { + // The one case the "no container or snapshot image yet" message may + // still describe. + assert_eq!(pick_probe_source(None, false), ProbeSource::Nothing); + } + #[test] fn byte_sizes_read_the_way_a_disk_warning_should() { assert_eq!(human_bytes(512), "512 B"); diff --git a/app/src-tauri/src/docker/container.rs b/app/src-tauri/src/docker/container.rs index 038e837..6bda607 100644 --- a/app/src-tauri/src/docker/container.rs +++ b/app/src-tauri/src/docker/container.rs @@ -3052,6 +3052,118 @@ fn blanked_secret_env() -> Vec { .collect() } +/// Image-name prefix for the throwaway commit a staleness probe of a stopped +/// container makes. The reaper's only handle on a leftover — see +/// [`crate::docker::migration::reap_probe_images`] — so nothing else may use it. +pub const PROBE_IMAGE_PREFIX: &str = "triple-c-probe-"; + +/// The throwaway image a staleness probe of a **stopped** container commits to. +/// +/// **Unique per call**, and both halves of the name earn their place: the +/// container id prefix makes a leftover traceable in `docker images`, and the +/// counter makes two overlapping probes independent. +/// +/// An earlier version of this was deliberately *stable* per container, on the +/// theory that the next probe would move the tag off an abandoned image and +/// leave it dangling for [`sweep_orphaned_snapshots`]. That was wrong twice +/// over. A container id does not survive a recreate, so for most leftovers +/// there is no "next probe of the same container" and the image was stranded +/// permanently; and a stable name made two concurrent probes fight over one +/// tag, where whichever finished first force-removed the image the other was +/// still reading and turned a healthy project into a bogus `probe_error`. +/// Uniqueness fixes both, and [`crate::docker::migration::reap_probe_images`] +/// is what collects the leftovers instead. +pub fn get_probe_image_name(container_id: &str) -> String { + use std::sync::atomic::{AtomicU64, Ordering}; + static SEQ: AtomicU64 = AtomicU64::new(0); + + let short: String = container_id.chars().take(12).collect(); + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0); + format!( + "{}{}-{}-{}:latest", + PROBE_IMAGE_PREFIX, + short, + nanos, + SEQ.fetch_add(1, Ordering::Relaxed) + ) +} + +/// Commit a **stopped** container's filesystem to a throwaway image, returning +/// its name. The caller owns the image and must remove it. +/// +/// This exists so a stopped project can be read at all. `docker exec` needs a +/// running container and the snapshot image is not a checkpoint — see +/// [`crate::commands::migration_commands`]'s probe-source pick — so without +/// this there is no way to see inside a project that is merely stopped. +/// +/// ## Why it is tagged at all +/// +/// An untagged commit would be tidier: untagged plus the `triple-c.managed=true` +/// that `docker commit` copies off the container is exactly the pair +/// [`sweep_orphaned_snapshots`] already collects, so a leftover would self-heal +/// with no new machinery. **It is not available.** `bollard`'s `Commit` response +/// model deserialises `"ID"` while the daemon sends `"Id"`, so +/// `commit_container` hands back `id: None` every time and there is no +/// reference left to probe. Neither existing commit site notices, because both +/// discard the response. Verified against Engine 29.6, bollard 0.18.1. +/// +/// So the image needs a name, a tagged image is not dangling, and the sweep +/// therefore cannot be the safety net. [`crate::docker::migration::reap_probe_images`] +/// is, and [`get_probe_image_name`] carries the rest of that argument. +/// +/// ## What is in the image, and what is not +/// +/// `pause: false` because nothing is running — pausing a stopped container is +/// an error, the same reason [`recommit_without_secrets`]'s scratch commit +/// passes `false`. +/// +/// Secrets are blanked from the env for the same reason +/// [`commit_container_snapshot`] blanks them: the commit bakes the container's +/// full ENV into the image, and "it only lives a few seconds" is not a property +/// this function can promise after a crash. +/// +/// **The writable layer is committed unscrubbed, and that is unavoidable here.** +/// [`commit_container_snapshot`] runs [`scrub_writable_layer`] first precisely +/// because a commit stacks a layer and never rewrites one — but that scrub is a +/// `docker exec`, which is exactly what a stopped container cannot serve, and +/// scrubbing is not wanted anyway: the probe's whole job is to report the +/// filesystem as it actually is. What makes it acceptable is that this copies +/// bytes that are *already on this disk* in the container's own writable layer, +/// into an image that is never pushed, never created from, and reaped — so it +/// duplicates data inside one trust domain rather than widening it. That +/// argument depends on the reaping actually happening; treat +/// [`crate::docker::migration::reap_probe_images`] as load-bearing, not tidying. +pub async fn commit_container_for_probe(container_id: &str) -> Result { + let docker = get_docker()?; + let image_name = get_probe_image_name(container_id); + let (repo, tag) = image_name + .rsplit_once(':') + .map(|(r, t)| (r.to_string(), t.to_string())) + .expect("get_probe_image_name always emits a tag"); + + docker + .commit_container( + CommitContainerOptions { + container: container_id.to_string(), + repo, + tag, + pause: false, + ..Default::default() + }, + Config:: { + env: Some(blanked_secret_env()), + ..Default::default() + }, + ) + .await + .map_err(|e| format!("Failed to commit stopped container {}: {}", container_id, e))?; + + Ok(image_name) +} + /// Whether `env` (an image's `Config.Env`) holds a non-empty value for any /// name in [`SECRET_ENV_KEYS`]. fn env_holds_a_secret(env: &[String]) -> bool { @@ -3518,9 +3630,10 @@ pub async fn remove_snapshot_image(project: &Project) -> Result<(), String> { remove_image_by_name(&get_snapshot_image_name(project)).await } -/// Remove a Docker image by name/tag, treating "does not exist" as success. -/// Shared by [`remove_snapshot_image`] and the pending-cleanup retry, which -/// only has the image name (the project record is already gone by then). +/// Remove a Docker image by name, tag or **id**, treating "does not exist" as +/// success. Shared by [`remove_snapshot_image`], the pending-cleanup retry +/// (which only has the image name — the project record is already gone by +/// then), and the staleness probe's throwaway commit, which has only an id. pub async fn remove_image_by_name(image_name: &str) -> Result<(), String> { let docker = get_docker()?; @@ -3536,7 +3649,7 @@ pub async fn remove_image_by_name(image_name: &str) -> Result<(), String> { .await { Ok(_) => { - log::info!("Removed snapshot image {}", image_name); + log::info!("Removed image {}", image_name); Ok(()) } Err(bollard::errors::Error::DockerResponseServerError { @@ -4464,6 +4577,29 @@ mod tests { assert!(env_holds_a_secret(&env)); } + /// The probe image's name must be **unique per call**. A stable name was + /// tried and is wrong twice over: a container id does not survive a + /// recreate, so a crashed probe's leftover would never be reclaimed by "the + /// next probe of the same container"; and two concurrent probes sharing one + /// tag means whichever finishes first force-removes the image the other is + /// still reading. See `commit_container_for_probe` and `reap_probe_images`. + #[test] + fn probe_image_names_are_unique_per_call_and_reapable_by_prefix() { + let id = "75993e6d5e1ab473b029a408c5ff0339"; + let a = get_probe_image_name(id); + let b = get_probe_image_name(id); + assert_ne!(a, b, "two probes of one container must not share a tag"); + + // The prefix is the reaper's only handle on a leftover, so every name + // has to carry it — and it must not be the snapshot namespace, which is + // what a project is rebuilt from. + assert!(a.starts_with(PROBE_IMAGE_PREFIX), "{}", a); + assert!(!a.starts_with("triple-c-snapshot-"), "{}", a); + // Traceable back to its container, which is the point of the prefix. + assert!(a.contains("75993e6d5e1a"), "{}", a); + assert!(a.ends_with(":latest"), "{}", a); + } + #[test] fn the_scrub_report_only_claims_success_when_nothing_is_left() { let clean = SnapshotScrubReport { diff --git a/app/src-tauri/src/docker/migration.rs b/app/src-tauri/src/docker/migration.rs index 3425970..a5332c0 100644 --- a/app/src-tauri/src/docker/migration.rs +++ b/app/src-tauri/src/docker/migration.rs @@ -886,6 +886,100 @@ pub async fn reap_probe_containers() { } } +/// Remove throwaway images left behind by a staleness probe of a stopped +/// container — [`super::container::commit_container_for_probe`]'s commits. +/// +/// **Load-bearing, not tidying.** A probe image is *tagged*, because bollard +/// gives no image id back from a commit and there has to be something to probe. +/// Tagged means not dangling, so [`super::container::sweep_orphaned_snapshots`] +/// — which collects every other kind of orphan this app can leave — will never +/// see one. Without this, a probe that dies between its commit and its own +/// cleanup (SIGKILL, a crash, a 409 from a concurrent remove) strands a +/// multi-gigabyte image that **no code path can ever reclaim**, and there is no +/// UI to find it either. That is the one leak in this app with no floor on it, +/// so this runs at startup beside [`reap_probe_containers`]. +/// +/// Age-gated for exactly the reason that one is: `reference=` is a daemon-wide +/// filter, so a second copy of the app probing a project on the same daemon has +/// images matching this glob, and removing one mid-capture fails that probe with +/// "No such image" — the bogus `probe_error` the staleness work exists to get +/// rid of. In-process state cannot see the other instance, so age is the only +/// brake, and [`PROBE_REAP_MIN_AGE_SECS`] is already the right one: a probe is a +/// `find` over a root filesystem, not a multi-minute job. +/// +/// Never fails the caller. Housekeeping, like every other sweep here. +pub async fn reap_probe_images() { + use bollard::image::{ListImagesOptions, RemoveImageOptions}; + + let docker = match get_docker() { + Ok(d) => d, + Err(e) => { + log::warn!("Could not reap leftover probe images: {}", e); + return; + } + }; + + let filters = HashMap::from([( + "reference".to_string(), + vec![format!("{}*", super::container::PROBE_IMAGE_PREFIX)], + )]); + let images = match docker + .list_images(Some(ListImagesOptions { + all: false, + filters, + ..Default::default() + })) + .await + { + Ok(images) => images, + Err(e) => { + log::warn!("Could not list leftover probe images: {}", e); + return; + } + }; + + let now = chrono::Utc::now().timestamp(); + for image in images { + // Unlike a container summary, an image summary always carries a + // `Created`, so there is no unknown-age case to defend against here. + if now - image.created < PROBE_REAP_MIN_AGE_SECS { + log::info!( + "Leaving probe image {:?} alone — it is younger than {} minutes, so it may belong \ + to another Triple-C instance's live probe", + image.repo_tags, + PROBE_REAP_MIN_AGE_SECS / 60 + ); + continue; + } + // By **tag**, never by image id. A `force` removal by id untags an + // image everywhere, so an id that happens to carry another name loses + // that name too — which is how a test fixture that tagged + // `alpine:latest` into this namespace deleted the user's alpine. A real + // leftover has exactly the one probe tag, so removing the tag removes + // the image; anything else keeps whatever other names it has. + for tag in image + .repo_tags + .iter() + .filter(|t| t.starts_with(super::container::PROBE_IMAGE_PREFIX)) + { + log::info!("Removing leftover probe image {}", tag); + if let Err(e) = docker + .remove_image( + tag, + Some(RemoveImageOptions { + force: true, + noprune: false, + }), + None, + ) + .await + { + log::warn!("Could not remove leftover probe image {}: {}", tag, e); + } + } + } +} + /// How old a `triple-c.probe=migration` container must be before /// [`reap_probe_containers`] will force-remove it, in seconds. /// @@ -993,6 +1087,36 @@ pub async fn manifest_from_container(container_id: &str) -> Result Result { + let image = super::container::commit_container_for_probe(container_id).await?; + + let manifest = manifest_from_image(&image) + .await + .map_err(|e| format!("Probe of the stopped container did not complete: {}", e)); + + if let Err(e) = super::container::remove_image_by_name(&image).await { + log::warn!( + "Could not remove the staleness probe's throwaway image {}: {} — the next probe of \ + this container reuses the name, which leaves this one dangling for the orphan sweep", + image, + e + ); + } + + manifest +} + /// The image ID (`sha256:…`) of a local image, or `None` if it is not present. /// /// Deliberately the **ID**, not a repo digest: locally built images and custom @@ -2146,4 +2270,169 @@ mod tests { assert!(!pin_is_reapable("pre-migration-handmade", false, ancient, &now)); assert!(!pin_is_reapable("latest", false, ancient, &now)); } + + // ── Live Docker ───────────────────────────────────────────────────────── + + /// The reaper finds a leftover probe image by prefix and — crucially — + /// refuses to remove a young one, because that image may be another + /// Triple-C instance's live probe. Only a real daemon can say whether the + /// `reference=` glob matches the names `get_probe_image_name` produces. + /// + /// The fixture is **committed**, not tagged and not built. An image's + /// `Created` is its own, not its tag's, so tagging something already on disk + /// into this namespace yields a fixture the reaper is right to call ancient + /// — and BuildKit stamps a fixed epoch on `docker build` output, so a built + /// one looks ancient too. A commit stamps *now*, verified against Engine + /// 29.6, which is also how real probe images get their age. + /// + /// Both of those mistakes were made here first, and one of them deleted an + /// unrelated `alpine:latest` — which is why `reap_probe_images` removes by + /// tag rather than by image id. + /// + /// ```text + /// cargo test -- --ignored --nocapture reaper_spares + /// ``` + #[cfg(unix)] + #[tokio::test] + #[ignore = "needs a Docker daemon; builds and removes a throwaway image"] + async fn the_reaper_spares_a_probe_image_young_enough_to_be_someone_elses() { + use std::process::Command; + + fn docker_out(args: &[&str]) -> std::process::Output { + Command::new("docker").args(args).output().expect("docker CLI") + } + + let base = std::env::var("TRIPLE_C_TEST_IMAGE") + .unwrap_or_else(|_| "alpine:latest".to_string()); + let name = crate::docker::container::get_probe_image_name("reapertest01234"); + + // A never-started container is enough to commit from, and leaves the + // daemon's run state alone entirely. + let created = docker_out(&["create", &base, "true"]); + assert!( + created.status.success(), + "could not create the fixture container from {}: {}", + base, + String::from_utf8_lossy(&created.stderr) + ); + let cid = String::from_utf8_lossy(&created.stdout).trim().to_string(); + + let committed = docker_out(&["commit", "--pause=false", &cid, &name]); + let _ = docker_out(&["rm", "-f", &cid]); + assert!( + committed.status.success(), + "could not commit the fixture image: {}", + String::from_utf8_lossy(&committed.stderr) + ); + + reap_probe_images().await; + + let still_there = Command::new("docker") + .args(["image", "inspect", &name]) + .output() + .expect("docker image inspect") + .status + .success(); + + let _ = Command::new("docker").args(["rmi", &name]).output(); + + assert!( + still_there, + "a probe image committed seconds ago was reaped — that is another \ + instance's live probe being broken, see PROBE_REAP_MIN_AGE_SECS" + ); + } + + /// A *stopped* container is readable, and what comes back is its writable + /// layer rather than the image it was created from. This is the whole point + /// of the function: the base image cannot answer it, and the project may + /// well have no snapshot image at all. + /// + /// Also asserts the throwaway commit leaves nothing behind, which no unit + /// test can. It has to assert on the `triple-c-probe-*` tags specifically: + /// the probe image is *tagged*, so a leak never shows up as a dangling + /// image and a dangling-set assertion here would pass either way. + /// + /// Ignored because it needs Docker and commits a container; run it with + /// + /// ```text + /// cargo test -- --ignored --nocapture stopped_container + /// ``` + #[cfg(unix)] + #[tokio::test] + #[ignore = "needs a Docker daemon; creates, commits and removes a throwaway container"] + async fn a_stopped_container_is_read_from_its_writable_layer() { + fn docker_cli(args: &[&str]) -> String { + let out = std::process::Command::new("docker") + .args(args) + .output() + .expect("docker CLI"); + assert!( + out.status.success(), + "docker {:?} failed: {}", + args, + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8_lossy(&out.stdout).trim().to_string() + } + fn probe_images() -> Vec { + let mut ids: Vec = docker_cli(&[ + "images", "-q", + "--filter", + &format!("reference={}*", crate::docker::container::PROBE_IMAGE_PREFIX), + ]) + .lines() + .map(|l| l.trim().to_string()) + .filter(|l| !l.is_empty()) + .collect(); + ids.sort(); + ids + } + + let image = std::env::var("TRIPLE_C_TEST_IMAGE") + .unwrap_or_else(|_| "ghcr.io/shadowdao/triple-c-sandbox:latest".to_string()); + // A marker only the writable layer can carry, under a MANIFEST_ROOTS root. + let marker = format!("/opt/probe-marker-{}", std::process::id()); + + // Another instance's live probe images are allowed to exist; what must + // hold is that this probe adds none of its own. + let before = probe_images(); + + let id = docker_cli(&[ + "run", "-d", "--label", "triple-c.managed=true", + "--entrypoint", "/bin/sh", + &image, "-c", "sleep 300", + ]); + let cleanup = |id: &str| { + let _ = std::process::Command::new("docker") + .args(["rm", "-f", id]) + .output(); + }; + + docker_cli(&["exec", &id, "mkdir", "-p", &marker]); + docker_cli(&["stop", "-t", "1", &id]); + + let result = manifest_from_stopped_container(&id).await; + + cleanup(&id); + + let manifest = result.expect("a stopped container must be probeable"); + assert!( + manifest.paths.iter().any(|e| e.path == marker && e.is_dir()), + "the probe read the image, not the container's writable layer: {} missing", + marker + ); + // Non-empty package sets prove the probe script really ran, rather than + // parsing an empty transcript into an empty-but-Ok manifest. + assert!( + !manifest.apt_manual.is_empty(), + "apt-mark showmanual came back empty, so the probe did not run" + ); + + assert_eq!( + probe_images(), + before, + "the throwaway probe image was not cleaned up" + ); + } } diff --git a/app/src-tauri/src/lib.rs b/app/src-tauri/src/lib.rs index 950cf35..dc62069 100644 --- a/app/src-tauri/src/lib.rs +++ b/app/src-tauri/src/lib.rs @@ -263,12 +263,20 @@ pub fn run() { // logged warning rather than a failed start. // // Ordering matters. Probes are removed first because a probe holds - // an image open and the sweep will not force; pins are untagged + // an image open and the sweep will not force — both the probe + // containers and the probe images, the latter being the one orphan + // the sweep can never reach on its own; pins are untagged // second so the images they were holding are dangling by the time // the sweep lists them; the sweep runs last and collects both. let projects_store_for_cleanup = projects_store_setup.clone(); tauri::async_runtime::spawn(async move { crate::docker::reap_probe_containers().await; + // Probe *images* too, and for a sharper reason: a probe + // container merely pins an image the sweep then refuses to + // touch, whereas a leftover probe image is tagged and so + // nothing else in this app can ever collect it. See + // `reap_probe_images`. + crate::docker::reap_probe_images().await; let reaped = crate::docker::reap_stale_migration_pins().await; if reaped > 0 { log::info!("Startup housekeeping dropped {} stale rollback pin(s)", reaped); From 95a78fe9a319552bc33ef5b2bca9d7ba20bb4e77 Mon Sep 17 00:00:00 2001 From: Josh Knapp Date: Thu, 10 Sep 2026 20:48:10 -0700 Subject: [PATCH 2/2] Take the review: cache the stopped probe, and never let it cost an answer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six findings, all real. The one that mattered: `getContainerStaleness` is called from a `useEffect` that fires whenever the container settles, so merely opening a stopped project's Overview now committed its whole writable layer — 44 s on a real project, against ~3 s for the snapshot probe it replaced. Shipping that would have traded one bad banner for a bad page. A stopped container's writable layer cannot change, so the probe is exactly cacheable: `STOPPED_MANIFEST_CACHE` keys on the container's `FinishedAt`, which moves on every stop. Cold 2967 ms, warm 1 ms, measured. A live test asserts the restart case as well as the hit, because a cache that failed to invalidate would plan a migration against a filesystem the project no longer has — verified by breaking the token and watching that assertion fail. Skipping the probe for projects that are not stale looked like the cheaper fix and is unsafe: the deltas would be empty while `probeSettled` stayed true, and the migrate action in the project menu is not gated on the banner, so the pre-flight would report nothing to copy while the backend was told to copy nothing. That is the hazard `canMigrate`'s comment already warns about. Not done, and written down so it is not tried again. Also from the review: - A failed commit no longer costs an answer the snapshot could have given. Before this feature a stopped project read its snapshot directly, so surfacing this error would have made the banner worse than it was — and the failure modes are where the fallback earns its keep: a full disk (the commit allocates the whole layer, the snapshot probe allocates nothing) and a 409 from a concurrent claim. - The probe no longer commits while the project is claimed. The collision is not symmetric: the probe losing is a retryable `probe_error`, but `start_project_container` removes the old container with a hard `?`, so a remove that raced a commit would fail the user's Start with an opaque error. `stopped_probe_policy` reads `project_lock::held` and probes the snapshot instead, or defers with a message that says so. - The cleanup-failure warning claimed the next probe of the same container would reclaim the leftover. Unique names made that false the moment they landed; it is `reap_probe_images` that collects it. - The TS binding still called the command read-only, which is how the auto-refresh got added in the first place. - CLAUDE.md still documented the stable `triple-c-probe-{cid}:latest` name this PR removed as unsafe. 548 unit tests, 752 frontend tests, 4 live-Docker tests. Clippy unchanged at 44 warnings. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019RSaoDLovVV2wmH4H8VVxz --- CLAUDE.md | 26 ++- .../src/commands/migration_commands.rs | 95 +++++++++- app/src-tauri/src/docker/migration.rs | 176 +++++++++++++++++- app/src/lib/tauri-commands.ts | 18 +- 4 files changed, 308 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 545f969..b4addc2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -441,7 +441,7 @@ security update. Migration is the non-destructive way out; Reset is the destruct or inside a migration. **Never on stop.** So a project in daily use for a year can legitimately have no `triple-c-snapshot-{id}:latest` at all, and one that has is stale by everything installed since. `pick_probe_source` therefore reads a *stopped* container directly — commit its writable - layer to `triple-c-probe-{cid}:latest`, probe that, drop it — and ranks it **above** the snapshot, + layer to a unique `triple-c-probe-*` image, probe that, drop it — and ranks it **above** the snapshot, for the same reason a running container already outranked it. Assuming a snapshot existed is what made a stopped, never-recreated project report "no container or snapshot image yet" with its container sitting right there, and left Update disabled on the projects furthest behind. @@ -466,6 +466,30 @@ security update. Migration is the non-destructive way out; Reset is the destruct other was still reading, reporting a bogus `probe_error` on a healthy project. `get_container_staleness` takes no `project_lock` claim (the migration banner needs it to answer *during* a migration), so uniqueness is what makes overlapping probes safe. +- **The stopped-container probe is cached per stop, and that is not an optimisation you may drop.** + `getContainerStaleness` is called from a `useEffect` that fires whenever the container settles, so + merely opening a stopped project's Overview probes it. Uncached that is a `docker commit` of the + whole writable layer per visit — measured at 44 s on a real project, against ~3 s for the snapshot + probe it replaced. `STOPPED_MANIFEST_CACHE` is keyed on the container's `FinishedAt`, which is + exact rather than merely plausible: nothing can write to a stopped container's writable layer, and + `FinishedAt` moves on every stop. A live test asserts the restart case, because a cache that + failed to invalidate would plan a migration against a filesystem the project no longer has. +- **Do not "skip the probe when the project is not stale" to save that cost.** It was tried. The + deltas would be empty while `probeSettled` (`!probing && staleness && !probe_error`) stayed *true*, + which leaves the migrate action in the project menu enabled — that action is not gated on the + banner — so the pre-flight would report nothing to copy while the backend was told to copy + nothing. That is the exact hazard `ProjectHome.tsx`'s `canMigrate` comment already warns about. +- **A failed stopped-container probe falls back to the snapshot whenever one exists.** Before this + feature a stopped project read its snapshot directly, so surfacing a commit failure where the + snapshot could have answered would make the banner *worse* than it was — and the failure modes are + exactly the ones where the fallback earns its keep: a full disk (the commit allocates the whole + writable layer; the snapshot probe allocates nothing) and a 409 from a concurrent claim. +- **`get_container_staleness` never commits while the project is claimed.** It takes no + `project_lock` claim itself, deliberately — the banner has to answer *during* a migration — so it + reads `project_lock::held` instead and probes the snapshot rather than the container. The + collision is not symmetric: the probe losing is a retryable `probe_error`, but + `start_project_container` removes the old container with a hard `?`, so a remove that raced a + commit would fail the user's Start with an opaque error. - **An image's `Created` is the image's own, not its tag's.** Tagging an existing image gives you that image's age; BuildKit stamps `docker build` output with a fixed epoch. Only `docker commit` stamps *now* — which is what real probe images do, and what any fixture for them must do. diff --git a/app/src-tauri/src/commands/migration_commands.rs b/app/src-tauri/src/commands/migration_commands.rs index e97fc31..5d4c0de 100644 --- a/app/src-tauri/src/commands/migration_commands.rs +++ b/app/src-tauri/src/commands/migration_commands.rs @@ -142,6 +142,45 @@ fn pick_probe_source(container_running: Option, snapshot_exists: bool) -> } } +/// Reported as `probe_error` when another operation owns the project and there +/// is no snapshot image to read instead. Deliberately not a claim about the +/// container: nothing is wrong with it, the answer is simply not safe to take +/// right now. See [`stopped_probe_policy`]. +const PROJECT_BUSY: &str = "Another operation is running on this project, so its contents could not be inspected. Try again once it finishes."; + +/// What to do about a stopped container, whose probe is the expensive one: it +/// commits the writable layer before it can read anything. +#[derive(Debug, PartialEq, Eq)] +enum StoppedProbe { + /// Commit and probe. The current answer, and the default. + Commit, + /// Probe the snapshot image instead. Less current — it lags the container by + /// everything installed since the last commit — but it allocates nothing and + /// touches nothing, which is what makes it the right answer while another + /// operation owns the container. + SnapshotInstead, + /// Report rather than guess. + Defer, +} + +/// Pick what to do about a stopped container. +/// +/// **Never commits while the project is claimed.** `get_container_staleness` +/// takes no [`crate::project_lock`] claim of its own, by design, so a commit +/// here can overlap a Recreate or Reset — and the collision is not symmetric. +/// The probe losing is harmless: a surfaced `probe_error` the user retries. The +/// *recreate* losing is not, because `start_project_container` removes the old +/// container with a hard `?`, so a non-404 from a remove that raced this commit +/// fails the whole Start with an opaque "Failed to remove container". Reading +/// the claim costs nothing and takes that failure off the table. +fn stopped_probe_policy(project_is_busy: bool, snapshot_exists: bool) -> StoppedProbe { + match (project_is_busy, snapshot_exists) { + (false, _) => StoppedProbe::Commit, + (true, true) => StoppedProbe::SnapshotInstead, + (true, false) => StoppedProbe::Defer, + } +} + /// Runs two filesystem probes (~3 s each) and is therefore meant to be called /// on demand, not polled. /// @@ -220,7 +259,44 @@ pub async fn get_container_staleness( ) { (ProbeSource::RunningContainer, Some(id)) => mig::manifest_from_container(id).await, (ProbeSource::StoppedContainer, Some(id)) => { - mig::manifest_from_stopped_container(id).await + let busy = crate::project_lock::held(&project_id).is_some(); + match stopped_probe_policy(busy, snapshot_exists) { + StoppedProbe::Commit => { + match mig::manifest_from_stopped_container_cached(id).await { + Ok(m) => Ok(m), + // **Never let a failed commit cost an answer the + // snapshot could have given.** Before stopped + // containers were readable at all, a stopped project + // fell straight through to its snapshot, so surfacing + // this error where the snapshot exists would make the + // banner *worse* than it was — and the ways this fails + // are the ones where the fallback matters most: a full + // disk (the commit has to allocate the whole writable + // layer; the snapshot probe allocates nothing) and a + // 409 from an operation that claimed the project after + // the check above. + Err(e) if snapshot_exists => { + log::warn!( + "Probing the stopped container for project {} failed ({}) — \ + falling back to its snapshot image, which may lag it", + project_id, + e + ); + mig::manifest_from_image(&snapshot_image).await + } + Err(e) => Err(e), + } + } + StoppedProbe::SnapshotInstead => { + log::info!( + "Project {} is claimed by another operation — probing its snapshot image \ + rather than committing the container", + project_id + ); + mig::manifest_from_image(&snapshot_image).await + } + StoppedProbe::Defer => Err(PROJECT_BUSY.to_string()), + } } (ProbeSource::Snapshot, _) => mig::manifest_from_image(&snapshot_image).await, // `container_running` is `Some` exactly when `container_id` is, so the @@ -2073,6 +2149,23 @@ mod tests { assert_eq!(pick_probe_source(None, false), ProbeSource::Nothing); } + #[test] + fn a_stopped_container_is_committed_only_when_nothing_else_owns_the_project() { + assert_eq!(stopped_probe_policy(false, false), StoppedProbe::Commit); + assert_eq!(stopped_probe_policy(false, true), StoppedProbe::Commit); + } + + #[test] + fn a_busy_project_falls_back_rather_than_racing_a_recreate() { + // The snapshot lags, but a stale answer beats failing someone's Start. + assert_eq!( + stopped_probe_policy(true, true), + StoppedProbe::SnapshotInstead + ); + // Nothing to fall back to: say so instead of committing anyway. + assert_eq!(stopped_probe_policy(true, false), StoppedProbe::Defer); + } + #[test] fn byte_sizes_read_the_way_a_disk_warning_should() { assert_eq!(human_bytes(512), "512 B"); diff --git a/app/src-tauri/src/docker/migration.rs b/app/src-tauri/src/docker/migration.rs index a5332c0..67c242d 100644 --- a/app/src-tauri/src/docker/migration.rs +++ b/app/src-tauri/src/docker/migration.rs @@ -1087,6 +1087,89 @@ pub async fn manifest_from_container(container_id: &str) -> Result>, +> = std::sync::Mutex::new(None); + +/// How many stopped-container manifests [`STOPPED_MANIFEST_CACHE`] keeps. +const STOPPED_MANIFEST_CACHE_MAX: usize = 4; + +/// `FinishedAt` for a container, the cache's validity token. `None` when it +/// cannot be read, which is never treated as a hit. +async fn container_finished_at(container_id: &str) -> Option { + let docker = get_docker().ok()?; + docker + .inspect_container(container_id, None) + .await + .ok()? + .state? + .finished_at + .filter(|s| !s.is_empty()) +} + +/// Capture a [`Manifest`] from a **stopped** container, reusing a cached one +/// when the container has not been started since it was taken. +/// +/// See [`STOPPED_MANIFEST_CACHE`] for why this is exact and why it is needed. +pub async fn manifest_from_stopped_container_cached( + container_id: &str, +) -> Result { + let finished_at = container_finished_at(container_id).await; + + if let Some(token) = &finished_at { + let guard = STOPPED_MANIFEST_CACHE.lock(); + if let Ok(cache) = guard { + if let Some(entries) = cache.as_ref() { + if let Some((_, _, manifest)) = entries + .iter() + .find(|(id, tok, _)| id == container_id && tok == token) + { + log::debug!( + "Reusing the cached manifest for stopped container {}", + container_id + ); + return Ok(manifest.clone()); + } + } + } + } + + let manifest = manifest_from_stopped_container(container_id).await?; + + // Only cacheable if the container's state could be read at all; an unknown + // `FinishedAt` means there is no token that could later be compared. + if let Some(token) = finished_at { + if let Ok(mut cache) = STOPPED_MANIFEST_CACHE.lock() { + let entries = cache.get_or_insert_with(Vec::new); + entries.retain(|(id, _, _)| id != container_id); + entries.push((container_id.to_string(), token, manifest.clone())); + while entries.len() > STOPPED_MANIFEST_CACHE_MAX { + entries.remove(0); + } + } + } + + Ok(manifest) +} + /// Capture a [`Manifest`] from a **stopped** container. /// /// Commits the container's writable layer to a throwaway image, probes that, @@ -1107,8 +1190,8 @@ pub async fn manifest_from_stopped_container(container_id: &str) -> Result String { + let out = std::process::Command::new("docker") + .args(args) + .output() + .expect("docker CLI"); + assert!( + out.status.success(), + "docker {:?} failed: {}", + args, + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8_lossy(&out.stdout).trim().to_string() + } + + let image = std::env::var("TRIPLE_C_TEST_IMAGE") + .unwrap_or_else(|_| "ghcr.io/shadowdao/triple-c-sandbox:latest".to_string()); + let first = format!("/opt/cache-marker-a-{}", std::process::id()); + let second = format!("/opt/cache-marker-b-{}", std::process::id()); + + let id = docker_cli(&[ + "run", "-d", "--label", "triple-c.managed=true", + "--entrypoint", "/bin/sh", + &image, "-c", "sleep 600", + ]); + let cleanup = || { + let _ = std::process::Command::new("docker") + .args(["rm", "-f", &id]) + .output(); + }; + + docker_cli(&["exec", &id, "mkdir", "-p", &first]); + docker_cli(&["stop", "-t", "1", &id]); + + let t0 = std::time::Instant::now(); + let cold = manifest_from_stopped_container_cached(&id).await; + let cold_ms = t0.elapsed().as_millis(); + + let t1 = std::time::Instant::now(); + let warm = manifest_from_stopped_container_cached(&id).await; + let warm_ms = t1.elapsed().as_millis(); + + // Restart, change the filesystem, stop again — `FinishedAt` moves. + docker_cli(&["start", &id]); + docker_cli(&["exec", &id, "mkdir", "-p", &second]); + docker_cli(&["stop", "-t", "1", &id]); + let after_restart = manifest_from_stopped_container_cached(&id).await; + + cleanup(); + + let has = |m: &Manifest, p: &str| m.paths.iter().any(|e| e.path == p && e.is_dir()); + + let cold = cold.expect("cold read"); + let warm = warm.expect("warm read"); + let after_restart = after_restart.expect("read after restart"); + + assert!(has(&cold, &first), "cold read missed {}", first); + assert!(has(&warm, &first), "warm read missed {}", first); + println!("cold {} ms, warm {} ms", cold_ms, warm_ms); + assert!( + warm_ms * 5 < cold_ms.max(5), + "the second read cost {} ms against a cold {} ms — it re-committed \ + instead of using the cache", + warm_ms, + cold_ms + ); + + // The restart must have invalidated it: the new directory has to show up. + assert!( + has(&after_restart, &second), + "a restart did not invalidate the cache — {} is missing, so this is \ + a stale manifest of a filesystem the container no longer has", + second + ); + assert!(has(&after_restart, &first), "the restart lost {}", first); + } + /// The reaper finds a leftover probe image by prefix and — crucially — /// refuses to remove a young one, because that image may be another /// Triple-C instance's live probe. Only a real daemon can say whether the diff --git a/app/src/lib/tauri-commands.ts b/app/src/lib/tauri-commands.ts index 49349e8..d6934f8 100644 --- a/app/src/lib/tauri-commands.ts +++ b/app/src/lib/tauri-commands.ts @@ -350,8 +350,8 @@ export const sweepClaudeTokenSnapshots = () => // without deleting its volumes. Reset is the destructive alternative: it wipes // ~/.claude, the OAuth credential, installed skills and every transcript. // -// Flow: getContainerStaleness (read-only, ~6s — two filesystem probes, so call -// it on demand rather than polling) → migrateProjectToBase → the project sits +// Flow: getContainerStaleness (~6s — two filesystem probes, so call it on demand +// rather than polling) → migrateProjectToBase → the project sits // in "awaiting-confirmation" while the user tries it → confirmMigration or // rollbackMigration. // @@ -361,7 +361,19 @@ export const sweepClaudeTokenSnapshots = () => // // Progress arrives on the existing `container-progress` event. -/** Read-only. Runs two container/image filesystem probes; not for polling. */ +/** + * Runs two container/image filesystem probes; not for polling. + * + * **Not read-only, despite only reporting.** When the container is *stopped* + * the backend has to commit its writable layer to a throwaway image before it + * can read anything — `docker exec` needs a running container — so this writes + * (and then removes) an image. The result is cached per stop, so repeat calls + * while the container stays stopped are cheap, but the first one after each stop + * pays for a commit of the whole layer: seconds on a small project, tens of + * seconds on a large one. Do not add a caller that fires more often than "the + * container settled into a new state" without re-reading + * `get_container_staleness`'s doc comment first. + */ export const getContainerStaleness = (projectId: string) => invoke("get_container_staleness", { projectId });