From eba0d957807b74fcac6d361fb285b42dd44ca3e2 Mon Sep 17 00:00:00 2001 From: Josh Knapp Date: Mon, 2 Nov 2026 15:57:12 -0800 Subject: [PATCH 1/2] Keep containers alive as root; notice containers that exit on their own The entrypoint ended in `exec su -s /bin/bash claude -c "exec sleep infinity"`, so a project container's whole lifetime was a sleep owned by the unprivileged claude user. Claude Code runs as claude and routinely runs `pkill sleep`, `pkill -f`, `kill -9 -1` while testing, and any of those stopped the container: every terminal disconnected and the app kept showing Running. Entrypoint: stay resident as root and wait on a root-owned sleep run as `triple-c-keepalive infinity`, respawned if killed, with a TERM/INT/HUP trap so `docker stop` returns at once (550 ms, exit 0, vs 2.6 s / 143 before). Unprivileged kills now get EPERM; `sudo pkill sleep` is survived by the respawn. The drop to claude dated from the initial commit with no stated purpose; its only reader was the marketplace readiness probe, which now accepts either generation of entrypoint. App: container_watch.rs holds one Docker event subscription filtered daemon-side to container `die`. A die naming a Running project's container is confirmed with an inspect, then ProjectsStore::mark_exited flips it to Stopped as a compare-and-set that skips any project a command holds the project lock on. Each (re)subscription re-checks Running projects and asks for events since just before it. The frontend applies the new `project-status-changed` event and toasts why the sessions died. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 13 +- app/src-tauri/src/container_watch.rs | 278 +++++++++++++++++++ app/src-tauri/src/lib.rs | 11 + app/src-tauri/src/marketplace/sync.rs | 25 +- app/src-tauri/src/storage/projects_store.rs | 131 +++++++++ app/src/App.tsx | 2 + app/src/hooks/useProjectStatusEvents.test.ts | 48 ++++ app/src/hooks/useProjectStatusEvents.ts | 39 +++ container/entrypoint.sh | 37 ++- 9 files changed, 574 insertions(+), 10 deletions(-) create mode 100644 app/src-tauri/src/container_watch.rs create mode 100644 app/src/hooks/useProjectStatusEvents.test.ts create mode 100644 app/src/hooks/useProjectStatusEvents.ts diff --git a/CLAUDE.md b/CLAUDE.md index 56f6cba..8c72fff 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -313,7 +313,13 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li approach converges on the worst state: a 400 MB browser present with its libraries gone. - The layer sits immediately after Node (npx is its only prerequisite) and well above the shim `COPY`s, so editing a shim does not re-run a multi-hundred-megabyte apt install. -- **`entrypoint.sh`** — UID/GID remapping to match host user, SSH key setup (`sync_host_ssh` *merges* the host dir into `~/.ssh`; never wipe it — container-made keys live there on the home volume), git config, docker socket permissions, Claude Code settings.json injection, then `sleep infinity` +- **`entrypoint.sh`** — UID/GID remapping to match host user, SSH key setup (`sync_host_ssh` *merges* the host dir into `~/.ssh`; never wipe it — container-made keys live there on the home volume), git config, docker socket permissions, Claude Code settings.json injection, then stays resident **as root** waiting on a root-owned + `triple-c-keepalive` sleep (respawned if killed; SIGTERM trap so `docker stop` is immediate). + **Never hand the container's lifetime to a process the `claude` user owns** — it used to end in + `exec su … claude -c "exec sleep infinity"`, and any `pkill sleep` / `kill -9 -1` a Claude + session ran killed the whole container. `container_watch.rs` (one Docker `die`-event + subscription) is what flips a project whose container exits on its own to Stopped and emits + `project-status-changed`; `useProjectStatusEvents` applies it - **`triple-c-scheduler`** — Bash-based scheduled task system for recurring Claude Code invocations **`/home/claude` in the image is seed-only.** It is the mount point of the named volume @@ -672,8 +678,9 @@ Anthropic and Bedrock deliberately keep Claude Code's own defaults. now", the host builds the project's effective set (`global − disabled ∪ project`), then uploads `payload.tar` **and the app-embedded script `src/marketplace/sync.sh`** (`include_str!`, not a file in `container/`) to `~/.claude/triple-c/marketplace/incoming/` and runs it as `claude`, - once the entrypoint has finished (`pgrep -x -f 'su -s /bin/bash claude -c exec sleep - infinity'`). The script is re-uploaded on every sync rather than baked into the image, so every + once the entrypoint has finished (`READY_PROBE`: a root `triple-c-keepalive infinity` process + on current images, `su -s /bin/bash claude -c exec sleep infinity` on older ones — keep both). + The script is re-uploaded on every sync rather than baked into the image, so every existing project always gets the version that matches the running app — `container/` is never touched for this feature. The script only removes files and hook entries it recorded in `~/.claude/triple-c/marketplace/state.json`; it must never overwrite or delete user-created diff --git a/app/src-tauri/src/container_watch.rs b/app/src-tauri/src/container_watch.rs new file mode 100644 index 0000000..cfb1508 --- /dev/null +++ b/app/src-tauri/src/container_watch.rs @@ -0,0 +1,278 @@ +//! Notices a project container that stopped without the app asking. +//! +//! Until this existed the app only ever learned a container's state from its +//! own commands and from `reconcile_project_statuses`, which the frontend runs +//! once, when Docker first answers. A container that died underneath a running +//! app — the entrypoint's keepalive killed, an OOM, `docker stop` from a shell, +//! a daemon restart — left its project at Running indefinitely: every terminal +//! disconnected, while the sidebar, Start/Stop and Project Home all still +//! described a live container. +//! +//! One subscription to Docker's event stream, filtered daemon-side to container +//! `die` events, so an idle daemon costs nothing and the app hears about a death +//! as it happens. No polling loop. Each `die` naming a project's container is +//! confirmed with one inspect before anything changes (an event can be +//! replayed, and a restart can reuse the id), and the flip itself is +//! [`ProjectsStore::mark_exited`]'s compare-and-set, which never touches a +//! project a command owns. The frontend hears it on [`PROJECT_STATUS_EVENT`]. +//! +//! The stream drops whenever the daemon does. Each (re)subscription asks for +//! events since just before it, and first re-checks every Running project once — +//! the only window that replay cannot cover is a daemon restart, which forgets +//! its event history, and that is exactly when containers go away. + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::Duration; + +use bollard::errors::Error as DockerError; +use bollard::models::{EventMessage, EventMessageTypeEnum}; +use bollard::system::EventsOptions; +use bollard::Docker; +use futures_util::StreamExt; +use tauri::{AppHandle, Emitter}; +use tokio::sync::watch; + +use crate::models::{Project, ProjectStatus}; +use crate::storage::projects_store::ProjectsStore; + +/// Emitted with the updated `Project` whenever this module changes a status. +pub const PROJECT_STATUS_EVENT: &str = "project-status-changed"; + +const RETRY_MIN: Duration = Duration::from_secs(5); +const RETRY_MAX: Duration = Duration::from_secs(60); + +/// The container id a Docker event reports as having died, if it is one. +pub(crate) fn died_container(msg: &EventMessage) -> Option<&str> { + if msg.typ != Some(EventMessageTypeEnum::CONTAINER) || msg.action.as_deref() != Some("die") { + return None; + } + msg.actor.as_ref()?.id.as_deref().filter(|id| !id.is_empty()) +} + +/// `true` only when Docker positively says the container is not running: it +/// inspects as stopped, or it no longer exists. Any other failure is "don't +/// know" — `docker::is_container_running` maps every error to `false`, which +/// is right for its callers and wrong here, where it would mark every project +/// Stopped the moment the daemon hiccups. +async fn has_exited(docker: &Docker, container_id: &str) -> bool { + match docker.inspect_container(container_id, None).await { + Ok(info) => !info.state.and_then(|s| s.running).unwrap_or(false), + Err(DockerError::DockerResponseServerError { status_code: 404, .. }) => true, + Err(_) => false, + } +} + +fn project_busy(project_id: &str) -> bool { + crate::project_lock::held(project_id).is_some() +} + +/// Confirm `container_id` really stopped and, if a project was Running on it, +/// flip that project. Returns what changed. +async fn settle(docker: &Docker, store: &ProjectsStore, container_id: &str) -> Option { + // The cheap filter first: most containers that die are not ours (probe and + // scratch containers, other software on the daemon), and those cost no + // inspect at all. + let ours = store.list().into_iter().any(|p| { + p.status == ProjectStatus::Running + && p.container_id + .as_deref() + .is_some_and(|c| c.starts_with(container_id) || container_id.starts_with(c)) + }); + if !ours || !has_exited(docker, container_id).await { + return None; + } + store.mark_exited(container_id, project_busy) +} + +/// Re-check every Running project once. Used on each (re)subscription. +async fn sweep(docker: &Docker, store: &ProjectsStore) -> Vec { + let mut changed = Vec::new(); + for p in store.list() { + if p.status != ProjectStatus::Running { + continue; + } + let Some(cid) = p.container_id.as_deref() else { continue }; + if let Some(c) = settle(docker, store, cid).await { + changed.push(c); + } + } + changed +} + +fn die_filter() -> HashMap> { + HashMap::from([ + ("type".to_string(), vec!["container".to_string()]), + ("event".to_string(), vec!["die".to_string()]), + ]) +} + +/// Watch until `cancel` flips, calling `on_change` for every project whose +/// container is found to have stopped on its own. Separate from [`spawn`] so a +/// test can drive it without a Tauri app. +pub(crate) async fn run( + store: Arc, + mut cancel: watch::Receiver, + on_change: impl Fn(&Project), +) { + let mut retry = RETRY_MIN; + loop { + if *cancel.borrow() { + return; + } + let docker = match crate::docker::get_docker() { + Ok(d) if d.ping().await.is_ok() => Some(d), + _ => None, + }; + if let Some(docker) = docker { + retry = RETRY_MIN; + // A second's overlap: the daemon replays from `since`, so a death + // during the sweep below is still delivered. Duplicates are + // harmless — `settle` confirms with an inspect every time. + let since = (chrono::Utc::now().timestamp() - 1).to_string(); + for p in sweep(&docker, &store).await { + on_change(&p); + } + let mut events = docker.events(Some(EventsOptions:: { + since: Some(since), + until: None, + filters: die_filter(), + })); + loop { + tokio::select! { + _ = cancel.changed() => return, + next = events.next() => match next { + Some(Ok(msg)) => { + if let Some(cid) = died_container(&msg) { + if let Some(p) = settle(&docker, &store, cid).await { + on_change(&p); + } + } + } + Some(Err(e)) => { + log::info!("Container watch: event stream ended ({e}) — reconnecting"); + break; + } + None => { + log::info!("Container watch: event stream closed — reconnecting"); + break; + } + }, + } + } + } + tokio::select! { + _ = cancel.changed() => return, + _ = tokio::time::sleep(retry) => {} + } + retry = (retry * 2).min(RETRY_MAX); + } +} + +/// Start the watcher for the life of the app. +pub fn spawn( + app: AppHandle, + store: Arc, + cancel: watch::Receiver, +) -> tauri::async_runtime::JoinHandle<()> { + tauri::async_runtime::spawn(async move { + run(store, cancel, |p| { + log::warn!( + "Project '{}' ({}): its container stopped without the app asking — marking Stopped", + p.name, + p.id + ); + let _ = app.emit(PROJECT_STATUS_EVENT, p); + }) + .await; + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use bollard::models::EventActor; + + fn event(typ: EventMessageTypeEnum, action: &str, id: Option<&str>) -> EventMessage { + EventMessage { + typ: Some(typ), + action: Some(action.to_string()), + actor: Some(EventActor { id: id.map(str::to_string), attributes: None }), + ..Default::default() + } + } + + #[test] + fn only_a_container_die_names_a_container() { + let die = event(EventMessageTypeEnum::CONTAINER, "die", Some("abc")); + assert_eq!(died_container(&die), Some("abc")); + + for (typ, action, id) in [ + (EventMessageTypeEnum::CONTAINER, "stop", Some("abc")), + (EventMessageTypeEnum::CONTAINER, "start", Some("abc")), + (EventMessageTypeEnum::CONTAINER, "exec_die", Some("abc")), + (EventMessageTypeEnum::NETWORK, "die", Some("abc")), + (EventMessageTypeEnum::CONTAINER, "die", None), + (EventMessageTypeEnum::CONTAINER, "die", Some("")), + ] { + assert_eq!(died_container(&event(typ, action, id)), None, "{typ:?} {action} {id:?}"); + } + } + + #[test] + fn the_subscription_is_filtered_to_container_deaths_daemon_side() { + let f = die_filter(); + assert_eq!(f.get("type"), Some(&vec!["container".to_string()])); + assert_eq!(f.get("event"), Some(&vec!["die".to_string()])); + assert_eq!(f.len(), 2); + } + + /// End to end against a real daemon: a Running project whose container is + /// killed from outside the app is flipped to Stopped and reported, and one + /// whose container is still up is not. + #[tokio::test] + #[ignore = "needs a Docker daemon; creates and removes throwaway containers"] + async fn a_container_killed_outside_the_app_flips_its_project_to_stopped() { + use std::process::Command; + fn docker_cli(args: &[&str]) -> String { + let out = Command::new("docker").args(args).output().expect("docker cli"); + assert!(out.status.success(), "docker {:?}: {}", args, String::from_utf8_lossy(&out.stderr)); + String::from_utf8_lossy(&out.stdout).trim().to_string() + } + let tag = uuid::Uuid::new_v4().simple().to_string(); + let doomed = docker_cli(&["run", "-d", "--name", &format!("tc-watch-a-{tag}"), "alpine:3", "sleep", "300"]); + let alive = docker_cli(&["run", "-d", "--name", &format!("tc-watch-b-{tag}"), "alpine:3", "sleep", "300"]); + + let dir = std::env::temp_dir().join(format!("triple-c-watch-{tag}")); + std::fs::create_dir_all(&dir).unwrap(); + let mut a = Project::new("a".to_string(), Vec::new()); + a.container_id = Some(doomed.clone()); + a.status = ProjectStatus::Running; + let mut b = Project::new("b".to_string(), Vec::new()); + b.container_id = Some(alive.clone()); + b.status = ProjectStatus::Running; + let (a_id, b_id) = (a.id.clone(), b.id.clone()); + let store = Arc::new(ProjectsStore::over_file_for_test(dir.join("projects.json"), vec![a, b])); + + let (cancel_tx, cancel_rx) = watch::channel(false); + let (seen_tx, mut seen_rx) = tokio::sync::mpsc::unbounded_channel::(); + let watcher = tokio::spawn(run(store.clone(), cancel_rx, move |p| { + let _ = seen_tx.send(p.id.clone()); + })); + // Let it subscribe, then kill from "outside". + tokio::time::sleep(Duration::from_secs(2)).await; + docker_cli(&["kill", &doomed]); + + let seen = tokio::time::timeout(Duration::from_secs(15), seen_rx.recv()).await; + let _ = cancel_tx.send(true); + let _ = watcher.await; + let a_status = store.get(&a_id).unwrap().status; + let b_status = store.get(&b_id).unwrap().status; + docker_cli(&["rm", "-f", &doomed, &alive]); + std::fs::remove_dir_all(&dir).ok(); + + assert_eq!(seen.expect("reported within 15s"), Some(a_id)); + assert_eq!(a_status, ProjectStatus::Stopped); + assert_eq!(b_status, ProjectStatus::Running); + } +} diff --git a/app/src-tauri/src/lib.rs b/app/src-tauri/src/lib.rs index 70f905f..08a2bba 100644 --- a/app/src-tauri/src/lib.rs +++ b/app/src-tauri/src/lib.rs @@ -3,6 +3,7 @@ mod browser_view; #[cfg(test)] mod command_census; mod commands; +mod container_watch; mod docker; pub mod file_viewer; mod install_helper; @@ -307,6 +308,16 @@ pub fn run() { .await; }); + // Notice a project container that stops without the app asking + // (its keepalive killed, an OOM, a `docker stop` from a shell) and + // flip the project to Stopped. One daemon-side-filtered event + // subscription for the app's lifetime — see `container_watch`. + lifecycle_setup.track(container_watch::spawn( + app.handle().clone(), + projects_store_setup.clone(), + lifecycle_setup.cancellation(), + )); + // Marketplaces: refresh each once at startup, in the background. // Failures are logged, not toasted — the Marketplace tab shows them. { diff --git a/app/src-tauri/src/marketplace/sync.rs b/app/src-tauri/src/marketplace/sync.rs index f2380f7..4221533 100644 --- a/app/src-tauri/src/marketplace/sync.rs +++ b/app/src-tauri/src/marketplace/sync.rs @@ -14,10 +14,17 @@ pub const INCOMING_DIR: &str = "/home/claude/.claude/triple-c/marketplace/incomi /// app version reaches existing containers without an image migration. pub const SYNC_SCRIPT: &str = include_str!("sync.sh"); -/// True once the entrypoint has finished: its last step execs this exact -/// command line. Before that it may still be merging `settings.json` or running -/// `claude update`, both of which the sync would race. -const READY_PROBE: &str = "pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null"; +/// True once the entrypoint has finished: its last step starts a process with +/// one of these exact command lines. Before that it may still be merging +/// `settings.json` or running `claude update`, both of which the sync would race. +/// +/// Two, because there are two generations of entrypoint in the field. Current +/// images keep the container alive with a root-owned `triple-c-keepalive` +/// sleep (so a `pkill sleep` run as `claude` cannot stop the container — see +/// the end of `container/entrypoint.sh`); older ones `exec su … claude -c exec +/// sleep infinity`, and a project keeps its base image until it is migrated. +const READY_PROBE: &str = "pgrep -u root -x -f 'triple-c-keepalive infinity' >/dev/null \ + || pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null"; const READY_TIMEOUT: Duration = Duration::from_secs(180); const READY_POLL: Duration = Duration::from_secs(2); @@ -196,14 +203,22 @@ mod tests { #[test] fn readiness_probes_the_entrypoints_final_exec() { + // Both generations of entrypoint: a project keeps the base image it + // was built from until it migrates, so the old `su … sleep` ending is + // the common case for years and must still read as ready. assert_eq!( ready_probe_cmd(), vec![ "sh", "-c", - "pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null" + "pgrep -u root -x -f 'triple-c-keepalive infinity' >/dev/null \ + || pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null" ] ); + // The marker name is shared with the entrypoint; a rename on one side + // only would leave every new container "not ready" for 180 s. + let entrypoint = include_str!("../../../../container/entrypoint.sh"); + assert!(entrypoint.contains("exec -a triple-c-keepalive sleep infinity")); assert_eq!(READY_POLL, Duration::from_secs(2)); assert_eq!(READY_TIMEOUT, Duration::from_secs(180)); } diff --git a/app/src-tauri/src/storage/projects_store.rs b/app/src-tauri/src/storage/projects_store.rs index 6a261c5..86dd83c 100644 --- a/app/src-tauri/src/storage/projects_store.rs +++ b/app/src-tauri/src/storage/projects_store.rs @@ -161,6 +161,12 @@ impl ProjectsStore { Ok(store) } + /// A store over a file the test owns (`new()` insists on the real data dir). + #[cfg(test)] + pub(crate) fn over_file_for_test(file_path: PathBuf, projects: Vec) -> Self { + Self { projects: Mutex::new(projects), file_path } + } + fn lock(&self) -> std::sync::MutexGuard<'_, Vec> { self.projects.lock().unwrap_or_else(|e| e.into_inner()) } @@ -237,6 +243,47 @@ impl ProjectsStore { Ok(()) } + /// A container stopped without the app asking: flip the project that is + /// `Running` on it to `Stopped` and return the updated record, or `None` + /// when there is nothing to flip. + /// + /// Compare-and-set under the store lock rather than a read followed by + /// `update_status`, so it cannot land on top of a command that got there + /// first: only a project that is *still* `Running` on *this* container is + /// touched, and never one `is_busy` says a command owns (the project lock + /// — Stop, Start, Reset and migration all hold it across their own + /// container's death and write the status they mean themselves). A start + /// that takes the project lock just after this check still blocks on the + /// store lock before it writes `Starting`, so the two serialise. + /// + /// `container_id` may be the short form; the store holds the full id. + pub fn mark_exited(&self, container_id: &str, is_busy: impl Fn(&str) -> bool) -> Option { + if container_id.len() < 12 { + return None; + } + let mut projects = self.lock(); + let p = projects.iter_mut().find(|p| { + p.status == crate::models::ProjectStatus::Running + && p.container_id + .as_deref() + .filter(|stored| stored.len() >= 12) + .is_some_and(|stored| stored.starts_with(container_id) || container_id.starts_with(stored)) + })?; + if is_busy(&p.id) { + return None; + } + p.status = crate::models::ProjectStatus::Stopped; + p.updated_at = chrono::Utc::now().to_rfc3339(); + let changed = p.clone(); + if let Err(e) = self.save(&projects) { + // The in-memory record is what the UI is told about and what every + // later command reads, so it stays flipped; the next save of any + // kind writes it out. + log::warn!("Could not persist Stopped for project {}: {}", changed.id, e); + } + Some(changed) + } + pub fn update_status(&self, id: &str, status: crate::models::ProjectStatus) -> Result<(), String> { let mut projects = self.lock(); if let Some(p) = projects.iter_mut().find(|p| p.id == id) { @@ -604,4 +651,88 @@ mod tests { fs::remove_dir_all(&dir).ok(); } + + // ── mark_exited: a container that died on its own ──────────────────── + + /// A full-length id, as Docker's events and the store both carry. + const C0FFEE: &str = "c0ffee0000000000000000000000000000000000000000000000000000000000"; + + fn running_on(container_id: &str) -> Project { + let mut p = Project::new("p".to_string(), Vec::new()); + p.container_id = Some(container_id.to_string()); + p.status = crate::models::ProjectStatus::Running; + p + } + + #[test] + fn a_running_project_whose_container_exited_is_marked_stopped_and_saved() { + let dir = temp_dir("exited"); + let p = running_on(C0FFEE); + let id = p.id.clone(); + let store = store_over(&dir, vec![p]); + + let changed = store.mark_exited(C0FFEE, |_| false).expect("flipped"); + + assert_eq!(changed.id, id); + assert_eq!(changed.status, crate::models::ProjectStatus::Stopped); + assert_eq!(store.get(&id).unwrap().status, crate::models::ProjectStatus::Stopped); + let on_disk = fs::read_to_string(dir.join("projects.json")).unwrap(); + assert!(on_disk.contains("\"stopped\""), "the flip must be persisted: {on_disk}"); + fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn mark_exited_leaves_a_project_alone_while_a_command_owns_it() { + // Stop, Start-with-recreate, Reset and migration all hold the project + // lock across their own container's death; the command writes the + // status it means, and the watcher must not race it. + let dir = temp_dir("busy"); + let p = running_on(C0FFEE); + let id = p.id.clone(); + let store = store_over(&dir, vec![p]); + + assert!(store.mark_exited(C0FFEE, |pid| pid == id).is_none()); + assert_eq!(store.get(&id).unwrap().status, crate::models::ProjectStatus::Running); + // ...and it was only the claim that held it back. + assert!(store.mark_exited(C0FFEE, |_| false).is_some()); + fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn mark_exited_only_touches_running_projects_on_that_container() { + let dir = temp_dir("other"); + let mut starting = running_on(C0FFEE); + starting.status = crate::models::ProjectStatus::Starting; + let other = running_on(&"b".repeat(64)); + let (starting_id, other_id) = (starting.id.clone(), other.id.clone()); + let store = store_over(&dir, vec![starting, other]); + + assert!(store.mark_exited(C0FFEE, |_| false).is_none(), "Starting is a command's"); + assert!(store.mark_exited(&"d".repeat(64), |_| false).is_none(), "nobody's container"); + assert!(store.mark_exited("", |_| false).is_none()); + // A blank stored id is a prefix of everything; it must match nothing. + let dir2 = temp_dir("blank"); + let blank = running_on(""); + let store2 = store_over(&dir2, vec![blank]); + assert!(store2.mark_exited(&"f".repeat(64), |_| false).is_none()); + fs::remove_dir_all(&dir2).ok(); + assert_eq!( + store.get(&starting_id).unwrap().status, + crate::models::ProjectStatus::Starting + ); + assert_eq!(store.get(&other_id).unwrap().status, crate::models::ProjectStatus::Running); + fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn mark_exited_matches_a_short_id_against_the_stored_full_one() { + let dir = temp_dir("short"); + let full = "a".repeat(12) + &"b".repeat(52); + let p = running_on(&full); + let store = store_over(&dir, vec![p]); + + assert!(store.mark_exited("aaaa", |_| false).is_none(), "too short to be an id"); + assert!(store.mark_exited(&full[..12], |_| false).is_some()); + fs::remove_dir_all(&dir).ok(); + } } diff --git a/app/src/App.tsx b/app/src/App.tsx index 23f64f1..8454652 100644 --- a/app/src/App.tsx +++ b/app/src/App.tsx @@ -22,6 +22,7 @@ import { useSTT } from "./hooks/useSTT"; import { useContainerProgress } from "./hooks/useContainerProgress"; import { useKeyboardShortcuts } from "./hooks/useKeyboardShortcuts"; import { useMarketplaceSyncToasts } from "./hooks/useMarketplace"; +import { useProjectStatusEvents } from "./hooks/useProjectStatusEvents"; import MarketplaceView from "./components/marketplace/MarketplaceView"; import { useAppState, isHomeTab, tabKeyId, homeTabKey, MARKETPLACE_TAB_KEY } from "./store/appState"; import { reconcileProjectStatuses } from "./lib/tauri-commands"; @@ -75,6 +76,7 @@ export default function App() { useContainerProgress(); useKeyboardShortcuts(); useMarketplaceSyncToasts(); + useProjectStatusEvents(); // Initialize on mount useEffect(() => { diff --git a/app/src/hooks/useProjectStatusEvents.test.ts b/app/src/hooks/useProjectStatusEvents.test.ts new file mode 100644 index 0000000..b3e2c53 --- /dev/null +++ b/app/src/hooks/useProjectStatusEvents.test.ts @@ -0,0 +1,48 @@ +import { describe, it, expect, vi, beforeEach } from "vitest"; +import { renderHook } from "@testing-library/react"; +import { useAppState } from "../store/appState"; +import type { Project } from "../lib/types"; + +let handler: ((e: { payload: unknown }) => void) | null = null; +const unlisten = vi.fn(); +vi.mock("@tauri-apps/api/event", () => ({ + listen: vi.fn(async (name: string, cb: (e: { payload: unknown }) => void) => { + if (name === "project-status-changed") handler = cb; + return unlisten; + }), +})); + +import { useProjectStatusEvents } from "./useProjectStatusEvents"; + +const project = (over: Partial): Project => + ({ id: "p1", name: "demo", status: "running", container_id: "c1", ...over }) as Project; + +describe("useProjectStatusEvents", () => { + beforeEach(() => { + handler = null; + unlisten.mockClear(); + useAppState.setState({ projects: [project({}), project({ id: "p2", name: "other" })], toasts: [] }); + }); + + it("replaces the project the backend reports, so a dead container stops reading Running", async () => { + renderHook(() => useProjectStatusEvents()); + await vi.waitFor(() => expect(handler).not.toBeNull()); + + handler!({ payload: project({ status: "stopped" }) }); + + const { projects, toasts } = useAppState.getState(); + expect(projects.find((p) => p.id === "p1")?.status).toBe("stopped"); + expect(projects.find((p) => p.id === "p2")?.status).toBe("running"); + // The terminals just went dead with no explanation; say why. + expect(toasts).toHaveLength(1); + expect(toasts[0].message).toContain("demo"); + expect(toasts[0].dedupeKey).toBe("container-exited-p1"); + }); + + it("stops listening on unmount", async () => { + const { unmount } = renderHook(() => useProjectStatusEvents()); + await vi.waitFor(() => expect(handler).not.toBeNull()); + unmount(); + expect(unlisten).toHaveBeenCalled(); + }); +}); diff --git a/app/src/hooks/useProjectStatusEvents.ts b/app/src/hooks/useProjectStatusEvents.ts new file mode 100644 index 0000000..5483c8b --- /dev/null +++ b/app/src/hooks/useProjectStatusEvents.ts @@ -0,0 +1,39 @@ +import { useEffect } from "react"; +import { listen, type UnlistenFn } from "@tauri-apps/api/event"; +import { useAppState } from "../store/appState"; +import type { Project } from "../lib/types"; + +/** + * App-wide: apply status changes the backend makes on its own. Today that is + * `container_watch.rs` marking a project Stopped when its container exited + * without the app asking — before it, the project kept reading Running, with + * every terminal dead, until the app was restarted. + */ +export function useProjectStatusEvents() { + useEffect(() => { + let cancelled = false; + let unlisten: UnlistenFn | null = null; + void listen("project-status-changed", (event) => { + const project = event.payload; + const state = useAppState.getState(); + state.updateProjectInList(project); + if (project.status === "stopped") { + state.pushToast({ + kind: "error", + message: `The container for “${project.name}” stopped`, + detail: + "It exited without Triple-C stopping it — open sessions were disconnected. " + + "Start the project again to reconnect.", + dedupeKey: `container-exited-${project.id}`, + }); + } + }).then((fn) => { + if (cancelled) fn(); + else unlisten = fn; + }); + return () => { + cancelled = true; + unlisten?.(); + }; + }, []); +} diff --git a/container/entrypoint.sh b/container/entrypoint.sh index 41e4438..e1fd83c 100644 --- a/container/entrypoint.sh +++ b/container/entrypoint.sh @@ -736,6 +736,39 @@ timeout 120 su -s /bin/bash claude -c 'export PATH="/home/claude/.claude/bin:/ho && echo "entrypoint: Claude Code is up to date" \ || echo "entrypoint: warning — Claude Code update skipped or failed (continuing)" -# ── Stay alive as claude ───────────────────────────────────────────────────── +# ── Stay alive, as root ────────────────────────────────────────────────────── +# The container lives exactly as long as this script: docker-init is PID 1 and +# exits when its one child does. So what keeps it alive must be something the +# claude user cannot kill. This used to end in +# exec su -s /bin/bash claude -c "exec sleep infinity" +# which hung the container's whole lifetime on a `sleep` owned by claude — and +# Claude Code runs as claude and routinely runs `pkill sleep`, `pkill -f …`, +# `killall` or `kill -9 -1` while testing something. Any of those stopped the +# container: every terminal disconnected and the app still said Running. +# Dropping to claude there bought nothing (it dates from the initial commit, +# with no stated reason, and nothing reads that process's owner); its one +# reader is the app's marketplace readiness probe, which now also matches the +# marker below (app/src-tauri/src/marketplace/sync.rs, READY_PROBE). +# +# * Unprivileged kills get EPERM on both this script and its sleep. Even +# `sudo pkill sleep` only kills the child, which the loop replaces; it takes +# a root kill of this script itself. +# * `docker stop`: docker-init forwards SIGTERM here, and bash runs the trap +# as soon as `wait` returns, so the container stops at once rather than +# sitting out Docker's grace period and being SIGKILLed. +# * The sleep runs as `triple-c-keepalive infinity` (exec -a), a root-owned +# command line nothing else has, and it exists only once every step above +# has finished — that is the readiness marker. Keep the name in step with +# READY_PROBE. echo "Triple-C container ready." -exec su -s /bin/bash claude -c "exec sleep infinity" +keepalive_pid="" +stop_keepalive() { + [ -n "$keepalive_pid" ] && kill "$keepalive_pid" 2>/dev/null + exit 0 +} +trap stop_keepalive TERM INT HUP +while :; do + (exec -a triple-c-keepalive sleep infinity) & + keepalive_pid=$! + wait "$keepalive_pid" +done -- 2.52.0 From 2eabf6972b9c9649d61f4a7e885922dc5d4ead33 Mon Sep 17 00:00:00 2001 From: Josh Knapp Date: Mon, 2 Nov 2026 16:15:31 -0800 Subject: [PATCH 2/2] Address review of the keepalive / container-watch change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Event renamed to `project-container-exited` and carries a reason: `exited` (died while the app was watching) toasts; `found_stopped` (the first look of a run, e.g. after a host reboot) updates silently. Reconnect sweeps report `exited`, since the app was running then. - listen() in useProjectStatusEvents gets a .catch, like its sibling. - Keepalive respawns are throttled to ~1/s with a fork-free pause (`read -t` on a FIFO, `sleep 1` fallback), so a failing fork or an instantly-dying sleep cannot spin a core. Measured: 10748 respawns in 3 s unthrottled vs 3. - Watcher backoff resets only after a working connection (an event delivered, or up for 60 s); ping-OK-but-events-fail now backs off to 60 s instead of re-sweeping every 5 s. - Every Docker call in the watcher (ping, sweep, inspect, stream) races the app's cancellation, so a wedged daemon cannot hold shutdown. - TOCTOU between the inspect and the flip: ProjectsStore now counts writes; running_on() returns a sighting at a revision and mark_exited() refuses with Changed if anything was written since (a stop+start of the same container id included), and settle() re-confirms. - The "Running on this container" rule lives once, in the store, with the 12-char guard on both ids, matched under the lock without cloning. - Marketplace readiness fallback also requires a root-owned process. - TECHNICAL.md no longer documents the `su … sleep infinity` ending. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- TECHNICAL.md | 6 +- app/src-tauri/src/container_watch.rs | 300 +++++++++++++------ app/src-tauri/src/marketplace/sync.rs | 6 +- app/src-tauri/src/storage/projects_store.rs | 200 ++++++++++--- app/src/hooks/useProjectStatusEvents.test.ts | 40 ++- app/src/hooks/useProjectStatusEvents.ts | 52 ++-- container/entrypoint.sh | 19 +- 8 files changed, 456 insertions(+), 169 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 8c72fff..5b8c435 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -319,7 +319,7 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li `exec su … claude -c "exec sleep infinity"`, and any `pkill sleep` / `kill -9 -1` a Claude session ran killed the whole container. `container_watch.rs` (one Docker `die`-event subscription) is what flips a project whose container exits on its own to Stopped and emits - `project-status-changed`; `useProjectStatusEvents` applies it + `project-container-exited` with a `reason` (`exited` toasts; `found_stopped`, the first look of a run, is silent); `useProjectStatusEvents` applies it - **`triple-c-scheduler`** — Bash-based scheduled task system for recurring Claude Code invocations **`/home/claude` in the image is seed-only.** It is the mount point of the named volume diff --git a/TECHNICAL.md b/TECHNICAL.md index 3111fd9..a751d85 100644 --- a/TECHNICAL.md +++ b/TECHNICAL.md @@ -137,7 +137,7 @@ Implementation gotchas for the terminal view and its global controls (merged in │ │ │ User: claude (UID/GID remapped to match host) │ │ Entrypoint: UID/GID remap → SSH setup → git config → │ -│ docker socket perms → sleep infinity │ +│ docker socket perms → root keepalive wait │ └──────────────────────────────────────────────────────────┘ ``` @@ -164,7 +164,7 @@ Terminal resize follows the same pattern: `ResizeObserver` detects container siz Containers follow a **stop/start** model, not create/destroy: -1. **First start**: A new container is created with bind mounts, named volumes, environment variables, and labels. The entrypoint remaps UID/GID, configures SSH and git, rebuilds the scheduler crontab, then runs `sleep infinity` to keep the container alive. +1. **First start**: A new container is created with bind mounts, named volumes, environment variables, and labels. The entrypoint remaps UID/GID, configures SSH and git, rebuilds the scheduler crontab, then stays resident as root waiting on a root-owned `triple-c-keepalive` sleep — never a process the `claude` user can kill, since any `pkill sleep` a Claude session runs would otherwise stop the container. 2. **Terminal open**: `docker exec` launches `claude` with a PTY in the running container, with the permission-mode flags from `PermissionMode::cli_args()` (or `bash -l` for a shell session). 3. **Stop**: `docker stop` halts the container but preserves its filesystem. Any packages Claude installed via `apt`, `pip`, `cargo`, etc. survive. 4. **Restart**: `docker start` resumes the existing container — unless `container_needs_recreation()` finds a `triple-c.*` label that no longer matches the project's settings, in which case the container is committed to a snapshot image (`triple-c-snapshot-{projectId}:latest`), removed, and recreated from that snapshot. Installed tools survive; the named volumes are untouched. @@ -348,7 +348,7 @@ The entrypoint solves this by: 3. Relocating any existing system user/group that conflicts with the target UID/GID. 4. Fixing ownership of `/home/claude` after the change. -This runs as root in the entrypoint, then the final `exec su -s /bin/bash claude -c "exec sleep infinity"` drops to the remapped user. +This runs as root in the entrypoint. The entrypoint itself then stays running as root to keep the container alive; sessions reach the remapped `claude` user through `docker exec --user`. (Older images ended in `exec su -s /bin/bash claude -c "exec sleep infinity"`, which let any `claude` process stop the container by killing that sleep.) ### SSH Key Handling diff --git a/app/src-tauri/src/container_watch.rs b/app/src-tauri/src/container_watch.rs index cfb1508..52136f2 100644 --- a/app/src-tauri/src/container_watch.rs +++ b/app/src-tauri/src/container_watch.rs @@ -14,7 +14,8 @@ //! confirmed with one inspect before anything changes (an event can be //! replayed, and a restart can reuse the id), and the flip itself is //! [`ProjectsStore::mark_exited`]'s compare-and-set, which never touches a -//! project a command owns. The frontend hears it on [`PROJECT_STATUS_EVENT`]. +//! project a command owns and refuses if the store changed while the inspect +//! was in flight. The frontend hears it on [`CONTAINER_EXITED_EVENT`]. //! //! The stream drops whenever the daemon does. Each (re)subscription asks for //! events since just before it, and first re-checks every Running project once — @@ -22,25 +23,54 @@ //! its event history, and that is exactly when containers go away. use std::collections::HashMap; +use std::future::Future; use std::sync::Arc; -use std::time::Duration; +use std::time::{Duration, Instant}; use bollard::errors::Error as DockerError; use bollard::models::{EventMessage, EventMessageTypeEnum}; use bollard::system::EventsOptions; use bollard::Docker; use futures_util::StreamExt; +use serde::Serialize; use tauri::{AppHandle, Emitter}; use tokio::sync::watch; use crate::models::{Project, ProjectStatus}; -use crate::storage::projects_store::ProjectsStore; +use crate::storage::projects_store::{ExitMark, ProjectsStore}; -/// Emitted with the updated `Project` whenever this module changes a status. -pub const PROJECT_STATUS_EVENT: &str = "project-status-changed"; +/// Emitted as a [`ContainerExited`] whenever this module flips a project to +/// Stopped. +pub const CONTAINER_EXITED_EVENT: &str = "project-container-exited"; const RETRY_MIN: Duration = Duration::from_secs(5); const RETRY_MAX: Duration = Duration::from_secs(60); +/// A stream that stayed up this long (or delivered anything) was a working +/// connection, so its loss starts the backoff over. Anything shorter — the +/// daemon answers ping but the events call fails — keeps backing off, or it +/// would re-sweep and re-log every five seconds forever. +const STABLE_AFTER: Duration = Duration::from_secs(60); +/// How often `settle` re-confirms when the store keeps changing under it. +const SETTLE_ATTEMPTS: usize = 3; + +/// How the app found out, which decides whether it is news to the user. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum ExitReason { + /// The container died while this app was running and watching — any + /// sessions it had were just cut off. + Exited, + /// Found already stopped on the first look of this run: it went down + /// while the app was closed (a host reboot, typically). Nothing the user + /// can see was interrupted, so it is not news. + FoundStopped, +} + +#[derive(Debug, Clone, Serialize)] +pub struct ContainerExited<'a> { + pub project: &'a Project, + pub reason: ExitReason, +} /// The container id a Docker event reports as having died, if it is one. pub(crate) fn died_container(msg: &EventMessage) -> Option<&str> { @@ -70,19 +100,23 @@ fn project_busy(project_id: &str) -> bool { /// Confirm `container_id` really stopped and, if a project was Running on it, /// flip that project. Returns what changed. async fn settle(docker: &Docker, store: &ProjectsStore, container_id: &str) -> Option { - // The cheap filter first: most containers that die are not ours (probe and - // scratch containers, other software on the daemon), and those cost no - // inspect at all. - let ours = store.list().into_iter().any(|p| { - p.status == ProjectStatus::Running - && p.container_id - .as_deref() - .is_some_and(|c| c.starts_with(container_id) || container_id.starts_with(c)) - }); - if !ours || !has_exited(docker, container_id).await { - return None; + for _ in 0..SETTLE_ATTEMPTS { + // Cheap and allocation-free: most containers that die are not ours + // (probe and scratch containers, other software on the daemon), and + // those cost no inspect at all. + let seen = store.running_on(container_id)?; + if !has_exited(docker, container_id).await { + return None; + } + match store.mark_exited(container_id, &seen, project_busy) { + ExitMark::Flipped(p) => return Some(*p), + ExitMark::NotApplicable => return None, + // The store moved while Docker was answering — possibly a stop and + // start of this very container. The inspect is stale; ask again. + ExitMark::Changed => continue, + } } - store.mark_exited(container_id, project_busy) + None } /// Re-check every Running project once. Used on each (re)subscription. @@ -107,65 +141,104 @@ fn die_filter() -> HashMap> { ]) } +/// The wait before connection attempt `n + 1`, after `n` consecutive +/// unproductive ones: 5 s, 10 s, 20 s, 40 s, then 60 s for good. +fn retry_delay(consecutive_failures: u32) -> Duration { + let doublings = consecutive_failures.saturating_sub(1).min(8); + (RETRY_MIN * 2u32.pow(doublings)).min(RETRY_MAX) +} + +/// `fut`, unless the app starts shutting down first. Every Docker call in here +/// goes through this: a wedged daemon must not hold shutdown for its full +/// budget. +async fn unless_cancelled( + cancel: &mut watch::Receiver, + fut: impl Future, +) -> Option { + if *cancel.borrow() { + return None; + } + tokio::select! { + // `Err` means the sender is gone, i.e. so is the app. + _ = cancel.changed() => None, + v = fut => Some(v), + } +} + +/// One connection: ping, sweep, then follow the event stream until it ends. +/// `None` when cancelled; otherwise whether the connection was a working one +/// (see [`STABLE_AFTER`]). +async fn watch_once( + store: &ProjectsStore, + cancel: &mut watch::Receiver, + sweep_reason: ExitReason, + on_change: &impl Fn(&Project, ExitReason), +) -> Option { + let Ok(docker) = crate::docker::get_docker() else { return Some(false) }; + if unless_cancelled(cancel, docker.ping()).await?.is_err() { + return Some(false); + } + // A second's overlap: the daemon replays from `since`, so a death during + // the sweep below is still delivered. Duplicates are harmless — `settle` + // confirms with an inspect every time. + let since = (chrono::Utc::now().timestamp() - 1).to_string(); + for p in unless_cancelled(cancel, sweep(&docker, store)).await? { + on_change(&p, sweep_reason); + } + let mut events = docker.events(Some(EventsOptions:: { + since: Some(since), + until: None, + filters: die_filter(), + })); + let connected = Instant::now(); + let mut delivered = false; + loop { + match unless_cancelled(cancel, events.next()).await? { + Some(Ok(msg)) => { + delivered = true; + if let Some(cid) = died_container(&msg) { + if let Some(p) = unless_cancelled(cancel, settle(&docker, store, cid)).await? { + on_change(&p, ExitReason::Exited); + } + } + } + Some(Err(e)) => { + log::info!("Container watch: event stream ended ({e}) — reconnecting"); + break; + } + None => { + log::info!("Container watch: event stream closed — reconnecting"); + break; + } + } + } + Some(delivered || connected.elapsed() >= STABLE_AFTER) +} + /// Watch until `cancel` flips, calling `on_change` for every project whose /// container is found to have stopped on its own. Separate from [`spawn`] so a /// test can drive it without a Tauri app. pub(crate) async fn run( store: Arc, mut cancel: watch::Receiver, - on_change: impl Fn(&Project), + on_change: impl Fn(&Project, ExitReason), ) { - let mut retry = RETRY_MIN; + // Only the first look of the run finds containers that stopped while the + // app was closed. A sweep after a reconnect finds ones that died while we + // were running — the daemon restarting is the usual cause — and those + // cut off live sessions just as a `die` event would. + let mut sweep_reason = ExitReason::FoundStopped; + let mut failures: u32 = 0; loop { + let Some(healthy) = watch_once(&store, &mut cancel, sweep_reason, &on_change).await else { + return; + }; + sweep_reason = ExitReason::Exited; + failures = if healthy { 1 } else { failures + 1 }; + unless_cancelled(&mut cancel, tokio::time::sleep(retry_delay(failures))).await; if *cancel.borrow() { return; } - let docker = match crate::docker::get_docker() { - Ok(d) if d.ping().await.is_ok() => Some(d), - _ => None, - }; - if let Some(docker) = docker { - retry = RETRY_MIN; - // A second's overlap: the daemon replays from `since`, so a death - // during the sweep below is still delivered. Duplicates are - // harmless — `settle` confirms with an inspect every time. - let since = (chrono::Utc::now().timestamp() - 1).to_string(); - for p in sweep(&docker, &store).await { - on_change(&p); - } - let mut events = docker.events(Some(EventsOptions:: { - since: Some(since), - until: None, - filters: die_filter(), - })); - loop { - tokio::select! { - _ = cancel.changed() => return, - next = events.next() => match next { - Some(Ok(msg)) => { - if let Some(cid) = died_container(&msg) { - if let Some(p) = settle(&docker, &store, cid).await { - on_change(&p); - } - } - } - Some(Err(e)) => { - log::info!("Container watch: event stream ended ({e}) — reconnecting"); - break; - } - None => { - log::info!("Container watch: event stream closed — reconnecting"); - break; - } - }, - } - } - } - tokio::select! { - _ = cancel.changed() => return, - _ = tokio::time::sleep(retry) => {} - } - retry = (retry * 2).min(RETRY_MAX); } } @@ -176,13 +249,14 @@ pub fn spawn( cancel: watch::Receiver, ) -> tauri::async_runtime::JoinHandle<()> { tauri::async_runtime::spawn(async move { - run(store, cancel, |p| { + run(store, cancel, |p, reason| { log::warn!( - "Project '{}' ({}): its container stopped without the app asking — marking Stopped", + "Project '{}' ({}): its container stopped without the app asking ({:?}) — marking Stopped", p.name, - p.id + p.id, + reason ); - let _ = app.emit(PROJECT_STATUS_EVENT, p); + let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: p, reason }); }) .await; }) @@ -227,9 +301,47 @@ mod tests { assert_eq!(f.len(), 2); } + #[test] + fn retries_back_off_to_a_minute_and_stay_there() { + let secs: Vec = (1..=7).map(|n| retry_delay(n).as_secs()).collect(); + assert_eq!(secs, vec![5, 10, 20, 40, 60, 60, 60]); + assert_eq!(retry_delay(0), RETRY_MIN); + assert_eq!(retry_delay(u32::MAX), RETRY_MAX); + } + + #[test] + fn the_event_says_why_so_the_ui_can_tell_news_from_housekeeping() { + let p = Project::new("demo".to_string(), Vec::new()); + let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited }).unwrap(); + assert_eq!(v["reason"], "exited"); + assert_eq!(v["project"]["name"], "demo"); + let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::FoundStopped }) + .unwrap(); + assert_eq!(v["reason"], "found_stopped"); + assert_eq!(CONTAINER_EXITED_EVENT, "project-container-exited"); + } + + /// The shutdown path: a Docker call that never answers (a wedged daemon) + /// must give way to cancellation at once, not hold teardown. + #[tokio::test] + async fn a_hung_docker_call_yields_to_cancellation() { + let (tx, mut rx) = watch::channel(false); + let canceller = tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(20)).await; + let _ = tx.send(true); + }); + let hung = std::future::pending::<()>(); + let out = tokio::time::timeout(Duration::from_secs(2), unless_cancelled(&mut rx, hung)).await; + assert_eq!(out.expect("returned promptly"), None); + canceller.await.unwrap(); + // Already cancelled: it does not even start the call. + assert_eq!(unless_cancelled(&mut rx, async { 1 }).await, None); + } + /// End to end against a real daemon: a Running project whose container is - /// killed from outside the app is flipped to Stopped and reported, and one - /// whose container is still up is not. + /// killed from outside the app is flipped to Stopped and reported as a live + /// exit; one found already stopped on the first look is reported as such; + /// and one whose container is still up is left alone. #[tokio::test] #[ignore = "needs a Docker daemon; creates and removes throwaway containers"] async fn a_container_killed_outside_the_app_flips_its_project_to_stopped() { @@ -242,37 +354,43 @@ mod tests { let tag = uuid::Uuid::new_v4().simple().to_string(); let doomed = docker_cli(&["run", "-d", "--name", &format!("tc-watch-a-{tag}"), "alpine:3", "sleep", "300"]); let alive = docker_cli(&["run", "-d", "--name", &format!("tc-watch-b-{tag}"), "alpine:3", "sleep", "300"]); + let already = docker_cli(&["create", "--name", &format!("tc-watch-c-{tag}"), "alpine:3", "true"]); let dir = std::env::temp_dir().join(format!("triple-c-watch-{tag}")); std::fs::create_dir_all(&dir).unwrap(); - let mut a = Project::new("a".to_string(), Vec::new()); - a.container_id = Some(doomed.clone()); - a.status = ProjectStatus::Running; - let mut b = Project::new("b".to_string(), Vec::new()); - b.container_id = Some(alive.clone()); - b.status = ProjectStatus::Running; - let (a_id, b_id) = (a.id.clone(), b.id.clone()); - let store = Arc::new(ProjectsStore::over_file_for_test(dir.join("projects.json"), vec![a, b])); + let running = |name: &str, cid: &str| { + let mut p = Project::new(name.to_string(), Vec::new()); + p.container_id = Some(cid.to_string()); + p.status = ProjectStatus::Running; + p + }; + let (a, b, c) = (running("a", &doomed), running("b", &alive), running("c", &already)); + let (a_id, b_id, c_id) = (a.id.clone(), b.id.clone(), c.id.clone()); + let store = Arc::new(ProjectsStore::over_file_for_test(dir.join("projects.json"), vec![a, b, c])); let (cancel_tx, cancel_rx) = watch::channel(false); - let (seen_tx, mut seen_rx) = tokio::sync::mpsc::unbounded_channel::(); - let watcher = tokio::spawn(run(store.clone(), cancel_rx, move |p| { - let _ = seen_tx.send(p.id.clone()); + let (seen_tx, mut seen_rx) = tokio::sync::mpsc::unbounded_channel::<(String, ExitReason)>(); + let watcher = tokio::spawn(run(store.clone(), cancel_rx, move |p, why| { + let _ = seen_tx.send((p.id.clone(), why)); })); - // Let it subscribe, then kill from "outside". - tokio::time::sleep(Duration::from_secs(2)).await; + let first = tokio::time::timeout(Duration::from_secs(10), seen_rx.recv()).await; + // Subscribed by now (the sweep precedes it, and `since` overlaps); + // kill from "outside". docker_cli(&["kill", &doomed]); + let second = tokio::time::timeout(Duration::from_secs(15), seen_rx.recv()).await; - let seen = tokio::time::timeout(Duration::from_secs(15), seen_rx.recv()).await; + let t0 = Instant::now(); let _ = cancel_tx.send(true); let _ = watcher.await; - let a_status = store.get(&a_id).unwrap().status; - let b_status = store.get(&b_id).unwrap().status; - docker_cli(&["rm", "-f", &doomed, &alive]); + let shutdown = t0.elapsed(); + let statuses: Vec = + [&a_id, &b_id, &c_id].iter().map(|id| store.get(id).unwrap().status).collect(); + docker_cli(&["rm", "-f", &doomed, &alive, &already]); std::fs::remove_dir_all(&dir).ok(); - assert_eq!(seen.expect("reported within 15s"), Some(a_id)); - assert_eq!(a_status, ProjectStatus::Stopped); - assert_eq!(b_status, ProjectStatus::Running); + assert_eq!(first.expect("sweep within 10s"), Some((c_id, ExitReason::FoundStopped))); + assert_eq!(second.expect("die within 15s"), Some((a_id, ExitReason::Exited))); + assert_eq!(statuses, vec![ProjectStatus::Stopped, ProjectStatus::Running, ProjectStatus::Stopped]); + assert!(shutdown < Duration::from_secs(1), "stopped watching in {shutdown:?}"); } } diff --git a/app/src-tauri/src/marketplace/sync.rs b/app/src-tauri/src/marketplace/sync.rs index 4221533..7ce9607 100644 --- a/app/src-tauri/src/marketplace/sync.rs +++ b/app/src-tauri/src/marketplace/sync.rs @@ -23,8 +23,10 @@ pub const SYNC_SCRIPT: &str = include_str!("sync.sh"); /// sleep (so a `pkill sleep` run as `claude` cannot stop the container — see /// the end of `container/entrypoint.sh`); older ones `exec su … claude -c exec /// sleep infinity`, and a project keeps its base image until it is migrated. +/// Both are matched as root-owned only (the old image's `su` runs as root), so +/// a `claude` process wearing either command line cannot fake readiness. const READY_PROBE: &str = "pgrep -u root -x -f 'triple-c-keepalive infinity' >/dev/null \ - || pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null"; + || pgrep -u root -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null"; const READY_TIMEOUT: Duration = Duration::from_secs(180); const READY_POLL: Duration = Duration::from_secs(2); @@ -212,7 +214,7 @@ mod tests { "sh", "-c", "pgrep -u root -x -f 'triple-c-keepalive infinity' >/dev/null \ - || pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null" + || pgrep -u root -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null" ] ); // The marker name is shared with the entrypoint; a rename on one side diff --git a/app/src-tauri/src/storage/projects_store.rs b/app/src-tauri/src/storage/projects_store.rs index 86dd83c..2f3b8ff 100644 --- a/app/src-tauri/src/storage/projects_store.rs +++ b/app/src-tauri/src/storage/projects_store.rs @@ -1,10 +1,42 @@ use std::fs; use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Mutex; use crate::models::marketplace::{MarketplaceInstall, MarketplaceItemRef}; use crate::models::Project; +/// Proof that, at some revision of the store, a project was `Running` on a +/// given container. See [`ProjectsStore::running_on`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct RunningSighting { + revision: u64, +} + +/// What [`ProjectsStore::mark_exited`] did. +#[derive(Debug)] +pub enum ExitMark { + /// Flipped to `Stopped`; the updated record. + Flipped(Box), + /// The store changed since the sighting while a project is still Running + /// on that container — the caller's Docker check may be stale. Re-check. + Changed, + /// No project is Running on that container, or a command owns it. + NotApplicable, +} + +/// The one rule for "this project is Running on that container". Both ids +/// must be at least a short id long: a blank or truncated value is a prefix +/// of everything and would match every container on the daemon. +fn is_running_on(p: &Project, container_id: &str) -> bool { + p.status == crate::models::ProjectStatus::Running + && container_id.len() >= 12 + && p.container_id + .as_deref() + .filter(|stored| stored.len() >= 12) + .is_some_and(|stored| stored.starts_with(container_id) || container_id.starts_with(stored)) +} + /// The sticky marker for `projects.json`: `projects.json.corrupt`, beside it. /// /// Derived from the file rather than from `dirs::data_dir()` so the marker @@ -64,6 +96,8 @@ fn record_corrupt_load(file_path: &Path, now: &chrono::DateTime) { pub struct ProjectsStore { projects: Mutex>, file_path: PathBuf, + /// Bumped by every `save()`; see [`ProjectsStore::mark_exited`]. + revision: AtomicU64, } impl ProjectsStore { @@ -147,6 +181,7 @@ impl ProjectsStore { let store = Self { projects: Mutex::new(projects), file_path, + revision: AtomicU64::new(0), }; // Persist migrated/reconciled format back to disk @@ -164,7 +199,7 @@ impl ProjectsStore { /// A store over a file the test owns (`new()` insists on the real data dir). #[cfg(test)] pub(crate) fn over_file_for_test(file_path: PathBuf, projects: Vec) -> Self { - Self { projects: Mutex::new(projects), file_path } + Self { projects: Mutex::new(projects), file_path, revision: AtomicU64::new(0) } } fn lock(&self) -> std::sync::MutexGuard<'_, Vec> { @@ -172,6 +207,10 @@ impl ProjectsStore { } fn save(&self, projects: &[Project]) -> Result<(), String> { + // Every mutation saves with the lock held, so this counts every change + // to the in-memory list — bumped first, because a failed write has + // still changed what the list holds. + self.revision.fetch_add(1, Ordering::SeqCst); let data = serde_json::to_string_pretty(projects) .map_err(|e| format!("Failed to serialize projects: {}", e))?; @@ -243,34 +282,56 @@ impl ProjectsStore { Ok(()) } - /// A container stopped without the app asking: flip the project that is - /// `Running` on it to `Stopped` and return the updated record, or `None` - /// when there is nothing to flip. + /// Is some project `Running` on `container_id`? Checked under the lock + /// without cloning anything — the container watcher asks this for every + /// container death on the daemon, nearly all of them not ours. /// - /// Compare-and-set under the store lock rather than a read followed by - /// `update_status`, so it cannot land on top of a command that got there - /// first: only a project that is *still* `Running` on *this* container is - /// touched, and never one `is_busy` says a command owns (the project lock - /// — Stop, Start, Reset and migration all hold it across their own - /// container's death and write the status they mean themselves). A start - /// that takes the project lock just after this check still blocks on the - /// store lock before it writes `Starting`, so the two serialise. + /// The answer carries the store's revision, which [`Self::mark_exited`] + /// needs: the caller confirms with Docker that the container really is + /// down *between* the two calls, and that inspect can be slow. + pub fn running_on(&self, container_id: &str) -> Option { + let projects = self.lock(); + projects + .iter() + .any(|p| is_running_on(p, container_id)) + .then(|| RunningSighting { revision: self.revision.load(Ordering::SeqCst) }) + } + + /// A container stopped without the app asking: flip the project that is + /// `Running` on it to `Stopped`. + /// + /// A compare-and-set under the store lock rather than a read followed by + /// `update_status`, and against two things: + /// + /// * **The store's revision as of `seen`.** Between [`Self::running_on`] + /// and here the caller waited on a Docker inspect. If the user stopped + /// and restarted the project meanwhile — same container id, stop/start + /// model — that inspect's "not running" is about a container that has + /// since come back, and flipping would mark a healthy project Stopped. + /// Any write to the store in that window bumps the revision, and the + /// answer is [`ExitMark::Changed`]: confirm again and retry. + /// * **`is_busy`** — the project lock. Stop, Start, Reset and migration + /// hold it across their own container's death and write the status they + /// mean themselves. A start that takes the project lock just after this + /// check still blocks on the store lock before it writes `Starting`. /// /// `container_id` may be the short form; the store holds the full id. - pub fn mark_exited(&self, container_id: &str, is_busy: impl Fn(&str) -> bool) -> Option { - if container_id.len() < 12 { - return None; - } + pub fn mark_exited( + &self, + container_id: &str, + seen: &RunningSighting, + is_busy: impl Fn(&str) -> bool, + ) -> ExitMark { let mut projects = self.lock(); - let p = projects.iter_mut().find(|p| { - p.status == crate::models::ProjectStatus::Running - && p.container_id - .as_deref() - .filter(|stored| stored.len() >= 12) - .is_some_and(|stored| stored.starts_with(container_id) || container_id.starts_with(stored)) - })?; + let unchanged = self.revision.load(Ordering::SeqCst) == seen.revision; + let Some(p) = projects.iter_mut().find(|p| is_running_on(p, container_id)) else { + return ExitMark::NotApplicable; + }; + if !unchanged { + return ExitMark::Changed; + } if is_busy(&p.id) { - return None; + return ExitMark::NotApplicable; } p.status = crate::models::ProjectStatus::Stopped; p.updated_at = chrono::Utc::now().to_rfc3339(); @@ -281,7 +342,7 @@ impl ProjectsStore { // kind writes it out. log::warn!("Could not persist Stopped for project {}: {}", changed.id, e); } - Some(changed) + ExitMark::Flipped(Box::new(changed)) } pub fn update_status(&self, id: &str, status: crate::models::ProjectStatus) -> Result<(), String> { @@ -484,6 +545,7 @@ mod tests { ProjectsStore { projects: Mutex::new(projects), file_path: dir.join("projects.json"), + revision: AtomicU64::new(0), } } @@ -652,7 +714,7 @@ mod tests { fs::remove_dir_all(&dir).ok(); } - // ── mark_exited: a container that died on its own ──────────────────── + // ── running_on / mark_exited: a container that died on its own ─────── /// A full-length id, as Docker's events and the store both carry. const C0FFEE: &str = "c0ffee0000000000000000000000000000000000000000000000000000000000"; @@ -664,6 +726,15 @@ mod tests { p } + /// Sighting and flip back to back, as when Docker answers instantly. + fn sight_and_mark(store: &ProjectsStore, cid: &str, busy: impl Fn(&str) -> bool) -> Option { + let seen = store.running_on(cid)?; + match store.mark_exited(cid, &seen, busy) { + ExitMark::Flipped(p) => Some(*p), + _ => None, + } + } + #[test] fn a_running_project_whose_container_exited_is_marked_stopped_and_saved() { let dir = temp_dir("exited"); @@ -671,7 +742,7 @@ mod tests { let id = p.id.clone(); let store = store_over(&dir, vec![p]); - let changed = store.mark_exited(C0FFEE, |_| false).expect("flipped"); + let changed = sight_and_mark(&store, C0FFEE, |_| false).expect("flipped"); assert_eq!(changed.id, id); assert_eq!(changed.status, crate::models::ProjectStatus::Stopped); @@ -691,15 +762,52 @@ mod tests { let id = p.id.clone(); let store = store_over(&dir, vec![p]); - assert!(store.mark_exited(C0FFEE, |pid| pid == id).is_none()); + assert!(sight_and_mark(&store, C0FFEE, |pid| pid == id).is_none()); assert_eq!(store.get(&id).unwrap().status, crate::models::ProjectStatus::Running); // ...and it was only the claim that held it back. - assert!(store.mark_exited(C0FFEE, |_| false).is_some()); + assert!(sight_and_mark(&store, C0FFEE, |_| false).is_some()); + fs::remove_dir_all(&dir).ok(); + } + + /// The slow-inspect race. The watcher sights the project Running, then + /// waits on Docker; meanwhile the user stops and starts it again on the + /// *same* container id (stop/start model). Docker's "not running" is now + /// about a container that has come back, so the flip must not land. + #[test] + fn a_stop_and_start_during_the_docker_check_invalidates_the_sighting() { + let dir = temp_dir("restarted"); + let p = running_on(C0FFEE); + let id = p.id.clone(); + let store = store_over(&dir, vec![p]); + + let seen = store.running_on(C0FFEE).expect("sighted"); + // ...the inspect is in flight... + store.update_status(&id, crate::models::ProjectStatus::Stopping).unwrap(); + store.update_status(&id, crate::models::ProjectStatus::Stopped).unwrap(); + store.update_status(&id, crate::models::ProjectStatus::Starting).unwrap(); + store.update_status(&id, crate::models::ProjectStatus::Running).unwrap(); + + assert!(matches!(store.mark_exited(C0FFEE, &seen, |_| false), ExitMark::Changed)); + assert_eq!(store.get(&id).unwrap().status, crate::models::ProjectStatus::Running); fs::remove_dir_all(&dir).ok(); } #[test] - fn mark_exited_only_touches_running_projects_on_that_container() { + fn a_stale_sighting_of_a_project_no_longer_running_is_simply_not_applicable() { + let dir = temp_dir("gone"); + let p = running_on(C0FFEE); + let id = p.id.clone(); + let store = store_over(&dir, vec![p]); + + let seen = store.running_on(C0FFEE).expect("sighted"); + store.update_status(&id, crate::models::ProjectStatus::Stopped).unwrap(); + + assert!(matches!(store.mark_exited(C0FFEE, &seen, |_| false), ExitMark::NotApplicable)); + fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn only_running_projects_on_that_container_are_sighted_or_touched() { let dir = temp_dir("other"); let mut starting = running_on(C0FFEE); starting.status = crate::models::ProjectStatus::Starting; @@ -707,15 +815,10 @@ mod tests { let (starting_id, other_id) = (starting.id.clone(), other.id.clone()); let store = store_over(&dir, vec![starting, other]); - assert!(store.mark_exited(C0FFEE, |_| false).is_none(), "Starting is a command's"); - assert!(store.mark_exited(&"d".repeat(64), |_| false).is_none(), "nobody's container"); - assert!(store.mark_exited("", |_| false).is_none()); - // A blank stored id is a prefix of everything; it must match nothing. - let dir2 = temp_dir("blank"); - let blank = running_on(""); - let store2 = store_over(&dir2, vec![blank]); - assert!(store2.mark_exited(&"f".repeat(64), |_| false).is_none()); - fs::remove_dir_all(&dir2).ok(); + assert!(store.running_on(C0FFEE).is_none(), "Starting is a command's"); + assert!(store.running_on(&"d".repeat(64)).is_none(), "nobody's container"); + assert!(store.running_on("").is_none()); + assert!(sight_and_mark(&store, C0FFEE, |_| false).is_none()); assert_eq!( store.get(&starting_id).unwrap().status, crate::models::ProjectStatus::Starting @@ -725,14 +828,23 @@ mod tests { } #[test] - fn mark_exited_matches_a_short_id_against_the_stored_full_one() { + fn a_blank_or_truncated_stored_id_matches_no_container() { + // A blank stored id is a prefix of everything. + let dir = temp_dir("blank"); + let store = store_over(&dir, vec![running_on(""), running_on("abc")]); + assert!(store.running_on(&"f".repeat(64)).is_none()); + assert!(store.running_on("abcdef0123456789").is_none()); + fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn a_short_id_is_matched_against_the_stored_full_one() { let dir = temp_dir("short"); let full = "a".repeat(12) + &"b".repeat(52); - let p = running_on(&full); - let store = store_over(&dir, vec![p]); + let store = store_over(&dir, vec![running_on(&full)]); - assert!(store.mark_exited("aaaa", |_| false).is_none(), "too short to be an id"); - assert!(store.mark_exited(&full[..12], |_| false).is_some()); + assert!(store.running_on("aaaa").is_none(), "too short to be an id"); + assert!(sight_and_mark(&store, &full[..12], |_| false).is_some()); fs::remove_dir_all(&dir).ok(); } } diff --git a/app/src/hooks/useProjectStatusEvents.test.ts b/app/src/hooks/useProjectStatusEvents.test.ts index b3e2c53..dee0635 100644 --- a/app/src/hooks/useProjectStatusEvents.test.ts +++ b/app/src/hooks/useProjectStatusEvents.test.ts @@ -5,11 +5,12 @@ import type { Project } from "../lib/types"; let handler: ((e: { payload: unknown }) => void) | null = null; const unlisten = vi.fn(); +const listen = vi.fn(async (name: string, cb: (e: { payload: unknown }) => void) => { + if (name === "project-container-exited") handler = cb; + return unlisten; +}); vi.mock("@tauri-apps/api/event", () => ({ - listen: vi.fn(async (name: string, cb: (e: { payload: unknown }) => void) => { - if (name === "project-status-changed") handler = cb; - return unlisten; - }), + listen: (name: string, cb: (e: { payload: unknown }) => void) => listen(name, cb), })); import { useProjectStatusEvents } from "./useProjectStatusEvents"; @@ -21,28 +22,53 @@ describe("useProjectStatusEvents", () => { beforeEach(() => { handler = null; unlisten.mockClear(); + listen.mockClear(); useAppState.setState({ projects: [project({}), project({ id: "p2", name: "other" })], toasts: [] }); }); - it("replaces the project the backend reports, so a dead container stops reading Running", async () => { + it("applies a live exit and says why the sessions just died", async () => { renderHook(() => useProjectStatusEvents()); await vi.waitFor(() => expect(handler).not.toBeNull()); - handler!({ payload: project({ status: "stopped" }) }); + handler!({ payload: { project: project({ status: "stopped" }), reason: "exited" } }); const { projects, toasts } = useAppState.getState(); expect(projects.find((p) => p.id === "p1")?.status).toBe("stopped"); expect(projects.find((p) => p.id === "p2")?.status).toBe("running"); - // The terminals just went dead with no explanation; say why. expect(toasts).toHaveLength(1); + expect(toasts[0].kind).toBe("error"); expect(toasts[0].message).toContain("demo"); expect(toasts[0].dedupeKey).toBe("container-exited-p1"); }); + it("applies a container found already stopped at startup silently", async () => { + // After a host reboot every project's container is down before the app + // opens; nothing was interrupted this run, so nothing to alarm anyone about. + renderHook(() => useProjectStatusEvents()); + await vi.waitFor(() => expect(handler).not.toBeNull()); + + handler!({ payload: { project: project({ status: "stopped" }), reason: "found_stopped" } }); + + const { projects, toasts } = useAppState.getState(); + expect(projects.find((p) => p.id === "p1")?.status).toBe("stopped"); + expect(toasts).toHaveLength(0); + }); + it("stops listening on unmount", async () => { const { unmount } = renderHook(() => useProjectStatusEvents()); await vi.waitFor(() => expect(handler).not.toBeNull()); unmount(); expect(unlisten).toHaveBeenCalled(); }); + + it("survives listen() rejecting (no unhandled rejection)", async () => { + const err = vi.spyOn(console, "error").mockImplementation(() => {}); + listen.mockImplementationOnce(async () => { + throw new Error("no IPC"); + }); + const { unmount } = renderHook(() => useProjectStatusEvents()); + await vi.waitFor(() => expect(err).toHaveBeenCalled()); + unmount(); + err.mockRestore(); + }); }); diff --git a/app/src/hooks/useProjectStatusEvents.ts b/app/src/hooks/useProjectStatusEvents.ts index 5483c8b..87a66f8 100644 --- a/app/src/hooks/useProjectStatusEvents.ts +++ b/app/src/hooks/useProjectStatusEvents.ts @@ -3,34 +3,46 @@ import { listen, type UnlistenFn } from "@tauri-apps/api/event"; import { useAppState } from "../store/appState"; import type { Project } from "../lib/types"; +/** `container_watch.rs`'s `ContainerExited`. */ +export interface ContainerExitedEvent { + project: Project; + /** + * `exited`: it died while the app was running — sessions were just cut off. + * `found_stopped`: found down on the app's first look this run (it stopped + * while the app was closed, e.g. a host reboot); nothing was interrupted. + */ + reason: "exited" | "found_stopped"; +} + /** - * App-wide: apply status changes the backend makes on its own. Today that is - * `container_watch.rs` marking a project Stopped when its container exited - * without the app asking — before it, the project kept reading Running, with - * every terminal dead, until the app was restarted. + * App-wide: apply `project-container-exited` — `container_watch.rs` marking a + * project Stopped because its container exited without the app asking. Before + * it, the project kept reading Running, with every terminal dead, until the + * app was restarted. */ export function useProjectStatusEvents() { useEffect(() => { let cancelled = false; let unlisten: UnlistenFn | null = null; - void listen("project-status-changed", (event) => { - const project = event.payload; + listen("project-container-exited", (event) => { + const { project, reason } = event.payload; const state = useAppState.getState(); state.updateProjectInList(project); - if (project.status === "stopped") { - state.pushToast({ - kind: "error", - message: `The container for “${project.name}” stopped`, - detail: - "It exited without Triple-C stopping it — open sessions were disconnected. " + - "Start the project again to reconnect.", - dedupeKey: `container-exited-${project.id}`, - }); - } - }).then((fn) => { - if (cancelled) fn(); - else unlisten = fn; - }); + if (reason !== "exited") return; + state.pushToast({ + kind: "error", + message: `The container for “${project.name}” stopped`, + detail: + "It exited without Triple-C stopping it — open sessions were disconnected. " + + "Start the project again to reconnect.", + dedupeKey: `container-exited-${project.id}`, + }); + }) + .then((fn) => { + if (cancelled) fn(); + else unlisten = fn; + }) + .catch((e) => console.error("Failed to listen for container exits:", e)); return () => { cancelled = true; unlisten?.(); diff --git a/container/entrypoint.sh b/container/entrypoint.sh index e1fd83c..1da6d4a 100644 --- a/container/entrypoint.sh +++ b/container/entrypoint.sh @@ -760,6 +760,12 @@ timeout 120 su -s /bin/bash claude -c 'export PATH="/home/claude/.claude/bin:/ho # command line nothing else has, and it exists only once every step above # has finished — that is the readiness marker. Keep the name in step with # READY_PROBE. +# * Respawns are throttled to about one a second. If the sleep cannot start +# (fork failing under a pids limit or a fork bomb) or dies at once, an +# unthrottled loop would spin a core at 100% exactly when the container is +# already in trouble. The pause is `read -t` on a FIFO nobody writes — +# a builtin, so it still works when fork does not — with `sleep 1` as the +# fallback should the FIFO be unavailable. echo "Triple-C container ready." keepalive_pid="" stop_keepalive() { @@ -767,8 +773,19 @@ stop_keepalive() { exit 0 } trap stop_keepalive TERM INT HUP +throttle_fifo=/run/triple-c-keepalive.throttle +rm -f "$throttle_fifo" +mkfifo -m 600 "$throttle_fifo" 2>/dev/null && exec 3<>"$throttle_fifo" +rm -f "$throttle_fifo" +respawn_pause() { + read -r -t 1 -u 3 _ 2>/dev/null + # >128 is read's timeout, i.e. it paused; anything else means no FIFO. + [ $? -gt 128 ] || sleep 1 +} while :; do - (exec -a triple-c-keepalive sleep infinity) & + started=$SECONDS + (exec -a triple-c-keepalive sleep infinity 3>&-) & keepalive_pid=$! wait "$keepalive_pid" + [ $((SECONDS - started)) -ge 2 ] || respawn_pause done -- 2.52.0