Address review of the keepalive / container-watch change
Secret Scan / scan (push) Successful in 9s
Build App (Preview) / compute-version (pull_request) Successful in 9s
Secret Scan / scan (pull_request) Successful in 8s
Build App (Preview) / create-release (pull_request) Successful in 2s
Build App (Preview) / test (pull_request) Successful in 4m10s
Build App (Preview) / build-macos (pull_request) Successful in 3m12s
Build App (Preview) / build-linux (pull_request) Successful in 9m31s
Build App (Preview) / build-windows (pull_request) Successful in 6m47s
Build App (Preview) / prune-previews (pull_request) Successful in 3s
Build Container / build-container (pull_request) Successful in 17m21s
Secret Scan / scan (push) Successful in 9s
Build App (Preview) / compute-version (pull_request) Successful in 9s
Secret Scan / scan (pull_request) Successful in 8s
Build App (Preview) / create-release (pull_request) Successful in 2s
Build App (Preview) / test (pull_request) Successful in 4m10s
Build App (Preview) / build-macos (pull_request) Successful in 3m12s
Build App (Preview) / build-linux (pull_request) Successful in 9m31s
Build App (Preview) / build-windows (pull_request) Successful in 6m47s
Build App (Preview) / prune-previews (pull_request) Successful in 3s
Build Container / build-container (pull_request) Successful in 17m21s
- Event renamed to `project-container-exited` and carries a reason: `exited` (died while the app was watching) toasts; `found_stopped` (the first look of a run, e.g. after a host reboot) updates silently. Reconnect sweeps report `exited`, since the app was running then. - listen() in useProjectStatusEvents gets a .catch, like its sibling. - Keepalive respawns are throttled to ~1/s with a fork-free pause (`read -t` on a FIFO, `sleep 1` fallback), so a failing fork or an instantly-dying sleep cannot spin a core. Measured: 10748 respawns in 3 s unthrottled vs 3. - Watcher backoff resets only after a working connection (an event delivered, or up for 60 s); ping-OK-but-events-fail now backs off to 60 s instead of re-sweeping every 5 s. - Every Docker call in the watcher (ping, sweep, inspect, stream) races the app's cancellation, so a wedged daemon cannot hold shutdown. - TOCTOU between the inspect and the flip: ProjectsStore now counts writes; running_on() returns a sighting at a revision and mark_exited() refuses with Changed if anything was written since (a stop+start of the same container id included), and settle() re-confirms. - The "Running on this container" rule lives once, in the store, with the 12-char guard on both ids, matched under the lock without cloning. - Marketplace readiness fallback also requires a root-owned process. - TECHNICAL.md no longer documents the `su … sleep infinity` ending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
eba0d95780
commit
2eabf6972b
8 files changed
+456
-169
No files matched your search
@@ -319,7 +319,7 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li
|
||||
`exec su … claude -c "exec sleep infinity"`, and any `pkill sleep` / `kill -9 -1` a Claude
|
||||
session ran killed the whole container. `container_watch.rs` (one Docker `die`-event
|
||||
subscription) is what flips a project whose container exits on its own to Stopped and emits
|
||||
`project-status-changed`; `useProjectStatusEvents` applies it
|
||||
`project-container-exited` with a `reason` (`exited` toasts; `found_stopped`, the first look of a run, is silent); `useProjectStatusEvents` applies it
|
||||
- **`triple-c-scheduler`** — Bash-based scheduled task system for recurring Claude Code invocations
|
||||
|
||||
**`/home/claude` in the image is seed-only.** It is the mount point of the named volume
|
||||
|
||||
+3
-3
@@ -137,7 +137,7 @@ Implementation gotchas for the terminal view and its global controls (merged in
|
||||
│ │
|
||||
│ User: claude (UID/GID remapped to match host) │
|
||||
│ Entrypoint: UID/GID remap → SSH setup → git config → │
|
||||
│ docker socket perms → sleep infinity │
|
||||
│ docker socket perms → root keepalive wait │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -164,7 +164,7 @@ Terminal resize follows the same pattern: `ResizeObserver` detects container siz
|
||||
|
||||
Containers follow a **stop/start** model, not create/destroy:
|
||||
|
||||
1. **First start**: A new container is created with bind mounts, named volumes, environment variables, and labels. The entrypoint remaps UID/GID, configures SSH and git, rebuilds the scheduler crontab, then runs `sleep infinity` to keep the container alive.
|
||||
1. **First start**: A new container is created with bind mounts, named volumes, environment variables, and labels. The entrypoint remaps UID/GID, configures SSH and git, rebuilds the scheduler crontab, then stays resident as root waiting on a root-owned `triple-c-keepalive` sleep — never a process the `claude` user can kill, since any `pkill sleep` a Claude session runs would otherwise stop the container.
|
||||
2. **Terminal open**: `docker exec` launches `claude` with a PTY in the running container, with the permission-mode flags from `PermissionMode::cli_args()` (or `bash -l` for a shell session).
|
||||
3. **Stop**: `docker stop` halts the container but preserves its filesystem. Any packages Claude installed via `apt`, `pip`, `cargo`, etc. survive.
|
||||
4. **Restart**: `docker start` resumes the existing container — unless `container_needs_recreation()` finds a `triple-c.*` label that no longer matches the project's settings, in which case the container is committed to a snapshot image (`triple-c-snapshot-{projectId}:latest`), removed, and recreated from that snapshot. Installed tools survive; the named volumes are untouched.
|
||||
@@ -348,7 +348,7 @@ The entrypoint solves this by:
|
||||
3. Relocating any existing system user/group that conflicts with the target UID/GID.
|
||||
4. Fixing ownership of `/home/claude` after the change.
|
||||
|
||||
This runs as root in the entrypoint, then the final `exec su -s /bin/bash claude -c "exec sleep infinity"` drops to the remapped user.
|
||||
This runs as root in the entrypoint. The entrypoint itself then stays running as root to keep the container alive; sessions reach the remapped `claude` user through `docker exec --user`. (Older images ended in `exec su -s /bin/bash claude -c "exec sleep infinity"`, which let any `claude` process stop the container by killing that sleep.)
|
||||
|
||||
### SSH Key Handling
|
||||
|
||||
|
||||
@@ -14,7 +14,8 @@
|
||||
//! confirmed with one inspect before anything changes (an event can be
|
||||
//! replayed, and a restart can reuse the id), and the flip itself is
|
||||
//! [`ProjectsStore::mark_exited`]'s compare-and-set, which never touches a
|
||||
//! project a command owns. The frontend hears it on [`PROJECT_STATUS_EVENT`].
|
||||
//! project a command owns and refuses if the store changed while the inspect
|
||||
//! was in flight. The frontend hears it on [`CONTAINER_EXITED_EVENT`].
|
||||
//!
|
||||
//! The stream drops whenever the daemon does. Each (re)subscription asks for
|
||||
//! events since just before it, and first re-checks every Running project once —
|
||||
@@ -22,25 +23,54 @@
|
||||
//! its event history, and that is exactly when containers go away.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::future::Future;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bollard::errors::Error as DockerError;
|
||||
use bollard::models::{EventMessage, EventMessageTypeEnum};
|
||||
use bollard::system::EventsOptions;
|
||||
use bollard::Docker;
|
||||
use futures_util::StreamExt;
|
||||
use serde::Serialize;
|
||||
use tauri::{AppHandle, Emitter};
|
||||
use tokio::sync::watch;
|
||||
|
||||
use crate::models::{Project, ProjectStatus};
|
||||
use crate::storage::projects_store::ProjectsStore;
|
||||
use crate::storage::projects_store::{ExitMark, ProjectsStore};
|
||||
|
||||
/// Emitted with the updated `Project` whenever this module changes a status.
|
||||
pub const PROJECT_STATUS_EVENT: &str = "project-status-changed";
|
||||
/// Emitted as a [`ContainerExited`] whenever this module flips a project to
|
||||
/// Stopped.
|
||||
pub const CONTAINER_EXITED_EVENT: &str = "project-container-exited";
|
||||
|
||||
const RETRY_MIN: Duration = Duration::from_secs(5);
|
||||
const RETRY_MAX: Duration = Duration::from_secs(60);
|
||||
/// A stream that stayed up this long (or delivered anything) was a working
|
||||
/// connection, so its loss starts the backoff over. Anything shorter — the
|
||||
/// daemon answers ping but the events call fails — keeps backing off, or it
|
||||
/// would re-sweep and re-log every five seconds forever.
|
||||
const STABLE_AFTER: Duration = Duration::from_secs(60);
|
||||
/// How often `settle` re-confirms when the store keeps changing under it.
|
||||
const SETTLE_ATTEMPTS: usize = 3;
|
||||
|
||||
/// How the app found out, which decides whether it is news to the user.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ExitReason {
|
||||
/// The container died while this app was running and watching — any
|
||||
/// sessions it had were just cut off.
|
||||
Exited,
|
||||
/// Found already stopped on the first look of this run: it went down
|
||||
/// while the app was closed (a host reboot, typically). Nothing the user
|
||||
/// can see was interrupted, so it is not news.
|
||||
FoundStopped,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct ContainerExited<'a> {
|
||||
pub project: &'a Project,
|
||||
pub reason: ExitReason,
|
||||
}
|
||||
|
||||
/// The container id a Docker event reports as having died, if it is one.
|
||||
pub(crate) fn died_container(msg: &EventMessage) -> Option<&str> {
|
||||
@@ -70,19 +100,23 @@ fn project_busy(project_id: &str) -> bool {
|
||||
/// Confirm `container_id` really stopped and, if a project was Running on it,
|
||||
/// flip that project. Returns what changed.
|
||||
async fn settle(docker: &Docker, store: &ProjectsStore, container_id: &str) -> Option<Project> {
|
||||
// The cheap filter first: most containers that die are not ours (probe and
|
||||
// scratch containers, other software on the daemon), and those cost no
|
||||
// inspect at all.
|
||||
let ours = store.list().into_iter().any(|p| {
|
||||
p.status == ProjectStatus::Running
|
||||
&& p.container_id
|
||||
.as_deref()
|
||||
.is_some_and(|c| c.starts_with(container_id) || container_id.starts_with(c))
|
||||
});
|
||||
if !ours || !has_exited(docker, container_id).await {
|
||||
return None;
|
||||
for _ in 0..SETTLE_ATTEMPTS {
|
||||
// Cheap and allocation-free: most containers that die are not ours
|
||||
// (probe and scratch containers, other software on the daemon), and
|
||||
// those cost no inspect at all.
|
||||
let seen = store.running_on(container_id)?;
|
||||
if !has_exited(docker, container_id).await {
|
||||
return None;
|
||||
}
|
||||
match store.mark_exited(container_id, &seen, project_busy) {
|
||||
ExitMark::Flipped(p) => return Some(*p),
|
||||
ExitMark::NotApplicable => return None,
|
||||
// The store moved while Docker was answering — possibly a stop and
|
||||
// start of this very container. The inspect is stale; ask again.
|
||||
ExitMark::Changed => continue,
|
||||
}
|
||||
}
|
||||
store.mark_exited(container_id, project_busy)
|
||||
None
|
||||
}
|
||||
|
||||
/// Re-check every Running project once. Used on each (re)subscription.
|
||||
@@ -107,65 +141,104 @@ fn die_filter() -> HashMap<String, Vec<String>> {
|
||||
])
|
||||
}
|
||||
|
||||
/// The wait before connection attempt `n + 1`, after `n` consecutive
|
||||
/// unproductive ones: 5 s, 10 s, 20 s, 40 s, then 60 s for good.
|
||||
fn retry_delay(consecutive_failures: u32) -> Duration {
|
||||
let doublings = consecutive_failures.saturating_sub(1).min(8);
|
||||
(RETRY_MIN * 2u32.pow(doublings)).min(RETRY_MAX)
|
||||
}
|
||||
|
||||
/// `fut`, unless the app starts shutting down first. Every Docker call in here
|
||||
/// goes through this: a wedged daemon must not hold shutdown for its full
|
||||
/// budget.
|
||||
async fn unless_cancelled<T>(
|
||||
cancel: &mut watch::Receiver<bool>,
|
||||
fut: impl Future<Output = T>,
|
||||
) -> Option<T> {
|
||||
if *cancel.borrow() {
|
||||
return None;
|
||||
}
|
||||
tokio::select! {
|
||||
// `Err` means the sender is gone, i.e. so is the app.
|
||||
_ = cancel.changed() => None,
|
||||
v = fut => Some(v),
|
||||
}
|
||||
}
|
||||
|
||||
/// One connection: ping, sweep, then follow the event stream until it ends.
|
||||
/// `None` when cancelled; otherwise whether the connection was a working one
|
||||
/// (see [`STABLE_AFTER`]).
|
||||
async fn watch_once(
|
||||
store: &ProjectsStore,
|
||||
cancel: &mut watch::Receiver<bool>,
|
||||
sweep_reason: ExitReason,
|
||||
on_change: &impl Fn(&Project, ExitReason),
|
||||
) -> Option<bool> {
|
||||
let Ok(docker) = crate::docker::get_docker() else { return Some(false) };
|
||||
if unless_cancelled(cancel, docker.ping()).await?.is_err() {
|
||||
return Some(false);
|
||||
}
|
||||
// A second's overlap: the daemon replays from `since`, so a death during
|
||||
// the sweep below is still delivered. Duplicates are harmless — `settle`
|
||||
// confirms with an inspect every time.
|
||||
let since = (chrono::Utc::now().timestamp() - 1).to_string();
|
||||
for p in unless_cancelled(cancel, sweep(&docker, store)).await? {
|
||||
on_change(&p, sweep_reason);
|
||||
}
|
||||
let mut events = docker.events(Some(EventsOptions::<String> {
|
||||
since: Some(since),
|
||||
until: None,
|
||||
filters: die_filter(),
|
||||
}));
|
||||
let connected = Instant::now();
|
||||
let mut delivered = false;
|
||||
loop {
|
||||
match unless_cancelled(cancel, events.next()).await? {
|
||||
Some(Ok(msg)) => {
|
||||
delivered = true;
|
||||
if let Some(cid) = died_container(&msg) {
|
||||
if let Some(p) = unless_cancelled(cancel, settle(&docker, store, cid)).await? {
|
||||
on_change(&p, ExitReason::Exited);
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(Err(e)) => {
|
||||
log::info!("Container watch: event stream ended ({e}) — reconnecting");
|
||||
break;
|
||||
}
|
||||
None => {
|
||||
log::info!("Container watch: event stream closed — reconnecting");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(delivered || connected.elapsed() >= STABLE_AFTER)
|
||||
}
|
||||
|
||||
/// Watch until `cancel` flips, calling `on_change` for every project whose
|
||||
/// container is found to have stopped on its own. Separate from [`spawn`] so a
|
||||
/// test can drive it without a Tauri app.
|
||||
pub(crate) async fn run(
|
||||
store: Arc<ProjectsStore>,
|
||||
mut cancel: watch::Receiver<bool>,
|
||||
on_change: impl Fn(&Project),
|
||||
on_change: impl Fn(&Project, ExitReason),
|
||||
) {
|
||||
let mut retry = RETRY_MIN;
|
||||
// Only the first look of the run finds containers that stopped while the
|
||||
// app was closed. A sweep after a reconnect finds ones that died while we
|
||||
// were running — the daemon restarting is the usual cause — and those
|
||||
// cut off live sessions just as a `die` event would.
|
||||
let mut sweep_reason = ExitReason::FoundStopped;
|
||||
let mut failures: u32 = 0;
|
||||
loop {
|
||||
let Some(healthy) = watch_once(&store, &mut cancel, sweep_reason, &on_change).await else {
|
||||
return;
|
||||
};
|
||||
sweep_reason = ExitReason::Exited;
|
||||
failures = if healthy { 1 } else { failures + 1 };
|
||||
unless_cancelled(&mut cancel, tokio::time::sleep(retry_delay(failures))).await;
|
||||
if *cancel.borrow() {
|
||||
return;
|
||||
}
|
||||
let docker = match crate::docker::get_docker() {
|
||||
Ok(d) if d.ping().await.is_ok() => Some(d),
|
||||
_ => None,
|
||||
};
|
||||
if let Some(docker) = docker {
|
||||
retry = RETRY_MIN;
|
||||
// A second's overlap: the daemon replays from `since`, so a death
|
||||
// during the sweep below is still delivered. Duplicates are
|
||||
// harmless — `settle` confirms with an inspect every time.
|
||||
let since = (chrono::Utc::now().timestamp() - 1).to_string();
|
||||
for p in sweep(&docker, &store).await {
|
||||
on_change(&p);
|
||||
}
|
||||
let mut events = docker.events(Some(EventsOptions::<String> {
|
||||
since: Some(since),
|
||||
until: None,
|
||||
filters: die_filter(),
|
||||
}));
|
||||
loop {
|
||||
tokio::select! {
|
||||
_ = cancel.changed() => return,
|
||||
next = events.next() => match next {
|
||||
Some(Ok(msg)) => {
|
||||
if let Some(cid) = died_container(&msg) {
|
||||
if let Some(p) = settle(&docker, &store, cid).await {
|
||||
on_change(&p);
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(Err(e)) => {
|
||||
log::info!("Container watch: event stream ended ({e}) — reconnecting");
|
||||
break;
|
||||
}
|
||||
None => {
|
||||
log::info!("Container watch: event stream closed — reconnecting");
|
||||
break;
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
tokio::select! {
|
||||
_ = cancel.changed() => return,
|
||||
_ = tokio::time::sleep(retry) => {}
|
||||
}
|
||||
retry = (retry * 2).min(RETRY_MAX);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -176,13 +249,14 @@ pub fn spawn(
|
||||
cancel: watch::Receiver<bool>,
|
||||
) -> tauri::async_runtime::JoinHandle<()> {
|
||||
tauri::async_runtime::spawn(async move {
|
||||
run(store, cancel, |p| {
|
||||
run(store, cancel, |p, reason| {
|
||||
log::warn!(
|
||||
"Project '{}' ({}): its container stopped without the app asking — marking Stopped",
|
||||
"Project '{}' ({}): its container stopped without the app asking ({:?}) — marking Stopped",
|
||||
p.name,
|
||||
p.id
|
||||
p.id,
|
||||
reason
|
||||
);
|
||||
let _ = app.emit(PROJECT_STATUS_EVENT, p);
|
||||
let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: p, reason });
|
||||
})
|
||||
.await;
|
||||
})
|
||||
@@ -227,9 +301,47 @@ mod tests {
|
||||
assert_eq!(f.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retries_back_off_to_a_minute_and_stay_there() {
|
||||
let secs: Vec<u64> = (1..=7).map(|n| retry_delay(n).as_secs()).collect();
|
||||
assert_eq!(secs, vec![5, 10, 20, 40, 60, 60, 60]);
|
||||
assert_eq!(retry_delay(0), RETRY_MIN);
|
||||
assert_eq!(retry_delay(u32::MAX), RETRY_MAX);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_event_says_why_so_the_ui_can_tell_news_from_housekeeping() {
|
||||
let p = Project::new("demo".to_string(), Vec::new());
|
||||
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited }).unwrap();
|
||||
assert_eq!(v["reason"], "exited");
|
||||
assert_eq!(v["project"]["name"], "demo");
|
||||
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::FoundStopped })
|
||||
.unwrap();
|
||||
assert_eq!(v["reason"], "found_stopped");
|
||||
assert_eq!(CONTAINER_EXITED_EVENT, "project-container-exited");
|
||||
}
|
||||
|
||||
/// The shutdown path: a Docker call that never answers (a wedged daemon)
|
||||
/// must give way to cancellation at once, not hold teardown.
|
||||
#[tokio::test]
|
||||
async fn a_hung_docker_call_yields_to_cancellation() {
|
||||
let (tx, mut rx) = watch::channel(false);
|
||||
let canceller = tokio::spawn(async move {
|
||||
tokio::time::sleep(Duration::from_millis(20)).await;
|
||||
let _ = tx.send(true);
|
||||
});
|
||||
let hung = std::future::pending::<()>();
|
||||
let out = tokio::time::timeout(Duration::from_secs(2), unless_cancelled(&mut rx, hung)).await;
|
||||
assert_eq!(out.expect("returned promptly"), None);
|
||||
canceller.await.unwrap();
|
||||
// Already cancelled: it does not even start the call.
|
||||
assert_eq!(unless_cancelled(&mut rx, async { 1 }).await, None);
|
||||
}
|
||||
|
||||
/// End to end against a real daemon: a Running project whose container is
|
||||
/// killed from outside the app is flipped to Stopped and reported, and one
|
||||
/// whose container is still up is not.
|
||||
/// killed from outside the app is flipped to Stopped and reported as a live
|
||||
/// exit; one found already stopped on the first look is reported as such;
|
||||
/// and one whose container is still up is left alone.
|
||||
#[tokio::test]
|
||||
#[ignore = "needs a Docker daemon; creates and removes throwaway containers"]
|
||||
async fn a_container_killed_outside_the_app_flips_its_project_to_stopped() {
|
||||
@@ -242,37 +354,43 @@ mod tests {
|
||||
let tag = uuid::Uuid::new_v4().simple().to_string();
|
||||
let doomed = docker_cli(&["run", "-d", "--name", &format!("tc-watch-a-{tag}"), "alpine:3", "sleep", "300"]);
|
||||
let alive = docker_cli(&["run", "-d", "--name", &format!("tc-watch-b-{tag}"), "alpine:3", "sleep", "300"]);
|
||||
let already = docker_cli(&["create", "--name", &format!("tc-watch-c-{tag}"), "alpine:3", "true"]);
|
||||
|
||||
let dir = std::env::temp_dir().join(format!("triple-c-watch-{tag}"));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let mut a = Project::new("a".to_string(), Vec::new());
|
||||
a.container_id = Some(doomed.clone());
|
||||
a.status = ProjectStatus::Running;
|
||||
let mut b = Project::new("b".to_string(), Vec::new());
|
||||
b.container_id = Some(alive.clone());
|
||||
b.status = ProjectStatus::Running;
|
||||
let (a_id, b_id) = (a.id.clone(), b.id.clone());
|
||||
let store = Arc::new(ProjectsStore::over_file_for_test(dir.join("projects.json"), vec![a, b]));
|
||||
let running = |name: &str, cid: &str| {
|
||||
let mut p = Project::new(name.to_string(), Vec::new());
|
||||
p.container_id = Some(cid.to_string());
|
||||
p.status = ProjectStatus::Running;
|
||||
p
|
||||
};
|
||||
let (a, b, c) = (running("a", &doomed), running("b", &alive), running("c", &already));
|
||||
let (a_id, b_id, c_id) = (a.id.clone(), b.id.clone(), c.id.clone());
|
||||
let store = Arc::new(ProjectsStore::over_file_for_test(dir.join("projects.json"), vec![a, b, c]));
|
||||
|
||||
let (cancel_tx, cancel_rx) = watch::channel(false);
|
||||
let (seen_tx, mut seen_rx) = tokio::sync::mpsc::unbounded_channel::<String>();
|
||||
let watcher = tokio::spawn(run(store.clone(), cancel_rx, move |p| {
|
||||
let _ = seen_tx.send(p.id.clone());
|
||||
let (seen_tx, mut seen_rx) = tokio::sync::mpsc::unbounded_channel::<(String, ExitReason)>();
|
||||
let watcher = tokio::spawn(run(store.clone(), cancel_rx, move |p, why| {
|
||||
let _ = seen_tx.send((p.id.clone(), why));
|
||||
}));
|
||||
// Let it subscribe, then kill from "outside".
|
||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
||||
let first = tokio::time::timeout(Duration::from_secs(10), seen_rx.recv()).await;
|
||||
// Subscribed by now (the sweep precedes it, and `since` overlaps);
|
||||
// kill from "outside".
|
||||
docker_cli(&["kill", &doomed]);
|
||||
let second = tokio::time::timeout(Duration::from_secs(15), seen_rx.recv()).await;
|
||||
|
||||
let seen = tokio::time::timeout(Duration::from_secs(15), seen_rx.recv()).await;
|
||||
let t0 = Instant::now();
|
||||
let _ = cancel_tx.send(true);
|
||||
let _ = watcher.await;
|
||||
let a_status = store.get(&a_id).unwrap().status;
|
||||
let b_status = store.get(&b_id).unwrap().status;
|
||||
docker_cli(&["rm", "-f", &doomed, &alive]);
|
||||
let shutdown = t0.elapsed();
|
||||
let statuses: Vec<ProjectStatus> =
|
||||
[&a_id, &b_id, &c_id].iter().map(|id| store.get(id).unwrap().status).collect();
|
||||
docker_cli(&["rm", "-f", &doomed, &alive, &already]);
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
|
||||
assert_eq!(seen.expect("reported within 15s"), Some(a_id));
|
||||
assert_eq!(a_status, ProjectStatus::Stopped);
|
||||
assert_eq!(b_status, ProjectStatus::Running);
|
||||
assert_eq!(first.expect("sweep within 10s"), Some((c_id, ExitReason::FoundStopped)));
|
||||
assert_eq!(second.expect("die within 15s"), Some((a_id, ExitReason::Exited)));
|
||||
assert_eq!(statuses, vec![ProjectStatus::Stopped, ProjectStatus::Running, ProjectStatus::Stopped]);
|
||||
assert!(shutdown < Duration::from_secs(1), "stopped watching in {shutdown:?}");
|
||||
}
|
||||
}
|
||||
@@ -23,8 +23,10 @@ pub const SYNC_SCRIPT: &str = include_str!("sync.sh");
|
||||
/// sleep (so a `pkill sleep` run as `claude` cannot stop the container — see
|
||||
/// the end of `container/entrypoint.sh`); older ones `exec su … claude -c exec
|
||||
/// sleep infinity`, and a project keeps its base image until it is migrated.
|
||||
/// Both are matched as root-owned only (the old image's `su` runs as root), so
|
||||
/// a `claude` process wearing either command line cannot fake readiness.
|
||||
const READY_PROBE: &str = "pgrep -u root -x -f 'triple-c-keepalive infinity' >/dev/null \
|
||||
|| pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null";
|
||||
|| pgrep -u root -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null";
|
||||
const READY_TIMEOUT: Duration = Duration::from_secs(180);
|
||||
const READY_POLL: Duration = Duration::from_secs(2);
|
||||
|
||||
@@ -212,7 +214,7 @@ mod tests {
|
||||
"sh",
|
||||
"-c",
|
||||
"pgrep -u root -x -f 'triple-c-keepalive infinity' >/dev/null \
|
||||
|| pgrep -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null"
|
||||
|| pgrep -u root -x -f 'su -s /bin/bash claude -c exec sleep infinity' >/dev/null"
|
||||
]
|
||||
);
|
||||
// The marker name is shared with the entrypoint; a rename on one side
|
||||
|
||||
@@ -1,10 +1,42 @@
|
||||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::Mutex;
|
||||
|
||||
use crate::models::marketplace::{MarketplaceInstall, MarketplaceItemRef};
|
||||
use crate::models::Project;
|
||||
|
||||
/// Proof that, at some revision of the store, a project was `Running` on a
|
||||
/// given container. See [`ProjectsStore::running_on`].
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct RunningSighting {
|
||||
revision: u64,
|
||||
}
|
||||
|
||||
/// What [`ProjectsStore::mark_exited`] did.
|
||||
#[derive(Debug)]
|
||||
pub enum ExitMark {
|
||||
/// Flipped to `Stopped`; the updated record.
|
||||
Flipped(Box<Project>),
|
||||
/// The store changed since the sighting while a project is still Running
|
||||
/// on that container — the caller's Docker check may be stale. Re-check.
|
||||
Changed,
|
||||
/// No project is Running on that container, or a command owns it.
|
||||
NotApplicable,
|
||||
}
|
||||
|
||||
/// The one rule for "this project is Running on that container". Both ids
|
||||
/// must be at least a short id long: a blank or truncated value is a prefix
|
||||
/// of everything and would match every container on the daemon.
|
||||
fn is_running_on(p: &Project, container_id: &str) -> bool {
|
||||
p.status == crate::models::ProjectStatus::Running
|
||||
&& container_id.len() >= 12
|
||||
&& p.container_id
|
||||
.as_deref()
|
||||
.filter(|stored| stored.len() >= 12)
|
||||
.is_some_and(|stored| stored.starts_with(container_id) || container_id.starts_with(stored))
|
||||
}
|
||||
|
||||
/// The sticky marker for `projects.json`: `projects.json.corrupt`, beside it.
|
||||
///
|
||||
/// Derived from the file rather than from `dirs::data_dir()` so the marker
|
||||
@@ -64,6 +96,8 @@ fn record_corrupt_load(file_path: &Path, now: &chrono::DateTime<chrono::Utc>) {
|
||||
pub struct ProjectsStore {
|
||||
projects: Mutex<Vec<Project>>,
|
||||
file_path: PathBuf,
|
||||
/// Bumped by every `save()`; see [`ProjectsStore::mark_exited`].
|
||||
revision: AtomicU64,
|
||||
}
|
||||
|
||||
impl ProjectsStore {
|
||||
@@ -147,6 +181,7 @@ impl ProjectsStore {
|
||||
let store = Self {
|
||||
projects: Mutex::new(projects),
|
||||
file_path,
|
||||
revision: AtomicU64::new(0),
|
||||
};
|
||||
|
||||
// Persist migrated/reconciled format back to disk
|
||||
@@ -164,7 +199,7 @@ impl ProjectsStore {
|
||||
/// A store over a file the test owns (`new()` insists on the real data dir).
|
||||
#[cfg(test)]
|
||||
pub(crate) fn over_file_for_test(file_path: PathBuf, projects: Vec<Project>) -> Self {
|
||||
Self { projects: Mutex::new(projects), file_path }
|
||||
Self { projects: Mutex::new(projects), file_path, revision: AtomicU64::new(0) }
|
||||
}
|
||||
|
||||
fn lock(&self) -> std::sync::MutexGuard<'_, Vec<Project>> {
|
||||
@@ -172,6 +207,10 @@ impl ProjectsStore {
|
||||
}
|
||||
|
||||
fn save(&self, projects: &[Project]) -> Result<(), String> {
|
||||
// Every mutation saves with the lock held, so this counts every change
|
||||
// to the in-memory list — bumped first, because a failed write has
|
||||
// still changed what the list holds.
|
||||
self.revision.fetch_add(1, Ordering::SeqCst);
|
||||
let data = serde_json::to_string_pretty(projects)
|
||||
.map_err(|e| format!("Failed to serialize projects: {}", e))?;
|
||||
|
||||
@@ -243,34 +282,56 @@ impl ProjectsStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A container stopped without the app asking: flip the project that is
|
||||
/// `Running` on it to `Stopped` and return the updated record, or `None`
|
||||
/// when there is nothing to flip.
|
||||
/// Is some project `Running` on `container_id`? Checked under the lock
|
||||
/// without cloning anything — the container watcher asks this for every
|
||||
/// container death on the daemon, nearly all of them not ours.
|
||||
///
|
||||
/// Compare-and-set under the store lock rather than a read followed by
|
||||
/// `update_status`, so it cannot land on top of a command that got there
|
||||
/// first: only a project that is *still* `Running` on *this* container is
|
||||
/// touched, and never one `is_busy` says a command owns (the project lock
|
||||
/// — Stop, Start, Reset and migration all hold it across their own
|
||||
/// container's death and write the status they mean themselves). A start
|
||||
/// that takes the project lock just after this check still blocks on the
|
||||
/// store lock before it writes `Starting`, so the two serialise.
|
||||
/// The answer carries the store's revision, which [`Self::mark_exited`]
|
||||
/// needs: the caller confirms with Docker that the container really is
|
||||
/// down *between* the two calls, and that inspect can be slow.
|
||||
pub fn running_on(&self, container_id: &str) -> Option<RunningSighting> {
|
||||
let projects = self.lock();
|
||||
projects
|
||||
.iter()
|
||||
.any(|p| is_running_on(p, container_id))
|
||||
.then(|| RunningSighting { revision: self.revision.load(Ordering::SeqCst) })
|
||||
}
|
||||
|
||||
/// A container stopped without the app asking: flip the project that is
|
||||
/// `Running` on it to `Stopped`.
|
||||
///
|
||||
/// A compare-and-set under the store lock rather than a read followed by
|
||||
/// `update_status`, and against two things:
|
||||
///
|
||||
/// * **The store's revision as of `seen`.** Between [`Self::running_on`]
|
||||
/// and here the caller waited on a Docker inspect. If the user stopped
|
||||
/// and restarted the project meanwhile — same container id, stop/start
|
||||
/// model — that inspect's "not running" is about a container that has
|
||||
/// since come back, and flipping would mark a healthy project Stopped.
|
||||
/// Any write to the store in that window bumps the revision, and the
|
||||
/// answer is [`ExitMark::Changed`]: confirm again and retry.
|
||||
/// * **`is_busy`** — the project lock. Stop, Start, Reset and migration
|
||||
/// hold it across their own container's death and write the status they
|
||||
/// mean themselves. A start that takes the project lock just after this
|
||||
/// check still blocks on the store lock before it writes `Starting`.
|
||||
///
|
||||
/// `container_id` may be the short form; the store holds the full id.
|
||||
pub fn mark_exited(&self, container_id: &str, is_busy: impl Fn(&str) -> bool) -> Option<Project> {
|
||||
if container_id.len() < 12 {
|
||||
return None;
|
||||
}
|
||||
pub fn mark_exited(
|
||||
&self,
|
||||
container_id: &str,
|
||||
seen: &RunningSighting,
|
||||
is_busy: impl Fn(&str) -> bool,
|
||||
) -> ExitMark {
|
||||
let mut projects = self.lock();
|
||||
let p = projects.iter_mut().find(|p| {
|
||||
p.status == crate::models::ProjectStatus::Running
|
||||
&& p.container_id
|
||||
.as_deref()
|
||||
.filter(|stored| stored.len() >= 12)
|
||||
.is_some_and(|stored| stored.starts_with(container_id) || container_id.starts_with(stored))
|
||||
})?;
|
||||
let unchanged = self.revision.load(Ordering::SeqCst) == seen.revision;
|
||||
let Some(p) = projects.iter_mut().find(|p| is_running_on(p, container_id)) else {
|
||||
return ExitMark::NotApplicable;
|
||||
};
|
||||
if !unchanged {
|
||||
return ExitMark::Changed;
|
||||
}
|
||||
if is_busy(&p.id) {
|
||||
return None;
|
||||
return ExitMark::NotApplicable;
|
||||
}
|
||||
p.status = crate::models::ProjectStatus::Stopped;
|
||||
p.updated_at = chrono::Utc::now().to_rfc3339();
|
||||
@@ -281,7 +342,7 @@ impl ProjectsStore {
|
||||
// kind writes it out.
|
||||
log::warn!("Could not persist Stopped for project {}: {}", changed.id, e);
|
||||
}
|
||||
Some(changed)
|
||||
ExitMark::Flipped(Box::new(changed))
|
||||
}
|
||||
|
||||
pub fn update_status(&self, id: &str, status: crate::models::ProjectStatus) -> Result<(), String> {
|
||||
@@ -484,6 +545,7 @@ mod tests {
|
||||
ProjectsStore {
|
||||
projects: Mutex::new(projects),
|
||||
file_path: dir.join("projects.json"),
|
||||
revision: AtomicU64::new(0),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -652,7 +714,7 @@ mod tests {
|
||||
fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
// ── mark_exited: a container that died on its own ────────────────────
|
||||
// ── running_on / mark_exited: a container that died on its own ───────
|
||||
|
||||
/// A full-length id, as Docker's events and the store both carry.
|
||||
const C0FFEE: &str = "c0ffee0000000000000000000000000000000000000000000000000000000000";
|
||||
@@ -664,6 +726,15 @@ mod tests {
|
||||
p
|
||||
}
|
||||
|
||||
/// Sighting and flip back to back, as when Docker answers instantly.
|
||||
fn sight_and_mark(store: &ProjectsStore, cid: &str, busy: impl Fn(&str) -> bool) -> Option<Project> {
|
||||
let seen = store.running_on(cid)?;
|
||||
match store.mark_exited(cid, &seen, busy) {
|
||||
ExitMark::Flipped(p) => Some(*p),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_running_project_whose_container_exited_is_marked_stopped_and_saved() {
|
||||
let dir = temp_dir("exited");
|
||||
@@ -671,7 +742,7 @@ mod tests {
|
||||
let id = p.id.clone();
|
||||
let store = store_over(&dir, vec![p]);
|
||||
|
||||
let changed = store.mark_exited(C0FFEE, |_| false).expect("flipped");
|
||||
let changed = sight_and_mark(&store, C0FFEE, |_| false).expect("flipped");
|
||||
|
||||
assert_eq!(changed.id, id);
|
||||
assert_eq!(changed.status, crate::models::ProjectStatus::Stopped);
|
||||
@@ -691,15 +762,52 @@ mod tests {
|
||||
let id = p.id.clone();
|
||||
let store = store_over(&dir, vec![p]);
|
||||
|
||||
assert!(store.mark_exited(C0FFEE, |pid| pid == id).is_none());
|
||||
assert!(sight_and_mark(&store, C0FFEE, |pid| pid == id).is_none());
|
||||
assert_eq!(store.get(&id).unwrap().status, crate::models::ProjectStatus::Running);
|
||||
// ...and it was only the claim that held it back.
|
||||
assert!(store.mark_exited(C0FFEE, |_| false).is_some());
|
||||
assert!(sight_and_mark(&store, C0FFEE, |_| false).is_some());
|
||||
fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
/// The slow-inspect race. The watcher sights the project Running, then
|
||||
/// waits on Docker; meanwhile the user stops and starts it again on the
|
||||
/// *same* container id (stop/start model). Docker's "not running" is now
|
||||
/// about a container that has come back, so the flip must not land.
|
||||
#[test]
|
||||
fn a_stop_and_start_during_the_docker_check_invalidates_the_sighting() {
|
||||
let dir = temp_dir("restarted");
|
||||
let p = running_on(C0FFEE);
|
||||
let id = p.id.clone();
|
||||
let store = store_over(&dir, vec![p]);
|
||||
|
||||
let seen = store.running_on(C0FFEE).expect("sighted");
|
||||
// ...the inspect is in flight...
|
||||
store.update_status(&id, crate::models::ProjectStatus::Stopping).unwrap();
|
||||
store.update_status(&id, crate::models::ProjectStatus::Stopped).unwrap();
|
||||
store.update_status(&id, crate::models::ProjectStatus::Starting).unwrap();
|
||||
store.update_status(&id, crate::models::ProjectStatus::Running).unwrap();
|
||||
|
||||
assert!(matches!(store.mark_exited(C0FFEE, &seen, |_| false), ExitMark::Changed));
|
||||
assert_eq!(store.get(&id).unwrap().status, crate::models::ProjectStatus::Running);
|
||||
fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mark_exited_only_touches_running_projects_on_that_container() {
|
||||
fn a_stale_sighting_of_a_project_no_longer_running_is_simply_not_applicable() {
|
||||
let dir = temp_dir("gone");
|
||||
let p = running_on(C0FFEE);
|
||||
let id = p.id.clone();
|
||||
let store = store_over(&dir, vec![p]);
|
||||
|
||||
let seen = store.running_on(C0FFEE).expect("sighted");
|
||||
store.update_status(&id, crate::models::ProjectStatus::Stopped).unwrap();
|
||||
|
||||
assert!(matches!(store.mark_exited(C0FFEE, &seen, |_| false), ExitMark::NotApplicable));
|
||||
fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_running_projects_on_that_container_are_sighted_or_touched() {
|
||||
let dir = temp_dir("other");
|
||||
let mut starting = running_on(C0FFEE);
|
||||
starting.status = crate::models::ProjectStatus::Starting;
|
||||
@@ -707,15 +815,10 @@ mod tests {
|
||||
let (starting_id, other_id) = (starting.id.clone(), other.id.clone());
|
||||
let store = store_over(&dir, vec![starting, other]);
|
||||
|
||||
assert!(store.mark_exited(C0FFEE, |_| false).is_none(), "Starting is a command's");
|
||||
assert!(store.mark_exited(&"d".repeat(64), |_| false).is_none(), "nobody's container");
|
||||
assert!(store.mark_exited("", |_| false).is_none());
|
||||
// A blank stored id is a prefix of everything; it must match nothing.
|
||||
let dir2 = temp_dir("blank");
|
||||
let blank = running_on("");
|
||||
let store2 = store_over(&dir2, vec![blank]);
|
||||
assert!(store2.mark_exited(&"f".repeat(64), |_| false).is_none());
|
||||
fs::remove_dir_all(&dir2).ok();
|
||||
assert!(store.running_on(C0FFEE).is_none(), "Starting is a command's");
|
||||
assert!(store.running_on(&"d".repeat(64)).is_none(), "nobody's container");
|
||||
assert!(store.running_on("").is_none());
|
||||
assert!(sight_and_mark(&store, C0FFEE, |_| false).is_none());
|
||||
assert_eq!(
|
||||
store.get(&starting_id).unwrap().status,
|
||||
crate::models::ProjectStatus::Starting
|
||||
@@ -725,14 +828,23 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mark_exited_matches_a_short_id_against_the_stored_full_one() {
|
||||
fn a_blank_or_truncated_stored_id_matches_no_container() {
|
||||
// A blank stored id is a prefix of everything.
|
||||
let dir = temp_dir("blank");
|
||||
let store = store_over(&dir, vec![running_on(""), running_on("abc")]);
|
||||
assert!(store.running_on(&"f".repeat(64)).is_none());
|
||||
assert!(store.running_on("abcdef0123456789").is_none());
|
||||
fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_short_id_is_matched_against_the_stored_full_one() {
|
||||
let dir = temp_dir("short");
|
||||
let full = "a".repeat(12) + &"b".repeat(52);
|
||||
let p = running_on(&full);
|
||||
let store = store_over(&dir, vec![p]);
|
||||
let store = store_over(&dir, vec![running_on(&full)]);
|
||||
|
||||
assert!(store.mark_exited("aaaa", |_| false).is_none(), "too short to be an id");
|
||||
assert!(store.mark_exited(&full[..12], |_| false).is_some());
|
||||
assert!(store.running_on("aaaa").is_none(), "too short to be an id");
|
||||
assert!(sight_and_mark(&store, &full[..12], |_| false).is_some());
|
||||
fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
}
|
||||
@@ -5,11 +5,12 @@ import type { Project } from "../lib/types";
|
||||
|
||||
let handler: ((e: { payload: unknown }) => void) | null = null;
|
||||
const unlisten = vi.fn();
|
||||
const listen = vi.fn(async (name: string, cb: (e: { payload: unknown }) => void) => {
|
||||
if (name === "project-container-exited") handler = cb;
|
||||
return unlisten;
|
||||
});
|
||||
vi.mock("@tauri-apps/api/event", () => ({
|
||||
listen: vi.fn(async (name: string, cb: (e: { payload: unknown }) => void) => {
|
||||
if (name === "project-status-changed") handler = cb;
|
||||
return unlisten;
|
||||
}),
|
||||
listen: (name: string, cb: (e: { payload: unknown }) => void) => listen(name, cb),
|
||||
}));
|
||||
|
||||
import { useProjectStatusEvents } from "./useProjectStatusEvents";
|
||||
@@ -21,28 +22,53 @@ describe("useProjectStatusEvents", () => {
|
||||
beforeEach(() => {
|
||||
handler = null;
|
||||
unlisten.mockClear();
|
||||
listen.mockClear();
|
||||
useAppState.setState({ projects: [project({}), project({ id: "p2", name: "other" })], toasts: [] });
|
||||
});
|
||||
|
||||
it("replaces the project the backend reports, so a dead container stops reading Running", async () => {
|
||||
it("applies a live exit and says why the sessions just died", async () => {
|
||||
renderHook(() => useProjectStatusEvents());
|
||||
await vi.waitFor(() => expect(handler).not.toBeNull());
|
||||
|
||||
handler!({ payload: project({ status: "stopped" }) });
|
||||
handler!({ payload: { project: project({ status: "stopped" }), reason: "exited" } });
|
||||
|
||||
const { projects, toasts } = useAppState.getState();
|
||||
expect(projects.find((p) => p.id === "p1")?.status).toBe("stopped");
|
||||
expect(projects.find((p) => p.id === "p2")?.status).toBe("running");
|
||||
// The terminals just went dead with no explanation; say why.
|
||||
expect(toasts).toHaveLength(1);
|
||||
expect(toasts[0].kind).toBe("error");
|
||||
expect(toasts[0].message).toContain("demo");
|
||||
expect(toasts[0].dedupeKey).toBe("container-exited-p1");
|
||||
});
|
||||
|
||||
it("applies a container found already stopped at startup silently", async () => {
|
||||
// After a host reboot every project's container is down before the app
|
||||
// opens; nothing was interrupted this run, so nothing to alarm anyone about.
|
||||
renderHook(() => useProjectStatusEvents());
|
||||
await vi.waitFor(() => expect(handler).not.toBeNull());
|
||||
|
||||
handler!({ payload: { project: project({ status: "stopped" }), reason: "found_stopped" } });
|
||||
|
||||
const { projects, toasts } = useAppState.getState();
|
||||
expect(projects.find((p) => p.id === "p1")?.status).toBe("stopped");
|
||||
expect(toasts).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("stops listening on unmount", async () => {
|
||||
const { unmount } = renderHook(() => useProjectStatusEvents());
|
||||
await vi.waitFor(() => expect(handler).not.toBeNull());
|
||||
unmount();
|
||||
expect(unlisten).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("survives listen() rejecting (no unhandled rejection)", async () => {
|
||||
const err = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
listen.mockImplementationOnce(async () => {
|
||||
throw new Error("no IPC");
|
||||
});
|
||||
const { unmount } = renderHook(() => useProjectStatusEvents());
|
||||
await vi.waitFor(() => expect(err).toHaveBeenCalled());
|
||||
unmount();
|
||||
err.mockRestore();
|
||||
});
|
||||
});
|
||||
@@ -3,34 +3,46 @@ import { listen, type UnlistenFn } from "@tauri-apps/api/event";
|
||||
import { useAppState } from "../store/appState";
|
||||
import type { Project } from "../lib/types";
|
||||
|
||||
/** `container_watch.rs`'s `ContainerExited`. */
|
||||
export interface ContainerExitedEvent {
|
||||
project: Project;
|
||||
/**
|
||||
* `exited`: it died while the app was running — sessions were just cut off.
|
||||
* `found_stopped`: found down on the app's first look this run (it stopped
|
||||
* while the app was closed, e.g. a host reboot); nothing was interrupted.
|
||||
*/
|
||||
reason: "exited" | "found_stopped";
|
||||
}
|
||||
|
||||
/**
|
||||
* App-wide: apply status changes the backend makes on its own. Today that is
|
||||
* `container_watch.rs` marking a project Stopped when its container exited
|
||||
* without the app asking — before it, the project kept reading Running, with
|
||||
* every terminal dead, until the app was restarted.
|
||||
* App-wide: apply `project-container-exited` — `container_watch.rs` marking a
|
||||
* project Stopped because its container exited without the app asking. Before
|
||||
* it, the project kept reading Running, with every terminal dead, until the
|
||||
* app was restarted.
|
||||
*/
|
||||
export function useProjectStatusEvents() {
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
let unlisten: UnlistenFn | null = null;
|
||||
void listen<Project>("project-status-changed", (event) => {
|
||||
const project = event.payload;
|
||||
listen<ContainerExitedEvent>("project-container-exited", (event) => {
|
||||
const { project, reason } = event.payload;
|
||||
const state = useAppState.getState();
|
||||
state.updateProjectInList(project);
|
||||
if (project.status === "stopped") {
|
||||
state.pushToast({
|
||||
kind: "error",
|
||||
message: `The container for “${project.name}” stopped`,
|
||||
detail:
|
||||
"It exited without Triple-C stopping it — open sessions were disconnected. " +
|
||||
"Start the project again to reconnect.",
|
||||
dedupeKey: `container-exited-${project.id}`,
|
||||
});
|
||||
}
|
||||
}).then((fn) => {
|
||||
if (cancelled) fn();
|
||||
else unlisten = fn;
|
||||
});
|
||||
if (reason !== "exited") return;
|
||||
state.pushToast({
|
||||
kind: "error",
|
||||
message: `The container for “${project.name}” stopped`,
|
||||
detail:
|
||||
"It exited without Triple-C stopping it — open sessions were disconnected. " +
|
||||
"Start the project again to reconnect.",
|
||||
dedupeKey: `container-exited-${project.id}`,
|
||||
});
|
||||
})
|
||||
.then((fn) => {
|
||||
if (cancelled) fn();
|
||||
else unlisten = fn;
|
||||
})
|
||||
.catch((e) => console.error("Failed to listen for container exits:", e));
|
||||
return () => {
|
||||
cancelled = true;
|
||||
unlisten?.();
|
||||
|
||||
+18
-1
@@ -760,6 +760,12 @@ timeout 120 su -s /bin/bash claude -c 'export PATH="/home/claude/.claude/bin:/ho
|
||||
# command line nothing else has, and it exists only once every step above
|
||||
# has finished — that is the readiness marker. Keep the name in step with
|
||||
# READY_PROBE.
|
||||
# * Respawns are throttled to about one a second. If the sleep cannot start
|
||||
# (fork failing under a pids limit or a fork bomb) or dies at once, an
|
||||
# unthrottled loop would spin a core at 100% exactly when the container is
|
||||
# already in trouble. The pause is `read -t` on a FIFO nobody writes —
|
||||
# a builtin, so it still works when fork does not — with `sleep 1` as the
|
||||
# fallback should the FIFO be unavailable.
|
||||
echo "Triple-C container ready."
|
||||
keepalive_pid=""
|
||||
stop_keepalive() {
|
||||
@@ -767,8 +773,19 @@ stop_keepalive() {
|
||||
exit 0
|
||||
}
|
||||
trap stop_keepalive TERM INT HUP
|
||||
throttle_fifo=/run/triple-c-keepalive.throttle
|
||||
rm -f "$throttle_fifo"
|
||||
mkfifo -m 600 "$throttle_fifo" 2>/dev/null && exec 3<>"$throttle_fifo"
|
||||
rm -f "$throttle_fifo"
|
||||
respawn_pause() {
|
||||
read -r -t 1 -u 3 _ 2>/dev/null
|
||||
# >128 is read's timeout, i.e. it paused; anything else means no FIFO.
|
||||
[ $? -gt 128 ] || sleep 1
|
||||
}
|
||||
while :; do
|
||||
(exec -a triple-c-keepalive sleep infinity) &
|
||||
started=$SECONDS
|
||||
(exec -a triple-c-keepalive sleep infinity 3>&-) &
|
||||
keepalive_pid=$!
|
||||
wait "$keepalive_pid"
|
||||
[ $((SECONDS - started)) -ge 2 ] || respawn_pause
|
||||
done
|
||||
Reference in new issue
Block a user