diff --git a/CLAUDE.md b/CLAUDE.md index 2cf8bea..df6be51 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -399,6 +399,12 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li user's index is untouched; needs the task definition's `stopTimeout` raised, 120 s) unless the stop marker says the app handled it. The idle watcher runs as root under a supervisor and signals the entrypoint's own PID (not PID 1, which is docker-init), so idle and ECS stops share one trap. + Idle = the agent's `last_activity_unix` older than N minutes (open-but-silent attaches/tunnels + don't count; requests with `X-Triple-C-Background: 1`, e.g. the auth bridge's poll, never do), + no `claude`, no scheduler job, no running clone, and no unsaved git work: with the WIP push on the + watcher pushes *before* signalling and stays up if that fails; with it off, dirty repos keep the + workspace up. An idle stop exits 75 (`aws::tasks::IDLE_EXIT_CODE`) so the death watch can tell it + from a crash (`classify_stop` → the `ContainerExited` toast). **The `TRIPLEC_` env prefix is reserved** on both hosts (`RESERVED_ENV_PREFIXES`; `workspace.rs` drops custom variables in it): the entrypoint's settings live there. - **Browser runtime libraries are baked in; browser *binaries* are not.** A layer runs diff --git a/app/src-tauri/src/aws/connection.rs b/app/src-tauri/src/aws/connection.rs index 1e14579..bd91b1c 100644 --- a/app/src-tauri/src/aws/connection.rs +++ b/app/src-tauri/src/aws/connection.rs @@ -5,6 +5,7 @@ use aws_config::identity::IdentityCache; use aws_config::profile::ProfileFileCredentialsProvider; use aws_config::provider_config::ProviderConfig; use aws_config::retry::RetryConfig; +use aws_config::timeout::TimeoutConfig; use aws_config::{BehaviorVersion, Region, SdkConfig}; use aws_credential_types::provider::error::CredentialsError; use aws_credential_types::provider::{future, ProvideCredentials, SharedCredentialsProvider}; @@ -81,9 +82,9 @@ fn keys_credentials(keys: &secure::AwsKeys) -> SharedCredentialsProvider { /// stray endpoint there would send every connection's signed requests (and /// the permission check) somewhere other than AWS. Connections talk to the /// real regional endpoints; a profile's own `endpoint_url` is ignored too. -/// The HTTP client, timeouts and stalled-stream protection are each service -/// client's defaults (the `default-https-client` stack every AWS crate here -/// is built with). +/// The HTTP client and stalled-stream protection are each service client's +/// defaults (the `default-https-client` stack every AWS crate here is built +/// with). Timeouts are ours: [`OPERATION_TIMEOUT`] / [`ATTEMPT_TIMEOUT`]. fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> SdkConfig { let mut builder = SdkConfig::builder() .behavior_version(BehaviorVersion::latest()) @@ -91,6 +92,7 @@ fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> S .credentials_provider(credentials) .identity_cache(IdentityCache::lazy().build()) .retry_config(RetryConfig::standard()) + .timeout_config(timeouts()) .time_source(SharedTimeSource::default()); if let Some(sleep) = default_async_sleep() { builder = builder.sleep_impl(sleep); @@ -98,6 +100,22 @@ fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> S builder.build() } +/// Every call the app makes through the SDK is a control-plane call with a +/// small answer (ECS, Secrets Manager, EFS, IAM, SSM, CloudWatch Logs, STS, +/// and S3 reads of state objects capped at 2 KiB); OpenTofu talks to AWS +/// itself, not through these clients. Without a bound a half-open +/// connection would hang a start, a stop or the death watch for minutes. +/// The operation bound covers every retry; the attempt bound is one try. +pub const OPERATION_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30); +pub const ATTEMPT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + +fn timeouts() -> TimeoutConfig { + TimeoutConfig::builder() + .operation_timeout(OPERATION_TIMEOUT) + .operation_attempt_timeout(ATTEMPT_TIMEOUT) + .build() +} + /// Puts the profile's name into the source chain of every credentials error, /// so a failure deep inside the SDK can be reported against the profile. #[derive(Debug)] @@ -297,4 +315,18 @@ sts = assert_isolated(&cfg); assert!(cfg.credentials_provider().is_some()); } + + #[test] + fn every_sdk_call_is_bounded() { + let keys = crate::storage::secure::AwsKeys { + access_key_id: "AKIDTEST".into(), + secret_access_key: "s".into(), + session_token: None, + }; + let cfg = build_sdk_config(Region::new("us-west-2"), keys_credentials(&keys)); + let t = cfg.timeout_config().expect("a timeout config"); + assert_eq!(t.operation_timeout(), Some(OPERATION_TIMEOUT)); + assert_eq!(t.operation_attempt_timeout(), Some(ATTEMPT_TIMEOUT)); + assert!(ATTEMPT_TIMEOUT < OPERATION_TIMEOUT); + } } diff --git a/app/src-tauri/src/aws/runtime/tests.rs b/app/src-tauri/src/aws/runtime/tests.rs index 0319993..7c9111c 100644 --- a/app/src-tauri/src/aws/runtime/tests.rs +++ b/app/src-tauri/src/aws/runtime/tests.rs @@ -247,6 +247,7 @@ fn live(id: &str) -> TaskInfo { ), runtime_id: None, exec_agent_running: false, + exit_code: None, owner: Some(USER.into()), project: Some(PROJECT.into()), } diff --git a/app/src-tauri/src/aws/tasks.rs b/app/src-tauri/src/aws/tasks.rs index 41b0e89..b4167cb 100644 --- a/app/src-tauri/src/aws/tasks.rs +++ b/app/src-tauri/src/aws/tasks.rs @@ -64,6 +64,8 @@ pub struct TaskInfo { pub runtime_id: Option, /// The `workspace` container's ExecuteCommandAgent is RUNNING. pub exec_agent_running: bool, + /// The `workspace` container's exit code, once it has one. + pub exit_code: Option, /// Its `triple-c:owner` / `triple-c:project` tags (DescribeTasks with /// TAGS; RunTask's answer carries the tags it was given). pub owner: Option, @@ -116,6 +118,7 @@ pub(crate) fn task_info(t: &Task) -> TaskInfo { && a.last_status() == Some("RUNNING") }) }), + exit_code: workspace.and_then(|c| c.exit_code()), owner: tag_value(t.tags(), TAG_OWNER), project: tag_value(t.tags(), TAG_PROJECT), } @@ -337,11 +340,6 @@ pub fn quota_or(e: AwsError) -> AwsError { } } -#[allow(dead_code)] // for the UI's start errors (Task 8) -pub fn is_quota_error(e: &AwsError) -> bool { - matches!(e, AwsError::QuotaExceeded { .. }) -} - /// RunTask's answer: its one task, or an error from its `failures` /// (`(reason, detail)`), a quota one when they say so. pub fn run_task_outcome( @@ -480,13 +478,19 @@ async fn stopped_error(api: &A, logs: &TaskLogs, t: &TaskInfo) -> Aw // ----------------------------------------------------------- stop / watch +/// The entrypoint's exit status after an idle stop (`container/entrypoint.sh`, +/// `stop_keepalive`). A stop from ECS (SIGTERM) exits 0; this one is distinct +/// so an idle stop (`EssentialContainerExited`) is not mistaken for a crash. +pub const IDLE_EXIT_CODE: i32 = 75; + /// Why a task stopped, as the app reports it (spec, "Lifecycle and status"). -#[allow(dead_code)] // the stop reasons the UI shows (Task 8) #[derive(Debug, Clone, PartialEq, Eq)] pub enum StopKind { - /// Exited on its own (the essential container ended, idle stop, or AWS - /// retired it). - Exited, + /// Exited on its own (the essential container ended, or AWS retired it), + /// with ECS's stopped reason when it gave one. + Exited(Option), + /// The workspace's own idle stop ([`IDLE_EXIT_CODE`]). + Idle, /// A Fargate Spot interruption (P8). Spot, /// Never got going: the project goes to Error with this reason. @@ -495,7 +499,6 @@ pub enum StopKind { UserInitiated, } -#[allow(dead_code)] // Task 8 pub fn classify_stop(info: &TaskInfo) -> StopKind { match info.stop_code.as_deref() { Some("SpotInterruption") => StopKind::Spot, @@ -506,25 +509,28 @@ pub fn classify_stop(info: &TaskInfo) -> StopKind { .unwrap_or_else(|| "The task failed to start".into()), ), Some("UserInitiated") => StopKind::UserInitiated, - _ => StopKind::Exited, + Some("EssentialContainerExited") if info.exit_code == Some(IDLE_EXIT_CODE) => StopKind::Idle, + _ => StopKind::Exited(info.stopped_reason.clone().filter(|r| !r.trim().is_empty())), } } /// Of `arns`, the ones that are STOPPED or that ECS no longer knows (it -/// forgets stopped tasks after about an hour). A task merely being stopped -/// is not yet reported. For the AWS death watch. +/// forgets stopped tasks after about an hour), each with why it stopped (a +/// forgotten task: [`StopKind::Exited`] without a reason). A task merely +/// being stopped is not yet reported. For the AWS death watch. pub async fn stopped_among( api: &A, cluster: &str, arns: &[String], -) -> Result, AwsError> { +) -> Result, AwsError> { let mut stopped = Vec::new(); for chunk in arns.chunks(DESCRIBE_BATCH) { let known = api.describe(cluster, chunk).await?; for arn in chunk { match known.iter().find(|t| &t.arn == arn) { Some(t) if t.last_status != "STOPPED" => {} - _ => stopped.push(arn.clone()), + Some(t) => stopped.push((arn.clone(), classify_stop(t))), + None => stopped.push((arn.clone(), StopKind::Exited(None))), } } } diff --git a/app/src-tauri/src/aws/tasks/tests.rs b/app/src-tauri/src/aws/tasks/tests.rs index 340399a..fdd6eb2 100644 --- a/app/src-tauri/src/aws/tasks/tests.rs +++ b/app/src-tauri/src/aws/tasks/tests.rs @@ -44,6 +44,10 @@ fn env_info() -> EnvironmentInfo { } } +fn is_quota_error(e: &AwsError) -> bool { + matches!(e, AwsError::QuotaExceeded { .. }) +} + fn info(id: &str, last: &str) -> TaskInfo { TaskInfo { arn: task_arn(id), @@ -60,6 +64,7 @@ fn info(id: &str, last: &str) -> TaskInfo { task_definition_arn: td_arn(3), runtime_id: None, exec_agent_running: false, + exit_code: None, owner: Some(USER.into()), project: Some(PROJECT_ID.into()), } @@ -620,8 +625,14 @@ async fn stopped_among_reports_stopped_and_vanished_tasks_only() { task_arn("gone"), ]; let mut got = stopped_among(&api, &cluster(), &arns).await.unwrap(); - got.sort(); - assert_eq!(got, [task_arn("down"), task_arn("gone")]); + got.sort_by(|a, b| a.0.cmp(&b.0)); + assert_eq!( + got, + [ + (task_arn("down"), StopKind::Exited(None)), + (task_arn("gone"), StopKind::Exited(None)), + ] + ); } // ------------------------------------------------------------ classify @@ -634,9 +645,17 @@ fn classify_stop_follows_the_spec_table() { ..info("t", "STOPPED") }; assert_eq!( - classify_stop(&with(Some("EssentialContainerExited"), None)), - StopKind::Exited + classify_stop(&with(Some("EssentialContainerExited"), Some("Essential container in task exited"))), + StopKind::Exited(Some("Essential container in task exited".into())) ); + // The entrypoint's idle stop exits with its own status. + let idle = TaskInfo { exit_code: Some(IDLE_EXIT_CODE), ..with(Some("EssentialContainerExited"), None) }; + assert_eq!(classify_stop(&idle), StopKind::Idle); + let crashed = TaskInfo { exit_code: Some(137), ..with(Some("EssentialContainerExited"), None) }; + assert_eq!(classify_stop(&crashed), StopKind::Exited(None)); + // Only with that stop code: a user stop of a task that exits 75 is a user stop. + let user = TaskInfo { exit_code: Some(IDLE_EXIT_CODE), ..with(Some("UserInitiated"), None) }; + assert_eq!(classify_stop(&user), StopKind::UserInitiated); assert_eq!( classify_stop(&with(Some("SpotInterruption"), None)), StopKind::Spot @@ -658,13 +677,13 @@ fn classify_stop_follows_the_spec_table() { ); assert_eq!( classify_stop(&with(Some("TerminationNotice"), None)), - StopKind::Exited + StopKind::Exited(None) ); assert_eq!( classify_stop(&with(Some("ServiceSchedulerInitiated"), None)), - StopKind::Exited + StopKind::Exited(None) ); - assert_eq!(classify_stop(&with(None, None)), StopKind::Exited); + assert_eq!(classify_stop(&with(None, Some(" "))), StopKind::Exited(None)); } // ------------------------------------------------------------ SDK mapping @@ -683,12 +702,14 @@ fn task_info_reads_the_workspace_container_and_its_exec_agent() { Container::builder() .name("sidecar") .runtime_id("wrong") + .exit_code(1) .build(), ) .containers( Container::builder() .name("workspace") .runtime_id("rt-1") + .exit_code(75) .managed_agents( ManagedAgent::builder() .name(ManagedAgentName::ExecuteCommandAgent) @@ -711,11 +732,13 @@ fn task_info_reads_the_workspace_container_and_its_exec_agent() { assert_eq!(i.stop_code.as_deref(), Some("SpotInterruption")); assert_eq!(i.runtime_id.as_deref(), Some("rt-1")); assert!(i.exec_agent_running); + assert_eq!(i.exit_code, Some(75), "the workspace container's, not the sidecar's"); assert_eq!(i.owner.as_deref(), Some(USER)); assert_eq!(i.project.as_deref(), Some(PROJECT_ID)); let bare = task_info(&Task::builder().task_arn("x").last_status("PENDING").build()); assert!(!bare.exec_agent_running); assert_eq!(bare.runtime_id, None); + assert_eq!(bare.exit_code, None); assert_eq!(bare.owner, None); } diff --git a/app/src-tauri/src/container_watch.rs b/app/src-tauri/src/container_watch.rs index 296d1d7..0ab8399 100644 --- a/app/src-tauri/src/container_watch.rs +++ b/app/src-tauri/src/container_watch.rs @@ -32,6 +32,7 @@ use serde::Serialize; use tauri::{AppHandle, Emitter}; use tokio::sync::watch; +use crate::aws::tasks::StopKind; use crate::host::watch::ExitSource; use crate::host::ContainerTarget; use crate::models::{Project, ProjectStatus}; @@ -68,6 +69,32 @@ pub enum ExitReason { pub struct ContainerExited<'a> { pub project: &'a Project, pub reason: ExitReason, + /// AWS: why ECS says the task stopped (P8). `None` on Docker, or when + /// the watch has no reason on record. + pub stop: Option, +} + +/// Why an AWS task stopped, for the UI's toast. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct StopNote { + /// `spot`, `idle`, `failed` (the project goes to Error), `user` (someone + /// stopped it: no toast) or `exited` (on its own). + pub kind: &'static str, + /// ECS's stopped reason, or a fixed text. + pub message: Option, +} + +impl StopNote { + pub fn of(kind: &StopKind) -> StopNote { + let (kind, message) = match kind { + StopKind::Spot => ("spot", Some("Spot interruption".to_string())), + StopKind::Idle => ("idle", None), + StopKind::FailedToStart(r) => ("failed", Some(r.clone())), + StopKind::UserInitiated => ("user", None), + StopKind::Exited(r) => ("exited", r.clone()), + }; + StopNote { kind, message } + } } fn project_busy(project_id: &str) -> bool { @@ -227,6 +254,19 @@ pub fn spawn( p.id, reason ); + // Why ECS stopped it, read before the tunnel (and with it the + // registry's record) goes. + let kind = p + .is_aws() + .then(|| p.container_id.as_deref().and_then(crate::host::tunnels::stop_kind)) + .flatten(); + let mut shown = p.clone(); + // A task that never got going is an error, not a stop (P8). + if let Some(StopKind::FailedToStart(_)) = kind { + if store_for_aws.update_status(&p.id, ProjectStatus::Error).is_ok() { + shown.status = ProjectStatus::Error; + } + } // An AWS task that stopped: its tunnel leads nowhere now, and its // task ARN is history (the data's access point stays). if p.is_aws() { @@ -245,7 +285,11 @@ pub fn spawn( }); } } - let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: p, reason }); + if let Some(store_p) = store_for_aws.get(&p.id) { + shown = store_p; + } + let stop = kind.as_ref().map(StopNote::of); + let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: &shown, reason, stop }); }) .await; }) @@ -266,15 +310,38 @@ mod tests { #[test] fn the_event_says_why_so_the_ui_can_tell_news_from_housekeeping() { let p = Project::new("demo".to_string(), Vec::new()); - let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited }).unwrap(); + let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited, stop: None }) + .unwrap(); assert_eq!(v["reason"], "exited"); assert_eq!(v["project"]["name"], "demo"); - let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::FoundStopped }) - .unwrap(); + assert!(v["stop"].is_null()); + let v = serde_json::to_value(ContainerExited { + project: &p, + reason: ExitReason::FoundStopped, + stop: Some(StopNote::of(&StopKind::Spot)), + }) + .unwrap(); assert_eq!(v["reason"], "found_stopped"); + assert_eq!(v["stop"], serde_json::json!({ "kind": "spot", "message": "Spot interruption" })); assert_eq!(CONTAINER_EXITED_EVENT, "project-container-exited"); } + #[test] + fn every_stop_kind_has_a_note_for_the_ui() { + let cases = [ + (StopKind::Spot, "spot", Some("Spot interruption")), + (StopKind::Idle, "idle", None), + (StopKind::FailedToStart("CannotPull".into()), "failed", Some("CannotPull")), + (StopKind::UserInitiated, "user", None), + (StopKind::Exited(Some("OOM".into())), "exited", Some("OOM")), + (StopKind::Exited(None), "exited", None), + ]; + for (k, kind, message) in cases { + let n = StopNote::of(&k); + assert_eq!((n.kind, n.message.as_deref()), (kind, message), "{k:?}"); + } + } + /// The shutdown path: a host call that never answers (a wedged daemon) /// must give way to cancellation at once, not hold teardown. #[tokio::test] diff --git a/app/src-tauri/src/docker/aws_mode_tests.rs b/app/src-tauri/src/docker/aws_mode_tests.rs index 1801cc2..3fe8964 100644 --- a/app/src-tauri/src/docker/aws_mode_tests.rs +++ b/app/src-tauri/src/docker/aws_mode_tests.rs @@ -698,20 +698,20 @@ fn is_idle(args: &[&str]) -> bool { #[test] fn is_idle_needs_every_condition() { - // now last execs tunnels claude_running jobs_running minutes - assert!(is_idle(&["10000", "9400", "0", "0", "0", "0", "10"]), "exactly N minutes"); - assert!(is_idle(&["10000", "1000", "0", "0", "0", "0", "10"])); - assert!(!is_idle(&["10000", "9401", "0", "0", "0", "0", "10"]), "one second short"); - assert!(!is_idle(&["10000", "1000", "1", "0", "0", "0", "10"]), "an exec is open"); - assert!(!is_idle(&["10000", "1000", "0", "2", "0", "0", "10"]), "a tunnel is open"); - assert!(!is_idle(&["10000", "1000", "0", "0", "1", "0", "10"]), "claude is running"); - assert!(!is_idle(&["10000", "1000", "0", "0", "0", "1", "10"]), "a scheduler job runs"); + // now, last activity, claude running, jobs running, minutes. + assert!(is_idle(&["10000", "9400", "0", "0", "10"]), "exactly N minutes"); + assert!(is_idle(&["10000", "1000", "0", "0", "10"])); + assert!(!is_idle(&["10000", "9401", "0", "0", "10"]), "one second short"); + assert!(!is_idle(&["10000", "1000", "1", "0", "10"]), "claude is running"); + assert!(!is_idle(&["10000", "1000", "0", "1", "10"]), "a scheduler job runs"); // Anything unreadable is "not idle". - assert!(!is_idle(&["10000", "", "0", "0", "0", "0", "10"])); - assert!(!is_idle(&["10000", "x", "0", "0", "0", "0", "10"])); - assert!(!is_idle(&["10000", "1000", "", "0", "0", "0", "10"])); - assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", "0"]), "0 minutes is off"); - assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", ""])); + assert!(!is_idle(&["10000", "", "0", "0", "10"])); + assert!(!is_idle(&["10000", "x", "0", "0", "10"])); + assert!(!is_idle(&["10000", "1000", "", "0", "10"])); + assert!(!is_idle(&["10000", "1000", "0", "0", "0"]), "0 minutes is off"); + assert!(!is_idle(&["10000", "1000", "0", "0", ""])); + // The old arity (with exec/tunnel counters) is refused, not misread. + assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", "10"])); } struct Watch { @@ -723,6 +723,18 @@ impl Watch { let tmp = tempfile::tempdir().unwrap(); write_script(tmp.path(), "triple-c-idle-watch", IDLE_WATCH); write_script(tmp.path(), "triple-c-scheduler", SCHEDULER); + // A stand-in WIP push: `--dry-run` prints the file `dry` (nothing: + // clean repos), a push records its arguments in `pushes` and exits + // with the number in `push_rc` (0 when absent). + let d = tmp.path().display(); + write_script( + tmp.path(), + "triple-c-wip-push", + &format!( + "#!/bin/bash\nif [ \"$1\" = --dry-run ]; then cat '{d}/dry' 2>/dev/null; exit 0; fi\n\ + echo \"$*\" >> '{d}/pushes'\nexit \"$(cat '{d}/push_rc' 2>/dev/null || echo 0)\"\n" + ), + ); fs::write(tmp.path().join("token"), "test-token-not-real").unwrap(); fs::create_dir_all(tmp.path().join("home")).unwrap(); Watch { tmp } @@ -743,10 +755,56 @@ impl Watch { .env("TRIPLEC_STOP_REASON_FILE", p.join("reason")) .env("TRIPLEC_SCHEDULER_BIN", p.join("triple-c-scheduler")) .env("TRIPLEC_SCHEDULER_USER", "") + .env("TRIPLEC_WIP_PUSH_BIN", p.join("triple-c-wip-push")) // The test host may well be running a `claude` of its own. .env("TRIPLEC_IDLE_PROC_NAME", "triple-c-no-such-process"); c } + + fn dirty(&self) { + fs::write( + self.tmp.path().join("dry"), + "{\"folder\":\"api\",\"dirty\":true,\"unpushed\":0,\"branch\":\"main\"}\n\ + {\"folder\":\"web\",\"dirty\":false,\"unpushed\":0,\"branch\":\"main\"}\n", + ) + .unwrap(); + } + + fn pushes(&self) -> String { + fs::read_to_string(self.tmp.path().join("pushes")).unwrap_or_default() + } +} + +/// Stands in for the entrypoint: records the TERM it gets in `got`. +fn fake_entrypoint(dir: &Path) -> std::process::Child { + Command::new("bash") + .arg("-c") + .arg("trap 'echo term > \"$1\"; exit 0' TERM; while :; do sleep 0.1; done") + .arg("t") + .arg(dir.join("got")) + .spawn() + .unwrap() +} + +/// Run the watcher loop for at most `secs`; `None` when it was still running +/// (it had not signalled) and was killed. Never hangs a test run. +fn run_watch_for(mut cmd: Command, secs: u64) -> Option { + let mut child = cmd + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .unwrap(); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(secs); + while std::time::Instant::now() < deadline { + if child.try_wait().unwrap().is_some() { + return Some(child.wait_with_output().unwrap()); + } + std::thread::sleep(std::time::Duration::from_millis(100)); + } + child.kill().ok(); + let out = child.wait_with_output().unwrap(); + eprintln!("watcher (killed): {}", text(&out)); + None } /// A one-shot fake agent: answers `/v1/activity` with `body` when the bearer @@ -782,10 +840,12 @@ s.serve_forever() (format!("http://127.0.0.1:{}", line.trim()), child) } +const OLD_ACTIVITY: &str = r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#; + #[test] fn idle_watch_check_counts_an_unfinished_clone_as_busy() { let w = Watch::new(); - let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#); + let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY); let clone = w.tmp.path().join("clone"); fs::create_dir_all(&clone).unwrap(); let busy = w.cmd(&url, 0).env("TRIPLEC_CLONE_STATUS_DIR", &clone).arg("--check").output().unwrap(); @@ -806,40 +866,115 @@ fn idle_watch_check_treats_an_unreachable_agent_as_busy() { #[test] fn idle_watch_check_reads_activity_with_the_token() { let w = Watch::new(); - let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#); + let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY); let idle = w.cmd(&url, 0).arg("--check").output().unwrap(); - let (busy_url, mut busy_agent) = - fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":1,"active_tunnels":0}"#); + let recent = format!( + r#"{{"last_activity_unix":{},"active_execs":0,"active_tunnels":0}}"#, + std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_secs() + ); + let (busy_url, mut busy_agent) = fake_agent(w.tmp.path(), &recent); let busy = w.cmd(&busy_url, 0).arg("--check").output().unwrap(); fs::write(w.tmp.path().join("token"), "wrong").unwrap(); let unauth = w.cmd(&url, 0).arg("--check").output().unwrap(); agent.kill().ok(); busy_agent.kill().ok(); assert_eq!(idle.status.code(), Some(0), "{}", text(&idle)); - assert_eq!(busy.status.code(), Some(1), "{}", text(&busy)); + assert_eq!(busy.status.code(), Some(1), "recent activity: {}", text(&busy)); assert_eq!(unauth.status.code(), Some(1), "a 401 is not idle: {}", text(&unauth)); } +#[test] +fn idle_watch_ignores_open_but_silent_execs_and_tunnels() { + // R11: an open terminal tab (an attach) or a tunnel no longer keeps the + // workspace up; only their traffic moves the clock. + let w = Watch::new(); + let (url, mut agent) = + fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":3,"active_tunnels":2}"#); + let out = w.cmd(&url, 0).arg("--check").output().unwrap(); + agent.kill().ok(); + assert_eq!(out.status.code(), Some(0), "{}", text(&out)); +} + +#[test] +fn idle_watch_check_with_unsaved_work_depends_on_the_wip_push() { + let w = Watch::new(); + w.dirty(); + let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY); + let off = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "0").arg("--check").output().unwrap(); + let on = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "1").arg("--check").output().unwrap(); + // A dry run that fails is a doubt: not idle. + write_script(w.tmp.path(), "triple-c-wip-push", "#!/bin/bash\nexit 1\n"); + let unknown = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "1").arg("--check").output().unwrap(); + agent.kill().ok(); + assert_eq!(off.status.code(), Some(1), "unsaved work, WIP push off: {}", text(&off)); + assert_eq!(on.status.code(), Some(0), "the WIP push would take it: {}", text(&on)); + assert_eq!(unknown.status.code(), Some(1), "{}", text(&unknown)); + assert_eq!(w.pushes(), "", "--check never pushes"); +} + #[test] fn idle_watch_signals_the_entrypoint_with_reason_idle() { let w = Watch::new(); - let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#); - // Stands in for the entrypoint: records the TERM it gets. - let target = Command::new("bash") - .arg("-c") - .arg("trap 'echo term > \"$1\"; exit 0' TERM; while :; do sleep 0.1; done") - .arg("t") - .arg(w.tmp.path().join("got")) - .spawn() - .unwrap(); - let mut target = target; - let out = w.cmd(&url, target.id()).output().unwrap(); + let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY); + let mut target = fake_entrypoint(w.tmp.path()); + let out = run_watch_for(w.cmd(&url, target.id()), 20).expect("it signals"); let status = target.wait().unwrap(); agent.kill().ok(); assert!(out.status.success(), "{}", text(&out)); assert!(status.success()); assert_eq!(fs::read_to_string(w.tmp.path().join("got")).unwrap().trim(), "term"); assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "idle"); + assert_eq!(w.pushes(), "", "nothing unsaved, nothing pushed"); +} + +#[test] +fn idle_watch_with_unsaved_work_and_the_wip_push_off_stays_up() { + let w = Watch::new(); + w.dirty(); + let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY); + let mut target = fake_entrypoint(w.tmp.path()); + let mut cmd = w.cmd(&url, target.id()); + cmd.env("TRIPLEC_WIP_PUSH", "0"); + let ran = run_watch_for(cmd, 4); + target.kill().ok(); + target.wait().ok(); + agent.kill().ok(); + assert!(ran.is_none(), "it must not stop: {:?}", ran.map(|o| text(&o))); + assert!(!w.tmp.path().join("got").exists(), "no TERM"); + assert!(!w.tmp.path().join("reason").exists(), "no marker"); + assert_eq!(w.pushes(), ""); +} + +#[test] +fn idle_watch_pushes_before_an_idle_stop_and_stays_up_while_the_push_fails() { + // The push fails: the workspace stays up and retries, backing off. + let w = Watch::new(); + w.dirty(); + fs::write(w.tmp.path().join("push_rc"), "1").unwrap(); + let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY); + let mut target = fake_entrypoint(w.tmp.path()); + let mut cmd = w.cmd(&url, target.id()); + cmd.env("TRIPLEC_WIP_PUSH", "1").env("TRIPLEC_WIP_BACKOFF_MAX", "2"); + let ran = run_watch_for(cmd, 6); + assert!(ran.is_none(), "a failed push must not stop the workspace"); + assert!(!w.tmp.path().join("got").exists(), "no TERM"); + assert!(!w.tmp.path().join("reason").exists(), "no marker"); + let tries = w.pushes().lines().count(); + assert!((1..=3).contains(&tries), "retried with a back-off, not every second: {tries}"); + assert!(w.pushes().lines().all(|l| l == "--reason idle"), "{}", w.pushes()); + + // Now the push works: it pushes once, then signals with the idle marker. + fs::write(w.tmp.path().join("push_rc"), "0").unwrap(); + fs::remove_file(w.tmp.path().join("pushes")).unwrap(); + let mut cmd = w.cmd(&url, target.id()); + cmd.env("TRIPLEC_WIP_PUSH", "1"); + let out = run_watch_for(cmd, 20).expect("it stops once the push succeeded"); + target.wait().unwrap(); + agent.kill().ok(); + assert!(out.status.success(), "{}", text(&out)); + assert_eq!(w.pushes(), "--reason idle\n"); + assert_eq!(fs::read_to_string(w.tmp.path().join("got")).unwrap().trim(), "term"); + assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "idle"); } #[test] @@ -847,13 +982,13 @@ fn idle_watch_keeps_a_stop_guard_marker_already_recorded() { for marker in ["pushed", "discard"] { let w = Watch::new(); fs::write(w.tmp.path().join("reason"), format!("{marker}\n")).unwrap(); - let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#); + let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY); let mut target = Command::new("bash") .arg("-c") .arg("trap 'exit 0' TERM; while :; do sleep 0.1; done") .spawn() .unwrap(); - let out = w.cmd(&url, target.id()).output().unwrap(); + let out = run_watch_for(w.cmd(&url, target.id()), 20).expect("it signals"); target.wait().unwrap(); agent.kill().ok(); assert!(out.status.success(), "{}", text(&out)); @@ -864,7 +999,7 @@ fn idle_watch_keeps_a_stop_guard_marker_already_recorded() { // ------------------------------------------------------- stop (SIGTERM) path #[test] -fn aws_stop_runs_the_wip_push_with_the_recorded_reason_once() { +fn aws_stop_pushes_on_sigterm_once_and_leaves_an_idle_stop_to_the_watcher() { let tmp = tempfile::tempdir().unwrap(); let log = tmp.path().join("calls"); let fake = write_script( @@ -882,13 +1017,18 @@ fn aws_stop_runs_the_wip_push_with_the_recorded_reason_once() { assert!(out.status.success(), "{}", text(&out)); assert_eq!(fs::read_to_string(&log).unwrap(), "--reason sigterm\n"); + // An idle stop: the watcher already pushed, so the trap does not, and it + // exits with the idle status the app reads from ECS. fs::remove_file(&log).unwrap(); fs::write(&reason, "idle\n").unwrap(); - let out = bash(&script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "1")], &[]); + let idle_script = format!("{blocks}\naws_before_stop\necho \"status=${{triplec_exit_status:-0}}\"\n"); + let out = bash(&idle_script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "1")], &[]); assert!(out.status.success(), "{}", text(&out)); - assert_eq!(fs::read_to_string(&log).unwrap(), "--reason idle\n"); + assert!(!log.exists(), "no second push after the watcher's"); + assert!(text(&out).contains("entrypoint: stopping (idle)"), "{}", text(&out)); + assert!(text(&out).contains("status=75"), "{}", text(&out)); + fs::remove_file(&reason).unwrap(); - fs::remove_file(&log).unwrap(); let out = bash(&script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "0")], &[]); assert!(out.status.success(), "{}", text(&out)); assert!(!log.exists(), "WIP push off"); @@ -922,6 +1062,32 @@ fn aws_stop_skips_its_push_when_the_app_already_pushed_or_the_user_declined() { assert_eq!(fs::read_to_string(&log).unwrap(), "--reason sigterm\n"); } +#[test] +fn the_idle_exit_status_is_the_one_the_app_reads() { + assert!( + block(ENTRYPOINT, "aws_stop") + .contains(&format!("triplec_idle_exit={}\n", crate::aws::tasks::IDLE_EXIT_CODE)), + "entrypoint.sh and aws::tasks::IDLE_EXIT_CODE disagree" + ); + assert!(ENTRYPOINT.contains("exit \"${triplec_exit_status:-0}\""), "stop_keepalive exits with it"); + // The watcher gets the WIP setting through its scrubbed environment. + assert!(block(ENTRYPOINT, "aws_stop").contains("TRIPLEC_WIP_PUSH=\"${TRIPLEC_WIP_PUSH:-0}\"")); +} + +#[test] +fn start_clone_repos_marks_done_however_the_clone_ends() { + let tmp = tempfile::tempdir().unwrap(); + let status = tmp.path().join("status"); + // clone_repos replaced by one that fails half-way. + let script = format!( + "{}\nclone_repos() {{ exit 3; }}\nstart_clone_repos '[x]' \"$1\"\nwait\n", + aws_blocks(&["aws_env", "aws_repos"]) + ); + let out = bash(&script, &[&status], &[], &[]); + assert!(out.status.success(), "{}", text(&out)); + assert!(status.join("done").exists(), "a failed clone still ends: {}", text(&out)); +} + #[test] fn docker_mode_never_reaches_the_aws_blocks() { // Every new step sits behind TRIPLEC_HOST=aws; the scripts are only diff --git a/app/src-tauri/src/host/aws_host.rs b/app/src-tauri/src/host/aws_host.rs index a7ba3c0..5cb827f 100644 --- a/app/src-tauri/src/host/aws_host.rs +++ b/app/src-tauri/src/host/aws_host.rs @@ -35,7 +35,7 @@ use crate::aws::runtime::{ }; use crate::aws::tasks::{ find_or_run, find_project_task, stopped_among, task_log_stream, wait_ready, SdkTaskApi, - TaskApi, TaskInfo, TaskLogs, WaitTiming, LOG_TAIL_LINES, STOP_REASON, + StopKind, TaskApi, TaskInfo, TaskLogs, WaitTiming, LOG_TAIL_LINES, STOP_REASON, }; use crate::aws::tofu_modules::{ environment_cluster_name, log_stream_prefix, owner_key, task_family, @@ -75,6 +75,11 @@ const HEALTH_TIMEOUT: Duration = Duration::from_secs(30); const HEALTH_INTERVAL: Duration = Duration::from_secs(1); /// How long a terminal waits for a background re-attach. pub const REATTACH_WAIT: Duration = Duration::from_secs(90); +/// The most one AWS death-watch poll of a group may take. +const WATCH_CALL_TIMEOUT: Duration = Duration::from_secs(60); +/// The most a StopTask (credentials included) may take before the stop is +/// reported as failed; the SDK's own bound is per call. +const STOP_TIMEOUT: Duration = Duration::from_secs(45); fn s(e: E) -> String { e.to_string() @@ -730,7 +735,18 @@ pub async fn task_log(project: &Project, settings: &AppSettings, limit: i32) -> pub async fn stop_task_arn(api: &T, task_arn: &str) -> Result<(), String> { let cluster = cluster_of_task(task_arn) .ok_or_else(|| format!("{task_arn} is not a task ARN with a cluster"))?; - api.stop_task(cluster, task_arn, STOP_REASON).await.map_err(s) + bounded_stop(api.stop_task(cluster, task_arn, STOP_REASON)).await +} + +/// A StopTask, bounded by [`STOP_TIMEOUT`]: a stop that cannot reach AWS +/// fails (and the project keeps its task ARN) instead of hanging the UI. +async fn bounded_stop( + call: impl std::future::Future>, +) -> Result<(), String> { + tokio::time::timeout(STOP_TIMEOUT, call) + .await + .map_err(|_| format!("StopTask did not answer within {STOP_TIMEOUT:?}; try again"))? + .map_err(s) } /// `ProjectHost::stop` for an agent target: the task behind a registered @@ -738,10 +754,7 @@ pub async fn stop_task_arn(api: &T, task_arn: &str) -> Result<(), St pub async fn stop_endpoint(ep: &AgentEndpoint) -> Option> { let (_connection, _region, cluster, arn) = tunnels::task_of(ep)?; let (project_id, sdk) = tunnels::owner_of(ep)?; - let res = SdkTaskApi::new(&sdk) - .stop_task(&cluster, &arn, STOP_REASON) - .await - .map_err(s); + let res = bounded_stop(SdkTaskApi::new(&sdk).stop_task(&cluster, &arn, STOP_REASON)).await; if res.is_ok() { tunnels::close(&project_id).await; } @@ -1089,22 +1102,29 @@ impl AwsTaskLookup for StoreTaskLookup { Box::pin(async move { let settings = self.settings.get(); let conn = crate::commands::host_commands::aws_conn(&group.connection_id, &settings)?; - let sdk = sdk_config(&conn, &group.region).await.map_err(s)?; - let stopped = stopped_among(&SdkTaskApi::new(&sdk), &group.cluster_arn, arns) - .await - .map_err(s)?; + // The SDK bounds each call; this bounds the whole poll of a group + // (credentials included, which an SSO refresh can stall), so one + // stuck group never stalls the watch of every other. + let stopped = tokio::time::timeout(WATCH_CALL_TIMEOUT, async { + let sdk = sdk_config(&conn, &group.region).await.map_err(s)?; + stopped_among(&SdkTaskApi::new(&sdk), &group.cluster_arn, arns) + .await + .map_err(s) + }) + .await + .map_err(|_| format!("ECS did not answer within {WATCH_CALL_TIMEOUT:?}"))??; record_stopped(&self.projects.list(), &stopped); - Ok(stopped) + Ok(stopped.into_iter().map(|(arn, _)| arn).collect()) }) } } -/// Tell the registry which targets' tasks ECS reports stopped, so -/// `has_exited` confirms them (R3). -pub fn record_stopped(projects: &[Project], stopped: &[String]) { +/// Tell the registry which targets' tasks ECS reports stopped, and why, so +/// `has_exited` confirms them (R3) and the death report can say why (P8). +pub fn record_stopped(projects: &[Project], stopped: &[(String, StopKind)]) { for w in projects.iter().filter_map(watched_task) { - if stopped.contains(&w.task_arn) { - tunnels::mark_stopped(w.target.id()); + if let Some((_, why)) = stopped.iter().find(|(arn, _)| *arn == w.task_arn) { + tunnels::mark_stopped(w.target.id(), why.clone()); } } } diff --git a/app/src-tauri/src/host/aws_host/tests.rs b/app/src-tauri/src/host/aws_host/tests.rs index a4560cb..ecfeb00 100644 --- a/app/src-tauri/src/host/aws_host/tests.rs +++ b/app/src-tauri/src/host/aws_host/tests.rs @@ -57,6 +57,7 @@ fn task(p: &Project, id: &str, last: &str, ready: bool) -> TaskInfo { task_definition_arn: format!("arn:aws:ecs:{REGION}:{ACCOUNT}:task-definition/{}:1", family(p)), runtime_id: ready.then(|| "rt-1".into()), exec_agent_running: ready, + exit_code: None, owner: Some(USER.into()), project: Some(p.id.clone()), } @@ -572,8 +573,9 @@ async fn the_watch_reports_a_running_aws_project_as_its_stored_target_and_confir assert_eq!(w.group.cluster_arn, cluster_arn()); assert_eq!(w.group.connection_id, "c1"); assert!(!w.target.has_exited().await, "nothing recorded yet"); - record_stopped(std::slice::from_ref(&p), &[task_arn("w1")]); + record_stopped(std::slice::from_ref(&p), &[(task_arn("w1"), StopKind::Idle)]); assert!(w.target.has_exited().await); + assert_eq!(crate::host::tunnels::stop_kind(w.target.id()), Some(StopKind::Idle)); let mut stopped = p.clone(); stopped.status = ProjectStatus::Stopped; diff --git a/app/src-tauri/src/host/tunnels.rs b/app/src-tauri/src/host/tunnels.rs index 49c760e..0232bde 100644 --- a/app/src-tauri/src/host/tunnels.rs +++ b/app/src-tauri/src/host/tunnels.rs @@ -20,6 +20,7 @@ use futures_util::future::BoxFuture; use tokio::sync::watch; use super::agent::AgentEndpoint; +use crate::aws::tasks::StopKind; use super::ssm::{EcsTask, PortForward}; /// Something that owns a tunnel's resources: the real [`PortForward`], or a @@ -58,8 +59,9 @@ pub type TaskOf = (String, String, String, String); #[derive(Default)] struct Registry { live: HashMap, - /// Target ids (endpoint base URLs) whose task the AWS watch saw STOPPED. - stopped: HashSet, + /// Target ids (endpoint base URLs) whose task the AWS watch saw STOPPED, + /// and why. + stopped: HashMap, /// Projects whose tunnel is being re-opened in the background. pending: HashSet, /// Bumped by every [`close`] (a stop or remove), every [`register`] (a @@ -245,14 +247,19 @@ pub async fn close_all() { futures_util::future::join_all(all.into_iter().map(|t| t.forward.close())).await; } -/// The AWS watch saw the task behind target `target_id` STOPPED. -pub fn mark_stopped(target_id: &str) { - lock().stopped.insert(target_id.to_string()); +/// The AWS watch saw the task behind target `target_id` STOPPED, for `why`. +pub fn mark_stopped(target_id: &str, why: StopKind) { + lock().stopped.insert(target_id.to_string(), why); } /// Whether the AWS watch has confirmed the task behind `target_id` stopped. pub fn is_stopped(target_id: &str) -> bool { - lock().stopped.contains(target_id) + lock().stopped.contains_key(target_id) +} + +/// Why the task behind `target_id` stopped, once the watch has seen it. +pub fn stop_kind(target_id: &str) -> Option { + lock().stopped.get(target_id).cloned() } pub fn is_pending(project_id: &str) -> bool { @@ -341,7 +348,8 @@ pub(crate) mod tests { let p = pid("rereg"); register(&p, fake_tunnel(61002, closes.clone())); let id = AgentEndpoint::loopback(61003, "tok").base_url.to_string(); - mark_stopped(&id); + mark_stopped(&id, StopKind::Spot); + assert_eq!(stop_kind(&id), Some(StopKind::Spot)); let old = register(&p, fake_tunnel(61003, closes.clone())).unwrap(); assert_eq!(old.endpoint.base_url.port(), Some(61002)); assert!(!is_stopped(&id), "a new task on a reused port is not stopped"); diff --git a/app/src/hooks/useProjectStatusEvents.test.ts b/app/src/hooks/useProjectStatusEvents.test.ts index dee0635..128242b 100644 --- a/app/src/hooks/useProjectStatusEvents.test.ts +++ b/app/src/hooks/useProjectStatusEvents.test.ts @@ -54,6 +54,35 @@ describe("useProjectStatusEvents", () => { expect(toasts).toHaveLength(0); }); + it("says why an AWS task stopped, and keeps a user's stop quiet", async () => { + renderHook(() => useProjectStatusEvents()); + await vi.waitFor(() => expect(handler).not.toBeNull()); + const fire = (stop: unknown, reason = "exited") => { + useAppState.setState({ toasts: [] }); + handler!({ payload: { project: project({ status: "stopped" }), reason, stop } }); + return useAppState.getState().toasts; + }; + + let t = fire({ kind: "spot", message: "Spot interruption" }); + expect(t[0].kind).toBe("info"); + expect(t[0].message).toContain("Spot interruption"); + + t = fire({ kind: "idle", message: null }); + expect(t[0].kind).toBe("info"); + expect(t[0].message).toContain("idle"); + + expect(fire({ kind: "user", message: null })).toHaveLength(0); + + // A task that never started is news even when found at startup. + t = fire({ kind: "failed", message: "CannotPullContainerError: denied" }, "found_stopped"); + expect(t[0].kind).toBe("error"); + expect(t[0].detail).toContain("CannotPullContainerError"); + + t = fire({ kind: "exited", message: "OutOfMemoryError" }); + expect(t[0].kind).toBe("error"); + expect(t[0].detail).toContain("OutOfMemoryError"); + }); + it("stops listening on unmount", async () => { const { unmount } = renderHook(() => useProjectStatusEvents()); await vi.waitFor(() => expect(handler).not.toBeNull()); diff --git a/app/src/hooks/useProjectStatusEvents.ts b/app/src/hooks/useProjectStatusEvents.ts index 87a66f8..02c94f6 100644 --- a/app/src/hooks/useProjectStatusEvents.ts +++ b/app/src/hooks/useProjectStatusEvents.ts @@ -12,6 +12,56 @@ export interface ContainerExitedEvent { * while the app was closed, e.g. a host reboot); nothing was interrupted. */ reason: "exited" | "found_stopped"; + /** AWS: why ECS says the task stopped (`StopNote`); absent on Docker. */ + stop?: StopNote | null; +} + +/** `container_watch.rs`'s `StopNote`. */ +export interface StopNote { + /** `user`: someone stopped it (console, CLI, another machine) — no toast. */ + kind: "spot" | "idle" | "failed" | "user" | "exited"; + message: string | null; +} + +/** The toast for an exit, or null for none. */ +export function exitToast( + name: string, + reason: ContainerExitedEvent["reason"], + stop: StopNote | null | undefined, +): { kind: "error" | "info"; message: string; detail?: string } | null { + if (stop?.kind === "user") return null; + if (stop?.kind === "failed") { + return { + kind: "error", + message: `The AWS workspace for “${name}” failed to start`, + detail: stop.message ?? undefined, + }; + } + if (reason !== "exited") return null; + switch (stop?.kind) { + case "spot": + return { + kind: "info", + message: `“${name}” stopped: Spot interruption`, + detail: + "AWS reclaimed the Spot capacity it ran on. Start the project again to get a new workspace.", + }; + case "idle": + return { + kind: "info", + message: `“${name}” stopped after being idle`, + detail: "Its AWS workspace stopped itself to save cost. Start the project again to continue.", + }; + default: + return { + kind: "error", + message: `The container for “${name}” stopped`, + detail: + "It exited without Triple-C stopping it — open sessions were disconnected. " + + "Start the project again to reconnect." + + (stop?.message ? ` (${stop.message})` : ""), + }; + } } /** @@ -25,18 +75,12 @@ export function useProjectStatusEvents() { let cancelled = false; let unlisten: UnlistenFn | null = null; listen("project-container-exited", (event) => { - const { project, reason } = event.payload; + const { project, reason, stop } = event.payload; const state = useAppState.getState(); state.updateProjectInList(project); - if (reason !== "exited") return; - state.pushToast({ - kind: "error", - message: `The container for “${project.name}” stopped`, - detail: - "It exited without Triple-C stopping it — open sessions were disconnected. " + - "Start the project again to reconnect.", - dedupeKey: `container-exited-${project.id}`, - }); + const toast = exitToast(project.name, reason, stop); + if (!toast) return; + state.pushToast({ ...toast, dedupeKey: `container-exited-${project.id}` }); }) .then((fn) => { if (cancelled) fn(); diff --git a/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md b/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md index 60edfcc..9bfcb0b 100644 --- a/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md +++ b/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md @@ -349,7 +349,9 @@ updates use the existing "new image available" flow: the next start uses the new `git status --porcelain` and any unpushed commits in each repo. If anything would be lost, it offers: **Push to a `triple-c/wip/` branch** (the preselected default), **Stop anyway**, or **Cancel**. The same check runs before idle auto-stop. If anything is dirty, idle auto-stop - pushes a WIP branch instead of throwing work away. + pushes a WIP branch instead of throwing work away, and stops only once that push succeeded (a + failed push keeps the workspace up and is retried, backing off). With the WIP push off, unsaved + work keeps the workspace up. - **Fargate task retirement and Spot interruption** send `SIGTERM` with a short deadline (2 minutes for Spot). The entrypoint's `SIGTERM` trap runs the same WIP push, best-effort, when the project has it enabled. @@ -497,9 +499,12 @@ way. - user-initiated → Stopped. EventBridge → SQS for push notifications is a later improvement. -- **Idle auto-stop** runs **inside** the task: the agent tracks activity (terminal I/O, agent - calls, running `claude` processes and scheduler jobs) and stops the task by exiting the - entrypoint. It doesn't depend on the app being open. Because a running scheduler job counts as +- **Idle auto-stop** runs **inside** the task: the agent tracks activity (terminal I/O and agent + calls; an open but silent terminal or tunnel is not activity, and the app's background polls + are marked `X-Triple-C-Background` and never count — ruling R11), the watcher adds running + `claude` processes, scheduler jobs and unsaved git work, and stops the task by exiting the + entrypoint with status 75, which the app reads back from `DescribeTasks` to say "stopped after + being idle". It doesn't depend on the app being open. Because a running scheduler job counts as activity, scheduled tasks keep a workspace alive. The admin's maximum lifetime is a hard cap either way. - **Health:** the task definition carries an ECS container health check against the agent, run