fix(aws): say why a task stopped, and bound every AWS call

- classify_stop is wired into the death watch: stopped_among returns
  each task's StopKind, the tunnel registry keeps it, and ContainerExited
  carries a StopNote. Spot -> 'Spot interruption'; TaskFailedToStart ->
  Error with the reason; the entrypoint's idle exit (75, read from the
  workspace container's exit code) -> 'stopped after being idle'; a user
  stop is applied silently (P8). is_quota_error, dead, moved to tests.
- The SDK config has a TimeoutConfig (30 s operation / 10 s attempt); the
  watch's per-group poll and StopTask also have tokio bounds.
- Watcher tests cover silent sessions, unsaved work with the WIP push on
  and off, a failing push and the idle exit status; docs updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
shadowdaoandClaude Opus 5.5 committed 2026-11-15 12:31:28 -08:00
1 parent 151b7690df
commit 5e8e67a0a4
13 files changed
+511 -102

No files matched your search

+6
View File
@@ -399,6 +399,12 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li
user's index is untouched; needs the task definition's `stopTimeout` raised, 120 s) unless the stop
marker says the app handled it. The idle watcher runs as root under a supervisor and signals the
entrypoint's own PID (not PID 1, which is docker-init), so idle and ECS stops share one trap.
Idle = the agent's `last_activity_unix` older than N minutes (open-but-silent attaches/tunnels
don't count; requests with `X-Triple-C-Background: 1`, e.g. the auth bridge's poll, never do),
no `claude`, no scheduler job, no running clone, and no unsaved git work: with the WIP push on the
watcher pushes *before* signalling and stays up if that fails; with it off, dirty repos keep the
workspace up. An idle stop exits 75 (`aws::tasks::IDLE_EXIT_CODE`) so the death watch can tell it
from a crash (`classify_stop` → the `ContainerExited` toast).
**The `TRIPLEC_` env prefix is reserved** on both hosts (`RESERVED_ENV_PREFIXES`; `workspace.rs`
drops custom variables in it): the entrypoint's settings live there.
- **Browser runtime libraries are baked in; browser *binaries* are not.** A layer runs
+35 -3
View File
@@ -5,6 +5,7 @@ use aws_config::identity::IdentityCache;
use aws_config::profile::ProfileFileCredentialsProvider;
use aws_config::provider_config::ProviderConfig;
use aws_config::retry::RetryConfig;
use aws_config::timeout::TimeoutConfig;
use aws_config::{BehaviorVersion, Region, SdkConfig};
use aws_credential_types::provider::error::CredentialsError;
use aws_credential_types::provider::{future, ProvideCredentials, SharedCredentialsProvider};
@@ -81,9 +82,9 @@ fn keys_credentials(keys: &secure::AwsKeys) -> SharedCredentialsProvider {
/// stray endpoint there would send every connection's signed requests (and
/// the permission check) somewhere other than AWS. Connections talk to the
/// real regional endpoints; a profile's own `endpoint_url` is ignored too.
/// The HTTP client, timeouts and stalled-stream protection are each service
/// client's defaults (the `default-https-client` stack every AWS crate here
/// is built with).
/// The HTTP client and stalled-stream protection are each service client's
/// defaults (the `default-https-client` stack every AWS crate here is built
/// with). Timeouts are ours: [`OPERATION_TIMEOUT`] / [`ATTEMPT_TIMEOUT`].
fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> SdkConfig {
let mut builder = SdkConfig::builder()
.behavior_version(BehaviorVersion::latest())
@@ -91,6 +92,7 @@ fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> S
.credentials_provider(credentials)
.identity_cache(IdentityCache::lazy().build())
.retry_config(RetryConfig::standard())
.timeout_config(timeouts())
.time_source(SharedTimeSource::default());
if let Some(sleep) = default_async_sleep() {
builder = builder.sleep_impl(sleep);
@@ -98,6 +100,22 @@ fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> S
builder.build()
}
/// Every call the app makes through the SDK is a control-plane call with a
/// small answer (ECS, Secrets Manager, EFS, IAM, SSM, CloudWatch Logs, STS,
/// and S3 reads of state objects capped at 2 KiB); OpenTofu talks to AWS
/// itself, not through these clients. Without a bound a half-open
/// connection would hang a start, a stop or the death watch for minutes.
/// The operation bound covers every retry; the attempt bound is one try.
pub const OPERATION_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
pub const ATTEMPT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
fn timeouts() -> TimeoutConfig {
TimeoutConfig::builder()
.operation_timeout(OPERATION_TIMEOUT)
.operation_attempt_timeout(ATTEMPT_TIMEOUT)
.build()
}
/// Puts the profile's name into the source chain of every credentials error,
/// so a failure deep inside the SDK can be reported against the profile.
#[derive(Debug)]
@@ -297,4 +315,18 @@ sts =
assert_isolated(&cfg);
assert!(cfg.credentials_provider().is_some());
}
#[test]
fn every_sdk_call_is_bounded() {
let keys = crate::storage::secure::AwsKeys {
access_key_id: "AKIDTEST".into(),
secret_access_key: "s".into(),
session_token: None,
};
let cfg = build_sdk_config(Region::new("us-west-2"), keys_credentials(&keys));
let t = cfg.timeout_config().expect("a timeout config");
assert_eq!(t.operation_timeout(), Some(OPERATION_TIMEOUT));
assert_eq!(t.operation_attempt_timeout(), Some(ATTEMPT_TIMEOUT));
assert!(ATTEMPT_TIMEOUT < OPERATION_TIMEOUT);
}
}
+1
View File
@@ -247,6 +247,7 @@ fn live(id: &str) -> TaskInfo {
),
runtime_id: None,
exec_agent_running: false,
exit_code: None,
owner: Some(USER.into()),
project: Some(PROJECT.into()),
}
+21 -15
View File
@@ -64,6 +64,8 @@ pub struct TaskInfo {
pub runtime_id: Option<String>,
/// The `workspace` container's ExecuteCommandAgent is RUNNING.
pub exec_agent_running: bool,
/// The `workspace` container's exit code, once it has one.
pub exit_code: Option<i32>,
/// Its `triple-c:owner` / `triple-c:project` tags (DescribeTasks with
/// TAGS; RunTask's answer carries the tags it was given).
pub owner: Option<String>,
@@ -116,6 +118,7 @@ pub(crate) fn task_info(t: &Task) -> TaskInfo {
&& a.last_status() == Some("RUNNING")
})
}),
exit_code: workspace.and_then(|c| c.exit_code()),
owner: tag_value(t.tags(), TAG_OWNER),
project: tag_value(t.tags(), TAG_PROJECT),
}
@@ -337,11 +340,6 @@ pub fn quota_or(e: AwsError) -> AwsError {
}
}
#[allow(dead_code)] // for the UI's start errors (Task 8)
pub fn is_quota_error(e: &AwsError) -> bool {
matches!(e, AwsError::QuotaExceeded { .. })
}
/// RunTask's answer: its one task, or an error from its `failures`
/// (`(reason, detail)`), a quota one when they say so.
pub fn run_task_outcome(
@@ -480,13 +478,19 @@ async fn stopped_error<A: TaskApi>(api: &A, logs: &TaskLogs, t: &TaskInfo) -> Aw
// ----------------------------------------------------------- stop / watch
/// The entrypoint's exit status after an idle stop (`container/entrypoint.sh`,
/// `stop_keepalive`). A stop from ECS (SIGTERM) exits 0; this one is distinct
/// so an idle stop (`EssentialContainerExited`) is not mistaken for a crash.
pub const IDLE_EXIT_CODE: i32 = 75;
/// Why a task stopped, as the app reports it (spec, "Lifecycle and status").
#[allow(dead_code)] // the stop reasons the UI shows (Task 8)
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum StopKind {
/// Exited on its own (the essential container ended, idle stop, or AWS
/// retired it).
Exited,
/// Exited on its own (the essential container ended, or AWS retired it),
/// with ECS's stopped reason when it gave one.
Exited(Option<String>),
/// The workspace's own idle stop ([`IDLE_EXIT_CODE`]).
Idle,
/// A Fargate Spot interruption (P8).
Spot,
/// Never got going: the project goes to Error with this reason.
@@ -495,7 +499,6 @@ pub enum StopKind {
UserInitiated,
}
#[allow(dead_code)] // Task 8
pub fn classify_stop(info: &TaskInfo) -> StopKind {
match info.stop_code.as_deref() {
Some("SpotInterruption") => StopKind::Spot,
@@ -506,25 +509,28 @@ pub fn classify_stop(info: &TaskInfo) -> StopKind {
.unwrap_or_else(|| "The task failed to start".into()),
),
Some("UserInitiated") => StopKind::UserInitiated,
_ => StopKind::Exited,
Some("EssentialContainerExited") if info.exit_code == Some(IDLE_EXIT_CODE) => StopKind::Idle,
_ => StopKind::Exited(info.stopped_reason.clone().filter(|r| !r.trim().is_empty())),
}
}
/// Of `arns`, the ones that are STOPPED or that ECS no longer knows (it
/// forgets stopped tasks after about an hour). A task merely being stopped
/// is not yet reported. For the AWS death watch.
/// forgets stopped tasks after about an hour), each with why it stopped (a
/// forgotten task: [`StopKind::Exited`] without a reason). A task merely
/// being stopped is not yet reported. For the AWS death watch.
pub async fn stopped_among<A: TaskApi>(
api: &A,
cluster: &str,
arns: &[String],
) -> Result<Vec<String>, AwsError> {
) -> Result<Vec<(String, StopKind)>, AwsError> {
let mut stopped = Vec::new();
for chunk in arns.chunks(DESCRIBE_BATCH) {
let known = api.describe(cluster, chunk).await?;
for arn in chunk {
match known.iter().find(|t| &t.arn == arn) {
Some(t) if t.last_status != "STOPPED" => {}
_ => stopped.push(arn.clone()),
Some(t) => stopped.push((arn.clone(), classify_stop(t))),
None => stopped.push((arn.clone(), StopKind::Exited(None))),
}
}
}
+30 -7
View File
@@ -44,6 +44,10 @@ fn env_info() -> EnvironmentInfo {
}
}
fn is_quota_error(e: &AwsError) -> bool {
matches!(e, AwsError::QuotaExceeded { .. })
}
fn info(id: &str, last: &str) -> TaskInfo {
TaskInfo {
arn: task_arn(id),
@@ -60,6 +64,7 @@ fn info(id: &str, last: &str) -> TaskInfo {
task_definition_arn: td_arn(3),
runtime_id: None,
exec_agent_running: false,
exit_code: None,
owner: Some(USER.into()),
project: Some(PROJECT_ID.into()),
}
@@ -620,8 +625,14 @@ async fn stopped_among_reports_stopped_and_vanished_tasks_only() {
task_arn("gone"),
];
let mut got = stopped_among(&api, &cluster(), &arns).await.unwrap();
got.sort();
assert_eq!(got, [task_arn("down"), task_arn("gone")]);
got.sort_by(|a, b| a.0.cmp(&b.0));
assert_eq!(
got,
[
(task_arn("down"), StopKind::Exited(None)),
(task_arn("gone"), StopKind::Exited(None)),
]
);
}
// ------------------------------------------------------------ classify
@@ -634,9 +645,17 @@ fn classify_stop_follows_the_spec_table() {
..info("t", "STOPPED")
};
assert_eq!(
classify_stop(&with(Some("EssentialContainerExited"), None)),
StopKind::Exited
classify_stop(&with(Some("EssentialContainerExited"), Some("Essential container in task exited"))),
StopKind::Exited(Some("Essential container in task exited".into()))
);
// The entrypoint's idle stop exits with its own status.
let idle = TaskInfo { exit_code: Some(IDLE_EXIT_CODE), ..with(Some("EssentialContainerExited"), None) };
assert_eq!(classify_stop(&idle), StopKind::Idle);
let crashed = TaskInfo { exit_code: Some(137), ..with(Some("EssentialContainerExited"), None) };
assert_eq!(classify_stop(&crashed), StopKind::Exited(None));
// Only with that stop code: a user stop of a task that exits 75 is a user stop.
let user = TaskInfo { exit_code: Some(IDLE_EXIT_CODE), ..with(Some("UserInitiated"), None) };
assert_eq!(classify_stop(&user), StopKind::UserInitiated);
assert_eq!(
classify_stop(&with(Some("SpotInterruption"), None)),
StopKind::Spot
@@ -658,13 +677,13 @@ fn classify_stop_follows_the_spec_table() {
);
assert_eq!(
classify_stop(&with(Some("TerminationNotice"), None)),
StopKind::Exited
StopKind::Exited(None)
);
assert_eq!(
classify_stop(&with(Some("ServiceSchedulerInitiated"), None)),
StopKind::Exited
StopKind::Exited(None)
);
assert_eq!(classify_stop(&with(None, None)), StopKind::Exited);
assert_eq!(classify_stop(&with(None, Some(" "))), StopKind::Exited(None));
}
// ------------------------------------------------------------ SDK mapping
@@ -683,12 +702,14 @@ fn task_info_reads_the_workspace_container_and_its_exec_agent() {
Container::builder()
.name("sidecar")
.runtime_id("wrong")
.exit_code(1)
.build(),
)
.containers(
Container::builder()
.name("workspace")
.runtime_id("rt-1")
.exit_code(75)
.managed_agents(
ManagedAgent::builder()
.name(ManagedAgentName::ExecuteCommandAgent)
@@ -711,11 +732,13 @@ fn task_info_reads_the_workspace_container_and_its_exec_agent() {
assert_eq!(i.stop_code.as_deref(), Some("SpotInterruption"));
assert_eq!(i.runtime_id.as_deref(), Some("rt-1"));
assert!(i.exec_agent_running);
assert_eq!(i.exit_code, Some(75), "the workspace container's, not the sidecar's");
assert_eq!(i.owner.as_deref(), Some(USER));
assert_eq!(i.project.as_deref(), Some(PROJECT_ID));
let bare = task_info(&Task::builder().task_arn("x").last_status("PENDING").build());
assert!(!bare.exec_agent_running);
assert_eq!(bare.runtime_id, None);
assert_eq!(bare.exit_code, None);
assert_eq!(bare.owner, None);
}
+71 -4
View File
@@ -32,6 +32,7 @@ use serde::Serialize;
use tauri::{AppHandle, Emitter};
use tokio::sync::watch;
use crate::aws::tasks::StopKind;
use crate::host::watch::ExitSource;
use crate::host::ContainerTarget;
use crate::models::{Project, ProjectStatus};
@@ -68,6 +69,32 @@ pub enum ExitReason {
pub struct ContainerExited<'a> {
pub project: &'a Project,
pub reason: ExitReason,
/// AWS: why ECS says the task stopped (P8). `None` on Docker, or when
/// the watch has no reason on record.
pub stop: Option<StopNote>,
}
/// Why an AWS task stopped, for the UI's toast.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct StopNote {
/// `spot`, `idle`, `failed` (the project goes to Error), `user` (someone
/// stopped it: no toast) or `exited` (on its own).
pub kind: &'static str,
/// ECS's stopped reason, or a fixed text.
pub message: Option<String>,
}
impl StopNote {
pub fn of(kind: &StopKind) -> StopNote {
let (kind, message) = match kind {
StopKind::Spot => ("spot", Some("Spot interruption".to_string())),
StopKind::Idle => ("idle", None),
StopKind::FailedToStart(r) => ("failed", Some(r.clone())),
StopKind::UserInitiated => ("user", None),
StopKind::Exited(r) => ("exited", r.clone()),
};
StopNote { kind, message }
}
}
fn project_busy(project_id: &str) -> bool {
@@ -227,6 +254,19 @@ pub fn spawn(
p.id,
reason
);
// Why ECS stopped it, read before the tunnel (and with it the
// registry's record) goes.
let kind = p
.is_aws()
.then(|| p.container_id.as_deref().and_then(crate::host::tunnels::stop_kind))
.flatten();
let mut shown = p.clone();
// A task that never got going is an error, not a stop (P8).
if let Some(StopKind::FailedToStart(_)) = kind {
if store_for_aws.update_status(&p.id, ProjectStatus::Error).is_ok() {
shown.status = ProjectStatus::Error;
}
}
// An AWS task that stopped: its tunnel leads nowhere now, and its
// task ARN is history (the data's access point stays).
if p.is_aws() {
@@ -245,7 +285,11 @@ pub fn spawn(
});
}
}
let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: p, reason });
if let Some(store_p) = store_for_aws.get(&p.id) {
shown = store_p;
}
let stop = kind.as_ref().map(StopNote::of);
let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: &shown, reason, stop });
})
.await;
})
@@ -266,15 +310,38 @@ mod tests {
#[test]
fn the_event_says_why_so_the_ui_can_tell_news_from_housekeeping() {
let p = Project::new("demo".to_string(), Vec::new());
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited }).unwrap();
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited, stop: None })
.unwrap();
assert_eq!(v["reason"], "exited");
assert_eq!(v["project"]["name"], "demo");
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::FoundStopped })
.unwrap();
assert!(v["stop"].is_null());
let v = serde_json::to_value(ContainerExited {
project: &p,
reason: ExitReason::FoundStopped,
stop: Some(StopNote::of(&StopKind::Spot)),
})
.unwrap();
assert_eq!(v["reason"], "found_stopped");
assert_eq!(v["stop"], serde_json::json!({ "kind": "spot", "message": "Spot interruption" }));
assert_eq!(CONTAINER_EXITED_EVENT, "project-container-exited");
}
#[test]
fn every_stop_kind_has_a_note_for_the_ui() {
let cases = [
(StopKind::Spot, "spot", Some("Spot interruption")),
(StopKind::Idle, "idle", None),
(StopKind::FailedToStart("CannotPull".into()), "failed", Some("CannotPull")),
(StopKind::UserInitiated, "user", None),
(StopKind::Exited(Some("OOM".into())), "exited", Some("OOM")),
(StopKind::Exited(None), "exited", None),
];
for (k, kind, message) in cases {
let n = StopNote::of(&k);
assert_eq!((n.kind, n.message.as_deref()), (kind, message), "{k:?}");
}
}
/// The shutdown path: a host call that never answers (a wedged daemon)
/// must give way to cancellation at once, not hold teardown.
#[tokio::test]
+201 -35
View File
@@ -698,20 +698,20 @@ fn is_idle(args: &[&str]) -> bool {
#[test]
fn is_idle_needs_every_condition() {
// now last execs tunnels claude_running jobs_running minutes
assert!(is_idle(&["10000", "9400", "0", "0", "0", "0", "10"]), "exactly N minutes");
assert!(is_idle(&["10000", "1000", "0", "0", "0", "0", "10"]));
assert!(!is_idle(&["10000", "9401", "0", "0", "0", "0", "10"]), "one second short");
assert!(!is_idle(&["10000", "1000", "1", "0", "0", "0", "10"]), "an exec is open");
assert!(!is_idle(&["10000", "1000", "0", "2", "0", "0", "10"]), "a tunnel is open");
assert!(!is_idle(&["10000", "1000", "0", "0", "1", "0", "10"]), "claude is running");
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "1", "10"]), "a scheduler job runs");
// now, last activity, claude running, jobs running, minutes.
assert!(is_idle(&["10000", "9400", "0", "0", "10"]), "exactly N minutes");
assert!(is_idle(&["10000", "1000", "0", "0", "10"]));
assert!(!is_idle(&["10000", "9401", "0", "0", "10"]), "one second short");
assert!(!is_idle(&["10000", "1000", "1", "0", "10"]), "claude is running");
assert!(!is_idle(&["10000", "1000", "0", "1", "10"]), "a scheduler job runs");
// Anything unreadable is "not idle".
assert!(!is_idle(&["10000", "", "0", "0", "0", "0", "10"]));
assert!(!is_idle(&["10000", "x", "0", "0", "0", "0", "10"]));
assert!(!is_idle(&["10000", "1000", "", "0", "0", "0", "10"]));
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", "0"]), "0 minutes is off");
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", ""]));
assert!(!is_idle(&["10000", "", "0", "0", "10"]));
assert!(!is_idle(&["10000", "x", "0", "0", "10"]));
assert!(!is_idle(&["10000", "1000", "", "0", "10"]));
assert!(!is_idle(&["10000", "1000", "0", "0", "0"]), "0 minutes is off");
assert!(!is_idle(&["10000", "1000", "0", "0", ""]));
// The old arity (with exec/tunnel counters) is refused, not misread.
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", "10"]));
}
struct Watch {
@@ -723,6 +723,18 @@ impl Watch {
let tmp = tempfile::tempdir().unwrap();
write_script(tmp.path(), "triple-c-idle-watch", IDLE_WATCH);
write_script(tmp.path(), "triple-c-scheduler", SCHEDULER);
// A stand-in WIP push: `--dry-run` prints the file `dry` (nothing:
// clean repos), a push records its arguments in `pushes` and exits
// with the number in `push_rc` (0 when absent).
let d = tmp.path().display();
write_script(
tmp.path(),
"triple-c-wip-push",
&format!(
"#!/bin/bash\nif [ \"$1\" = --dry-run ]; then cat '{d}/dry' 2>/dev/null; exit 0; fi\n\
echo \"$*\" >> '{d}/pushes'\nexit \"$(cat '{d}/push_rc' 2>/dev/null || echo 0)\"\n"
),
);
fs::write(tmp.path().join("token"), "test-token-not-real").unwrap();
fs::create_dir_all(tmp.path().join("home")).unwrap();
Watch { tmp }
@@ -743,10 +755,56 @@ impl Watch {
.env("TRIPLEC_STOP_REASON_FILE", p.join("reason"))
.env("TRIPLEC_SCHEDULER_BIN", p.join("triple-c-scheduler"))
.env("TRIPLEC_SCHEDULER_USER", "")
.env("TRIPLEC_WIP_PUSH_BIN", p.join("triple-c-wip-push"))
// The test host may well be running a `claude` of its own.
.env("TRIPLEC_IDLE_PROC_NAME", "triple-c-no-such-process");
c
}
fn dirty(&self) {
fs::write(
self.tmp.path().join("dry"),
"{\"folder\":\"api\",\"dirty\":true,\"unpushed\":0,\"branch\":\"main\"}\n\
{\"folder\":\"web\",\"dirty\":false,\"unpushed\":0,\"branch\":\"main\"}\n",
)
.unwrap();
}
fn pushes(&self) -> String {
fs::read_to_string(self.tmp.path().join("pushes")).unwrap_or_default()
}
}
/// Stands in for the entrypoint: records the TERM it gets in `got`.
fn fake_entrypoint(dir: &Path) -> std::process::Child {
Command::new("bash")
.arg("-c")
.arg("trap 'echo term > \"$1\"; exit 0' TERM; while :; do sleep 0.1; done")
.arg("t")
.arg(dir.join("got"))
.spawn()
.unwrap()
}
/// Run the watcher loop for at most `secs`; `None` when it was still running
/// (it had not signalled) and was killed. Never hangs a test run.
fn run_watch_for(mut cmd: Command, secs: u64) -> Option<Output> {
let mut child = cmd
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.unwrap();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(secs);
while std::time::Instant::now() < deadline {
if child.try_wait().unwrap().is_some() {
return Some(child.wait_with_output().unwrap());
}
std::thread::sleep(std::time::Duration::from_millis(100));
}
child.kill().ok();
let out = child.wait_with_output().unwrap();
eprintln!("watcher (killed): {}", text(&out));
None
}
/// A one-shot fake agent: answers `/v1/activity` with `body` when the bearer
@@ -782,10 +840,12 @@ s.serve_forever()
(format!("http://127.0.0.1:{}", line.trim()), child)
}
const OLD_ACTIVITY: &str = r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#;
#[test]
fn idle_watch_check_counts_an_unfinished_clone_as_busy() {
let w = Watch::new();
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
let clone = w.tmp.path().join("clone");
fs::create_dir_all(&clone).unwrap();
let busy = w.cmd(&url, 0).env("TRIPLEC_CLONE_STATUS_DIR", &clone).arg("--check").output().unwrap();
@@ -806,40 +866,115 @@ fn idle_watch_check_treats_an_unreachable_agent_as_busy() {
#[test]
fn idle_watch_check_reads_activity_with_the_token() {
let w = Watch::new();
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
let idle = w.cmd(&url, 0).arg("--check").output().unwrap();
let (busy_url, mut busy_agent) =
fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":1,"active_tunnels":0}"#);
let recent = format!(
r#"{{"last_activity_unix":{},"active_execs":0,"active_tunnels":0}}"#,
std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_secs()
);
let (busy_url, mut busy_agent) = fake_agent(w.tmp.path(), &recent);
let busy = w.cmd(&busy_url, 0).arg("--check").output().unwrap();
fs::write(w.tmp.path().join("token"), "wrong").unwrap();
let unauth = w.cmd(&url, 0).arg("--check").output().unwrap();
agent.kill().ok();
busy_agent.kill().ok();
assert_eq!(idle.status.code(), Some(0), "{}", text(&idle));
assert_eq!(busy.status.code(), Some(1), "{}", text(&busy));
assert_eq!(busy.status.code(), Some(1), "recent activity: {}", text(&busy));
assert_eq!(unauth.status.code(), Some(1), "a 401 is not idle: {}", text(&unauth));
}
#[test]
fn idle_watch_ignores_open_but_silent_execs_and_tunnels() {
// R11: an open terminal tab (an attach) or a tunnel no longer keeps the
// workspace up; only their traffic moves the clock.
let w = Watch::new();
let (url, mut agent) =
fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":3,"active_tunnels":2}"#);
let out = w.cmd(&url, 0).arg("--check").output().unwrap();
agent.kill().ok();
assert_eq!(out.status.code(), Some(0), "{}", text(&out));
}
#[test]
fn idle_watch_check_with_unsaved_work_depends_on_the_wip_push() {
let w = Watch::new();
w.dirty();
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
let off = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "0").arg("--check").output().unwrap();
let on = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "1").arg("--check").output().unwrap();
// A dry run that fails is a doubt: not idle.
write_script(w.tmp.path(), "triple-c-wip-push", "#!/bin/bash\nexit 1\n");
let unknown = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "1").arg("--check").output().unwrap();
agent.kill().ok();
assert_eq!(off.status.code(), Some(1), "unsaved work, WIP push off: {}", text(&off));
assert_eq!(on.status.code(), Some(0), "the WIP push would take it: {}", text(&on));
assert_eq!(unknown.status.code(), Some(1), "{}", text(&unknown));
assert_eq!(w.pushes(), "", "--check never pushes");
}
#[test]
fn idle_watch_signals_the_entrypoint_with_reason_idle() {
let w = Watch::new();
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
// Stands in for the entrypoint: records the TERM it gets.
let target = Command::new("bash")
.arg("-c")
.arg("trap 'echo term > \"$1\"; exit 0' TERM; while :; do sleep 0.1; done")
.arg("t")
.arg(w.tmp.path().join("got"))
.spawn()
.unwrap();
let mut target = target;
let out = w.cmd(&url, target.id()).output().unwrap();
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
let mut target = fake_entrypoint(w.tmp.path());
let out = run_watch_for(w.cmd(&url, target.id()), 20).expect("it signals");
let status = target.wait().unwrap();
agent.kill().ok();
assert!(out.status.success(), "{}", text(&out));
assert!(status.success());
assert_eq!(fs::read_to_string(w.tmp.path().join("got")).unwrap().trim(), "term");
assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "idle");
assert_eq!(w.pushes(), "", "nothing unsaved, nothing pushed");
}
#[test]
fn idle_watch_with_unsaved_work_and_the_wip_push_off_stays_up() {
let w = Watch::new();
w.dirty();
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
let mut target = fake_entrypoint(w.tmp.path());
let mut cmd = w.cmd(&url, target.id());
cmd.env("TRIPLEC_WIP_PUSH", "0");
let ran = run_watch_for(cmd, 4);
target.kill().ok();
target.wait().ok();
agent.kill().ok();
assert!(ran.is_none(), "it must not stop: {:?}", ran.map(|o| text(&o)));
assert!(!w.tmp.path().join("got").exists(), "no TERM");
assert!(!w.tmp.path().join("reason").exists(), "no marker");
assert_eq!(w.pushes(), "");
}
#[test]
fn idle_watch_pushes_before_an_idle_stop_and_stays_up_while_the_push_fails() {
// The push fails: the workspace stays up and retries, backing off.
let w = Watch::new();
w.dirty();
fs::write(w.tmp.path().join("push_rc"), "1").unwrap();
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
let mut target = fake_entrypoint(w.tmp.path());
let mut cmd = w.cmd(&url, target.id());
cmd.env("TRIPLEC_WIP_PUSH", "1").env("TRIPLEC_WIP_BACKOFF_MAX", "2");
let ran = run_watch_for(cmd, 6);
assert!(ran.is_none(), "a failed push must not stop the workspace");
assert!(!w.tmp.path().join("got").exists(), "no TERM");
assert!(!w.tmp.path().join("reason").exists(), "no marker");
let tries = w.pushes().lines().count();
assert!((1..=3).contains(&tries), "retried with a back-off, not every second: {tries}");
assert!(w.pushes().lines().all(|l| l == "--reason idle"), "{}", w.pushes());
// Now the push works: it pushes once, then signals with the idle marker.
fs::write(w.tmp.path().join("push_rc"), "0").unwrap();
fs::remove_file(w.tmp.path().join("pushes")).unwrap();
let mut cmd = w.cmd(&url, target.id());
cmd.env("TRIPLEC_WIP_PUSH", "1");
let out = run_watch_for(cmd, 20).expect("it stops once the push succeeded");
target.wait().unwrap();
agent.kill().ok();
assert!(out.status.success(), "{}", text(&out));
assert_eq!(w.pushes(), "--reason idle\n");
assert_eq!(fs::read_to_string(w.tmp.path().join("got")).unwrap().trim(), "term");
assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "idle");
}
#[test]
@@ -847,13 +982,13 @@ fn idle_watch_keeps_a_stop_guard_marker_already_recorded() {
for marker in ["pushed", "discard"] {
let w = Watch::new();
fs::write(w.tmp.path().join("reason"), format!("{marker}\n")).unwrap();
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
let mut target = Command::new("bash")
.arg("-c")
.arg("trap 'exit 0' TERM; while :; do sleep 0.1; done")
.spawn()
.unwrap();
let out = w.cmd(&url, target.id()).output().unwrap();
let out = run_watch_for(w.cmd(&url, target.id()), 20).expect("it signals");
target.wait().unwrap();
agent.kill().ok();
assert!(out.status.success(), "{}", text(&out));
@@ -864,7 +999,7 @@ fn idle_watch_keeps_a_stop_guard_marker_already_recorded() {
// ------------------------------------------------------- stop (SIGTERM) path
#[test]
fn aws_stop_runs_the_wip_push_with_the_recorded_reason_once() {
fn aws_stop_pushes_on_sigterm_once_and_leaves_an_idle_stop_to_the_watcher() {
let tmp = tempfile::tempdir().unwrap();
let log = tmp.path().join("calls");
let fake = write_script(
@@ -882,13 +1017,18 @@ fn aws_stop_runs_the_wip_push_with_the_recorded_reason_once() {
assert!(out.status.success(), "{}", text(&out));
assert_eq!(fs::read_to_string(&log).unwrap(), "--reason sigterm\n");
// An idle stop: the watcher already pushed, so the trap does not, and it
// exits with the idle status the app reads from ECS.
fs::remove_file(&log).unwrap();
fs::write(&reason, "idle\n").unwrap();
let out = bash(&script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "1")], &[]);
let idle_script = format!("{blocks}\naws_before_stop\necho \"status=${{triplec_exit_status:-0}}\"\n");
let out = bash(&idle_script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "1")], &[]);
assert!(out.status.success(), "{}", text(&out));
assert_eq!(fs::read_to_string(&log).unwrap(), "--reason idle\n");
assert!(!log.exists(), "no second push after the watcher's");
assert!(text(&out).contains("entrypoint: stopping (idle)"), "{}", text(&out));
assert!(text(&out).contains("status=75"), "{}", text(&out));
fs::remove_file(&reason).unwrap();
fs::remove_file(&log).unwrap();
let out = bash(&script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "0")], &[]);
assert!(out.status.success(), "{}", text(&out));
assert!(!log.exists(), "WIP push off");
@@ -922,6 +1062,32 @@ fn aws_stop_skips_its_push_when_the_app_already_pushed_or_the_user_declined() {
assert_eq!(fs::read_to_string(&log).unwrap(), "--reason sigterm\n");
}
#[test]
fn the_idle_exit_status_is_the_one_the_app_reads() {
assert!(
block(ENTRYPOINT, "aws_stop")
.contains(&format!("triplec_idle_exit={}\n", crate::aws::tasks::IDLE_EXIT_CODE)),
"entrypoint.sh and aws::tasks::IDLE_EXIT_CODE disagree"
);
assert!(ENTRYPOINT.contains("exit \"${triplec_exit_status:-0}\""), "stop_keepalive exits with it");
// The watcher gets the WIP setting through its scrubbed environment.
assert!(block(ENTRYPOINT, "aws_stop").contains("TRIPLEC_WIP_PUSH=\"${TRIPLEC_WIP_PUSH:-0}\""));
}
#[test]
fn start_clone_repos_marks_done_however_the_clone_ends() {
let tmp = tempfile::tempdir().unwrap();
let status = tmp.path().join("status");
// clone_repos replaced by one that fails half-way.
let script = format!(
"{}\nclone_repos() {{ exit 3; }}\nstart_clone_repos '[x]' \"$1\"\nwait\n",
aws_blocks(&["aws_env", "aws_repos"])
);
let out = bash(&script, &[&status], &[], &[]);
assert!(out.status.success(), "{}", text(&out));
assert!(status.join("done").exists(), "a failed clone still ends: {}", text(&out));
}
#[test]
fn docker_mode_never_reaches_the_aws_blocks() {
// Every new step sits behind TRIPLEC_HOST=aws; the scripts are only
+36 -16
View File
@@ -35,7 +35,7 @@ use crate::aws::runtime::{
};
use crate::aws::tasks::{
find_or_run, find_project_task, stopped_among, task_log_stream, wait_ready, SdkTaskApi,
TaskApi, TaskInfo, TaskLogs, WaitTiming, LOG_TAIL_LINES, STOP_REASON,
StopKind, TaskApi, TaskInfo, TaskLogs, WaitTiming, LOG_TAIL_LINES, STOP_REASON,
};
use crate::aws::tofu_modules::{
environment_cluster_name, log_stream_prefix, owner_key, task_family,
@@ -75,6 +75,11 @@ const HEALTH_TIMEOUT: Duration = Duration::from_secs(30);
const HEALTH_INTERVAL: Duration = Duration::from_secs(1);
/// How long a terminal waits for a background re-attach.
pub const REATTACH_WAIT: Duration = Duration::from_secs(90);
/// The most one AWS death-watch poll of a group may take.
const WATCH_CALL_TIMEOUT: Duration = Duration::from_secs(60);
/// The most a StopTask (credentials included) may take before the stop is
/// reported as failed; the SDK's own bound is per call.
const STOP_TIMEOUT: Duration = Duration::from_secs(45);
fn s<E: std::fmt::Display>(e: E) -> String {
e.to_string()
@@ -730,7 +735,18 @@ pub async fn task_log(project: &Project, settings: &AppSettings, limit: i32) ->
pub async fn stop_task_arn<T: TaskApi>(api: &T, task_arn: &str) -> Result<(), String> {
let cluster = cluster_of_task(task_arn)
.ok_or_else(|| format!("{task_arn} is not a task ARN with a cluster"))?;
api.stop_task(cluster, task_arn, STOP_REASON).await.map_err(s)
bounded_stop(api.stop_task(cluster, task_arn, STOP_REASON)).await
}
/// A StopTask, bounded by [`STOP_TIMEOUT`]: a stop that cannot reach AWS
/// fails (and the project keeps its task ARN) instead of hanging the UI.
async fn bounded_stop(
call: impl std::future::Future<Output = Result<(), crate::aws::AwsError>>,
) -> Result<(), String> {
tokio::time::timeout(STOP_TIMEOUT, call)
.await
.map_err(|_| format!("StopTask did not answer within {STOP_TIMEOUT:?}; try again"))?
.map_err(s)
}
/// `ProjectHost::stop` for an agent target: the task behind a registered
@@ -738,10 +754,7 @@ pub async fn stop_task_arn<T: TaskApi>(api: &T, task_arn: &str) -> Result<(), St
pub async fn stop_endpoint(ep: &AgentEndpoint) -> Option<Result<(), String>> {
let (_connection, _region, cluster, arn) = tunnels::task_of(ep)?;
let (project_id, sdk) = tunnels::owner_of(ep)?;
let res = SdkTaskApi::new(&sdk)
.stop_task(&cluster, &arn, STOP_REASON)
.await
.map_err(s);
let res = bounded_stop(SdkTaskApi::new(&sdk).stop_task(&cluster, &arn, STOP_REASON)).await;
if res.is_ok() {
tunnels::close(&project_id).await;
}
@@ -1089,22 +1102,29 @@ impl AwsTaskLookup for StoreTaskLookup {
Box::pin(async move {
let settings = self.settings.get();
let conn = crate::commands::host_commands::aws_conn(&group.connection_id, &settings)?;
let sdk = sdk_config(&conn, &group.region).await.map_err(s)?;
let stopped = stopped_among(&SdkTaskApi::new(&sdk), &group.cluster_arn, arns)
.await
.map_err(s)?;
// The SDK bounds each call; this bounds the whole poll of a group
// (credentials included, which an SSO refresh can stall), so one
// stuck group never stalls the watch of every other.
let stopped = tokio::time::timeout(WATCH_CALL_TIMEOUT, async {
let sdk = sdk_config(&conn, &group.region).await.map_err(s)?;
stopped_among(&SdkTaskApi::new(&sdk), &group.cluster_arn, arns)
.await
.map_err(s)
})
.await
.map_err(|_| format!("ECS did not answer within {WATCH_CALL_TIMEOUT:?}"))??;
record_stopped(&self.projects.list(), &stopped);
Ok(stopped)
Ok(stopped.into_iter().map(|(arn, _)| arn).collect())
})
}
}
/// Tell the registry which targets' tasks ECS reports stopped, so
/// `has_exited` confirms them (R3).
pub fn record_stopped(projects: &[Project], stopped: &[String]) {
/// Tell the registry which targets' tasks ECS reports stopped, and why, so
/// `has_exited` confirms them (R3) and the death report can say why (P8).
pub fn record_stopped(projects: &[Project], stopped: &[(String, StopKind)]) {
for w in projects.iter().filter_map(watched_task) {
if stopped.contains(&w.task_arn) {
tunnels::mark_stopped(w.target.id());
if let Some((_, why)) = stopped.iter().find(|(arn, _)| *arn == w.task_arn) {
tunnels::mark_stopped(w.target.id(), why.clone());
}
}
}
+3 -1
View File
@@ -57,6 +57,7 @@ fn task(p: &Project, id: &str, last: &str, ready: bool) -> TaskInfo {
task_definition_arn: format!("arn:aws:ecs:{REGION}:{ACCOUNT}:task-definition/{}:1", family(p)),
runtime_id: ready.then(|| "rt-1".into()),
exec_agent_running: ready,
exit_code: None,
owner: Some(USER.into()),
project: Some(p.id.clone()),
}
@@ -572,8 +573,9 @@ async fn the_watch_reports_a_running_aws_project_as_its_stored_target_and_confir
assert_eq!(w.group.cluster_arn, cluster_arn());
assert_eq!(w.group.connection_id, "c1");
assert!(!w.target.has_exited().await, "nothing recorded yet");
record_stopped(std::slice::from_ref(&p), &[task_arn("w1")]);
record_stopped(std::slice::from_ref(&p), &[(task_arn("w1"), StopKind::Idle)]);
assert!(w.target.has_exited().await);
assert_eq!(crate::host::tunnels::stop_kind(w.target.id()), Some(StopKind::Idle));
let mut stopped = p.clone();
stopped.status = ProjectStatus::Stopped;
+15 -7
View File
@@ -20,6 +20,7 @@ use futures_util::future::BoxFuture;
use tokio::sync::watch;
use super::agent::AgentEndpoint;
use crate::aws::tasks::StopKind;
use super::ssm::{EcsTask, PortForward};
/// Something that owns a tunnel's resources: the real [`PortForward`], or a
@@ -58,8 +59,9 @@ pub type TaskOf = (String, String, String, String);
#[derive(Default)]
struct Registry {
live: HashMap<String, LiveTunnel>,
/// Target ids (endpoint base URLs) whose task the AWS watch saw STOPPED.
stopped: HashSet<String>,
/// Target ids (endpoint base URLs) whose task the AWS watch saw STOPPED,
/// and why.
stopped: HashMap<String, StopKind>,
/// Projects whose tunnel is being re-opened in the background.
pending: HashSet<String>,
/// Bumped by every [`close`] (a stop or remove), every [`register`] (a
@@ -245,14 +247,19 @@ pub async fn close_all() {
futures_util::future::join_all(all.into_iter().map(|t| t.forward.close())).await;
}
/// The AWS watch saw the task behind target `target_id` STOPPED.
pub fn mark_stopped(target_id: &str) {
lock().stopped.insert(target_id.to_string());
/// The AWS watch saw the task behind target `target_id` STOPPED, for `why`.
pub fn mark_stopped(target_id: &str, why: StopKind) {
lock().stopped.insert(target_id.to_string(), why);
}
/// Whether the AWS watch has confirmed the task behind `target_id` stopped.
pub fn is_stopped(target_id: &str) -> bool {
lock().stopped.contains(target_id)
lock().stopped.contains_key(target_id)
}
/// Why the task behind `target_id` stopped, once the watch has seen it.
pub fn stop_kind(target_id: &str) -> Option<StopKind> {
lock().stopped.get(target_id).cloned()
}
pub fn is_pending(project_id: &str) -> bool {
@@ -341,7 +348,8 @@ pub(crate) mod tests {
let p = pid("rereg");
register(&p, fake_tunnel(61002, closes.clone()));
let id = AgentEndpoint::loopback(61003, "tok").base_url.to_string();
mark_stopped(&id);
mark_stopped(&id, StopKind::Spot);
assert_eq!(stop_kind(&id), Some(StopKind::Spot));
let old = register(&p, fake_tunnel(61003, closes.clone())).unwrap();
assert_eq!(old.endpoint.base_url.port(), Some(61002));
assert!(!is_stopped(&id), "a new task on a reused port is not stopped");
@@ -54,6 +54,35 @@ describe("useProjectStatusEvents", () => {
expect(toasts).toHaveLength(0);
});
it("says why an AWS task stopped, and keeps a user's stop quiet", async () => {
renderHook(() => useProjectStatusEvents());
await vi.waitFor(() => expect(handler).not.toBeNull());
const fire = (stop: unknown, reason = "exited") => {
useAppState.setState({ toasts: [] });
handler!({ payload: { project: project({ status: "stopped" }), reason, stop } });
return useAppState.getState().toasts;
};
let t = fire({ kind: "spot", message: "Spot interruption" });
expect(t[0].kind).toBe("info");
expect(t[0].message).toContain("Spot interruption");
t = fire({ kind: "idle", message: null });
expect(t[0].kind).toBe("info");
expect(t[0].message).toContain("idle");
expect(fire({ kind: "user", message: null })).toHaveLength(0);
// A task that never started is news even when found at startup.
t = fire({ kind: "failed", message: "CannotPullContainerError: denied" }, "found_stopped");
expect(t[0].kind).toBe("error");
expect(t[0].detail).toContain("CannotPullContainerError");
t = fire({ kind: "exited", message: "OutOfMemoryError" });
expect(t[0].kind).toBe("error");
expect(t[0].detail).toContain("OutOfMemoryError");
});
it("stops listening on unmount", async () => {
const { unmount } = renderHook(() => useProjectStatusEvents());
await vi.waitFor(() => expect(handler).not.toBeNull());
+54 -10
View File
@@ -12,6 +12,56 @@ export interface ContainerExitedEvent {
* while the app was closed, e.g. a host reboot); nothing was interrupted.
*/
reason: "exited" | "found_stopped";
/** AWS: why ECS says the task stopped (`StopNote`); absent on Docker. */
stop?: StopNote | null;
}
/** `container_watch.rs`'s `StopNote`. */
export interface StopNote {
/** `user`: someone stopped it (console, CLI, another machine) — no toast. */
kind: "spot" | "idle" | "failed" | "user" | "exited";
message: string | null;
}
/** The toast for an exit, or null for none. */
export function exitToast(
name: string,
reason: ContainerExitedEvent["reason"],
stop: StopNote | null | undefined,
): { kind: "error" | "info"; message: string; detail?: string } | null {
if (stop?.kind === "user") return null;
if (stop?.kind === "failed") {
return {
kind: "error",
message: `The AWS workspace for “${name}” failed to start`,
detail: stop.message ?? undefined,
};
}
if (reason !== "exited") return null;
switch (stop?.kind) {
case "spot":
return {
kind: "info",
message: `“${name}” stopped: Spot interruption`,
detail:
"AWS reclaimed the Spot capacity it ran on. Start the project again to get a new workspace.",
};
case "idle":
return {
kind: "info",
message: `“${name}” stopped after being idle`,
detail: "Its AWS workspace stopped itself to save cost. Start the project again to continue.",
};
default:
return {
kind: "error",
message: `The container for “${name}” stopped`,
detail:
"It exited without Triple-C stopping it — open sessions were disconnected. " +
"Start the project again to reconnect." +
(stop?.message ? ` (${stop.message})` : ""),
};
}
}
/**
@@ -25,18 +75,12 @@ export function useProjectStatusEvents() {
let cancelled = false;
let unlisten: UnlistenFn | null = null;
listen<ContainerExitedEvent>("project-container-exited", (event) => {
const { project, reason } = event.payload;
const { project, reason, stop } = event.payload;
const state = useAppState.getState();
state.updateProjectInList(project);
if (reason !== "exited") return;
state.pushToast({
kind: "error",
message: `The container for “${project.name}” stopped`,
detail:
"It exited without Triple-C stopping it — open sessions were disconnected. " +
"Start the project again to reconnect.",
dedupeKey: `container-exited-${project.id}`,
});
const toast = exitToast(project.name, reason, stop);
if (!toast) return;
state.pushToast({ ...toast, dedupeKey: `container-exited-${project.id}` });
})
.then((fn) => {
if (cancelled) fn();
@@ -349,7 +349,9 @@ updates use the existing "new image available" flow: the next start uses the new
`git status --porcelain` and any unpushed commits in each repo. If anything would be lost, it
offers: **Push to a `triple-c/wip/<date>` branch** (the preselected default), **Stop anyway**, or
**Cancel**. The same check runs before idle auto-stop. If anything is dirty, idle auto-stop
pushes a WIP branch instead of throwing work away.
pushes a WIP branch instead of throwing work away, and stops only once that push succeeded (a
failed push keeps the workspace up and is retried, backing off). With the WIP push off, unsaved
work keeps the workspace up.
- **Fargate task retirement and Spot interruption** send `SIGTERM` with a short deadline (2 minutes
for Spot). The entrypoint's `SIGTERM` trap runs the same WIP push, best-effort, when the project
has it enabled.
@@ -497,9 +499,12 @@ way.
- user-initiated → Stopped.
EventBridge → SQS for push notifications is a later improvement.
- **Idle auto-stop** runs **inside** the task: the agent tracks activity (terminal I/O, agent
calls, running `claude` processes and scheduler jobs) and stops the task by exiting the
entrypoint. It doesn't depend on the app being open. Because a running scheduler job counts as
- **Idle auto-stop** runs **inside** the task: the agent tracks activity (terminal I/O and agent
calls; an open but silent terminal or tunnel is not activity, and the app's background polls
are marked `X-Triple-C-Background` and never count — ruling R11), the watcher adds running
`claude` processes, scheduler jobs and unsaved git work, and stops the task by exiting the
entrypoint with status 75, which the app reads back from `DescribeTasks` to say "stopped after
being idle". It doesn't depend on the app being open. Because a running scheduler job counts as
activity, scheduled tasks keep a workspace alive. The admin's maximum lifetime is a hard cap
either way.
- **Health:** the task definition carries an ECS container health check against the agent, run