fix(aws): say why a task stopped, and bound every AWS call
- classify_stop is wired into the death watch: stopped_among returns each task's StopKind, the tunnel registry keeps it, and ContainerExited carries a StopNote. Spot -> 'Spot interruption'; TaskFailedToStart -> Error with the reason; the entrypoint's idle exit (75, read from the workspace container's exit code) -> 'stopped after being idle'; a user stop is applied silently (P8). is_quota_error, dead, moved to tests. - The SDK config has a TimeoutConfig (30 s operation / 10 s attempt); the watch's per-group poll and StopTask also have tokio bounds. - Watcher tests cover silent sessions, unsaved work with the WIP push on and off, a failing push and the idle exit status; docs updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
151b7690df
commit
5e8e67a0a4
13 files changed
+511
-102
No files matched your search
@@ -399,6 +399,12 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li
|
||||
user's index is untouched; needs the task definition's `stopTimeout` raised, 120 s) unless the stop
|
||||
marker says the app handled it. The idle watcher runs as root under a supervisor and signals the
|
||||
entrypoint's own PID (not PID 1, which is docker-init), so idle and ECS stops share one trap.
|
||||
Idle = the agent's `last_activity_unix` older than N minutes (open-but-silent attaches/tunnels
|
||||
don't count; requests with `X-Triple-C-Background: 1`, e.g. the auth bridge's poll, never do),
|
||||
no `claude`, no scheduler job, no running clone, and no unsaved git work: with the WIP push on the
|
||||
watcher pushes *before* signalling and stays up if that fails; with it off, dirty repos keep the
|
||||
workspace up. An idle stop exits 75 (`aws::tasks::IDLE_EXIT_CODE`) so the death watch can tell it
|
||||
from a crash (`classify_stop` → the `ContainerExited` toast).
|
||||
**The `TRIPLEC_` env prefix is reserved** on both hosts (`RESERVED_ENV_PREFIXES`; `workspace.rs`
|
||||
drops custom variables in it): the entrypoint's settings live there.
|
||||
- **Browser runtime libraries are baked in; browser *binaries* are not.** A layer runs
|
||||
|
||||
@@ -5,6 +5,7 @@ use aws_config::identity::IdentityCache;
|
||||
use aws_config::profile::ProfileFileCredentialsProvider;
|
||||
use aws_config::provider_config::ProviderConfig;
|
||||
use aws_config::retry::RetryConfig;
|
||||
use aws_config::timeout::TimeoutConfig;
|
||||
use aws_config::{BehaviorVersion, Region, SdkConfig};
|
||||
use aws_credential_types::provider::error::CredentialsError;
|
||||
use aws_credential_types::provider::{future, ProvideCredentials, SharedCredentialsProvider};
|
||||
@@ -81,9 +82,9 @@ fn keys_credentials(keys: &secure::AwsKeys) -> SharedCredentialsProvider {
|
||||
/// stray endpoint there would send every connection's signed requests (and
|
||||
/// the permission check) somewhere other than AWS. Connections talk to the
|
||||
/// real regional endpoints; a profile's own `endpoint_url` is ignored too.
|
||||
/// The HTTP client, timeouts and stalled-stream protection are each service
|
||||
/// client's defaults (the `default-https-client` stack every AWS crate here
|
||||
/// is built with).
|
||||
/// The HTTP client and stalled-stream protection are each service client's
|
||||
/// defaults (the `default-https-client` stack every AWS crate here is built
|
||||
/// with). Timeouts are ours: [`OPERATION_TIMEOUT`] / [`ATTEMPT_TIMEOUT`].
|
||||
fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> SdkConfig {
|
||||
let mut builder = SdkConfig::builder()
|
||||
.behavior_version(BehaviorVersion::latest())
|
||||
@@ -91,6 +92,7 @@ fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> S
|
||||
.credentials_provider(credentials)
|
||||
.identity_cache(IdentityCache::lazy().build())
|
||||
.retry_config(RetryConfig::standard())
|
||||
.timeout_config(timeouts())
|
||||
.time_source(SharedTimeSource::default());
|
||||
if let Some(sleep) = default_async_sleep() {
|
||||
builder = builder.sleep_impl(sleep);
|
||||
@@ -98,6 +100,22 @@ fn build_sdk_config(region: Region, credentials: SharedCredentialsProvider) -> S
|
||||
builder.build()
|
||||
}
|
||||
|
||||
/// Every call the app makes through the SDK is a control-plane call with a
|
||||
/// small answer (ECS, Secrets Manager, EFS, IAM, SSM, CloudWatch Logs, STS,
|
||||
/// and S3 reads of state objects capped at 2 KiB); OpenTofu talks to AWS
|
||||
/// itself, not through these clients. Without a bound a half-open
|
||||
/// connection would hang a start, a stop or the death watch for minutes.
|
||||
/// The operation bound covers every retry; the attempt bound is one try.
|
||||
pub const OPERATION_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||
pub const ATTEMPT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);
|
||||
|
||||
fn timeouts() -> TimeoutConfig {
|
||||
TimeoutConfig::builder()
|
||||
.operation_timeout(OPERATION_TIMEOUT)
|
||||
.operation_attempt_timeout(ATTEMPT_TIMEOUT)
|
||||
.build()
|
||||
}
|
||||
|
||||
/// Puts the profile's name into the source chain of every credentials error,
|
||||
/// so a failure deep inside the SDK can be reported against the profile.
|
||||
#[derive(Debug)]
|
||||
@@ -297,4 +315,18 @@ sts =
|
||||
assert_isolated(&cfg);
|
||||
assert!(cfg.credentials_provider().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_sdk_call_is_bounded() {
|
||||
let keys = crate::storage::secure::AwsKeys {
|
||||
access_key_id: "AKIDTEST".into(),
|
||||
secret_access_key: "s".into(),
|
||||
session_token: None,
|
||||
};
|
||||
let cfg = build_sdk_config(Region::new("us-west-2"), keys_credentials(&keys));
|
||||
let t = cfg.timeout_config().expect("a timeout config");
|
||||
assert_eq!(t.operation_timeout(), Some(OPERATION_TIMEOUT));
|
||||
assert_eq!(t.operation_attempt_timeout(), Some(ATTEMPT_TIMEOUT));
|
||||
assert!(ATTEMPT_TIMEOUT < OPERATION_TIMEOUT);
|
||||
}
|
||||
}
|
||||
@@ -247,6 +247,7 @@ fn live(id: &str) -> TaskInfo {
|
||||
),
|
||||
runtime_id: None,
|
||||
exec_agent_running: false,
|
||||
exit_code: None,
|
||||
owner: Some(USER.into()),
|
||||
project: Some(PROJECT.into()),
|
||||
}
|
||||
|
||||
@@ -64,6 +64,8 @@ pub struct TaskInfo {
|
||||
pub runtime_id: Option<String>,
|
||||
/// The `workspace` container's ExecuteCommandAgent is RUNNING.
|
||||
pub exec_agent_running: bool,
|
||||
/// The `workspace` container's exit code, once it has one.
|
||||
pub exit_code: Option<i32>,
|
||||
/// Its `triple-c:owner` / `triple-c:project` tags (DescribeTasks with
|
||||
/// TAGS; RunTask's answer carries the tags it was given).
|
||||
pub owner: Option<String>,
|
||||
@@ -116,6 +118,7 @@ pub(crate) fn task_info(t: &Task) -> TaskInfo {
|
||||
&& a.last_status() == Some("RUNNING")
|
||||
})
|
||||
}),
|
||||
exit_code: workspace.and_then(|c| c.exit_code()),
|
||||
owner: tag_value(t.tags(), TAG_OWNER),
|
||||
project: tag_value(t.tags(), TAG_PROJECT),
|
||||
}
|
||||
@@ -337,11 +340,6 @@ pub fn quota_or(e: AwsError) -> AwsError {
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(dead_code)] // for the UI's start errors (Task 8)
|
||||
pub fn is_quota_error(e: &AwsError) -> bool {
|
||||
matches!(e, AwsError::QuotaExceeded { .. })
|
||||
}
|
||||
|
||||
/// RunTask's answer: its one task, or an error from its `failures`
|
||||
/// (`(reason, detail)`), a quota one when they say so.
|
||||
pub fn run_task_outcome(
|
||||
@@ -480,13 +478,19 @@ async fn stopped_error<A: TaskApi>(api: &A, logs: &TaskLogs, t: &TaskInfo) -> Aw
|
||||
|
||||
// ----------------------------------------------------------- stop / watch
|
||||
|
||||
/// The entrypoint's exit status after an idle stop (`container/entrypoint.sh`,
|
||||
/// `stop_keepalive`). A stop from ECS (SIGTERM) exits 0; this one is distinct
|
||||
/// so an idle stop (`EssentialContainerExited`) is not mistaken for a crash.
|
||||
pub const IDLE_EXIT_CODE: i32 = 75;
|
||||
|
||||
/// Why a task stopped, as the app reports it (spec, "Lifecycle and status").
|
||||
#[allow(dead_code)] // the stop reasons the UI shows (Task 8)
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum StopKind {
|
||||
/// Exited on its own (the essential container ended, idle stop, or AWS
|
||||
/// retired it).
|
||||
Exited,
|
||||
/// Exited on its own (the essential container ended, or AWS retired it),
|
||||
/// with ECS's stopped reason when it gave one.
|
||||
Exited(Option<String>),
|
||||
/// The workspace's own idle stop ([`IDLE_EXIT_CODE`]).
|
||||
Idle,
|
||||
/// A Fargate Spot interruption (P8).
|
||||
Spot,
|
||||
/// Never got going: the project goes to Error with this reason.
|
||||
@@ -495,7 +499,6 @@ pub enum StopKind {
|
||||
UserInitiated,
|
||||
}
|
||||
|
||||
#[allow(dead_code)] // Task 8
|
||||
pub fn classify_stop(info: &TaskInfo) -> StopKind {
|
||||
match info.stop_code.as_deref() {
|
||||
Some("SpotInterruption") => StopKind::Spot,
|
||||
@@ -506,25 +509,28 @@ pub fn classify_stop(info: &TaskInfo) -> StopKind {
|
||||
.unwrap_or_else(|| "The task failed to start".into()),
|
||||
),
|
||||
Some("UserInitiated") => StopKind::UserInitiated,
|
||||
_ => StopKind::Exited,
|
||||
Some("EssentialContainerExited") if info.exit_code == Some(IDLE_EXIT_CODE) => StopKind::Idle,
|
||||
_ => StopKind::Exited(info.stopped_reason.clone().filter(|r| !r.trim().is_empty())),
|
||||
}
|
||||
}
|
||||
|
||||
/// Of `arns`, the ones that are STOPPED or that ECS no longer knows (it
|
||||
/// forgets stopped tasks after about an hour). A task merely being stopped
|
||||
/// is not yet reported. For the AWS death watch.
|
||||
/// forgets stopped tasks after about an hour), each with why it stopped (a
|
||||
/// forgotten task: [`StopKind::Exited`] without a reason). A task merely
|
||||
/// being stopped is not yet reported. For the AWS death watch.
|
||||
pub async fn stopped_among<A: TaskApi>(
|
||||
api: &A,
|
||||
cluster: &str,
|
||||
arns: &[String],
|
||||
) -> Result<Vec<String>, AwsError> {
|
||||
) -> Result<Vec<(String, StopKind)>, AwsError> {
|
||||
let mut stopped = Vec::new();
|
||||
for chunk in arns.chunks(DESCRIBE_BATCH) {
|
||||
let known = api.describe(cluster, chunk).await?;
|
||||
for arn in chunk {
|
||||
match known.iter().find(|t| &t.arn == arn) {
|
||||
Some(t) if t.last_status != "STOPPED" => {}
|
||||
_ => stopped.push(arn.clone()),
|
||||
Some(t) => stopped.push((arn.clone(), classify_stop(t))),
|
||||
None => stopped.push((arn.clone(), StopKind::Exited(None))),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -44,6 +44,10 @@ fn env_info() -> EnvironmentInfo {
|
||||
}
|
||||
}
|
||||
|
||||
fn is_quota_error(e: &AwsError) -> bool {
|
||||
matches!(e, AwsError::QuotaExceeded { .. })
|
||||
}
|
||||
|
||||
fn info(id: &str, last: &str) -> TaskInfo {
|
||||
TaskInfo {
|
||||
arn: task_arn(id),
|
||||
@@ -60,6 +64,7 @@ fn info(id: &str, last: &str) -> TaskInfo {
|
||||
task_definition_arn: td_arn(3),
|
||||
runtime_id: None,
|
||||
exec_agent_running: false,
|
||||
exit_code: None,
|
||||
owner: Some(USER.into()),
|
||||
project: Some(PROJECT_ID.into()),
|
||||
}
|
||||
@@ -620,8 +625,14 @@ async fn stopped_among_reports_stopped_and_vanished_tasks_only() {
|
||||
task_arn("gone"),
|
||||
];
|
||||
let mut got = stopped_among(&api, &cluster(), &arns).await.unwrap();
|
||||
got.sort();
|
||||
assert_eq!(got, [task_arn("down"), task_arn("gone")]);
|
||||
got.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
assert_eq!(
|
||||
got,
|
||||
[
|
||||
(task_arn("down"), StopKind::Exited(None)),
|
||||
(task_arn("gone"), StopKind::Exited(None)),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------ classify
|
||||
@@ -634,9 +645,17 @@ fn classify_stop_follows_the_spec_table() {
|
||||
..info("t", "STOPPED")
|
||||
};
|
||||
assert_eq!(
|
||||
classify_stop(&with(Some("EssentialContainerExited"), None)),
|
||||
StopKind::Exited
|
||||
classify_stop(&with(Some("EssentialContainerExited"), Some("Essential container in task exited"))),
|
||||
StopKind::Exited(Some("Essential container in task exited".into()))
|
||||
);
|
||||
// The entrypoint's idle stop exits with its own status.
|
||||
let idle = TaskInfo { exit_code: Some(IDLE_EXIT_CODE), ..with(Some("EssentialContainerExited"), None) };
|
||||
assert_eq!(classify_stop(&idle), StopKind::Idle);
|
||||
let crashed = TaskInfo { exit_code: Some(137), ..with(Some("EssentialContainerExited"), None) };
|
||||
assert_eq!(classify_stop(&crashed), StopKind::Exited(None));
|
||||
// Only with that stop code: a user stop of a task that exits 75 is a user stop.
|
||||
let user = TaskInfo { exit_code: Some(IDLE_EXIT_CODE), ..with(Some("UserInitiated"), None) };
|
||||
assert_eq!(classify_stop(&user), StopKind::UserInitiated);
|
||||
assert_eq!(
|
||||
classify_stop(&with(Some("SpotInterruption"), None)),
|
||||
StopKind::Spot
|
||||
@@ -658,13 +677,13 @@ fn classify_stop_follows_the_spec_table() {
|
||||
);
|
||||
assert_eq!(
|
||||
classify_stop(&with(Some("TerminationNotice"), None)),
|
||||
StopKind::Exited
|
||||
StopKind::Exited(None)
|
||||
);
|
||||
assert_eq!(
|
||||
classify_stop(&with(Some("ServiceSchedulerInitiated"), None)),
|
||||
StopKind::Exited
|
||||
StopKind::Exited(None)
|
||||
);
|
||||
assert_eq!(classify_stop(&with(None, None)), StopKind::Exited);
|
||||
assert_eq!(classify_stop(&with(None, Some(" "))), StopKind::Exited(None));
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------ SDK mapping
|
||||
@@ -683,12 +702,14 @@ fn task_info_reads_the_workspace_container_and_its_exec_agent() {
|
||||
Container::builder()
|
||||
.name("sidecar")
|
||||
.runtime_id("wrong")
|
||||
.exit_code(1)
|
||||
.build(),
|
||||
)
|
||||
.containers(
|
||||
Container::builder()
|
||||
.name("workspace")
|
||||
.runtime_id("rt-1")
|
||||
.exit_code(75)
|
||||
.managed_agents(
|
||||
ManagedAgent::builder()
|
||||
.name(ManagedAgentName::ExecuteCommandAgent)
|
||||
@@ -711,11 +732,13 @@ fn task_info_reads_the_workspace_container_and_its_exec_agent() {
|
||||
assert_eq!(i.stop_code.as_deref(), Some("SpotInterruption"));
|
||||
assert_eq!(i.runtime_id.as_deref(), Some("rt-1"));
|
||||
assert!(i.exec_agent_running);
|
||||
assert_eq!(i.exit_code, Some(75), "the workspace container's, not the sidecar's");
|
||||
assert_eq!(i.owner.as_deref(), Some(USER));
|
||||
assert_eq!(i.project.as_deref(), Some(PROJECT_ID));
|
||||
|
||||
let bare = task_info(&Task::builder().task_arn("x").last_status("PENDING").build());
|
||||
assert!(!bare.exec_agent_running);
|
||||
assert_eq!(bare.runtime_id, None);
|
||||
assert_eq!(bare.exit_code, None);
|
||||
assert_eq!(bare.owner, None);
|
||||
}
|
||||
@@ -32,6 +32,7 @@ use serde::Serialize;
|
||||
use tauri::{AppHandle, Emitter};
|
||||
use tokio::sync::watch;
|
||||
|
||||
use crate::aws::tasks::StopKind;
|
||||
use crate::host::watch::ExitSource;
|
||||
use crate::host::ContainerTarget;
|
||||
use crate::models::{Project, ProjectStatus};
|
||||
@@ -68,6 +69,32 @@ pub enum ExitReason {
|
||||
pub struct ContainerExited<'a> {
|
||||
pub project: &'a Project,
|
||||
pub reason: ExitReason,
|
||||
/// AWS: why ECS says the task stopped (P8). `None` on Docker, or when
|
||||
/// the watch has no reason on record.
|
||||
pub stop: Option<StopNote>,
|
||||
}
|
||||
|
||||
/// Why an AWS task stopped, for the UI's toast.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||
pub struct StopNote {
|
||||
/// `spot`, `idle`, `failed` (the project goes to Error), `user` (someone
|
||||
/// stopped it: no toast) or `exited` (on its own).
|
||||
pub kind: &'static str,
|
||||
/// ECS's stopped reason, or a fixed text.
|
||||
pub message: Option<String>,
|
||||
}
|
||||
|
||||
impl StopNote {
|
||||
pub fn of(kind: &StopKind) -> StopNote {
|
||||
let (kind, message) = match kind {
|
||||
StopKind::Spot => ("spot", Some("Spot interruption".to_string())),
|
||||
StopKind::Idle => ("idle", None),
|
||||
StopKind::FailedToStart(r) => ("failed", Some(r.clone())),
|
||||
StopKind::UserInitiated => ("user", None),
|
||||
StopKind::Exited(r) => ("exited", r.clone()),
|
||||
};
|
||||
StopNote { kind, message }
|
||||
}
|
||||
}
|
||||
|
||||
fn project_busy(project_id: &str) -> bool {
|
||||
@@ -227,6 +254,19 @@ pub fn spawn(
|
||||
p.id,
|
||||
reason
|
||||
);
|
||||
// Why ECS stopped it, read before the tunnel (and with it the
|
||||
// registry's record) goes.
|
||||
let kind = p
|
||||
.is_aws()
|
||||
.then(|| p.container_id.as_deref().and_then(crate::host::tunnels::stop_kind))
|
||||
.flatten();
|
||||
let mut shown = p.clone();
|
||||
// A task that never got going is an error, not a stop (P8).
|
||||
if let Some(StopKind::FailedToStart(_)) = kind {
|
||||
if store_for_aws.update_status(&p.id, ProjectStatus::Error).is_ok() {
|
||||
shown.status = ProjectStatus::Error;
|
||||
}
|
||||
}
|
||||
// An AWS task that stopped: its tunnel leads nowhere now, and its
|
||||
// task ARN is history (the data's access point stays).
|
||||
if p.is_aws() {
|
||||
@@ -245,7 +285,11 @@ pub fn spawn(
|
||||
});
|
||||
}
|
||||
}
|
||||
let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: p, reason });
|
||||
if let Some(store_p) = store_for_aws.get(&p.id) {
|
||||
shown = store_p;
|
||||
}
|
||||
let stop = kind.as_ref().map(StopNote::of);
|
||||
let _ = app.emit(CONTAINER_EXITED_EVENT, ContainerExited { project: &shown, reason, stop });
|
||||
})
|
||||
.await;
|
||||
})
|
||||
@@ -266,15 +310,38 @@ mod tests {
|
||||
#[test]
|
||||
fn the_event_says_why_so_the_ui_can_tell_news_from_housekeeping() {
|
||||
let p = Project::new("demo".to_string(), Vec::new());
|
||||
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited }).unwrap();
|
||||
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::Exited, stop: None })
|
||||
.unwrap();
|
||||
assert_eq!(v["reason"], "exited");
|
||||
assert_eq!(v["project"]["name"], "demo");
|
||||
let v = serde_json::to_value(ContainerExited { project: &p, reason: ExitReason::FoundStopped })
|
||||
.unwrap();
|
||||
assert!(v["stop"].is_null());
|
||||
let v = serde_json::to_value(ContainerExited {
|
||||
project: &p,
|
||||
reason: ExitReason::FoundStopped,
|
||||
stop: Some(StopNote::of(&StopKind::Spot)),
|
||||
})
|
||||
.unwrap();
|
||||
assert_eq!(v["reason"], "found_stopped");
|
||||
assert_eq!(v["stop"], serde_json::json!({ "kind": "spot", "message": "Spot interruption" }));
|
||||
assert_eq!(CONTAINER_EXITED_EVENT, "project-container-exited");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_stop_kind_has_a_note_for_the_ui() {
|
||||
let cases = [
|
||||
(StopKind::Spot, "spot", Some("Spot interruption")),
|
||||
(StopKind::Idle, "idle", None),
|
||||
(StopKind::FailedToStart("CannotPull".into()), "failed", Some("CannotPull")),
|
||||
(StopKind::UserInitiated, "user", None),
|
||||
(StopKind::Exited(Some("OOM".into())), "exited", Some("OOM")),
|
||||
(StopKind::Exited(None), "exited", None),
|
||||
];
|
||||
for (k, kind, message) in cases {
|
||||
let n = StopNote::of(&k);
|
||||
assert_eq!((n.kind, n.message.as_deref()), (kind, message), "{k:?}");
|
||||
}
|
||||
}
|
||||
|
||||
/// The shutdown path: a host call that never answers (a wedged daemon)
|
||||
/// must give way to cancellation at once, not hold teardown.
|
||||
#[tokio::test]
|
||||
|
||||
@@ -698,20 +698,20 @@ fn is_idle(args: &[&str]) -> bool {
|
||||
|
||||
#[test]
|
||||
fn is_idle_needs_every_condition() {
|
||||
// now last execs tunnels claude_running jobs_running minutes
|
||||
assert!(is_idle(&["10000", "9400", "0", "0", "0", "0", "10"]), "exactly N minutes");
|
||||
assert!(is_idle(&["10000", "1000", "0", "0", "0", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "9401", "0", "0", "0", "0", "10"]), "one second short");
|
||||
assert!(!is_idle(&["10000", "1000", "1", "0", "0", "0", "10"]), "an exec is open");
|
||||
assert!(!is_idle(&["10000", "1000", "0", "2", "0", "0", "10"]), "a tunnel is open");
|
||||
assert!(!is_idle(&["10000", "1000", "0", "0", "1", "0", "10"]), "claude is running");
|
||||
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "1", "10"]), "a scheduler job runs");
|
||||
// now, last activity, claude running, jobs running, minutes.
|
||||
assert!(is_idle(&["10000", "9400", "0", "0", "10"]), "exactly N minutes");
|
||||
assert!(is_idle(&["10000", "1000", "0", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "9401", "0", "0", "10"]), "one second short");
|
||||
assert!(!is_idle(&["10000", "1000", "1", "0", "10"]), "claude is running");
|
||||
assert!(!is_idle(&["10000", "1000", "0", "1", "10"]), "a scheduler job runs");
|
||||
// Anything unreadable is "not idle".
|
||||
assert!(!is_idle(&["10000", "", "0", "0", "0", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "x", "0", "0", "0", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "1000", "", "0", "0", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", "0"]), "0 minutes is off");
|
||||
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", ""]));
|
||||
assert!(!is_idle(&["10000", "", "0", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "x", "0", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "1000", "", "0", "10"]));
|
||||
assert!(!is_idle(&["10000", "1000", "0", "0", "0"]), "0 minutes is off");
|
||||
assert!(!is_idle(&["10000", "1000", "0", "0", ""]));
|
||||
// The old arity (with exec/tunnel counters) is refused, not misread.
|
||||
assert!(!is_idle(&["10000", "1000", "0", "0", "0", "0", "10"]));
|
||||
}
|
||||
|
||||
struct Watch {
|
||||
@@ -723,6 +723,18 @@ impl Watch {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
write_script(tmp.path(), "triple-c-idle-watch", IDLE_WATCH);
|
||||
write_script(tmp.path(), "triple-c-scheduler", SCHEDULER);
|
||||
// A stand-in WIP push: `--dry-run` prints the file `dry` (nothing:
|
||||
// clean repos), a push records its arguments in `pushes` and exits
|
||||
// with the number in `push_rc` (0 when absent).
|
||||
let d = tmp.path().display();
|
||||
write_script(
|
||||
tmp.path(),
|
||||
"triple-c-wip-push",
|
||||
&format!(
|
||||
"#!/bin/bash\nif [ \"$1\" = --dry-run ]; then cat '{d}/dry' 2>/dev/null; exit 0; fi\n\
|
||||
echo \"$*\" >> '{d}/pushes'\nexit \"$(cat '{d}/push_rc' 2>/dev/null || echo 0)\"\n"
|
||||
),
|
||||
);
|
||||
fs::write(tmp.path().join("token"), "test-token-not-real").unwrap();
|
||||
fs::create_dir_all(tmp.path().join("home")).unwrap();
|
||||
Watch { tmp }
|
||||
@@ -743,10 +755,56 @@ impl Watch {
|
||||
.env("TRIPLEC_STOP_REASON_FILE", p.join("reason"))
|
||||
.env("TRIPLEC_SCHEDULER_BIN", p.join("triple-c-scheduler"))
|
||||
.env("TRIPLEC_SCHEDULER_USER", "")
|
||||
.env("TRIPLEC_WIP_PUSH_BIN", p.join("triple-c-wip-push"))
|
||||
// The test host may well be running a `claude` of its own.
|
||||
.env("TRIPLEC_IDLE_PROC_NAME", "triple-c-no-such-process");
|
||||
c
|
||||
}
|
||||
|
||||
fn dirty(&self) {
|
||||
fs::write(
|
||||
self.tmp.path().join("dry"),
|
||||
"{\"folder\":\"api\",\"dirty\":true,\"unpushed\":0,\"branch\":\"main\"}\n\
|
||||
{\"folder\":\"web\",\"dirty\":false,\"unpushed\":0,\"branch\":\"main\"}\n",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
fn pushes(&self) -> String {
|
||||
fs::read_to_string(self.tmp.path().join("pushes")).unwrap_or_default()
|
||||
}
|
||||
}
|
||||
|
||||
/// Stands in for the entrypoint: records the TERM it gets in `got`.
|
||||
fn fake_entrypoint(dir: &Path) -> std::process::Child {
|
||||
Command::new("bash")
|
||||
.arg("-c")
|
||||
.arg("trap 'echo term > \"$1\"; exit 0' TERM; while :; do sleep 0.1; done")
|
||||
.arg("t")
|
||||
.arg(dir.join("got"))
|
||||
.spawn()
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
/// Run the watcher loop for at most `secs`; `None` when it was still running
|
||||
/// (it had not signalled) and was killed. Never hangs a test run.
|
||||
fn run_watch_for(mut cmd: Command, secs: u64) -> Option<Output> {
|
||||
let mut child = cmd
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(secs);
|
||||
while std::time::Instant::now() < deadline {
|
||||
if child.try_wait().unwrap().is_some() {
|
||||
return Some(child.wait_with_output().unwrap());
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
}
|
||||
child.kill().ok();
|
||||
let out = child.wait_with_output().unwrap();
|
||||
eprintln!("watcher (killed): {}", text(&out));
|
||||
None
|
||||
}
|
||||
|
||||
/// A one-shot fake agent: answers `/v1/activity` with `body` when the bearer
|
||||
@@ -782,10 +840,12 @@ s.serve_forever()
|
||||
(format!("http://127.0.0.1:{}", line.trim()), child)
|
||||
}
|
||||
|
||||
const OLD_ACTIVITY: &str = r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#;
|
||||
|
||||
#[test]
|
||||
fn idle_watch_check_counts_an_unfinished_clone_as_busy() {
|
||||
let w = Watch::new();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let clone = w.tmp.path().join("clone");
|
||||
fs::create_dir_all(&clone).unwrap();
|
||||
let busy = w.cmd(&url, 0).env("TRIPLEC_CLONE_STATUS_DIR", &clone).arg("--check").output().unwrap();
|
||||
@@ -806,40 +866,115 @@ fn idle_watch_check_treats_an_unreachable_agent_as_busy() {
|
||||
#[test]
|
||||
fn idle_watch_check_reads_activity_with_the_token() {
|
||||
let w = Watch::new();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let idle = w.cmd(&url, 0).arg("--check").output().unwrap();
|
||||
let (busy_url, mut busy_agent) =
|
||||
fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":1,"active_tunnels":0}"#);
|
||||
let recent = format!(
|
||||
r#"{{"last_activity_unix":{},"active_execs":0,"active_tunnels":0}}"#,
|
||||
std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_secs()
|
||||
);
|
||||
let (busy_url, mut busy_agent) = fake_agent(w.tmp.path(), &recent);
|
||||
let busy = w.cmd(&busy_url, 0).arg("--check").output().unwrap();
|
||||
fs::write(w.tmp.path().join("token"), "wrong").unwrap();
|
||||
let unauth = w.cmd(&url, 0).arg("--check").output().unwrap();
|
||||
agent.kill().ok();
|
||||
busy_agent.kill().ok();
|
||||
assert_eq!(idle.status.code(), Some(0), "{}", text(&idle));
|
||||
assert_eq!(busy.status.code(), Some(1), "{}", text(&busy));
|
||||
assert_eq!(busy.status.code(), Some(1), "recent activity: {}", text(&busy));
|
||||
assert_eq!(unauth.status.code(), Some(1), "a 401 is not idle: {}", text(&unauth));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idle_watch_ignores_open_but_silent_execs_and_tunnels() {
|
||||
// R11: an open terminal tab (an attach) or a tunnel no longer keeps the
|
||||
// workspace up; only their traffic moves the clock.
|
||||
let w = Watch::new();
|
||||
let (url, mut agent) =
|
||||
fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":3,"active_tunnels":2}"#);
|
||||
let out = w.cmd(&url, 0).arg("--check").output().unwrap();
|
||||
agent.kill().ok();
|
||||
assert_eq!(out.status.code(), Some(0), "{}", text(&out));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idle_watch_check_with_unsaved_work_depends_on_the_wip_push() {
|
||||
let w = Watch::new();
|
||||
w.dirty();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let off = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "0").arg("--check").output().unwrap();
|
||||
let on = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "1").arg("--check").output().unwrap();
|
||||
// A dry run that fails is a doubt: not idle.
|
||||
write_script(w.tmp.path(), "triple-c-wip-push", "#!/bin/bash\nexit 1\n");
|
||||
let unknown = w.cmd(&url, 0).env("TRIPLEC_WIP_PUSH", "1").arg("--check").output().unwrap();
|
||||
agent.kill().ok();
|
||||
assert_eq!(off.status.code(), Some(1), "unsaved work, WIP push off: {}", text(&off));
|
||||
assert_eq!(on.status.code(), Some(0), "the WIP push would take it: {}", text(&on));
|
||||
assert_eq!(unknown.status.code(), Some(1), "{}", text(&unknown));
|
||||
assert_eq!(w.pushes(), "", "--check never pushes");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idle_watch_signals_the_entrypoint_with_reason_idle() {
|
||||
let w = Watch::new();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
|
||||
// Stands in for the entrypoint: records the TERM it gets.
|
||||
let target = Command::new("bash")
|
||||
.arg("-c")
|
||||
.arg("trap 'echo term > \"$1\"; exit 0' TERM; while :; do sleep 0.1; done")
|
||||
.arg("t")
|
||||
.arg(w.tmp.path().join("got"))
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let mut target = target;
|
||||
let out = w.cmd(&url, target.id()).output().unwrap();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let mut target = fake_entrypoint(w.tmp.path());
|
||||
let out = run_watch_for(w.cmd(&url, target.id()), 20).expect("it signals");
|
||||
let status = target.wait().unwrap();
|
||||
agent.kill().ok();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(status.success());
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("got")).unwrap().trim(), "term");
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "idle");
|
||||
assert_eq!(w.pushes(), "", "nothing unsaved, nothing pushed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idle_watch_with_unsaved_work_and_the_wip_push_off_stays_up() {
|
||||
let w = Watch::new();
|
||||
w.dirty();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let mut target = fake_entrypoint(w.tmp.path());
|
||||
let mut cmd = w.cmd(&url, target.id());
|
||||
cmd.env("TRIPLEC_WIP_PUSH", "0");
|
||||
let ran = run_watch_for(cmd, 4);
|
||||
target.kill().ok();
|
||||
target.wait().ok();
|
||||
agent.kill().ok();
|
||||
assert!(ran.is_none(), "it must not stop: {:?}", ran.map(|o| text(&o)));
|
||||
assert!(!w.tmp.path().join("got").exists(), "no TERM");
|
||||
assert!(!w.tmp.path().join("reason").exists(), "no marker");
|
||||
assert_eq!(w.pushes(), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idle_watch_pushes_before_an_idle_stop_and_stays_up_while_the_push_fails() {
|
||||
// The push fails: the workspace stays up and retries, backing off.
|
||||
let w = Watch::new();
|
||||
w.dirty();
|
||||
fs::write(w.tmp.path().join("push_rc"), "1").unwrap();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let mut target = fake_entrypoint(w.tmp.path());
|
||||
let mut cmd = w.cmd(&url, target.id());
|
||||
cmd.env("TRIPLEC_WIP_PUSH", "1").env("TRIPLEC_WIP_BACKOFF_MAX", "2");
|
||||
let ran = run_watch_for(cmd, 6);
|
||||
assert!(ran.is_none(), "a failed push must not stop the workspace");
|
||||
assert!(!w.tmp.path().join("got").exists(), "no TERM");
|
||||
assert!(!w.tmp.path().join("reason").exists(), "no marker");
|
||||
let tries = w.pushes().lines().count();
|
||||
assert!((1..=3).contains(&tries), "retried with a back-off, not every second: {tries}");
|
||||
assert!(w.pushes().lines().all(|l| l == "--reason idle"), "{}", w.pushes());
|
||||
|
||||
// Now the push works: it pushes once, then signals with the idle marker.
|
||||
fs::write(w.tmp.path().join("push_rc"), "0").unwrap();
|
||||
fs::remove_file(w.tmp.path().join("pushes")).unwrap();
|
||||
let mut cmd = w.cmd(&url, target.id());
|
||||
cmd.env("TRIPLEC_WIP_PUSH", "1");
|
||||
let out = run_watch_for(cmd, 20).expect("it stops once the push succeeded");
|
||||
target.wait().unwrap();
|
||||
agent.kill().ok();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(w.pushes(), "--reason idle\n");
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("got")).unwrap().trim(), "term");
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "idle");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -847,13 +982,13 @@ fn idle_watch_keeps_a_stop_guard_marker_already_recorded() {
|
||||
for marker in ["pushed", "discard"] {
|
||||
let w = Watch::new();
|
||||
fs::write(w.tmp.path().join("reason"), format!("{marker}\n")).unwrap();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), r#"{"last_activity_unix":1,"active_execs":0,"active_tunnels":0}"#);
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let mut target = Command::new("bash")
|
||||
.arg("-c")
|
||||
.arg("trap 'exit 0' TERM; while :; do sleep 0.1; done")
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let out = w.cmd(&url, target.id()).output().unwrap();
|
||||
let out = run_watch_for(w.cmd(&url, target.id()), 20).expect("it signals");
|
||||
target.wait().unwrap();
|
||||
agent.kill().ok();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
@@ -864,7 +999,7 @@ fn idle_watch_keeps_a_stop_guard_marker_already_recorded() {
|
||||
// ------------------------------------------------------- stop (SIGTERM) path
|
||||
|
||||
#[test]
|
||||
fn aws_stop_runs_the_wip_push_with_the_recorded_reason_once() {
|
||||
fn aws_stop_pushes_on_sigterm_once_and_leaves_an_idle_stop_to_the_watcher() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let log = tmp.path().join("calls");
|
||||
let fake = write_script(
|
||||
@@ -882,13 +1017,18 @@ fn aws_stop_runs_the_wip_push_with_the_recorded_reason_once() {
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(fs::read_to_string(&log).unwrap(), "--reason sigterm\n");
|
||||
|
||||
// An idle stop: the watcher already pushed, so the trap does not, and it
|
||||
// exits with the idle status the app reads from ECS.
|
||||
fs::remove_file(&log).unwrap();
|
||||
fs::write(&reason, "idle\n").unwrap();
|
||||
let out = bash(&script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "1")], &[]);
|
||||
let idle_script = format!("{blocks}\naws_before_stop\necho \"status=${{triplec_exit_status:-0}}\"\n");
|
||||
let out = bash(&idle_script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "1")], &[]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(fs::read_to_string(&log).unwrap(), "--reason idle\n");
|
||||
assert!(!log.exists(), "no second push after the watcher's");
|
||||
assert!(text(&out).contains("entrypoint: stopping (idle)"), "{}", text(&out));
|
||||
assert!(text(&out).contains("status=75"), "{}", text(&out));
|
||||
fs::remove_file(&reason).unwrap();
|
||||
|
||||
fs::remove_file(&log).unwrap();
|
||||
let out = bash(&script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "0")], &[]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(!log.exists(), "WIP push off");
|
||||
@@ -922,6 +1062,32 @@ fn aws_stop_skips_its_push_when_the_app_already_pushed_or_the_user_declined() {
|
||||
assert_eq!(fs::read_to_string(&log).unwrap(), "--reason sigterm\n");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_idle_exit_status_is_the_one_the_app_reads() {
|
||||
assert!(
|
||||
block(ENTRYPOINT, "aws_stop")
|
||||
.contains(&format!("triplec_idle_exit={}\n", crate::aws::tasks::IDLE_EXIT_CODE)),
|
||||
"entrypoint.sh and aws::tasks::IDLE_EXIT_CODE disagree"
|
||||
);
|
||||
assert!(ENTRYPOINT.contains("exit \"${triplec_exit_status:-0}\""), "stop_keepalive exits with it");
|
||||
// The watcher gets the WIP setting through its scrubbed environment.
|
||||
assert!(block(ENTRYPOINT, "aws_stop").contains("TRIPLEC_WIP_PUSH=\"${TRIPLEC_WIP_PUSH:-0}\""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn start_clone_repos_marks_done_however_the_clone_ends() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let status = tmp.path().join("status");
|
||||
// clone_repos replaced by one that fails half-way.
|
||||
let script = format!(
|
||||
"{}\nclone_repos() {{ exit 3; }}\nstart_clone_repos '[x]' \"$1\"\nwait\n",
|
||||
aws_blocks(&["aws_env", "aws_repos"])
|
||||
);
|
||||
let out = bash(&script, &[&status], &[], &[]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(status.join("done").exists(), "a failed clone still ends: {}", text(&out));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_mode_never_reaches_the_aws_blocks() {
|
||||
// Every new step sits behind TRIPLEC_HOST=aws; the scripts are only
|
||||
|
||||
@@ -35,7 +35,7 @@ use crate::aws::runtime::{
|
||||
};
|
||||
use crate::aws::tasks::{
|
||||
find_or_run, find_project_task, stopped_among, task_log_stream, wait_ready, SdkTaskApi,
|
||||
TaskApi, TaskInfo, TaskLogs, WaitTiming, LOG_TAIL_LINES, STOP_REASON,
|
||||
StopKind, TaskApi, TaskInfo, TaskLogs, WaitTiming, LOG_TAIL_LINES, STOP_REASON,
|
||||
};
|
||||
use crate::aws::tofu_modules::{
|
||||
environment_cluster_name, log_stream_prefix, owner_key, task_family,
|
||||
@@ -75,6 +75,11 @@ const HEALTH_TIMEOUT: Duration = Duration::from_secs(30);
|
||||
const HEALTH_INTERVAL: Duration = Duration::from_secs(1);
|
||||
/// How long a terminal waits for a background re-attach.
|
||||
pub const REATTACH_WAIT: Duration = Duration::from_secs(90);
|
||||
/// The most one AWS death-watch poll of a group may take.
|
||||
const WATCH_CALL_TIMEOUT: Duration = Duration::from_secs(60);
|
||||
/// The most a StopTask (credentials included) may take before the stop is
|
||||
/// reported as failed; the SDK's own bound is per call.
|
||||
const STOP_TIMEOUT: Duration = Duration::from_secs(45);
|
||||
|
||||
fn s<E: std::fmt::Display>(e: E) -> String {
|
||||
e.to_string()
|
||||
@@ -730,7 +735,18 @@ pub async fn task_log(project: &Project, settings: &AppSettings, limit: i32) ->
|
||||
pub async fn stop_task_arn<T: TaskApi>(api: &T, task_arn: &str) -> Result<(), String> {
|
||||
let cluster = cluster_of_task(task_arn)
|
||||
.ok_or_else(|| format!("{task_arn} is not a task ARN with a cluster"))?;
|
||||
api.stop_task(cluster, task_arn, STOP_REASON).await.map_err(s)
|
||||
bounded_stop(api.stop_task(cluster, task_arn, STOP_REASON)).await
|
||||
}
|
||||
|
||||
/// A StopTask, bounded by [`STOP_TIMEOUT`]: a stop that cannot reach AWS
|
||||
/// fails (and the project keeps its task ARN) instead of hanging the UI.
|
||||
async fn bounded_stop(
|
||||
call: impl std::future::Future<Output = Result<(), crate::aws::AwsError>>,
|
||||
) -> Result<(), String> {
|
||||
tokio::time::timeout(STOP_TIMEOUT, call)
|
||||
.await
|
||||
.map_err(|_| format!("StopTask did not answer within {STOP_TIMEOUT:?}; try again"))?
|
||||
.map_err(s)
|
||||
}
|
||||
|
||||
/// `ProjectHost::stop` for an agent target: the task behind a registered
|
||||
@@ -738,10 +754,7 @@ pub async fn stop_task_arn<T: TaskApi>(api: &T, task_arn: &str) -> Result<(), St
|
||||
pub async fn stop_endpoint(ep: &AgentEndpoint) -> Option<Result<(), String>> {
|
||||
let (_connection, _region, cluster, arn) = tunnels::task_of(ep)?;
|
||||
let (project_id, sdk) = tunnels::owner_of(ep)?;
|
||||
let res = SdkTaskApi::new(&sdk)
|
||||
.stop_task(&cluster, &arn, STOP_REASON)
|
||||
.await
|
||||
.map_err(s);
|
||||
let res = bounded_stop(SdkTaskApi::new(&sdk).stop_task(&cluster, &arn, STOP_REASON)).await;
|
||||
if res.is_ok() {
|
||||
tunnels::close(&project_id).await;
|
||||
}
|
||||
@@ -1089,22 +1102,29 @@ impl AwsTaskLookup for StoreTaskLookup {
|
||||
Box::pin(async move {
|
||||
let settings = self.settings.get();
|
||||
let conn = crate::commands::host_commands::aws_conn(&group.connection_id, &settings)?;
|
||||
let sdk = sdk_config(&conn, &group.region).await.map_err(s)?;
|
||||
let stopped = stopped_among(&SdkTaskApi::new(&sdk), &group.cluster_arn, arns)
|
||||
.await
|
||||
.map_err(s)?;
|
||||
// The SDK bounds each call; this bounds the whole poll of a group
|
||||
// (credentials included, which an SSO refresh can stall), so one
|
||||
// stuck group never stalls the watch of every other.
|
||||
let stopped = tokio::time::timeout(WATCH_CALL_TIMEOUT, async {
|
||||
let sdk = sdk_config(&conn, &group.region).await.map_err(s)?;
|
||||
stopped_among(&SdkTaskApi::new(&sdk), &group.cluster_arn, arns)
|
||||
.await
|
||||
.map_err(s)
|
||||
})
|
||||
.await
|
||||
.map_err(|_| format!("ECS did not answer within {WATCH_CALL_TIMEOUT:?}"))??;
|
||||
record_stopped(&self.projects.list(), &stopped);
|
||||
Ok(stopped)
|
||||
Ok(stopped.into_iter().map(|(arn, _)| arn).collect())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Tell the registry which targets' tasks ECS reports stopped, so
|
||||
/// `has_exited` confirms them (R3).
|
||||
pub fn record_stopped(projects: &[Project], stopped: &[String]) {
|
||||
/// Tell the registry which targets' tasks ECS reports stopped, and why, so
|
||||
/// `has_exited` confirms them (R3) and the death report can say why (P8).
|
||||
pub fn record_stopped(projects: &[Project], stopped: &[(String, StopKind)]) {
|
||||
for w in projects.iter().filter_map(watched_task) {
|
||||
if stopped.contains(&w.task_arn) {
|
||||
tunnels::mark_stopped(w.target.id());
|
||||
if let Some((_, why)) = stopped.iter().find(|(arn, _)| *arn == w.task_arn) {
|
||||
tunnels::mark_stopped(w.target.id(), why.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,6 +57,7 @@ fn task(p: &Project, id: &str, last: &str, ready: bool) -> TaskInfo {
|
||||
task_definition_arn: format!("arn:aws:ecs:{REGION}:{ACCOUNT}:task-definition/{}:1", family(p)),
|
||||
runtime_id: ready.then(|| "rt-1".into()),
|
||||
exec_agent_running: ready,
|
||||
exit_code: None,
|
||||
owner: Some(USER.into()),
|
||||
project: Some(p.id.clone()),
|
||||
}
|
||||
@@ -572,8 +573,9 @@ async fn the_watch_reports_a_running_aws_project_as_its_stored_target_and_confir
|
||||
assert_eq!(w.group.cluster_arn, cluster_arn());
|
||||
assert_eq!(w.group.connection_id, "c1");
|
||||
assert!(!w.target.has_exited().await, "nothing recorded yet");
|
||||
record_stopped(std::slice::from_ref(&p), &[task_arn("w1")]);
|
||||
record_stopped(std::slice::from_ref(&p), &[(task_arn("w1"), StopKind::Idle)]);
|
||||
assert!(w.target.has_exited().await);
|
||||
assert_eq!(crate::host::tunnels::stop_kind(w.target.id()), Some(StopKind::Idle));
|
||||
|
||||
let mut stopped = p.clone();
|
||||
stopped.status = ProjectStatus::Stopped;
|
||||
|
||||
@@ -20,6 +20,7 @@ use futures_util::future::BoxFuture;
|
||||
use tokio::sync::watch;
|
||||
|
||||
use super::agent::AgentEndpoint;
|
||||
use crate::aws::tasks::StopKind;
|
||||
use super::ssm::{EcsTask, PortForward};
|
||||
|
||||
/// Something that owns a tunnel's resources: the real [`PortForward`], or a
|
||||
@@ -58,8 +59,9 @@ pub type TaskOf = (String, String, String, String);
|
||||
#[derive(Default)]
|
||||
struct Registry {
|
||||
live: HashMap<String, LiveTunnel>,
|
||||
/// Target ids (endpoint base URLs) whose task the AWS watch saw STOPPED.
|
||||
stopped: HashSet<String>,
|
||||
/// Target ids (endpoint base URLs) whose task the AWS watch saw STOPPED,
|
||||
/// and why.
|
||||
stopped: HashMap<String, StopKind>,
|
||||
/// Projects whose tunnel is being re-opened in the background.
|
||||
pending: HashSet<String>,
|
||||
/// Bumped by every [`close`] (a stop or remove), every [`register`] (a
|
||||
@@ -245,14 +247,19 @@ pub async fn close_all() {
|
||||
futures_util::future::join_all(all.into_iter().map(|t| t.forward.close())).await;
|
||||
}
|
||||
|
||||
/// The AWS watch saw the task behind target `target_id` STOPPED.
|
||||
pub fn mark_stopped(target_id: &str) {
|
||||
lock().stopped.insert(target_id.to_string());
|
||||
/// The AWS watch saw the task behind target `target_id` STOPPED, for `why`.
|
||||
pub fn mark_stopped(target_id: &str, why: StopKind) {
|
||||
lock().stopped.insert(target_id.to_string(), why);
|
||||
}
|
||||
|
||||
/// Whether the AWS watch has confirmed the task behind `target_id` stopped.
|
||||
pub fn is_stopped(target_id: &str) -> bool {
|
||||
lock().stopped.contains(target_id)
|
||||
lock().stopped.contains_key(target_id)
|
||||
}
|
||||
|
||||
/// Why the task behind `target_id` stopped, once the watch has seen it.
|
||||
pub fn stop_kind(target_id: &str) -> Option<StopKind> {
|
||||
lock().stopped.get(target_id).cloned()
|
||||
}
|
||||
|
||||
pub fn is_pending(project_id: &str) -> bool {
|
||||
@@ -341,7 +348,8 @@ pub(crate) mod tests {
|
||||
let p = pid("rereg");
|
||||
register(&p, fake_tunnel(61002, closes.clone()));
|
||||
let id = AgentEndpoint::loopback(61003, "tok").base_url.to_string();
|
||||
mark_stopped(&id);
|
||||
mark_stopped(&id, StopKind::Spot);
|
||||
assert_eq!(stop_kind(&id), Some(StopKind::Spot));
|
||||
let old = register(&p, fake_tunnel(61003, closes.clone())).unwrap();
|
||||
assert_eq!(old.endpoint.base_url.port(), Some(61002));
|
||||
assert!(!is_stopped(&id), "a new task on a reused port is not stopped");
|
||||
|
||||
@@ -54,6 +54,35 @@ describe("useProjectStatusEvents", () => {
|
||||
expect(toasts).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("says why an AWS task stopped, and keeps a user's stop quiet", async () => {
|
||||
renderHook(() => useProjectStatusEvents());
|
||||
await vi.waitFor(() => expect(handler).not.toBeNull());
|
||||
const fire = (stop: unknown, reason = "exited") => {
|
||||
useAppState.setState({ toasts: [] });
|
||||
handler!({ payload: { project: project({ status: "stopped" }), reason, stop } });
|
||||
return useAppState.getState().toasts;
|
||||
};
|
||||
|
||||
let t = fire({ kind: "spot", message: "Spot interruption" });
|
||||
expect(t[0].kind).toBe("info");
|
||||
expect(t[0].message).toContain("Spot interruption");
|
||||
|
||||
t = fire({ kind: "idle", message: null });
|
||||
expect(t[0].kind).toBe("info");
|
||||
expect(t[0].message).toContain("idle");
|
||||
|
||||
expect(fire({ kind: "user", message: null })).toHaveLength(0);
|
||||
|
||||
// A task that never started is news even when found at startup.
|
||||
t = fire({ kind: "failed", message: "CannotPullContainerError: denied" }, "found_stopped");
|
||||
expect(t[0].kind).toBe("error");
|
||||
expect(t[0].detail).toContain("CannotPullContainerError");
|
||||
|
||||
t = fire({ kind: "exited", message: "OutOfMemoryError" });
|
||||
expect(t[0].kind).toBe("error");
|
||||
expect(t[0].detail).toContain("OutOfMemoryError");
|
||||
});
|
||||
|
||||
it("stops listening on unmount", async () => {
|
||||
const { unmount } = renderHook(() => useProjectStatusEvents());
|
||||
await vi.waitFor(() => expect(handler).not.toBeNull());
|
||||
|
||||
@@ -12,6 +12,56 @@ export interface ContainerExitedEvent {
|
||||
* while the app was closed, e.g. a host reboot); nothing was interrupted.
|
||||
*/
|
||||
reason: "exited" | "found_stopped";
|
||||
/** AWS: why ECS says the task stopped (`StopNote`); absent on Docker. */
|
||||
stop?: StopNote | null;
|
||||
}
|
||||
|
||||
/** `container_watch.rs`'s `StopNote`. */
|
||||
export interface StopNote {
|
||||
/** `user`: someone stopped it (console, CLI, another machine) — no toast. */
|
||||
kind: "spot" | "idle" | "failed" | "user" | "exited";
|
||||
message: string | null;
|
||||
}
|
||||
|
||||
/** The toast for an exit, or null for none. */
|
||||
export function exitToast(
|
||||
name: string,
|
||||
reason: ContainerExitedEvent["reason"],
|
||||
stop: StopNote | null | undefined,
|
||||
): { kind: "error" | "info"; message: string; detail?: string } | null {
|
||||
if (stop?.kind === "user") return null;
|
||||
if (stop?.kind === "failed") {
|
||||
return {
|
||||
kind: "error",
|
||||
message: `The AWS workspace for “${name}” failed to start`,
|
||||
detail: stop.message ?? undefined,
|
||||
};
|
||||
}
|
||||
if (reason !== "exited") return null;
|
||||
switch (stop?.kind) {
|
||||
case "spot":
|
||||
return {
|
||||
kind: "info",
|
||||
message: `“${name}” stopped: Spot interruption`,
|
||||
detail:
|
||||
"AWS reclaimed the Spot capacity it ran on. Start the project again to get a new workspace.",
|
||||
};
|
||||
case "idle":
|
||||
return {
|
||||
kind: "info",
|
||||
message: `“${name}” stopped after being idle`,
|
||||
detail: "Its AWS workspace stopped itself to save cost. Start the project again to continue.",
|
||||
};
|
||||
default:
|
||||
return {
|
||||
kind: "error",
|
||||
message: `The container for “${name}” stopped`,
|
||||
detail:
|
||||
"It exited without Triple-C stopping it — open sessions were disconnected. " +
|
||||
"Start the project again to reconnect." +
|
||||
(stop?.message ? ` (${stop.message})` : ""),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -25,18 +75,12 @@ export function useProjectStatusEvents() {
|
||||
let cancelled = false;
|
||||
let unlisten: UnlistenFn | null = null;
|
||||
listen<ContainerExitedEvent>("project-container-exited", (event) => {
|
||||
const { project, reason } = event.payload;
|
||||
const { project, reason, stop } = event.payload;
|
||||
const state = useAppState.getState();
|
||||
state.updateProjectInList(project);
|
||||
if (reason !== "exited") return;
|
||||
state.pushToast({
|
||||
kind: "error",
|
||||
message: `The container for “${project.name}” stopped`,
|
||||
detail:
|
||||
"It exited without Triple-C stopping it — open sessions were disconnected. " +
|
||||
"Start the project again to reconnect.",
|
||||
dedupeKey: `container-exited-${project.id}`,
|
||||
});
|
||||
const toast = exitToast(project.name, reason, stop);
|
||||
if (!toast) return;
|
||||
state.pushToast({ ...toast, dedupeKey: `container-exited-${project.id}` });
|
||||
})
|
||||
.then((fn) => {
|
||||
if (cancelled) fn();
|
||||
|
||||
@@ -349,7 +349,9 @@ updates use the existing "new image available" flow: the next start uses the new
|
||||
`git status --porcelain` and any unpushed commits in each repo. If anything would be lost, it
|
||||
offers: **Push to a `triple-c/wip/<date>` branch** (the preselected default), **Stop anyway**, or
|
||||
**Cancel**. The same check runs before idle auto-stop. If anything is dirty, idle auto-stop
|
||||
pushes a WIP branch instead of throwing work away.
|
||||
pushes a WIP branch instead of throwing work away, and stops only once that push succeeded (a
|
||||
failed push keeps the workspace up and is retried, backing off). With the WIP push off, unsaved
|
||||
work keeps the workspace up.
|
||||
- **Fargate task retirement and Spot interruption** send `SIGTERM` with a short deadline (2 minutes
|
||||
for Spot). The entrypoint's `SIGTERM` trap runs the same WIP push, best-effort, when the project
|
||||
has it enabled.
|
||||
@@ -497,9 +499,12 @@ way.
|
||||
- user-initiated → Stopped.
|
||||
|
||||
EventBridge → SQS for push notifications is a later improvement.
|
||||
- **Idle auto-stop** runs **inside** the task: the agent tracks activity (terminal I/O, agent
|
||||
calls, running `claude` processes and scheduler jobs) and stops the task by exiting the
|
||||
entrypoint. It doesn't depend on the app being open. Because a running scheduler job counts as
|
||||
- **Idle auto-stop** runs **inside** the task: the agent tracks activity (terminal I/O and agent
|
||||
calls; an open but silent terminal or tunnel is not activity, and the app's background polls
|
||||
are marked `X-Triple-C-Background` and never count — ruling R11), the watcher adds running
|
||||
`claude` processes, scheduler jobs and unsaved git work, and stops the task by exiting the
|
||||
entrypoint with status 75, which the app reads back from `DescribeTasks` to say "stopped after
|
||||
being idle". It doesn't depend on the app being open. Because a running scheduler job counts as
|
||||
activity, scheduled tasks keep a workspace alive. The admin's maximum lifetime is a hard cap
|
||||
either way.
|
||||
- **Health:** the task definition carries an ECS container health check against the agent, run
|
||||
|
||||
Reference in new issue
Block a user