feat(container): setup script, lifetime cap and one WIP branch per run
TRIPLEC_SETUP_SCRIPT runs as claude after the background clone (bash -e, stdin /dev/null, 30 min timeout) with status/log in /run/triple-c-setup; the idle watcher counts it as busy and enforces TRIPLEC_MAX_LIFETIME_MINUTES (best-effort WIP push, marker lifetime, exit 76 = StopKind::LifetimeCap). WIP pushes go to one triple-c/wip/<run-id> branch per task run, skipping an unchanged tree. New triple-c-workspace-status prints clone + setup JSON. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ad447555d9
commit
618cac1a49
14 files changed
+906
-53
No files matched your search
@@ -355,7 +355,7 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li
|
||||
through the agent, closes the project's terminals and auth bridge (so nothing is written after the
|
||||
push's snapshot), may push first (under an in-container `timeout`, since the agent does not kill an
|
||||
exec the client gave up on), then writes `pushed` or `discard` to `/run/triple-c/stop-reason`
|
||||
(root-owned; the entrypoint's TERM hook reads it, and the idle watcher writes `idle`) so the hook
|
||||
(root-owned; the entrypoint's TERM hook reads it, and the idle watcher writes `idle` or `lifetime`) so the hook
|
||||
doesn't push twice; the marker is cleared if StopTask fails. Claude Code's sandbox is forced off in
|
||||
the generated settings (bubblewrap can't create namespaces on Fargate, verified live).
|
||||
**Environments are OpenTofu modules** (`aws/terraform/{environment,user-onboard,permissions-boundary}`;
|
||||
@@ -405,13 +405,26 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li
|
||||
start, and `triple-c-wip-push` skips scratch.
|
||||
Switching EFS → Hybrid/Ephemeral keeps `workspace_access_point_id` (store-owned) until "Remove AWS
|
||||
resources" with discard deletes both access points. On
|
||||
TERM the hook pushes unsaved work to a `triple-c/wip/<stamp>` branch via a temporary index (the
|
||||
user's index is untouched; needs the task definition's `stopTimeout` raised, 120 s) unless the stop
|
||||
marker says the app handled it. The idle watcher runs as root under a supervisor and signals the
|
||||
TERM the hook pushes unsaved work to the run's `triple-c/wip/<run-id>` branch via a temporary index
|
||||
(the user's index is untouched; needs the task definition's `stopTimeout` raised, 120 s) unless the
|
||||
stop marker says the app handled it. **One WIP branch per task run:** `aws_prepare_run` writes
|
||||
`/run/triple-c/run-id` (`<UTC stamp>-<task id 8 | local>`, task id from ECS task metadata) and
|
||||
`start-epoch` at the very start; `/run/triple-c` is root 0711 (the stop guard keeps that mode — at
|
||||
0700 `claude` could not read the run id). Later pushes force (`+`) only that branch, and a tree equal
|
||||
to `/run/triple-c/wip-last/<folder>` (claude-owned dir, written after a successful push) is skipped;
|
||||
no usable run id (Docker) keeps the per-push stamp. **Setup script** (`TRIPLEC_SETUP_SCRIPT`, unset
|
||||
before the snapshot like `TRIPLEC_FILES`): written to `/run/triple-c-setup/script` (root 0755 dir;
|
||||
root opens `log` and hands the descriptor over, so claude writes nothing there), run as `claude`,
|
||||
`bash -e`, stdin `/dev/null`, `timeout -k 10 1800`, chained after the background clone's `done` in
|
||||
the same background subshell; `status` is `running` from the start until `ok` / `failed <code>`.
|
||||
`triple-c-workspace-status` (python, no secrets) prints `{clone:{done,repos},setup:{state,code,tail}}`.
|
||||
**Lifetime cap** (`TRIPLEC_MAX_LIFETIME_MINUTES`, advisory — sudo can kill it): the idle watcher
|
||||
(started when idle stop *or* the cap is set) pushes best effort, records `lifetime` and TERMs; the
|
||||
hook exits 76 (`aws::tasks::LIFETIME_EXIT_CODE` → `StopKind::LifetimeCap` → `lifetime` toast). The idle watcher runs as root under a supervisor and signals the
|
||||
entrypoint's own PID (not PID 1, which is docker-init), so idle and ECS stops share one trap.
|
||||
Idle = the agent's `last_activity_unix` older than N minutes (open-but-silent attaches/tunnels
|
||||
don't count; requests with `X-Triple-C-Background: 1` — the auth bridge's and the file viewer's polls — never do),
|
||||
no `claude`, no scheduler job, no running clone, and no unsaved git work: with the WIP push on the
|
||||
no `claude`, no scheduler job, no running clone or setup script, and no unsaved git work: with the WIP push on the
|
||||
watcher pushes *before* signalling and stays up if that fails; with it off, dirty repos keep the
|
||||
workspace up; activity during the watcher's push sends it back to watching. A repo the background
|
||||
clone is still writing (`/run/triple-c-clone/repos` last says `cloning`, no `done`) is skipped by
|
||||
|
||||
@@ -483,6 +483,11 @@ async fn stopped_error<A: TaskApi>(api: &A, logs: &TaskLogs, t: &TaskInfo) -> Aw
|
||||
/// so an idle stop (`EssentialContainerExited`) is not mistaken for a crash.
|
||||
pub const IDLE_EXIT_CODE: i32 = 75;
|
||||
|
||||
/// The entrypoint's exit status after the environment's lifetime cap stopped
|
||||
/// the workspace (`TRIPLEC_MAX_LIFETIME_MINUTES`, enforced by the in-container
|
||||
/// idle watcher — advisory, see the phase 6 constraints, Q8).
|
||||
pub const LIFETIME_EXIT_CODE: i32 = 76;
|
||||
|
||||
/// Why a task stopped, as the app reports it (spec, "Lifecycle and status").
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum StopKind {
|
||||
@@ -491,6 +496,8 @@ pub enum StopKind {
|
||||
Exited(Option<String>),
|
||||
/// The workspace's own idle stop ([`IDLE_EXIT_CODE`]).
|
||||
Idle,
|
||||
/// The environment's maximum workspace lifetime ([`LIFETIME_EXIT_CODE`]).
|
||||
LifetimeCap,
|
||||
/// A Fargate Spot interruption (P8).
|
||||
Spot,
|
||||
/// Never got going: the project goes to Error with this reason.
|
||||
@@ -510,6 +517,9 @@ pub fn classify_stop(info: &TaskInfo) -> StopKind {
|
||||
),
|
||||
Some("UserInitiated") => StopKind::UserInitiated,
|
||||
Some("EssentialContainerExited") if info.exit_code == Some(IDLE_EXIT_CODE) => StopKind::Idle,
|
||||
Some("EssentialContainerExited") if info.exit_code == Some(LIFETIME_EXIT_CODE) => {
|
||||
StopKind::LifetimeCap
|
||||
}
|
||||
_ => StopKind::Exited(info.stopped_reason.clone().filter(|r| !r.trim().is_empty())),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -652,6 +652,11 @@ fn classify_stop_follows_the_spec_table() {
|
||||
// The entrypoint's idle stop exits with its own status.
|
||||
let idle = TaskInfo { exit_code: Some(IDLE_EXIT_CODE), ..with(Some("EssentialContainerExited"), None) };
|
||||
assert_eq!(classify_stop(&idle), StopKind::Idle);
|
||||
// So does its lifetime-cap stop.
|
||||
let capped = TaskInfo { exit_code: Some(LIFETIME_EXIT_CODE), ..with(Some("EssentialContainerExited"), None) };
|
||||
assert_eq!(classify_stop(&capped), StopKind::LifetimeCap);
|
||||
let user_capped = TaskInfo { exit_code: Some(LIFETIME_EXIT_CODE), ..with(Some("UserInitiated"), None) };
|
||||
assert_eq!(classify_stop(&user_capped), StopKind::UserInitiated);
|
||||
let crashed = TaskInfo { exit_code: Some(137), ..with(Some("EssentialContainerExited"), None) };
|
||||
assert_eq!(classify_stop(&crashed), StopKind::Exited(None));
|
||||
// Only with that stop code: a user stop of a task that exits 75 is a user stop.
|
||||
|
||||
@@ -115,13 +115,15 @@ pub async fn forget_choice<R: WipRunner>(runner: Option<&R>, choice: Option<WipC
|
||||
}
|
||||
|
||||
/// The root command that records `marker`; the value travels as `$1`, never
|
||||
/// inside the script.
|
||||
/// inside the script. The directory stays 0711 (the entrypoint's mode): the
|
||||
/// `claude` user reads the run id and its WIP record in it by name (the
|
||||
/// marker itself is 0600).
|
||||
pub fn stop_reason_command(marker: &str) -> Vec<String> {
|
||||
vec![
|
||||
"sh".into(),
|
||||
"-c".into(),
|
||||
format!(
|
||||
"umask 077 && mkdir -p /run/triple-c && chmod 700 /run/triple-c && printf '%s\\n' \"$1\" > {STOP_REASON_FILE}"
|
||||
"umask 077 && mkdir -p /run/triple-c && chmod 711 /run/triple-c && printf '%s\\n' \"$1\" > {STOP_REASON_FILE}"
|
||||
),
|
||||
"sh".into(),
|
||||
marker.into(),
|
||||
|
||||
@@ -77,7 +77,7 @@ pub struct ContainerExited<'a> {
|
||||
/// Why an AWS task stopped, for the UI's toast.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||
pub struct StopNote {
|
||||
/// `spot`, `idle`, `failed` (the project goes to Error), `user` (someone
|
||||
/// `spot`, `idle`, `lifetime`, `failed` (the project goes to Error), `user` (someone
|
||||
/// stopped it: no toast) or `exited` (on its own).
|
||||
pub kind: &'static str,
|
||||
/// ECS's stopped reason, or a fixed text.
|
||||
@@ -89,6 +89,7 @@ impl StopNote {
|
||||
let (kind, message) = match kind {
|
||||
StopKind::Spot => ("spot", Some("Spot interruption".to_string())),
|
||||
StopKind::Idle => ("idle", None),
|
||||
StopKind::LifetimeCap => ("lifetime", None),
|
||||
StopKind::FailedToStart(r) => ("failed", Some(r.clone())),
|
||||
StopKind::UserInitiated => ("user", None),
|
||||
StopKind::Exited(r) => ("exited", r.clone()),
|
||||
@@ -332,6 +333,7 @@ mod tests {
|
||||
let cases = [
|
||||
(StopKind::Spot, "spot", Some("Spot interruption")),
|
||||
(StopKind::Idle, "idle", None),
|
||||
(StopKind::LifetimeCap, "lifetime", None),
|
||||
(StopKind::FailedToStart("CannotPull".into()), "failed", Some("CannotPull")),
|
||||
(StopKind::UserInitiated, "user", None),
|
||||
(StopKind::Exited(Some("OOM".into())), "exited", Some("OOM")),
|
||||
|
||||
@@ -13,6 +13,7 @@ const ENTRYPOINT: &str = include_str!("../../../../container/entrypoint.sh");
|
||||
const WIP_PUSH: &str = include_str!("../../../../container/triple-c-wip-push");
|
||||
const IDLE_WATCH: &str = include_str!("../../../../container/triple-c-idle-watch");
|
||||
const SCHEDULER: &str = include_str!("../../../../container/triple-c-scheduler");
|
||||
const WORKSPACE_STATUS: &str = include_str!("../../../../container/triple-c-workspace-status");
|
||||
|
||||
fn block<'a>(text: &'a str, name: &str) -> &'a str {
|
||||
let start = text
|
||||
@@ -101,7 +102,8 @@ fn files_script(tmp: &Path) -> String {
|
||||
"{blocks}\n{agent}\n\
|
||||
env -0 > \"$1/env\"\n\
|
||||
printf '%s\\0' \"${{agent_env[@]}}\" > \"$1/agent_env\"\n\
|
||||
printf '%s' \"${{triplec_repos_json:-}}\" > \"$1/repos\"\n"
|
||||
printf '%s' \"${{triplec_repos_json:-}}\" > \"$1/repos\"\n\
|
||||
printf '%s' \"${{triplec_setup_script:-}}\" > \"$1/setup\"\n"
|
||||
)
|
||||
}
|
||||
|
||||
@@ -129,6 +131,7 @@ fn aws_files_decode_to_the_host_dirs_mode_600_and_leave_the_environment() {
|
||||
("TRIPLEC_HOST", "aws"),
|
||||
("TRIPLEC_FILES", &files),
|
||||
("TRIPLEC_REPOS", repos),
|
||||
("TRIPLEC_SETUP_SCRIPT", "echo SETUP-BODY"),
|
||||
("TRIPLEC_AGENT_TOKEN", "test-token-not-real"),
|
||||
],
|
||||
&[],
|
||||
@@ -178,6 +181,10 @@ fn aws_files_decode_to_the_host_dirs_mode_600_and_leave_the_environment() {
|
||||
assert!(!agent_env.contains(&b64("PRIVATE-KEY-BODY")));
|
||||
// The repo list survives as an unexported shell variable for clone_repos.
|
||||
assert_eq!(fs::read_to_string(tmp.path().join("repos")).unwrap(), repos);
|
||||
// So does the setup script, for aws_prepare_run to write to its file.
|
||||
assert!(!env.contains("TRIPLEC_SETUP_SCRIPT=") && !env.contains("SETUP-BODY"), "{env}");
|
||||
assert!(!agent_env.contains("TRIPLEC_SETUP_SCRIPT") && !agent_env.contains("SETUP-BODY"));
|
||||
assert_eq!(fs::read_to_string(tmp.path().join("setup")).unwrap(), "echo SETUP-BODY");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -610,6 +617,8 @@ fn wip_push_with_clone_dir(ws: &Path, args: &[&str], clone_dir: &Path) -> Output
|
||||
.args(args)
|
||||
.env("TRIPLEC_WORKSPACE_ROOT", ws)
|
||||
.env("TRIPLEC_CLONE_STATUS_DIR", clone_dir)
|
||||
.env("TRIPLEC_RUN_ID_FILE", tmp.join("no-run-id"))
|
||||
.env("TRIPLEC_WIP_LAST_DIR", tmp.join("no-wip-last"))
|
||||
.env("GIT_CONFIG_NOSYSTEM", "1")
|
||||
.env("HOME", tmp)
|
||||
.output()
|
||||
@@ -1204,7 +1213,7 @@ fn idle_watch_goes_back_to_watching_when_someone_comes_back_during_the_push() {
|
||||
|
||||
#[test]
|
||||
fn idle_watch_keeps_a_stop_guard_marker_already_recorded() {
|
||||
for marker in ["pushed", "discard"] {
|
||||
for marker in ["pushed", "discard", "lifetime"] {
|
||||
let w = Watch::new();
|
||||
fs::write(w.tmp.path().join("reason"), format!("{marker}\n")).unwrap();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
@@ -1252,6 +1261,14 @@ fn aws_stop_pushes_on_sigterm_once_and_leaves_an_idle_stop_to_the_watcher() {
|
||||
assert!(!log.exists(), "no second push after the watcher's");
|
||||
assert!(text(&out).contains("entrypoint: stopping (idle)"), "{}", text(&out));
|
||||
assert!(text(&out).contains("status=75"), "{}", text(&out));
|
||||
|
||||
// A lifetime-cap stop: the watcher already tried its push; exit 76.
|
||||
fs::write(&reason, "lifetime\n").unwrap();
|
||||
let out = bash(&idle_script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "1")], &[]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(!log.exists(), "no second push after the watcher's");
|
||||
assert!(text(&out).contains("entrypoint: stopping (lifetime)"), "{}", text(&out));
|
||||
assert!(text(&out).contains("status=76"), "{}", text(&out));
|
||||
fs::remove_file(&reason).unwrap();
|
||||
|
||||
let out = bash(&script, &[], &[("TRIPLEC_HOST", "aws"), ("TRIPLEC_WIP_PUSH", "0")], &[]);
|
||||
@@ -1294,9 +1311,17 @@ fn the_idle_exit_status_is_the_one_the_app_reads() {
|
||||
.contains(&format!("triplec_idle_exit={}\n", crate::aws::tasks::IDLE_EXIT_CODE)),
|
||||
"entrypoint.sh and aws::tasks::IDLE_EXIT_CODE disagree"
|
||||
);
|
||||
assert!(
|
||||
block(ENTRYPOINT, "aws_stop")
|
||||
.contains(&format!("triplec_lifetime_exit={}\n", crate::aws::tasks::LIFETIME_EXIT_CODE)),
|
||||
"entrypoint.sh and aws::tasks::LIFETIME_EXIT_CODE disagree"
|
||||
);
|
||||
assert!(ENTRYPOINT.contains("exit \"${triplec_exit_status:-0}\""), "stop_keepalive exits with it");
|
||||
// The watcher gets the WIP setting through its scrubbed environment.
|
||||
assert!(block(ENTRYPOINT, "aws_stop").contains("TRIPLEC_WIP_PUSH=\"${TRIPLEC_WIP_PUSH:-0}\""));
|
||||
assert!(block(ENTRYPOINT, "aws_stop")
|
||||
.contains("TRIPLEC_MAX_LIFETIME_MINUTES=\"${TRIPLEC_MAX_LIFETIME_MINUTES:-}\""));
|
||||
assert!(IDLE_WATCH.contains("printf 'lifetime\\n' > \"$reason_file\""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1306,7 +1331,7 @@ fn start_clone_repos_marks_done_however_the_clone_ends() {
|
||||
// clone_repos replaced by one that fails half-way.
|
||||
let script = format!(
|
||||
"{}\nclone_repos() {{ exit 3; }}\nstart_clone_repos '[x]' \"$1\"\nwait\n",
|
||||
aws_blocks(&["aws_env", "aws_repos"])
|
||||
aws_run_blocks(tmp.path(), &["aws_env", "aws_run", "aws_repos"])
|
||||
);
|
||||
let out = bash(&script, &[&status], &[], &[]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
@@ -1317,13 +1342,13 @@ fn start_clone_repos_marks_done_however_the_clone_ends() {
|
||||
fn docker_mode_never_reaches_the_aws_blocks() {
|
||||
// Every new step sits behind TRIPLEC_HOST=aws; the scripts are only
|
||||
// started from those branches.
|
||||
for name in ["aws_env", "aws_home", "aws_repos", "aws_stop"] {
|
||||
for name in ["aws_env", "aws_run", "aws_home", "aws_repos", "aws_stop"] {
|
||||
let b = block(ENTRYPOINT, name);
|
||||
assert!(b.contains("aws"), "{name}");
|
||||
}
|
||||
let outside: String = {
|
||||
let mut s = ENTRYPOINT.to_string();
|
||||
for name in ["aws_env", "aws_home", "aws_repos", "aws_stop"] {
|
||||
for name in ["aws_env", "aws_run", "aws_home", "aws_repos", "aws_stop"] {
|
||||
s = s.replace(block(ENTRYPOINT, name), "");
|
||||
}
|
||||
s
|
||||
@@ -1338,6 +1363,9 @@ fn docker_mode_never_reaches_the_aws_blocks() {
|
||||
"aws_before_stop",
|
||||
"start_idle_watch",
|
||||
"aws_efs_workspace_check",
|
||||
"aws_prepare_run",
|
||||
"aws_run_setup",
|
||||
"triple-c-workspace-status",
|
||||
];
|
||||
let guarded = |s: &str| s.contains("\"$triplec_aws\" = 1");
|
||||
let mut seen = 0;
|
||||
@@ -1354,8 +1382,9 @@ fn docker_mode_never_reaches_the_aws_blocks() {
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
seen, 6,
|
||||
"the call sites: Hybrid/EFS link, EFS /workspace check, Hybrid ~/.ssh, clone, idle watch, stop"
|
||||
seen, 7,
|
||||
"the call sites: run record + setup file, Hybrid/EFS link, EFS /workspace check, Hybrid ~/.ssh, \
|
||||
clone (+ setup), idle watch, stop"
|
||||
);
|
||||
// Clones start in the background *after* the agent: the app gives a
|
||||
// starting task's agent only seconds to answer.
|
||||
@@ -1378,4 +1407,453 @@ fn docker_mode_never_reaches_the_aws_blocks() {
|
||||
// environment snapshot is taken.
|
||||
assert!(ENTRYPOINT.find("unset TRIPLEC_FILES").unwrap() < ENTRYPOINT.find("# >>> agent_token").unwrap());
|
||||
assert!(ENTRYPOINT.find("unset TRIPLEC_REPOS").unwrap() < ENTRYPOINT.find("# >>> agent_token").unwrap());
|
||||
assert!(
|
||||
ENTRYPOINT.find("unset TRIPLEC_SETUP_SCRIPT").unwrap() < ENTRYPOINT.find("# >>> agent_token").unwrap()
|
||||
);
|
||||
// The setup runs in the clone's background chain, never in the agent's way.
|
||||
assert!(block(ENTRYPOINT, "aws_repos").contains("aws_run_setup"));
|
||||
}
|
||||
|
||||
// ------------------------------------------- run record and setup script
|
||||
|
||||
/// The named blocks with the run and setup dirs moved into `tmp` (`run`,
|
||||
/// `setup`) and the container user switched off.
|
||||
fn aws_run_blocks(tmp: &Path, names: &[&str]) -> String {
|
||||
aws_blocks(names)
|
||||
.replace(
|
||||
"triplec_run_dir=/run/triple-c\n",
|
||||
&format!("triplec_run_dir='{}'\n", tmp.join("run").display()),
|
||||
)
|
||||
.replace(
|
||||
"triplec_setup_dir=/run/triple-c-setup\n",
|
||||
&format!("triplec_setup_dir='{}'\n", tmp.join("setup").display()),
|
||||
)
|
||||
}
|
||||
|
||||
/// A one-shot fake ECS task metadata endpoint answering `/task` with `body`.
|
||||
fn fake_metadata(body: &str) -> (String, std::process::Child) {
|
||||
let server = r#"
|
||||
import http.server, sys, socketserver
|
||||
body = sys.argv[1].encode()
|
||||
class H(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
self.send_response(200 if self.path == "/v4/x/task" else 404)
|
||||
self.end_headers()
|
||||
if self.path == "/v4/x/task": self.wfile.write(body)
|
||||
def log_message(self, *a): pass
|
||||
s = socketserver.TCPServer(("127.0.0.1", 0), H)
|
||||
print(s.server_address[1], flush=True)
|
||||
s.serve_forever()
|
||||
"#;
|
||||
let mut child = Command::new("python3")
|
||||
.arg("-I")
|
||||
.arg("-c")
|
||||
.arg(server)
|
||||
.arg(body)
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let mut line = String::new();
|
||||
use std::io::BufRead;
|
||||
std::io::BufReader::new(child.stdout.take().unwrap()).read_line(&mut line).unwrap();
|
||||
(format!("http://127.0.0.1:{}/v4/x", line.trim()), child)
|
||||
}
|
||||
|
||||
fn is_run_id(s: &str, suffix: &str) -> bool {
|
||||
let b = s.as_bytes();
|
||||
s.len() == 16 + suffix.len()
|
||||
&& b[8] == b'-'
|
||||
&& b[15] == b'-'
|
||||
&& s[..8].chars().chain(s[9..15].chars()).all(|c| c.is_ascii_digit())
|
||||
&& &s[16..] == suffix
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn aws_prepare_run_records_the_run_and_writes_the_setup_script() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let script = format!(
|
||||
"{}\ntriplec_setup_script=\"$SETUP\"\naws_prepare_run\n",
|
||||
aws_run_blocks(tmp.path(), &["aws_env", "aws_run"])
|
||||
);
|
||||
let before = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_secs();
|
||||
let out = bash(&script, &[], &[("SETUP", "echo hi\n")], &["ECS_CONTAINER_METADATA_URI_V4", "TRIPLEC_HOST"]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
let run = tmp.path().join("run");
|
||||
let setup = tmp.path().join("setup");
|
||||
let id = fs::read_to_string(run.join("run-id")).unwrap();
|
||||
assert!(is_run_id(id.trim_end(), "local"), "{id:?}");
|
||||
let epoch: u64 = fs::read_to_string(run.join("start-epoch")).unwrap().trim().parse().unwrap();
|
||||
assert!(epoch >= before && epoch <= before + 5);
|
||||
assert_eq!(mode(&run), 0o711, "claude may read the run id, not list the dir");
|
||||
assert_eq!(mode(&run.join("run-id")), 0o644);
|
||||
assert_eq!(mode(&run.join("wip-last")), 0o700);
|
||||
assert_eq!(mode(&setup), 0o755);
|
||||
assert_eq!(fs::read_to_string(setup.join("script")).unwrap(), "echo hi\n");
|
||||
assert_eq!(mode(&setup.join("script")), 0o644);
|
||||
assert_eq!(fs::read_to_string(setup.join("status")).unwrap(), "running\n");
|
||||
assert_eq!(fs::read_to_string(setup.join("log")).unwrap(), "");
|
||||
|
||||
// No script: nothing for the setup to run, no status (state `none`).
|
||||
let out = bash(&script, &[], &[("SETUP", "")], &["ECS_CONTAINER_METADATA_URI_V4"]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(setup.is_dir() && !setup.join("script").exists() && !setup.join("status").exists());
|
||||
|
||||
// On ECS the run id carries the task id's first 8 characters.
|
||||
let (url, mut meta) = fake_metadata(
|
||||
r#"{"TaskARN":"arn:aws:ecs:eu-west-1:123456789012:task/triple-c/0123abcd4567ef890123abcd4567ef89"}"#,
|
||||
);
|
||||
let out = bash(&script, &[], &[("SETUP", ""), ("ECS_CONTAINER_METADATA_URI_V4", &url)], &[]);
|
||||
meta.kill().ok();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
let id = fs::read_to_string(run.join("run-id")).unwrap();
|
||||
assert!(is_run_id(id.trim_end(), "0123abcd"), "{id:?}");
|
||||
// An answer that is not a task id, or none at all, falls back to `local`.
|
||||
let (bad_url, mut bad_meta) = fake_metadata(r#"{"TaskARN":"arn:aws:ecs:x:1:task/c/a-b"}"#);
|
||||
for uri in [bad_url.as_str(), "http://127.0.0.1:9/v4"] {
|
||||
let out = bash(&script, &[], &[("SETUP", ""), ("ECS_CONTAINER_METADATA_URI_V4", uri)], &[]);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
let id = fs::read_to_string(run.join("run-id")).unwrap();
|
||||
assert!(is_run_id(id.trim_end(), "local"), "{uri}: {id:?}");
|
||||
}
|
||||
bad_meta.kill().ok();
|
||||
}
|
||||
|
||||
/// Runs `start_clone_repos` with `setup_script` (and a clone that takes a
|
||||
/// moment), as the entrypoint would; `_triplec_as_user` records its argv.
|
||||
fn run_setup(tmp: &Path, setup_script: &str, timeout: &str) -> Output {
|
||||
let blocks = aws_run_blocks(tmp, &["aws_env", "aws_run", "aws_repos"])
|
||||
.replace("triplec_setup_timeout=1800\n", &format!("triplec_setup_timeout={timeout}\n"));
|
||||
let script = format!(
|
||||
"{blocks}\n\
|
||||
_triplec_as_user() {{ printf '%s\\n' \"$*\" >> \"$CALLS\"; \"$@\"; }}\n\
|
||||
clone_repos() {{ sleep 1; echo cloned > \"$3/marker\"; }}\n\
|
||||
triplec_setup_script=\"$SETUP\"\n\
|
||||
aws_prepare_run\n\
|
||||
start_clone_repos '[]' \"$STATUS\"\n\
|
||||
wait\n"
|
||||
);
|
||||
bash(
|
||||
&script,
|
||||
&[],
|
||||
&[
|
||||
("SETUP", setup_script),
|
||||
("CALLS", tmp.join("calls").to_str().unwrap()),
|
||||
("STATUS", tmp.join("clone").to_str().unwrap()),
|
||||
],
|
||||
&["ECS_CONTAINER_METADATA_URI_V4"],
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_setup_script_runs_after_the_clone_as_the_user_with_no_stdin() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let clone = tmp.path().join("clone");
|
||||
let body = format!(
|
||||
"echo started\nreadlink /proc/self/fd/0\n\
|
||||
if [ -e '{c}/done' ] && [ -e '{c}/marker' ]; then echo clone-was-done; fi\n",
|
||||
c = clone.display()
|
||||
);
|
||||
let out = run_setup(tmp.path(), &body, "1800");
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
let setup = tmp.path().join("setup");
|
||||
let log = fs::read_to_string(setup.join("log")).unwrap();
|
||||
assert_eq!(log, "started\n/dev/null\nclone-was-done\n");
|
||||
assert_eq!(fs::read_to_string(setup.join("status")).unwrap(), "ok\n");
|
||||
let calls = fs::read_to_string(tmp.path().join("calls")).unwrap();
|
||||
assert_eq!(
|
||||
calls,
|
||||
format!("timeout -k 10 1800 bash -e {}/script\n", setup.display()),
|
||||
"as the user, bounded, under bash -e"
|
||||
);
|
||||
assert!(text(&out).contains("entrypoint: setup script ok"), "{}", text(&out));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_setup_script_fails_under_bash_e_and_a_hang_is_killed_as_failed_124() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let out = run_setup(tmp.path(), "echo one\nfalse\necho never\n", "1800");
|
||||
assert!(out.status.success(), "a failed setup never fails the start: {}", text(&out));
|
||||
let setup = tmp.path().join("setup");
|
||||
assert_eq!(fs::read_to_string(setup.join("status")).unwrap(), "failed 1\n");
|
||||
assert_eq!(fs::read_to_string(setup.join("log")).unwrap(), "one\n");
|
||||
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let started = std::time::Instant::now();
|
||||
let out = run_setup(tmp.path(), "echo waiting\nsleep 60\n", "1");
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(started.elapsed() < std::time::Duration::from_secs(20), "the hang was killed");
|
||||
let setup = tmp.path().join("setup");
|
||||
assert_eq!(fs::read_to_string(setup.join("status")).unwrap(), "failed 124\n");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_setup_script_means_no_setup_run() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let out = run_setup(tmp.path(), "", "1800");
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(!tmp.path().join("calls").exists(), "nothing ran as the user");
|
||||
assert!(!tmp.path().join("setup/status").exists());
|
||||
// The clone status is still made, and ends `done`, with no repos at all.
|
||||
assert!(tmp.path().join("clone/done").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn idle_watch_check_counts_a_running_setup_as_busy() {
|
||||
let w = Watch::new();
|
||||
let (url, mut agent) = fake_agent(w.tmp.path(), OLD_ACTIVITY);
|
||||
let setup = w.tmp.path().join("setup");
|
||||
fs::create_dir_all(&setup).unwrap();
|
||||
fs::write(setup.join("status"), "running\n").unwrap();
|
||||
let check = || w.cmd(&url, 0).env("TRIPLEC_SETUP_STATUS_DIR", &setup).arg("--check").output().unwrap();
|
||||
let busy = check();
|
||||
fs::write(setup.join("status"), "failed 124\n").unwrap();
|
||||
let failed = check();
|
||||
fs::write(setup.join("status"), "ok\n").unwrap();
|
||||
let ok = check();
|
||||
agent.kill().ok();
|
||||
assert_eq!(busy.status.code(), Some(1), "{}", text(&busy));
|
||||
assert_eq!(failed.status.code(), Some(0), "{}", text(&failed));
|
||||
assert_eq!(ok.status.code(), Some(0), "{}", text(&ok));
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------- lifetime cap
|
||||
|
||||
impl Watch {
|
||||
/// A watcher for the lifetime cap alone (no idle stop), its run started
|
||||
/// `age_secs` ago.
|
||||
fn lifetime_cmd(&self, signal_pid: u32, minutes: &str, age_secs: u64) -> Command {
|
||||
let p = self.tmp.path();
|
||||
let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_secs();
|
||||
fs::write(p.join("start-epoch"), format!("{}\n", now - age_secs)).unwrap();
|
||||
// An agent that never answers: the idle check alone would never stop it.
|
||||
let mut c = self.cmd("http://127.0.0.1:9", signal_pid);
|
||||
c.env_remove("TRIPLEC_IDLE_STOP_MINUTES")
|
||||
.env("TRIPLEC_MAX_LIFETIME_MINUTES", minutes)
|
||||
.env("TRIPLEC_START_EPOCH_FILE", p.join("start-epoch"));
|
||||
c
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lifetime_cap_pushes_then_stops_with_its_marker_even_when_the_push_fails() {
|
||||
let w = Watch::new();
|
||||
w.dirty();
|
||||
fs::write(w.tmp.path().join("push_rc"), "1").unwrap();
|
||||
let mut target = fake_entrypoint(w.tmp.path());
|
||||
let mut cmd = w.lifetime_cmd(target.id(), "1", 61);
|
||||
cmd.env("TRIPLEC_WIP_PUSH", "1");
|
||||
let out = run_watch_for(cmd, 20).expect("it stops at the cap");
|
||||
target.wait().unwrap();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(w.pushes(), "--reason lifetime\n", "one push, best effort");
|
||||
assert!(text(&out).contains("stopping anyway"), "{}", text(&out));
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("got")).unwrap().trim(), "term");
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "lifetime");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lifetime_cap_with_the_wip_push_off_stops_without_pushing() {
|
||||
let w = Watch::new();
|
||||
w.dirty();
|
||||
let mut target = fake_entrypoint(w.tmp.path());
|
||||
let mut cmd = w.lifetime_cmd(target.id(), "1", 120);
|
||||
cmd.env("TRIPLEC_WIP_PUSH", "0");
|
||||
let out = run_watch_for(cmd, 20).expect("it stops at the cap");
|
||||
target.wait().unwrap();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(w.pushes(), "");
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), "lifetime");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lifetime_cap_not_reached_keeps_the_workspace_up() {
|
||||
let w = Watch::new();
|
||||
let mut target = fake_entrypoint(w.tmp.path());
|
||||
let ran = run_watch_for(w.lifetime_cmd(target.id(), "60", 30), 3);
|
||||
target.kill().ok();
|
||||
target.wait().ok();
|
||||
assert!(ran.is_none(), "{:?}", ran.map(|o| text(&o)));
|
||||
assert!(!w.tmp.path().join("got").exists());
|
||||
// Neither an idle stop nor a cap: nothing to watch, refused.
|
||||
let none = w.cmd("http://127.0.0.1:9", target.id()).env_remove("TRIPLEC_IDLE_STOP_MINUTES").output().unwrap();
|
||||
assert_eq!(none.status.code(), Some(2), "{}", text(&none));
|
||||
let zero = w.lifetime_cmd(target.id(), "0", 30).output().unwrap();
|
||||
assert_eq!(zero.status.code(), Some(2), "0 is no cap: {}", text(&zero));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lifetime_cap_leaves_a_stop_guard_decision_alone() {
|
||||
for marker in ["pushed", "discard"] {
|
||||
let w = Watch::new();
|
||||
w.dirty();
|
||||
fs::write(w.tmp.path().join("reason"), format!("{marker}\n")).unwrap();
|
||||
let mut target = fake_entrypoint(w.tmp.path());
|
||||
let mut cmd = w.lifetime_cmd(target.id(), "1", 120);
|
||||
cmd.env("TRIPLEC_WIP_PUSH", "1");
|
||||
let out = run_watch_for(cmd, 20).expect("it signals");
|
||||
target.wait().unwrap();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(w.pushes(), "", "{marker}: the app already decided");
|
||||
assert_eq!(fs::read_to_string(w.tmp.path().join("reason")).unwrap().trim(), marker);
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------- WIP branch per run
|
||||
|
||||
fn wip_push_in_run(ws: &Path, run_id: Option<&str>) -> Output {
|
||||
let tmp = ws.parent().unwrap();
|
||||
let run = tmp.join("run");
|
||||
fs::create_dir_all(run.join("wip-last")).unwrap();
|
||||
match run_id {
|
||||
Some(id) => fs::write(run.join("run-id"), format!("{id}\n")).unwrap(),
|
||||
None => {
|
||||
let _ = fs::remove_file(run.join("run-id"));
|
||||
}
|
||||
}
|
||||
let script = write_script(tmp, "triple-c-wip-push", WIP_PUSH);
|
||||
Command::new("bash")
|
||||
.arg(script)
|
||||
.args(["--reason", "lifetime"])
|
||||
.env("TRIPLEC_WORKSPACE_ROOT", ws)
|
||||
.env("TRIPLEC_CLONE_STATUS_DIR", tmp.join("no-clone-status"))
|
||||
.env("TRIPLEC_RUN_ID_FILE", run.join("run-id"))
|
||||
.env("TRIPLEC_WIP_LAST_DIR", run.join("wip-last"))
|
||||
.env("GIT_CONFIG_NOSYSTEM", "1")
|
||||
.env("HOME", tmp)
|
||||
.output()
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wip_pushes_in_one_run_update_one_branch_and_skip_an_unchanged_tree() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let (ws, repo, remote) = workspace_with_clone(tmp.path());
|
||||
let run_id = "20261115-093000-0123abcd";
|
||||
fs::write(repo.join("a.txt"), "one").unwrap();
|
||||
let out = wip_push_in_run(&ws, Some(run_id));
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
let branch = format!("refs/heads/triple-c/wip/{run_id}");
|
||||
assert_eq!(wip_refs(&remote), [branch.clone()]);
|
||||
let first = git(&remote, &["rev-parse", &branch]);
|
||||
assert_eq!(git(&remote, &["show", &format!("{branch}:a.txt")]), "one");
|
||||
let last = tmp.path().join("run/wip-last/proj");
|
||||
assert_eq!(
|
||||
fs::read_to_string(&last).unwrap(),
|
||||
format!("{} origin\n", git(&remote, &["rev-parse", &format!("{branch}^{{tree}}")]))
|
||||
);
|
||||
|
||||
// Nothing changed: skipped, the branch stays where it was.
|
||||
let out = wip_push_in_run(&ws, Some(run_id));
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert!(text(&out).contains("unchanged since the last push"), "{}", text(&out));
|
||||
assert_eq!(git(&remote, &["rev-parse", &branch]), first);
|
||||
|
||||
// A change: the same branch moves (forced — the WIP commit is not a
|
||||
// descendant of the last one), still one branch for the run.
|
||||
fs::write(repo.join("a.txt"), "two").unwrap();
|
||||
let out = wip_push_in_run(&ws, Some(run_id));
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(wip_refs(&remote), [branch.clone()]);
|
||||
assert_ne!(git(&remote, &["rev-parse", &branch]), first);
|
||||
assert_eq!(git(&remote, &["show", &format!("{branch}:a.txt")]), "two");
|
||||
assert_eq!(git(&remote, &["log", "-1", "--format=%s", &branch]).contains("(lifetime)"), true);
|
||||
|
||||
// A failed push records nothing, so the next one tries again.
|
||||
fs::write(repo.join("a.txt"), "three").unwrap();
|
||||
let before = fs::read_to_string(&last).unwrap();
|
||||
let saved = tmp.path().join("origin.git.away");
|
||||
fs::rename(&remote, &saved).unwrap();
|
||||
let out = wip_push_in_run(&ws, Some(run_id));
|
||||
assert!(!out.status.success(), "{}", text(&out));
|
||||
assert_eq!(fs::read_to_string(&last).unwrap(), before);
|
||||
fs::rename(&saved, &remote).unwrap();
|
||||
let out = wip_push_in_run(&ws, Some(run_id));
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
assert_eq!(git(&remote, &["show", &format!("{branch}:a.txt")]), "three");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wip_push_without_a_usable_run_id_keeps_the_per_push_stamp() {
|
||||
for run_id in [None, Some("../../evil"), Some("not a run id")] {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let (ws, repo, remote) = workspace_with_clone(tmp.path());
|
||||
fs::write(repo.join("a.txt"), "one").unwrap();
|
||||
let out = wip_push_in_run(&ws, run_id);
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
let refs = wip_refs(&remote);
|
||||
assert_eq!(refs.len(), 1, "{refs:?}");
|
||||
let name = refs[0].trim_start_matches("refs/heads/triple-c/wip/");
|
||||
assert!(name.len() == 15 && name.as_bytes()[8] == b'-', "{run_id:?}: {name}");
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------- workspace status
|
||||
|
||||
fn workspace_status(clone: &Path, setup: &Path) -> serde_json::Value {
|
||||
let tmp = clone.parent().unwrap();
|
||||
let script = write_script(tmp, "triple-c-workspace-status", WORKSPACE_STATUS);
|
||||
let out = Command::new("python3")
|
||||
.arg("-I")
|
||||
.arg(script)
|
||||
.env("TRIPLEC_CLONE_STATUS_DIR", clone)
|
||||
.env("TRIPLEC_SETUP_STATUS_DIR", setup)
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(out.status.success(), "{}", text(&out));
|
||||
let stdout = String::from_utf8(out.stdout).unwrap();
|
||||
assert_eq!(stdout.lines().count(), 1, "one JSON object on one line: {stdout}");
|
||||
serde_json::from_str(&stdout).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn workspace_status_reports_clone_and_setup_and_survives_missing_files() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let clone = tmp.path().join("clone");
|
||||
let setup = tmp.path().join("setup");
|
||||
// Nothing yet: the clone has not started, no setup.
|
||||
assert_eq!(
|
||||
workspace_status(&clone, &setup),
|
||||
serde_json::json!({"clone": {"done": false, "repos": []},
|
||||
"setup": {"state": "none", "code": null, "tail": []}})
|
||||
);
|
||||
|
||||
fs::create_dir_all(&clone).unwrap();
|
||||
fs::write(clone.join("repos"), "api\tcloning\napi\tcloned\nweb\tcloning\n").unwrap();
|
||||
fs::create_dir_all(&setup).unwrap();
|
||||
fs::write(setup.join("status"), "running\n").unwrap();
|
||||
let v = workspace_status(&clone, &setup);
|
||||
assert_eq!(
|
||||
v["clone"],
|
||||
serde_json::json!({"done": false, "repos": [
|
||||
{"folder": "api", "state": "cloned"}, {"folder": "web", "state": "cloning"}]})
|
||||
);
|
||||
assert_eq!(v["setup"], serde_json::json!({"state": "running", "code": null, "tail": []}));
|
||||
|
||||
fs::write(clone.join("repos"), "api\tcloning\napi\tcloned\nweb\tcloning\nweb\tfailed\n").unwrap();
|
||||
fs::write(clone.join("done"), "").unwrap();
|
||||
let lines: String = (1..=25).map(|i| format!("line {i}\n")).collect();
|
||||
fs::write(setup.join("log"), format!("{lines}partial \u{e9}\u{ff}")).unwrap();
|
||||
fs::write(setup.join("status"), "ok\n").unwrap();
|
||||
let v = workspace_status(&clone, &setup);
|
||||
assert_eq!(v["clone"]["done"], true);
|
||||
assert_eq!(v["clone"]["repos"][1], serde_json::json!({"folder": "web", "state": "failed"}));
|
||||
assert_eq!(v["setup"]["state"], "ok");
|
||||
assert_eq!(v["setup"]["code"], 0);
|
||||
let tail: Vec<String> =
|
||||
v["setup"]["tail"].as_array().unwrap().iter().map(|l| l.as_str().unwrap().to_string()).collect();
|
||||
assert_eq!(tail.len(), 20);
|
||||
assert_eq!(tail[0], "line 7");
|
||||
assert!(tail[19].starts_with("partial \u{e9}"), "{:?}", tail[19]);
|
||||
|
||||
fs::write(setup.join("status"), "failed 124\n").unwrap();
|
||||
let v = workspace_status(&clone, &setup);
|
||||
assert_eq!((v["setup"]["state"].as_str(), v["setup"]["code"].as_i64()), (Some("failed"), Some(124)));
|
||||
// Garbled status: a failure with no code, never a crash.
|
||||
fs::write(setup.join("status"), "what\n").unwrap();
|
||||
let v = workspace_status(&clone, &setup);
|
||||
assert_eq!(v["setup"], serde_json::json!({"state": "failed", "code": null, "tail": v["setup"]["tail"]}));
|
||||
// An unreadable log is an empty tail.
|
||||
fs::remove_file(setup.join("log")).unwrap();
|
||||
fs::create_dir(setup.join("log")).unwrap();
|
||||
assert_eq!(workspace_status(&clone, &setup)["setup"]["tail"], serde_json::json!([]));
|
||||
}
|
||||
@@ -19,6 +19,7 @@ const TRIPLE_C_OPEN: &str = include_str!("../../../../container/triple-c-open");
|
||||
const PLAYWRIGHT_HEAL: &str = include_str!("../../../../container/triple-c-playwright-heal");
|
||||
const WIP_PUSH: &str = include_str!("../../../../container/triple-c-wip-push");
|
||||
const IDLE_WATCH: &str = include_str!("../../../../container/triple-c-idle-watch");
|
||||
const WORKSPACE_STATUS: &str = include_str!("../../../../container/triple-c-workspace-status");
|
||||
|
||||
static MISSION_CONTROL_DIR: Dir = include_dir!("$CARGO_MANIFEST_DIR/../../container/mission-control");
|
||||
static SKILLS_DIR: Dir = include_dir!("$CARGO_MANIFEST_DIR/../../container/skills");
|
||||
@@ -249,6 +250,12 @@ fn create_build_context() -> Result<Vec<u8>, std::io::Error> {
|
||||
append_file_to_archive(&mut archive, "triple-c-playwright-heal", PLAYWRIGHT_HEAL.as_bytes(), 0o755)?;
|
||||
append_file_to_archive(&mut archive, "triple-c-wip-push", WIP_PUSH.as_bytes(), 0o755)?;
|
||||
append_file_to_archive(&mut archive, "triple-c-idle-watch", IDLE_WATCH.as_bytes(), 0o755)?;
|
||||
append_file_to_archive(
|
||||
&mut archive,
|
||||
"triple-c-workspace-status",
|
||||
WORKSPACE_STATUS.as_bytes(),
|
||||
0o755,
|
||||
)?;
|
||||
|
||||
append_embedded_dir(&mut archive, &MISSION_CONTROL_DIR, "mission-control")?;
|
||||
append_embedded_dir(&mut archive, &SKILLS_DIR, "skills")?;
|
||||
|
||||
@@ -71,6 +71,10 @@ describe("useProjectStatusEvents", () => {
|
||||
expect(t[0].kind).toBe("info");
|
||||
expect(t[0].message).toContain("idle");
|
||||
|
||||
t = fire({ kind: "lifetime", message: null });
|
||||
expect(t[0].kind).toBe("info");
|
||||
expect(t[0].message).toContain("maximum workspace lifetime was reached");
|
||||
|
||||
expect(fire({ kind: "user", message: null })).toHaveLength(0);
|
||||
|
||||
// A task that never started is news even when found at startup.
|
||||
|
||||
@@ -19,7 +19,7 @@ export interface ContainerExitedEvent {
|
||||
/** `container_watch.rs`'s `StopNote`. */
|
||||
export interface StopNote {
|
||||
/** `user`: someone stopped it (console, CLI, another machine) — no toast. */
|
||||
kind: "spot" | "idle" | "failed" | "user" | "exited";
|
||||
kind: "spot" | "idle" | "lifetime" | "failed" | "user" | "exited";
|
||||
message: string | null;
|
||||
}
|
||||
|
||||
@@ -52,6 +52,14 @@ export function exitToast(
|
||||
message: `“${name}” stopped after being idle`,
|
||||
detail: "Its AWS workspace stopped itself to save cost. Start the project again to continue.",
|
||||
};
|
||||
case "lifetime":
|
||||
return {
|
||||
kind: "info",
|
||||
message: `“${name}” stopped: the environment's maximum workspace lifetime was reached`,
|
||||
detail:
|
||||
"Its AWS environment caps how long a workspace runs. With the WIP push on, unsaved work " +
|
||||
"was pushed (best effort) to a triple-c/wip/ branch. Start the project again to continue.",
|
||||
};
|
||||
default:
|
||||
return {
|
||||
kind: "error",
|
||||
|
||||
@@ -489,10 +489,13 @@ RUN chmod +x /usr/local/bin/triple-c-playwright-heal
|
||||
COPY triple-c-task-runner /usr/local/bin/triple-c-task-runner
|
||||
RUN chmod +x /usr/local/bin/triple-c-task-runner
|
||||
# AWS workspaces only (entrypoint.sh, TRIPLEC_HOST=aws): the WIP push run on
|
||||
# stop and the idle auto-stop watcher. Inert in a Docker container.
|
||||
# stop, the idle auto-stop / lifetime-cap watcher and the start-up status
|
||||
# (clone + setup script) the app polls. Inert in a Docker container.
|
||||
COPY triple-c-wip-push /usr/local/bin/triple-c-wip-push
|
||||
COPY triple-c-idle-watch /usr/local/bin/triple-c-idle-watch
|
||||
RUN chmod +x /usr/local/bin/triple-c-wip-push /usr/local/bin/triple-c-idle-watch
|
||||
COPY triple-c-workspace-status /usr/local/bin/triple-c-workspace-status
|
||||
RUN chmod 755 /usr/local/bin/triple-c-wip-push /usr/local/bin/triple-c-idle-watch \
|
||||
/usr/local/bin/triple-c-workspace-status
|
||||
# Root-owned in /usr/local/bin: entrypoint.sh starts it as root (only when a
|
||||
# token is configured), and it rides the snapshot and migration like the other
|
||||
# shims here.
|
||||
|
||||
+140
-22
@@ -22,11 +22,14 @@
|
||||
# Docker bind mounts use, so the SSH merge and the CA install
|
||||
# below run unchanged;
|
||||
# TRIPLEC_REPOS [{"url","branch"|null,"folder"}] — cloned by clone_repos
|
||||
# once git credentials and ~/.ssh are in place.
|
||||
# Both are unset here, *before* the agent-token block below snapshots the
|
||||
# once git credentials and ~/.ssh are in place;
|
||||
# TRIPLEC_SETUP_SCRIPT the project's setup script (≤ 32 KiB), written to
|
||||
# /run/triple-c-setup/script by aws_prepare_run and run as
|
||||
# `claude` after the clone (aws_run_setup).
|
||||
# All three are unset here, *before* the agent-token block below snapshots the
|
||||
# environment for the agent: that snapshot is every terminal's environment on
|
||||
# AWS, and the files hold private keys. The repo list is kept only as an
|
||||
# unexported shell variable, which no child process inherits.
|
||||
# AWS, and the files hold private keys. The repo list and the setup script are
|
||||
# kept only as unexported shell variables, which no child process inherits.
|
||||
#
|
||||
# Nothing in this block (or any other aws block) runs without TRIPLEC_HOST=aws;
|
||||
# a Docker container sees the same script it always did.
|
||||
@@ -34,6 +37,7 @@
|
||||
triplec_aws=0
|
||||
triplec_run_as=claude
|
||||
triplec_repos_json=
|
||||
triplec_setup_script=
|
||||
# Where Hybrid storage mounts the project's EFS access point.
|
||||
triplec_home_mnt=/mnt/triple-c-home
|
||||
|
||||
@@ -112,9 +116,106 @@ if [ "${TRIPLEC_HOST:-}" = aws ]; then
|
||||
fi
|
||||
triplec_repos_json="${TRIPLEC_REPOS:-}"
|
||||
unset TRIPLEC_REPOS
|
||||
triplec_setup_script="${TRIPLEC_SETUP_SCRIPT:-}"
|
||||
unset TRIPLEC_SETUP_SCRIPT
|
||||
fi
|
||||
# <<< aws_env
|
||||
|
||||
# ── AWS: this run's record, and the setup script's files ───────────────────
|
||||
# aws_prepare_run, as root at the very start of an AWS task:
|
||||
# /run/triple-c root, 0711 — users may open a file they know the
|
||||
# name of, never list it (the stop marker in it is
|
||||
# 0600; the stop guard and the idle watcher keep the
|
||||
# mode);
|
||||
# /run/triple-c/run-id `<UTC yyyymmdd-HHMMSS>-<task id, first 8>` (task id
|
||||
# from the ECS task metadata, else `local`), 0644 —
|
||||
# triple-c-wip-push names this run's one WIP branch
|
||||
# after it (triple-c/wip/<run-id>);
|
||||
# /run/triple-c/start-epoch the start in Unix seconds, for the idle watcher's
|
||||
# lifetime cap (TRIPLEC_MAX_LIFETIME_MINUTES);
|
||||
# /run/triple-c/wip-last 0700, owned by `claude`: triple-c-wip-push (run as
|
||||
# claude) records there the tree it last pushed per
|
||||
# folder and skips an unchanged one. Root makes it
|
||||
# and never looks inside;
|
||||
# /run/triple-c-setup root, 0755: `script` (0644) when the project has a
|
||||
# setup script, `log` (0644; root opens it and hands
|
||||
# the script the descriptor, so the user never needs
|
||||
# write access to anything here and root never opens
|
||||
# a path the user controls) and `status` — `running`
|
||||
# from now until the script ends (waiting for the
|
||||
# clone included), then `ok` or `failed <code>`. No
|
||||
# status file: no setup script.
|
||||
# aws_run_setup runs the script as `claude`, `bash -e`, stdin /dev/null, cwd
|
||||
# /workspace, under `timeout -k 10 1800`, chained after the background clone
|
||||
# (start_clone_repos) so the agent's start is never held up. A failure is
|
||||
# recorded, never fatal. triple-c-workspace-status reads all of this back.
|
||||
# >>> aws_run
|
||||
triplec_run_dir=/run/triple-c
|
||||
triplec_setup_dir=/run/triple-c-setup
|
||||
triplec_setup_timeout=1800
|
||||
|
||||
# The first 8 characters of this task's id, from the ECS task metadata
|
||||
# endpoint ECS sets for every container; `local` without one or on any doubt.
|
||||
aws_task_id8() {
|
||||
local uri="${ECS_CONTAINER_METADATA_URI_V4:-}" arn id
|
||||
if [ -n "$uri" ]; then
|
||||
arn=$(curl -sf --max-time 3 "$uri/task" 2>/dev/null | jq -r '.TaskARN // empty' 2>/dev/null)
|
||||
id=${arn##*/}
|
||||
case "$id" in
|
||||
''|*[!A-Za-z0-9]*) ;;
|
||||
*) [ "${#id}" -ge 8 ] && { printf '%s\n' "${id:0:8}"; return; } ;;
|
||||
esac
|
||||
fi
|
||||
echo local
|
||||
}
|
||||
|
||||
aws_prepare_run() {
|
||||
local now root=0
|
||||
[ "$(id -u)" = 0 ] && root=1
|
||||
now=$(date +%s)
|
||||
if mkdir -p "$triplec_run_dir"; then
|
||||
[ "$root" = 1 ] && chown root:root "$triplec_run_dir"
|
||||
chmod 711 "$triplec_run_dir"
|
||||
printf '%s-%s\n' "$(date -u -d "@$now" +%Y%m%d-%H%M%S)" "$(aws_task_id8)" > "$triplec_run_dir/run-id"
|
||||
chmod 644 "$triplec_run_dir/run-id"
|
||||
printf '%s\n' "$now" > "$triplec_run_dir/start-epoch"
|
||||
rm -rf "$triplec_run_dir/wip-last"
|
||||
if mkdir -m 700 "$triplec_run_dir/wip-last" && [ -n "${triplec_run_as:-}" ]; then
|
||||
chown "$triplec_run_as:" "$triplec_run_dir/wip-last"
|
||||
fi
|
||||
else
|
||||
echo "entrypoint: warning — could not create $triplec_run_dir"
|
||||
fi
|
||||
rm -rf "$triplec_setup_dir"
|
||||
mkdir -m 755 "$triplec_setup_dir" || { echo "entrypoint: warning — could not create $triplec_setup_dir; no setup"; return 0; }
|
||||
[ "$root" = 1 ] && chown root:root "$triplec_setup_dir"
|
||||
[ -n "$triplec_setup_script" ] || return 0
|
||||
printf '%s' "$triplec_setup_script" > "$triplec_setup_dir/script"
|
||||
chmod 644 "$triplec_setup_dir/script"
|
||||
: > "$triplec_setup_dir/log"
|
||||
chmod 644 "$triplec_setup_dir/log"
|
||||
printf 'running\n' > "$triplec_setup_dir/status"
|
||||
triplec_setup_script=
|
||||
}
|
||||
|
||||
# Runs the setup script, if there is one (see above). Never fails.
|
||||
aws_run_setup() {
|
||||
local dir="$triplec_setup_dir" rc state
|
||||
[ -f "$dir/script" ] || return 0
|
||||
echo "entrypoint: running the setup script..."
|
||||
(
|
||||
cd /workspace 2>/dev/null || cd /
|
||||
_triplec_as_user timeout -k 10 "$triplec_setup_timeout" bash -e "$dir/script"
|
||||
) </dev/null >>"$dir/log" 2>&1
|
||||
rc=$?
|
||||
if [ "$rc" -eq 0 ]; then state=ok; else state="failed $rc"; fi
|
||||
printf '%s\n' "$state" > "$dir/status.tmp" && mv -f "$dir/status.tmp" "$dir/status"
|
||||
echo "entrypoint: setup script $state"
|
||||
return 0
|
||||
}
|
||||
# <<< aws_run
|
||||
if [ "$triplec_aws" = 1 ]; then aws_prepare_run; fi
|
||||
|
||||
# ── triple-c-agent token: off the environment before anything else runs ───
|
||||
# The agent (started at the very end, see start_agent) authenticates every
|
||||
# request with this token, and it runs as root, so the token is effectively
|
||||
@@ -769,18 +870,22 @@ for i, r in enumerate(repos):
|
||||
return 0
|
||||
}
|
||||
|
||||
# Runs clone_repos in the background, recording progress in $2.
|
||||
# Runs clone_repos in the background, recording progress in $2, then the
|
||||
# setup script (aws_run, which must see the repositories). The status dir is
|
||||
# made even with no repositories, so `done` always ends up there.
|
||||
start_clone_repos() {
|
||||
local json="$1" status_dir="$2"
|
||||
[ -n "$json" ] || return 0
|
||||
rm -rf "$status_dir"
|
||||
mkdir -m 755 "$status_dir" || status_dir=""
|
||||
(
|
||||
trap '' HUP
|
||||
# `done` however the clone ends (an error, a signal), or the idle
|
||||
# watcher would count it as still running forever.
|
||||
[ -z "$status_dir" ] || trap ': > "$status_dir/done"' EXIT
|
||||
clone_repos "$json" /workspace "$status_dir"
|
||||
(
|
||||
# `done` however the clone ends (an error, a signal), or the idle
|
||||
# watcher would count it as still running forever.
|
||||
[ -z "$status_dir" ] || trap ': > "$status_dir/done"' EXIT
|
||||
clone_repos "$json" /workspace "$status_dir"
|
||||
)
|
||||
aws_run_setup
|
||||
) &
|
||||
}
|
||||
# <<< aws_repos
|
||||
@@ -1220,19 +1325,21 @@ start_agent() {
|
||||
# <<< start_agent
|
||||
start_agent
|
||||
|
||||
# ── AWS: WIP push on stop, idle auto-stop ───────────────────────────────────
|
||||
# ── AWS: WIP push on stop, idle auto-stop, lifetime cap ─────────────────────
|
||||
# aws_before_stop runs from the TERM trap below, before the keepalive goes:
|
||||
# with TRIPLEC_WIP_PUSH=1 it pushes every repo's unsaved work to a
|
||||
# triple-c/wip/<stamp> branch (as `claude`, bounded at 90 s), unless the app's
|
||||
# with TRIPLEC_WIP_PUSH=1 it pushes every repo's unsaved work to the run's
|
||||
# triple-c/wip/<run-id> branch (as `claude`, bounded at 90 s), unless the app's
|
||||
# stop guard recorded `pushed` or `discard` in /run/triple-c/stop-reason
|
||||
# (written as root through the agent just before StopTask), or the idle
|
||||
# watcher recorded `idle` (it pushes *before* signalling and only signals once
|
||||
# nothing unsaved is left, so an idle stop never loses work). ECS gives a task
|
||||
# nothing unsaved is left, so an idle stop never loses work) or `lifetime`
|
||||
# (it tried its push, best effort, before signalling). ECS gives a task
|
||||
# `stopTimeout` seconds between SIGTERM and SIGKILL — 30 by default, 120 at
|
||||
# most on Fargate — so the task definition must raise it for the push to
|
||||
# finish. Once only: a second TERM during the push changes nothing.
|
||||
#
|
||||
# The idle watcher (TRIPLEC_IDLE_STOP_MINUTES set) runs as root under a
|
||||
# The idle watcher (TRIPLEC_IDLE_STOP_MINUTES and/or a non-zero
|
||||
# TRIPLEC_MAX_LIFETIME_MINUTES set) runs as root under a
|
||||
# supervisor loop like the agent's, so a claude-user `pkill` cannot switch
|
||||
# auto-stop off. When the workspace is idle it records the reason "idle" and
|
||||
# sends TERM to *this* script ($$) — not PID 1, which under ECS's
|
||||
@@ -1240,10 +1347,14 @@ start_agent
|
||||
# this same trap and push at most once. An idle stop exits with status 75
|
||||
# (triplec_idle_exit; Rust `aws::tasks::IDLE_EXIT_CODE`), so ECS reports it as
|
||||
# EssentialContainerExited with that exit code and the app can say "stopped
|
||||
# after being idle" rather than "crashed". Its environment is built from
|
||||
# scratch: it needs none of the container's but TRIPLEC_WIP_PUSH.
|
||||
# after being idle" rather than "crashed". A lifetime-cap stop (the run's
|
||||
# start-epoch plus TRIPLEC_MAX_LIFETIME_MINUTES reached; advisory — a user
|
||||
# with sudo can switch it off) exits 76 (triplec_lifetime_exit; Rust
|
||||
# `aws::tasks::LIFETIME_EXIT_CODE`). Its environment is built from scratch: it
|
||||
# needs none of the container's but TRIPLEC_WIP_PUSH and the lifetime cap.
|
||||
# >>> aws_stop
|
||||
triplec_idle_exit=75
|
||||
triplec_lifetime_exit=76
|
||||
aws_before_stop() {
|
||||
[ "${TRIPLEC_HOST:-}" = aws ] || return 0
|
||||
[ -z "${triplec_stopping:-}" ] || return 0
|
||||
@@ -1256,6 +1367,11 @@ aws_before_stop() {
|
||||
triplec_exit_status=$triplec_idle_exit
|
||||
echo "entrypoint: stopping (idle) — work in progress already handled by the idle watcher"
|
||||
return 0 ;;
|
||||
lifetime)
|
||||
# The watcher tried its push (best effort) before signalling.
|
||||
triplec_exit_status=$triplec_lifetime_exit
|
||||
echo "entrypoint: stopping (lifetime) — the environment's maximum workspace lifetime was reached; work in progress already handled by the idle watcher"
|
||||
return 0 ;;
|
||||
# The app's stop guard already pushed, or the user chose to stop
|
||||
# without pushing: honour that instead of pushing again.
|
||||
pushed|discard)
|
||||
@@ -1272,9 +1388,9 @@ aws_before_stop() {
|
||||
|
||||
start_idle_watch() {
|
||||
local minutes="$1" signal_pid="$2"
|
||||
mkdir -p /run/triple-c && chmod 700 /run/triple-c
|
||||
mkdir -p /run/triple-c && chmod 711 /run/triple-c
|
||||
rm -f /run/triple-c/stop-reason
|
||||
if [ -z "${TRIPLEC_AGENT_TOKEN_FILE:-}" ]; then
|
||||
if [ -n "$minutes" ] && [ -z "${TRIPLEC_AGENT_TOKEN_FILE:-}" ]; then
|
||||
echo "triple-c-idle-watch: no agent token, so no activity to read — idle stop will never fire" >&2
|
||||
fi
|
||||
(
|
||||
@@ -1286,6 +1402,7 @@ start_idle_watch() {
|
||||
TRIPLEC_IDLE_SIGNAL_PID="$signal_pid" \
|
||||
TRIPLEC_AGENT_TOKEN_FILE="${TRIPLEC_AGENT_TOKEN_FILE:-/dev/shm/triple-c/agent-token}" \
|
||||
TRIPLEC_WIP_PUSH="${TRIPLEC_WIP_PUSH:-0}" \
|
||||
TRIPLEC_MAX_LIFETIME_MINUTES="${TRIPLEC_MAX_LIFETIME_MINUTES:-}" \
|
||||
/usr/local/bin/triple-c-idle-watch
|
||||
rc=$?
|
||||
# 0: it signalled the stop. 2: misconfigured, and a restart would
|
||||
@@ -1308,10 +1425,11 @@ stop_keepalive() {
|
||||
exit "${triplec_exit_status:-0}"
|
||||
}
|
||||
trap stop_keepalive TERM INT HUP
|
||||
if [ "$triplec_aws" = 1 ] && [ -n "${TRIPLEC_IDLE_STOP_MINUTES:-}" ]; then
|
||||
start_idle_watch "$TRIPLEC_IDLE_STOP_MINUTES" "$$"
|
||||
if [ "$triplec_aws" = 1 ] && { [ -n "${TRIPLEC_IDLE_STOP_MINUTES:-}" ] || [ "${TRIPLEC_MAX_LIFETIME_MINUTES:-0}" != 0 ]; }; then
|
||||
start_idle_watch "${TRIPLEC_IDLE_STOP_MINUTES:-}" "$$"
|
||||
fi
|
||||
# Repos are cloned in the background from here on — see clone_repos.
|
||||
# Repos are cloned in the background from here on — see clone_repos — and the
|
||||
# setup script runs after them (aws_run_setup).
|
||||
if [ "$triplec_aws" = 1 ]; then
|
||||
start_clone_repos "$triplec_repos_json" /run/triple-c-clone
|
||||
fi
|
||||
|
||||
@@ -15,7 +15,8 @@
|
||||
# * no `claude` process is running;
|
||||
# * no scheduler job is running (`triple-c-scheduler status`, as claude);
|
||||
# * the start's background repository clone has finished (entrypoint.sh,
|
||||
# start_clone_repos: /run/triple-c-clone exists without `done`);
|
||||
# start_clone_repos: /run/triple-c-clone exists without `done`), and so
|
||||
# has the setup script (/run/triple-c-setup/status says `running`);
|
||||
# * no work would be lost: `triple-c-wip-push --dry-run` (as claude) finds
|
||||
# no dirty repository and no unpushed commit. With the WIP push on
|
||||
# (TRIPLEC_WIP_PUSH=1) such work is pushed here, before the stop, and the
|
||||
@@ -39,13 +40,26 @@
|
||||
# forward the signal, but signalling the entrypoint directly does not depend on
|
||||
# that and also works without an init.
|
||||
#
|
||||
# Lifetime cap (phase 6, Q8; TRIPLEC_MAX_LIFETIME_MINUTES, 0 or empty = none):
|
||||
# once the run's start (the entrypoint's /run/triple-c/start-epoch) is that
|
||||
# many minutes ago, whatever the activity, it pushes work in progress first
|
||||
# when the WIP push is on (best effort, bounded by TRIPLEC_WIP_TIMEOUT) and
|
||||
# then records "lifetime" and sends SIGTERM even if that push failed; the
|
||||
# entrypoint exits 76. A `pushed`/`discard` marker (the app is stopping it
|
||||
# already) is kept, with no push of its own. The cap is advisory: a workspace
|
||||
# user has sudo and could stop this watcher.
|
||||
#
|
||||
# Either the idle stop or the lifetime cap must be configured; the watcher
|
||||
# runs when at least one is.
|
||||
#
|
||||
# --check evaluates once and exits 0 (idle and nothing unsaved, or unsaved
|
||||
# work the WIP push would take) or 1 (not idle); for tests and for looking at
|
||||
# a live task. It never pushes.
|
||||
#
|
||||
# Configuration (environment; the entrypoint sets the first four, the rest
|
||||
# exist for tests):
|
||||
# TRIPLEC_IDLE_STOP_MINUTES N (required, > 0)
|
||||
# TRIPLEC_IDLE_STOP_MINUTES N (> 0; empty: no idle stop)
|
||||
# TRIPLEC_MAX_LIFETIME_MINUTES the lifetime cap (empty or 0: none)
|
||||
# TRIPLEC_IDLE_SIGNAL_PID the entrypoint's PID
|
||||
# TRIPLEC_AGENT_TOKEN_FILE default /dev/shm/triple-c/agent-token
|
||||
# TRIPLEC_WIP_PUSH 1: push unsaved work before an idle stop
|
||||
@@ -60,6 +74,8 @@
|
||||
# TRIPLEC_WIP_BACKOFF_MAX longest wait between push retries, default 1800
|
||||
# TRIPLEC_IDLE_PROC_NAME default claude
|
||||
# TRIPLEC_CLONE_STATUS_DIR default /run/triple-c-clone
|
||||
# TRIPLEC_SETUP_STATUS_DIR default /run/triple-c-setup
|
||||
# TRIPLEC_START_EPOCH_FILE default /run/triple-c/start-epoch
|
||||
|
||||
set -u
|
||||
|
||||
@@ -77,6 +93,9 @@ wip_push="${TRIPLEC_WIP_PUSH:-0}"
|
||||
wip_bin="${TRIPLEC_WIP_PUSH_BIN:-/usr/local/bin/triple-c-wip-push}"
|
||||
wip_timeout="${TRIPLEC_WIP_TIMEOUT:-90}"
|
||||
backoff_max="${TRIPLEC_WIP_BACKOFF_MAX:-1800}"
|
||||
setup_dir="${TRIPLEC_SETUP_STATUS_DIR:-/run/triple-c-setup}"
|
||||
lifetime="${TRIPLEC_MAX_LIFETIME_MINUTES:-}"
|
||||
start_file="${TRIPLEC_START_EPOCH_FILE:-/run/triple-c/start-epoch}"
|
||||
|
||||
# >>> is_idle
|
||||
# is_idle now last_activity_unix claude_running jobs_running minutes
|
||||
@@ -131,6 +150,11 @@ clone_running() {
|
||||
[ -d "$clone_dir" ] && [ ! -e "$clone_dir/done" ]
|
||||
}
|
||||
|
||||
# So does the setup script (entrypoint.sh, aws_run_setup).
|
||||
setup_running() {
|
||||
[ "$(head -n 1 "$setup_dir/status" 2>/dev/null)" = running ]
|
||||
}
|
||||
|
||||
# The folders with unsaved work (dirty, or commits no remote has), one per
|
||||
# line; nothing when there is none. Fails when it cannot tell.
|
||||
unsaved_work() {
|
||||
@@ -146,6 +170,7 @@ quiet() {
|
||||
if pgrep -x "$proc_name" >/dev/null 2>&1; then claude=1; else claude=0; fi
|
||||
jobs=$(jobs_running)
|
||||
clone_running && jobs=1
|
||||
setup_running && jobs=1
|
||||
is_idle "$(date +%s)" "${last:-}" "$claude" "$jobs" "$minutes"
|
||||
}
|
||||
|
||||
@@ -166,22 +191,69 @@ check() {
|
||||
}
|
||||
|
||||
case "$minutes" in
|
||||
''|*[!0-9]*|0) echo "triple-c-idle-watch: TRIPLEC_IDLE_STOP_MINUTES must be a positive integer" >&2; exit 2 ;;
|
||||
'') ;;
|
||||
*[!0-9]*|0) echo "triple-c-idle-watch: TRIPLEC_IDLE_STOP_MINUTES must be a positive integer" >&2; exit 2 ;;
|
||||
esac
|
||||
case "$lifetime" in
|
||||
''|0) lifetime="" ;;
|
||||
*[!0-9]*) echo "triple-c-idle-watch: TRIPLEC_MAX_LIFETIME_MINUTES must be a whole number of minutes" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
if [ "${1:-}" = "--check" ]; then
|
||||
[ -n "$minutes" ] || { echo "triple-c-idle-watch: --check needs TRIPLEC_IDLE_STOP_MINUTES" >&2; exit 2; }
|
||||
check
|
||||
exit $?
|
||||
fi
|
||||
|
||||
if [ -z "$minutes" ] && [ -z "$lifetime" ]; then
|
||||
echo "triple-c-idle-watch: neither TRIPLEC_IDLE_STOP_MINUTES nor TRIPLEC_MAX_LIFETIME_MINUTES is set — nothing to watch" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
case "$signal_pid" in
|
||||
''|*[!0-9]*|0|1) echo "triple-c-idle-watch: TRIPLEC_IDLE_SIGNAL_PID must be the entrypoint's PID" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
# The run's start, for the lifetime cap. The entrypoint wrote it before
|
||||
# starting this watcher; should it be unreadable the cap counts from now
|
||||
# (later rather than never).
|
||||
started_at=$(head -n 1 "$start_file" 2>/dev/null)
|
||||
case "$started_at" in
|
||||
''|*[!0-9]*)
|
||||
[ -z "$lifetime" ] || echo "triple-c-idle-watch: no start time in $start_file — the lifetime cap counts from now" >&2
|
||||
started_at=$(date +%s) ;;
|
||||
esac
|
||||
|
||||
lifetime_reached() {
|
||||
[ -n "$lifetime" ] && [ $(( $(date +%s) - started_at )) -ge $((lifetime * 60)) ]
|
||||
}
|
||||
|
||||
# The cap is reached: push (best effort) unless the app has decided already,
|
||||
# record "lifetime", signal the entrypoint and end.
|
||||
lifetime_stop() {
|
||||
echo "triple-c-idle-watch: the environment's maximum workspace lifetime (${lifetime} minutes) was reached — stopping the workspace"
|
||||
mkdir -p "$(dirname "$reason_file")" 2>/dev/null
|
||||
case "$(cat "$reason_file" 2>/dev/null)" in
|
||||
pushed|discard)
|
||||
echo "triple-c-idle-watch: Triple-C is already stopping it — keeping its decision" ;;
|
||||
*)
|
||||
if [ "$wip_push" = 1 ]; then
|
||||
echo "triple-c-idle-watch: pushing work in progress before the stop..."
|
||||
as_user timeout -k 5 "$wip_timeout" "$wip_bin" --reason lifetime \
|
||||
|| echo "triple-c-idle-watch: the WIP push failed — stopping anyway (the lifetime cap is reached)"
|
||||
fi
|
||||
printf 'lifetime\n' > "$reason_file" ;;
|
||||
esac
|
||||
kill -TERM "$signal_pid"
|
||||
exit 0
|
||||
}
|
||||
|
||||
push_failures=0
|
||||
next_push_at=0
|
||||
while :; do
|
||||
sleep "$interval"
|
||||
lifetime_reached && lifetime_stop
|
||||
[ -n "$minutes" ] || continue
|
||||
if ! quiet; then
|
||||
said=""
|
||||
continue
|
||||
@@ -220,8 +292,9 @@ while :; do
|
||||
mkdir -p "$(dirname "$reason_file")" 2>/dev/null
|
||||
# A pushed/discard marker means the app's stop guard has decided and
|
||||
# its stop is under way: keep that decision rather than overwrite it.
|
||||
# (Nor a lifetime stop's marker, should one be there.)
|
||||
case "$(cat "$reason_file" 2>/dev/null)" in
|
||||
pushed|discard) ;;
|
||||
pushed|discard|lifetime) ;;
|
||||
*) printf 'idle\n' > "$reason_file" ;;
|
||||
esac
|
||||
kill -TERM "$signal_pid"
|
||||
|
||||
@@ -1,11 +1,22 @@
|
||||
#!/bin/bash
|
||||
# triple-c-wip-push [--dry-run] [--reason idle|stop|sigterm]
|
||||
# triple-c-wip-push [--dry-run] [--reason idle|stop|sigterm|lifetime]
|
||||
#
|
||||
# Save unpushed work before an AWS workspace stops. For every git repository
|
||||
# directly under /workspace whose working tree is dirty or which holds commits
|
||||
# no remote has, push a snapshot of the *whole* working tree (tracked changes,
|
||||
# staged or not, and untracked files .gitignore does not exclude) to
|
||||
# refs/heads/triple-c/wip/<UTC yyyymmdd-HHMMSS> on the repo's remote.
|
||||
# refs/heads/triple-c/wip/<run-id> on the repo's remote.
|
||||
#
|
||||
# One branch per task run (phase 6, Q9): <run-id> is the entrypoint's
|
||||
# /run/triple-c/run-id, `<UTC yyyymmdd-HHMMSS>-<task id, first 8 | local>`.
|
||||
# Later pushes in the same run move that branch (a forced push — the branch is
|
||||
# Triple-C's alone, and each WIP commit sits on the user's HEAD, not on the
|
||||
# last WIP commit), and a push whose tree equals the one last pushed for that
|
||||
# folder in this run is skipped: the tree is recorded after each successful
|
||||
# push in /run/triple-c/wip-last/<folder> (made for `claude` by the
|
||||
# entrypoint). Without a usable run id (Docker, or a run that recorded none)
|
||||
# every push gets its own refs/heads/triple-c/wip/<UTC yyyymmdd-HHMMSS>, never
|
||||
# forced.
|
||||
#
|
||||
# It never touches the user's index, HEAD, branches or working tree: the
|
||||
# snapshot is built in a temporary index (GIT_INDEX_FILE) seeded from a copy of
|
||||
@@ -32,12 +43,15 @@
|
||||
# any failed, 2 on bad usage.
|
||||
#
|
||||
# TRIPLEC_WORKSPACE_ROOT overrides /workspace, TRIPLEC_CLONE_STATUS_DIR
|
||||
# /run/triple-c-clone (tests).
|
||||
# /run/triple-c-clone, TRIPLEC_RUN_ID_FILE /run/triple-c/run-id and
|
||||
# TRIPLEC_WIP_LAST_DIR /run/triple-c/wip-last (tests).
|
||||
|
||||
set -u
|
||||
|
||||
root="${TRIPLEC_WORKSPACE_ROOT:-/workspace}"
|
||||
clone_dir="${TRIPLEC_CLONE_STATUS_DIR:-/run/triple-c-clone}"
|
||||
run_id_file="${TRIPLEC_RUN_ID_FILE:-/run/triple-c/run-id}"
|
||||
last_dir="${TRIPLEC_WIP_LAST_DIR:-/run/triple-c/wip-last}"
|
||||
dry_run=0
|
||||
reason=stop
|
||||
|
||||
@@ -46,10 +60,10 @@ while [ $# -gt 0 ]; do
|
||||
--dry-run) dry_run=1; shift ;;
|
||||
--reason)
|
||||
case "${2:-}" in
|
||||
idle|stop|sigterm) reason="$2"; shift 2 ;;
|
||||
*) echo "triple-c-wip-push: --reason must be idle, stop or sigterm" >&2; exit 2 ;;
|
||||
idle|stop|sigterm|lifetime) reason="$2"; shift 2 ;;
|
||||
*) echo "triple-c-wip-push: --reason must be idle, stop, sigterm or lifetime" >&2; exit 2 ;;
|
||||
esac ;;
|
||||
*) echo "usage: triple-c-wip-push [--dry-run] [--reason idle|stop|sigterm]" >&2; exit 2 ;;
|
||||
*) echo "usage: triple-c-wip-push [--dry-run] [--reason idle|stop|sigterm|lifetime]" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
@@ -60,6 +74,17 @@ shopt -s dotglob nullglob
|
||||
stamp=$(date -u +%Y%m%d-%H%M%S)
|
||||
failed=0
|
||||
|
||||
# This run's branch, when the entrypoint recorded a well-formed run id; the
|
||||
# per-push stamp otherwise. Only our own per-run branch is ever force-pushed.
|
||||
run_id=$(head -n 1 "$run_id_file" 2>/dev/null)
|
||||
if [[ "$run_id" =~ ^[0-9]{8}-[0-9]{6}-[A-Za-z0-9]{1,32}$ ]]; then
|
||||
ref="refs/heads/triple-c/wip/$run_id"
|
||||
force=+
|
||||
else
|
||||
ref="refs/heads/triple-c/wip/$stamp"
|
||||
force=
|
||||
fi
|
||||
|
||||
# The remote to push to: the branch's upstream remote, else origin, else the
|
||||
# only remote. Empty when there is none.
|
||||
pick_remote() {
|
||||
@@ -142,6 +167,7 @@ for dir in "$root"/*/; do
|
||||
fi
|
||||
|
||||
commit="$head"
|
||||
tree=""
|
||||
if [ "$dirty" = true ]; then
|
||||
tmp_index=$(mktemp) || exit 1
|
||||
trap 'rm -f "$tmp_index"' EXIT
|
||||
@@ -159,16 +185,28 @@ for dir in "$root"/*/; do
|
||||
git config user.email >/dev/null 2>&1 || export GIT_AUTHOR_EMAIL=triple-c@localhost GIT_COMMITTER_EMAIL=triple-c@localhost
|
||||
git add -A . >/dev/null 2>&1 || { echo "triple-c-wip-push: $folder: could not stage the working tree" >&2; exit 1; }
|
||||
tree=$(git write-tree) || exit 1
|
||||
unset GIT_INDEX_FILE
|
||||
else
|
||||
tree=$(git rev-parse -q --verify "$head^{tree}") || exit 1
|
||||
fi
|
||||
|
||||
# Pushed already in this run, unchanged since: nothing new to save.
|
||||
last_file="$last_dir/$folder"
|
||||
if [ -n "$tree" ] && [ -f "$last_file" ] \
|
||||
&& [ "$(head -n 1 "$last_file" 2>/dev/null)" = "$tree $remote" ]; then
|
||||
echo "triple-c-wip-push: $folder: unchanged since the last push of this run — skipped"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$dirty" = true ]; then
|
||||
msg="Triple-C WIP ($reason) of $branch at $stamp UTC"
|
||||
if [ -n "$head" ]; then
|
||||
commit=$(git commit-tree "$tree" -p "$head" -m "$msg") || exit 1
|
||||
else
|
||||
commit=$(git commit-tree "$tree" -m "$msg") || exit 1
|
||||
fi
|
||||
unset GIT_INDEX_FILE
|
||||
fi
|
||||
|
||||
ref="refs/heads/triple-c/wip/$stamp"
|
||||
# --no-verify skips every pre-push hook, Git LFS's upload among them.
|
||||
# So when git-lfs is here and the snapshot has LFS files, upload their
|
||||
# objects first — best effort: the push still goes, with a warning.
|
||||
@@ -180,8 +218,16 @@ for dir in "$root"/*/; do
|
||||
echo "triple-c-wip-push: $folder: warning — LFS upload failed, large files may be missing on the branch: $(printf '%s' "$lfs_err" | tail -n 1)" >&2
|
||||
fi
|
||||
fi
|
||||
if err=$(git push --no-verify -q "$remote" "$commit:$ref" 2>&1); then
|
||||
if err=$(git push --no-verify -q "$remote" "$force$commit:$ref" 2>&1); then
|
||||
echo "triple-c-wip-push: $folder: pushed to $remote $ref"
|
||||
# Remember what this run pushed (best effort: without it the next
|
||||
# push just pushes again). Only where the entrypoint made the dir.
|
||||
if [ -d "$last_dir" ] && [ -w "$last_dir" ]; then
|
||||
tmp_last=$(mktemp "$last_dir/.tmp.XXXXXX" 2>/dev/null) \
|
||||
&& printf '%s %s\n' "$tree" "$remote" > "$tmp_last" \
|
||||
&& mv -f "$tmp_last" "$last_file" \
|
||||
|| rm -f "${tmp_last:-}"
|
||||
fi
|
||||
else
|
||||
echo "triple-c-wip-push: $folder: push to $remote failed: $(printf '%s' "$err" | tail -n 1)" >&2
|
||||
exit 1
|
||||
|
||||
Executable
+84
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/python3 -I
|
||||
# triple-c-workspace-status
|
||||
#
|
||||
# An AWS workspace's start-up progress as one JSON object on one line, for the
|
||||
# app's start command and Project Home (phase 6, Q4):
|
||||
#
|
||||
# {"clone": {"done": bool, "repos": [{"folder": str, "state": str}, ...]},
|
||||
# "setup": {"state": "none"|"running"|"ok"|"failed",
|
||||
# "code": int|null, "tail": [the setup log's last 20 lines]}}
|
||||
#
|
||||
# Read from what entrypoint.sh writes (root-owned, world-readable, so this
|
||||
# works as `claude` and as root):
|
||||
# /run/triple-c-clone `repos` ("<folder>\t<state>" lines; a folder's last
|
||||
# line is its state: cloning, cloned, failed, exists)
|
||||
# and `done`. No directory yet: the clone has not
|
||||
# started (done: false).
|
||||
# /run/triple-c-setup `status` (`running`, `ok`, `failed <code>`; absent:
|
||||
# no setup script) and `log`.
|
||||
# A missing or unreadable file is reported as its empty state, never an
|
||||
# error; a status line it does not know is a failure without a code. Nothing
|
||||
# else is read: no environment, no secrets — the tail is the setup script's
|
||||
# own output and nothing more.
|
||||
#
|
||||
# TRIPLEC_CLONE_STATUS_DIR and TRIPLEC_SETUP_STATUS_DIR override the two
|
||||
# directories (tests).
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
TAIL_LINES = 20
|
||||
# Enough for 20 long lines; the log can be large and only its end matters.
|
||||
TAIL_BYTES = 64 * 1024
|
||||
MAX_LINE = 1000
|
||||
|
||||
clone_dir = os.environ.get("TRIPLEC_CLONE_STATUS_DIR") or "/run/triple-c-clone"
|
||||
setup_dir = os.environ.get("TRIPLEC_SETUP_STATUS_DIR") or "/run/triple-c-setup"
|
||||
|
||||
|
||||
def read_text(path, limit=None):
|
||||
"""The file's text (its last `limit` bytes when given), or None."""
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
if limit is not None:
|
||||
f.seek(0, os.SEEK_END)
|
||||
size = f.tell()
|
||||
f.seek(max(0, size - limit))
|
||||
data = f.read() if limit is not None else f.read(1024 * 1024)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return data.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def clone_status():
|
||||
done = os.path.exists(os.path.join(clone_dir, "done"))
|
||||
states = {}
|
||||
for line in (read_text(os.path.join(clone_dir, "repos")) or "").splitlines():
|
||||
folder, sep, state = line.partition("\t")
|
||||
if sep and folder:
|
||||
states[folder] = state # dicts keep first-seen order
|
||||
return {"done": done, "repos": [{"folder": f, "state": s} for f, s in states.items()]}
|
||||
|
||||
|
||||
def setup_status():
|
||||
raw = read_text(os.path.join(setup_dir, "status"))
|
||||
if raw is None:
|
||||
state, code = "none", None
|
||||
else:
|
||||
word, _, rest = raw.strip().partition(" ")
|
||||
if word == "running" and not rest:
|
||||
state, code = "running", None
|
||||
elif word == "ok" and not rest:
|
||||
state, code = "ok", 0
|
||||
else:
|
||||
state = "failed"
|
||||
code = int(rest) if word == "failed" and rest.strip().isdigit() else None
|
||||
log = read_text(os.path.join(setup_dir, "log"), TAIL_BYTES) or ""
|
||||
tail = [l[:MAX_LINE] for l in log.splitlines()[-TAIL_LINES:]]
|
||||
return {"state": state, "code": code, "tail": tail}
|
||||
|
||||
|
||||
json.dump({"clone": clone_status(), "setup": setup_status()}, sys.stdout, ensure_ascii=False,
|
||||
separators=(",", ":"))
|
||||
sys.stdout.write("\n")
|
||||
Reference in new issue
Block a user