diff --git a/CLAUDE.md b/CLAUDE.md index ae7a67b..c0ee221 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -327,7 +327,27 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li admin credentials for a disposable account; `cargo test aws::live_tests -- --ignored --test-threads=1`) runs the check and real calls against throwaway `triple-c-e2e-*` roles, secrets and keys, and cleans them up. Re-run it after any policy change; the results are in - `aws/README.md` ("Live verification"). + `aws/README.md` ("Live verification"). Its `environment` tests apply the real modules as a role + holding only the admin files (`cargo build` first: the runner they start is the built app + binary), in a throwaway state bucket. + **Environments are OpenTofu modules** (`aws/terraform/{environment,user-onboard,permissions-boundary}`; + spec D2–D10). `tofu_modules.rs` embeds them and is the **only** place the shared names are built + (environment, `/triple-c/environments/` record, owner key, per-user roles, state bucket + `triple-c-tfstate--`, state keys); its tests pin the HCL to those names, the + versions and tags, and `MODULE_CALLS` (every provider call) to the admin policy. + `tofu.rs` downloads the pinned OpenTofu (SHA-256 checked **before** extraction), writes a work + folder (`materialize`), builds tofu's environment from scratch (`child_env`: no inherited + `AWS_*`/`TF_*`, keys only in the environment) and runs `init` + `apply|destroy` **detached + (D10)**: the app binary re-executed with `--tofu-runner` (`main.rs` checks that first), in its + own session, logging to `apply.log`; liveness is the runner's `apply.lock`, never a PID; closing + the app never stops a run and the next start re-follows the log. `environments.rs` is detection + (records + state objects + locks; records are untrusted and capped; an empty state, which + `tofu destroy` leaves behind, is no deployment) and `ensure_state_bucket`; + `commands/environment_commands.rs` sequences ensure_tofu → ensure_state_bucket → materialize → + start → follow. **D2: strict tenancy gives each user their own EFS file system** (its policy + admits only their task role) from their onboarding module; the environment keeps only the shared + pieces. The app destroys only incomplete deployments (D6); a working environment is removed with + `tofu destroy` from an admin bundle (D5). - **`web_terminal/`** — Remote terminal access via axum HTTP+WebSocket server: - `server.rs` — Axum server lifecycle (start/stop), serves embedded HTML and handles WS upgrades - `ws_handler.rs` — Per-connection WebSocket handler with JSON protocol, session management, cleanup on disconnect diff --git a/app/src-tauri/src/aws/live_tests.rs b/app/src-tauri/src/aws/live_tests.rs index 9385318..77be47b 100644 --- a/app/src-tauri/src/aws/live_tests.rs +++ b/app/src-tauri/src/aws/live_tests.rs @@ -289,6 +289,11 @@ impl E2e { /// An SDK config for a session of `role`, once IAM lets it be assumed /// and its policies have propagated. async fn assume(&self, role: &Role) -> SdkConfig { + sdk_config_for_keys(&self.assume_keys(role).await, &self.region) + } + + /// A session of `role` as keys (see [`E2e::assume`]). + async fn assume_keys(&self, role: &Role) -> AwsKeys { let mut last = None; for _ in 0..20 { match self @@ -309,7 +314,7 @@ impl E2e { // New sessions see a just-attached policy after a few // seconds; denials in that window would be noise. tokio::time::sleep(Duration::from_secs(10)).await; - return sdk_config_for_keys(&keys, &self.region); + return keys; } Err(e) => { last = Some(classify_sdk_error("sts:AssumeRole", &e).to_string()); @@ -957,3 +962,705 @@ async fn the_admin_tier_creates_only_bounded_roles_and_tagged_keys() { }) .await; } + +/// Phase 4: OpenTofu environments, deployed for real by a role that holds +/// only the environment-admin policies. +/// +/// ```sh +/// cd app/src-tauri +/// cargo build # the runner is the app binary re-executed (see below) +/// TRIPLEC_AWS_E2E_PROFILE=… TRIPLEC_AWS_E2E_REGION=us-west-2 \ +/// cargo test aws::live_tests::environment -- --ignored --test-threads=1 +/// ``` +/// +/// What runs the way the app runs it: `ensure_tofu` (the pinned download), +/// `ensure_state_bucket`'s configuration, `materialize`, `child_env` (a keys +/// connection for the environment admin, a profile connection for the +/// account admin), the detached runner — the **built app binary** +/// (`target/debug/triple-c`, or `TRIPLEC_E2E_APP_BIN`) re-executed with +/// `--tofu-runner`, because the test harness is not the app and can't +/// re-execute itself as one — `follow`, `scan`, the refusal checks the +/// commands make, and the conditional lock delete. What doesn't: the Tauri +/// commands themselves (they need an `AppHandle` and saved settings), so the +/// slot/follower bookkeeping and the event stream are not exercised here. +/// +/// Extra variables: +/// - `TRIPLEC_E2E_BUCKET_SUFFIX`: the state bucket is +/// `triple-c-tfstate---` (default: random), so the +/// account's real one is never touched; +/// - `TRIPLEC_E2E_TOFU_ROOT`: where OpenTofu, the provider cache and the work +/// folders go (default: `/triple-c-e2e-tofu`, kept between runs). +mod environment { + use std::path::PathBuf; + use std::sync::Arc; + use std::time::Duration; + + use aws_config::SdkConfig; + + use super::{e2e, ok, E2e, Role}; + use crate::aws::environments::{ + ensure_named_state_bucket, is_empty_state, scan_in_bucket, EnvironmentAction, EnvironmentState, Knowable, + OnboardingState, SdkDetectApi, + }; + use crate::aws::policies::{policy_files, Tier, PERMISSIONS_BOUNDARY_NAME}; + use crate::aws::tofu::{ + self, child_env, ensure_tofu, follow, materialize, start_detached_with, TofuOp, + TofuOutcome, Workdir, + }; + use crate::aws::tofu_modules::{ + boundary_vars, environment_vars, onboarding_vars, owner_key, state_bucket_name, state_key, + user_role_names, Deployment, ModuleKind, NetworkChoice, + }; + use crate::aws::{caller_identity, sdk_config_for_keys, CallerIdentity}; + use crate::commands::environment_commands::{ + delete_lock_if_unchanged, deploy_refusal, network_from_record, onboard_refusal, + validate_owner_id, LOCK_CHANGED, + }; + use crate::models::host_connection::{AwsAuth, HostConnection, HostKind}; + use crate::storage::secure::AwsKeys; + + /// The environment admin: a role holding the admin tier's policy files + /// as managed policies, and nothing else. + struct Admin { + role: Role, + } + + struct Env { + e: Arc, + root: PathBuf, + tofu: PathBuf, + app_bin: PathBuf, + bucket: String, + /// Deployments applied (work folder, still to destroy), newest last. + applied: std::sync::Mutex>, + /// The test created the boundary (through its module). + boundary_created: std::sync::Mutex, + bucket_created: std::sync::Mutex, + } + + fn app_bin() -> PathBuf { + if let Ok(p) = std::env::var("TRIPLEC_E2E_APP_BIN") { + return PathBuf::from(p); + } + // target/debug/deps/ -> target/debug/triple-c + let exe = std::env::current_exe().expect("current exe"); + let dir = exe.parent().and_then(|d| d.parent()).expect("target dir"); + dir.join(format!("triple-c{}", std::env::consts::EXE_SUFFIX)) + } + + fn keys_connection(region: &str) -> HostConnection { + HostConnection { + id: "e2e-envadmin".into(), + name: "e2e-envadmin".into(), + kind: HostKind::Aws { + auth: AwsAuth::Keys { + keyring_ref: "unused".into(), + }, + default_region: region.to_string(), + }, + } + } + + fn profile_connection(region: &str) -> HostConnection { + HostConnection { + id: "e2e".into(), + name: "e2e".into(), + kind: HostKind::Aws { + auth: AwsAuth::Profile { + name: super::var("TRIPLEC_AWS_E2E_PROFILE"), + }, + default_region: region.to_string(), + }, + } + } + + /// `not authorized to perform: ` (and the KMS/S3 phrasings) in a + /// log message. + fn denied_actions(message: &str) -> Vec { + let mut out = Vec::new(); + for marker in ["not authorized to perform: ", "is not authorized to perform "] { + let mut rest = message; + while let Some(i) = rest.find(marker) { + rest = &rest[i + marker.len()..]; + let action: String = rest + .chars() + .take_while(|c| c.is_ascii_alphanumeric() || *c == ':' || *c == '*') + .collect(); + if !action.is_empty() && !out.contains(&action) { + out.push(action); + } + } + } + if out.is_empty() + && (message.contains("AccessDenied") || message.contains("UnauthorizedOperation")) + { + out.push("(unnamed: see the message above)".into()); + } + out + } + + impl Env { + async fn new(e: Arc) -> Arc { + let root = std::env::var("TRIPLEC_E2E_TOFU_ROOT") + .map(PathBuf::from) + .unwrap_or_else(|_| std::env::temp_dir().join("triple-c-e2e-tofu")); + std::fs::create_dir_all(&root).unwrap(); + let app_bin = app_bin(); + assert!( + app_bin.is_file(), + "{} is missing: run `cargo build` first (the runner is the app binary)", + app_bin.display() + ); + let tofu = ensure_tofu(&root).await.expect("ensure_tofu"); + let suffix = std::env::var("TRIPLEC_E2E_BUCKET_SUFFIX").unwrap_or_else(|_| { + format!("e2e{}", &uuid::Uuid::new_v4().simple().to_string()[..8]) + }); + let bucket = format!("{}-{suffix}", state_bucket_name(&e.account, &e.region)); + assert!(bucket.len() <= 63, "{bucket} is too long for a bucket name"); + Arc::new(Env { + e, + root, + tofu, + app_bin, + bucket, + applied: Default::default(), + boundary_created: Default::default(), + bucket_created: Default::default(), + }) + } + + /// The environment admin role (no boundary on the role itself: the + /// boundary allows only what tasks do, so an admin under it could + /// administer nothing). + async fn admin(&self) -> Admin { + let role = self.e.role("envadmin", None, &[]).await; + for (i, (file, json)) in policy_files(Tier::EnvironmentAdmin).iter().enumerate() { + let arn = self.e.managed_policy(&format!("envadmin{i}"), json).await; + eprintln!("envadmin: {file} -> {arn}"); + self.e.attach(&role, &arn).await; + } + Admin { role } + } + + /// A fresh session of the admin role (role chaining caps one at an + /// hour, so each phase takes its own). + async fn session(&self, admin: &Admin) -> (AwsKeys, SdkConfig, CallerIdentity) { + let keys = self.e.assume_keys(&admin.role).await; + let cfg = sdk_config_for_keys(&keys, &self.e.region); + let who = caller_identity(&cfg).await.expect("envadmin identity"); + (keys, cfg, who) + } + + async fn scan(&self, cfg: &SdkConfig, who: &CallerIdentity) -> crate::aws::environments::EnvironmentScan { + let api = SdkDetectApi::new(cfg, &who.account); + let (root, bucket) = (self.root.clone(), self.bucket.clone()); + let running = move |d: Deployment| { + tofu::workdir_for(&root, d, &bucket) + .map(|wd| tofu::is_running(&wd)) + .unwrap_or(false) + }; + scan_in_bucket(&api, who, &self.e.region, &self.bucket, &running) + .await + .expect("scan") + } + + /// Write the module and run `op` through the detached runner, printing + /// the log and every denied action. + async fn run( + &self, + kind: ModuleKind, + d: Deployment<'_>, + vars: &serde_json::Value, + op: TofuOp, + env: Vec<(String, String)>, + ) -> (Workdir, TofuOutcome, Vec) { + let wd = materialize(&self.root, kind, d, &self.bucket, &self.e.region, vars) + .expect("materialize"); + if op == TofuOp::Apply { + let mut applied = self.applied.lock().unwrap(); + if !applied.iter().any(|(w, _)| *w == wd) { + applied.push((wd.clone(), true)); + } + } + let label = state_key(d); + eprintln!("=== tofu {op:?} {label} ({})", wd.dir.display()); + let (bin, tofu_bin, start_wd) = (self.app_bin.clone(), self.tofu.clone(), wd.clone()); + let started = tokio::task::spawn_blocking(move || { + start_detached_with(&bin, Duration::from_secs(30), &tofu_bin, &start_wd, op, &env) + }) + .await + .unwrap() + .unwrap_or_else(|e| panic!("start {label}: {e}")); + assert!(started.pid.is_some(), "the runner wrote no PID"); + let mut denied = Vec::new(); + let outcome = follow(&wd, Duration::from_millis(500), |events| { + for ev in events { + let resource = ev.resource.as_deref().unwrap_or(""); + let short: String = ev.message.chars().take(300).collect(); + eprintln!(" [{}] {} {} {}", ev.level, ev.kind, resource, short); + if ev.level == "error" { + eprintln!(" ERROR DETAIL: {}", ev.message); + for a in denied_actions(&ev.message) { + eprintln!(" DENIED: {a} ({resource})"); + if !denied.contains(&a) { + denied.push(a); + } + } + } + } + }) + .await + .expect("follow"); + eprintln!("=== {label}: {outcome:?}"); + if op == TofuOp::Destroy && outcome.ok { + for (w, pending) in self.applied.lock().unwrap().iter_mut() { + if *w == wd { + *pending = false; + } + } + } + (wd, outcome, denied) + } + + fn keys_env(&self, keys: &AwsKeys) -> Vec<(String, String)> { + child_env( + &keys_connection(&self.e.region), + Some(keys), + &self.root, + std::env::vars(), + ) + .expect("child env") + } + + fn admin_env(&self) -> Vec<(String, String)> { + child_env(&profile_connection(&self.e.region), None, &self.root, std::env::vars()) + .expect("child env") + } + + async fn ensure_bucket(&self, cfg: &SdkConfig, who: &CallerIdentity) { + *self.bucket_created.lock().unwrap() = true; + let b = ensure_named_state_bucket(cfg, &who.account, &self.e.region, &self.bucket) + .await + .unwrap_or_else(|e| panic!("ensure_state_bucket as the environment admin: {e}")); + assert_eq!(b, self.bucket); + } + + /// The boundary, through its module as the account admin, if the + /// account has none. + async fn ensure_boundary(&self) { + let arn = format!( + "arn:aws:iam::{}:policy/{PERMISSIONS_BOUNDARY_NAME}", + self.e.account + ); + if self.e.iam.get_policy().policy_arn(&arn).send().await.is_ok() { + eprintln!("the account already has the boundary; using it"); + return; + } + *self.boundary_created.lock().unwrap() = true; + let (_, outcome, denied) = self + .run( + ModuleKind::PermissionsBoundary, + Deployment::Boundary, + &boundary_vars(&self.e.region), + TofuOp::Apply, + self.admin_env(), + ) + .await; + assert!(outcome.ok, "boundary apply: {outcome:?} denied {denied:?}"); + } + + /// As the account admin: destroy whatever is still applied (newest + /// first), the bucket with every version, and the boundary if this + /// test made it. Logs rather than fails, so one error strands nothing + /// else. + async fn cleanup(&self) { + let pending: Vec = self + .applied + .lock() + .unwrap() + .iter() + .rev() + .filter(|(_, p)| *p) + .map(|(w, _)| w.clone()) + .collect(); + for wd in pending { + eprintln!("cleanup: destroying {} as the account admin", wd.dir.display()); + let env = self.admin_env(); + let (bin, tofu_bin, w) = (self.app_bin.clone(), self.tofu.clone(), wd.clone()); + let started = tokio::task::spawn_blocking(move || { + start_detached_with(&bin, Duration::from_secs(30), &tofu_bin, &w, TofuOp::Destroy, &env) + }) + .await; + match started { + Ok(Ok(_)) => { + let outcome = follow(&wd, Duration::from_millis(500), |events| { + for ev in events.iter().filter(|e| e.level == "error") { + eprintln!(" cleanup error: {}", ev.message); + } + }) + .await; + eprintln!("cleanup: destroy {}: {outcome:?}", wd.dir.display()); + } + other => eprintln!("cleanup: could not start destroy: {other:?}"), + } + } + if *self.boundary_created.lock().unwrap() { + let arn = format!( + "arn:aws:iam::{}:policy/{PERMISSIONS_BOUNDARY_NAME}", + self.e.account + ); + if self.e.iam.get_policy().policy_arn(&arn).send().await.is_ok() { + eprintln!("cleanup: deleting the boundary directly"); + if let Ok(out) = self.e.iam.list_entities_for_policy().policy_arn(&arn).send().await { + for r in out.policy_roles() { + if let Some(name) = r.role_name() { + let _ = self + .e + .iam + .delete_role_permissions_boundary() + .role_name(name) + .send() + .await; + } + } + } + if let Err(e) = self.e.iam.delete_policy().policy_arn(&arn).send().await { + eprintln!("cleanup: boundary: {e:?}"); + } + } + } + if *self.bucket_created.lock().unwrap() { + self.delete_bucket().await; + } + } + + async fn delete_bucket(&self) { + let s3 = aws_sdk_s3::Client::new(&self.e.cfg); + loop { + let page = match s3.list_object_versions().bucket(&self.bucket).send().await { + Ok(p) => p, + Err(e) => { + use aws_sdk_s3::error::ProvideErrorMetadata; + if e.code() != Some("NoSuchBucket") { + eprintln!("cleanup: list versions of {}: {e:?}", self.bucket); + } + return; + } + }; + let mut ids = Vec::new(); + for v in page.versions() { + ids.push((v.key().unwrap_or_default().to_string(), v.version_id().map(str::to_string))); + } + for m in page.delete_markers() { + ids.push((m.key().unwrap_or_default().to_string(), m.version_id().map(str::to_string))); + } + if ids.is_empty() { + break; + } + for (key, version) in ids { + if let Err(e) = s3 + .delete_object() + .bucket(&self.bucket) + .key(&key) + .set_version_id(version) + .send() + .await + { + eprintln!("cleanup: delete {key}: {e:?}"); + return; + } + } + } + match s3.delete_bucket().bucket(&self.bucket).send().await { + Ok(_) => eprintln!("cleanup: deleted bucket {}", self.bucket), + Err(e) => eprintln!("cleanup: delete bucket {}: {e:?}", self.bucket), + } + } + } + + /// `e2e` plus the environment fixture's own cleanup, which runs first + /// (it needs the account admin, and the roles it destroys hold the + /// boundary). + async fn env_e2e(body: F) + where + F: FnOnce(Arc) -> Fut, + Fut: std::future::Future, + { + use futures_util::FutureExt; + e2e(|e| async move { + let env = Env::new(e).await; + let result = std::panic::AssertUnwindSafe(body(env.clone())) + .catch_unwind() + .await; + env.cleanup().await; + if let Err(panic) = result { + std::panic::resume_unwind(panic); + } + }) + .await; + } + + fn assert_ok(what: &str, outcome: &TofuOutcome, denied: &[String]) { + assert!( + outcome.ok, + "{what} failed: {:?}; denied actions: {denied:?}", + outcome.summary + ); + assert!(denied.is_empty(), "{what} reported denied actions: {denied:?}"); + } + + #[tokio::test] + #[ignore] + async fn an_environment_admin_can_deploy_join_update_and_remove_an_environment() { + env_e2e(|t| async move { + let admin = t.admin().await; + let (keys, cfg, who) = t.session(&admin).await; + assert!(who.arn.contains(":assumed-role/"), "{}", who.arn); + + // The state bucket, created and configured by the environment + // admin (s3:CreateBucket with the request tag, in-account). + t.ensure_bucket(&cfg, &who).await; + // Idempotent. + t.ensure_bucket(&cfg, &who).await; + + t.ensure_boundary().await; + + let name = format!("e2e{}", &uuid::Uuid::new_v4().simple().to_string()[..6]); + let found = t.scan(&cfg, &who).await; + assert_eq!(found.state_bucket, Knowable::Yes); + assert_eq!(found.boundary, Knowable::Yes); + deploy_refusal(&found, &name).expect("deploy is offered"); + + // Deploy, public network, as the environment admin only. + let vars = environment_vars(&name, &t.e.region, &NetworkChoice::Public).unwrap(); + let (_, outcome, denied) = t + .run( + ModuleKind::Environment, + Deployment::Environment(&name), + &vars, + TofuOp::Apply, + t.keys_env(&keys), + ) + .await; + assert_ok("deploy", &outcome, &denied); + + // Detected Ready with a valid record, joinable, not joined yet. + let found = t.scan(&cfg, &who).await; + let env = found + .environments + .iter() + .find(|x| x.name == name) + .unwrap_or_else(|| panic!("{name} not detected: {found:#?}")); + assert_eq!(env.state, EnvironmentState::Ready, "{env:#?}"); + let info = env.info.clone().expect("a valid record"); + assert_eq!(info.network_mode, "public"); + assert!(info.assign_public_ip); + assert_eq!(info.subnet_ids.len(), 2); + assert!(env.actions.contains(&EnvironmentAction::Join), "{env:#?}"); + assert_eq!(env.onboarding, Some(OnboardingState::NotJoined)); + + // A stale lock's conditional delete: a mismatched ETag is refused + // (412), the current one removes it. + { + let s3 = aws_sdk_s3::Client::new(&cfg); + let lock_key = format!("{}.tflock", state_key(Deployment::Environment(&name))); + let put = ok( + "put a fake lock", + s3.put_object() + .bucket(&t.bucket) + .key(&lock_key) + .body(r#"{"ID":"e2e","Operation":"OperationTypeApply","Who":"e2e@test","Created":"2026-01-01T00:00:00Z"}"#.as_bytes().to_vec().into()) + .expected_bucket_owner(&who.account) + .send() + .await, + ); + let etag = put.e_tag().expect("etag").to_string(); + let head = ok( + "head the lock", + s3.head_object().bucket(&t.bucket).key(&lock_key).send().await, + ); + assert_eq!(head.e_tag(), Some(etag.as_str())); + let found = t.scan(&cfg, &who).await; + let env = found.environments.iter().find(|x| x.name == name).unwrap(); + assert!( + matches!(&env.state, EnvironmentState::Deploying { lock: Some(l) } if l.who == "e2e@test"), + "a fresh foreign lock reads as Deploying: {env:#?}" + ); + let wrong = delete_lock_if_unchanged( + &s3, + &t.bucket, + &who.account, + &lock_key, + "\"00000000000000000000000000000000\"", + ) + .await + .expect_err("a mismatched ETag must not delete the lock"); + assert!(wrong.to_string().contains(LOCK_CHANGED), "{wrong}"); + ok( + "the lock survives a mismatched delete", + s3.head_object().bucket(&t.bucket).key(&lock_key).send().await, + ); + delete_lock_if_unchanged(&s3, &t.bucket, &who.account, &lock_key, &etag) + .await + .unwrap_or_else(|e| panic!("conditional delete with the current ETag: {e}")); + let gone = s3.head_object().bucket(&t.bucket).key(&lock_key).send().await; + assert!(gone.is_err(), "the lock is still there after the delete"); + eprintln!("conditional unlock: 412 on a mismatched ETag, deleted with the current one"); + } + + // Self-onboarding as the environment admin. + let (keys, cfg, who) = t.session(&admin).await; + let found = t.scan(&cfg, &who).await; + onboard_refusal(&found, &name).expect("join is offered"); + let owner = validate_owner_id(&who.user_id).expect("owner id"); + let okey = owner_key(&owner); + let (_, outcome, denied) = t + .run( + ModuleKind::UserOnboard, + Deployment::User { env: &name, owner_key: &okey }, + &onboarding_vars(&name, &t.e.region, &owner), + TofuOp::Apply, + t.keys_env(&keys), + ) + .await; + assert_ok("onboard", &outcome, &denied); + let found = t.scan(&cfg, &who).await; + let env = found.environments.iter().find(|x| x.name == name).unwrap(); + assert_eq!(env.onboarding, Some(OnboardingState::Joined), "{env:#?}"); + let (task_role, exec_role) = user_role_names(&name, &okey); + for r in [&task_role, &exec_role] { + let role = ok("get the onboarded role", t.e.iam.get_role().role_name(r).send().await); + let boundary = role + .role() + .and_then(|r| r.permissions_boundary()) + .and_then(|b| b.permissions_boundary_arn()) + .unwrap_or_default() + .to_string(); + assert!(boundary.ends_with(&format!(":policy/{PERMISSIONS_BOUNDARY_NAME}")), "{r}: {boundary}"); + } + + // Update: the recorded network, the same module version: nothing + // to change. + let network = network_from_record(&info, None).expect("network from record"); + let vars = environment_vars(&name, &t.e.region, &network).unwrap(); + let (_, outcome, denied) = t + .run( + ModuleKind::Environment, + Deployment::Environment(&name), + &vars, + TofuOp::Apply, + t.keys_env(&keys), + ) + .await; + assert_ok("update", &outcome, &denied); + let summary = outcome.summary.clone().unwrap_or_default(); + assert!( + summary.contains("0 added, 0 changed, 0 destroyed"), + "the update changed something: {summary}" + ); + + // Remove: the onboarding, then the environment (an admin's + // `tofu destroy` from a bundle; the same module and state). + let (keys, _, _) = t.session(&admin).await; + let (_, outcome, denied) = t + .run( + ModuleKind::UserOnboard, + Deployment::User { env: &name, owner_key: &okey }, + &onboarding_vars(&name, &t.e.region, &owner), + TofuOp::Destroy, + t.keys_env(&keys), + ) + .await; + assert_ok("destroy the onboarding", &outcome, &denied); + let (_, outcome, denied) = t + .run( + ModuleKind::Environment, + Deployment::Environment(&name), + &vars, + TofuOp::Destroy, + t.keys_env(&keys), + ) + .await; + assert_ok("destroy the environment", &outcome, &denied); + + // `tofu destroy` leaves an empty state object, which detection + // must read as no deployment at all. + let state = ok( + "read the destroyed state", + aws_sdk_s3::Client::new(&cfg) + .get_object() + .bucket(&t.bucket) + .key(state_key(Deployment::Environment(&name))) + .send() + .await, + ); + let bytes = state.body.collect().await.unwrap().into_bytes(); + assert!( + is_empty_state(&String::from_utf8_lossy(&bytes)), + "the destroyed state is not empty: {}", + String::from_utf8_lossy(&bytes) + ); + let found = t.scan(&cfg, &who).await; + assert!( + found.environments.iter().all(|x| x.name != name), + "{name} still detected after destroy: {found:#?}" + ); + + // The boundary, by its module, if this test made it. + if *t.boundary_created.lock().unwrap() { + let (_, outcome, denied) = t + .run( + ModuleKind::PermissionsBoundary, + Deployment::Boundary, + &boundary_vars(&t.e.region), + TofuOp::Destroy, + t.admin_env(), + ) + .await; + assert_ok("destroy the boundary", &outcome, &denied); + } + }) + .await; + } + + /// About $0.05: one NAT gateway and its address for a few minutes. + #[tokio::test] + #[ignore] + async fn the_private_nat_mode_deploys() { + env_e2e(|t| async move { + let admin = t.admin().await; + let (keys, cfg, who) = t.session(&admin).await; + t.ensure_bucket(&cfg, &who).await; + let name = format!("e2e{}", &uuid::Uuid::new_v4().simple().to_string()[..6]); + let vars = environment_vars(&name, &t.e.region, &NetworkChoice::PrivateNat).unwrap(); + let (_, outcome, denied) = t + .run( + ModuleKind::Environment, + Deployment::Environment(&name), + &vars, + TofuOp::Apply, + t.keys_env(&keys), + ) + .await; + assert_ok("deploy private-nat", &outcome, &denied); + let found = t.scan(&cfg, &who).await; + let env = found.environments.iter().find(|x| x.name == name).unwrap(); + assert_eq!(env.state, EnvironmentState::Ready, "{env:#?}"); + let info = env.info.as_ref().unwrap(); + assert_eq!(info.network_mode, "private-nat"); + assert!(!info.assign_public_ip); + let (keys, _, _) = t.session(&admin).await; + let (_, outcome, denied) = t + .run( + ModuleKind::Environment, + Deployment::Environment(&name), + &vars, + TofuOp::Destroy, + t.keys_env(&keys), + ) + .await; + assert_ok("destroy private-nat", &outcome, &denied); + }) + .await; + } +} diff --git a/app/src-tauri/src/aws/tofu.rs b/app/src-tauri/src/aws/tofu.rs index 61fc691..3ab431c 100644 --- a/app/src-tauri/src/aws/tofu.rs +++ b/app/src-tauri/src/aws/tofu.rs @@ -706,7 +706,10 @@ pub fn start_detached( /// After the runner takes its lock, how long to wait for its PID file. const PID_GRACE: Duration = Duration::from_secs(2); -fn start_detached_with( +/// [`start_detached`] with the runner program and start timeout given: the +/// unit tests pass a fake runner, the live tests the built app binary (the +/// test harness is not the app, so it can't re-execute itself as one). +pub(crate) fn start_detached_with( program: &Path, timeout: Duration, tofu: &Path, diff --git a/app/src-tauri/src/commands/environment_commands.rs b/app/src-tauri/src/commands/environment_commands.rs index b071f57..17e38cd 100644 --- a/app/src-tauri/src/commands/environment_commands.rs +++ b/app/src-tauri/src/commands/environment_commands.rs @@ -1095,24 +1095,38 @@ pub async fn unlock_aws_state( .map(|t| now.saturating_sub(t.secs()).max(0) as u64) .ok_or_else(|| other("The state lock's age is unknown, so it can't be removed safely"))?; stale_lock_refusal(age).map_err(other)?; + delete_lock_if_unchanged(&s3, &ctx.bucket(), &ctx.who.account, &lock_key, &etag).await?; + log::info!("Removed the stale state lock {lock_key}"); + Ok(()) +} + +/// The lock was replaced since it was read: its ETag no longer matches. +pub(crate) const LOCK_CHANGED: &str = "The state lock changed since it was checked; detect again"; + +/// Delete `lock_key` only if its ETag is still `etag` (S3 conditional +/// delete, `If-Match`); [`LOCK_CHANGED`] when it is not. +pub(crate) async fn delete_lock_if_unchanged( + s3: &aws_sdk_s3::Client, + bucket: &str, + account: &str, + lock_key: &str, + etag: &str, +) -> Result<(), AwsError> { if let Err(e) = s3 .delete_object() - .bucket(ctx.bucket()) - .key(&lock_key) - .if_match(&etag) - .expected_bucket_owner(&ctx.who.account) + .bucket(bucket) + .key(lock_key) + .if_match(etag) + .expected_bucket_owner(account) .send() .await { let status = e.raw_response().map(|r| r.status().as_u16()); if is_precondition_failed(e.code(), status) { - return Err(other( - "The state lock changed since it was checked; detect again", - )); + return Err(other(LOCK_CHANGED)); } return Err(crate::aws::classify_sdk_error("s3:DeleteObject", &e)); } - log::info!("Removed the stale state lock {lock_key}"); Ok(()) } diff --git a/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md b/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md index 69ea4aa..5215ae9 100644 --- a/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md +++ b/docs/superpowers/specs/2026-11-13-aws-ecs-host-design.md @@ -1,7 +1,7 @@ # Triple-C on AWS (ECS on EC2 / Fargate) — design Date: 2026-11-13 -Status: **draft for review** (user decisions recorded 2026-11-13). The user set the direction (local or AWS, login like the AWS CLI, +Status: **draft for review** (user decisions recorded 2026-11-13 and 2026-11-14; phase 4 done). The user set the direction (local or AWS, login like the AWS CLI, permission check, hosting options, secrets pushed automatically, shared accounts). Everything else here is a proposal. @@ -31,6 +31,48 @@ The AWS flow, in the user's words: | Tenancy | **Strict from v1, and the default.** Most people will set this up in their own accounts. Strict costs a single user nothing extra (they onboard themselves automatically), and it means the setup is already safe the day a second person joins | | Connection model | **C, hybrid.** A connection is a login (profile or keys) plus a **default region**. Environments are detected per region when needed; each project records `(connection, region, environment)`. See [Connection model](#connection-model) | +### Decisions (user, 2026-11-14): environments are OpenTofu + +The user chose **Terraform over CloudFormation**, the **app runs it** itself, with **OpenTofu** +(MPL-2.0, safe for the app to download and run), and **state in an S3 bucket** in the account. +Phase 4 added these decisions: + +- **D2. Strict tenancy gives each user their own EFS file system**, created by their onboarding + module. EFS can't bind an access point to its owner's task role (`ClientMount` sees the access + point's ARN, not its tags, and access-point IDs are random), so the file-system policy admits + only that user's task role. An empty file system costs nothing. The environment keeps the shared + pieces: cluster, security group, KMS key, log group and network. +- **D3. No ECR in v1.** Tasks pull `ghcr.io/shadowdao/triple-c-sandbox` straight from GHCR (the + default when an environment has no ECR mirror). +- **D4. Network modes** (`network_mode`): `public` (**default**: a new VPC, two public subnets, + tasks get a public IP, the security group has no inbound rules, no NAT, $0); `private-nat` (a new + VPC with two public and two private subnets and one NAT gateway; the UI says about $33/month); + `existing` (the user's `vpc_id`, exactly two `subnet_ids` in different AZs, and + `assign_public_ip`). +- **D5. An admin request is a folder bundle**, written into a folder picked through a Rust-side + dialog: the module's files, `backend.hcl`, `terraform.tfvars.json` and a `README.txt` with the + exact commands (`tofu init -backend-config=backend.hcl`, then `tofu apply`; Terraform 1.10+ + works too). +- **D6. The app destroys only incomplete deployments** (state without an environment record, or + a failed apply). A working environment is destroyed by an admin with `tofu destroy` from a + bundle, which is the irreversible case. +- **D7. The admin tier spans several policy files** (`environment-admin.json` and + `environment-admin-network.json`), all required; `aws_policy_document` returns every file of a + tier. +- **D8. CloudFormation left the policies**, with its actions, checks and probes. Detection for a + workspace user reads only `ssm:GetParameter*` on `/triple-c/*` and `iam:GetRole` on + `role/triple-c-*`; the bucket (state objects and locks) is admin-only, so a workspace user sees + only environments that have a record. +- **D9. One state key per deployment**, in the per-account, per-region bucket + `triple-c-tfstate--`: `environments//terraform.tfstate`, + `users///terraform.tfstate`, `boundary/terraform.tfstate`. The lock is the + matching `.tflock` object (S3-native locking, `use_lockfile = true`); a held lock is a deploy in + progress, and its `Who`/`Created`/`Operation` are shown. +- **D10. Applies are detached.** `tofu apply` runs in its own process group, output to + `/apply.log`, PID in `/apply.pid`. Closing the app doesn't interrupt it; on the + next run the app sees the live runner or the held lock and reattaches by following the log. It + interrupts only when the user presses Cancel, and OpenTofu then stops gracefully and saves state. + ### Non-goals (v1) - Kubernetes/EKS, other clouds, or remote Docker hosts. The host abstraction below allows them @@ -83,40 +125,48 @@ users don't need the AWS CLI. Only SSO login needs it, until the device flow lan Triple-C looks for existing Triple-C infrastructure in the region in two ways, so it finds an environment even if one of them is missing: -- the SSM parameters `/triple-c/environments/*`, written by the environment stack, whose JSON - value lists the environment's ARNs and IDs; -- CloudFormation stacks tagged `triple-c:component=environment`. This catches a stack whose - parameter was deleted, or one that is half-deployed (`*_IN_PROGRESS`, `*_FAILED`). Those are - shown with their state and a "Repair or delete" action instead of being silently ignored. +- the SSM parameters `/triple-c/environments/*`, written last by the environment module, whose + JSON value (`EnvironmentInfo`, schema 1) lists the environment's ARNs and IDs. The record is + untrusted input: every field is validated and the value capped at 4 KiB; +- the OpenTofu state objects and `.tflock` locks in the state bucket (D9; admins only, D8). This + catches a deployment whose record is missing (a failed or interrupted apply) or one in progress. + A state without a record, a failed apply or a stale lock (over two hours old) is shown as + **Incomplete** with Repair, Unlock or, without a record, Destroy (D6), instead of being silently + ignored; a fresh lock is **Deploying** (Follow when it is this machine's run). An empty state + (what `tofu destroy` leaves behind) counts as no deployment. -Detection also reads the template version each stack was deployed from, so an environment deployed -by an older Triple-C shows **"Update available"**. +The record carries the module version it was deployed from, so an environment deployed by an older +Triple-C shows **"Update available"**. What the user sees depends on what was found and what they're allowed to do: -| Found | Can deploy (stack permissions) | Offered | +| Found | Can deploy (admin tier) | Offered | |---|---|---| | Nothing | Yes | **Deploy a new environment** (the default path for a personal account) | -| Nothing | No | Explanation, plus the policy JSON and a pre-filled CloudFormation console link to send to whoever owns the account | +| Nothing | No | Explanation, plus the policy JSON and an admin bundle (D5) to send to whoever owns the account | | One or more | Yes | **Join** `` (default) or **Deploy a new one** (for example a separate region or cost boundary) | | One or more | No | **Join** `` | -**Deploy** uses the **CloudFormation template that ships with the app** -(`aws/triple-c-environment.yaml`). A CloudFormation stack is auditable, an admin can deploy it -without Triple-C, and deleting the stack removes everything. The app never creates environment -resources one API call at a time. Deploy progress streams from the stack events into the same -progress view used for local image pulls. +**Deploy** applies the **OpenTofu module that ships with the app** (`aws/terraform/environment`) +with the caller's own credentials. The app downloads a pinned OpenTofu (checked against its SHA-256 +before it is unpacked), creates the state bucket if needed, writes a work folder and starts a +detached `tofu init` + `tofu apply` (D10); progress streams from OpenTofu's JSON log into the +deploy view. An admin can apply the same files without Triple-C from a bundle (D5), and +`tofu destroy` removes everything the module created. The app never creates environment resources +one API call at a time. -**Joining in strict tenancy** (the default) also needs the user's **onboarding stack**: their own -task role, EFS access-point policy entry and, if they use EC2, a dedicated capacity provider. The -app checks whether the caller already has one: +**Joining in strict tenancy** (the default) also needs the user's **onboarding** +(`aws/terraform/user-onboard`): their own task and execution roles (with the permissions +boundary) and their own EFS file system (D2), and later, if they use EC2, a dedicated capacity +provider. The app checks whether the caller already has one (their state object, or their task +role): -- **The caller can deploy it** (always true in their own account): Triple-C deploys - `triple-c-user-onboard.yaml` for them as part of joining. For the person who just deployed the +- **The caller can deploy it** (an environment admin; always true in their own account): Triple-C + applies the onboarding module for them as part of joining. For the person who just deployed the environment, this happens in the same flow, so a single user never sees "onboarding" as a step. - **The caller can't**: Triple-C produces an onboarding request for whoever runs the environment — - a pre-filled CloudFormation console link plus the exact parameters — and the user's "Join" stays - pending until the stack exists. The app re-checks whenever the user opens the dialog. + an admin bundle (D5) with the user's `aws:userid` filled in — and the user's "Join" stays + pending until their roles exist. The app re-checks whenever the user opens the dialog. ### 3. Permission check @@ -124,7 +174,7 @@ There are three tiers, and the check reports which one the user is in: | Tier | Can do | |---|---| -| **Environment admin** | Deploy or update the environment stack, onboard users (strict tenancy) | +| **Environment admin** | Deploy or update an environment (the OpenTofu modules), onboard users (strict tenancy) | | **Workspace user** | Create, start, stop, connect to and delete **their own** workspaces in an existing environment | | **Insufficient** | Lists exactly which actions are missing, and offers the IAM policy JSON to send to an admin | @@ -270,7 +320,7 @@ other. - ECR pull-through cache of `ghcr.io/shadowdao/triple-c-sandbox` (GHCR as an upstream needs a GitHub token stored in Secrets Manager), or -- the environment stack's ECR repo, filled by an admin or CI with `crane copy`. +- an environment's ECR repo (not in v1, D3), filled by an admin or CI with `crane copy`. Pulling straight from GHCR over NAT also works, and is the default when the environment has no ECR mirror. Image updates use the existing "new image available" flow: the next start uses the new digest. @@ -390,7 +440,7 @@ before release.** | Mode | For | Task role | EC2 options | Isolation | |---|---|---|---|---| -| **Strict** (**default, v1**) | Everyone, from a solo account to a team | **Per-user** task role, created by the user's onboarding stack (self-deployed, or by an admin) | Privileged and Docker allowed **only on a per-user capacity provider** (dedicated EC2 instances, never shared with another owner) | Per-user IAM all the way down. EFS file system policy allows each access point only to its owner's task role | +| **Strict** (**default, v1**) | Everyone, from a solo account to a team | **Per-user** task role, created by the user's onboarding module (self-deployed, or by an admin) | Privileged and Docker allowed **only on a per-user capacity provider** (dedicated EC2 instances, never shared with another owner) | Per-user IAM all the way down. EFS file system policy allows each access point only to its owner's task role | | **Standard** (opt-in, later) | Large teams that want fewer IAM roles | Shared, minimal (logs, SSM messages) | **Fargate only**, or EC2 with **no** privileged mode, Docker socket or `NET_ADMIN` | Containers are unprivileged and can't mount NFS, can't read secrets, and can't reach each other over the network | There is no separate "personal" mode. A solo user on strict tenancy is simply the only onboarded @@ -422,7 +472,7 @@ fails because of a quota, the app says so and doesn't retry. | Start time | About 30–90 s (image pull every time; SOCI lazy-loading **to evaluate**) | Fast when the image is cached on a warm instance | | Docker access / privileged / VPN | No | Yes, on dedicated capacity | | GPU | No | Yes | -| Ops | None | ASG + capacity provider, defined in the environment stack. The app only picks a capacity provider | +| Ops | None | ASG + capacity provider, defined in the environment module. The app only picks a capacity provider | | Cost when idle | 0 (task stopped) | Instances keep running unless the ASG scales in | | Max size | 16 vCPU / 120 GB | Instance type | @@ -560,19 +610,23 @@ Existing `projects.json` files deserialize unchanged (`host_id` missing → Loca ``` aws/ - triple-c-environment.yaml # CloudFormation: cluster, capacity providers, SG, EFS + policy, - # KMS key, ECR repo / pull-through rule, log group, exec+task roles, - # SSM environment parameter. Params: VPC/subnets (or create), - # tenancy mode, limits, Bedrock-on-task-role. - triple-c-user-onboard.yaml # strict tenancy: per-user task role + capacity provider + README.md # the policies, the modules, running a bundle by hand, live results + terraform/ # OpenTofu modules (hashicorp/aws pinned, .terraform.lock.hcl) + permissions-boundary/ # the account's triple-c-permissions-boundary policy + environment/ # cluster, tasks SG (no ingress), KMS key + alias, log group, + # network (public | private-nat | existing), SSM record (last) + user-onboard/ # per user: task + exec roles (bounded), own EFS file system, + # its SG, two mount targets, file-system policy (D2) policies/ - environment-admin.json workspace-user.json - permissions-boundary.json # every triple-c-* role carries it (phase 3) - task-execution-role.json - task-role-standard.json + environment-admin.json # the admin tier, part 1 (D7) + environment-admin-network.json # the admin tier, part 2: the network + permissions-boundary.json # every triple-c-* role carries it ``` +State lives in `triple-c-tfstate--` (versioned, SSE-S3, public access blocked, +TLS only, tagged `triple-c:component=state`), one object per deployment (D9). + ## Phases 1. **Host abstraction refactor.** `ProjectHost` trait, `DockerHost`, every call site moved. No @@ -582,13 +636,15 @@ aws/ 3. **AWS connection + permission check.** Settings UI, SDK wiring, `GetCallerIdentity`, the tiered simulation and fallback probes, policy files. **Done 2026-11-14**; the policies were verified against real IAM (see "Phase 3: policy verification and rulings"). -4. **Environment + onboarding stacks.** CloudFormation templates for both, detection (SSM - parameters + tagged stacks, including half-deployed and outdated ones), join or deploy, - self-onboarding or an admin request, **strict tenancy**. +4. **Environment + onboarding modules.** OpenTofu modules for both and the boundary, run by the + app (detached) or by an admin from a bundle; detection (SSM records + state objects and locks, + including incomplete, in-progress and outdated ones), join or deploy, self-onboarding or an + admin request, **strict tenancy**. **Done 2026-11-14**; deployed, joined, updated and destroyed + for real as a role holding only the admin policies (see "Phase 4: live verification"). 5. **Fargate workspaces.** Ephemeral and Hybrid storage, Secrets Manager delivery, git bootstrap, start/stop/status, stop guard with WIP push on by default, idle auto-stop, logs. 6. **EFS mode, Copy-at-connect, setup scripts, cost estimate.** -7. **ECS on EC2.** Per-user capacity providers in the onboarding stack, privileged/Docker/VPN on +7. **ECS on EC2.** Per-user capacity providers in the onboarding module, privileged/Docker/VPN on dedicated capacity. 8. **Standard tenancy** (opt-in), only if demand appears. @@ -664,6 +720,29 @@ Left for the phases with real ECS/EFS/EC2 resources: `RunTask`'s `aws:TagKeys` ( `aws:ecs:*` or propagated tags), `iam:PassRole` with `iam:ResourceTag` on a real run, the owner tag on a real task for stop/exec/session, session ownership for an SSO-started session, EFS `CreateAction`/`CreateAccessPoint`/`ClientMount`, and `security-group-rule` authorization. +(Phase 4 has since created security groups and rules for real; see below.) + +### Phase 4: live verification (2026-11-14) + +`app/src-tauri/src/aws/live_tests.rs` (`environment`) applies the modules for real, in us-west-2, +as a role holding **only** the admin files: state bucket, a `public` environment through the real +detached runner, detection, self-onboarding, an update ("No changes"), destroy; and a +`private-nat` environment. What reality changed (details in `aws/README.md`): + +- `s3:CreateBucket` with tags also needs `s3:TagResource`. Its `aws:RequestTag` and + `s3:ResourceAccount` conditions are evaluated and pass. +- `ec2:CreateSecurityGroup` is authorized against the VPC without the request's tags, so the VPC + side of the grant is unconditioned (the group still needs the tag). +- Destroying the NAT gateway's address disassociates it after the association is gone; EC2 + authorizes that against the literal `*/*`, which is all the policy grants (`${*}/${*}`). +- `tofu destroy` leaves an empty state object; detection treats it as no deployment (otherwise a + destroyed environment read as Incomplete for ever). +- Confirmed as designed: EFS sets `aws:CalledViaLast` for mount-target interfaces; the EFS + file-system policy accepted the just-created task role; conditional `DeleteObject` (`If-Match`) + on the versioned bucket returns 412 on a stale ETag. + +Still to verify: the `existing` network mode against a user's own VPC, and Fargate tasks in the +deployed environment (phase 5). ## Connection model @@ -684,7 +763,7 @@ The options: | Several regions or environments, same login | Duplicate connections with the same credentials. Keys are stored twice; SSO shares a token cache, so it's only clutter | One connection | One connection | | Permission check | Once per connection, cached | Per environment, run lazily the first time a project uses it | Per environment, run lazily | | Detection cost | One region | Every region the user might use (`DescribeRegions` + a scan, slow) or a hand-kept list | Default region straight away; other regions only when the user picks one | -| Risk of deploying to the wrong place | Low: the place is in the connection's name | Higher: an easy region-picker mistake leaves a stack in a region nobody looks at | Low: default region preselected, others must be chosen deliberately | +| Risk of deploying to the wrong place | Low: the place is in the connection's name | Higher: an easy region-picker mistake leaves an environment in a region nobody looks at | Low: default region preselected, others must be chosen deliberately | | Moving a project to another region or environment | Re-point the project to another connection | Change a field | Change a field | | Settings export/import | Each connection is self-contained | Projects reference environments that may not exist on the importing machine | Same as B | | Data model / UI | Smallest | Largest (environment picker and per-environment status everywhere) | Medium |