test(aws): live environment lifecycle with OpenTofu as an environment admin; docs
The environment live tests apply the real modules through the real detached runner (the built app binary) as a role holding only the admin files: state bucket, public deploy, scan, self-onboard, update with no changes, destroy; the conditional lock delete (412 on a stale ETag); and a private-nat deploy. A guard destroys what is left as the account admin and deletes the throwaway bucket. CLAUDE.md and the spec describe the OpenTofu design (D2-D10) in place of CloudFormation; phase 4 is done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
shadowdaoandClaude Opus 5.5 committed
1 parent
5f38e0d0d2
commit
ba88323feb
5 files changed
+874
-51
No files matched your search
@@ -327,7 +327,27 @@ docker exec stdout → tokio task → emit("terminal-output-{sessionId}") → li
|
||||
admin credentials for a disposable account; `cargo test aws::live_tests -- --ignored
|
||||
--test-threads=1`) runs the check and real calls against throwaway `triple-c-e2e-*` roles,
|
||||
secrets and keys, and cleans them up. Re-run it after any policy change; the results are in
|
||||
`aws/README.md` ("Live verification").
|
||||
`aws/README.md` ("Live verification"). Its `environment` tests apply the real modules as a role
|
||||
holding only the admin files (`cargo build` first: the runner they start is the built app
|
||||
binary), in a throwaway state bucket.
|
||||
**Environments are OpenTofu modules** (`aws/terraform/{environment,user-onboard,permissions-boundary}`;
|
||||
spec D2–D10). `tofu_modules.rs` embeds them and is the **only** place the shared names are built
|
||||
(environment, `/triple-c/environments/<env>` record, owner key, per-user roles, state bucket
|
||||
`triple-c-tfstate-<account>-<region>`, state keys); its tests pin the HCL to those names, the
|
||||
versions and tags, and `MODULE_CALLS` (every provider call) to the admin policy.
|
||||
`tofu.rs` downloads the pinned OpenTofu (SHA-256 checked **before** extraction), writes a work
|
||||
folder (`materialize`), builds tofu's environment from scratch (`child_env`: no inherited
|
||||
`AWS_*`/`TF_*`, keys only in the environment) and runs `init` + `apply|destroy` **detached
|
||||
(D10)**: the app binary re-executed with `--tofu-runner` (`main.rs` checks that first), in its
|
||||
own session, logging to `apply.log`; liveness is the runner's `apply.lock`, never a PID; closing
|
||||
the app never stops a run and the next start re-follows the log. `environments.rs` is detection
|
||||
(records + state objects + locks; records are untrusted and capped; an empty state, which
|
||||
`tofu destroy` leaves behind, is no deployment) and `ensure_state_bucket`;
|
||||
`commands/environment_commands.rs` sequences ensure_tofu → ensure_state_bucket → materialize →
|
||||
start → follow. **D2: strict tenancy gives each user their own EFS file system** (its policy
|
||||
admits only their task role) from their onboarding module; the environment keeps only the shared
|
||||
pieces. The app destroys only incomplete deployments (D6); a working environment is removed with
|
||||
`tofu destroy` from an admin bundle (D5).
|
||||
- **`web_terminal/`** — Remote terminal access via axum HTTP+WebSocket server:
|
||||
- `server.rs` — Axum server lifecycle (start/stop), serves embedded HTML and handles WS upgrades
|
||||
- `ws_handler.rs` — Per-connection WebSocket handler with JSON protocol, session management, cleanup on disconnect
|
||||
|
||||
@@ -289,6 +289,11 @@ impl E2e {
|
||||
/// An SDK config for a session of `role`, once IAM lets it be assumed
|
||||
/// and its policies have propagated.
|
||||
async fn assume(&self, role: &Role) -> SdkConfig {
|
||||
sdk_config_for_keys(&self.assume_keys(role).await, &self.region)
|
||||
}
|
||||
|
||||
/// A session of `role` as keys (see [`E2e::assume`]).
|
||||
async fn assume_keys(&self, role: &Role) -> AwsKeys {
|
||||
let mut last = None;
|
||||
for _ in 0..20 {
|
||||
match self
|
||||
@@ -309,7 +314,7 @@ impl E2e {
|
||||
// New sessions see a just-attached policy after a few
|
||||
// seconds; denials in that window would be noise.
|
||||
tokio::time::sleep(Duration::from_secs(10)).await;
|
||||
return sdk_config_for_keys(&keys, &self.region);
|
||||
return keys;
|
||||
}
|
||||
Err(e) => {
|
||||
last = Some(classify_sdk_error("sts:AssumeRole", &e).to_string());
|
||||
@@ -957,3 +962,705 @@ async fn the_admin_tier_creates_only_bounded_roles_and_tagged_keys() {
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Phase 4: OpenTofu environments, deployed for real by a role that holds
|
||||
/// only the environment-admin policies.
|
||||
///
|
||||
/// ```sh
|
||||
/// cd app/src-tauri
|
||||
/// cargo build # the runner is the app binary re-executed (see below)
|
||||
/// TRIPLEC_AWS_E2E_PROFILE=… TRIPLEC_AWS_E2E_REGION=us-west-2 \
|
||||
/// cargo test aws::live_tests::environment -- --ignored --test-threads=1
|
||||
/// ```
|
||||
///
|
||||
/// What runs the way the app runs it: `ensure_tofu` (the pinned download),
|
||||
/// `ensure_state_bucket`'s configuration, `materialize`, `child_env` (a keys
|
||||
/// connection for the environment admin, a profile connection for the
|
||||
/// account admin), the detached runner — the **built app binary**
|
||||
/// (`target/debug/triple-c`, or `TRIPLEC_E2E_APP_BIN`) re-executed with
|
||||
/// `--tofu-runner`, because the test harness is not the app and can't
|
||||
/// re-execute itself as one — `follow`, `scan`, the refusal checks the
|
||||
/// commands make, and the conditional lock delete. What doesn't: the Tauri
|
||||
/// commands themselves (they need an `AppHandle` and saved settings), so the
|
||||
/// slot/follower bookkeeping and the event stream are not exercised here.
|
||||
///
|
||||
/// Extra variables:
|
||||
/// - `TRIPLEC_E2E_BUCKET_SUFFIX`: the state bucket is
|
||||
/// `triple-c-tfstate-<account>-<region>-<suffix>` (default: random), so the
|
||||
/// account's real one is never touched;
|
||||
/// - `TRIPLEC_E2E_TOFU_ROOT`: where OpenTofu, the provider cache and the work
|
||||
/// folders go (default: `<tmp>/triple-c-e2e-tofu`, kept between runs).
|
||||
mod environment {
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use aws_config::SdkConfig;
|
||||
|
||||
use super::{e2e, ok, E2e, Role};
|
||||
use crate::aws::environments::{
|
||||
ensure_named_state_bucket, is_empty_state, scan_in_bucket, EnvironmentAction, EnvironmentState, Knowable,
|
||||
OnboardingState, SdkDetectApi,
|
||||
};
|
||||
use crate::aws::policies::{policy_files, Tier, PERMISSIONS_BOUNDARY_NAME};
|
||||
use crate::aws::tofu::{
|
||||
self, child_env, ensure_tofu, follow, materialize, start_detached_with, TofuOp,
|
||||
TofuOutcome, Workdir,
|
||||
};
|
||||
use crate::aws::tofu_modules::{
|
||||
boundary_vars, environment_vars, onboarding_vars, owner_key, state_bucket_name, state_key,
|
||||
user_role_names, Deployment, ModuleKind, NetworkChoice,
|
||||
};
|
||||
use crate::aws::{caller_identity, sdk_config_for_keys, CallerIdentity};
|
||||
use crate::commands::environment_commands::{
|
||||
delete_lock_if_unchanged, deploy_refusal, network_from_record, onboard_refusal,
|
||||
validate_owner_id, LOCK_CHANGED,
|
||||
};
|
||||
use crate::models::host_connection::{AwsAuth, HostConnection, HostKind};
|
||||
use crate::storage::secure::AwsKeys;
|
||||
|
||||
/// The environment admin: a role holding the admin tier's policy files
|
||||
/// as managed policies, and nothing else.
|
||||
struct Admin {
|
||||
role: Role,
|
||||
}
|
||||
|
||||
struct Env {
|
||||
e: Arc<E2e>,
|
||||
root: PathBuf,
|
||||
tofu: PathBuf,
|
||||
app_bin: PathBuf,
|
||||
bucket: String,
|
||||
/// Deployments applied (work folder, still to destroy), newest last.
|
||||
applied: std::sync::Mutex<Vec<(Workdir, bool)>>,
|
||||
/// The test created the boundary (through its module).
|
||||
boundary_created: std::sync::Mutex<bool>,
|
||||
bucket_created: std::sync::Mutex<bool>,
|
||||
}
|
||||
|
||||
fn app_bin() -> PathBuf {
|
||||
if let Ok(p) = std::env::var("TRIPLEC_E2E_APP_BIN") {
|
||||
return PathBuf::from(p);
|
||||
}
|
||||
// target/debug/deps/<test binary> -> target/debug/triple-c
|
||||
let exe = std::env::current_exe().expect("current exe");
|
||||
let dir = exe.parent().and_then(|d| d.parent()).expect("target dir");
|
||||
dir.join(format!("triple-c{}", std::env::consts::EXE_SUFFIX))
|
||||
}
|
||||
|
||||
fn keys_connection(region: &str) -> HostConnection {
|
||||
HostConnection {
|
||||
id: "e2e-envadmin".into(),
|
||||
name: "e2e-envadmin".into(),
|
||||
kind: HostKind::Aws {
|
||||
auth: AwsAuth::Keys {
|
||||
keyring_ref: "unused".into(),
|
||||
},
|
||||
default_region: region.to_string(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn profile_connection(region: &str) -> HostConnection {
|
||||
HostConnection {
|
||||
id: "e2e".into(),
|
||||
name: "e2e".into(),
|
||||
kind: HostKind::Aws {
|
||||
auth: AwsAuth::Profile {
|
||||
name: super::var("TRIPLEC_AWS_E2E_PROFILE"),
|
||||
},
|
||||
default_region: region.to_string(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// `not authorized to perform: <action>` (and the KMS/S3 phrasings) in a
|
||||
/// log message.
|
||||
fn denied_actions(message: &str) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
for marker in ["not authorized to perform: ", "is not authorized to perform "] {
|
||||
let mut rest = message;
|
||||
while let Some(i) = rest.find(marker) {
|
||||
rest = &rest[i + marker.len()..];
|
||||
let action: String = rest
|
||||
.chars()
|
||||
.take_while(|c| c.is_ascii_alphanumeric() || *c == ':' || *c == '*')
|
||||
.collect();
|
||||
if !action.is_empty() && !out.contains(&action) {
|
||||
out.push(action);
|
||||
}
|
||||
}
|
||||
}
|
||||
if out.is_empty()
|
||||
&& (message.contains("AccessDenied") || message.contains("UnauthorizedOperation"))
|
||||
{
|
||||
out.push("(unnamed: see the message above)".into());
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
impl Env {
|
||||
async fn new(e: Arc<E2e>) -> Arc<Env> {
|
||||
let root = std::env::var("TRIPLEC_E2E_TOFU_ROOT")
|
||||
.map(PathBuf::from)
|
||||
.unwrap_or_else(|_| std::env::temp_dir().join("triple-c-e2e-tofu"));
|
||||
std::fs::create_dir_all(&root).unwrap();
|
||||
let app_bin = app_bin();
|
||||
assert!(
|
||||
app_bin.is_file(),
|
||||
"{} is missing: run `cargo build` first (the runner is the app binary)",
|
||||
app_bin.display()
|
||||
);
|
||||
let tofu = ensure_tofu(&root).await.expect("ensure_tofu");
|
||||
let suffix = std::env::var("TRIPLEC_E2E_BUCKET_SUFFIX").unwrap_or_else(|_| {
|
||||
format!("e2e{}", &uuid::Uuid::new_v4().simple().to_string()[..8])
|
||||
});
|
||||
let bucket = format!("{}-{suffix}", state_bucket_name(&e.account, &e.region));
|
||||
assert!(bucket.len() <= 63, "{bucket} is too long for a bucket name");
|
||||
Arc::new(Env {
|
||||
e,
|
||||
root,
|
||||
tofu,
|
||||
app_bin,
|
||||
bucket,
|
||||
applied: Default::default(),
|
||||
boundary_created: Default::default(),
|
||||
bucket_created: Default::default(),
|
||||
})
|
||||
}
|
||||
|
||||
/// The environment admin role (no boundary on the role itself: the
|
||||
/// boundary allows only what tasks do, so an admin under it could
|
||||
/// administer nothing).
|
||||
async fn admin(&self) -> Admin {
|
||||
let role = self.e.role("envadmin", None, &[]).await;
|
||||
for (i, (file, json)) in policy_files(Tier::EnvironmentAdmin).iter().enumerate() {
|
||||
let arn = self.e.managed_policy(&format!("envadmin{i}"), json).await;
|
||||
eprintln!("envadmin: {file} -> {arn}");
|
||||
self.e.attach(&role, &arn).await;
|
||||
}
|
||||
Admin { role }
|
||||
}
|
||||
|
||||
/// A fresh session of the admin role (role chaining caps one at an
|
||||
/// hour, so each phase takes its own).
|
||||
async fn session(&self, admin: &Admin) -> (AwsKeys, SdkConfig, CallerIdentity) {
|
||||
let keys = self.e.assume_keys(&admin.role).await;
|
||||
let cfg = sdk_config_for_keys(&keys, &self.e.region);
|
||||
let who = caller_identity(&cfg).await.expect("envadmin identity");
|
||||
(keys, cfg, who)
|
||||
}
|
||||
|
||||
async fn scan(&self, cfg: &SdkConfig, who: &CallerIdentity) -> crate::aws::environments::EnvironmentScan {
|
||||
let api = SdkDetectApi::new(cfg, &who.account);
|
||||
let (root, bucket) = (self.root.clone(), self.bucket.clone());
|
||||
let running = move |d: Deployment| {
|
||||
tofu::workdir_for(&root, d, &bucket)
|
||||
.map(|wd| tofu::is_running(&wd))
|
||||
.unwrap_or(false)
|
||||
};
|
||||
scan_in_bucket(&api, who, &self.e.region, &self.bucket, &running)
|
||||
.await
|
||||
.expect("scan")
|
||||
}
|
||||
|
||||
/// Write the module and run `op` through the detached runner, printing
|
||||
/// the log and every denied action.
|
||||
async fn run(
|
||||
&self,
|
||||
kind: ModuleKind,
|
||||
d: Deployment<'_>,
|
||||
vars: &serde_json::Value,
|
||||
op: TofuOp,
|
||||
env: Vec<(String, String)>,
|
||||
) -> (Workdir, TofuOutcome, Vec<String>) {
|
||||
let wd = materialize(&self.root, kind, d, &self.bucket, &self.e.region, vars)
|
||||
.expect("materialize");
|
||||
if op == TofuOp::Apply {
|
||||
let mut applied = self.applied.lock().unwrap();
|
||||
if !applied.iter().any(|(w, _)| *w == wd) {
|
||||
applied.push((wd.clone(), true));
|
||||
}
|
||||
}
|
||||
let label = state_key(d);
|
||||
eprintln!("=== tofu {op:?} {label} ({})", wd.dir.display());
|
||||
let (bin, tofu_bin, start_wd) = (self.app_bin.clone(), self.tofu.clone(), wd.clone());
|
||||
let started = tokio::task::spawn_blocking(move || {
|
||||
start_detached_with(&bin, Duration::from_secs(30), &tofu_bin, &start_wd, op, &env)
|
||||
})
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap_or_else(|e| panic!("start {label}: {e}"));
|
||||
assert!(started.pid.is_some(), "the runner wrote no PID");
|
||||
let mut denied = Vec::new();
|
||||
let outcome = follow(&wd, Duration::from_millis(500), |events| {
|
||||
for ev in events {
|
||||
let resource = ev.resource.as_deref().unwrap_or("");
|
||||
let short: String = ev.message.chars().take(300).collect();
|
||||
eprintln!(" [{}] {} {} {}", ev.level, ev.kind, resource, short);
|
||||
if ev.level == "error" {
|
||||
eprintln!(" ERROR DETAIL: {}", ev.message);
|
||||
for a in denied_actions(&ev.message) {
|
||||
eprintln!(" DENIED: {a} ({resource})");
|
||||
if !denied.contains(&a) {
|
||||
denied.push(a);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("follow");
|
||||
eprintln!("=== {label}: {outcome:?}");
|
||||
if op == TofuOp::Destroy && outcome.ok {
|
||||
for (w, pending) in self.applied.lock().unwrap().iter_mut() {
|
||||
if *w == wd {
|
||||
*pending = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
(wd, outcome, denied)
|
||||
}
|
||||
|
||||
fn keys_env(&self, keys: &AwsKeys) -> Vec<(String, String)> {
|
||||
child_env(
|
||||
&keys_connection(&self.e.region),
|
||||
Some(keys),
|
||||
&self.root,
|
||||
std::env::vars(),
|
||||
)
|
||||
.expect("child env")
|
||||
}
|
||||
|
||||
fn admin_env(&self) -> Vec<(String, String)> {
|
||||
child_env(&profile_connection(&self.e.region), None, &self.root, std::env::vars())
|
||||
.expect("child env")
|
||||
}
|
||||
|
||||
async fn ensure_bucket(&self, cfg: &SdkConfig, who: &CallerIdentity) {
|
||||
*self.bucket_created.lock().unwrap() = true;
|
||||
let b = ensure_named_state_bucket(cfg, &who.account, &self.e.region, &self.bucket)
|
||||
.await
|
||||
.unwrap_or_else(|e| panic!("ensure_state_bucket as the environment admin: {e}"));
|
||||
assert_eq!(b, self.bucket);
|
||||
}
|
||||
|
||||
/// The boundary, through its module as the account admin, if the
|
||||
/// account has none.
|
||||
async fn ensure_boundary(&self) {
|
||||
let arn = format!(
|
||||
"arn:aws:iam::{}:policy/{PERMISSIONS_BOUNDARY_NAME}",
|
||||
self.e.account
|
||||
);
|
||||
if self.e.iam.get_policy().policy_arn(&arn).send().await.is_ok() {
|
||||
eprintln!("the account already has the boundary; using it");
|
||||
return;
|
||||
}
|
||||
*self.boundary_created.lock().unwrap() = true;
|
||||
let (_, outcome, denied) = self
|
||||
.run(
|
||||
ModuleKind::PermissionsBoundary,
|
||||
Deployment::Boundary,
|
||||
&boundary_vars(&self.e.region),
|
||||
TofuOp::Apply,
|
||||
self.admin_env(),
|
||||
)
|
||||
.await;
|
||||
assert!(outcome.ok, "boundary apply: {outcome:?} denied {denied:?}");
|
||||
}
|
||||
|
||||
/// As the account admin: destroy whatever is still applied (newest
|
||||
/// first), the bucket with every version, and the boundary if this
|
||||
/// test made it. Logs rather than fails, so one error strands nothing
|
||||
/// else.
|
||||
async fn cleanup(&self) {
|
||||
let pending: Vec<Workdir> = self
|
||||
.applied
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.rev()
|
||||
.filter(|(_, p)| *p)
|
||||
.map(|(w, _)| w.clone())
|
||||
.collect();
|
||||
for wd in pending {
|
||||
eprintln!("cleanup: destroying {} as the account admin", wd.dir.display());
|
||||
let env = self.admin_env();
|
||||
let (bin, tofu_bin, w) = (self.app_bin.clone(), self.tofu.clone(), wd.clone());
|
||||
let started = tokio::task::spawn_blocking(move || {
|
||||
start_detached_with(&bin, Duration::from_secs(30), &tofu_bin, &w, TofuOp::Destroy, &env)
|
||||
})
|
||||
.await;
|
||||
match started {
|
||||
Ok(Ok(_)) => {
|
||||
let outcome = follow(&wd, Duration::from_millis(500), |events| {
|
||||
for ev in events.iter().filter(|e| e.level == "error") {
|
||||
eprintln!(" cleanup error: {}", ev.message);
|
||||
}
|
||||
})
|
||||
.await;
|
||||
eprintln!("cleanup: destroy {}: {outcome:?}", wd.dir.display());
|
||||
}
|
||||
other => eprintln!("cleanup: could not start destroy: {other:?}"),
|
||||
}
|
||||
}
|
||||
if *self.boundary_created.lock().unwrap() {
|
||||
let arn = format!(
|
||||
"arn:aws:iam::{}:policy/{PERMISSIONS_BOUNDARY_NAME}",
|
||||
self.e.account
|
||||
);
|
||||
if self.e.iam.get_policy().policy_arn(&arn).send().await.is_ok() {
|
||||
eprintln!("cleanup: deleting the boundary directly");
|
||||
if let Ok(out) = self.e.iam.list_entities_for_policy().policy_arn(&arn).send().await {
|
||||
for r in out.policy_roles() {
|
||||
if let Some(name) = r.role_name() {
|
||||
let _ = self
|
||||
.e
|
||||
.iam
|
||||
.delete_role_permissions_boundary()
|
||||
.role_name(name)
|
||||
.send()
|
||||
.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Err(e) = self.e.iam.delete_policy().policy_arn(&arn).send().await {
|
||||
eprintln!("cleanup: boundary: {e:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
if *self.bucket_created.lock().unwrap() {
|
||||
self.delete_bucket().await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn delete_bucket(&self) {
|
||||
let s3 = aws_sdk_s3::Client::new(&self.e.cfg);
|
||||
loop {
|
||||
let page = match s3.list_object_versions().bucket(&self.bucket).send().await {
|
||||
Ok(p) => p,
|
||||
Err(e) => {
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
if e.code() != Some("NoSuchBucket") {
|
||||
eprintln!("cleanup: list versions of {}: {e:?}", self.bucket);
|
||||
}
|
||||
return;
|
||||
}
|
||||
};
|
||||
let mut ids = Vec::new();
|
||||
for v in page.versions() {
|
||||
ids.push((v.key().unwrap_or_default().to_string(), v.version_id().map(str::to_string)));
|
||||
}
|
||||
for m in page.delete_markers() {
|
||||
ids.push((m.key().unwrap_or_default().to_string(), m.version_id().map(str::to_string)));
|
||||
}
|
||||
if ids.is_empty() {
|
||||
break;
|
||||
}
|
||||
for (key, version) in ids {
|
||||
if let Err(e) = s3
|
||||
.delete_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.set_version_id(version)
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
eprintln!("cleanup: delete {key}: {e:?}");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
match s3.delete_bucket().bucket(&self.bucket).send().await {
|
||||
Ok(_) => eprintln!("cleanup: deleted bucket {}", self.bucket),
|
||||
Err(e) => eprintln!("cleanup: delete bucket {}: {e:?}", self.bucket),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `e2e` plus the environment fixture's own cleanup, which runs first
|
||||
/// (it needs the account admin, and the roles it destroys hold the
|
||||
/// boundary).
|
||||
async fn env_e2e<F, Fut>(body: F)
|
||||
where
|
||||
F: FnOnce(Arc<Env>) -> Fut,
|
||||
Fut: std::future::Future<Output = ()>,
|
||||
{
|
||||
use futures_util::FutureExt;
|
||||
e2e(|e| async move {
|
||||
let env = Env::new(e).await;
|
||||
let result = std::panic::AssertUnwindSafe(body(env.clone()))
|
||||
.catch_unwind()
|
||||
.await;
|
||||
env.cleanup().await;
|
||||
if let Err(panic) = result {
|
||||
std::panic::resume_unwind(panic);
|
||||
}
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
fn assert_ok(what: &str, outcome: &TofuOutcome, denied: &[String]) {
|
||||
assert!(
|
||||
outcome.ok,
|
||||
"{what} failed: {:?}; denied actions: {denied:?}",
|
||||
outcome.summary
|
||||
);
|
||||
assert!(denied.is_empty(), "{what} reported denied actions: {denied:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore]
|
||||
async fn an_environment_admin_can_deploy_join_update_and_remove_an_environment() {
|
||||
env_e2e(|t| async move {
|
||||
let admin = t.admin().await;
|
||||
let (keys, cfg, who) = t.session(&admin).await;
|
||||
assert!(who.arn.contains(":assumed-role/"), "{}", who.arn);
|
||||
|
||||
// The state bucket, created and configured by the environment
|
||||
// admin (s3:CreateBucket with the request tag, in-account).
|
||||
t.ensure_bucket(&cfg, &who).await;
|
||||
// Idempotent.
|
||||
t.ensure_bucket(&cfg, &who).await;
|
||||
|
||||
t.ensure_boundary().await;
|
||||
|
||||
let name = format!("e2e{}", &uuid::Uuid::new_v4().simple().to_string()[..6]);
|
||||
let found = t.scan(&cfg, &who).await;
|
||||
assert_eq!(found.state_bucket, Knowable::Yes);
|
||||
assert_eq!(found.boundary, Knowable::Yes);
|
||||
deploy_refusal(&found, &name).expect("deploy is offered");
|
||||
|
||||
// Deploy, public network, as the environment admin only.
|
||||
let vars = environment_vars(&name, &t.e.region, &NetworkChoice::Public).unwrap();
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::Environment,
|
||||
Deployment::Environment(&name),
|
||||
&vars,
|
||||
TofuOp::Apply,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("deploy", &outcome, &denied);
|
||||
|
||||
// Detected Ready with a valid record, joinable, not joined yet.
|
||||
let found = t.scan(&cfg, &who).await;
|
||||
let env = found
|
||||
.environments
|
||||
.iter()
|
||||
.find(|x| x.name == name)
|
||||
.unwrap_or_else(|| panic!("{name} not detected: {found:#?}"));
|
||||
assert_eq!(env.state, EnvironmentState::Ready, "{env:#?}");
|
||||
let info = env.info.clone().expect("a valid record");
|
||||
assert_eq!(info.network_mode, "public");
|
||||
assert!(info.assign_public_ip);
|
||||
assert_eq!(info.subnet_ids.len(), 2);
|
||||
assert!(env.actions.contains(&EnvironmentAction::Join), "{env:#?}");
|
||||
assert_eq!(env.onboarding, Some(OnboardingState::NotJoined));
|
||||
|
||||
// A stale lock's conditional delete: a mismatched ETag is refused
|
||||
// (412), the current one removes it.
|
||||
{
|
||||
let s3 = aws_sdk_s3::Client::new(&cfg);
|
||||
let lock_key = format!("{}.tflock", state_key(Deployment::Environment(&name)));
|
||||
let put = ok(
|
||||
"put a fake lock",
|
||||
s3.put_object()
|
||||
.bucket(&t.bucket)
|
||||
.key(&lock_key)
|
||||
.body(r#"{"ID":"e2e","Operation":"OperationTypeApply","Who":"e2e@test","Created":"2026-01-01T00:00:00Z"}"#.as_bytes().to_vec().into())
|
||||
.expected_bucket_owner(&who.account)
|
||||
.send()
|
||||
.await,
|
||||
);
|
||||
let etag = put.e_tag().expect("etag").to_string();
|
||||
let head = ok(
|
||||
"head the lock",
|
||||
s3.head_object().bucket(&t.bucket).key(&lock_key).send().await,
|
||||
);
|
||||
assert_eq!(head.e_tag(), Some(etag.as_str()));
|
||||
let found = t.scan(&cfg, &who).await;
|
||||
let env = found.environments.iter().find(|x| x.name == name).unwrap();
|
||||
assert!(
|
||||
matches!(&env.state, EnvironmentState::Deploying { lock: Some(l) } if l.who == "e2e@test"),
|
||||
"a fresh foreign lock reads as Deploying: {env:#?}"
|
||||
);
|
||||
let wrong = delete_lock_if_unchanged(
|
||||
&s3,
|
||||
&t.bucket,
|
||||
&who.account,
|
||||
&lock_key,
|
||||
"\"00000000000000000000000000000000\"",
|
||||
)
|
||||
.await
|
||||
.expect_err("a mismatched ETag must not delete the lock");
|
||||
assert!(wrong.to_string().contains(LOCK_CHANGED), "{wrong}");
|
||||
ok(
|
||||
"the lock survives a mismatched delete",
|
||||
s3.head_object().bucket(&t.bucket).key(&lock_key).send().await,
|
||||
);
|
||||
delete_lock_if_unchanged(&s3, &t.bucket, &who.account, &lock_key, &etag)
|
||||
.await
|
||||
.unwrap_or_else(|e| panic!("conditional delete with the current ETag: {e}"));
|
||||
let gone = s3.head_object().bucket(&t.bucket).key(&lock_key).send().await;
|
||||
assert!(gone.is_err(), "the lock is still there after the delete");
|
||||
eprintln!("conditional unlock: 412 on a mismatched ETag, deleted with the current one");
|
||||
}
|
||||
|
||||
// Self-onboarding as the environment admin.
|
||||
let (keys, cfg, who) = t.session(&admin).await;
|
||||
let found = t.scan(&cfg, &who).await;
|
||||
onboard_refusal(&found, &name).expect("join is offered");
|
||||
let owner = validate_owner_id(&who.user_id).expect("owner id");
|
||||
let okey = owner_key(&owner);
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::UserOnboard,
|
||||
Deployment::User { env: &name, owner_key: &okey },
|
||||
&onboarding_vars(&name, &t.e.region, &owner),
|
||||
TofuOp::Apply,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("onboard", &outcome, &denied);
|
||||
let found = t.scan(&cfg, &who).await;
|
||||
let env = found.environments.iter().find(|x| x.name == name).unwrap();
|
||||
assert_eq!(env.onboarding, Some(OnboardingState::Joined), "{env:#?}");
|
||||
let (task_role, exec_role) = user_role_names(&name, &okey);
|
||||
for r in [&task_role, &exec_role] {
|
||||
let role = ok("get the onboarded role", t.e.iam.get_role().role_name(r).send().await);
|
||||
let boundary = role
|
||||
.role()
|
||||
.and_then(|r| r.permissions_boundary())
|
||||
.and_then(|b| b.permissions_boundary_arn())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
assert!(boundary.ends_with(&format!(":policy/{PERMISSIONS_BOUNDARY_NAME}")), "{r}: {boundary}");
|
||||
}
|
||||
|
||||
// Update: the recorded network, the same module version: nothing
|
||||
// to change.
|
||||
let network = network_from_record(&info, None).expect("network from record");
|
||||
let vars = environment_vars(&name, &t.e.region, &network).unwrap();
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::Environment,
|
||||
Deployment::Environment(&name),
|
||||
&vars,
|
||||
TofuOp::Apply,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("update", &outcome, &denied);
|
||||
let summary = outcome.summary.clone().unwrap_or_default();
|
||||
assert!(
|
||||
summary.contains("0 added, 0 changed, 0 destroyed"),
|
||||
"the update changed something: {summary}"
|
||||
);
|
||||
|
||||
// Remove: the onboarding, then the environment (an admin's
|
||||
// `tofu destroy` from a bundle; the same module and state).
|
||||
let (keys, _, _) = t.session(&admin).await;
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::UserOnboard,
|
||||
Deployment::User { env: &name, owner_key: &okey },
|
||||
&onboarding_vars(&name, &t.e.region, &owner),
|
||||
TofuOp::Destroy,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("destroy the onboarding", &outcome, &denied);
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::Environment,
|
||||
Deployment::Environment(&name),
|
||||
&vars,
|
||||
TofuOp::Destroy,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("destroy the environment", &outcome, &denied);
|
||||
|
||||
// `tofu destroy` leaves an empty state object, which detection
|
||||
// must read as no deployment at all.
|
||||
let state = ok(
|
||||
"read the destroyed state",
|
||||
aws_sdk_s3::Client::new(&cfg)
|
||||
.get_object()
|
||||
.bucket(&t.bucket)
|
||||
.key(state_key(Deployment::Environment(&name)))
|
||||
.send()
|
||||
.await,
|
||||
);
|
||||
let bytes = state.body.collect().await.unwrap().into_bytes();
|
||||
assert!(
|
||||
is_empty_state(&String::from_utf8_lossy(&bytes)),
|
||||
"the destroyed state is not empty: {}",
|
||||
String::from_utf8_lossy(&bytes)
|
||||
);
|
||||
let found = t.scan(&cfg, &who).await;
|
||||
assert!(
|
||||
found.environments.iter().all(|x| x.name != name),
|
||||
"{name} still detected after destroy: {found:#?}"
|
||||
);
|
||||
|
||||
// The boundary, by its module, if this test made it.
|
||||
if *t.boundary_created.lock().unwrap() {
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::PermissionsBoundary,
|
||||
Deployment::Boundary,
|
||||
&boundary_vars(&t.e.region),
|
||||
TofuOp::Destroy,
|
||||
t.admin_env(),
|
||||
)
|
||||
.await;
|
||||
assert_ok("destroy the boundary", &outcome, &denied);
|
||||
}
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
/// About $0.05: one NAT gateway and its address for a few minutes.
|
||||
#[tokio::test]
|
||||
#[ignore]
|
||||
async fn the_private_nat_mode_deploys() {
|
||||
env_e2e(|t| async move {
|
||||
let admin = t.admin().await;
|
||||
let (keys, cfg, who) = t.session(&admin).await;
|
||||
t.ensure_bucket(&cfg, &who).await;
|
||||
let name = format!("e2e{}", &uuid::Uuid::new_v4().simple().to_string()[..6]);
|
||||
let vars = environment_vars(&name, &t.e.region, &NetworkChoice::PrivateNat).unwrap();
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::Environment,
|
||||
Deployment::Environment(&name),
|
||||
&vars,
|
||||
TofuOp::Apply,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("deploy private-nat", &outcome, &denied);
|
||||
let found = t.scan(&cfg, &who).await;
|
||||
let env = found.environments.iter().find(|x| x.name == name).unwrap();
|
||||
assert_eq!(env.state, EnvironmentState::Ready, "{env:#?}");
|
||||
let info = env.info.as_ref().unwrap();
|
||||
assert_eq!(info.network_mode, "private-nat");
|
||||
assert!(!info.assign_public_ip);
|
||||
let (keys, _, _) = t.session(&admin).await;
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::Environment,
|
||||
Deployment::Environment(&name),
|
||||
&vars,
|
||||
TofuOp::Destroy,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("destroy private-nat", &outcome, &denied);
|
||||
})
|
||||
.await;
|
||||
}
|
||||
}
|
||||
@@ -706,7 +706,10 @@ pub fn start_detached(
|
||||
/// After the runner takes its lock, how long to wait for its PID file.
|
||||
const PID_GRACE: Duration = Duration::from_secs(2);
|
||||
|
||||
fn start_detached_with(
|
||||
/// [`start_detached`] with the runner program and start timeout given: the
|
||||
/// unit tests pass a fake runner, the live tests the built app binary (the
|
||||
/// test harness is not the app, so it can't re-execute itself as one).
|
||||
pub(crate) fn start_detached_with(
|
||||
program: &Path,
|
||||
timeout: Duration,
|
||||
tofu: &Path,
|
||||
|
||||
@@ -1095,24 +1095,38 @@ pub async fn unlock_aws_state(
|
||||
.map(|t| now.saturating_sub(t.secs()).max(0) as u64)
|
||||
.ok_or_else(|| other("The state lock's age is unknown, so it can't be removed safely"))?;
|
||||
stale_lock_refusal(age).map_err(other)?;
|
||||
delete_lock_if_unchanged(&s3, &ctx.bucket(), &ctx.who.account, &lock_key, &etag).await?;
|
||||
log::info!("Removed the stale state lock {lock_key}");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The lock was replaced since it was read: its ETag no longer matches.
|
||||
pub(crate) const LOCK_CHANGED: &str = "The state lock changed since it was checked; detect again";
|
||||
|
||||
/// Delete `lock_key` only if its ETag is still `etag` (S3 conditional
|
||||
/// delete, `If-Match`); [`LOCK_CHANGED`] when it is not.
|
||||
pub(crate) async fn delete_lock_if_unchanged(
|
||||
s3: &aws_sdk_s3::Client,
|
||||
bucket: &str,
|
||||
account: &str,
|
||||
lock_key: &str,
|
||||
etag: &str,
|
||||
) -> Result<(), AwsError> {
|
||||
if let Err(e) = s3
|
||||
.delete_object()
|
||||
.bucket(ctx.bucket())
|
||||
.key(&lock_key)
|
||||
.if_match(&etag)
|
||||
.expected_bucket_owner(&ctx.who.account)
|
||||
.bucket(bucket)
|
||||
.key(lock_key)
|
||||
.if_match(etag)
|
||||
.expected_bucket_owner(account)
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
let status = e.raw_response().map(|r| r.status().as_u16());
|
||||
if is_precondition_failed(e.code(), status) {
|
||||
return Err(other(
|
||||
"The state lock changed since it was checked; detect again",
|
||||
));
|
||||
return Err(other(LOCK_CHANGED));
|
||||
}
|
||||
return Err(crate::aws::classify_sdk_error("s3:DeleteObject", &e));
|
||||
}
|
||||
log::info!("Removed the stale state lock {lock_key}");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Triple-C on AWS (ECS on EC2 / Fargate) — design
|
||||
|
||||
Date: 2026-11-13
|
||||
Status: **draft for review** (user decisions recorded 2026-11-13). The user set the direction (local or AWS, login like the AWS CLI,
|
||||
Status: **draft for review** (user decisions recorded 2026-11-13 and 2026-11-14; phase 4 done). The user set the direction (local or AWS, login like the AWS CLI,
|
||||
permission check, hosting options, secrets pushed automatically, shared accounts). Everything else
|
||||
here is a proposal.
|
||||
|
||||
@@ -31,6 +31,48 @@ The AWS flow, in the user's words:
|
||||
| Tenancy | **Strict from v1, and the default.** Most people will set this up in their own accounts. Strict costs a single user nothing extra (they onboard themselves automatically), and it means the setup is already safe the day a second person joins |
|
||||
| Connection model | **C, hybrid.** A connection is a login (profile or keys) plus a **default region**. Environments are detected per region when needed; each project records `(connection, region, environment)`. See [Connection model](#connection-model) |
|
||||
|
||||
### Decisions (user, 2026-11-14): environments are OpenTofu
|
||||
|
||||
The user chose **Terraform over CloudFormation**, the **app runs it** itself, with **OpenTofu**
|
||||
(MPL-2.0, safe for the app to download and run), and **state in an S3 bucket** in the account.
|
||||
Phase 4 added these decisions:
|
||||
|
||||
- **D2. Strict tenancy gives each user their own EFS file system**, created by their onboarding
|
||||
module. EFS can't bind an access point to its owner's task role (`ClientMount` sees the access
|
||||
point's ARN, not its tags, and access-point IDs are random), so the file-system policy admits
|
||||
only that user's task role. An empty file system costs nothing. The environment keeps the shared
|
||||
pieces: cluster, security group, KMS key, log group and network.
|
||||
- **D3. No ECR in v1.** Tasks pull `ghcr.io/shadowdao/triple-c-sandbox` straight from GHCR (the
|
||||
default when an environment has no ECR mirror).
|
||||
- **D4. Network modes** (`network_mode`): `public` (**default**: a new VPC, two public subnets,
|
||||
tasks get a public IP, the security group has no inbound rules, no NAT, $0); `private-nat` (a new
|
||||
VPC with two public and two private subnets and one NAT gateway; the UI says about $33/month);
|
||||
`existing` (the user's `vpc_id`, exactly two `subnet_ids` in different AZs, and
|
||||
`assign_public_ip`).
|
||||
- **D5. An admin request is a folder bundle**, written into a folder picked through a Rust-side
|
||||
dialog: the module's files, `backend.hcl`, `terraform.tfvars.json` and a `README.txt` with the
|
||||
exact commands (`tofu init -backend-config=backend.hcl`, then `tofu apply`; Terraform 1.10+
|
||||
works too).
|
||||
- **D6. The app destroys only incomplete deployments** (state without an environment record, or
|
||||
a failed apply). A working environment is destroyed by an admin with `tofu destroy` from a
|
||||
bundle, which is the irreversible case.
|
||||
- **D7. The admin tier spans several policy files** (`environment-admin.json` and
|
||||
`environment-admin-network.json`), all required; `aws_policy_document` returns every file of a
|
||||
tier.
|
||||
- **D8. CloudFormation left the policies**, with its actions, checks and probes. Detection for a
|
||||
workspace user reads only `ssm:GetParameter*` on `/triple-c/*` and `iam:GetRole` on
|
||||
`role/triple-c-*`; the bucket (state objects and locks) is admin-only, so a workspace user sees
|
||||
only environments that have a record.
|
||||
- **D9. One state key per deployment**, in the per-account, per-region bucket
|
||||
`triple-c-tfstate-<account>-<region>`: `environments/<env>/terraform.tfstate`,
|
||||
`users/<env>/<ownerkey>/terraform.tfstate`, `boundary/terraform.tfstate`. The lock is the
|
||||
matching `.tflock` object (S3-native locking, `use_lockfile = true`); a held lock is a deploy in
|
||||
progress, and its `Who`/`Created`/`Operation` are shown.
|
||||
- **D10. Applies are detached.** `tofu apply` runs in its own process group, output to
|
||||
`<workdir>/apply.log`, PID in `<workdir>/apply.pid`. Closing the app doesn't interrupt it; on the
|
||||
next run the app sees the live runner or the held lock and reattaches by following the log. It
|
||||
interrupts only when the user presses Cancel, and OpenTofu then stops gracefully and saves state.
|
||||
|
||||
### Non-goals (v1)
|
||||
|
||||
- Kubernetes/EKS, other clouds, or remote Docker hosts. The host abstraction below allows them
|
||||
@@ -83,40 +125,48 @@ users don't need the AWS CLI. Only SSO login needs it, until the device flow lan
|
||||
Triple-C looks for existing Triple-C infrastructure in the region in two ways, so it finds an
|
||||
environment even if one of them is missing:
|
||||
|
||||
- the SSM parameters `/triple-c/environments/*`, written by the environment stack, whose JSON
|
||||
value lists the environment's ARNs and IDs;
|
||||
- CloudFormation stacks tagged `triple-c:component=environment`. This catches a stack whose
|
||||
parameter was deleted, or one that is half-deployed (`*_IN_PROGRESS`, `*_FAILED`). Those are
|
||||
shown with their state and a "Repair or delete" action instead of being silently ignored.
|
||||
- the SSM parameters `/triple-c/environments/*`, written last by the environment module, whose
|
||||
JSON value (`EnvironmentInfo`, schema 1) lists the environment's ARNs and IDs. The record is
|
||||
untrusted input: every field is validated and the value capped at 4 KiB;
|
||||
- the OpenTofu state objects and `.tflock` locks in the state bucket (D9; admins only, D8). This
|
||||
catches a deployment whose record is missing (a failed or interrupted apply) or one in progress.
|
||||
A state without a record, a failed apply or a stale lock (over two hours old) is shown as
|
||||
**Incomplete** with Repair, Unlock or, without a record, Destroy (D6), instead of being silently
|
||||
ignored; a fresh lock is **Deploying** (Follow when it is this machine's run). An empty state
|
||||
(what `tofu destroy` leaves behind) counts as no deployment.
|
||||
|
||||
Detection also reads the template version each stack was deployed from, so an environment deployed
|
||||
by an older Triple-C shows **"Update available"**.
|
||||
The record carries the module version it was deployed from, so an environment deployed by an older
|
||||
Triple-C shows **"Update available"**.
|
||||
|
||||
What the user sees depends on what was found and what they're allowed to do:
|
||||
|
||||
| Found | Can deploy (stack permissions) | Offered |
|
||||
| Found | Can deploy (admin tier) | Offered |
|
||||
|---|---|---|
|
||||
| Nothing | Yes | **Deploy a new environment** (the default path for a personal account) |
|
||||
| Nothing | No | Explanation, plus the policy JSON and a pre-filled CloudFormation console link to send to whoever owns the account |
|
||||
| Nothing | No | Explanation, plus the policy JSON and an admin bundle (D5) to send to whoever owns the account |
|
||||
| One or more | Yes | **Join** `<name>` (default) or **Deploy a new one** (for example a separate region or cost boundary) |
|
||||
| One or more | No | **Join** `<name>` |
|
||||
|
||||
**Deploy** uses the **CloudFormation template that ships with the app**
|
||||
(`aws/triple-c-environment.yaml`). A CloudFormation stack is auditable, an admin can deploy it
|
||||
without Triple-C, and deleting the stack removes everything. The app never creates environment
|
||||
resources one API call at a time. Deploy progress streams from the stack events into the same
|
||||
progress view used for local image pulls.
|
||||
**Deploy** applies the **OpenTofu module that ships with the app** (`aws/terraform/environment`)
|
||||
with the caller's own credentials. The app downloads a pinned OpenTofu (checked against its SHA-256
|
||||
before it is unpacked), creates the state bucket if needed, writes a work folder and starts a
|
||||
detached `tofu init` + `tofu apply` (D10); progress streams from OpenTofu's JSON log into the
|
||||
deploy view. An admin can apply the same files without Triple-C from a bundle (D5), and
|
||||
`tofu destroy` removes everything the module created. The app never creates environment resources
|
||||
one API call at a time.
|
||||
|
||||
**Joining in strict tenancy** (the default) also needs the user's **onboarding stack**: their own
|
||||
task role, EFS access-point policy entry and, if they use EC2, a dedicated capacity provider. The
|
||||
app checks whether the caller already has one:
|
||||
**Joining in strict tenancy** (the default) also needs the user's **onboarding**
|
||||
(`aws/terraform/user-onboard`): their own task and execution roles (with the permissions
|
||||
boundary) and their own EFS file system (D2), and later, if they use EC2, a dedicated capacity
|
||||
provider. The app checks whether the caller already has one (their state object, or their task
|
||||
role):
|
||||
|
||||
- **The caller can deploy it** (always true in their own account): Triple-C deploys
|
||||
`triple-c-user-onboard.yaml` for them as part of joining. For the person who just deployed the
|
||||
- **The caller can deploy it** (an environment admin; always true in their own account): Triple-C
|
||||
applies the onboarding module for them as part of joining. For the person who just deployed the
|
||||
environment, this happens in the same flow, so a single user never sees "onboarding" as a step.
|
||||
- **The caller can't**: Triple-C produces an onboarding request for whoever runs the environment —
|
||||
a pre-filled CloudFormation console link plus the exact parameters — and the user's "Join" stays
|
||||
pending until the stack exists. The app re-checks whenever the user opens the dialog.
|
||||
an admin bundle (D5) with the user's `aws:userid` filled in — and the user's "Join" stays
|
||||
pending until their roles exist. The app re-checks whenever the user opens the dialog.
|
||||
|
||||
### 3. Permission check
|
||||
|
||||
@@ -124,7 +174,7 @@ There are three tiers, and the check reports which one the user is in:
|
||||
|
||||
| Tier | Can do |
|
||||
|---|---|
|
||||
| **Environment admin** | Deploy or update the environment stack, onboard users (strict tenancy) |
|
||||
| **Environment admin** | Deploy or update an environment (the OpenTofu modules), onboard users (strict tenancy) |
|
||||
| **Workspace user** | Create, start, stop, connect to and delete **their own** workspaces in an existing environment |
|
||||
| **Insufficient** | Lists exactly which actions are missing, and offers the IAM policy JSON to send to an admin |
|
||||
|
||||
@@ -270,7 +320,7 @@ other.
|
||||
|
||||
- ECR pull-through cache of `ghcr.io/shadowdao/triple-c-sandbox` (GHCR as an upstream needs a
|
||||
GitHub token stored in Secrets Manager), or
|
||||
- the environment stack's ECR repo, filled by an admin or CI with `crane copy`.
|
||||
- an environment's ECR repo (not in v1, D3), filled by an admin or CI with `crane copy`.
|
||||
|
||||
Pulling straight from GHCR over NAT also works, and is the default when the environment has no ECR mirror. Image
|
||||
updates use the existing "new image available" flow: the next start uses the new digest.
|
||||
@@ -390,7 +440,7 @@ before release.**
|
||||
|
||||
| Mode | For | Task role | EC2 options | Isolation |
|
||||
|---|---|---|---|---|
|
||||
| **Strict** (**default, v1**) | Everyone, from a solo account to a team | **Per-user** task role, created by the user's onboarding stack (self-deployed, or by an admin) | Privileged and Docker allowed **only on a per-user capacity provider** (dedicated EC2 instances, never shared with another owner) | Per-user IAM all the way down. EFS file system policy allows each access point only to its owner's task role |
|
||||
| **Strict** (**default, v1**) | Everyone, from a solo account to a team | **Per-user** task role, created by the user's onboarding module (self-deployed, or by an admin) | Privileged and Docker allowed **only on a per-user capacity provider** (dedicated EC2 instances, never shared with another owner) | Per-user IAM all the way down. EFS file system policy allows each access point only to its owner's task role |
|
||||
| **Standard** (opt-in, later) | Large teams that want fewer IAM roles | Shared, minimal (logs, SSM messages) | **Fargate only**, or EC2 with **no** privileged mode, Docker socket or `NET_ADMIN` | Containers are unprivileged and can't mount NFS, can't read secrets, and can't reach each other over the network |
|
||||
|
||||
There is no separate "personal" mode. A solo user on strict tenancy is simply the only onboarded
|
||||
@@ -422,7 +472,7 @@ fails because of a quota, the app says so and doesn't retry.
|
||||
| Start time | About 30–90 s (image pull every time; SOCI lazy-loading **to evaluate**) | Fast when the image is cached on a warm instance |
|
||||
| Docker access / privileged / VPN | No | Yes, on dedicated capacity |
|
||||
| GPU | No | Yes |
|
||||
| Ops | None | ASG + capacity provider, defined in the environment stack. The app only picks a capacity provider |
|
||||
| Ops | None | ASG + capacity provider, defined in the environment module. The app only picks a capacity provider |
|
||||
| Cost when idle | 0 (task stopped) | Instances keep running unless the ASG scales in |
|
||||
| Max size | 16 vCPU / 120 GB | Instance type |
|
||||
|
||||
@@ -560,19 +610,23 @@ Existing `projects.json` files deserialize unchanged (`host_id` missing → Loca
|
||||
|
||||
```
|
||||
aws/
|
||||
triple-c-environment.yaml # CloudFormation: cluster, capacity providers, SG, EFS + policy,
|
||||
# KMS key, ECR repo / pull-through rule, log group, exec+task roles,
|
||||
# SSM environment parameter. Params: VPC/subnets (or create),
|
||||
# tenancy mode, limits, Bedrock-on-task-role.
|
||||
triple-c-user-onboard.yaml # strict tenancy: per-user task role + capacity provider
|
||||
README.md # the policies, the modules, running a bundle by hand, live results
|
||||
terraform/ # OpenTofu modules (hashicorp/aws pinned, .terraform.lock.hcl)
|
||||
permissions-boundary/ # the account's triple-c-permissions-boundary policy
|
||||
environment/ # cluster, tasks SG (no ingress), KMS key + alias, log group,
|
||||
# network (public | private-nat | existing), SSM record (last)
|
||||
user-onboard/ # per user: task + exec roles (bounded), own EFS file system,
|
||||
# its SG, two mount targets, file-system policy (D2)
|
||||
policies/
|
||||
environment-admin.json
|
||||
workspace-user.json
|
||||
permissions-boundary.json # every triple-c-* role carries it (phase 3)
|
||||
task-execution-role.json
|
||||
task-role-standard.json
|
||||
environment-admin.json # the admin tier, part 1 (D7)
|
||||
environment-admin-network.json # the admin tier, part 2: the network
|
||||
permissions-boundary.json # every triple-c-* role carries it
|
||||
```
|
||||
|
||||
State lives in `triple-c-tfstate-<account>-<region>` (versioned, SSE-S3, public access blocked,
|
||||
TLS only, tagged `triple-c:component=state`), one object per deployment (D9).
|
||||
|
||||
## Phases
|
||||
|
||||
1. **Host abstraction refactor.** `ProjectHost` trait, `DockerHost`, every call site moved. No
|
||||
@@ -582,13 +636,15 @@ aws/
|
||||
3. **AWS connection + permission check.** Settings UI, SDK wiring, `GetCallerIdentity`, the tiered
|
||||
simulation and fallback probes, policy files. **Done 2026-11-14**; the policies were verified
|
||||
against real IAM (see "Phase 3: policy verification and rulings").
|
||||
4. **Environment + onboarding stacks.** CloudFormation templates for both, detection (SSM
|
||||
parameters + tagged stacks, including half-deployed and outdated ones), join or deploy,
|
||||
self-onboarding or an admin request, **strict tenancy**.
|
||||
4. **Environment + onboarding modules.** OpenTofu modules for both and the boundary, run by the
|
||||
app (detached) or by an admin from a bundle; detection (SSM records + state objects and locks,
|
||||
including incomplete, in-progress and outdated ones), join or deploy, self-onboarding or an
|
||||
admin request, **strict tenancy**. **Done 2026-11-14**; deployed, joined, updated and destroyed
|
||||
for real as a role holding only the admin policies (see "Phase 4: live verification").
|
||||
5. **Fargate workspaces.** Ephemeral and Hybrid storage, Secrets Manager delivery, git bootstrap,
|
||||
start/stop/status, stop guard with WIP push on by default, idle auto-stop, logs.
|
||||
6. **EFS mode, Copy-at-connect, setup scripts, cost estimate.**
|
||||
7. **ECS on EC2.** Per-user capacity providers in the onboarding stack, privileged/Docker/VPN on
|
||||
7. **ECS on EC2.** Per-user capacity providers in the onboarding module, privileged/Docker/VPN on
|
||||
dedicated capacity.
|
||||
8. **Standard tenancy** (opt-in), only if demand appears.
|
||||
|
||||
@@ -664,6 +720,29 @@ Left for the phases with real ECS/EFS/EC2 resources: `RunTask`'s `aws:TagKeys` (
|
||||
`aws:ecs:*` or propagated tags), `iam:PassRole` with `iam:ResourceTag` on a real run, the owner
|
||||
tag on a real task for stop/exec/session, session ownership for an SSO-started session, EFS
|
||||
`CreateAction`/`CreateAccessPoint`/`ClientMount`, and `security-group-rule` authorization.
|
||||
(Phase 4 has since created security groups and rules for real; see below.)
|
||||
|
||||
### Phase 4: live verification (2026-11-14)
|
||||
|
||||
`app/src-tauri/src/aws/live_tests.rs` (`environment`) applies the modules for real, in us-west-2,
|
||||
as a role holding **only** the admin files: state bucket, a `public` environment through the real
|
||||
detached runner, detection, self-onboarding, an update ("No changes"), destroy; and a
|
||||
`private-nat` environment. What reality changed (details in `aws/README.md`):
|
||||
|
||||
- `s3:CreateBucket` with tags also needs `s3:TagResource`. Its `aws:RequestTag` and
|
||||
`s3:ResourceAccount` conditions are evaluated and pass.
|
||||
- `ec2:CreateSecurityGroup` is authorized against the VPC without the request's tags, so the VPC
|
||||
side of the grant is unconditioned (the group still needs the tag).
|
||||
- Destroying the NAT gateway's address disassociates it after the association is gone; EC2
|
||||
authorizes that against the literal `*/*`, which is all the policy grants (`${*}/${*}`).
|
||||
- `tofu destroy` leaves an empty state object; detection treats it as no deployment (otherwise a
|
||||
destroyed environment read as Incomplete for ever).
|
||||
- Confirmed as designed: EFS sets `aws:CalledViaLast` for mount-target interfaces; the EFS
|
||||
file-system policy accepted the just-created task role; conditional `DeleteObject` (`If-Match`)
|
||||
on the versioned bucket returns 412 on a stale ETag.
|
||||
|
||||
Still to verify: the `existing` network mode against a user's own VPC, and Fargate tasks in the
|
||||
deployed environment (phase 5).
|
||||
|
||||
## Connection model
|
||||
|
||||
@@ -684,7 +763,7 @@ The options:
|
||||
| Several regions or environments, same login | Duplicate connections with the same credentials. Keys are stored twice; SSO shares a token cache, so it's only clutter | One connection | One connection |
|
||||
| Permission check | Once per connection, cached | Per environment, run lazily the first time a project uses it | Per environment, run lazily |
|
||||
| Detection cost | One region | Every region the user might use (`DescribeRegions` + a scan, slow) or a hand-kept list | Default region straight away; other regions only when the user picks one |
|
||||
| Risk of deploying to the wrong place | Low: the place is in the connection's name | Higher: an easy region-picker mistake leaves a stack in a region nobody looks at | Low: default region preselected, others must be chosen deliberately |
|
||||
| Risk of deploying to the wrong place | Low: the place is in the connection's name | Higher: an easy region-picker mistake leaves an environment in a region nobody looks at | Low: default region preselected, others must be chosen deliberately |
|
||||
| Moving a project to another region or environment | Re-point the project to another connection | Change a field | Change a field |
|
||||
| Settings export/import | Each connection is self-contained | Projects reference environments that may not exist on the importing machine | Same as B |
|
||||
| Data model / UI | Smallest | Largest (environment picker and per-environment status everywhere) | Medium |
|
||||
|
||||
Reference in new issue
Block a user