test(aws): live phase 7 EC2 checks
a_phase7_ec2_workspace_round_trips: onboarding with per-user EC2 capacity through prepare_onboarding (tagged workspace role, all four admin files), its collision check and refusal, admin-tier scaling probes, a cold and a warm EC2 start (Docker access, VPN, ECS Exec into the bridge task), in-container checks (docker ps, wg/tun, src_valid_mark, bwrap, IMDS, introspection), RunTask on another user's cluster, stop and scale-in, removal, re-apply without EC2 with an instance up, offboard and destroy. The guard (ec2_cleanup) force-deletes any per-user EC2 capacity. Results in aws/README.md and the spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6c7c2ea4ef
commit
fc3456c44e
3 files changed
+912
-66
No files matched your search
@@ -1773,6 +1773,13 @@ mod environment {
|
||||
config_file_before: Option<Option<std::ffi::OsString>>,
|
||||
/// `TRIPLEC_E2E_ECR_REPO`: the throwaway repository to force-delete.
|
||||
ecr_repo: Option<String>,
|
||||
/// Phase 7: the environment whose per-user EC2 capacity
|
||||
/// (`triple-c-<env>--u-*`) the guard tears down, and the user's
|
||||
/// owner key (instance role, user record).
|
||||
ec2_env: Option<String>,
|
||||
ec2_okey: Option<String>,
|
||||
/// Auto Scaling groups a probe made outside that prefix.
|
||||
probe_groups: Vec<String>,
|
||||
}
|
||||
|
||||
/// The whole phase-5 body: a hang anywhere becomes a panic the guard
|
||||
@@ -1802,6 +1809,9 @@ mod environment {
|
||||
Some(None) => std::env::remove_var("AWS_CONFIG_FILE"),
|
||||
None => {}
|
||||
}
|
||||
if let Some(env) = &p.ec2_env {
|
||||
ec2_cleanup(e, env, p.ec2_okey.as_deref(), &p.probe_groups).await;
|
||||
}
|
||||
let ecs = aws_sdk_ecs::Client::new(&e.cfg);
|
||||
if let Some(cluster) = &p.cluster {
|
||||
let mut arns = Vec::new();
|
||||
@@ -1918,6 +1928,201 @@ mod environment {
|
||||
}
|
||||
}
|
||||
|
||||
/// The AWS CLI as the e2e profile (`keys`: as that session instead):
|
||||
/// (success, stdout, stderr). The app has no EC2 SDK crate.
|
||||
async fn aws_cli(keys: Option<&AwsKeys>, region: &str, args: &[&str]) -> (bool, String, String) {
|
||||
let mut cmd = tokio::process::Command::new("aws");
|
||||
cmd.args(["--region", region, "--output", "json"]);
|
||||
match keys {
|
||||
None => {
|
||||
cmd.args(["--profile", &super::var("TRIPLEC_AWS_E2E_PROFILE")]);
|
||||
}
|
||||
Some(k) => {
|
||||
cmd.env_remove("AWS_PROFILE")
|
||||
.env("AWS_ACCESS_KEY_ID", &k.access_key_id)
|
||||
.env("AWS_SECRET_ACCESS_KEY", &k.secret_access_key);
|
||||
match &k.session_token {
|
||||
Some(t) => cmd.env("AWS_SESSION_TOKEN", t),
|
||||
None => cmd.env_remove("AWS_SESSION_TOKEN"),
|
||||
};
|
||||
}
|
||||
}
|
||||
cmd.args(args);
|
||||
match tokio::time::timeout(Duration::from_secs(120), cmd.output()).await {
|
||||
Ok(Ok(o)) => (
|
||||
o.status.success(),
|
||||
String::from_utf8_lossy(&o.stdout).into_owned(),
|
||||
String::from_utf8_lossy(&o.stderr).into_owned(),
|
||||
),
|
||||
Ok(Err(err)) => (false, String::new(), format!("aws cli: {err}")),
|
||||
Err(_) => (false, String::new(), "aws cli: timed out".into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Phase 7's guard, as the account admin: everything of `env`'s per-user
|
||||
/// EC2 capacity (`triple-c-<env>--u-*`) and the probes' groups, so no
|
||||
/// instance outlives a failed run. Tasks on the per-user clusters are
|
||||
/// stopped, groups force-deleted (terminating their instances) and waited
|
||||
/// for, leftover tagged instances terminated, then capacity providers,
|
||||
/// container instances, clusters, launch templates, the user record and
|
||||
/// the instance role/profile deleted. A resource already gone is fine
|
||||
/// (the modules' destroy then finds nothing). Logs, never fails.
|
||||
async fn ec2_cleanup(e: &E2e, env: &str, okey: Option<&str>, probe_groups: &[String]) {
|
||||
let prefix = format!("triple-c-{env}--u-");
|
||||
let ecs = aws_sdk_ecs::Client::new(&e.cfg);
|
||||
let asg = aws_sdk_autoscaling::Client::new(&e.cfg);
|
||||
// Tasks on the per-user clusters.
|
||||
let mut clusters = Vec::new();
|
||||
if let Ok(out) = ecs.list_clusters().send().await {
|
||||
clusters.extend(out.cluster_arns().iter().filter(|a| a.contains(&format!(":cluster/{prefix}"))).cloned());
|
||||
}
|
||||
for cluster in &clusters {
|
||||
if let Ok(out) = ecs.list_tasks().cluster(cluster).send().await {
|
||||
for arn in out.task_arns() {
|
||||
eprintln!("cleanup: stopping EC2 task {arn}");
|
||||
let _ = ecs.stop_task().cluster(cluster).task(arn).reason("triple-c-e2e cleanup").send().await;
|
||||
}
|
||||
}
|
||||
}
|
||||
// Groups (force: terminates their instances, ignores protection).
|
||||
let mut groups: Vec<String> = probe_groups.to_vec();
|
||||
if let Ok(out) = asg.describe_auto_scaling_groups().send().await {
|
||||
for g in out.auto_scaling_groups() {
|
||||
let n = g.auto_scaling_group_name().unwrap_or_default();
|
||||
if n.starts_with(&prefix) && !groups.iter().any(|x| x == n) {
|
||||
groups.push(n.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
for g in &groups {
|
||||
match asg.delete_auto_scaling_group().auto_scaling_group_name(g).force_delete(true).send().await {
|
||||
Ok(_) => eprintln!("cleanup: force-deleting Auto Scaling group {g}"),
|
||||
Err(err) => {
|
||||
let msg = format!("{err:?}");
|
||||
if !msg.contains("not found") && !msg.contains("ValidationError") {
|
||||
eprintln!("cleanup: delete group {g}: {msg}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !groups.is_empty() {
|
||||
for _ in 0..60 {
|
||||
let left = asg
|
||||
.describe_auto_scaling_groups()
|
||||
.set_auto_scaling_group_names(Some(groups.clone()))
|
||||
.send()
|
||||
.await
|
||||
.map(|o| o.auto_scaling_groups().len())
|
||||
.unwrap_or(0);
|
||||
if left == 0 {
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(10)).await;
|
||||
}
|
||||
}
|
||||
// Any instance still tagged with this environment.
|
||||
let (ok_, out, err) = aws_cli(
|
||||
None,
|
||||
&e.region,
|
||||
&[
|
||||
"ec2", "describe-instances",
|
||||
"--filters", &format!("Name=tag:triple-c:environment,Values={env}"),
|
||||
"Name=instance-state-name,Values=pending,running,stopping,stopped",
|
||||
"--query", "Reservations[].Instances[].InstanceId",
|
||||
],
|
||||
)
|
||||
.await;
|
||||
if ok_ {
|
||||
let ids: Vec<String> = serde_json::from_str(&out).unwrap_or_default();
|
||||
if !ids.is_empty() {
|
||||
let mut args = vec!["ec2", "terminate-instances", "--instance-ids"];
|
||||
args.extend(ids.iter().map(String::as_str));
|
||||
let r = aws_cli(None, &e.region, &args).await;
|
||||
eprintln!("cleanup: terminate {ids:?}: ok {}", r.0);
|
||||
let mut args = vec!["ec2", "wait", "instance-terminated", "--instance-ids"];
|
||||
args.extend(ids.iter().map(String::as_str));
|
||||
let _ = aws_cli(None, &e.region, &args).await;
|
||||
}
|
||||
} else {
|
||||
eprintln!("cleanup: describe instances: {err}");
|
||||
}
|
||||
// Capacity providers: off their clusters, then deleted.
|
||||
for cluster in &clusters {
|
||||
let _ = ecs.put_cluster_capacity_providers().cluster(cluster).set_capacity_providers(Some(vec![])).set_default_capacity_provider_strategy(Some(vec![])).send().await;
|
||||
}
|
||||
if let Ok(out) = ecs.describe_capacity_providers().send().await {
|
||||
for cp in out.capacity_providers() {
|
||||
let n = cp.name().unwrap_or_default();
|
||||
if n.starts_with(&prefix) && cp.status() != Some(&aws_sdk_ecs::types::CapacityProviderStatus::Inactive) {
|
||||
match ecs.delete_capacity_provider().capacity_provider(n).send().await {
|
||||
Ok(_) => eprintln!("cleanup: deleted capacity provider {n}"),
|
||||
Err(err) => eprintln!("cleanup: delete capacity provider {n}: {err:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for cluster in &clusters {
|
||||
if let Ok(out) = ecs.list_container_instances().cluster(cluster).send().await {
|
||||
for ci in out.container_instance_arns() {
|
||||
let _ = ecs.deregister_container_instance().cluster(cluster).container_instance(ci).force(true).send().await;
|
||||
}
|
||||
}
|
||||
// Busy (UpdateInProgress) while the provider detaches: retry.
|
||||
for attempt in 0..12 {
|
||||
match ecs.delete_cluster().cluster(cluster).send().await {
|
||||
Ok(_) => {
|
||||
eprintln!("cleanup: deleted cluster {cluster}");
|
||||
break;
|
||||
}
|
||||
Err(err) if attempt < 11 && format!("{err:?}").contains("UpdateInProgress") => {
|
||||
tokio::time::sleep(Duration::from_secs(10)).await;
|
||||
}
|
||||
Err(err) => {
|
||||
eprintln!("cleanup: delete cluster {cluster}: {err:?}");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Launch templates.
|
||||
let (ok_, out, _) = aws_cli(
|
||||
None,
|
||||
&e.region,
|
||||
&["ec2", "describe-launch-templates", "--filters", &format!("Name=launch-template-name,Values={prefix}*"), "--query", "LaunchTemplates[].LaunchTemplateName"],
|
||||
)
|
||||
.await;
|
||||
if ok_ {
|
||||
let names: Vec<String> = serde_json::from_str(&out).unwrap_or_default();
|
||||
for n in names {
|
||||
let r = aws_cli(None, &e.region, &["ec2", "delete-launch-template", "--launch-template-name", &n]).await;
|
||||
eprintln!("cleanup: delete launch template {n}: ok {} {}", r.0, r.2.trim());
|
||||
}
|
||||
}
|
||||
if let Some(okey) = okey {
|
||||
let ssm = aws_sdk_ssm::Client::new(&e.cfg);
|
||||
let param = crate::aws::tofu_modules::user_record_parameter_name(env, okey);
|
||||
if ssm.delete_parameter().name(¶m).send().await.is_ok() {
|
||||
eprintln!("cleanup: deleted {param}");
|
||||
}
|
||||
let role = crate::aws::tofu_modules::user_instance_role_name(env, okey);
|
||||
if e.iam.remove_role_from_instance_profile().instance_profile_name(&role).role_name(&role).send().await.is_ok() {
|
||||
eprintln!("cleanup: removed {role} from its instance profile");
|
||||
}
|
||||
if e.iam.delete_instance_profile().instance_profile_name(&role).send().await.is_ok() {
|
||||
eprintln!("cleanup: deleted instance profile {role}");
|
||||
}
|
||||
if let Ok(out) = e.iam.list_role_policies().role_name(&role).send().await {
|
||||
for pn in out.policy_names() {
|
||||
let _ = e.iam.delete_role_policy().role_name(&role).policy_name(pn).send().await;
|
||||
}
|
||||
match e.iam.delete_role().role_name(&role).send().await {
|
||||
Ok(_) => eprintln!("cleanup: deleted role {role}"),
|
||||
Err(err) => eprintln!("cleanup: delete role {role}: {err:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Poll `f` every `every` until it returns `Some`, or `None` after `within`.
|
||||
async fn poll<T, F, Fut>(within: Duration, every: Duration, mut f: F) -> Option<T>
|
||||
where
|
||||
@@ -1960,6 +2165,8 @@ mod environment {
|
||||
conn: HostConnection,
|
||||
ws_cfg: SdkConfig,
|
||||
ws: CallerIdentity,
|
||||
/// The environment admin role (fresh sessions via `Env::session`).
|
||||
admin: Admin,
|
||||
}
|
||||
|
||||
/// The workspace tests' common start: the environment (with
|
||||
@@ -1973,6 +2180,7 @@ mod environment {
|
||||
p5: &std::sync::Mutex<Phase5>,
|
||||
image: &str,
|
||||
max_workspace_hours: u32,
|
||||
ec2: Option<&crate::aws::tofu_modules::Ec2Compute>,
|
||||
) -> Deployed {
|
||||
use crate::aws::tofu_modules::{environment_cluster_name, user_file_system_token};
|
||||
|
||||
@@ -2003,18 +2211,43 @@ mod environment {
|
||||
let fs_token = user_file_system_token(&name, &okey);
|
||||
p5_lock(p5).fs_token = Some(fs_token.clone());
|
||||
|
||||
// Onboarded by the admin (an onboarding from an admin bundle).
|
||||
let (keys, _, _) = t.session(&admin).await;
|
||||
// Onboarded by the admin (an onboarding from an admin bundle). With
|
||||
// EC2 capacity (phase 7): the user's role carries the triple-c-user
|
||||
// principal tag, and the vars come from the app's own pre-apply
|
||||
// step (record, choice, name-collision check) as the admin.
|
||||
let (keys, admin_cfg, _) = t.session(&admin).await;
|
||||
let vars = match ec2 {
|
||||
None => onboarding_vars(&name, ®ion, &owner, None).unwrap(),
|
||||
Some(c) => {
|
||||
use crate::aws::tofu_modules::EC2_USER_TAG_KEY;
|
||||
use crate::commands::environment_commands::{prepare_onboarding, Ec2Choice};
|
||||
t.e.tag_role(&user, EC2_USER_TAG_KEY, &c.user_tag).await;
|
||||
{
|
||||
let mut g = p5_lock(p5);
|
||||
g.ec2_env = Some(name.clone());
|
||||
g.ec2_okey = Some(okey.clone());
|
||||
}
|
||||
let api = crate::aws::environments::SdkOnboardApi::new(&admin_cfg);
|
||||
let t0 = std::time::Instant::now();
|
||||
let vars = prepare_onboarding(&api, &name, ®ion, &t.e.account, &owner, Some(Ec2Choice::Ec2(c.clone())))
|
||||
.await
|
||||
.unwrap_or_else(|e| panic!("the pre-apply onboarding step: {e}"));
|
||||
eprintln!("RESULT pre-apply EC2 collision check (free names) passed in {:?}: {vars}", t0.elapsed());
|
||||
vars
|
||||
}
|
||||
};
|
||||
let t0 = std::time::Instant::now();
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::UserOnboard,
|
||||
Deployment::User { env: &name, owner_key: &okey },
|
||||
&onboarding_vars(&name, ®ion, &owner, None).unwrap(),
|
||||
&vars,
|
||||
TofuOp::Apply,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("onboard the workspace user", &outcome, &denied);
|
||||
eprintln!("RESULT onboarding apply: {:?}", t0.elapsed());
|
||||
let (_, exec_role) = user_role_names(&name, &okey);
|
||||
|
||||
// Test-only: pull from the throwaway private ECR repository.
|
||||
@@ -2062,7 +2295,7 @@ mod environment {
|
||||
let ws = caller_identity(&ws_cfg).await.expect("workspace user identity");
|
||||
assert_eq!(ws.user_id, owner, "the profile session is the onboarded owner");
|
||||
eprintln!("workspace user: {}", ws.arn);
|
||||
Deployed { region, name, owner, okey, fs_token, conn, ws_cfg, ws }
|
||||
Deployed { region, name, owner, okey, fs_token, conn, ws_cfg, ws, admin }
|
||||
}
|
||||
|
||||
/// Phase 5: a Fargate workspace's whole life through `ProjectHost::Aws`,
|
||||
@@ -2119,8 +2352,8 @@ log_stream_prefix, project_secret_name, task_family};
|
||||
// Boxed: the body's state machine is far larger than a test
|
||||
// thread's 2 MiB stack once it is moved into the timeout.
|
||||
let inner = Box::pin(async {
|
||||
let Deployed { region, name, owner, okey, fs_token, conn, ws_cfg, ws } =
|
||||
deploy_for_workspace_user(&t, &p5, &image, 0).await;
|
||||
let Deployed { region, name, owner, okey, fs_token, conn, ws_cfg, ws, .. } =
|
||||
deploy_for_workspace_user(&t, &p5, &image, 0, None).await;
|
||||
|
||||
let mut settings = AppSettings::default();
|
||||
settings.hosts = vec![conn];
|
||||
@@ -2579,8 +2812,8 @@ log_stream_prefix, project_secret_name, task_family};
|
||||
let p5 = std::sync::Mutex::new(Phase5 { ecr_repo, ..Default::default() });
|
||||
let inner = Box::pin(async {
|
||||
// ── 1. the environment carries the lifetime cap ──
|
||||
let Deployed { region, name, owner: _, okey, fs_token: _, conn, ws_cfg, ws } =
|
||||
deploy_for_workspace_user(&t, &p5, &image, 1).await;
|
||||
let Deployed { region, name, okey, conn, ws_cfg, ws, .. } =
|
||||
deploy_for_workspace_user(&t, &p5, &image, 1, None).await;
|
||||
let env = crate::aws::runtime::read_environment(&SdkRuntimeApi::new(&ws_cfg), &name, &ws.account, ®ion)
|
||||
.await
|
||||
.expect("the environment record, as the workspace user");
|
||||
@@ -2927,4 +3160,572 @@ log_stream_prefix, project_secret_name, task_family};
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────── phase 7: ECS on EC2
|
||||
|
||||
/// The phase-7 body: two cold starts, a scale-in wait (managed scaling
|
||||
/// takes ~15 minutes to remove an idle instance) and three applies.
|
||||
const PHASE7_TIMEOUT: Duration = Duration::from_secs(110 * 60);
|
||||
|
||||
/// A group's (instance count, desired capacity, `id:state` list) as the
|
||||
/// account admin; `None` when it does not exist.
|
||||
async fn group_size(asg: &aws_sdk_autoscaling::Client, name: &str) -> Option<(usize, i32, Vec<String>)> {
|
||||
let out = asg.describe_auto_scaling_groups().auto_scaling_group_names(name).send().await.ok()?;
|
||||
let g = out.auto_scaling_groups().first()?;
|
||||
let ids = g
|
||||
.instances()
|
||||
.iter()
|
||||
.map(|i| format!("{}:{}", i.instance_id().unwrap_or_default(), i.lifecycle_state().map(|s| s.as_str()).unwrap_or("?")))
|
||||
.collect();
|
||||
Some((g.instances().len(), g.desired_capacity().unwrap_or(-1), ids))
|
||||
}
|
||||
|
||||
/// Instances still tagged with `env` and not terminated, as the account admin.
|
||||
async fn live_env_instances(region: &str, env: &str) -> Vec<String> {
|
||||
let (ok_, out, err) = aws_cli(
|
||||
None,
|
||||
region,
|
||||
&[
|
||||
"ec2", "describe-instances",
|
||||
"--filters", &format!("Name=tag:triple-c:environment,Values={env}"),
|
||||
"Name=instance-state-name,Values=pending,running,stopping,stopped,shutting-down",
|
||||
"--query", "Reservations[].Instances[].[InstanceId,State.Name]",
|
||||
],
|
||||
)
|
||||
.await;
|
||||
assert!(ok_, "describe-instances: {err}");
|
||||
let rows: Vec<Vec<String>> = serde_json::from_str(&out).unwrap_or_default();
|
||||
rows.into_iter().map(|r| r.join(":")).collect()
|
||||
}
|
||||
|
||||
/// Phase 7: an EC2 project's whole life on the user's own capacity
|
||||
/// (ECS on EC2, bridge networking, scale to zero), as a role holding
|
||||
/// only the workspace files and carrying the `triple-c-user` principal
|
||||
/// tag (a role tag: an assumed-role session's principal tags), onboarded
|
||||
/// with `m7i.large`, at most 1 instance, by the environment admin
|
||||
/// holding all four admin files. Run with `scripts/aws-e2e-workspace.sh`
|
||||
/// and `TRIPLEC_E2E_TEST=aws::live_tests::environment::a_phase7_ec2_workspace_round_trips`.
|
||||
///
|
||||
/// Covers: the onboarding's resources and its pre-apply collision check
|
||||
/// (and its refusal for another owner); admin-tier probes of the
|
||||
/// scaling file (`$Latest`, a pinned control, an unsupported type in a
|
||||
/// template version, a capacity provider over a non-Triple-C group);
|
||||
/// a cold start with Docker access, VPN and the sandbox on (instance
|
||||
/// launch, ECS Exec port forward into the bridge-mode task), `docker
|
||||
/// ps`, a WireGuard/TUN interface, `src_valid_mark`, bubblewrap, IMDS
|
||||
/// and the agent's introspection port from the container; RunTask on
|
||||
/// another user's cluster (the wrong-triple-c-user message); stop and
|
||||
/// the scale-in to zero; a second start; removal; re-applying the
|
||||
/// onboarding without EC2 (drain and delete with an instance still up);
|
||||
/// offboarding and the environment's destroy, all as the admin tier.
|
||||
/// Its guard (`ec2_cleanup`) force-deletes whatever EC2 capacity is
|
||||
/// left. About one m7i.large instance-hour.
|
||||
#[tokio::test]
|
||||
#[ignore]
|
||||
async fn a_phase7_ec2_workspace_round_trips() {
|
||||
use crate::aws::environments::{OnboardApi, SdkOnboardApi, UserCompute, UserRecord};
|
||||
use crate::aws::runtime::{read_user_record, SdkRuntimeApi};
|
||||
use crate::aws::tasks::{ec2_run_error, ec2_run_task_request, SdkTaskApi, TaskApi, EC2_TAG_MISMATCH};
|
||||
use crate::aws::tofu_modules::{user_ec2_stem, user_instance_role_name, Ec2Compute};
|
||||
use crate::aws::workspace::workspace_tags;
|
||||
use crate::commands::environment_commands::{prepare_onboarding, Ec2Choice};
|
||||
use crate::host::aws_host::PROGRESS_INSTANCE;
|
||||
use crate::host::lifecycle::ProjectHost;
|
||||
use crate::models::app_settings::ImageSource;
|
||||
use crate::models::project::{AwsPlacement, AwsWorkspaceConfig, LaunchType, StorageMode};
|
||||
use crate::models::{AppSettings, Project};
|
||||
use crate::storage::projects_store::ProjectsStore;
|
||||
|
||||
let image = super::var("TRIPLEC_E2E_IMAGE");
|
||||
let ecr_repo = std::env::var("TRIPLEC_E2E_ECR_REPO").ok().filter(|r| !r.is_empty());
|
||||
env_e2e(|t| async move {
|
||||
let p5 = std::sync::Mutex::new(Phase5 { ecr_repo, ..Default::default() });
|
||||
let inner = Box::pin(async {
|
||||
// TRIPLEC_E2E_PHASE7_SHORT=1: the life cycle only (no probes, no
|
||||
// in-container checks or isolation, no scale-in wait; the second
|
||||
// start is then warm, on the instance still up).
|
||||
let quick = std::env::var("TRIPLEC_E2E_PHASE7_SHORT").is_ok_and(|v| v == "1");
|
||||
let short = uuid::Uuid::new_v4().simple().to_string();
|
||||
let user_tag = format!("e2e{}", &short[..6]);
|
||||
let compute = Ec2Compute { instance_type: "m7i.large".into(), max_instances: 1, user_tag: user_tag.clone() };
|
||||
let Deployed { region, name, owner, okey, conn, ws_cfg, ws, admin, .. } =
|
||||
deploy_for_workspace_user(&t, &p5, &image, 0, Some(&compute)).await;
|
||||
let account = t.e.account.clone();
|
||||
let stem = user_ec2_stem(&name, &user_tag);
|
||||
let asg_admin = aws_sdk_autoscaling::Client::new(&t.e.cfg);
|
||||
let ecs_admin = aws_sdk_ecs::Client::new(&t.e.cfg);
|
||||
|
||||
// ── 1. what the onboarding made ──
|
||||
let groups = ok("describe the group", asg_admin.describe_auto_scaling_groups().auto_scaling_group_names(&stem).send().await);
|
||||
let group = groups.auto_scaling_groups().first().expect("the user's Auto Scaling group").clone();
|
||||
let subnets = group.vpc_zone_identifier().unwrap_or_default().to_string();
|
||||
let group_tags: Vec<(String, String)> = group
|
||||
.tags()
|
||||
.iter()
|
||||
.map(|t| (t.key().unwrap_or_default().to_string(), t.value().unwrap_or_default().to_string()))
|
||||
.collect();
|
||||
eprintln!(
|
||||
"RESULT group {stem}: min {:?} max {:?} desired {:?}, template {:?}, protect {:?}, tags {group_tags:?}",
|
||||
group.min_size(),
|
||||
group.max_size(),
|
||||
group.desired_capacity(),
|
||||
group.launch_template().map(|l| (l.launch_template_name(), l.version())),
|
||||
group.new_instances_protected_from_scale_in()
|
||||
);
|
||||
assert_eq!((group.min_size(), group.max_size()), (Some(0), Some(1)));
|
||||
let cl = ok(
|
||||
"describe the user's cluster",
|
||||
ecs_admin.describe_clusters().clusters(&stem).include(aws_sdk_ecs::types::ClusterField::Tags).send().await,
|
||||
);
|
||||
let cluster = cl.clusters().first().expect("the user's cluster");
|
||||
eprintln!(
|
||||
"RESULT cluster {stem}: {:?}, providers {:?}, default strategy {:?}, tags {:?}",
|
||||
cluster.status(),
|
||||
cluster.capacity_providers(),
|
||||
cluster.default_capacity_provider_strategy().iter().map(|s| s.capacity_provider()).collect::<Vec<_>>(),
|
||||
cluster.tags().iter().map(|t| (t.key().unwrap_or_default(), t.value().unwrap_or_default())).collect::<Vec<_>>()
|
||||
);
|
||||
let cps = ok("describe the capacity provider", ecs_admin.describe_capacity_providers().capacity_providers(&stem).send().await);
|
||||
let cp = cps.capacity_providers().first().expect("the capacity provider");
|
||||
eprintln!(
|
||||
"RESULT capacity provider {stem}: {:?}, managed scaling {:?}, termination protection {:?}",
|
||||
cp.status(),
|
||||
cp.auto_scaling_group_provider().and_then(|a| a.managed_scaling()).map(|m| (m.status(), m.target_capacity())),
|
||||
cp.auto_scaling_group_provider().map(|a| a.managed_termination_protection())
|
||||
);
|
||||
let (_, lt, _) = aws_cli(
|
||||
None,
|
||||
®ion,
|
||||
&["ec2", "describe-launch-template-versions", "--launch-template-name", &stem, "--query",
|
||||
"LaunchTemplateVersions[].[VersionNumber,LaunchTemplateData.InstanceType,LaunchTemplateData.ImageId,LaunchTemplateData.MetadataOptions]"],
|
||||
)
|
||||
.await;
|
||||
eprintln!("RESULT launch template {stem}: {}", lt.split_whitespace().collect::<Vec<_>>().join(" "));
|
||||
let role_name = user_instance_role_name(&name, &okey);
|
||||
let prof = ok("get the instance profile", t.e.iam.get_instance_profile().instance_profile_name(&role_name).send().await);
|
||||
eprintln!(
|
||||
"RESULT instance profile {role_name}: roles {:?}",
|
||||
prof.instance_profile().map(|p| p.roles().iter().map(|r| r.role_name().to_string()).collect::<Vec<_>>())
|
||||
);
|
||||
let record = read_user_record(&SdkRuntimeApi::new(&ws_cfg), &name, &okey, &account, ®ion).await.expect("read the user record");
|
||||
eprintln!("RESULT user record, as the workspace user: {record:?}");
|
||||
let Some(rec) = record.valid().cloned() else { panic!("the user record is not valid: {record:?}") };
|
||||
assert_eq!((rec.instance_type.as_str(), rec.max_instances, rec.user_tag.as_str()), ("m7i.large", 1, user_tag.as_str()));
|
||||
assert!(rec.cluster_arn.ends_with(&format!(":cluster/{stem}")), "{}", rec.cluster_arn);
|
||||
|
||||
// ── 1b / 6. the collision check's reads, and its refusal ──
|
||||
let (akeys, acfg, _) = t.session(&admin).await;
|
||||
let oapi = SdkOnboardApi::new(&acfg);
|
||||
let held = oapi.cluster(&stem).await.expect("DescribeClusters include=TAGS as the admin tier");
|
||||
eprintln!("RESULT DescribeClusters include=TAGS as the environment admin: {held:?}");
|
||||
let held = held.expect("the cluster is found");
|
||||
assert!(held.tags.iter().any(|(k, v)| k == "triple-c:owner" && *v == owner), "{held:?}");
|
||||
let missing = oapi.cluster(&format!("{stem}zz")).await;
|
||||
eprintln!("RESULT DescribeClusters on a missing name: {missing:?}");
|
||||
assert!(matches!(missing, Ok(None)), "{missing:?}");
|
||||
let missing_group = oapi.group(&format!("{stem}zz")).await;
|
||||
eprintln!("RESULT DescribeAutoScalingGroups on a missing name: {missing_group:?}");
|
||||
assert!(matches!(missing_group, Ok(None)), "{missing_group:?}");
|
||||
let refused = prepare_onboarding(&oapi, &name, ®ion, &account, "AROAEXAMPLEOTHER0000:bob", Some(Ec2Choice::Ec2(compute.clone()))).await;
|
||||
eprintln!("RESULT onboarding another owner with the same triple-c-user value: {refused:?}");
|
||||
let refused = refused.expect_err("another owner's onboarding with the same tag is refused");
|
||||
assert!(refused.to_string().contains("another onboarding"), "{refused}");
|
||||
|
||||
if !quick {
|
||||
// ── 6. the scaling file's open points, as the environment admin ──
|
||||
let asg_env = aws_sdk_autoscaling::Client::new(&acfg);
|
||||
let ecs_env = aws_sdk_ecs::Client::new(&acfg);
|
||||
let probe_tags = |n: &str| -> Vec<aws_sdk_autoscaling::types::Tag> {
|
||||
group_tags
|
||||
.iter()
|
||||
.filter(|(k, _)| k.starts_with("triple-c:"))
|
||||
.map(|(k, v)| {
|
||||
aws_sdk_autoscaling::types::Tag::builder()
|
||||
.key(k)
|
||||
.value(v)
|
||||
.resource_id(n)
|
||||
.resource_type("auto-scaling-group")
|
||||
.propagate_at_launch(false)
|
||||
.build()
|
||||
})
|
||||
.collect()
|
||||
};
|
||||
let create_group = |client: aws_sdk_autoscaling::Client, n: String, version: String, tags: Vec<aws_sdk_autoscaling::types::Tag>| {
|
||||
let subnets = subnets.clone();
|
||||
let lt_name = stem.clone();
|
||||
async move {
|
||||
client
|
||||
.create_auto_scaling_group()
|
||||
.auto_scaling_group_name(&n)
|
||||
.launch_template(
|
||||
aws_sdk_autoscaling::types::LaunchTemplateSpecification::builder()
|
||||
.launch_template_name(lt_name)
|
||||
.version(version)
|
||||
.build(),
|
||||
)
|
||||
.min_size(0)
|
||||
.max_size(0)
|
||||
.desired_capacity(0)
|
||||
.vpc_zone_identifier(subnets)
|
||||
.set_tags(Some(tags))
|
||||
.send()
|
||||
.await
|
||||
.map(|_| ())
|
||||
.map_err(|e| classify_sdk_error("autoscaling:CreateAutoScalingGroup", &e).to_string())
|
||||
}
|
||||
};
|
||||
let delete_group = |n: String| {
|
||||
let client = asg_admin.clone();
|
||||
async move {
|
||||
let _ = client.delete_auto_scaling_group().auto_scaling_group_name(&n).force_delete(true).send().await;
|
||||
}
|
||||
};
|
||||
// $Latest must be denied (LaunchTemplateVersionSpecified).
|
||||
let pa = format!("{stem}-pa");
|
||||
let r = create_group(asg_env.clone(), pa.clone(), "$Latest".into(), probe_tags(&pa)).await;
|
||||
eprintln!("RESULT probe: CreateAutoScalingGroup with $Latest as the environment admin: {r:?}");
|
||||
if r.is_ok() {
|
||||
eprintln!("RESULT WARNING: $Latest was allowed");
|
||||
delete_group(pa).await;
|
||||
}
|
||||
// The control: the same create pinned to version 1 is allowed
|
||||
// (InstanceTypes filled from the template).
|
||||
let pb = format!("{stem}-pb");
|
||||
let r = create_group(asg_env.clone(), pb.clone(), "1".into(), probe_tags(&pb)).await;
|
||||
eprintln!("RESULT probe: the same create pinned to version 1 (control): {r:?}");
|
||||
if r.is_ok() {
|
||||
let d = asg_env.delete_auto_scaling_group().auto_scaling_group_name(&pb).send().await;
|
||||
eprintln!("RESULT probe: DeleteAutoScalingGroup of it as the environment admin: {:?}", d.map(|_| ()).map_err(|e| classify_sdk_error("autoscaling:DeleteAutoScalingGroup", &e).to_string()));
|
||||
delete_group(pb).await;
|
||||
}
|
||||
// A template version naming an unsupported type, then a group pinned to it.
|
||||
let (made, out, err) = aws_cli(
|
||||
Some(&akeys),
|
||||
®ion,
|
||||
&["ec2", "create-launch-template-version", "--launch-template-name", &stem, "--source-version", "1",
|
||||
"--launch-template-data", r#"{"InstanceType":"p5.48xlarge"}"#, "--query", "LaunchTemplateVersion.VersionNumber"],
|
||||
)
|
||||
.await;
|
||||
eprintln!("RESULT probe: CreateLaunchTemplateVersion with p5.48xlarge as the environment admin: ok {made} {} {}", out.trim(), err.trim());
|
||||
if made {
|
||||
let pc = format!("{stem}-pc");
|
||||
let r = create_group(asg_env.clone(), pc.clone(), out.trim().to_string(), probe_tags(&pc)).await;
|
||||
eprintln!("RESULT probe: CreateAutoScalingGroup pinned to the p5.48xlarge version: {r:?}");
|
||||
if r.is_ok() {
|
||||
eprintln!("RESULT WARNING: a group of an unsupported type was allowed");
|
||||
delete_group(pc).await;
|
||||
}
|
||||
}
|
||||
// A Triple-C capacity provider over a group that isn't Triple-C's.
|
||||
let foreign_group = format!("e2e-untagged-{}", &short[..10]);
|
||||
p5_lock(&p5).probe_groups.push(foreign_group.clone());
|
||||
let r = create_group(asg_admin.clone(), foreign_group.clone(), "1".into(), Vec::new()).await;
|
||||
eprintln!("RESULT probe: account admin creates the untagged group {foreign_group}: {r:?}");
|
||||
if r.is_ok() {
|
||||
let arn = ok("describe the untagged group", asg_admin.describe_auto_scaling_groups().auto_scaling_group_names(&foreign_group).send().await)
|
||||
.auto_scaling_groups()
|
||||
.first()
|
||||
.and_then(|g| g.auto_scaling_group_arn().map(str::to_string))
|
||||
.unwrap();
|
||||
let pd = format!("{stem}-pd");
|
||||
let r = ecs_env
|
||||
.create_capacity_provider()
|
||||
.name(&pd)
|
||||
.auto_scaling_group_provider(
|
||||
aws_sdk_ecs::types::AutoScalingGroupProvider::builder()
|
||||
.auto_scaling_group_arn(arn)
|
||||
.managed_termination_protection(aws_sdk_ecs::types::ManagedTerminationProtection::Disabled)
|
||||
.build()
|
||||
.unwrap(),
|
||||
)
|
||||
.set_tags(Some(
|
||||
group_tags
|
||||
.iter()
|
||||
.filter(|(k, _)| k.starts_with("triple-c:"))
|
||||
.map(|(k, v)| aws_sdk_ecs::types::Tag::builder().key(k).value(v).build())
|
||||
.collect(),
|
||||
))
|
||||
.send()
|
||||
.await;
|
||||
eprintln!(
|
||||
"RESULT probe: CreateCapacityProvider {pd} over the untagged group, as the environment admin: {:?}",
|
||||
r.as_ref().map(|_| ()).map_err(|e| classify_sdk_error("ecs:CreateCapacityProvider", e).to_string())
|
||||
);
|
||||
if r.is_ok() {
|
||||
eprintln!("RESULT WARNING: a Triple-C capacity provider over a foreign group was allowed");
|
||||
let _ = ecs_admin.delete_capacity_provider().capacity_provider(&pd).send().await;
|
||||
}
|
||||
delete_group(foreign_group).await;
|
||||
}
|
||||
}
|
||||
|
||||
// ── 2. start 1: cold, with an instance launch ──
|
||||
let mut settings = AppSettings::default();
|
||||
settings.hosts = vec![conn];
|
||||
settings.image_source = ImageSource::Custom;
|
||||
settings.custom_image_name = Some(image.clone());
|
||||
let mut project = Project::new("e2e-ec2".into(), Vec::new());
|
||||
project.host_id = Some("e2e-wsuser".into());
|
||||
project.aws_placement = Some(AwsPlacement { region: region.clone(), environment: name.clone() });
|
||||
project.allow_docker_access = true;
|
||||
project.vpn_support_enabled = true;
|
||||
project.sandbox_mode_enabled = true;
|
||||
project.aws = Some(AwsWorkspaceConfig {
|
||||
launch_type: LaunchType::Ec2,
|
||||
cpu: 1024,
|
||||
memory_mib: 4096,
|
||||
storage: StorageMode::Hybrid,
|
||||
wip_push_on_stop: true,
|
||||
..Default::default()
|
||||
});
|
||||
let id = project.id.clone();
|
||||
{
|
||||
let mut g = p5_lock(&p5);
|
||||
g.project_id = Some(id.clone());
|
||||
g.secrets.push(crate::aws::tofu_modules::project_secret_name(&name, &okey, &id));
|
||||
}
|
||||
let store_dir = tempfile::tempdir().unwrap();
|
||||
let store = ProjectsStore::over_file_for_test(store_dir.path().join("projects.json"), vec![project.clone()]);
|
||||
let host = ProjectHost::for_project(&project);
|
||||
let current = |store: &ProjectsStore| store.get(&id).unwrap();
|
||||
let api = SdkTaskApi::new(&ws_cfg);
|
||||
let steps: Arc<std::sync::Mutex<Vec<(Duration, String)>>> = Default::default();
|
||||
let t0 = std::sync::Mutex::new(std::time::Instant::now());
|
||||
let progress = |m: &str| {
|
||||
let at = t0.lock().unwrap().elapsed();
|
||||
eprintln!(" progress +{at:?}: {m}");
|
||||
steps.lock().unwrap().push((at, m.to_string()));
|
||||
};
|
||||
|
||||
let started = host.ensure_started(¤t(&store), &settings, &store, &progress).await;
|
||||
if let Err(err) = &started {
|
||||
// Debugging aid: hold the workspace for an admin to look
|
||||
// at before the guard tears it down.
|
||||
let pause: u64 = std::env::var("TRIPLEC_E2E_PAUSE_SECS").ok().and_then(|v| v.parse().ok()).unwrap_or(0);
|
||||
if pause > 0 {
|
||||
eprintln!("RESULT start 1 failed ({err}); pausing {pause}s for inspection (task {:?})", current(&store).aws.and_then(|a| a.task_arn));
|
||||
tokio::time::sleep(Duration::from_secs(pause)).await;
|
||||
}
|
||||
}
|
||||
let target = started.expect("EC2 start 1");
|
||||
let first_start = t0.lock().unwrap().elapsed();
|
||||
let s1: Vec<(Duration, String)> = std::mem::take(&mut *steps.lock().unwrap());
|
||||
eprintln!("RESULT EC2 start 1 (cold, instance launch): {first_start:?}; steps {s1:?}");
|
||||
assert!(s1.iter().any(|(_, m)| m == PROGRESS_INSTANCE), "no \"Starting an instance\" step: {s1:?}");
|
||||
let aws1 = current(&store).aws.unwrap();
|
||||
let task1 = aws1.task_arn.clone().expect("the task ARN is stored");
|
||||
let revision1 = aws1.task_definition_arn.clone().expect("the revision is stored");
|
||||
assert!(task1.contains(&format!(":task/{stem}/")), "the task runs on the user's cluster: {task1}");
|
||||
let d = api.describe(&rec.cluster_arn, std::slice::from_ref(&task1)).await.expect("describe task 1");
|
||||
eprintln!("RESULT task 1: {:?}", d.first().map(|t| (&t.last_status, &t.container_instance_arn)));
|
||||
eprintln!("RESULT group after start 1: {:?}", group_size(&asg_admin, &stem).await);
|
||||
let td = ok("describe revision 1", ecs_admin.describe_task_definition().task_definition(&revision1).send().await);
|
||||
let td = td.task_definition().unwrap();
|
||||
eprintln!(
|
||||
"RESULT revision 1: network {:?}, compat {:?}, volumes {:?}",
|
||||
td.network_mode(),
|
||||
td.requires_compatibilities(),
|
||||
td.volumes().iter().map(|v| v.name().unwrap_or_default()).collect::<Vec<_>>()
|
||||
);
|
||||
|
||||
// ── through the agent (ECS Exec port forward into the bridge task) ──
|
||||
let (out, code) = sh(&target, "claude", "echo hello-from-ec2; id").await;
|
||||
eprintln!("RESULT agent tunnel over ECS Exec into the bridge-mode task: exit {code}: {}", out.trim());
|
||||
assert_eq!(code, 0, "{out}");
|
||||
let (mut docker_ok, mut vpn_ok, mut src_valid_mark, mut bwrap_ok) = (true, true, true, false);
|
||||
if !quick {
|
||||
let (out, code) = sh(&target, "claude", "ls -l /var/run/docker.sock; docker ps 2>&1; docker version --format '{{.Server.Version}}' 2>&1").await;
|
||||
eprintln!("RESULT docker ps as claude (exit {code}):\n{}", out.trim());
|
||||
docker_ok = code == 0;
|
||||
let (out, code) = sh(
|
||||
&target,
|
||||
"root",
|
||||
"ip link add wg0 type wireguard && ip link show wg0 && ip link del wg0 && echo WG-OK; \
|
||||
ip tuntap add tun0 mode tun && ip link show tun0 && ip link del tun0 && echo TUN-OK; \
|
||||
ls -l /dev/net/tun; sysctl net.ipv4.conf.all.src_valid_mark; grep CapEff /proc/self/status",
|
||||
)
|
||||
.await;
|
||||
eprintln!("RESULT VPN pieces as root (exit {code}):\n{}", out.trim());
|
||||
let (sudo_out, _) = sh(&target, "claude", "sudo -n ip link add wg1 type wireguard && sudo -n ip link del wg1 && echo SUDO-WG-OK 2>&1").await;
|
||||
eprintln!("RESULT wireguard via sudo as claude: {}", sudo_out.trim());
|
||||
vpn_ok = out.contains("WG-OK") || out.contains("TUN-OK");
|
||||
src_valid_mark = out.contains("src_valid_mark = 1");
|
||||
let (bwrap, _) = sh(
|
||||
&target,
|
||||
"claude",
|
||||
"bwrap --unshare-user --ro-bind / / true 2>&1; echo \"unshare-user rc=$?\"; \
|
||||
bwrap --ro-bind / / --dev /dev --proc /proc --unshare-all --die-with-parent true 2>&1; echo \"unshare-all rc=$?\"; \
|
||||
cat /proc/sys/user/max_user_namespaces 2>&1; uname -r",
|
||||
)
|
||||
.await;
|
||||
eprintln!("RESULT E8 bwrap on EC2:\n{}", bwrap.trim());
|
||||
bwrap_ok = bwrap.contains("unshare-user rc=0");
|
||||
let (imds, _) = sh(
|
||||
&target,
|
||||
"claude",
|
||||
"tok=$(curl -s -m 3 -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60'); \
|
||||
echo \"token rc=$? len=${#tok}\"; \
|
||||
curl -s -m 3 -o /dev/null -w 'imdsv1 http=%{http_code}\\n' http://169.254.169.254/latest/meta-data/; \
|
||||
gw=$(ip route | awk '/default/ {print $3}'); echo \"gateway $gw\"; \
|
||||
for h in \"$gw\" 127.0.0.1; do curl -s -m 3 -o /dev/null -w \"introspection $h:51678 http=%{http_code}\\n\" http://$h:51678/v1/metadata; done; \
|
||||
curl -s -m 3 http://$gw:51678/v1/metadata | head -c 300; echo; \
|
||||
test -n \"$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI\" && curl -s -m 3 http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | grep -q AccessKeyId && echo TASK-ROLE-CREDS-OK || echo task-role-creds-missing",
|
||||
)
|
||||
.await;
|
||||
eprintln!("RESULT IMDS / introspection from the container:\n{}", imds.trim());
|
||||
assert!(imds.contains("len=0"), "the container got an IMDS token: {imds}");
|
||||
|
||||
// ── 4. isolation: RunTask on another triple-c-user's cluster ──
|
||||
let other = format!("e2eother{}", &short[6..10]);
|
||||
let other_cluster = user_ec2_stem(&name, &other);
|
||||
let made = ok("admin creates another user's (empty) cluster", ecs_admin.create_cluster().cluster_name(&other_cluster).send().await);
|
||||
let other_arn = made.cluster().and_then(|c| c.cluster_arn()).unwrap().to_string();
|
||||
let fake = UserCompute {
|
||||
instance_type: "m7i.large".into(),
|
||||
max_instances: 1,
|
||||
user_tag: other.clone(),
|
||||
cluster_arn: other_arn.clone(),
|
||||
capacity_provider: other_cluster.clone(),
|
||||
};
|
||||
let r = api.run_task(ec2_run_task_request(&fake, &revision1, &workspace_tags(&owner, &id, &name))).await.map_err(ec2_run_error);
|
||||
eprintln!("RESULT RunTask on {other_cluster} as triple-c-user {user_tag}: {r:?}");
|
||||
match r {
|
||||
Err(crate::aws::error::AwsError::AccessDenied { message, .. }) => assert!(message.contains(EC2_TAG_MISMATCH), "{message}"),
|
||||
other => panic!("RunTask on another user's cluster: expected AccessDenied, got {other:?}"),
|
||||
}
|
||||
let _ = ecs_admin.delete_cluster().cluster(&other_arn).send().await;
|
||||
}
|
||||
|
||||
// ── 5. stop 1, then the scale-in to zero ──
|
||||
let t2 = std::time::Instant::now();
|
||||
host.stop_project(¤t(&store), &settings).await.expect("stop 1");
|
||||
poll(Duration::from_secs(300), Duration::from_secs(5), || async {
|
||||
let d = api.describe(&rec.cluster_arn, std::slice::from_ref(&task1)).await.ok()?;
|
||||
(d.first()?.last_status == "STOPPED").then_some(())
|
||||
})
|
||||
.await
|
||||
.expect("task 1 never stopped");
|
||||
let stopped_after = t2.elapsed();
|
||||
eprintln!("RESULT stop 1: STOPPED after {stopped_after:?}");
|
||||
store.set_aws_runtime(&id, None, aws1.access_point_id.clone(), Some(revision1.clone()), None).unwrap();
|
||||
let mut scale_in = Duration::ZERO;
|
||||
if !quick {
|
||||
let mut last: Option<(usize, i32)> = None;
|
||||
let mut scaled = None;
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(40 * 60);
|
||||
while std::time::Instant::now() < deadline {
|
||||
let now = group_size(&asg_admin, &stem).await;
|
||||
let key = now.as_ref().map(|n| (n.0, n.1));
|
||||
if key != last {
|
||||
eprintln!(" group +{:?} after STOPPED: {now:?}", t2.elapsed() - stopped_after);
|
||||
last = key;
|
||||
}
|
||||
if key == Some((0, 0)) {
|
||||
scaled = Some(());
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(30)).await;
|
||||
}
|
||||
scale_in = t2.elapsed() - stopped_after;
|
||||
eprintln!("RESULT scale-in to 0 instances: {scaled:?} after {:.1} min from STOPPED", scale_in.as_secs_f64() / 60.0);
|
||||
assert!(scaled.is_some(), "the group did not scale back to zero within 40 minutes");
|
||||
}
|
||||
|
||||
// ── start 2 ──
|
||||
*t0.lock().unwrap() = std::time::Instant::now();
|
||||
let target2 = host.ensure_started(¤t(&store), &settings, &store, &progress).await.expect("EC2 start 2");
|
||||
let second_start = t0.lock().unwrap().elapsed();
|
||||
let s2: Vec<(Duration, String)> = std::mem::take(&mut *steps.lock().unwrap());
|
||||
eprintln!("RESULT EC2 start 2 ({}): {second_start:?}; steps {s2:?}", if quick { "warm: right after stop 1" } else { "after scale-in" });
|
||||
let (out, code) = sh(&target2, "claude", "echo again; docker ps -q | wc -l").await;
|
||||
assert_eq!(code, 0, "{out}");
|
||||
let aws2 = current(&store).aws.unwrap();
|
||||
let task2 = aws2.task_arn.clone().unwrap();
|
||||
host.stop_project(¤t(&store), &settings).await.expect("stop 2");
|
||||
poll(Duration::from_secs(300), Duration::from_secs(5), || async {
|
||||
let d = api.describe(&rec.cluster_arn, std::slice::from_ref(&task2)).await.ok()?;
|
||||
(d.first()?.last_status == "STOPPED").then_some(())
|
||||
})
|
||||
.await
|
||||
.expect("task 2 never stopped");
|
||||
store.set_aws_runtime(&id, None, aws2.access_point_id.clone(), aws2.task_definition_arn.clone(), None).unwrap();
|
||||
let removed = host.remove_aws_resources(¤t(&store), &settings, true).await;
|
||||
eprintln!("RESULT remove_aws_resources: {:?}", removed.as_ref().map_err(|e| e.chars().take(400).collect::<String>()));
|
||||
removed.expect("remove the project's resources");
|
||||
|
||||
// ── 7. the onboarding re-applied without EC2, an instance still up ──
|
||||
eprintln!("RESULT group before the re-apply without EC2: {:?}", group_size(&asg_admin, &stem).await);
|
||||
let (keys, acfg, _) = t.session(&admin).await;
|
||||
let vars = prepare_onboarding(&SdkOnboardApi::new(&acfg), &name, ®ion, &account, &owner, Some(Ec2Choice::None))
|
||||
.await
|
||||
.expect("the pre-apply step without EC2");
|
||||
let t5 = std::time::Instant::now();
|
||||
let (_, outcome, denied) = t
|
||||
.run(ModuleKind::UserOnboard, Deployment::User { env: &name, owner_key: &okey }, &vars, TofuOp::Apply, t.keys_env(&keys))
|
||||
.await;
|
||||
eprintln!("RESULT re-apply without EC2: {:?} in {:?}", outcome.summary, t5.elapsed());
|
||||
assert_ok("re-apply the onboarding without EC2", &outcome, &denied);
|
||||
let g = group_size(&asg_admin, &stem).await;
|
||||
let inst = live_env_instances(®ion, &name).await;
|
||||
let cps = ecs_admin.describe_capacity_providers().capacity_providers(&stem).send().await;
|
||||
let cl = ecs_admin.describe_clusters().clusters(&stem).send().await;
|
||||
let (_, lt, lt_err) = aws_cli(None, ®ion, &["ec2", "describe-launch-templates", "--launch-template-names", &stem]).await;
|
||||
let prof = t.e.iam.get_instance_profile().instance_profile_name(&role_name).send().await;
|
||||
let role = t.e.iam.get_role().role_name(&role_name).send().await;
|
||||
let record = read_user_record(&SdkRuntimeApi::new(&ws_cfg), &name, &okey, &account, ®ion).await;
|
||||
eprintln!(
|
||||
"RESULT after the re-apply: group {g:?}; live instances {inst:?}; provider {:?}; cluster {:?}; template {} {}; profile gone {}; role gone {}; record {record:?}",
|
||||
cps.as_ref().map(|o| o.capacity_providers().iter().map(|c| c.status().cloned()).collect::<Vec<_>>()),
|
||||
cl.as_ref().map(|o| o.clusters().iter().map(|c| c.status().map(str::to_string)).collect::<Vec<_>>()),
|
||||
lt.trim().chars().take(100).collect::<String>(),
|
||||
lt_err.trim().chars().take(100).collect::<String>(),
|
||||
prof.is_err(),
|
||||
role.is_err(),
|
||||
);
|
||||
assert!(g.is_none(), "the group is gone");
|
||||
assert!(inst.is_empty(), "no instance is left: {inst:?}");
|
||||
assert!(prof.is_err() && role.is_err(), "the instance profile and role are gone");
|
||||
assert!(matches!(record, Ok(UserRecord::Absent)), "{record:?}");
|
||||
|
||||
// ── offboard, then destroy the environment, as the environment admin ──
|
||||
if let Some(role) = p5_lock(&p5).exec_role.take() {
|
||||
let _ = t.e.iam.delete_role_policy().role_name(&role).policy_name(ECR_POLICY).send().await;
|
||||
}
|
||||
let (keys, _, _) = t.session(&admin).await;
|
||||
let (_, outcome, denied) = t
|
||||
.run(ModuleKind::UserOnboard, Deployment::User { env: &name, owner_key: &okey }, &vars, TofuOp::Destroy, t.keys_env(&keys))
|
||||
.await;
|
||||
assert_ok("offboard", &outcome, &denied);
|
||||
let (_, outcome, denied) = t
|
||||
.run(
|
||||
ModuleKind::Environment,
|
||||
Deployment::Environment(&name),
|
||||
&environment_vars(&name, ®ion, &NetworkChoice::Public, 0).unwrap(),
|
||||
TofuOp::Destroy,
|
||||
t.keys_env(&keys),
|
||||
)
|
||||
.await;
|
||||
assert_ok("destroy the environment", &outcome, &denied);
|
||||
eprintln!(
|
||||
"RESULT summary: docker {docker_ok}, vpn {vpn_ok}, src_valid_mark {src_valid_mark}, bwrap {bwrap_ok}, start1 {first_start:?}, scale-in {:.1} min, start2 {second_start:?}",
|
||||
scale_in.as_secs_f64() / 60.0
|
||||
);
|
||||
let _ = ws;
|
||||
assert!(docker_ok, "docker ps failed as claude");
|
||||
assert!(vpn_ok && src_valid_mark, "the VPN pieces are missing");
|
||||
});
|
||||
let body = async {
|
||||
if tokio::time::timeout(PHASE7_TIMEOUT, inner).await.is_err() {
|
||||
panic!("the phase-7 body did not finish in {PHASE7_TIMEOUT:?}");
|
||||
}
|
||||
};
|
||||
let result = std::panic::AssertUnwindSafe(body).catch_unwind().await;
|
||||
let guard = std::mem::take(&mut *p5_lock(&p5));
|
||||
phase5_cleanup(&t.e, &guard).await;
|
||||
if let Err(panic) = result {
|
||||
std::panic::resume_unwind(panic);
|
||||
}
|
||||
})
|
||||
.await;
|
||||
}
|
||||
}
|
||||
+88
-58
@@ -554,66 +554,55 @@ aws iam attach-user-policy --user-name <user> \
|
||||
`containerOverrides`: every workspace user can read them through `ecs:Describe*` on `*`. Use
|
||||
Secrets Manager (`triple-c/*`) references instead.
|
||||
|
||||
**Phase 7 (per-user EC2 capacity): still open until the live run (Task 7)**
|
||||
**Phase 7 (per-user EC2 capacity): the live run's answers (Task 7, 2026-11-16)**
|
||||
|
||||
The unit tests' evaluator is not IAM; these points of the compute file and the boundary are
|
||||
expected, not verified:
|
||||
The unit tests' evaluator is not IAM. What the live run (see "Phase 7: ECS on EC2" below) settled,
|
||||
and what is still open:
|
||||
|
||||
- **Creating an Auto Scaling group checks the caller's launch permissions.** Expected: a dry run
|
||||
of `ec2:RunInstances` from the launch template as the admin, with `ec2:LaunchTemplate` set on
|
||||
every resource, the template's own tags visible, `ec2:Owner = amazon` on the image, and
|
||||
`iam:PassRole` of the instance role to `ec2.amazonaws.com`. If the dry run omits
|
||||
`ec2:LaunchTemplate` on some resource, the group's create is denied and that statement needs
|
||||
another condition.
|
||||
- **`iam:AddRoleToInstanceProfile`** is expected to need `iam:PassRole` of the role; which
|
||||
`iam:PassedToService` IAM sends then (if any) is unknown, and only `ec2.amazonaws.com` is allowed.
|
||||
- **`ecs:PutClusterCapacityProviders`** may be authorized against the capacity provider as well as
|
||||
the cluster (both are allowed); **`ecs:CreateCapacityProvider`** may check the caller's
|
||||
`autoscaling:Describe*` on the group (allowed) or create ECS's service-linked role (allowed).
|
||||
- **Service-linked roles.** The Auto Scaling and ECS roles exist in most accounts already;
|
||||
creating them is allowed only for those two services. Whether creating a group in an account
|
||||
without one creates it as the caller (needing this grant) is not verified.
|
||||
- **The instance role's agent calls** (`ecs:Poll`, `StartTelemetrySession`,
|
||||
`UpdateContainerInstancesState`) are expected to be authorized against
|
||||
`container-instance/<cluster>/<id>` and the rest against the cluster; the boundary allows both.
|
||||
- **The public AMI parameter's ARN** has no account (`arn:aws:ssm:<region>::parameter/aws/…`);
|
||||
`arn:aws:ssm:*::parameter/aws/service/ecs/optimized-ami/*` is expected to match it.
|
||||
- **Version pinning.** `autoscaling:LaunchTemplateVersionSpecified` is expected to be `true` for a
|
||||
numeric version and `false` for `$Latest`/`$Default`, and absent on an update that names no
|
||||
template (allowed by `BoolIfExists`, so the drain before destroy works). Probe: create a group
|
||||
with `$Latest` as the admin; it must be denied.
|
||||
- **`ec2:IsLaunchTemplateResource`** is expected to be `true` on the image and security group the
|
||||
template names, and `aws:RequestTag` to carry the template's tag specifications for the instance
|
||||
and volume. If the group's launch check doesn't send either, that statement denies the create.
|
||||
- ~~**`autoscaling:InstanceTypes`** required on a group create.~~ **Verified live (phase 7):** a
|
||||
template-only create sends **no** `autoscaling:InstanceTypes`, so the `Null: false` denied every
|
||||
group (the first live onboarding failed on it); it is dropped. Creating a group runs the caller's
|
||||
own `ec2:RunInstances` check on the template ("You are not authorized to use launch template"
|
||||
otherwise), so `ec2:InstanceType` holds the type; a `MixedInstancesPolicy` override of an
|
||||
unsupported type is denied by both checks.
|
||||
- **Instance types in templates.** `ec2:CreateLaunchTemplate`/`CreateLaunchTemplateVersion` are
|
||||
not known to support `ec2:InstanceType` (not in the service authorization reference as I know
|
||||
it), so a template may *name* any type; it can only be *launched* (directly or by a group) with a
|
||||
supported one. Probe: create a version with `p5.48xlarge` and pin a group to it; the group
|
||||
create must be denied.
|
||||
- **`aws:PrincipalTag/triple-c-user` from Identity Center** attributes for access control is
|
||||
expected in the SSO session's request context, and `${aws:PrincipalTag/triple-c-user}` in
|
||||
`ArnNotLike` to resolve from it. Probe: as a tagged and an untagged user, `RunTask` on a
|
||||
per-user cluster.
|
||||
- **Tags at group create.** If `CreateAutoScalingGroup` with tags also needs
|
||||
`autoscaling:CreateOrUpdateTags`, it is denied (that is allowed only on an already-tagged
|
||||
group); then add it to the create statement with the same conditions.
|
||||
- **A `triple-c-*` capacity provider pointing at someone else's group.** Whether
|
||||
`CreateCapacityProvider` checks the caller's rights on the named Auto Scaling group is unknown;
|
||||
if it does not, an admin could put a non-Triple-C group under a Triple-C capacity provider (and
|
||||
ECS's managed scaling would then scale it). Probe: create one naming an untagged group.
|
||||
- **`aws:PrincipalTag` for the instance role**: the role's session is expected to carry its role
|
||||
tags, so `${aws:PrincipalTag/triple-c:cluster}` resolves for the agent's calls.
|
||||
- **IMDS and Docker access.** Tasks use `awsvpc`, with `ECS_AWSVPC_BLOCK_IMDS=true` and a hop
|
||||
limit of 1 so a task can't read the instance role. A project with Docker access mounts the
|
||||
host's socket, so its user can start a host-network container and read IMDS (and so the
|
||||
instance role): accepted, since Docker access is root on the user's own instance; the boundary
|
||||
keeps that role on the user's own cluster.
|
||||
- ✓ **Creating an Auto Scaling group checks the caller's launch permissions.** The onboarding's
|
||||
group create as the environment admin (only the four admin files) passed with no denial. A group
|
||||
pinned to a template version naming `p5.48xlarge` was refused with "You are not authorized to
|
||||
use launch template", so the create does run the caller's `ec2:RunInstances` check, with
|
||||
`ec2:LaunchTemplate`, `ec2:IsLaunchTemplateResource`, the template's tags and the instance role's
|
||||
`iam:PassRole` all as the statements expect.
|
||||
- ✓ **`iam:AddRoleToInstanceProfile`, `ecs:PutClusterCapacityProviders`,
|
||||
`ecs:CreateCapacityProvider`**, the public AMI parameter read, and tags at group create
|
||||
(`autoscaling:CreateOrUpdateTags` was not needed separately): all passed in the real apply,
|
||||
with no denial. Which context keys each one sent is not visible; the policy is not narrowed further.
|
||||
- — **Service-linked roles.** The test account already had both. Creating them as the admin is
|
||||
still unverified.
|
||||
- ✓ **The instance role's agent calls**, under the boundary: the instance registered into the user's
|
||||
cluster, polled, and ran tasks. ECR pulls through the execution role and logs worked too. So the
|
||||
`triple-c:cluster` role tag resolves as `${aws:PrincipalTag/triple-c:cluster}`.
|
||||
- ✓ **Version pinning.** `$Latest` and `$Default` are denied, and a numeric version is allowed (a
|
||||
probe group with min/max 0, created and deleted as the admin).
|
||||
- ✗→fixed **`autoscaling:InstanceTypes` required on a group create.** A template-only create sends
|
||||
**no** `autoscaling:InstanceTypes`, so the `Null: false` denied every group (the first live
|
||||
onboarding failed on it). It is dropped. Creating a group runs the caller's own
|
||||
`ec2:RunInstances` check on the template, so `ec2:InstanceType` holds the type. A
|
||||
`MixedInstancesPolicy` override of an unsupported type is denied by both checks
|
||||
(`ForAllValues` on the key once present, and the launch check).
|
||||
- ✓ **Instance types in templates.** The admin *can* create a template version naming
|
||||
`p5.48xlarge` (as expected, no condition key). A group pinned to it is denied by the launch check.
|
||||
- ✓ **`aws:PrincipalTag/triple-c-user`** from a **role tag**: the tagged workspace role's session
|
||||
ran tasks on its own `--u-<tag>` cluster. `RunTask` on another `--u-<other>` cluster was denied,
|
||||
and the app shows the wrong-triple-c-user message. — An Identity Center attribute and an
|
||||
untagged user were not tried. The policy's tag test is the same for both, so this is expected
|
||||
to hold.
|
||||
- ✓ **A `triple-c-*` capacity provider over someone else's group** is refused. ECS checks the
|
||||
caller's `autoscaling:CreateOrUpdateTags` on the named group ("you do not have
|
||||
autoscaling:CreateOrUpdateTags permissions to create tags on the Auto Scaling group"), which the
|
||||
admin tier allows only on component-tagged `triple-c-*--u-*` groups.
|
||||
- **IMDS and Docker access.** EC2 tasks use `bridge` networking (R2), and the template sets IMDSv2
|
||||
with a hop limit of 1. Live: from the workspace container, the token `PUT` gets no answer
|
||||
(curl times out) and an IMDSv1 `GET` gets 401, so no instance-role credentials. The task role's
|
||||
credentials (169.254.170.2) work. The ECS agent's introspection API **is** reachable on the
|
||||
bridge gateway (`172.17.0.1:51678`, 200). It returns the cluster, the container instance and
|
||||
the instance's tasks, and no credentials. That is accepted: the instance runs only its user's
|
||||
tasks (E1). A project with Docker access has the host's socket (group `docker-host`, `docker
|
||||
ps` works as `claude`), so its user can start a host-network container and read IMDS (and so the
|
||||
instance role). That is accepted too: Docker access is root on the user's own instance, and
|
||||
the boundary keeps that role on the user's own cluster.
|
||||
|
||||
**Phase 6**
|
||||
|
||||
@@ -814,3 +803,44 @@ spend under $0.05.
|
||||
| Cost estimate | ✓ `fetch_rates` as the workspace user + `estimate`: $0.0495/hour for 1 vCPU / 2 GiB / 21 GiB x86, `efs_billed_separately` |
|
||||
| Removal with discard | ✓ `remove_aws_resources(…, true)` Ok; both access points gone; the secret deleted |
|
||||
| Cleanup | ✓ nothing `triple-c` left in us-west-2 (the run's environment KMS key pending deletion). The guard's task-definition retirement had never matched anything (ECS's `familyPrefix` is a whole family name); fixed, and 13 revisions earlier runs had left ACTIVE were deregistered and deleted |
|
||||
|
||||
### Phase 7: ECS on EC2 (2026-11-16)
|
||||
|
||||
```sh
|
||||
TRIPLEC_E2E_TEST=aws::live_tests::environment::a_phase7_ec2_workspace_round_trips \
|
||||
TRIPLEC_AWS_E2E_PROFILE=<admin profile> TRIPLEC_AWS_E2E_REGION=us-west-2 scripts/aws-e2e-workspace.sh
|
||||
```
|
||||
|
||||
Same fixture as phase 5. The account admin applies the current boundary (the account had none,
|
||||
so it is created and destroyed again). The environment admin role holds all four admin files.
|
||||
The workspace role carries the role tag `triple-c-user=<tag>`, which an assumed-role session has
|
||||
as a principal tag. The admin onboards it with `m7i.large`, at most 1 instance, through the app's
|
||||
own pre-apply step (`prepare_onboarding`: record, choice, name-collision check). The project is
|
||||
EC2, Hybrid, 1024 CPU units / 4096 MiB reserved, with Docker access, VPN and sandbox mode on, and
|
||||
no repos. The guard (`ec2_cleanup`, as the account admin) stops the per-user clusters' tasks,
|
||||
force-deletes their groups (waiting until they are gone), terminates any instance tagged with the
|
||||
environment, then deletes capacity providers, container instances, clusters (retried while busy),
|
||||
launch templates, the user record and the instance role and profile. After that the fixture
|
||||
destroys the modules. `TRIPLEC_E2E_PAUSE_SECS` holds a failed first start for inspection before
|
||||
the guard runs.
|
||||
|
||||
| Point | Result |
|
||||
|---|---|
|
||||
| Onboarding with EC2 | ✓ after the `InstanceTypes` fix (the first run's group create was denied, see above). Applied in 93 s: cluster `triple-c-<env>--u-<tag>` ACTIVE with exactly its capacity provider as the default strategy; group min 0, max 1, desired 0, template pinned to version 1, scale-in protection on, tagged with component/environment/owner/version; capacity provider with managed scaling at target 100 and managed termination protection; launch template `m7i.large`, the current ECS-optimized AL2023 AMI, IMDSv2 required, hop limit 1; instance profile holding the instance role; the user record reads `Valid` as the workspace user |
|
||||
| Collision check | ✓ `DescribeClusters include=TAGS` as the admin tier returns the cluster's `triple-c:owner` tag. A missing name answers `None` for both the cluster (a `failures` entry) and the group. Onboarding another owner with the same `triple-c-user` value is refused: "An ECS cluster named … already belongs to another onboarding" |
|
||||
| Admin-tier probes | ✓ `$Latest` denied; pinned version allowed and deleted; a `p5.48xlarge` template version created but a group pinned to it denied; a capacity provider over an untagged group denied (see "Phase 7" under Known gaps) |
|
||||
| **ECS Exec on a cold start** | ✗→fixed (app). ECS turns `enableExecuteCommand` **off** on a RunTask that has to wait for an instance (the task goes PROVISIONING behind the capacity provider). CloudTrail shows `true` in the request and `false` in the answer, and the task keeps it off once placed, so the ExecuteCommandAgent never starts and every cold start timed out ("ECS Exec agent not running yet"). The same RunTask with an instance up and room keeps it `true`, and one that has to wait gets `false` even with an instance up. A cold start now lets that first task bring the instance up. Once the task is placed, the app stops it, waits until it has stopped and runs the task again onto the instance (`rerun_once_placed`). A live task found with ECS Exec off is stopped and replaced |
|
||||
| Cold start | ✓ 249 s (run 4) and 203 s (run 5) from Start to connected. Run 4: RunTask at +4 s; "Starting an instance" until +186 s (instance launch, ECS registration and placement of the first task, which is then stopped); the second task ready and the tunnel at +248 s |
|
||||
| Agent tunnel, bridge mode | ✓ the ECS Exec port forward reaches the agent on the container's own loopback (`echo` through the agent as `claude`); the task definition is `bridge`/`EC2` with the `docker-socket` host volume |
|
||||
| Docker access | ✓ `/var/run/docker.sock` is `root:docker-host 0660`, and `claude` is in `docker-host`; `docker ps` as `claude` lists the workspace and the ECS agent (Docker 25.0.18) |
|
||||
| VPN | ✓ as root: `ip link add wg0 type wireguard` and `ip tuntap add tun0 mode tun` both work; `/dev/net/tun` is there; `net.ipv4.conf.all.src_valid_mark = 1`; `sudo ip link add … wireguard` works as `claude` |
|
||||
| E8 bubblewrap | ✗→decided. `bwrap --unshare-user` and `--unshare-all` fail with "No permissions to create new namespace", although the kernel (6.1, AL2023) allows user namespaces (`max_user_namespaces` 30937). Docker's default seccomp profile refuses `unshare` without `CAP_SYS_ADMIN`. The sandbox is kept off on EC2 as on Fargate (`sandbox_allowed`, the Runtime toggle) |
|
||||
| IMDS / introspection | see "IMDS and Docker access" above: no token (hop limit 1), IMDSv1 401, task-role credentials work, introspection reachable on the bridge gateway (metadata only) |
|
||||
| Isolation | ✓ `RunTask` on another user's cluster (`triple-c-<env>--u-<other>`, created empty by the admin) as the tagged workspace user is denied, and the app's error is the wrong-triple-c-user one ("This EC2 capacity belongs to a different triple-c-user value than your sign-in's …") |
|
||||
| Stop | ✓ STOPPED 6 s after `stop_project` |
|
||||
| Scale-in to zero | ✓ from the group's activity history (run 4). The task STOPPED at 00:55:42Z. At 01:11:08Z managed scaling's low alarm set desired capacity 1 → 0. Scale-in protection first held the instance ("all remaining instances are protected from scale-in"). ECS then removed it from the now-idle instance, which was terminated at 01:11:59–01:12:44Z. That is **about 15.5 minutes to the scale-in decision and 17 minutes until the instance is gone**. The cold start's scale-out is similar: the high alarm fired about 3 minutes after the RunTask, and the instance was running a minute later |
|
||||
| Second start | ✓ warm, right after stop 1 with the instance still up: 29 s (no instance wait, image cached, ECS Exec on at once). After a scale-in a start is cold again (≈ 3.5–4 min) |
|
||||
| Removal | ✓ `remove_aws_resources(…, true)` after stop 2: Ok (task, secret, access point) |
|
||||
| Re-apply without EC2, instance still up | ✓ `prepare_onboarding(…, None)` then the apply as the environment admin, 79 s: "0 added, 0 changed, 9 destroyed". The provider drained the group (the instance was InService, scale-in protected) and deleted it; after the apply no group, no live instance, capacity provider INACTIVE, cluster INACTIVE, launch template gone, instance profile and role gone, the user record Absent (as the workspace user). No denied action |
|
||||
| Offboard and destroy | ✓ as the environment admin: onboarding destroy (9) and environment destroy (15) with no denied action; the account admin's boundary destroy after |
|
||||
| Runs and cleanup | Five runs. Run 1: the `InstanceTypes` denial. Run 2: ECS Exec off on the cold start. Run 3: inspection over SSM (Session Manager core attached by hand to the instance role, boundary removed, both undone before the teardown), the ECS Exec diagnosis and the in-task checks. Run 4: everything to the scale-in, which the guard could not finish because the credentials expired. Run 5, `TRIPLEC_E2E_PHASE7_SHORT=1` (life cycle only): passed in 577 s. Every run's guard, or by hand with tofu destroy from the run's work folders once the credentials were renewed, left nothing: the final inventory of us-west-2 has no instance, group, launch template, Triple-C capacity provider, cluster, `triple-c*` role, policy or instance profile, `/triple-c` parameter, log group, ECR repository, file system, secret, task definition, VPC, state bucket or boundary. The environments' KMS keys are pending deletion (7 days). Spend: about 1.5 m7i.large instance-hours plus minutes of EFS and public IPv4, under $0.30 |
|
||||
@@ -834,6 +834,21 @@ detached runner, detection, self-onboarding, an update ("No changes"), destroy;
|
||||
file-system policy accepted the just-created task role; conditional `DeleteObject` (`If-Match`)
|
||||
on the versioned bucket returns 412 on a stale ETag.
|
||||
|
||||
- **ECS on EC2, measured 2026-11-16** (phase 7, `aws/README.md` "Phase 7: ECS on EC2"):
|
||||
- A per-user capacity (cluster, capacity provider, Auto Scaling group at 0, launch template)
|
||||
applies as the environment admin once the scaling file stops requiring
|
||||
`autoscaling:InstanceTypes`. A template-only group create doesn't send it, so the create's
|
||||
own launch check (`ec2:InstanceType`) is what holds the type.
|
||||
- ECS turns `enableExecuteCommand` off on a RunTask that has to wait for an instance, so a cold
|
||||
start runs the task a second time once the instance is up. A cold start takes about 4 minutes.
|
||||
- Bridge networking works for the agent tunnel (ECS Exec port forward), Docker access (host
|
||||
socket) and VPN (`NET_ADMIN`, `/dev/net/tun`, `src_valid_mark`).
|
||||
- Bubblewrap **does not** work in an ECS task on EC2 either (Docker's seccomp profile), so the
|
||||
sandbox stays off on AWS.
|
||||
- IMDS gives a bridge container no token. The agent's introspection port is reachable from it
|
||||
(metadata only).
|
||||
- The `triple-c-user` principal tag (a role tag) confines RunTask to the user's own cluster.
|
||||
|
||||
Still to verify: the `existing` network mode against a user's own VPC. Fargate tasks in the
|
||||
deployed environment were run in phase 5 (`aws/README.md`).
|
||||
|
||||
|
||||
Reference in new issue
Block a user