AWS host phase 7: ECS on EC2 (per-user capacity, Docker access, VPN) #84

Merged
jknapp merged 29 commits from feat/aws-ec2 into main 2026-11-17 14:28:45 +00:00
29 Commits
Author SHA1 Message Date
shadowdaoandClaude Opus 5.5 05ed1c37a5 fix(aws): phase 7 parked minors
Build App (Preview) / compute-version (pull_request) Successful in 9s
Secret Scan / scan (push) Successful in 21s
Secret Scan / scan (pull_request) Successful in 9s
Build App (Preview) / create-release (pull_request) Successful in 2s
Build Container / build-container (pull_request) Successful in 2m59s
Build App (Preview) / build-macos (pull_request) Successful in 6m17s
Build App (Preview) / build-windows (pull_request) Successful in 12m0s
Build App (Preview) / build-linux (pull_request) Successful in 21m5s
Build App (Preview) / prune-previews (pull_request) Successful in 4s
Build App (Preview) / test (pull_request) Successful in 24m1s
Interrupted-start notice gets its own toast topic and headline; the launch-type switch note only shows after a successful save and is dropped by later saves; README states the CreateLogStream scoping limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 05:24:56 -08:00
shadowdaoandClaude Opus 5.5 70cfa6b757 test(aws): Fargate logs under the narrowed scope
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:01:28 -08:00
shadowdaoandClaude Opus 5.5 6fa7ef5605 docs(aws): record phase 7 live run 7 (log scoping)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:44:47 -08:00
shadowdaoandClaude Opus 5.5 d85292ea00 fix(aws): add the EC2 root volume to the cost estimate; name public IPv4
The EC2 estimate adds the 50 GiB gp3 root volume at the region's gp3
GB-month price (Pricing API, cached per region like the instance price;
fixture recorded 2026-11-16). The note says whether it is included and
that a public-mode environment's IPv4 address adds $0.005/hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:40:12 -08:00
shadowdaoandClaude Opus 5.5 1d1e356fcd fix(aws): EC2 host settings polish
HostSection says what a launch-type switch changed besides the type,
checks the saved reservations against the instance type once on load,
and is re-indented. The EC2 capacity cache keeps none/unusable for one
minute only. RuntimeSection's VPN hint on EC2 speaks of an ECS task on
the user's instance. Joining over an incomplete onboarding with no
record warns that None removes half-built EC2 capacity, and starts from
the capacity last chosen in this run when there is one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:37:07 -08:00
shadowdaoandClaude Opus 5.5 77db6028d0 fix(aws): write logs only into the user's own streams; no log rights on the instance role
The instance role (reachable from a Docker-access project's host-network
container) could PutLogEvents into any user's streams. It now has no log
rights: awslogs writes with the execution role
(ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true). The task and execution
roles create streams in the environment's group but write only into
log-stream:<owner_key>/* (every awslogs stream's prefix); the Fargate
fingerprint moves deliberately. The live test asserts task logs are
written. README: per-user logs, orphan clock skew, verdict wording;
boundary version comments no longer mention Session Manager.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:33:57 -08:00
shadowdaoandClaude Opus 5.5 8fe0810901 fix(aws): never let the user record vouch for EC2 names tagged as another owner's
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:30:41 -08:00
shadowdaoandClaude Opus 5.5 8c9ffd4c4c fix(aws): never attach to an exec-off EC2 task after a restart; widen the orphan margin
Reconcile stops a placed or orphaned exec-off task (REPLACED_REASON,
checked; the ARN is cleared only once the stop succeeded) and marks a
recent one still waiting Error with its ARN kept and an 'interrupted'
warning. Reattach refuses an exec-off task. ORPHAN_MARGIN is 10 minutes
(clock skew). A failed EC2 record read no longer blocks attaching to a
live task. One helper builds the task cluster list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:30:05 -08:00
shadowdaoandClaude Sonnet 5.5 719a75b46d docs(aws): phase 7
Spec: phase 7 done, rulings (E1-E11 as built, R1-R4, cold-start rerun),
data model, feature table. CLAUDE.md: EC2 rules and the docker-socket
step; sandbox is off on all AWS. aws/README.md: triple-c-user admin
setup and caveats, measured sizes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-16 18:19:39 -08:00
shadowdaoandClaude Opus 5.5 6e24977cf0 fix(aws): replace an orphaned exec-off EC2 task instead of refusing for ever
A found task with ECS Exec off that still waits for an instance is
another start's only while one could still be waiting for it (created
within the EC2 placement wait plus 2 minutes); an older one is a crashed
start's orphan and is replaced (stopped with its own reason, checked,
waited for). INSTANCE_RACE now mentions Stop. The rerun's race records
the winner's task, as the first launch's race does; a first task another
start replaced says so. TaskInfo gains created_unix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:16:22 -08:00
shadowdaoandClaude Opus 5.5 374fb5a35d test(aws): harden the phase 7 live test; record the update probe
The guard paginates Auto Scaling groups and treats a failed read as not
yet gone; the test asserts exactly one live task after a cold start,
the sandbox off in the task's settings and bwrap failing (E8). README:
updating a group to an unsupported template version is denied by the
launch check (live probe); run 6 after the review fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:12:51 -08:00
shadowdaoandClaude Opus 5.5 10f473f3da fix(aws): show the sandbox as off wherever an AWS project can't run it
PermissionModeControl read the raw sandbox_mode_enabled, so an AWS
project showed "Sandbox ON — bubblewrap isolation" and no Bypass
caution although the host forces it off. A shared sandboxEffective()
(false for any AWS project) now feeds both it and the Runtime toggle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:59:44 -08:00
shadowdaoandClaude Opus 5.5 c25761789e fix(aws): never run a second task for a stopped or contested cold EC2 start
The cold-start rerun now runs again only once an instance has taken the
first task and it is still meant to run; a first task that stops by
itself (a user's Stop, a failed placement) ends the start with its
reason. The second launch goes through find_or_run, so a concurrent
start's task is reported as START_RACE instead of doubled. A found task
with ECS Exec off that is still waiting for an instance belongs to a
start in progress (INSTANCE_RACE); a placed one is stopped, the stop is
checked and waited for (its own 3-minute budget) before a new run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:59:09 -08:00
shadowdaoandClaude Opus 5.5 fc3456c44e test(aws): live phase 7 EC2 checks
a_phase7_ec2_workspace_round_trips: onboarding with per-user EC2
capacity through prepare_onboarding (tagged workspace role, all four
admin files), its collision check and refusal, admin-tier scaling
probes, a cold and a warm EC2 start (Docker access, VPN, ECS Exec into
the bridge task), in-container checks (docker ps, wg/tun,
src_valid_mark, bwrap, IMDS, introspection), RunTask on another user's
cluster, stop and scale-in, removal, re-apply without EC2 with an
instance up, offboard and destroy. The guard (ec2_cleanup) force-deletes
any per-user EC2 capacity. Results in aws/README.md and the spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:52:36 -08:00
shadowdaoandClaude Opus 5.5 6c7c2ea4ef fix(aws): keep Claude Code's sandbox off on EC2 too (E8)
Measured live on the ECS-optimized AL2023 AMI (kernel 6.1,
max_user_namespaces 30937): bwrap --unshare-user in the ECS task fails
with "No permissions to create new namespace", as on Fargate. Docker's
default seccomp profile refuses it without CAP_SYS_ADMIN. sandbox_allowed
is false for every AWS project, and the Runtime toggle is locked off on
EC2 as on Fargate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:35:30 -08:00
shadowdaoandClaude Opus 5.5 43fcc520ab fix(aws): run a cold EC2 task again once its instance is up
ECS turns enableExecuteCommand off on a RunTask that has to wait for an
instance (PROVISIONING behind a capacity provider), and the task keeps it
off once placed: verified live (CloudTrail shows true in the request,
false in the answer; a RunTask with an instance up keeps it). Such a
task can never be reached, so every cold EC2 start timed out.

A cold start now lets that first task bring the instance up, stops it as
soon as it is placed, waits for it to stop and runs the task again onto
the instance; a second task without ECS Exec is stopped and reported. A
live task found with ECS Exec off is stopped and replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:34:38 -08:00
shadowdaoandClaude Opus 5.5 bf5a810e0d fix(aws): allow template-only Auto Scaling group creates
A CreateAutoScalingGroup that names only a launch template sends no
autoscaling:InstanceTypes (verified live), so the scaling file's
Null:false on it denied every per-user group. Drop it; the type is held
by the caller's own ec2:RunInstances launch check on the template, which
also refuses MixedInstancesPolicy overrides of unsupported types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:24:55 -08:00
shadowdaoandClaude Opus 5.5 9e1a63cde5 feat(aws): EC2 projects in the UI, Docker access and VPN
Host settings get the launch-type control (Fargate | EC2 with the user's
instance type, disabled with the reason when their onboarding has no EC2
capacity), switching through launchTypePatch, which now also clears the
gateway sidecar and Spot for EC2 and snaps sizes per side. EC2 projects size
CPU/memory reservations checked by Rust's fit rule (ec2_fit_problem, new
command) against the user's record (aws_ec2_capacity, new command); no
ephemeral size, Spot or architecture; notes for scale-to-zero, Docker access
as root on the instance and no sidecar. The sandbox follows the project on EC2.

The cost estimate prices an EC2 project as its instance type's AmazonEC2
on-demand Linux shared-tenancy price (recorded fixture), cached per region and
type with the Fargate cache's rules. The entrypoint's Docker-socket group step
now runs in aws mode when the socket exists; the VPN skill follows
VPN_SUPPORT_ENABLED there as on Docker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:03:57 -08:00
shadowdaoandClaude Opus 5.5 d6ef6a5fd2 feat(aws): run projects on per-user EC2 capacity
EC2 projects register an EC2/bridge task definition (no runtime platform or
ephemeral storage; task-level reservations checked against the user's
instance type with a 10% memory margin), with the instance's Docker socket
and the Docker VPN pieces (NET_ADMIN, /dev/net/tun, src_valid_mark) when
enabled, and run on the user's cluster through their capacity provider.
The start reads the user record, waits up to 10 minutes with "Starting an
instance" progress, and maps RunTask access denied to the triple-c-user
mismatch. Stop, reconcile, removal and the watch follow the task's own
cluster, so a running EC2 task outlives its record. The sandbox stays on
for EC2; the gateway sidecar is refused there for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:46:06 -08:00
shadowdaoandClaude Opus 5.5 98e330060e fix(aws): EC2 name check by owner tag; keep unusable records; bundle warning
The pre-apply check now decides ownership by the resources' triple-c:owner
tag (record as fallback), so a failed apply of one's own capacity can be
re-applied; any cluster status but INACTIVE holds the name. An onboarding
seam (OnboardApi) makes the record read and the name lookups testable, and
prepare_onboarding settles everything before launch. With no choice given,
an unreadable or invalid record is refused instead of removing capacity.
Bundles warn when their cluster name looks taken; the README check prints
owner tags. The UI keeps the recorded type when the instance list fails and
says when an apply removes EC2 capacity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:31:30 -08:00
shadowdaoandClaude Opus 5.5 1a2c530be8 feat(aws): choose EC2 capacity when onboarding
Self-onboarding and the onboarding bundle take an EC2 capacity choice:
none, or an instance type from the app's table (grouped x86/ARM, default
m7i.large), 1-4 instances and the user's triple-c-user value. A re-apply
keeps the recorded capacity unless changed (the record gains
max_instances). Before any self-onboarding apply with EC2 the app refuses
per-user names (ACTIVE cluster or Auto Scaling group) that this owner's
record doesn't name (R3). OnboardingState carries the user's compute from
their record; the panel shows it. The bundle README's collision check now
comes before the apply step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:22:31 -08:00
shadowdaoandClaude Opus 5.5 e9c20287b7 fix(aws): EC2 name-collision step in onboarding bundles; bridge agent config
Task 3 fix round 1 (rulings R2, R3):
- Onboarding admin bundles' README gains an "EC2 capacity" step: before
  applying with ec2_instance_type, check with describe-clusters /
  describe-auto-scaling-groups that triple-c-<env>--u-<tag> is unused
  unless this owner's /triple-c/users/<env>/<owner key> record names it
  (exact names; <tag> when the bundle has none). Tested.
- User data for bridge tasks (R2): ECS_ENABLE_TASK_IAM_ROLE=true instead
  of ECS_AWSVPC_BLOCK_IMDS; IMDSv2 hop limit 1 kept, comment corrected;
  the test asserts no ECS_ENABLE_TASK_ENI / awsvpc settings.
- Reverted unrelated rustfmt churn in tofu_modules/tests.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:07:28 -08:00
shadowdaoandClaude Opus 5.5 9410c098ed feat(aws): optional per-user EC2 capacity in the onboarding module
Phase 7 Task 3 (E1-E4, E11; onboarding module version 2):
- ec2.tf: with ec2_instance_type set, the user's own cluster
  triple-c-<env>--u-<ec2_user_tag> with exactly one capacity provider
  (managed termination protection, managed scaling 100% one instance at
  a time, 60 s warm-up), an Auto Scaling group of 0..ec2_max_instances
  (desired left to ECS, protect_from_scale_in, force_delete off so the
  provider drains before deleting, fixed launch-template version, default
  tags + AmazonECSManaged), a launch template on the ECS-optimized AL2023
  AMI by architecture (IMDSv2 hop 1, 50 GiB gp3 encrypted root, the
  environment's tasks group, public IP as the environment's tasks get
  one, ECS_CLUSTER + ECS_AWSVPC_BLOCK_IMDS), the instance role
  triple-c-<env>-<owner key>-instance (bounded, tagged triple-c:cluster,
  ECS agent on its cluster only, ECR pull, logs; no SSM core) and its
  instance profile, and the user record /triple-c/users/<env>/<owner key>.
  Empty type (the default): count 0 everywhere, Fargate plan unchanged.
- Variables ec2_instance_type (the app's 30 types), ec2_max_instances
  (1-4), ec2_user_tag (as validate_ec2_user_tag, required with a type);
  onboarding_vars takes Option<&Ec2Compute> and validates the same.
- The environment record's assign_public_ip is now format-checked.
- tofu_modules.rs: ONBOARD_MODULE_VERSION 2, user_record_parameter_name,
  USER_RECORD_FIELDS, Ec2Compute, EC2_MAX_INSTANCES, ec2.tf embedded.
- Tests: EC2 calls merged into MODULE_CALLS under real types
  (all_module_calls removed); gating + version-1 resource fingerprint;
  launch template, group, capacity provider, instance role, AMI by arch,
  record fields, names, variable rules; the instance role inside the
  boundary with its cluster principal tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:00:23 -08:00
shadowdaoandClaude Opus 5.5 9829ff0c43 feat(aws): per-user EC2 clusters keyed by the triple-c-user principal tag
Fix round 2 for the phase 7 policies (amends plan E1):
- Per-user EC2 names: triple-c-<env>--u-<user tag> for the cluster,
  capacity provider, Auto Scaling group and launch template
  (user_ec2_stem, validate_ec2_user_tag; `--` can't occur in environment
  names or user tags, so the patterns are unambiguous); the instance
  role/profile stay owner-key named (user_instance_role_name, <= 64).
- workspace-user-tasks.json: RunTask/StartTask on a triple-c-*--u-*
  cluster is denied unless it is the caller's own
  (${aws:PrincipalTag/triple-c-user}), and denied without the tag.
- Admin: capacity providers and groups narrowed to triple-c-*--u-*; the
  boundary's agent cluster tag must match triple-c-*--u-*.
- Instance-type fail-open closed: ec2:InstanceType required (StringEquals,
  volumes in their own statement); group create requires
  autoscaling:InstanceTypes present.
- The compute file is split: environment-admin-scaling.json (groups,
  instance profiles, service-linked roles) is the admin tier's fourth file.
- Evaluator learns ArnNotLike. README: triple-c-user setup, sizes,
  still-open probes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 13:44:55 -08:00
shadowdaoandClaude Opus 5.5 e6fbf4e133 fix(aws): pin, type-limit and fence per-user EC2 capacity permissions
Review fix round 1 for the phase 7 admin compute file and boundary:
- Auto Scaling groups: create needs autoscaling:LaunchTemplateVersionSpecified
  (update BoolIfExists), no target groups / load balancers, supported
  instance types; CreateOrUpdateTags only on already-tagged groups.
  ModifyLaunchTemplate dropped (the default version never moves).
- RunInstances: instance and volume need the component request tag,
  the instance a supported type (the app's ec2_instance_spec table,
  pinned by a test); the security group must be the template's own and
  component-tagged, the image Amazon's and the template's.
- Boundary: the ECS agent only on the cluster its role's
  triple-c:cluster principal tag names (must be triple-c-*); Session
  Manager / ec2messages dropped (plan E4 amended).
- Evaluator learns Bool/BoolIfExists; the simulator sends the Boolean
  keys typed Boolean.
- README: instance types, pinning, -instance suffix is not an admin
  fence, RunInstances wording, still-open probes, IMDS/Docker note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 13:17:54 -08:00
shadowdaoandClaude Opus 5.5 7393143762 feat(aws): admin and boundary permissions for per-user EC2 capacity
Phase 7 Task 2 (E4, E5, E11):
- New third admin file environment-admin-compute.json (3,705 chars):
  capacity providers (triple-c-*), component-tagged launch templates,
  triple-c-* Auto Scaling groups (request tag to create, resource tag to
  change/protect/delete), RunInstances only from a tagged launch template
  with Amazon images, tags only at create, instance profiles (triple-c-*),
  PassRole of triple-c-*-instance roles to ec2.amazonaws.com only,
  service-linked roles for Auto Scaling and ECS only, the public
  ECS-optimized AMI parameter, autoscaling/capacity-provider reads.
- Boundary v2 (2,625 chars): the ECS agent's calls on triple-c-* clusters
  and container instances, TagResource only at RegisterContainerInstance,
  DiscoverPollEndpoint and Session Manager core. BOUNDARY_MODULE_VERSION=2
  with a template_version local and default tag, pinned by a test.
- Checks for every new action; tests for tag/name scoping, launch-only-
  through-templates, PassRole to EC2 only for instance roles, the
  workspace tier changing no compute, the boundary's agent fence.
- EC2_COMPUTE_CALLS in tofu_modules/tests.rs: the onboarding module's
  expected calls, evaluated against the admin tier and in the write drift
  test until Task 3 moves them into MODULE_CALLS.
- README: third file, sizes, boundary re-apply, resource table rows,
  scoping notes and "still open" points for the live run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 13:07:37 -08:00
shadowdaoandClaude Opus 5.5 c6b46a7347 fix(aws): refuse Docker/VPN on Fargate without grandfathering; launchTypePatch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:56:18 -08:00
shadowdaoandClaude Opus 5.5 86f253b7da feat(aws): EC2 launch type setting
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:54:24 -08:00
shadowdaoandClaude Opus 5.5 edef77feed docs(aws): phase 7 plan — ECS on EC2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:46:40 -08:00