Keep containers alive as root; notice containers that exit on their own #71

Merged

jknapp merged 2 commits from fix/container-keepalive into main 2026-11-03 00:41:14 +00:00

Conversation 0 Commits 2 Files Changed 10

+864 -13

jknapp commented 2026-11-02 23:57:54 +00:00

Owner

Copy link

Copy source

Problem

Project containers died "randomly": terminals showed disconnected, and the app kept showing the project as Running though the container had exited.

1. Fragile keepalive. The entrypoint ended in exec su -s /bin/bash claude -c "exec sleep infinity", so the container's lifetime was a sleep owned by the unprivileged claude user. Claude Code sessions run as claude and routinely run pkill sleep, pkill -f …, pkill -9 -x sleep, kill -9 -1 in tests. Any match killed the container (host journal: the scope just ended; no OOM, no docker stop).
2. Stale status. Nothing in the app noticed a container exiting by itself; status was only reconciled once at startup.

Fix

Entrypoint (container/entrypoint.sh)

The script now stays resident as root (docker-init stays PID 1) and waits on a root-owned sleep run as triple-c-keepalive infinity:

• Unprivileged kills get EPERM on both the script and the sleep.
• If the sleep is killed anyway (sudo pkill sleep), the loop respawns it.
• A TERM/INT/HUP trap makes docker stop return at once.

Why it dropped to claude: it came in with the initial commit (97a0745) with no comment or stated reason. Nothing depends on that process's owner. Env, cron, the scheduler .env and claude update are all set up earlier, as before, and are untouched. Its one reader was the marketplace readiness probe (marketplace/sync.rs READY_PROBE, which pgreps the exact su … sleep infinity command line). The probe now accepts either form: the new root triple-c-keepalive infinity or the old su line, which existing images will keep using until they migrate. A test pins the marker name against entrypoint.sh.

Stale Running status (container_watch.rs, new)

• There was no existing general watcher to reuse: the auth-bridge poll only runs for opted-in projects. So this adds one Docker event subscription for the app's lifetime, filtered daemon-side to type=container, event=die. An idle daemon costs nothing, and there's no polling loop.
• A die for a Running project's container is confirmed with one inspect. Only "not running" or a 404 counts; any other error means "don't know".
• The flip is then done by ProjectsStore::mark_exited, a compare-and-set under the store lock: only a project still Running on that container, and never one whose project lock is held. Stop, Start-with-recreate, Reset and migration hold that lock and write their own status.
• On each (re)subscription it re-checks Running projects once and asks for events since just before, to cover daemon restarts and reconnects. Retry backoff is 5→60 s, and it shuts down with the app's lifecycle cancellation.
• It emits project-container-exited with { project, reason }. The new useProjectStatusEvents hook (wired in App.tsx) applies it via updateProjectInList. Only reason: "exited" (died while the app was watching) shows an error toast; found_stopped (the first look of a run, e.g. after a host reboot) is silent.

Verification

Live, with a sandbox image built from this branch (triple-c-sandbox:keepalive-test, since removed) against a baseline container from the current ghcr.io/shadowdao/triple-c-sandbox:latest:

	current image	this branch
process tree	docker-init → su … claude -c exec sleep infinity → claude sleep	docker-init → root entrypoint.sh → root triple-c-keepalive infinity
su claude -c 'pkill -9 sleep; kill -9 -1'	container exited (137)	Operation not permitted, container keeps running
as claude: pkill -9 -x sleep, pkill -9 -f entrypoint	—	EPERM, keeps running
as claude: sudo pkill -9 sleep	—	sleep respawned (new pid), keeps running
docker stop	2624 ms, exit 143	550 ms, exit 0
stop → start → ready again	ok	ok
readiness probe (root)	—	exit 0

Tests:

• Rust: cargo test → 883 passed, 0 failed, 5 ignored (lib) + 3 (other targets). New tests: mark_exited ×4 (CAS, busy-lock, status/ID mismatch incl. blank stored ID, short-ID prefix), died_container parsing, daemon-side filter, updated readiness probe test. There is also a live #[ignore] test, a_container_killed_outside_the_app_flips_its_project_to_stopped: it docker kills one of two throwaway containers and asserts that only that project flips and is reported. It passes locally in 3.7 s (cargo test --lib -- --ignored a_container_killed_outside).
• cargo clippy --all-targets: no new warnings (only pre-existing ones remain).
• Frontend: npx vitest run → 1168 passed (91 files), incl. the new useProjectStatusEvents.test.ts; npx tsc --noEmit clean.

Review follow-up (2eabf69)

• Respawn throttle: about one respawn a second, using a fork-free pause (read -t on a FIFO, with sleep 1 as a fallback). Measured: 10748 respawns in 3 s unthrottled vs 3. Live, under a root kill storm every 50 ms: 5 keepalive PIDs in 4 s, and the container stays up.
• Backoff: resets only after a working connection (an event delivered, or up for 60 s). Ping-OK-but-events-fail now backs off to 60 s.
• Cancellation: every Docker call in the watcher races the app's cancellation.
• TOCTOU: the store counts writes. running_on returns a sighting at a revision; mark_exited returns Changed if anything was written since, and settle re-confirms. One shared matching rule with the 12-char guard, checked under the lock without cloning.
• Readiness probe: both alternatives now require -u root.
• Docs: TECHNICAL.md is updated.
• Re-verified live on a rebuilt image: unprivileged kills get EPERM and the container survives, sudo pkill sleep is respawned, docker stop takes 386 ms / exit 0, and docker stop during a respawn pause takes 475 ms / exit 0.
• Tests: cargo test 889 passed, plus the live ignored test (now also asserting the found_stopped vs exited reasons and that shutdown takes < 1 s). vitest 1170 passed; tsc and clippy are clean.

Do existing containers need anything?

• Status detection (app side): no. It works for every project as soon as the app is updated, whatever image the project is on.
• Keepalive (entrypoint): yes, it needs a new base image and a migration (or Reset). entrypoint.sh lives in the image, and a recreate builds from the project's own snapshot, which still has the old entrypoint. So existing projects keep the fragile keepalive until they're migrated to the new base (Project → Update/migrate) or Reset; new projects get it straight away. A plain stop/start or recreate is not enough. Until then the app at least flips such a container to Stopped when it dies, instead of showing Running.

Open questions

• sudo kill -9 -1 (or a root kill of entrypoint.sh) still stops the container. claude has passwordless sudo, so a full guarantee would need a keepalive the container's root can't reach, which isn't possible from inside it. This covers the unprivileged cases that were actually happening, plus sudo pkill sleep.
• kill -9 -1 as claude still kills all of claude's own processes (the Claude sessions included); only the container now survives it.
• Should the watcher also close the project's exec sessions or browser-view on a flip? The auth bridge already tears itself down, and terminal execs end on their own, so I left them alone.

🤖 Generated with Claude Code

## Problem Project containers died "randomly": terminals showed disconnected, and the app kept showing the project as **Running** though the container had exited. 1. **Fragile keepalive.** The entrypoint ended in `exec su -s /bin/bash claude -c "exec sleep infinity"`, so the container's lifetime was a `sleep` owned by the unprivileged `claude` user. Claude Code sessions run as `claude` and routinely run `pkill sleep`, `pkill -f …`, `pkill -9 -x sleep`, `kill -9 -1` in tests. Any match killed the container (host journal: the scope just ended; no OOM, no `docker stop`). 2. **Stale status.** Nothing in the app noticed a container exiting by itself; status was only reconciled once at startup. ## Fix ### Entrypoint (`container/entrypoint.sh`) The script now stays resident **as root** (docker-init stays PID 1) and waits on a root-owned sleep run as `triple-c-keepalive infinity`: - Unprivileged kills get `EPERM` on both the script and the sleep. - If the sleep is killed anyway (`sudo pkill sleep`), the loop respawns it. - A `TERM/INT/HUP` trap makes `docker stop` return at once. **Why it dropped to claude:** it came in with the initial commit (`97a0745`) with no comment or stated reason. Nothing depends on that process's owner. Env, cron, the scheduler `.env` and `claude update` are all set up earlier, as before, and are untouched. Its **one** reader was the marketplace readiness probe (`marketplace/sync.rs` `READY_PROBE`, which pgreps the exact `su … sleep infinity` command line). The probe now accepts either form: the new root `triple-c-keepalive infinity` or the old `su` line, which existing images will keep using until they migrate. A test pins the marker name against `entrypoint.sh`. ### Stale Running status (`container_watch.rs`, new) - There was no existing general watcher to reuse: the auth-bridge poll only runs for opted-in projects. So this adds **one Docker event subscription** for the app's lifetime, filtered daemon-side to `type=container, event=die`. An idle daemon costs nothing, and there's no polling loop. - A `die` for a Running project's container is confirmed with one inspect. Only "not running" or a 404 counts; any other error means "don't know". - The flip is then done by `ProjectsStore::mark_exited`, a compare-and-set under the store lock: only a project still `Running` on *that* container, and never one whose project lock is held. Stop, Start-with-recreate, Reset and migration hold that lock and write their own status. - On each (re)subscription it re-checks Running projects once and asks for events `since` just before, to cover daemon restarts and reconnects. Retry backoff is 5→60 s, and it shuts down with the app's lifecycle cancellation. - It emits `project-container-exited` with `{ project, reason }`. The new `useProjectStatusEvents` hook (wired in `App.tsx`) applies it via `updateProjectInList`. Only `reason: "exited"` (died while the app was watching) shows an error toast; `found_stopped` (the first look of a run, e.g. after a host reboot) is silent. ## Verification Live, with a sandbox image built from this branch (`triple-c-sandbox:keepalive-test`, since removed) against a baseline container from the current `ghcr.io/shadowdao/triple-c-sandbox:latest`: | | current image | this branch | |---|---|---| | process tree | `docker-init` → `su … claude -c exec sleep infinity` → `claude sleep` | `docker-init` → root `entrypoint.sh` → root `triple-c-keepalive infinity` | | `su claude -c 'pkill -9 sleep; kill -9 -1'` | **container exited (137)** | `Operation not permitted`, container keeps running | | as claude: `pkill -9 -x sleep`, `pkill -9 -f entrypoint` | — | EPERM, keeps running | | as claude: `sudo pkill -9 sleep` | — | sleep respawned (new pid), keeps running | | `docker stop` | 2624 ms, exit 143 | **550 ms, exit 0** | | stop → start → ready again | ok | ok | | readiness probe (root) | — | exit 0 | Tests: - Rust: `cargo test` → 883 passed, 0 failed, 5 ignored (lib) + 3 (other targets). New tests: `mark_exited` ×4 (CAS, busy-lock, status/ID mismatch incl. blank stored ID, short-ID prefix), `died_container` parsing, daemon-side filter, updated readiness probe test. There is also a live `#[ignore]` test, `a_container_killed_outside_the_app_flips_its_project_to_stopped`: it `docker kill`s one of two throwaway containers and asserts that only that project flips and is reported. It **passes** locally in 3.7 s (`cargo test --lib -- --ignored a_container_killed_outside`). - `cargo clippy --all-targets`: no new warnings (only pre-existing ones remain). - Frontend: `npx vitest run` → 1168 passed (91 files), incl. the new `useProjectStatusEvents.test.ts`; `npx tsc --noEmit` clean. ## Review follow-up (2eabf69) - **Respawn throttle:** about one respawn a second, using a fork-free pause (`read -t` on a FIFO, with `sleep 1` as a fallback). Measured: 10748 respawns in 3 s unthrottled vs 3. Live, under a root kill storm every 50 ms: 5 keepalive PIDs in 4 s, and the container stays up. - **Backoff:** resets only after a working connection (an event delivered, or up for 60 s). Ping-OK-but-events-fail now backs off to 60 s. - **Cancellation:** every Docker call in the watcher races the app's cancellation. - **TOCTOU:** the store counts writes. `running_on` returns a sighting at a revision; `mark_exited` returns `Changed` if anything was written since, and `settle` re-confirms. One shared matching rule with the 12-char guard, checked under the lock without cloning. - **Readiness probe:** both alternatives now require `-u root`. - **Docs:** TECHNICAL.md is updated. - **Re-verified live** on a rebuilt image: unprivileged kills get EPERM and the container survives, `sudo pkill sleep` is respawned, `docker stop` takes 386 ms / exit 0, and `docker stop` during a respawn pause takes 475 ms / exit 0. - **Tests:** `cargo test` 889 passed, plus the live ignored test (now also asserting the `found_stopped` vs `exited` reasons and that shutdown takes < 1 s). vitest 1170 passed; tsc and clippy are clean. ## Do existing containers need anything? - **Status detection (app side): no.** It works for every project as soon as the app is updated, whatever image the project is on. - **Keepalive (entrypoint): yes, it needs a new base image *and* a migration (or Reset).** `entrypoint.sh` lives in the image, and a recreate builds from the project's own snapshot, which still has the old entrypoint. So existing projects keep the fragile keepalive until they're **migrated to the new base** (Project → Update/migrate) or Reset; new projects get it straight away. A plain stop/start or recreate is not enough. Until then the app at least flips such a container to Stopped when it dies, instead of showing Running. ## Open questions - `sudo kill -9 -1` (or a root kill of `entrypoint.sh`) still stops the container. `claude` has passwordless sudo, so a full guarantee would need a keepalive the container's root can't reach, which isn't possible from inside it. This covers the unprivileged cases that were actually happening, plus `sudo pkill sleep`. - `kill -9 -1` as claude still kills all of claude's own processes (the Claude sessions included); only the container now survives it. - Should the watcher also close the project's exec sessions or browser-view on a flip? The auth bridge already tears itself down, and terminal execs end on their own, so I left them alone. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jknapp added 1 commit 2026-11-02 23:57:55 +00:00

Keep containers alive as root; notice containers that exit on their own ... eba0d95780

The entrypoint ended in `exec su -s /bin/bash claude -c "exec sleep
infinity"`, so a project container's whole lifetime was a sleep owned by
the unprivileged claude user. Claude Code runs as claude and routinely runs
`pkill sleep`, `pkill -f`, `kill -9 -1` while testing, and any of those
stopped the container: every terminal disconnected and the app kept
showing Running.

Entrypoint: stay resident as root and wait on a root-owned sleep run as
`triple-c-keepalive infinity`, respawned if killed, with a TERM/INT/HUP
trap so `docker stop` returns at once (550 ms, exit 0, vs 2.6 s / 143
before). Unprivileged kills now get EPERM; `sudo pkill sleep` is survived
by the respawn. The drop to claude dated from the initial commit with no
stated purpose; its only reader was the marketplace readiness probe, which
now accepts either generation of entrypoint.

App: container_watch.rs holds one Docker event subscription filtered
daemon-side to container `die`. A die naming a Running project's container
is confirmed with an inspect, then ProjectsStore::mark_exited flips it to
Stopped as a compare-and-set that skips any project a command holds the
project lock on. Each (re)subscription re-checks Running projects and asks
for events since just before it. The frontend applies the new
`project-status-changed` event and toasts why the sessions died.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jknapp added 1 commit 2026-11-03 00:15:37 +00:00

Address review of the keepalive / container-watch change ... 2eabf6972b

- Event renamed to `project-container-exited` and carries a reason:
  `exited` (died while the app was watching) toasts; `found_stopped` (the
  first look of a run, e.g. after a host reboot) updates silently.
  Reconnect sweeps report `exited`, since the app was running then.
- listen() in useProjectStatusEvents gets a .catch, like its sibling.
- Keepalive respawns are throttled to ~1/s with a fork-free pause
  (`read -t` on a FIFO, `sleep 1` fallback), so a failing fork or an
  instantly-dying sleep cannot spin a core. Measured: 10748 respawns in
  3 s unthrottled vs 3.
- Watcher backoff resets only after a working connection (an event
  delivered, or up for 60 s); ping-OK-but-events-fail now backs off to
  60 s instead of re-sweeping every 5 s.
- Every Docker call in the watcher (ping, sweep, inspect, stream) races
  the app's cancellation, so a wedged daemon cannot hold shutdown.
- TOCTOU between the inspect and the flip: ProjectsStore now counts
  writes; running_on() returns a sighting at a revision and mark_exited()
  refuses with Changed if anything was written since (a stop+start of the
  same container id included), and settle() re-confirms.
- The "Running on this container" rule lives once, in the store, with
  the 12-char guard on both ids, matched under the lock without cloning.
- Marketplace readiness fallback also requires a root-owned process.
- TECHNICAL.md no longer documents the `su … sleep infinity` ending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jknapp merged commit 4532412869 into main 2026-11-03 00:41:14 +00:00

jknapp deleted branch fix/container-keepalive 2026-11-03 00:41:25 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

jknapp

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: CyberCoveLLC/Triple-C#71