Fix review findings: secrets in snapshots, URL spoofing, migration data loss
Adversarial review of the branch produced findings across four areas. This addresses them, plus the Windows CI environment. Secrets. commit_container_snapshot baked the container's full env into the per-project snapshot image, so the shared OAuth token — and the AWS keys, git token and gateway master key — outlived revocation and were readable via docker inspect. Verified against Engine 29.6 that a commit body's config merges over the container's: keys cannot be dropped but can be overwritten, so all of them now commit as KEY=. clear_claude_token additionally rewrites images from earlier builds and reports honestly when a tag could not be rewritten. The recommendation to move the token out of env entirely was not taken, with reasoning: apiKeyHelper is a different auth method that outranks CLAUDE_CODE_OAUTH_TOKEN rather than a transport for it, and no file-based delivery exists. The durable exposure — the image — is what is closed here. Separately noted, not fixed: entrypoint.sh captures the token into the scheduler's .env inside the persisted volume. URL spoofing. Three call sites reached openUrl with container-controlled strings, one of which the review missed (the WebLinksAddon handler). The sign-in URL was scraped from container output with a longest-match tie-break and no userinfo check, so claude.ai@evil.tld rendered as "claude.ai…" in a truncating element. There is now one sanitizer in front of every sink — scheme allowlist, no userinfo, C0/C1 and quote rejection, host allowlist for the sign-in case, first-match — and the origin renders un-truncated. The toast is keyed so a changed URL remounts, closing a bait-and-switch where the user read one URL and clicked another. Migration. The rollback pin was best-effort: a tag failure was logged and the migration continued past remove_container, after which the final commit overwrote the only copy of the old system layer. It now aborts before anything destructive and reads the tag back. /var was destroyed while the ordinary recreate path preserves it — making the "safe" alternative to Reset more destructive than Reset's alternative; data-bearing subtrees are now detected and disclosed in the pre-flight rather than copied, since tarring a live database onto a different base's packages is a corruption risk. resume_migration now verifies the migration-state label instead of reporting success for a container that never swapped. dismiss actually resolves the record rather than leaving the feature permanently refusing to migrate. Start and Reset are guarded while a migration is live. Lifecycle. The gateway no longer publishes on 0.0.0.0 — bind address and advertised URL are derived together so they cannot drift. Disabling it now stops it. App exit runs teardown concurrently under a budget with a visible shutting-down state instead of blocking for minutes. Auto-starts retry when Docker is not up yet, and the polling-recovery path now reconciles, so interrupted migrations are still recovered. Auth-bridge forwards are capped, closing a container-driven fd exhaustion. Windows CI. build-windows failed on this branch with "linker link.exe not found". The runner had no MSVC build tools and the workflow assumed a hand-provisioned machine, so a bare runner registers, accepts jobs and fails at link time after downloading the whole crate graph. The job now installs the VC++ workload when vswhere cannot find it, matching how it already conditionally installs Rust and Node. 192 Rust tests, 274 frontend tests, both builds clean, zero warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -176,6 +176,39 @@ fn build_claude_instructions(
|
||||
/// stale-value neutralization pass can never disagree about the spelling.
|
||||
pub const CLAUDE_OAUTH_TOKEN_ENV: &str = "CLAUDE_CODE_OAUTH_TOKEN";
|
||||
|
||||
/// Every managed env var whose *value* is a credential.
|
||||
///
|
||||
/// These are the names that must never survive into a snapshot image. A
|
||||
/// container's env is visible to `docker inspect`, which is bad but bounded —
|
||||
/// the container is recreated whenever the credential rotates, and removed
|
||||
/// with the project. An **image**'s env is neither: `docker commit` copies the
|
||||
/// container's full environment into `triple-c-snapshot-{id}:latest`, that tag
|
||||
/// outlives every container built from it, and nothing about deleting a
|
||||
/// keychain entry touches it. A ~1-year OAuth token baked in that way is
|
||||
/// readable by `docker image inspect` for as long as the image exists, long
|
||||
/// after the user has clicked Revoke.
|
||||
///
|
||||
/// [`commit_container_snapshot`] therefore blanks all of them at commit time,
|
||||
/// and [`scrub_secrets_from_snapshots`] rewrites images committed before that
|
||||
/// was true.
|
||||
///
|
||||
/// Blanked rather than omitted, because Docker's commit endpoint *merges* the
|
||||
/// supplied config over the container's rather than replacing it: a key left
|
||||
/// out of the list is inherited with its original value, so `KEY=` is the only
|
||||
/// way to clear one. That matches how `MANAGED_AUTH_KEYS` already works at
|
||||
/// create time, and Claude Code, the AWS SDK and git all treat an empty value
|
||||
/// as unset.
|
||||
pub const SECRET_ENV_KEYS: &[&str] = &[
|
||||
CLAUDE_OAUTH_TOKEN_ENV,
|
||||
"ANTHROPIC_API_KEY",
|
||||
"ANTHROPIC_AUTH_TOKEN",
|
||||
"AWS_ACCESS_KEY_ID",
|
||||
"AWS_SECRET_ACCESS_KEY",
|
||||
"AWS_SESSION_TOKEN",
|
||||
"AWS_BEARER_TOKEN_BEDROCK",
|
||||
"GIT_TOKEN",
|
||||
];
|
||||
|
||||
/// Env var name prefixes Triple-C manages itself; users cannot set these by hand.
|
||||
const RESERVED_ENV_PREFIXES: &[&str] = &["ANTHROPIC_", "AWS_", "GIT_", "HOST_", "TRIPLE_C_"];
|
||||
|
||||
@@ -998,11 +1031,16 @@ pub async fn create_container(
|
||||
|
||||
// ── Neutralize stale backend auth env vars ──────────────────────────────
|
||||
// When a project switches backends (e.g. Bedrock → Anthropic) the container
|
||||
// is recreated *from a snapshot image* committed off the previous container.
|
||||
// `docker commit` always bakes the previous container's full ENV into that
|
||||
// image, and the commit API cannot strip it. So any auth var set under the
|
||||
// old backend (e.g. CLAUDE_CODE_USE_BEDROCK=1, AWS_*) survives in the image
|
||||
// ENV and stays active unless we explicitly override it at create time.
|
||||
// is recreated *from a snapshot image* committed off the previous container,
|
||||
// and `docker commit` copies that container's full ENV into the image. So
|
||||
// any auth var set under the old backend (e.g. CLAUDE_CODE_USE_BEDROCK=1,
|
||||
// AWS_PROFILE, a model alias) survives in the image ENV and stays active
|
||||
// unless we explicitly override it at create time.
|
||||
//
|
||||
// This pass is about *staleness*, not secrecy. It fixes the container it is
|
||||
// building and does nothing to the image, so it is not — and never was —
|
||||
// a defence against a credential baked into a snapshot. That is
|
||||
// `commit_container_snapshot`'s job, via SECRET_ENV_KEYS.
|
||||
// Create-time env takes precedence over image ENV, so we set every managed
|
||||
// auth key the *current* backend did NOT set to an empty value, clearing the
|
||||
// stale baked-in one.
|
||||
@@ -1514,14 +1552,29 @@ chmod 600 "$HOME/.aws/credentials""#;
|
||||
/// changes (apt/pip/npm installs, ~/.claude.json, etc.) survive container
|
||||
/// removal.
|
||||
///
|
||||
/// NOTE: `docker commit` always bakes the *running container's* full ENV into
|
||||
/// the resulting image — passing an empty Config here does NOT strip it, and
|
||||
/// the commit API gives no way to remove env vars. As a result auth vars (e.g.
|
||||
/// CLAUDE_CODE_USE_BEDROCK, AWS_*, CLAUDE_CODE_OAUTH_TOKEN) are present in this
|
||||
/// snapshot image's ENV — this image is local and per-project, never pushed.
|
||||
/// `create_container` defends against that by explicitly overriding every
|
||||
/// managed auth key for the active backend (see MANAGED_AUTH_KEYS), so a
|
||||
/// backend switch does not inherit the previous backend's stale credentials.
|
||||
/// ## Why this passes a Config instead of `Default::default()`
|
||||
///
|
||||
/// `docker commit` bakes the *running container's* full ENV into the resulting
|
||||
/// image. An earlier version of this function passed an empty `Config` and a
|
||||
/// comment asserting that the API "gives no way to remove env vars", with
|
||||
/// `MANAGED_AUTH_KEYS` cited as the defence. That was wrong on both counts.
|
||||
///
|
||||
/// `MANAGED_AUTH_KEYS` defends the *next container* — it overrides the image's
|
||||
/// stale value at create time — and does nothing whatsoever about the value
|
||||
/// sitting in the image. So `docker image inspect triple-c-snapshot-<id>:latest
|
||||
/// --format '{{json .Config.Env}}'` returned the shared ~1-year OAuth token,
|
||||
/// and kept returning it after the user revoked the token, because
|
||||
/// `clear_claude_token` only deletes a keychain entry.
|
||||
///
|
||||
/// And the API does allow it. Verified against Engine 29.6: the config in the
|
||||
/// commit body is **merged over** the container's config, key by key for `Env`,
|
||||
/// with unmentioned fields (`Cmd`, `WorkingDir`, `Labels`, …) inherited
|
||||
/// untouched. A key cannot be *dropped*, but it can be set — so every name in
|
||||
/// [`SECRET_ENV_KEYS`] is committed as `KEY=`, which is exactly the "empty
|
||||
/// means unset" convention the rest of the auth plumbing already uses.
|
||||
///
|
||||
/// Non-secret env (PATH, TZ, model aliases, instructions) is inherited as
|
||||
/// before, so nothing about the snapshot's behaviour changes.
|
||||
pub async fn commit_container_snapshot(container_id: &str, project: &Project) -> Result<(), String> {
|
||||
let docker = get_docker()?;
|
||||
let image_name = get_snapshot_image_name(project);
|
||||
@@ -1540,8 +1593,8 @@ pub async fn commit_container_snapshot(container_id: &str, project: &Project) ->
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
// Empty config — no env vars / cmd baked in
|
||||
let config = Config::<String> {
|
||||
env: Some(blanked_secret_env()),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
@@ -1554,6 +1607,258 @@ pub async fn commit_container_snapshot(container_id: &str, project: &Project) ->
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `KEY=` for every name in [`SECRET_ENV_KEYS`] — the env override handed to
|
||||
/// `docker commit` so no credential value reaches an image.
|
||||
fn blanked_secret_env() -> Vec<String> {
|
||||
SECRET_ENV_KEYS
|
||||
.iter()
|
||||
.map(|key| format!("{}=", key))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Whether `env` (an image's `Config.Env`) holds a non-empty value for any
|
||||
/// name in [`SECRET_ENV_KEYS`].
|
||||
fn env_holds_a_secret(env: &[String]) -> bool {
|
||||
env.iter().any(|entry| {
|
||||
let Some((key, value)) = entry.split_once('=') else {
|
||||
return false;
|
||||
};
|
||||
!value.is_empty() && SECRET_ENV_KEYS.contains(&key)
|
||||
})
|
||||
}
|
||||
|
||||
/// Outcome of [`scrub_secrets_from_snapshots`], so callers can tell the user
|
||||
/// what actually happened rather than guessing.
|
||||
#[derive(Debug, Default, Clone, serde::Serialize)]
|
||||
pub struct SnapshotScrubReport {
|
||||
/// Snapshot images that were found to hold a credential and were rewritten.
|
||||
pub scrubbed: Vec<String>,
|
||||
/// Snapshot images that hold a credential and could **not** be rewritten,
|
||||
/// each with the reason. A non-empty list means the tag every future
|
||||
/// container is built from still carries the credential.
|
||||
pub failed: Vec<(String, String)>,
|
||||
/// Tags that *were* rewritten, but whose superseded image object could not
|
||||
/// be deleted — almost always because a container is still running off it.
|
||||
///
|
||||
/// Much weaker than `failed`, and normal rather than exceptional. The tag
|
||||
/// is clean, so nothing new is built with the credential; what remains is
|
||||
/// an untagged image whose config is still readable by id, for exactly as
|
||||
/// long as the container using it survives — and that container already
|
||||
/// holds the same value in its own env, so it is not a new exposure. The
|
||||
/// rotation-id label mismatch recreates it on the next start, after which
|
||||
/// an image prune collects the leftover.
|
||||
pub superseded_retained: Vec<String>,
|
||||
/// Set when the image list itself could not be read (Docker not running,
|
||||
/// no permission). Nothing was scrubbed and nothing is known.
|
||||
pub unavailable: Option<String>,
|
||||
}
|
||||
|
||||
impl SnapshotScrubReport {
|
||||
/// True when a credential is known to still be reachable through something
|
||||
/// that will keep being used — a tag, or an unknown state.
|
||||
pub fn left_something_behind(&self) -> bool {
|
||||
!self.failed.is_empty() || self.unavailable.is_some()
|
||||
}
|
||||
}
|
||||
|
||||
/// Rewrite every `triple-c-snapshot-*` image whose ENV still carries a
|
||||
/// credential, blanking the values in place.
|
||||
///
|
||||
/// Revoking a shared token has to mean something. Deleting the keychain entry
|
||||
/// stops *new* containers getting it, but images committed before
|
||||
/// [`commit_container_snapshot`] learned to strip secrets still have the token
|
||||
/// in their config, and those images are the ones every future container of
|
||||
/// that project is built from. This is the cleanup for them.
|
||||
///
|
||||
/// Mechanics: create (do not start) a throwaway container from the image, then
|
||||
/// commit it straight back over the same tag with the secret keys blanked. The
|
||||
/// new image shares every layer with the old one, so this costs no meaningful
|
||||
/// disk and preserves the project's installed packages exactly. The superseded
|
||||
/// image is then removed by id; if Docker refuses (some storage drivers will
|
||||
/// not delete an image that is a parent of another), the report says so rather
|
||||
/// than pretending the secret is gone.
|
||||
///
|
||||
/// Never fails the caller: an unreachable Docker engine is reported in the
|
||||
/// return value, because the keychain deletion that precedes it must still
|
||||
/// stand.
|
||||
pub async fn scrub_secrets_from_snapshots() -> SnapshotScrubReport {
|
||||
use bollard::image::ListImagesOptions;
|
||||
|
||||
let mut report = SnapshotScrubReport::default();
|
||||
|
||||
let docker = match get_docker() {
|
||||
Ok(d) => d,
|
||||
Err(e) => {
|
||||
report.unavailable = Some(e);
|
||||
return report;
|
||||
}
|
||||
};
|
||||
|
||||
let filters: HashMap<String, Vec<String>> = HashMap::from([(
|
||||
"reference".to_string(),
|
||||
vec!["triple-c-snapshot-*".to_string()],
|
||||
)]);
|
||||
let images = match docker
|
||||
.list_images(Some(ListImagesOptions {
|
||||
filters,
|
||||
..Default::default()
|
||||
}))
|
||||
.await
|
||||
{
|
||||
Ok(images) => images,
|
||||
Err(e) => {
|
||||
report.unavailable = Some(format!("Could not list snapshot images: {}", e));
|
||||
return report;
|
||||
}
|
||||
};
|
||||
|
||||
for summary in images {
|
||||
// `list_images` does not return Config, so inspect each candidate.
|
||||
let details = match docker.inspect_image(&summary.id).await {
|
||||
Ok(d) => d,
|
||||
Err(e) => {
|
||||
report
|
||||
.failed
|
||||
.push((summary.id.clone(), format!("could not inspect: {}", e)));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let env = details
|
||||
.config
|
||||
.as_ref()
|
||||
.and_then(|c| c.env.clone())
|
||||
.unwrap_or_default();
|
||||
if !env_holds_a_secret(&env) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if summary.repo_tags.is_empty() {
|
||||
// The reference filter should make this impossible; if it happens,
|
||||
// say so rather than silently leaving a credential in place.
|
||||
report.failed.push((
|
||||
summary.id.clone(),
|
||||
"an untagged snapshot image holds a credential and cannot be rewritten"
|
||||
.to_string(),
|
||||
));
|
||||
continue;
|
||||
}
|
||||
|
||||
// Rewrite every tag this image answers to, so an old tag cannot keep
|
||||
// serving the un-scrubbed config.
|
||||
let mut all_tags_rewritten = true;
|
||||
for tag in summary.repo_tags.iter() {
|
||||
let (repo, tag_part) = match tag.rsplit_once(':') {
|
||||
Some((r, t)) => (r.to_string(), t.to_string()),
|
||||
None => (tag.clone(), "latest".to_string()),
|
||||
};
|
||||
if let Err(e) = rewrite_image_without_secrets(&docker, tag, &repo, &tag_part).await {
|
||||
all_tags_rewritten = false;
|
||||
report.failed.push((tag.clone(), e));
|
||||
} else {
|
||||
report.scrubbed.push(tag.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Drop the superseded image so its config stops being inspectable.
|
||||
// Best effort by design — see the doc comment.
|
||||
if all_tags_rewritten {
|
||||
if let Err(e) = docker
|
||||
.remove_image(
|
||||
&summary.id,
|
||||
Some(RemoveImageOptions {
|
||||
force: false,
|
||||
noprune: false,
|
||||
}),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
// Expected whenever the project's container is still around:
|
||||
// Docker will not delete an image a container was created
|
||||
// from. Not a failure of the scrub — see `superseded_retained`.
|
||||
log::info!(
|
||||
"Scrubbed snapshot {} but kept the superseded image {}: {}",
|
||||
summary.repo_tags.join(", "),
|
||||
summary.id,
|
||||
e
|
||||
);
|
||||
report
|
||||
.superseded_retained
|
||||
.push(summary.repo_tags.join(", "));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
report
|
||||
}
|
||||
|
||||
/// Commit `source_image` back over `repo:tag` with [`SECRET_ENV_KEYS`] blanked.
|
||||
async fn rewrite_image_without_secrets(
|
||||
docker: &bollard::Docker,
|
||||
source_image: &str,
|
||||
repo: &str,
|
||||
tag: &str,
|
||||
) -> Result<(), String> {
|
||||
let scratch_name = format!("triple-c-scrub-{}", uuid::Uuid::new_v4().simple());
|
||||
|
||||
let created = docker
|
||||
.create_container(
|
||||
Some(CreateContainerOptions {
|
||||
name: scratch_name.clone(),
|
||||
..Default::default()
|
||||
}),
|
||||
Config::<String> {
|
||||
image: Some(source_image.to_string()),
|
||||
// Deliberately nothing else. The container is never started;
|
||||
// its only job is to be a config to commit from, and every
|
||||
// field left unset here is inherited from the image and
|
||||
// inherited back out by the commit. Setting `cmd` to a
|
||||
// placeholder — the obvious way to satisfy an image with no
|
||||
// CMD — would write that placeholder into the rewritten
|
||||
// snapshot. The base image has an ENTRYPOINT, so `create`
|
||||
// needs no command; an image with neither fails here and is
|
||||
// reported rather than silently mangled.
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|e| format!("could not create a scratch container: {}", e))?;
|
||||
|
||||
let commit = docker
|
||||
.commit_container(
|
||||
CommitContainerOptions {
|
||||
container: created.id.clone(),
|
||||
repo: repo.to_string(),
|
||||
tag: tag.to_string(),
|
||||
// Nothing is running; pausing a created container is an error.
|
||||
pause: false,
|
||||
..Default::default()
|
||||
},
|
||||
Config::<String> {
|
||||
env: Some(blanked_secret_env()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|e| format!("could not re-commit without the credential: {}", e));
|
||||
|
||||
// Remove the scratch container whatever happened to the commit.
|
||||
if let Err(e) = docker
|
||||
.remove_container(
|
||||
&created.id,
|
||||
Some(RemoveContainerOptions {
|
||||
force: true,
|
||||
..Default::default()
|
||||
}),
|
||||
)
|
||||
.await
|
||||
{
|
||||
log::warn!("Could not remove scratch container {}: {}", scratch_name, e);
|
||||
}
|
||||
|
||||
commit.map(|_| ())
|
||||
}
|
||||
|
||||
/// Remove the snapshot image for a project (used on Reset / project removal).
|
||||
pub async fn remove_snapshot_image(project: &Project) -> Result<(), String> {
|
||||
let docker = get_docker()?;
|
||||
@@ -1806,8 +2111,10 @@ pub async fn container_needs_recreation(
|
||||
// re-acquired (rotated), revoked, or opted out of. Both "" means no token
|
||||
// is in play, which is also what a container predating this feature reports
|
||||
// — so existing installs are not recreated until a token actually exists.
|
||||
// Recreation is the only way to change container env, and it is also what
|
||||
// makes MANAGED_AUTH_KEYS blank a revoked token out of the snapshot image.
|
||||
// Recreation is the only way to change a container's env, so it is the only
|
||||
// way a revoked token stops being live in one. It does *not* clean the
|
||||
// snapshot image — `clear_claude_token` calls
|
||||
// `scrub_secrets_from_snapshots` for that.
|
||||
let expected_claude_token = claude_token_label(project);
|
||||
let container_claude_token = get_label("triple-c.claude-token-version").unwrap_or_default();
|
||||
if container_claude_token != expected_claude_token {
|
||||
@@ -2195,4 +2502,84 @@ mod tests {
|
||||
// The new per-backend haiku override also defaults cleanly.
|
||||
assert!(p.ollama_config.unwrap().haiku_model_id.is_none());
|
||||
}
|
||||
// ── Snapshot secret stripping ────────────────────────────────────────
|
||||
// The bug these cover: `docker commit` copies the container's whole
|
||||
// environment into `triple-c-snapshot-{id}:latest`, that image outlives
|
||||
// every container built from it, and revoking the shared token used to
|
||||
// touch only the keychain — so `docker image inspect` kept returning a
|
||||
// live ~1-year OAuth credential indefinitely.
|
||||
|
||||
#[test]
|
||||
fn the_commit_override_blanks_every_credential_bearing_key() {
|
||||
let blanked = blanked_secret_env();
|
||||
assert_eq!(blanked.len(), SECRET_ENV_KEYS.len());
|
||||
for key in SECRET_ENV_KEYS {
|
||||
assert!(
|
||||
blanked.contains(&format!("{}=", key)),
|
||||
"{} is not blanked at commit time",
|
||||
key
|
||||
);
|
||||
}
|
||||
// Blanked, never omitted: the commit endpoint merges this over the
|
||||
// container's env key by key, so a name left out is inherited with its
|
||||
// original value.
|
||||
assert!(blanked.iter().all(|e| e.ends_with('=')));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_shared_claude_token_is_one_of_the_stripped_keys() {
|
||||
assert!(SECRET_ENV_KEYS.contains(&CLAUDE_OAUTH_TOKEN_ENV));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_image_holding_a_credential_is_detected() {
|
||||
let env = vec![
|
||||
"PATH=/usr/bin".to_string(),
|
||||
format!("{}=sk-ant-oat01-{}", CLAUDE_OAUTH_TOKEN_ENV, "x".repeat(90)),
|
||||
];
|
||||
assert!(env_holds_a_secret(&env));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_blanked_or_secret_free_image_is_left_alone() {
|
||||
// Already scrubbed.
|
||||
assert!(!env_holds_a_secret(&blanked_secret_env()));
|
||||
// Never had one.
|
||||
assert!(!env_holds_a_secret(&[
|
||||
"PATH=/usr/bin".to_string(),
|
||||
"TZ=UTC".to_string(),
|
||||
format!("{}=claude-sonnet-4-5", ANTHROPIC_DEFAULT_SONNET_MODEL),
|
||||
]));
|
||||
// A non-secret var whose *name* merely contains a secret name.
|
||||
assert!(!env_holds_a_secret(&[
|
||||
format!("MY_{}=not-a-secret", CLAUDE_OAUTH_TOKEN_ENV)
|
||||
]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_value_containing_an_equals_sign_is_still_recognised() {
|
||||
let env = vec!["ANTHROPIC_AUTH_TOKEN=abc=def==".to_string()];
|
||||
assert!(env_holds_a_secret(&env));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_scrub_report_only_claims_success_when_nothing_is_left() {
|
||||
let clean = SnapshotScrubReport {
|
||||
scrubbed: vec!["triple-c-snapshot-a:latest".to_string()],
|
||||
..Default::default()
|
||||
};
|
||||
assert!(!clean.left_something_behind());
|
||||
|
||||
let partial = SnapshotScrubReport {
|
||||
failed: vec![("triple-c-snapshot-b:latest".to_string(), "nope".to_string())],
|
||||
..Default::default()
|
||||
};
|
||||
assert!(partial.left_something_behind());
|
||||
|
||||
let blind = SnapshotScrubReport {
|
||||
unavailable: Some("Docker is not running".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
assert!(blind.left_something_behind());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -432,11 +432,51 @@ pub async fn upload_bytes_to_container(
|
||||
Ok(format!("{}/{}", dest_dir.trim_end_matches('/'), file_name))
|
||||
}
|
||||
|
||||
/// Ceiling on how much container output a one-shot exec will buffer into the
|
||||
/// host process.
|
||||
///
|
||||
/// Every `exec_oneshot*` call reads the whole stream into a `String` before any
|
||||
/// caller sees a byte, and what it is reading is *container-controlled* — the
|
||||
/// scheduler notifications reader `cat`s up to 50 files with no size cap, and
|
||||
/// the auth bridge reads `/proc/net/tcp` every two seconds. Neither has an
|
||||
/// upstream bound, so this is where the bound goes. Generous enough that no
|
||||
/// legitimate reader (the largest is a package manifest of a full image) comes
|
||||
/// close.
|
||||
pub const MAX_ONESHOT_OUTPUT: usize = 8 * 1024 * 1024;
|
||||
|
||||
/// The auth bridge's per-tick budget. It reads two procfs files whose rows are
|
||||
/// ~150 bytes; a real container has tens of listeners, and the parser only ever
|
||||
/// yields at most one entry per port number. 1 MiB is thousands of rows — far
|
||||
/// past anything genuine, far short of a problem.
|
||||
pub const PROC_NET_OUTPUT_LIMIT: usize = 1024 * 1024;
|
||||
|
||||
/// Append to `buf` while it stays inside `limit`. Returns `false` once the
|
||||
/// limit is exceeded, at which point the caller must stop reading.
|
||||
fn push_capped(buf: &mut String, chunk: &str, limit: usize) -> bool {
|
||||
if buf.len() + chunk.len() > limit {
|
||||
return false;
|
||||
}
|
||||
buf.push_str(chunk);
|
||||
true
|
||||
}
|
||||
|
||||
/// Run a one-shot (non-interactive) exec command in a container and collect stdout.
|
||||
pub async fn exec_oneshot(container_id: &str, cmd: Vec<String>) -> Result<String, String> {
|
||||
exec_oneshot_env(container_id, cmd, Vec::new()).await
|
||||
}
|
||||
|
||||
/// [`exec_oneshot`] with a caller-chosen output ceiling, for readers whose
|
||||
/// input is fully container-controlled and whose legitimate output is small.
|
||||
pub async fn exec_oneshot_limited(
|
||||
container_id: &str,
|
||||
cmd: Vec<String>,
|
||||
limit: usize,
|
||||
) -> Result<String, String> {
|
||||
exec_oneshot_inner(container_id, "claude", cmd, Vec::new(), limit)
|
||||
.await
|
||||
.map(|(output, _)| output)
|
||||
}
|
||||
|
||||
/// Like `exec_oneshot`, but passes additional environment variables to the exec
|
||||
/// process. Secrets passed this way live only in `/proc/<pid>/environ` (readable
|
||||
/// by the same user / root) rather than in the process argv, so they are not
|
||||
@@ -477,6 +517,16 @@ pub async fn exec_oneshot_as(
|
||||
user: &str,
|
||||
cmd: Vec<String>,
|
||||
env: Vec<String>,
|
||||
) -> Result<(String, i64), String> {
|
||||
exec_oneshot_inner(container_id, user, cmd, env, MAX_ONESHOT_OUTPUT).await
|
||||
}
|
||||
|
||||
async fn exec_oneshot_inner(
|
||||
container_id: &str,
|
||||
user: &str,
|
||||
cmd: Vec<String>,
|
||||
env: Vec<String>,
|
||||
limit: usize,
|
||||
) -> Result<(String, i64), String> {
|
||||
let docker = get_docker()?;
|
||||
|
||||
@@ -505,7 +555,19 @@ pub async fn exec_oneshot_as(
|
||||
StartExecResults::Attached { mut output, .. } => {
|
||||
while let Some(msg) = output.next().await {
|
||||
match msg {
|
||||
Ok(data) => combined.push_str(&String::from_utf8_lossy(&data.into_bytes())),
|
||||
Ok(data) => {
|
||||
let chunk = String::from_utf8_lossy(&data.into_bytes()).into_owned();
|
||||
if !push_capped(&mut combined, &chunk, limit) {
|
||||
// Stop reading rather than truncate silently: every
|
||||
// caller parses this output, and a half-read
|
||||
// manifest or JSON array is worse than an error.
|
||||
// Dropping `output` kills the exec's stream.
|
||||
return Err(format!(
|
||||
"Command output exceeded {} bytes and was abandoned",
|
||||
limit
|
||||
));
|
||||
}
|
||||
}
|
||||
Err(e) => return Err(format!("Exec output error: {}", e)),
|
||||
}
|
||||
}
|
||||
@@ -540,3 +602,42 @@ pub async fn wait_for_exec_exit(exec_id: &str) -> Option<i64> {
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn output_under_the_limit_is_buffered_whole() {
|
||||
let mut buf = String::new();
|
||||
assert!(push_capped(&mut buf, "hello ", 16));
|
||||
assert!(push_capped(&mut buf, "world", 16));
|
||||
assert_eq!(buf, "hello world");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn output_over_the_limit_is_refused_rather_than_truncated() {
|
||||
// The abandoned chunk must not land in the buffer either: a caller that
|
||||
// ignored the error would otherwise parse a half-read document.
|
||||
let mut buf = String::new();
|
||||
assert!(push_capped(&mut buf, "0123456789", 12));
|
||||
assert!(!push_capped(&mut buf, "0123456789", 12));
|
||||
assert_eq!(buf, "0123456789");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_single_oversized_chunk_is_refused() {
|
||||
let mut buf = String::new();
|
||||
assert!(!push_capped(&mut buf, "0123456789", 4));
|
||||
assert!(buf.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_bridge_budget_is_far_smaller_than_the_general_one() {
|
||||
// The auth bridge re-reads container-controlled procfs every 2s, so it
|
||||
// gets a tighter ceiling than one-shot readers that run on demand.
|
||||
assert!(PROC_NET_OUTPUT_LIMIT < MAX_ONESHOT_OUTPUT);
|
||||
// …but still comfortably above a genuine /proc/net/tcp{,6} pair.
|
||||
assert!(PROC_NET_OUTPUT_LIMIT > 100 * 150);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,11 +7,14 @@
|
||||
//!
|
||||
//! Two things differ from STT, both deliberate:
|
||||
//!
|
||||
//! * **The port is published on `0.0.0.0`, not `127.0.0.1`.** STT is consumed
|
||||
//! by the Tauri host process, so loopback is enough. The gateway is consumed
|
||||
//! by *project containers*, which sit on Docker's default bridge and reach
|
||||
//! the host through the bridge gateway — a loopback-only bind is invisible to
|
||||
//! them. See [`gateway_base_url`].
|
||||
//! * **The published host address is *detected*, not fixed.** STT is consumed
|
||||
//! by the Tauri host process, so loopback is always enough. The gateway is
|
||||
//! consumed by *project containers*, and how a container reaches the host
|
||||
//! depends on the engine — so the bind address does too. See
|
||||
//! [`GatewayBinding`]. It is never `0.0.0.0`: the config behind this port
|
||||
//! holds a billed provider key, and Docker's published-port rules land in the
|
||||
//! `DOCKER` iptables chain *ahead* of a host firewall, so a wildcard bind is
|
||||
//! genuinely LAN-reachable even with `ufw` enabled.
|
||||
//! * **The rendered config is uploaded into the container over the Docker
|
||||
//! API** rather than passed as env. It holds the provider API key, and both
|
||||
//! env vars and labels are readable by anything on the host via
|
||||
@@ -23,10 +26,14 @@ use bollard::container::{
|
||||
};
|
||||
use bollard::image::BuildImageOptions;
|
||||
use bollard::models::{HostConfig, Mount, MountTypeEnum, PortBinding};
|
||||
use bollard::network::InspectNetworkOptions;
|
||||
use bollard::Docker;
|
||||
use futures_util::StreamExt;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::collections::HashMap;
|
||||
use std::io::Write;
|
||||
use std::sync::OnceLock;
|
||||
use tokio::sync::{Mutex, OnceCell};
|
||||
|
||||
use super::client::get_docker;
|
||||
use crate::models::gateway_settings::{GatewaySettings, GatewayStatus};
|
||||
@@ -58,21 +65,139 @@ const GATEWAY_INTERNAL_PORT: u16 = 4000;
|
||||
|
||||
const CONFIG_FINGERPRINT_LABEL: &str = "triple-c.gateway.config-fingerprint";
|
||||
|
||||
/// The value a project should use as its base URL (`ANTHROPIC_BASE_URL`).
|
||||
/// The default bridge gateway address on a stock native-Linux engine. Only a
|
||||
/// fallback: the real value is read from the `bridge` network's IPAM config.
|
||||
const DEFAULT_BRIDGE_GATEWAY: &str = "172.17.0.1";
|
||||
|
||||
/// Where the gateway's published port is bound on the host, and the address a
|
||||
/// *project container* uses to reach it.
|
||||
///
|
||||
/// Project containers run on Docker's default bridge with no user-defined
|
||||
/// network and no `--add-host`, so the only address they share with the
|
||||
/// gateway is the host itself. Publishing the gateway on `0.0.0.0:<port>`
|
||||
/// makes it reachable from every container network on the machine:
|
||||
/// network and no `--add-host`, so the only address they share with the gateway
|
||||
/// is the host itself — but *which* host address works is engine-specific, and
|
||||
/// the whole point of this type is that the two answers are derived together so
|
||||
/// they cannot drift apart:
|
||||
///
|
||||
/// * Docker Desktop (macOS / Windows / WSL2) resolves `host.docker.internal`
|
||||
/// from inside containers automatically — that is the portable value and the
|
||||
/// one already suggested by the existing OpenAI-compatible placeholder text.
|
||||
/// * On native Linux Docker `host.docker.internal` is not injected, and the
|
||||
/// equivalent address is the default bridge gateway, normally
|
||||
/// `http://172.17.0.1:<port>`.
|
||||
pub fn gateway_base_url(port: u16) -> String {
|
||||
format!("http://host.docker.internal:{}", port)
|
||||
/// * **Docker Desktop** (macOS / Windows / WSL2) resolves `host.docker.internal`
|
||||
/// from inside containers automatically, and its port forwarder reaches the
|
||||
/// host's *loopback*. So: bind `127.0.0.1`, hand out `host.docker.internal`.
|
||||
/// * **Native Linux Docker** injects no `host.docker.internal`, and the address
|
||||
/// containers share with the host is the default bridge gateway (normally
|
||||
/// `172.17.0.1`). So: bind that address, and hand out the same literal.
|
||||
///
|
||||
/// Neither case binds `0.0.0.0`. The bridge-gateway bind is reachable from
|
||||
/// every container on the default bridge — which is the requirement — without
|
||||
/// publishing a key-bearing proxy to the LAN.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct GatewayBinding {
|
||||
/// Host address the published port is bound to (`HostIp`).
|
||||
pub host_ip: String,
|
||||
/// Host address a project container should dial.
|
||||
pub container_host: String,
|
||||
}
|
||||
|
||||
impl GatewayBinding {
|
||||
fn desktop() -> Self {
|
||||
Self {
|
||||
host_ip: "127.0.0.1".to_string(),
|
||||
container_host: "host.docker.internal".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn bridge(gateway_ip: &str) -> Self {
|
||||
Self {
|
||||
host_ip: gateway_ip.to_string(),
|
||||
container_host: gateway_ip.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The value a project should use as its base URL (`ANTHROPIC_BASE_URL`).
|
||||
pub fn base_url(&self, port: u16) -> String {
|
||||
format!("http://{}:{}", self.container_host, port)
|
||||
}
|
||||
|
||||
/// The address the *host* process (health checks) should dial.
|
||||
fn host_url(&self, port: u16) -> String {
|
||||
format!("http://{}:{}", self.host_ip, port)
|
||||
}
|
||||
}
|
||||
|
||||
/// Decide the binding from what the daemon reports. Pure, so the engine-shape
|
||||
/// matrix is testable without a daemon.
|
||||
fn binding_for(operating_system: &str, bridge_gateway: Option<&str>) -> GatewayBinding {
|
||||
// Docker Desktop reports exactly "Docker Desktop" here on every platform it
|
||||
// ships for; matched loosely so a future suffix doesn't silently flip us
|
||||
// onto the bridge path.
|
||||
if operating_system.to_ascii_lowercase().contains("docker desktop") {
|
||||
return GatewayBinding::desktop();
|
||||
}
|
||||
GatewayBinding::bridge(
|
||||
bridge_gateway
|
||||
.map(str::trim)
|
||||
.filter(|g| !g.is_empty())
|
||||
.unwrap_or(DEFAULT_BRIDGE_GATEWAY),
|
||||
)
|
||||
}
|
||||
|
||||
/// Detection is one `info` + one `inspect_network` per process; the answer
|
||||
/// cannot change without the engine being replaced under us.
|
||||
static GATEWAY_BINDING: OnceCell<GatewayBinding> = OnceCell::const_new();
|
||||
|
||||
/// The gateway's host binding, detected once and cached.
|
||||
///
|
||||
/// When Docker is unreachable the *loopback* answer is returned without being
|
||||
/// cached: it is the conservative one (nothing is published anywhere yet, and
|
||||
/// the only caller in that state is status reporting), and the next call
|
||||
/// re-detects once the daemon is up.
|
||||
pub async fn gateway_binding() -> GatewayBinding {
|
||||
if let Some(binding) = GATEWAY_BINDING.get() {
|
||||
return binding.clone();
|
||||
}
|
||||
match detect_binding().await {
|
||||
Ok(binding) => {
|
||||
let _ = GATEWAY_BINDING.set(binding.clone());
|
||||
binding
|
||||
}
|
||||
Err(e) => {
|
||||
log::debug!("Gateway bind detection deferred ({}), assuming loopback", e);
|
||||
GatewayBinding::desktop()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn detect_binding() -> Result<GatewayBinding, String> {
|
||||
let docker = get_docker()?;
|
||||
let info = docker
|
||||
.info()
|
||||
.await
|
||||
.map_err(|e| format!("Failed to query the Docker daemon: {}", e))?;
|
||||
let operating_system = info.operating_system.unwrap_or_default();
|
||||
let gateway_ip = bridge_gateway_ip(&docker).await;
|
||||
let binding = binding_for(&operating_system, gateway_ip.as_deref());
|
||||
log::info!(
|
||||
"Model gateway will publish on {} (engine OS: {})",
|
||||
binding.host_ip,
|
||||
if operating_system.is_empty() {
|
||||
"unknown"
|
||||
} else {
|
||||
&operating_system
|
||||
}
|
||||
);
|
||||
Ok(binding)
|
||||
}
|
||||
|
||||
/// The default bridge's gateway address, straight from its IPAM config, so a
|
||||
/// host whose bridge subnet was customised still gets a reachable bind.
|
||||
async fn bridge_gateway_ip(docker: &Docker) -> Option<String> {
|
||||
let network = docker
|
||||
.inspect_network("bridge", None::<InspectNetworkOptions<String>>)
|
||||
.await
|
||||
.ok()?;
|
||||
network
|
||||
.ipam?
|
||||
.config?
|
||||
.into_iter()
|
||||
.find_map(|c| c.gateway.filter(|g| !g.trim().is_empty()))
|
||||
}
|
||||
|
||||
fn sha256_hex(input: &str) -> String {
|
||||
@@ -101,10 +226,31 @@ pub async fn get_gateway_status(settings: &GatewaySettings) -> Result<GatewaySta
|
||||
image_exists,
|
||||
model_count: settings.valid_models().len(),
|
||||
has_api_key: secure::has_gateway_api_key(),
|
||||
base_url: gateway_base_url(settings.port),
|
||||
base_url: gateway_binding().await.base_url(settings.port),
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether a gateway container exists, and whether it is running. Used by the
|
||||
/// settings reconcile, which must not start anything the user never started.
|
||||
pub async fn gateway_container_presence() -> Result<(bool, bool), String> {
|
||||
Ok(match find_gateway_container().await? {
|
||||
Some((_, state, _)) => (true, state == "running"),
|
||||
None => (false, false),
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether a container summary's names contain *exactly* our container.
|
||||
///
|
||||
/// Docker's `name` filter is an unanchored regex, so listing with it also
|
||||
/// returns `triple-c-gateway-backup`, `my-triple-c-gateway`, and anything else
|
||||
/// containing the string. Taking `.first()` of that would let this module
|
||||
/// adopt — and then force-remove — a container it does not own.
|
||||
/// `container::find_existing_container` matches exactly for the same reason.
|
||||
fn is_gateway_container(names: Option<&Vec<String>>) -> bool {
|
||||
let expected = format!("/{}", GATEWAY_CONTAINER_NAME);
|
||||
names.is_some_and(|names| names.iter().any(|n| n == &expected))
|
||||
}
|
||||
|
||||
/// `(id, state, config fingerprint label)` for the gateway container, if any.
|
||||
async fn find_gateway_container() -> Result<Option<(String, String, String)>, String> {
|
||||
let docker = get_docker()?;
|
||||
@@ -123,7 +269,11 @@ async fn find_gateway_container() -> Result<Option<(String, String, String)>, St
|
||||
.await
|
||||
.map_err(|e| format!("Failed to list containers: {}", e))?;
|
||||
|
||||
if let Some(container) = containers.first() {
|
||||
// The filter is a prefilter only — the exact-name check is what decides.
|
||||
for container in &containers {
|
||||
if !is_gateway_container(container.names.as_ref()) {
|
||||
continue;
|
||||
}
|
||||
let id = container.id.clone().unwrap_or_default();
|
||||
let state = container.state.clone().unwrap_or_default();
|
||||
let fingerprint = container
|
||||
@@ -169,17 +319,21 @@ fn yaml_str(value: &str) -> String {
|
||||
/// The parts of the config that are safe to hash into a Docker label — i.e.
|
||||
/// everything except the two secrets, whose changes are tracked by the
|
||||
/// keychain rotation id instead.
|
||||
fn config_shape(settings: &GatewaySettings) -> String {
|
||||
fn config_shape(settings: &GatewaySettings, binding: &GatewayBinding) -> String {
|
||||
let models: Vec<String> = settings
|
||||
.valid_models()
|
||||
.iter()
|
||||
.map(|m| format!("{}={}", m.name.trim(), m.model_id.trim()))
|
||||
.collect();
|
||||
// `bind` is part of the shape so that moving between engines (or a bridge
|
||||
// subnet change) recreates the container instead of leaving it published on
|
||||
// an address the new environment doesn't use.
|
||||
format!(
|
||||
"provider={};api_base={};port={};models={}",
|
||||
"provider={};api_base={};port={};bind={};models={}",
|
||||
settings.provider.trim(),
|
||||
settings.api_base.as_deref().unwrap_or("").trim(),
|
||||
settings.port,
|
||||
binding.host_ip,
|
||||
models.join(",")
|
||||
)
|
||||
}
|
||||
@@ -273,6 +427,7 @@ async fn upload_config(container_id: &str, config: &str) -> Result<(), String> {
|
||||
|
||||
async fn create_gateway_container(
|
||||
settings: &GatewaySettings,
|
||||
binding: &GatewayBinding,
|
||||
fingerprint: &str,
|
||||
) -> Result<String, String> {
|
||||
let docker = get_docker()?;
|
||||
@@ -299,9 +454,9 @@ async fn create_gateway_container(
|
||||
port_bindings.insert(
|
||||
format!("{}/tcp", GATEWAY_INTERNAL_PORT),
|
||||
Some(vec![PortBinding {
|
||||
// Not loopback — project containers reach this through the host.
|
||||
// See `gateway_base_url`.
|
||||
host_ip: Some("0.0.0.0".to_string()),
|
||||
// Never `0.0.0.0`: the narrowest host address project containers
|
||||
// can still reach. See `GatewayBinding`.
|
||||
host_ip: Some(binding.host_ip.clone()),
|
||||
host_port: Some(settings.port.to_string()),
|
||||
}]),
|
||||
);
|
||||
@@ -328,6 +483,7 @@ async fn create_gateway_container(
|
||||
"triple-c.gateway.port".to_string(),
|
||||
settings.port.to_string(),
|
||||
);
|
||||
labels.insert("triple-c.gateway.bind".to_string(), binding.host_ip.clone());
|
||||
labels.insert(
|
||||
"triple-c.gateway.provider".to_string(),
|
||||
settings.provider.trim().to_string(),
|
||||
@@ -365,7 +521,29 @@ async fn create_gateway_container(
|
||||
Ok(response.id)
|
||||
}
|
||||
|
||||
/// Serialises every mutation of the single fixed-name gateway container.
|
||||
///
|
||||
/// `ensure_gateway_running` is check-then-act over one container name, so two
|
||||
/// concurrent callers — the setup auto-start and the user's Start button is the
|
||||
/// realistic pair — would both see `None` and both try to create it, and the
|
||||
/// loser would surface a raw Docker 409. Migration guards the same shape with
|
||||
/// `ActiveGuard`; here the right behaviour is to *serialise* rather than
|
||||
/// refuse, because the second caller then observes the first's container, finds
|
||||
/// a matching fingerprint, and returns its status — which is exactly what it
|
||||
/// asked for.
|
||||
fn gateway_lock() -> &'static Mutex<()> {
|
||||
static LOCK: OnceLock<Mutex<()>> = OnceLock::new();
|
||||
LOCK.get_or_init(|| Mutex::new(()))
|
||||
}
|
||||
|
||||
pub async fn ensure_gateway_running(settings: &GatewaySettings) -> Result<GatewayStatus, String> {
|
||||
let _guard = gateway_lock().lock().await;
|
||||
ensure_gateway_running_locked(settings).await
|
||||
}
|
||||
|
||||
async fn ensure_gateway_running_locked(
|
||||
settings: &GatewaySettings,
|
||||
) -> Result<GatewayStatus, String> {
|
||||
let docker = get_docker()?;
|
||||
|
||||
if settings.valid_models().is_empty() {
|
||||
@@ -382,11 +560,13 @@ pub async fn ensure_gateway_running(settings: &GatewaySettings) -> Result<Gatewa
|
||||
})?;
|
||||
let master_key = secure::get_or_create_gateway_master_key()?;
|
||||
|
||||
let binding = gateway_binding().await;
|
||||
|
||||
// Rotation id, not a hash of either secret — see `storage::secure`.
|
||||
let secret_version = secure::get_gateway_secret_version()?.unwrap_or_default();
|
||||
let fingerprint = sha256_hex(&format!(
|
||||
"{}|{}",
|
||||
config_shape(settings),
|
||||
config_shape(settings, &binding),
|
||||
secret_version
|
||||
));
|
||||
|
||||
@@ -421,7 +601,7 @@ pub async fn ensure_gateway_running(settings: &GatewaySettings) -> Result<Gatewa
|
||||
.map_err(|e| format!("Failed to remove gateway container: {}", e))?;
|
||||
}
|
||||
|
||||
let id = create_gateway_container(settings, &fingerprint).await?;
|
||||
let id = create_gateway_container(settings, &binding, &fingerprint).await?;
|
||||
|
||||
// Upload before the first start: LiteLLM reads the config once at boot.
|
||||
let rendered = render_config(settings, &api_key, &master_key);
|
||||
@@ -446,7 +626,8 @@ pub async fn ensure_gateway_running(settings: &GatewaySettings) -> Result<Gatewa
|
||||
.map_err(|e| format!("Failed to start gateway container: {}", e))?;
|
||||
|
||||
log::info!(
|
||||
"Model gateway started on port {} ({} model(s))",
|
||||
"Model gateway started on {}:{} ({} model(s))",
|
||||
binding.host_ip,
|
||||
settings.port,
|
||||
settings.valid_models().len()
|
||||
);
|
||||
@@ -454,13 +635,25 @@ pub async fn ensure_gateway_running(settings: &GatewaySettings) -> Result<Gatewa
|
||||
get_gateway_status(settings).await
|
||||
}
|
||||
|
||||
/// Grace period given to LiteLLM on stop. The Docker default is 10s, which app
|
||||
/// exit cannot afford to spend on a proxy that holds no state worth flushing.
|
||||
const GATEWAY_STOP_GRACE_SECS: i64 = 3;
|
||||
|
||||
pub async fn stop_gateway_container() -> Result<(), String> {
|
||||
// Same lock as `ensure_gateway_running`, so a stop can't interleave with a
|
||||
// create/start and leave a container running behind a "stopped" return.
|
||||
let _guard = gateway_lock().lock().await;
|
||||
let docker = get_docker()?;
|
||||
|
||||
if let Some((id, state, _)) = find_gateway_container().await? {
|
||||
if state == "running" {
|
||||
docker
|
||||
.stop_container(&id, None::<StopContainerOptions>)
|
||||
.stop_container(
|
||||
&id,
|
||||
Some(StopContainerOptions {
|
||||
t: GATEWAY_STOP_GRACE_SECS,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to stop gateway container: {}", e))?;
|
||||
}
|
||||
@@ -477,8 +670,12 @@ pub async fn check_gateway_health(port: u16) -> Result<bool, String> {
|
||||
.build()
|
||||
.map_err(|e| format!("Failed to build HTTP client: {}", e))?;
|
||||
|
||||
// Dial whatever the container is actually published on — with a
|
||||
// bridge-gateway bind, the host's loopback answers nothing.
|
||||
let base = gateway_binding().await.host_url(port);
|
||||
|
||||
match client
|
||||
.get(format!("http://127.0.0.1:{}/health/liveliness", port))
|
||||
.get(format!("{}/health/liveliness", base))
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
@@ -627,18 +824,126 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn config_shape_excludes_secrets_and_tracks_changes() {
|
||||
let a = config_shape(&settings());
|
||||
let binding = GatewayBinding::desktop();
|
||||
let a = config_shape(&settings(), &binding);
|
||||
let mut s = settings();
|
||||
s.models[0].model_id = "gpt-4.1".to_string();
|
||||
assert_ne!(a, config_shape(&s));
|
||||
assert_ne!(a, config_shape(&s, &binding));
|
||||
assert!(!a.contains("sk-"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn base_url_points_at_the_host_not_loopback() {
|
||||
// A project container cannot reach the host's loopback interface.
|
||||
let url = gateway_base_url(4000);
|
||||
assert_eq!(url, "http://host.docker.internal:4000");
|
||||
assert!(!url.contains("127.0.0.1"));
|
||||
fn config_shape_tracks_the_bind_address() {
|
||||
// Moving between engines must recreate the container rather than leave
|
||||
// it published on an address the new environment doesn't use.
|
||||
let s = settings();
|
||||
assert_ne!(
|
||||
config_shape(&s, &GatewayBinding::desktop()),
|
||||
config_shape(&s, &GatewayBinding::bridge("172.17.0.1"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn docker_desktop_binds_loopback_and_hands_out_host_docker_internal() {
|
||||
let binding = binding_for("Docker Desktop", None);
|
||||
assert_eq!(binding.host_ip, "127.0.0.1");
|
||||
assert_eq!(binding.base_url(4000), "http://host.docker.internal:4000");
|
||||
// Detection must not depend on the bridge answer on this engine.
|
||||
assert_eq!(binding, binding_for("Docker Desktop", Some("172.17.0.1")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_linux_binds_the_bridge_gateway_it_reports() {
|
||||
// A project container can't reach the host's loopback here, but it can
|
||||
// reach the bridge gateway — and so can nothing on the LAN.
|
||||
let binding = binding_for("Ubuntu 24.04.1 LTS", Some("172.19.0.1"));
|
||||
assert_eq!(binding.host_ip, "172.19.0.1");
|
||||
assert_eq!(binding.base_url(4000), "http://172.19.0.1:4000");
|
||||
assert_eq!(binding.host_url(4000), "http://172.19.0.1:4000");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_bridge_answer_falls_back_to_the_documented_default() {
|
||||
for reported in [None, Some(""), Some(" ")] {
|
||||
assert_eq!(
|
||||
binding_for("Ubuntu 24.04.1 LTS", reported).host_ip,
|
||||
"172.17.0.1"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_engine_shape_ever_binds_a_wildcard_address() {
|
||||
// The regression this guards: the published port fronts a container
|
||||
// config holding a billed provider key, and Docker's rules sit ahead of
|
||||
// the host firewall.
|
||||
for os in ["Docker Desktop", "Ubuntu 24.04.1 LTS", "", "Rancher Desktop"] {
|
||||
for gw in [None, Some("172.17.0.1"), Some("10.0.0.1")] {
|
||||
let host_ip = binding_for(os, gw).host_ip;
|
||||
assert_ne!(host_ip, "0.0.0.0", "os={:?} gw={:?}", os, gw);
|
||||
assert_ne!(host_ip, "::", "os={:?} gw={:?}", os, gw);
|
||||
assert!(!host_ip.is_empty());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_the_exact_container_name_is_adopted() {
|
||||
// Docker's `name` filter is an unanchored regex: all of these come back
|
||||
// from a filtered list. Adopting one would force-remove a user's
|
||||
// container.
|
||||
assert!(is_gateway_container(Some(&vec![
|
||||
"/triple-c-gateway".to_string()
|
||||
])));
|
||||
assert!(is_gateway_container(Some(&vec![
|
||||
"/something-else".to_string(),
|
||||
"/triple-c-gateway".to_string(),
|
||||
])));
|
||||
for impostor in [
|
||||
"/triple-c-gateway-backup",
|
||||
"/my-triple-c-gateway",
|
||||
"/triple-c-gateway2",
|
||||
"triple-c-gateway",
|
||||
] {
|
||||
assert!(
|
||||
!is_gateway_container(Some(&vec![impostor.to_string()])),
|
||||
"{} must not be adopted",
|
||||
impostor
|
||||
);
|
||||
}
|
||||
assert!(!is_gateway_container(None));
|
||||
assert!(!is_gateway_container(Some(&vec![])));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_gateway_lock_serialises_concurrent_callers() {
|
||||
// The auto-start racing the Start button: both would otherwise see no
|
||||
// container and both create one, and the loser gets a Docker 409.
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
let inside = Arc::new(AtomicUsize::new(0));
|
||||
let overlaps = Arc::new(AtomicUsize::new(0));
|
||||
|
||||
let mut tasks = Vec::new();
|
||||
for _ in 0..8 {
|
||||
let inside = inside.clone();
|
||||
let overlaps = overlaps.clone();
|
||||
tasks.push(tokio::spawn(async move {
|
||||
let _guard = gateway_lock().lock().await;
|
||||
if inside.fetch_add(1, Ordering::SeqCst) != 0 {
|
||||
overlaps.fetch_add(1, Ordering::SeqCst);
|
||||
}
|
||||
tokio::task::yield_now().await;
|
||||
tokio::time::sleep(std::time::Duration::from_millis(1)).await;
|
||||
inside.fetch_sub(1, Ordering::SeqCst);
|
||||
}));
|
||||
}
|
||||
for t in tasks {
|
||||
t.await.unwrap();
|
||||
}
|
||||
|
||||
assert_eq!(overlaps.load(Ordering::SeqCst), 0);
|
||||
assert_eq!(inside.load(Ordering::SeqCst), 0);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -45,7 +45,7 @@ use bollard::models::HostConfig;
|
||||
use futures_util::StreamExt;
|
||||
|
||||
use super::client::get_docker;
|
||||
use crate::models::ProjectPath;
|
||||
use crate::models::{ProjectPath, UnpreservedData};
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Policy constants
|
||||
@@ -81,7 +81,35 @@ pub const COPY_EXCLUSIONS: &[&str] = &["/usr/local/aws-cli", "/opt/mission-contr
|
||||
/// Roots the filesystem manifest walks. Wider than [`COPY_ROOTS`] so the
|
||||
/// manifest stays useful for debugging; [`compute_verbatim_paths`] applies the
|
||||
/// narrower policy.
|
||||
pub const MANIFEST_ROOTS: &[&str] = &["/usr/local", "/opt", "/srv", "/workspace"];
|
||||
///
|
||||
/// [`DATA_ROOTS`] are in here for a different reason: they are never copied,
|
||||
/// but they *are* destroyed by the container swap, so the walk has to see them
|
||||
/// in order to warn about them.
|
||||
pub const MANIFEST_ROOTS: &[&str] = &[
|
||||
"/usr/local",
|
||||
"/opt",
|
||||
"/srv",
|
||||
"/workspace",
|
||||
"/var/lib",
|
||||
"/var/www",
|
||||
];
|
||||
|
||||
/// Roots holding **state a base-image swap destroys and no replay can put
|
||||
/// back**. Reported by [`unpreserved_data`], never copied.
|
||||
///
|
||||
/// A container running Postgres, MySQL, Redis or nginx keeps its actual data in
|
||||
/// `/var/lib/<service>` or `/var/www`. Replaying the apt delta reinstalls the
|
||||
/// *package* onto the new base and gets an empty data directory back — the
|
||||
/// database is gone. That is worse than the ordinary recreate path, which
|
||||
/// creates from the project's snapshot and therefore keeps `/var` intact.
|
||||
///
|
||||
/// These are deliberately **not** in [`COPY_ROOTS`]. A live database's on-disk
|
||||
/// files cannot be tarred out from under a running server and restored into a
|
||||
/// different base's version of the same package with any confidence — a copy
|
||||
/// that half-works is worse than a warning that lets the user take a proper
|
||||
/// dump first. So migration's answer is disclosure, loudly, before anything is
|
||||
/// touched.
|
||||
pub const DATA_ROOTS: &[&str] = &["/var/lib", "/var/www"];
|
||||
|
||||
/// Base-image capabilities worth telling the user they are missing, as
|
||||
/// `(path, human label)`.
|
||||
@@ -125,6 +153,16 @@ pub const LABEL_CREATE_IMAGE: &str = "triple-c.create-image";
|
||||
pub const LABEL_MIGRATION_STATE: &str = "triple-c.migration-state";
|
||||
/// Value of [`LABEL_MIGRATION_STATE`] while a migration is unfinished.
|
||||
pub const MIGRATION_LABEL_IN_PROGRESS: &str = "in-progress";
|
||||
/// Label stamped on the short-lived probe containers [`run_throwaway`] creates.
|
||||
///
|
||||
/// They are removed on every path including failure, but a hard crash of the
|
||||
/// app (or of Docker) between create and remove would otherwise leave a
|
||||
/// container that carries no `triple-c.*` marking at all — invisible to every
|
||||
/// cleanup this app has, and unattributable by hand. The label makes
|
||||
/// `docker ps -a --filter label=triple-c.probe=migration` find them.
|
||||
pub const LABEL_PROBE: &str = "triple-c.probe";
|
||||
/// Value of [`LABEL_PROBE`] on a migration manifest/pre-flight probe container.
|
||||
pub const PROBE_LABEL_MIGRATION: &str = "migration";
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Manifests
|
||||
@@ -437,6 +475,72 @@ pub fn verbatim_payload_bytes(from: &Manifest, verbatim: &[String]) -> u64 {
|
||||
.sum()
|
||||
}
|
||||
|
||||
/// The reporting unit under a [`DATA_ROOTS`] entry: the first path component
|
||||
/// below the root, e.g. `/var/lib/postgresql`. Directory-level, because that is
|
||||
/// the granularity a user can actually act on ("dump this database"), and
|
||||
/// because a per-file list of a Postgres cluster would be thousands of lines.
|
||||
fn data_unit(path: &str) -> Option<String> {
|
||||
for root in DATA_ROOTS {
|
||||
let prefix = format!("{}/", root);
|
||||
if let Some(rest) = path.strip_prefix(&prefix) {
|
||||
let first = rest.split('/').next()?;
|
||||
if first.is_empty() {
|
||||
return None;
|
||||
}
|
||||
return Some(format!("{}/{}", root, first));
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Data-bearing subtrees under [`DATA_ROOTS`] that the migration will destroy
|
||||
/// and cannot restore, with the size and file count of what is at risk.
|
||||
///
|
||||
/// A subtree qualifies when **all** of:
|
||||
/// * it is a first-level directory under a [`DATA_ROOTS`] entry,
|
||||
/// * the current base image does not have that directory **at all** — if the
|
||||
/// base ships it, it is the base's own machinery (`/var/lib/apt`,
|
||||
/// `/var/lib/dpkg`, `/var/lib/systemd`, …) and the base's copy is the right
|
||||
/// one, exactly as for `/etc`,
|
||||
/// * it contains at least one regular file that neither image's dpkg database
|
||||
/// owns — a package's own scaffolding is recreated by the apt replay, the
|
||||
/// data written into it is not.
|
||||
///
|
||||
/// That pair of filters is what keeps this quiet on an ordinary container and
|
||||
/// loud on one running a database: `/var/lib/postgresql` is absent from the
|
||||
/// base and full of unowned files, while `/var/lib/apt/lists` is present in the
|
||||
/// base and never reported.
|
||||
pub fn unpreserved_data(from: &Manifest, base: &Manifest) -> Vec<UnpreservedData> {
|
||||
let base_paths = base.path_set();
|
||||
let mut acc: BTreeMap<String, (u64, u32)> = BTreeMap::new();
|
||||
|
||||
for entry in &from.paths {
|
||||
let Some(unit) = data_unit(&entry.path) else {
|
||||
continue;
|
||||
};
|
||||
if base_paths.contains(unit.as_str()) {
|
||||
continue;
|
||||
}
|
||||
if entry.is_dir() {
|
||||
continue;
|
||||
}
|
||||
if from.dpkg_owned.contains(&entry.path) || base.dpkg_owned.contains(&entry.path) {
|
||||
continue;
|
||||
}
|
||||
let slot = acc.entry(unit).or_insert((0, 0));
|
||||
slot.0 = slot.0.saturating_add(entry.size);
|
||||
slot.1 += 1;
|
||||
}
|
||||
|
||||
acc.into_iter()
|
||||
.map(|(path, (bytes, file_count))| UnpreservedData {
|
||||
path,
|
||||
bytes,
|
||||
file_count,
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Base-image capabilities the container does not have, as
|
||||
/// `(concrete paths, human labels)`.
|
||||
///
|
||||
@@ -582,6 +686,13 @@ pub async fn run_throwaway(image: &str, script: &str) -> Result<ThrowawayResult,
|
||||
user: Some("root".to_string()),
|
||||
working_dir: Some("/".to_string()),
|
||||
tty: Some(false),
|
||||
// Written explicitly rather than inherited: a probe container that
|
||||
// outlives a crash has to be findable, and nothing else in the app
|
||||
// labels these.
|
||||
labels: Some(HashMap::from([(
|
||||
LABEL_PROBE.to_string(),
|
||||
PROBE_LABEL_MIGRATION.to_string(),
|
||||
)])),
|
||||
host_config: Some(HostConfig {
|
||||
// No mounts on purpose: this must observe the *image*, not the
|
||||
// project's volumes, which are exactly the state migration does
|
||||
@@ -1180,6 +1291,99 @@ mod tests {
|
||||
assert_eq!(verbatim_payload_bytes(&from, &verbatim), 300);
|
||||
}
|
||||
|
||||
// ── Data that migration destroys and cannot restore ─────────────────────
|
||||
|
||||
#[test]
|
||||
fn a_database_under_var_lib_is_reported_because_nothing_replays_it() {
|
||||
// The regression this exists for: replaying `postgresql` onto the new
|
||||
// base reinstalls the package and gets an empty cluster. The ordinary
|
||||
// recreate path keeps /var because it creates from the snapshot, so a
|
||||
// silent migration would be *more* destructive than the thing it
|
||||
// replaces.
|
||||
let from = manifest(
|
||||
&[
|
||||
('d', 4096, "/var/lib/postgresql"),
|
||||
('d', 4096, "/var/lib/postgresql/16/main"),
|
||||
('f', 8192, "/var/lib/postgresql/16/main/PG_VERSION"),
|
||||
('f', 1024, "/var/lib/postgresql/16/main/base/1/2"),
|
||||
('d', 4096, "/var/www"),
|
||||
('d', 4096, "/var/www/site"),
|
||||
('f', 500, "/var/www/site/index.html"),
|
||||
],
|
||||
&[],
|
||||
&[],
|
||||
);
|
||||
let got = unpreserved_data(&from, &Manifest::default());
|
||||
assert_eq!(
|
||||
got.iter().map(|d| d.path.as_str()).collect::<Vec<_>>(),
|
||||
vec!["/var/lib/postgresql", "/var/www/site"]
|
||||
);
|
||||
assert_eq!(got[0].bytes, 9216);
|
||||
assert_eq!(got[0].file_count, 2);
|
||||
// And it is emphatically not in the copy set — reporting is the whole
|
||||
// answer here, not a half-working copy of a live database.
|
||||
assert!(!COPY_ROOTS.iter().any(|r| is_under("/var/lib/postgresql", r)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn package_machinery_under_var_is_never_reported_as_data_at_risk() {
|
||||
// /var/lib/apt exists in the base too, so it is the base's to own —
|
||||
// the same rule /etc gets. Reporting apt's lists would bury the one
|
||||
// line that matters under noise on every single migration.
|
||||
let from = manifest(
|
||||
&[
|
||||
('d', 4096, "/var/lib/apt"),
|
||||
('f', 900_000, "/var/lib/apt/lists/some.mirror_InRelease"),
|
||||
('d', 4096, "/var/lib/dpkg"),
|
||||
('f', 4096, "/var/lib/dpkg/status"),
|
||||
],
|
||||
&[],
|
||||
&[],
|
||||
);
|
||||
let base = manifest(
|
||||
&[
|
||||
('d', 4096, "/var/lib/apt"),
|
||||
('d', 4096, "/var/lib/dpkg"),
|
||||
('f', 4096, "/var/lib/dpkg/status"),
|
||||
],
|
||||
&[],
|
||||
&[],
|
||||
);
|
||||
assert!(unpreserved_data(&from, &base).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_packages_own_scaffolding_under_var_is_not_data() {
|
||||
// nginx-common ships /var/www/html/index.nginx-debian.html. The apt
|
||||
// replay puts that back; only what the user wrote is at risk.
|
||||
let from = manifest(
|
||||
&[
|
||||
('d', 4096, "/var/www/html"),
|
||||
('f', 612, "/var/www/html/index.nginx-debian.html"),
|
||||
],
|
||||
&["/var/www/html/index.nginx-debian.html"],
|
||||
&[],
|
||||
);
|
||||
assert!(unpreserved_data(&from, &Manifest::default()).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn data_is_reported_per_directory_not_per_file() {
|
||||
assert_eq!(
|
||||
data_unit("/var/lib/mysql/ibdata1").as_deref(),
|
||||
Some("/var/lib/mysql")
|
||||
);
|
||||
// A first-level directory is its own unit.
|
||||
assert_eq!(
|
||||
data_unit("/var/lib/mysql").as_deref(),
|
||||
Some("/var/lib/mysql")
|
||||
);
|
||||
// The root itself is not: it exists in every image.
|
||||
assert_eq!(data_unit("/var/lib").as_deref(), None);
|
||||
assert_eq!(data_unit("/var/www").as_deref(), None);
|
||||
assert_eq!(data_unit("/usr/local/bin/tool"), None);
|
||||
}
|
||||
|
||||
// ── Bind-mount exclusion ────────────────────────────────────────────────
|
||||
|
||||
fn pp(mount: &str) -> ProjectPath {
|
||||
|
||||
Reference in New Issue
Block a user