Move existing gateways onto the pinned LiteLLM after an app update #69

Merged
jknapp merged 9 commits from fix/gateway-image-upgrade into main 2026-11-03 00:24:29 +00:00
Owner

Follow-up to #68. Bumping the LiteLLM pin didn't reach existing gateways. The recreate fingerprint ignored the image. An old local triple-c-gateway:latest build always won over the new pin and counted as "image present".

Changes

  • Recreate on image change. A gateway is recreated when its config changes, or when a pinned image is present and the container runs a different one (needs_recreate). The fingerprint keeps its old format (config + secret version), so the app update alone recreates nothing. The container is created from the resolved image id.
  • Local builds are tagged triple-c-gateway:<pin>-<hash of the Dockerfile's significant lines + config.yaml>, never latest. A build for an older pin or an older build context simply doesn't match.
  • Pulls and lookups go by digest. The registry image is pulled and resolved by its digest, the same digest as the Dockerfile's FROM (enforced by a test). After the pull it is tagged locally so docker image prune doesn't remove it. image::pull_image now supports digest references.
  • No automatic pull. If no pinned image is present, a working gateway keeps running on its current image, recreated with new config if that changed. A failed image lookup takes the same path. A gateway is never torn down over a missing image.
  • Status gains image_outdated, from the same predicate ensure uses. Settings shows "gateway image update available, Pull or Build to get it" or "new gateway image ready, Stop and Start to apply". Stop and Start stay reachable for an existing container.

User-visible effect

After an app update that moves the pin, Settings shows the update prompt. Pull (or Build), then Stop and Start, applies it.

Known follow-ups (deliberately out of scope)

  • Old local builds: triple-c-gateway:latest builds from earlier app versions are not reused, even at the same pin. Affected users see the update prompt and Pull or Build.
  • No automatic cleanup: old gateway images (latest, older pins and older build hashes) aren't removed. A daemon-wide reference= sweep can delete another installed app's build, so they're removed manually for now.
  • STT container: it has the same floating-latest precedence issue. It needs a shared helper in a separate PR.
  • Minor:
    • ensure re-queries Docker for its returned status.
    • A few thin wrappers could be removed.
    • The status label could be a small function.

Testing

  • Rust: 886 cargo test tests pass. New tests cover image choice, recreate/outdated agreement, the unchanged fingerprint format, tag format and comment-insensitivity, the digest ref, Dockerfile lockstep and digest ref parsing. No new clippy warnings.
  • Frontend: 1169 tests pass, including new Settings tests that fail on the old UI. tsc is clean.
  • Live against Docker: pull by digest plus tag, and digest lookup.
  • Review: 8 review rounds.
  • Not yet run: a full gateway start in the running app.

🤖 Generated with Claude Code

Follow-up to #68. Bumping the LiteLLM pin didn't reach existing gateways. The recreate fingerprint ignored the image. An old local `triple-c-gateway:latest` build always won over the new pin and counted as "image present". ## Changes - **Recreate on image change.** A gateway is recreated when its config changes, or when a pinned image is present and the container runs a different one (`needs_recreate`). The fingerprint keeps its old format (config + secret version), so the app update alone recreates nothing. The container is created from the resolved image id. - **Local builds are tagged `triple-c-gateway:<pin>-<hash of the Dockerfile's significant lines + config.yaml>`**, never `latest`. A build for an older pin or an older build context simply doesn't match. - **Pulls and lookups go by digest.** The registry image is pulled and resolved by its digest, the same digest as the Dockerfile's `FROM` (enforced by a test). After the pull it is tagged locally so `docker image prune` doesn't remove it. `image::pull_image` now supports digest references. - **No automatic pull.** If no pinned image is present, a working gateway keeps running on its current image, recreated with new config if that changed. A failed image lookup takes the same path. A gateway is never torn down over a missing image. - **Status gains `image_outdated`**, from the same predicate `ensure` uses. Settings shows "gateway image update available, Pull or Build to get it" or "new gateway image ready, Stop and Start to apply". Stop and Start stay reachable for an existing container. ## User-visible effect After an app update that moves the pin, Settings shows the update prompt. Pull (or Build), then Stop and Start, applies it. ## Known follow-ups (deliberately out of scope) - **Old local builds:** `triple-c-gateway:latest` builds from earlier app versions are not reused, even at the same pin. Affected users see the update prompt and Pull or Build. - **No automatic cleanup:** old gateway images (`latest`, older pins and older build hashes) aren't removed. A daemon-wide `reference=` sweep can delete another installed app's build, so they're removed manually for now. - **STT container:** it has the same floating-`latest` precedence issue. It needs a shared helper in a separate PR. - **Minor:** - `ensure` re-queries Docker for its returned status. - A few thin wrappers could be removed. - The status label could be a small function. ## Testing - **Rust:** 886 `cargo test` tests pass. New tests cover image choice, recreate/outdated agreement, the unchanged fingerprint format, tag format and comment-insensitivity, the digest ref, Dockerfile lockstep and digest ref parsing. No new clippy warnings. - **Frontend:** 1169 tests pass, including new Settings tests that fail on the old UI. `tsc` is clean. - **Live against Docker:** pull by digest plus tag, and digest lookup. - **Review:** 8 review rounds. - **Not yet run:** a full gateway start in the running app. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
jknapp added 1 commit 2026-11-02 23:00:41 +00:00
Move existing gateways onto the pinned LiteLLM after an app update
Secret Scan / scan (push) Successful in 5s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 4s
Build App (Preview) / create-release (pull_request) Successful in 3s
Build App (Preview) / build-macos (pull_request) Successful in 3m15s
Build App (Preview) / test (pull_request) Successful in 4m4s
Build App (Preview) / build-windows (pull_request) Successful in 6m49s
Build App (Preview) / build-linux (pull_request) Successful in 8m10s
Build App (Preview) / prune-previews (pull_request) Successful in 3s
9d2e5734b6
An app update that bumps the LiteLLM pin left existing gateways on the old
version: the container fingerprint covered only config and secrets, so the
old container kept matching, and a local `triple-c-gateway:latest` from an
earlier app version always won over the new pinned image.

- Resolve the gateway image once, and put its image id in the container
  fingerprint so a newly pulled or rebuilt image recreates the container.
- Label local builds with the pinned LiteLLM version and ignore a local
  build whose label doesn't match (older or unlabelled builds), falling back
  to the pinned upstream image; status reports no usable image so the UI
  offers a pull/build.
- Test that the gateway Dockerfile's FROM stays in lockstep with the pin,
  since the build label relies on it.

Every gateway is recreated once on its next start after this ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-02 23:08:50 +00:00
Gateway upgrade: auto-pull, never leave a stale gateway, probe old builds
Secret Scan / scan (push) Successful in 5s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 5s
Build App (Preview) / create-release (pull_request) Successful in 2s
Build App (Preview) / build-macos (pull_request) Successful in 3m13s
Build App (Preview) / test (pull_request) Successful in 4m12s
Build App (Preview) / build-windows (pull_request) Successful in 6m45s
Build App (Preview) / build-linux (pull_request) Successful in 7m57s
Build App (Preview) / prune-previews (pull_request) Successful in 3s
00b504063f
Review follow-ups:
- When no usable image is present, pull the pinned image instead of failing.
  If the pull fails, stop and remove the existing gateway rather than leave
  an old LiteLLM (and, after a settings change or key rotation, an old
  config and old keys) serving.
- A local build without the version label is probed for its LiteLLM
  version (cached by image id) instead of being treated as stale.
- An engine that reports no image id still gets a fingerprint component
  that changes on rebuild/re-pull.
- Inspect the two candidate images concurrently; log the ignored-build
  line at debug.
- Settings shows a running gateway, and its Stop button, even when its
  image is no longer the pinned one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-02 23:14:34 +00:00
Gateway upgrade: tag local builds with the pin; pull outside the lock
Build App (Preview) / compute-version (pull_request) Successful in 8s
Secret Scan / scan (push) Successful in 12s
Secret Scan / scan (pull_request) Successful in 9s
Build App (Preview) / create-release (pull_request) Successful in 1s
Build App (Preview) / test (pull_request) Successful in 4m18s
Build App (Preview) / build-macos (pull_request) Successful in 3m15s
Build App (Preview) / build-windows (pull_request) Successful in 6m51s
Build App (Preview) / build-linux (pull_request) Successful in 9m31s
Build App (Preview) / prune-previews (pull_request) Successful in 3s
03c77c5f55
Second-review follow-ups, replacing the label + probe approach:
- Local builds are tagged `triple-c-gateway:<pin>` instead of `latest`, so
  a build for an older pin simply doesn't match. Drops the version label,
  the probe container, its cache and version normalisation — and with them
  the case where a failed probe made a current build look stale.
- The pinned image is pulled before taking the gateway lock, so a long
  pull no longer blocks Stop, shutdown or settings saves; skipped when the
  start would fail on configuration anyway.
- If no pinned image is available, the old container is removed and the
  error keeps the pull failure, reporting a removal failure alongside it.
- Status gains `image_outdated` (container image id != pinned image id).
  Settings uses it for the label, keeps Stop for any running gateway, and
  always offers Start, since Start pulls the image.
- Reuse `migration::image_id` and one container-removal helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-02 23:19:44 +00:00
Gateway upgrade: pull only at launch, never tear down a working gateway
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (push) Successful in 7s
Secret Scan / scan (pull_request) Successful in 5s
Build App (Preview) / create-release (pull_request) Successful in 2s
Build App (Preview) / build-macos (pull_request) Successful in 3m14s
Build App (Preview) / test (pull_request) Successful in 5m34s
Build App (Preview) / build-linux (pull_request) Successful in 5m45s
Build App (Preview) / build-windows (pull_request) Successful in 6m56s
Build App (Preview) / prune-previews (pull_request) Successful in 6s
1ef4505a69
Third-review follow-ups:
- The pinned image is pulled in the background at app launch only, not
  inside every ensure, so Start and settings saves never stall on a
  download.
- With no pinned image present, ensure keeps the gateway on the image it
  already runs (recreated with new config if that changed) instead of
  removing it; status reports it as outdated.
- The container is created from the resolved image id, so a concurrent
  build/pull retag can't make it disagree with its fingerprint.
- "Outdated" means the container runs neither pinned image, so a later
  local build of the same pin doesn't flag a registry-image gateway.
- After starting on the pin, superseded triple-c-gateway:* tags (the old
  floating `latest`, earlier pins) are removed so they don't linger.
- Reuse container::remove_container (404-tolerant); fix misplaced docs.
- Settings: Start needs an image again (no silent pull); the outdated
  label says whether to Pull/Build or to Stop and Start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-02 23:25:08 +00:00
Gateway upgrade: start before pulling, graceful recreate, safer cleanup
Secret Scan / scan (push) Successful in 5s
Build App (Preview) / compute-version (pull_request) Successful in 4s
Build App (Preview) / create-release (pull_request) Successful in 3s
Secret Scan / scan (pull_request) Successful in 11s
Build App (Preview) / build-macos (pull_request) Successful in 3m15s
Build App (Preview) / test (pull_request) Successful in 5m39s
Build App (Preview) / build-linux (pull_request) Successful in 5m36s
Build App (Preview) / build-windows (pull_request) Successful in 6m46s
Build App (Preview) / prune-previews (pull_request) Successful in 2s
1ef5220961
Fourth-review follow-ups:
- Autostart starts the gateway on whatever image it has first, then pulls
  the new pin (only if the gateway is configured) and moves it over, so
  the gateway isn't down for the download; a first start with no image at
  all still gets the pull.
- Recreates stop the container gracefully before removing it again, so
  in-flight requests can drain.
- `image_outdated` now means exactly "ensure would recreate it", so status
  and the fingerprint can't disagree; the label says "new gateway image
  ready" rather than claiming a LiteLLM update.
- Cleanup removes only `latest` and builds for older pins, never a newer
  pin or a custom tag.
- Settings: Start is offered for a stopped container even without a pinned
  image (it restarts on its own image); outdated no longer uses the
  "starting" tone. Concurrent lookups via join.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-02 23:35:27 +00:00
Gateway upgrade: drop auto-pull, pull by digest, content-tag local builds
Secret Scan / scan (push) Successful in 6s
Build App (Preview) / compute-version (pull_request) Successful in 6s
Secret Scan / scan (pull_request) Successful in 5s
Build App (Preview) / create-release (pull_request) Successful in 3s
Build App (Preview) / build-macos (pull_request) Successful in 3m14s
Build App (Preview) / test (pull_request) Successful in 5m38s
Build App (Preview) / build-linux (pull_request) Successful in 5m45s
Build App (Preview) / build-windows (pull_request) Successful in 6m48s
Build App (Preview) / prune-previews (pull_request) Successful in 2s
2ea5bba830
Per the decision to drop the automatic pull and keep an outdated gateway
running (flagged) when the new image is absent:
- Remove the launch-time pull; lib.rs is unchanged from main. Updating is
  Pull or Build, then Stop/Start (or the next launch).
- Pull and look up the registry image by the pinned digest (the same one
  the Dockerfile's FROM uses, enforced by test), not the mutable tag.
  image::pull_image now handles digest references.
- Local builds are tagged `<pin>-<hash of Dockerfile+config>`, so a
  changed build context also stops an old build from winning.
- A failed lookup of one image no longer hides the other.
- Superseded-build cleanup runs off the lock, never forces, and also
  drops earlier builds of the current pin; recreate stops with the same
  grace as Stop; reuse update_commands::parse_semver.
- Settings: a stopped outdated gateway without a new image says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-02 23:48:36 +00:00
Gateway upgrade: review round 6 fixes
Secret Scan / scan (push) Successful in 5s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 5s
Build App (Preview) / create-release (pull_request) Successful in 4s
Build App (Preview) / build-macos (pull_request) Successful in 3m15s
Build App (Preview) / test (pull_request) Successful in 4m10s
Build App (Preview) / build-windows (pull_request) Successful in 6m50s
Build App (Preview) / build-linux (pull_request) Successful in 8m3s
Build App (Preview) / prune-previews (pull_request) Successful in 2s
b2cae374f9
- Hash only significant lines of the build context, so a comment edit in
  a later release doesn't mark every local build outdated.
- A failed image lookup no longer fails ensure for a gateway that is fine:
  it takes the keep-current-image path.
- Recreates stop with Docker's default grace again, as before; the 3s
  grace stays for app exit.
- Cleanup keeps every build of the current pin (a stable and a preview
  app no longer delete each other's), removing only `latest` and older
  pins; parse locally instead of depending on the commands layer.
- GatewayContainer struct instead of a 4-tuple; separate repo/version
  constants; cache the local tag; fix the outdated-check doc.
- UI says "gateway image update available" rather than claiming a LiteLLM
  update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-02 23:54:35 +00:00
Gateway upgrade: config-only fingerprint, tag digest pulls, drop cleanup
Secret Scan / scan (push) Successful in 10s
Build App (Preview) / compute-version (pull_request) Successful in 9s
Secret Scan / scan (pull_request) Successful in 10s
Build App (Preview) / create-release (pull_request) Successful in 2s
Build App (Preview) / build-macos (pull_request) Successful in 3m13s
Build App (Preview) / test (pull_request) Successful in 6m6s
Build App (Preview) / build-windows (pull_request) Successful in 6m46s
Build App (Preview) / build-linux (pull_request) Successful in 9m25s
Build App (Preview) / prune-previews (pull_request) Successful in 2s
7077d36f54
Review round 7:
- Keep the fingerprint in its earlier format (config + secret version)
  so the app update alone recreates no gateway; image changes go through
  one `needs_recreate` rule (pinned image present and container on another
  image), which the status `image_outdated` mirrors.
- Tag the image after a digest pull so `docker image prune` doesn't treat
  it as dangling; lookups still go by digest.
- Drop superseded-build cleanup: `reference=` is daemon-wide, so it could
  delete another installed app's build, and the version parsing didn't
  cope with suffixed LiteLLM tags.
- Derive the lockstep test's `repo:tag` from the repo/version constants;
  correct the `image_outdated` docs for the no-pinned-image case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp added 1 commit 2026-11-03 00:01:28 +00:00
Gateway upgrade: review round 8 fixes
Secret Scan / scan (push) Successful in 7s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Build App (Preview) / test (pull_request) Successful in 4m36s
Secret Scan / scan (pull_request) Successful in 7s
Build App (Preview) / create-release (pull_request) Successful in 4s
Build App (Preview) / build-macos (pull_request) Successful in 3m14s
Build App (Preview) / build-linux (pull_request) Successful in 9m29s
Build App (Preview) / build-windows (pull_request) Successful in 6m51s
Build App (Preview) / prune-previews (pull_request) Successful in 1s
53151dfd42
- A failed lookup of the preferred local build is "unknown", not "use the
  registry image", so a transient error can't swap a working local-build
  gateway onto the upstream image.
- One image predicate (`image_switch_pending`) behind both `needs_recreate`
  and status `image_outdated`, so they agree, including for a container
  whose image id is unknown.
- A failed tag after a successful digest pull is logged, not reported as a
  failed pull.
- Keep Dockerfile parser directives (`# syntax=` etc.) in the build-context
  hash; hash config.yaml whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jknapp merged commit 1fc1f44c35 into main 2026-11-03 00:24:29 +00:00
jknapp deleted branch fix/gateway-image-upgrade 2026-11-03 00:24:30 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: CyberCoveLLC/Triple-C#69