CyberCoveLLC/Triple-C
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 44 Wiki Activity

Fix disk/migration defects and add a real per-project lock

Browse Source 
The compaction panel's headline action had never worked, three reclaim
paths could delete data with no confirmation and no grace period, and the
app's only mutual-exclusion primitive was one-way.

**A per-project lock (`project_lock.rs`).** `ACTIVE_MIGRATIONS` was the
app's only exclusion and everything but migration merely *polled* it once
at entry. Compaction, start/stop/recreate, Reset and destroy now
**acquire** a `ProjectGuard` and hold it for the whole operation;
`is_migrating` is a view onto the same registry. Closes the three verified
interleavings where a compaction commits `flat(A)` over a `:latest` that a
migration, a recreate or a Reset had already moved. In-process only — the
two-instance case is documented in the module, not solved, and the
daemon-wide reapers gained age gates to bound it.

**H1: compaction never ran.** `fold_shell_script` joined the scrub script's
lines with a space, so every build died on `syntax error: unexpected "do"`.
Replaced with the JSON exec form, which carries any script verbatim;
`sh -n` and a real end-to-end build now cover it (159.5 MB / 9 layers ->
33.7 MB / 1 layer, setuid and multi-line env preserved).

**H2/H4:** `reclaim_migration_pins` and `survey_rollback_pins` apply
`parse_rollback_tag` and `pin_is_reapable` like every other path, and stop
double-counting an image with two pin tags. The 14-day grace period is
re-anchored from the tag's timestamp (when the migration *started*) to a
tombstone recording when the record went missing, with clock skew handled
in both directions.

**H3:** `migration_store::load` no longer renames a corrupt record aside —
that destroyed the `has_record` signal both pin reapers depend on. `save`
fsyncs the file and the directory, and corruption backups are timestamped.

**H2b/M2:** a crashed compaction's `:compacting` tag and `triple-c-compact-*`
container are reaped at startup; the stale-container sweep moved from the
end of a compaction to the start, where its doc always claimed it was.

**M5/M6:** orphan-volume deletion moved from a `Safety::Safe` tick to the
destructive path with a typed volume name; `project_store_trust` reads the
real `projects.json` so a second instance's project is not offered as an
orphan.

Numbers: `images_total_bytes` uses `df()`'s deduplicated `layers_size`; the
Total column is derived from the same figure the Snapshot column shows;
partial container/staging reclaims report their failure count; `human()`
no longer prints "1000.0 KB"; `docker_cli` has a timeout; blocking `fs`
calls moved to `spawn_blocking`.

Also fixes `ProbeContainerGuard::remove_now`, which disarmed before the
await and so did nothing on the cancellation path it exists for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GBq2rGum6GX7xXgsas1fDc
This commit is contained in:
shadow-test
2026-08-23 11:45:28 -07:00
co-authored by Claude Opus 5
parent 0003793abb
commit 6abc7f27a4
8 changed files with 2371 additions and 342 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/src-tauri/src/docker/migration.rs
+181 -16
View File
@@ -754,9 +754,16 @@ impl ProbeContainerGuard {
&self.id
}
/// **Disarm after the await, never before it.** Clearing `armed` first
/// looked equivalent and was the exact inverse of this guard's purpose: on
/// the one path it exists for — this future being dropped part-way through
/// the removal — `Drop` then saw a disarmed guard and did nothing, so the
/// container survived with no background removal queued behind it. Setting
/// it afterwards means a cancelled `remove_now` falls back to `Drop`'s
/// detached removal, and only a removal that actually completed disarms.
async fn remove_now(mut self) {
self.armed = false;
remove_probe_container(&self.id).await;
self.armed = false;
}
}
@@ -810,6 +817,17 @@ async fn remove_probe_container(id: &str) {
/// and no volumes, owned entirely by a `run_throwaway` call. If one is running
/// right now it belongs to this process — and this runs before any migration
/// can be started, so there is none to interrupt.
///
/// **Except that "belongs to this process" is not something this can know.**
/// The filter is a label, and labels are daemon-wide: a second copy of the app
/// migrating a project on the same daemon has probe containers carrying exactly
/// this label, and force-removing one mid-manifest-capture fails that
/// migration. In-process state cannot see the other instance, so the only
/// available brake is age — [`PROBE_REAP_MIN_AGE_SECS`]. A probe runs a `df`, an
/// `apt-get update` or a `find` over a root filesystem; none of those is a
/// multi-minute job, so anything younger than the gate is far more likely to be
/// someone's live probe than a leftover, and a leftover simply waits for the
/// next start.
pub async fn reap_probe_containers() {
let Ok(docker) = get_docker() else {
return;
@@ -835,7 +853,23 @@ pub async fn reap_probe_containers() {
}
};
let now = chrono::Utc::now().timestamp();
for c in containers {
// `created` is a unix timestamp; a summary without one is treated as
// too young to touch, because unknown is never permission.
let age = c.created.map(|created| now - created);
match age {
Some(age) if age >= PROBE_REAP_MIN_AGE_SECS => {}
_ => {
log::info!(
"Leaving migration probe container {} alone — it is younger than {} minutes, \
so it may belong to another Triple-C instance's live migration",
c.id.as_deref().unwrap_or("<unknown>"),
PROBE_REAP_MIN_AGE_SECS / 60
);
continue;
}
}
if let Some(id) = c.id {
log::info!("Removing leftover migration probe container {}", id);
remove_probe_container(&id).await;
@@ -843,6 +877,16 @@ pub async fn reap_probe_containers() {
}
}
/// How old a `triple-c.probe=migration` container must be before
/// [`reap_probe_containers`] will force-remove it, in seconds.
///
/// The label is daemon-wide and this process cannot tell its own leftovers from
/// another instance's live probe, so this is the whole guard. Generous against
/// the longest probe there is (an `apt-get update` inside a throwaway container
/// on a slow link) and still short enough that a crashed run's probe stops
/// pinning a multi-gigabyte base image within the hour.
pub const PROBE_REAP_MIN_AGE_SECS: i64 = 30 * 60;
async fn run_throwaway_inner(id: &str) -> Result<ThrowawayResult, String> {
let docker = get_docker()?;
@@ -1025,6 +1069,10 @@ pub fn rollback_tag(now: &chrono::DateTime<chrono::Utc>) -> String {
///
/// Two weeks, chosen to be far longer than anyone deliberates over a base
/// update and far shorter than "forever", which is what it was.
///
/// **Measured from when the record went missing, not from the tag.** See
/// [`pin_is_reapable`] and
/// [`crate::storage::migration_store::note_ownerless_since`].
pub const STALE_PIN_MAX_AGE_DAYS: i64 = 14;
/// Recover the timestamp encoded in a tag produced by [`rollback_tag`].
@@ -1051,26 +1099,59 @@ pub fn parse_snapshot_reference(reference: &str) -> Option<(String, String)> {
Some((project_id, tag))
}
/// Whether a rollback pin is safe to drop, given how old it is and whether the
/// project it belongs to still has a migration record.
/// Whether a rollback pin is safe to drop, given whether the project it belongs
/// to still has a migration record and how long it has been without one.
///
/// Pure so the decision can be tested without a daemon. The order of the two
/// Pure so the decision can be tested without a daemon. The order of the
/// conditions is the point: **a pin whose migration is still awaiting
/// confirmation is never reaped at any age**, because it is the only copy of
/// the rollback target and the user has not yet said they are happy with the
/// new base.
///
/// ## `ownerless_since`, and why it is not the tag's timestamp
///
/// This used to compute the age from `parse_rollback_tag(tag)` — the instant
/// the migration *started*. A migration is allowed to sit at
/// `awaiting-confirmation` for as long as the user likes; that is what
/// `keep_rollback` is for. A project parked there for a month whose record is
/// then lost had a tag a month old, so the pin was reapable on the very next
/// check and the startup sweep deleted the image immediately after. The
/// fourteen days were nominal: the real grace period for the case the constant
/// was written for was zero.
///
/// So the clock starts when the claim was lost, which is recorded by
/// [`crate::storage::migration_store::note_ownerless_since`] the first time a
/// reaper notices. `None` means no reaper has recorded a sighting yet, and that
/// is **not** "sighted now": returning false there is what gives a pin its
/// first full fourteen days instead of none.
///
/// ## Clock skew
///
/// A `now` earlier than `ownerless_since` — a host clock that ran fast and was
/// corrected, or a data directory carried between machines — yields a negative
/// elapsed time. That is treated as not reapable, and the marker writer
/// re-anchors it, rather than letting a negative `num_days()` mean "never" or
/// an inflated one mean "immediately".
pub fn pin_is_reapable(
tag: &str,
has_migration_record: bool,
ownerless_since: Option<chrono::DateTime<chrono::Utc>>,
now: &chrono::DateTime<chrono::Utc>,
) -> bool {
if has_migration_record {
return false;
}
let Some(created) = parse_rollback_tag(tag) else {
// Still required: the tag has to be one of ours. A hand-made
// `pre-migration-keepme` is somebody's deliberate pin and is never guessed
// at, whatever a marker beside it says.
if parse_rollback_tag(tag).is_none() {
return false;
}
let Some(since) = ownerless_since else {
return false;
};
(*now - created).num_days() >= STALE_PIN_MAX_AGE_DAYS
let elapsed = *now - since;
elapsed >= chrono::Duration::days(STALE_PIN_MAX_AGE_DAYS)
}
/// Drop `triple-c-snapshot-*:pre-migration-*` tags that no migration record
@@ -1138,15 +1219,35 @@ pub async fn reap_stale_migration_pins() -> usize {
// still count as "somebody may want this back".
let has_record =
crate::storage::migration_store::has_record(&project_id).unwrap_or(true);
if !pin_is_reapable(&tag, has_record, &now) {
if has_record {
// Owned again (or still owned): throw away any grace clock a
// previous pass started, so a pin that loses its record twice
// gets a fresh fourteen days rather than inheriting a stale one.
crate::storage::migration_store::clear_ownerless(&project_id, &tag);
continue;
}
// Only a *well-formed* pin gets a marker written for it — a tag
// that is not one of ours is left entirely alone, files included.
if parse_rollback_tag(&tag).is_none() {
continue;
}
// Records the first sighting when there is none, which is why this
// returns `None` on that pass and the pin survives it.
let ownerless_since =
crate::storage::migration_store::note_ownerless_since(&project_id, &tag, &now);
if !pin_is_reapable(&tag, has_record, ownerless_since, &now) {
continue;
}
match untag_image(reference).await {
Ok(()) => {
crate::storage::migration_store::clear_ownerless(&project_id, &tag);
log::info!(
"Dropped stale rollback pin {} ({:.2} GB) — no migration record has claimed it for {} days",
"Dropped stale rollback pin {} ({:.2} GB) — no migration record has claimed it since {}, more than {} days",
reference,
summary.size as f64 / 1_073_741_824.0,
ownerless_since
.map(|t| t.to_rfc3339())
.unwrap_or_else(|| "unknown".to_string()),
STALE_PIN_MAX_AGE_DAYS,
);
reaped += 1;
@@ -1915,29 +2016,93 @@ mod tests {
// image is the only copy of the rollback target and the user has not
// yet said they are happy on the new base.
let ancient = rollback_tag(&at(2020, 1, 1));
assert!(!pin_is_reapable(&ancient, true, &at(2026, 8, 23)));
assert!(!pin_is_reapable(
&ancient,
true,
Some(at(2020, 1, 1)),
&at(2026, 8, 23)
));
}
#[test]
fn an_unclaimed_pin_is_reaped_only_once_it_is_old() {
let made = at(2026, 8, 1);
let tag = rollback_tag(&made);
assert!(!pin_is_reapable(&tag, false, &at(2026, 8, 2)));
let tag = rollback_tag(&at(2026, 8, 1));
// The clock runs from when the record went missing, which here is well
// after the migration started.
let lost = at(2026, 8, 10);
assert!(!pin_is_reapable(&tag, false, Some(lost), &at(2026, 8, 11)));
assert!(!pin_is_reapable(
&tag,
false,
&(made + chrono::Duration::days(STALE_PIN_MAX_AGE_DAYS - 1))
Some(lost),
&(lost + chrono::Duration::days(STALE_PIN_MAX_AGE_DAYS) - chrono::Duration::seconds(1))
));
assert!(pin_is_reapable(
&tag,
false,
&(made + chrono::Duration::days(STALE_PIN_MAX_AGE_DAYS))
Some(lost),
&(lost + chrono::Duration::days(STALE_PIN_MAX_AGE_DAYS))
));
}
#[test]
fn the_grace_period_runs_from_the_lost_record_not_from_the_tag() {
// The bug this replaced, stated as a test. A migration parked at
// `awaiting-confirmation` for a month — supported, that is what
// `keep_rollback` is for — whose record is then lost had a
// month-old tag, so the old rule made its pin reapable on the very
// next app start with the startup sweep deleting the image two lines
// later. Zero grace, on the one case the fourteen days exist for.
let started = at(2026, 6, 1);
let tag = rollback_tag(&started);
let record_lost = at(2026, 7, 1);
let noticed_immediately_after = record_lost + chrono::Duration::minutes(5);
assert!(
!pin_is_reapable(&tag, false, Some(record_lost), &noticed_immediately_after),
"a tag a month old must still get its full grace period once orphaned"
);
}
#[test]
fn an_unsighted_pin_is_never_reaped_on_the_pass_that_first_sees_it() {
// `None` means no reaper has recorded a sighting. Treating that as
// "sighted now" would be harmless; treating it as "sighted long ago"
// would not, and neither is what it means — the marker is written on
// this pass and the pin becomes reapable fourteen days later.
let tag = rollback_tag(&at(2020, 1, 1));
assert!(!pin_is_reapable(&tag, false, None, &at(2026, 8, 23)));
}
#[test]
fn a_clock_that_ran_backwards_neither_reaps_nor_strands() {
// A marker dated after `now`: the host clock was fast and got
// corrected, or the data directory came from another machine. A
// negative elapsed time must read as "not yet", not as a huge age.
let tag = rollback_tag(&at(2026, 1, 1));
let marker = at(2026, 9, 1);
assert!(!pin_is_reapable(&tag, false, Some(marker), &at(2026, 8, 1)));
// The other direction is bounded by the marker rather than by the tag:
// a wildly future `now` can only expire a clock that was actually
// started, and a pin with no marker (the case above) still cannot be
// reaped at all — which is what stops a fast host clock from making
// *every* pin on the daemon instantly collectable.
assert!(pin_is_reapable(
&tag,
false,
Some(at(2026, 8, 20)),
&at(2030, 1, 1)
));
assert!(!pin_is_reapable(&tag, false, None, &at(2030, 1, 1)));
}
#[test]
fn a_tag_we_cannot_date_is_left_alone() {
assert!(!pin_is_reapable("pre-migration-handmade", false, &at(2026, 8, 23)));
assert!(!pin_is_reapable("latest", false, &at(2026, 8, 23)));
// Even with an ancient ownerless marker sitting beside it: a tag that
// merely *starts* `pre-migration-` is somebody's deliberate pin, and
// the reaper never writes a marker for one in the first place.
let ancient = Some(at(2020, 1, 1));
let now = at(2026, 8, 23);
assert!(!pin_is_reapable("pre-migration-handmade", false, ancient, &now));
assert!(!pin_is_reapable("latest", false, ancient, &now));
}
}