Stop the migration manifest overflowing on a large /workspace
Secret Scan / scan (push) Successful in 6s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 12s
Build App (Preview) / create-release (pull_request) Successful in 1s
Build App (Preview) / build-macos (pull_request) Successful in 3m16s
Build App (Preview) / test (pull_request) Successful in 4m9s
Build App (Preview) / build-windows (pull_request) Successful in 6m46s
Build App (Preview) / build-linux (pull_request) Successful in 10m1s
Build App (Preview) / prune-previews (pull_request) Successful in 2s
Secret Scan / scan (push) Successful in 6s
Build App (Preview) / compute-version (pull_request) Successful in 5s
Secret Scan / scan (pull_request) Successful in 12s
Build App (Preview) / create-release (pull_request) Successful in 1s
Build App (Preview) / build-macos (pull_request) Successful in 3m16s
Build App (Preview) / test (pull_request) Successful in 4m9s
Build App (Preview) / build-windows (pull_request) Successful in 6m46s
Build App (Preview) / build-linux (pull_request) Successful in 10m1s
Build App (Preview) / prune-previews (pull_request) Successful in 2s
get_container_staleness failed on every start/stop of a real container with "OUTPUT_LIMIT: Command output exceeded 8388608 bytes": the manifest's flat `find /workspace` walk alone was 8.6 MB (94k entries of worktrees, node_modules, vendored trees in the writable layer). Pruning node_modules and .git by name was measured on that container and only reached 6.3 MB. - Walk /workspace two levels deep; each non-empty directory at depth 2 is one `S` entry, sized from a new ###SUMMARY section (`du -sbx`). The entry comes from find, so a failed du costs the size, never the dir. Measured on the same container: 8.6 MB -> ~25 kB. - compute_verbatim_paths: summaries count as content (not an empty tree) and are exempt from presence-in-base, since a summary hides which paths are inside; restore's --skip-old-files keeps the per-file guarantee. The verbatim set is unchanged: it already pruned to /workspace/<x>. - manifest_from_container gets its own 32 MiB budget via a new exec_oneshot_as_limited, and an OUTPUT_LIMIT is reported as an actionable sentence (describe_manifest_failure) instead of the buffer message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e223a3f64b
commit
209ed00fdd
3 files changed
+383
-6
No files matched your search
@@ -579,6 +579,15 @@ security update. Migration is the non-destructive way out; Reset is the destruct
|
||||
the last commit. Migration diffs two filesystem manifests instead, filtered through dpkg
|
||||
ownership and presence-in-the-new-base. Measured on a real project, that turns 8,677 raw path
|
||||
differences into 2 genuinely user-authored ones.
|
||||
- **`/workspace` is walked two levels deep, never file by file** (`SUMMARISED_ROOTS`). Below that,
|
||||
each non-empty directory is one `S` entry sized by `du -sbx`. A user's worktrees and
|
||||
`node_modules` made a flat walk 8.6 MB — past the exec cap, so every staleness check failed on
|
||||
start and stop — and pruning `node_modules`/`.git` by name only reached 6.3 MB. Summaries lose
|
||||
nothing: below `/workspace/<x>` every per-path filter is vacuous, and `tar` copies a directory
|
||||
recursively. Two rules: the `S` line comes from `find`, never from `du`, so a failed size costs
|
||||
the estimate and not the directory; and a summary is exempt from presence-in-the-base (it hides
|
||||
which paths are inside — `--skip-old-files` on restore is the per-file answer). Depth 2, not 1,
|
||||
because `du -x` on a first-level bind-mount point would walk the host repository.
|
||||
|
||||
### Authentication
|
||||
|
||||
|
||||
@@ -642,6 +642,19 @@ pub async fn exec_oneshot_as(
|
||||
exec_oneshot_inner(container_id, user, cmd, env, MAX_ONESHOT_OUTPUT).await
|
||||
}
|
||||
|
||||
/// [`exec_oneshot_as`] with a caller-chosen output ceiling, for a reader that
|
||||
/// knows its legitimate output can pass [`MAX_ONESHOT_OUTPUT`] — the migration
|
||||
/// manifest of a container with a large `/opt` is the one today.
|
||||
pub async fn exec_oneshot_as_limited(
|
||||
container_id: &str,
|
||||
user: &str,
|
||||
cmd: Vec<String>,
|
||||
env: Vec<String>,
|
||||
limit: usize,
|
||||
) -> Result<(String, i64), String> {
|
||||
exec_oneshot_inner(container_id, user, cmd, env, limit).await
|
||||
}
|
||||
|
||||
/// What a one-shot exec printed, with the two streams still tellable apart.
|
||||
///
|
||||
/// `combined` is stdout and stderr interleaved in arrival order — the shape
|
||||
|
||||
@@ -94,6 +94,51 @@ pub const MANIFEST_ROOTS: &[&str] = &[
|
||||
"/var/www",
|
||||
];
|
||||
|
||||
/// [`MANIFEST_ROOTS`] entries walked only to [`SUMMARY_DEPTH`], with every
|
||||
/// non-empty directory at that depth reported as one [`SUMMARY_KIND`] entry
|
||||
/// carrying the subtree's total size.
|
||||
///
|
||||
/// **Why `/workspace`, and why it has to be.** It is the one root whose size is
|
||||
/// the user's rather than the image's: git worktrees, `node_modules`, a pnpm
|
||||
/// store and vendored trees all sit in the writable layer beside the bind
|
||||
/// mounts. A real container listed 94k entries — 8.6 MB of `find` output —
|
||||
/// under `/workspace` alone, which overflowed
|
||||
/// [`crate::docker::exec::MAX_ONESHOT_OUTPUT`] and failed the staleness check
|
||||
/// on every start and stop. Pruning by name (`node_modules`, `.git`) was
|
||||
/// measured on that container and only got it to 6.3 MB: most of it was a
|
||||
/// vendored phpMyAdmin in each of eight worktrees. Depth is what bounds it.
|
||||
///
|
||||
/// **Why summarising loses nothing.** Every per-path filter in
|
||||
/// [`compute_verbatim_paths`] is vacuous below `/workspace/<x>`: dpkg owns
|
||||
/// nothing there, no [`COPY_EXCLUSIONS`] entry is there, bind mounts are
|
||||
/// exactly `/workspace/<mount_name>` (a mount name cannot hold a `/`), and the
|
||||
/// base ships an empty `/workspace`. So the verbatim set was already pruned to
|
||||
/// first-level entries; a summary just stops listing what was going to be
|
||||
/// pruned away. Migration's `tar` still archives the directory recursively.
|
||||
///
|
||||
/// **Why depth 2 and not 1.** The first level holds the bind-mount points.
|
||||
/// `find -xdev` lists a mount point without descending, but measuring one with
|
||||
/// `du -x` would walk the whole host repository — so summaries start a level
|
||||
/// below, where `-xdev` has already refused to enter any mount.
|
||||
pub const SUMMARISED_ROOTS: &[&str] = &["/workspace"];
|
||||
|
||||
/// Depth below a [`SUMMARISED_ROOTS`] entry at which directories are
|
||||
/// summarised instead of walked. See there for why it is 2.
|
||||
pub const SUMMARY_DEPTH: usize = 2;
|
||||
|
||||
/// [`ManifestEntry::kind`] of a summarised directory. Not a `find` `%y` letter.
|
||||
pub const SUMMARY_KIND: char = 'S';
|
||||
|
||||
/// Output budget for one manifest exec against a running container.
|
||||
///
|
||||
/// Above [`crate::docker::exec::MAX_ONESHOT_OUTPUT`] on purpose. With
|
||||
/// [`SUMMARISED_ROOTS`] a real container's manifest is a few MB (`/usr/local`
|
||||
/// ~0.8 MB, `/var/lib` ~0.1 MB, the dpkg section a few hundred kB), but the
|
||||
/// non-summarised roots are still walked file by file and a user can install
|
||||
/// anything into `/opt`. Failing at 8 MiB cost the user every staleness check;
|
||||
/// 32 MiB keeps that rare while still bounding a container-controlled read.
|
||||
pub const MANIFEST_OUTPUT_LIMIT: usize = 32 * 1024 * 1024;
|
||||
|
||||
/// Roots holding **state a base-image swap destroys and no replay can put
|
||||
/// back**. Reported by [`unpreserved_data`], never copied.
|
||||
///
|
||||
@@ -173,16 +218,27 @@ pub const PROBE_LABEL_MIGRATION: &str = "migration";
|
||||
/// One entry from the filesystem walk.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ManifestEntry {
|
||||
/// `find`'s `%y`: `f` regular, `d` directory, `l` symlink, …
|
||||
/// `find`'s `%y`: `f` regular, `d` directory, `l` symlink, … — or
|
||||
/// [`SUMMARY_KIND`] for a directory under a [`SUMMARISED_ROOTS`] entry that
|
||||
/// was not walked file by file.
|
||||
pub kind: char,
|
||||
/// Bytes on disk; for a summary, the whole subtree's (`du -sb`), or 0 when
|
||||
/// that could not be measured.
|
||||
pub size: u64,
|
||||
pub path: String,
|
||||
}
|
||||
|
||||
impl ManifestEntry {
|
||||
/// A directory reported as itself, with nothing implied about content.
|
||||
/// A summary is deliberately *not* one: it stands for a non-empty subtree.
|
||||
pub fn is_dir(&self) -> bool {
|
||||
self.kind == 'd'
|
||||
}
|
||||
|
||||
/// A non-empty directory whose contents were aggregated rather than listed.
|
||||
pub fn is_summary(&self) -> bool {
|
||||
self.kind == SUMMARY_KIND
|
||||
}
|
||||
}
|
||||
|
||||
/// Everything one probe run learned about an image or a running container.
|
||||
@@ -226,9 +282,15 @@ pub fn manifest_script() -> String {
|
||||
.join(" ");
|
||||
let roots = MANIFEST_ROOTS
|
||||
.iter()
|
||||
.filter(|p| !SUMMARISED_ROOTS.contains(p))
|
||||
.map(|p| shell_single_quote(p))
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ");
|
||||
let summarised = SUMMARISED_ROOTS
|
||||
.iter()
|
||||
.map(|r| summarised_walk(r))
|
||||
.collect::<Vec<_>>()
|
||||
.join("\necho '###PATHS'\n");
|
||||
// The dpkg grep is anchored to the manifest roots so the section stays a
|
||||
// few hundred kB instead of the ~40 MB a full ownership dump would be.
|
||||
let dpkg_filter = MANIFEST_ROOTS
|
||||
@@ -240,6 +302,7 @@ pub fn manifest_script() -> String {
|
||||
r#"
|
||||
echo '###PATHS'
|
||||
find {roots} -xdev -printf '%y\t%s\t%p\n' 2>/dev/null
|
||||
{summarised}
|
||||
echo '###DPKG'
|
||||
cat /var/lib/dpkg/info/*.list 2>/dev/null | grep -E '^/({dpkg_filter})(/|$)'
|
||||
echo '###APT'
|
||||
@@ -260,6 +323,26 @@ exit 0
|
||||
)
|
||||
}
|
||||
|
||||
/// The `###PATHS` lines for one [`SUMMARISED_ROOTS`] entry, followed by a
|
||||
/// `###SUMMARY` section of `du -sb` lines (`bytes<TAB>path`) for its summaries.
|
||||
///
|
||||
/// The summary entry itself comes from `find`, not `du`: the directory's
|
||||
/// presence must never depend on the size measurement succeeding, or a `du`
|
||||
/// failure would make a whole subtree vanish from the migration. A missing size
|
||||
/// costs only the disk estimate. Empty directories at the boundary are printed
|
||||
/// as ordinary `d` entries, so the empty-tree filter still sees them as empty.
|
||||
fn summarised_walk(root: &str) -> String {
|
||||
let q = shell_single_quote(root);
|
||||
let boundary = shell_single_quote(&format!("{}{}", root, "/*".repeat(SUMMARY_DEPTH)));
|
||||
format!(
|
||||
"find {q} -xdev -maxdepth {d} \\( -type d -path {boundary} ! -empty -printf '{k}\\t0\\t%p\\n' \\) -o -printf '%y\\t%s\\t%p\\n' 2>/dev/null\n\
|
||||
echo '###SUMMARY'\n\
|
||||
find {q} -xdev -mindepth {d} -maxdepth {d} -type d ! -empty -exec du -sbx {{}} + 2>/dev/null",
|
||||
d = SUMMARY_DEPTH,
|
||||
k = SUMMARY_KIND,
|
||||
)
|
||||
}
|
||||
|
||||
/// Parse the output of [`manifest_script`].
|
||||
///
|
||||
/// Unknown sections and malformed lines are skipped rather than failing: the
|
||||
@@ -268,11 +351,15 @@ exit 0
|
||||
pub fn parse_manifest(raw: &str) -> Manifest {
|
||||
let mut m = Manifest::default();
|
||||
let mut section = "";
|
||||
// `du -sb` lines for summary entries, applied once everything has been
|
||||
// read so the two halves cannot depend on output order.
|
||||
let mut summary_sizes: HashMap<String, u64> = HashMap::new();
|
||||
for line in raw.lines() {
|
||||
let line = line.strip_suffix('\r').unwrap_or(line);
|
||||
if let Some(name) = line.strip_prefix("###") {
|
||||
section = match name {
|
||||
"PATHS" | "DPKG" | "APT" | "NPM" | "FEATURES" | "ETC" | "PKGVER" | "END" => name,
|
||||
"PATHS" | "SUMMARY" | "DPKG" | "APT" | "NPM" | "FEATURES" | "ETC" | "PKGVER"
|
||||
| "END" => name,
|
||||
_ => "",
|
||||
};
|
||||
continue;
|
||||
@@ -286,6 +373,13 @@ pub fn parse_manifest(raw: &str) -> Manifest {
|
||||
m.paths.push(entry);
|
||||
}
|
||||
}
|
||||
"SUMMARY" => {
|
||||
if let Some((bytes, path)) = line.split_once('\t') {
|
||||
if let Ok(bytes) = bytes.trim().parse::<u64>() {
|
||||
summary_sizes.insert(path.to_string(), bytes);
|
||||
}
|
||||
}
|
||||
}
|
||||
"DPKG" => {
|
||||
m.dpkg_owned.insert(line.to_string());
|
||||
}
|
||||
@@ -313,6 +407,11 @@ pub fn parse_manifest(raw: &str) -> Manifest {
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
for entry in m.paths.iter_mut().filter(|e| e.is_summary()) {
|
||||
if let Some(bytes) = summary_sizes.get(&entry.path) {
|
||||
entry.size = *bytes;
|
||||
}
|
||||
}
|
||||
m
|
||||
}
|
||||
|
||||
@@ -416,7 +515,8 @@ pub fn prune_to_roots(paths: &BTreeSet<String>) -> Vec<String> {
|
||||
/// * it lives under a [`COPY_ROOTS`] entry,
|
||||
/// * it is not under a [`COPY_EXCLUSIONS`] entry or a bind-mount target,
|
||||
/// * neither image's dpkg database owns it,
|
||||
/// * the current base image does not already have it.
|
||||
/// * the current base image does not already have it (a [`SUMMARY_KIND`]
|
||||
/// entry is exempt from this one — see the comment at the check).
|
||||
///
|
||||
/// The result is then pruned to subtree roots. An empty result means the copy
|
||||
/// step is skipped entirely.
|
||||
@@ -446,7 +546,12 @@ pub fn compute_verbatim_paths(
|
||||
if from.dpkg_owned.contains(p) || base.dpkg_owned.contains(p) {
|
||||
continue;
|
||||
}
|
||||
if base_paths.contains(p) {
|
||||
// Not for a summary: "the base already has this path" is a per-path
|
||||
// judgment, and a summary hides which paths are inside it. Skipping one
|
||||
// would drop every user file under any subtree the base happens to ship
|
||||
// too. Copying it is safe — the restore's `--skip-old-files` keeps the
|
||||
// base's copy of anything both have, which is the per-file answer.
|
||||
if base_paths.contains(p) && !entry.is_summary() {
|
||||
continue;
|
||||
}
|
||||
candidates.insert(entry.path.clone());
|
||||
@@ -1064,6 +1169,26 @@ pub async fn manifest_from_image(image: &str) -> Result<Manifest, String> {
|
||||
Ok(parse_manifest(&out.stdout))
|
||||
}
|
||||
|
||||
/// Turn a manifest exec's failure into something the user can act on.
|
||||
///
|
||||
/// The one case worth naming is the output ceiling: what reached the banner was
|
||||
/// "OUTPUT_LIMIT: Command output exceeded 8388608 bytes and was abandoned",
|
||||
/// true of a buffer and no help about a container. Same idea as
|
||||
/// `file_commands::describe_listing_failure`.
|
||||
pub fn describe_manifest_failure(error: String) -> String {
|
||||
if error.starts_with(super::exec::OUTPUT_LIMIT_MARKER) {
|
||||
return format!(
|
||||
"This container has too many files for Triple-C to inspect for an update (its \
|
||||
file listing passed {} MiB). The usual cause is a very large tree installed \
|
||||
under /usr/local, /opt, /srv or /var/lib, or a huge number of files directly \
|
||||
in a top-level /workspace folder. Removing or moving it into a mounted \
|
||||
project folder will let the check run.",
|
||||
MANIFEST_OUTPUT_LIMIT / (1024 * 1024)
|
||||
);
|
||||
}
|
||||
error
|
||||
}
|
||||
|
||||
/// Capture a [`Manifest`] from a *running* container.
|
||||
///
|
||||
/// Preferred over [`manifest_from_image`] for the "from" side whenever the
|
||||
@@ -1071,13 +1196,15 @@ pub async fn manifest_from_image(image: &str) -> Result<Manifest, String> {
|
||||
/// installed since the last commit, and a verbatim set computed from a stale
|
||||
/// manifest would silently fail to carry that work across.
|
||||
pub async fn manifest_from_container(container_id: &str) -> Result<Manifest, String> {
|
||||
let (out, code) = super::exec::exec_oneshot_as(
|
||||
let (out, code) = super::exec::exec_oneshot_as_limited(
|
||||
container_id,
|
||||
"root",
|
||||
vec!["/bin/sh".to_string(), "-c".to_string(), manifest_script()],
|
||||
Vec::new(),
|
||||
MANIFEST_OUTPUT_LIMIT,
|
||||
)
|
||||
.await?;
|
||||
.await
|
||||
.map_err(describe_manifest_failure)?;
|
||||
if !out.contains("###END") {
|
||||
return Err(format!(
|
||||
"Probe of the running container did not complete (exit {})",
|
||||
@@ -2220,6 +2347,234 @@ mod tests {
|
||||
assert_eq!(shell_single_quote("/opt/a'b"), r#"'/opt/a'\''b'"#);
|
||||
}
|
||||
|
||||
// ── /workspace summarisation (the 8 MiB overflow) ───────────────────────
|
||||
|
||||
#[test]
|
||||
fn a_summarised_subtree_takes_its_size_from_the_summary_section() {
|
||||
let raw = "###PATHS\n\
|
||||
d\t4096\t/workspace/wt\n\
|
||||
f\t10\t/workspace/wt/README.md\n\
|
||||
S\t0\t/workspace/wt/node_modules\n\
|
||||
###SUMMARY\n\
|
||||
123456\t/workspace/wt/node_modules\n\
|
||||
###DPKG\n\
|
||||
###END\n";
|
||||
let m = parse_manifest(raw);
|
||||
assert_eq!(m.paths.len(), 3);
|
||||
let s = &m.paths[2];
|
||||
assert!(s.is_summary());
|
||||
assert!(!s.is_dir(), "a summary stands for content, not an empty directory");
|
||||
assert_eq!(s.size, 123456);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_summary_with_no_size_line_is_kept_rather_than_dropped() {
|
||||
// `du` failing must cost the disk estimate, never the directory.
|
||||
let raw = "###PATHS\nS\t0\t/workspace/wt/vendor\n###SUMMARY\n###END\n";
|
||||
let m = parse_manifest(raw);
|
||||
assert_eq!(m.paths.len(), 1);
|
||||
assert_eq!(m.paths[0].path, "/workspace/wt/vendor");
|
||||
assert_eq!(m.paths[0].size, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_summarised_subtree_is_copied_whole_and_counted_in_the_payload() {
|
||||
// `/workspace/node_modules` holds nothing at its own level but deeper
|
||||
// summaries: it must not look like an empty tree and be dropped.
|
||||
let from = manifest(
|
||||
&[
|
||||
('d', 4096, "/workspace/node_modules"),
|
||||
('S', 50_000, "/workspace/node_modules/typescript"),
|
||||
('S', 7_000, "/workspace/node_modules/.bin"),
|
||||
('d', 4096, "/workspace/wt"),
|
||||
('f', 10, "/workspace/wt/README.md"),
|
||||
('S', 1_000, "/workspace/wt/src"),
|
||||
],
|
||||
&[],
|
||||
&[],
|
||||
);
|
||||
let verbatim = compute_verbatim_paths(&from, &Manifest::default(), &[]);
|
||||
assert_eq!(
|
||||
verbatim,
|
||||
vec!["/workspace/node_modules".to_string(), "/workspace/wt".to_string()]
|
||||
);
|
||||
assert_eq!(verbatim_payload_bytes(&from, &verbatim), 58_010);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_summary_the_base_also_has_is_still_copied() {
|
||||
// The presence-in-base filter is a per-path judgment, and a summary
|
||||
// hides which paths are inside it. Dropping it would lose the user's
|
||||
// files under any subtree the base happens to ship; copying it is safe
|
||||
// because the restore's `--skip-old-files` never clobbers the base.
|
||||
let from = manifest(
|
||||
&[
|
||||
('d', 4096, "/workspace/tools"),
|
||||
('S', 900, "/workspace/tools/lib"),
|
||||
],
|
||||
&[],
|
||||
&[],
|
||||
);
|
||||
let base = manifest(
|
||||
&[
|
||||
('d', 4096, "/workspace/tools"),
|
||||
('S', 100, "/workspace/tools/lib"),
|
||||
],
|
||||
&[],
|
||||
&[],
|
||||
);
|
||||
assert_eq!(
|
||||
compute_verbatim_paths(&from, &base, &[]),
|
||||
vec!["/workspace/tools/lib".to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_workspace_is_walked_to_a_bounded_depth_and_summarised_below_it() {
|
||||
let s = manifest_script();
|
||||
assert!(
|
||||
!s.contains("/workspace' -xdev -printf") && !s.contains("'/workspace' '/var/lib'"),
|
||||
"/workspace must not be walked file-by-file:\n{}",
|
||||
s
|
||||
);
|
||||
assert!(s.contains("###SUMMARY"));
|
||||
assert!(s.contains("-maxdepth 2"));
|
||||
}
|
||||
|
||||
/// Runs the real walk against a scratch tree shaped like the container that
|
||||
/// overflowed: a few large dependency trees under a writable-layer
|
||||
/// directory. GNU `find`/`du` are what the image has; the Linux CI runner
|
||||
/// has the same.
|
||||
#[cfg(target_os = "linux")]
|
||||
#[test]
|
||||
fn the_summarised_walk_stays_small_on_a_tree_of_dependencies() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let root = dir.path().join("workspace");
|
||||
for wt in 0..4 {
|
||||
let nm = root.join(format!("wt{}", wt)).join("node_modules");
|
||||
for pkg in 0..50 {
|
||||
let p = nm.join(format!("pkg{}", pkg)).join("lib");
|
||||
std::fs::create_dir_all(&p).unwrap();
|
||||
for f in 0..10 {
|
||||
std::fs::write(p.join(format!("f{}.js", f)), b"0123456789").unwrap();
|
||||
}
|
||||
}
|
||||
std::fs::write(root.join(format!("wt{}", wt)).join("README.md"), b"hi").unwrap();
|
||||
}
|
||||
std::fs::create_dir_all(root.join("wt0").join("empty")).unwrap();
|
||||
std::fs::write(root.join("loose.txt"), b"x").unwrap();
|
||||
let root_s = root.to_string_lossy().to_string();
|
||||
|
||||
let script = format!(
|
||||
"echo '###PATHS'\n{}\n###END",
|
||||
summarised_walk(&root_s).replace("###END", "")
|
||||
);
|
||||
let out = std::process::Command::new("/bin/sh")
|
||||
.arg("-c")
|
||||
.arg(&script)
|
||||
.output()
|
||||
.unwrap();
|
||||
let text = String::from_utf8_lossy(&out.stdout).to_string();
|
||||
let m = parse_manifest(&text);
|
||||
|
||||
// 2,000 files below; one line per first- and second-level entry above.
|
||||
assert!(m.paths.len() < 20, "walk was not summarised: {}", text);
|
||||
let nm0 = format!("{}/wt0/node_modules", root_s);
|
||||
let entry = m.paths.iter().find(|e| e.path == nm0).expect("summary present");
|
||||
assert!(entry.is_summary());
|
||||
assert!(entry.size >= 500 * 10, "du size missing: {:?}", entry);
|
||||
// An empty second-level directory is reported as one, not as content.
|
||||
let empty = format!("{}/wt0/empty", root_s);
|
||||
assert!(m.paths.iter().any(|e| e.path == empty && e.is_dir()));
|
||||
assert!(m.paths.iter().any(|e| e.path == format!("{}/loose.txt", root_s)));
|
||||
}
|
||||
|
||||
/// The overflow end to end, against a scratch container off the real base:
|
||||
/// 80k files in a writable-layer `/workspace` tree (past 8 MiB unsummarised)
|
||||
/// beside a bind-mounted project folder.
|
||||
///
|
||||
/// ```text
|
||||
/// cargo test -- --ignored --nocapture large_workspace
|
||||
/// ```
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
#[ignore = "needs a Docker daemon; creates and removes a throwaway container"]
|
||||
async fn a_large_workspace_is_probed_without_overflowing_and_still_copied() {
|
||||
fn docker_cli(args: &[&str]) -> String {
|
||||
let out = std::process::Command::new("docker")
|
||||
.args(args)
|
||||
.output()
|
||||
.expect("docker CLI");
|
||||
assert!(
|
||||
out.status.success(),
|
||||
"docker {:?} failed: {}",
|
||||
args,
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
String::from_utf8_lossy(&out.stdout).trim().to_string()
|
||||
}
|
||||
let image = std::env::var("TRIPLE_C_TEST_IMAGE")
|
||||
.unwrap_or_else(|_| "ghcr.io/shadowdao/triple-c-sandbox:latest".to_string());
|
||||
let host = tempfile::tempdir().unwrap();
|
||||
for i in 0..50 {
|
||||
std::fs::write(host.path().join(format!("host-{}.txt", i)), b"h").unwrap();
|
||||
}
|
||||
let mount = format!("{}:/workspace/proj", host.path().display());
|
||||
let id = docker_cli(&[
|
||||
"run", "-d", "--label", "triple-c.managed=true",
|
||||
"-v", &mount,
|
||||
"--entrypoint", "/bin/sh",
|
||||
&image, "-c", "sleep 300",
|
||||
]);
|
||||
// 80k files with long names, three levels down: ~9.2 MB as a flat walk.
|
||||
let populate = "for w in 0 1 2 3 4 5 6 7; do for p in $(seq 1 100); do \
|
||||
d=/workspace/worktree-number-$w/node_modules/package-with-a-long-name-$p/lib; \
|
||||
mkdir -p $d; cd $d && seq -f 'module-file-with-a-long-name-%g.js' 1 100 | xargs touch; \
|
||||
done; done; echo x > /workspace/worktree-number-0/README.md; \
|
||||
head -c 4096 /dev/zero > /workspace/worktree-number-0/node_modules/real.js";
|
||||
docker_cli(&["exec", "-u", "root", &id, "sh", "-c", populate]);
|
||||
|
||||
let result = manifest_from_container(&id).await;
|
||||
let _ = std::process::Command::new("docker").args(["rm", "-f", &id]).output();
|
||||
|
||||
let m = result.expect("the probe must not overflow");
|
||||
let ws: Vec<_> = m.paths.iter().filter(|e| is_under(&e.path, "/workspace")).collect();
|
||||
assert!(ws.len() < 100, "/workspace was walked file by file: {} entries", ws.len());
|
||||
assert!(
|
||||
!ws.iter().any(|e| is_under(&e.path, "/workspace/proj") && e.path != "/workspace/proj"),
|
||||
"the walk crossed into the bind mount"
|
||||
);
|
||||
let nm = m
|
||||
.paths
|
||||
.iter()
|
||||
.find(|e| e.path == "/workspace/worktree-number-0/node_modules")
|
||||
.expect("summary entry");
|
||||
assert!(nm.is_summary() && nm.size > 0, "{:?}", nm);
|
||||
|
||||
let excl = bind_mount_exclusions(&[pp("proj")]);
|
||||
let verbatim = compute_verbatim_paths(&m, &Manifest::default(), &excl);
|
||||
let ws_verbatim: Vec<_> =
|
||||
verbatim.iter().filter(|p| is_under(p, "/workspace")).cloned().collect();
|
||||
assert_eq!(
|
||||
ws_verbatim,
|
||||
(0..8).map(|w| format!("/workspace/worktree-number-{}", w)).collect::<Vec<_>>()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_overflowing_probe_says_what_to_do_rather_than_quoting_a_buffer() {
|
||||
let raw = format!(
|
||||
"{}: Command output exceeded 33554432 bytes and was abandoned",
|
||||
crate::docker::exec::OUTPUT_LIMIT_MARKER
|
||||
);
|
||||
let msg = describe_manifest_failure(raw);
|
||||
assert!(!msg.contains(crate::docker::exec::OUTPUT_LIMIT_MARKER), "{}", msg);
|
||||
assert!(!msg.contains("33554432"), "{}", msg);
|
||||
assert!(msg.contains("/usr/local"), "{}", msg);
|
||||
// Anything else passes through untouched.
|
||||
assert_eq!(describe_manifest_failure("boom".into()), "boom");
|
||||
}
|
||||
|
||||
|
||||
// ── Stale rollback pins (A5) ─────────────────────────────────────────────
|
||||
|
||||
|
||||
Reference in new issue
Block a user