Stop the migration manifest overflowing on a large /workspace #70

Merged

jknapp merged 2 commits from fix/manifest-probe-overflow into main 2026-11-03 00:31:39 +00:00

Conversation 0 Commits 2 Files Changed 4

+571 -24

2 Commits

Author	SHA1	Message	Date
shadowdaoandClaude Opus 5.5	89908b3b3d	Address review of the /workspace manifest summarisation ... - One find pass piped through awk replaces find + du: listing and sizes can no longer disagree under concurrent writes, and /workspace is walked once. A summary's size is now the sum of %s over its non-directories, exactly what verbatim_payload_bytes summed per file, so the disk gate neither under-counts (no du-missing -> 0) nor over-counts dir inodes. - A depth-2 dir with no file at any depth is reported as a plain `d`, so an empty scaffold (a/b/c with no files) is dropped as before instead of becoming a verbatim root. - The pipeline prints ###SUMMARISED-OK only when awk succeeds; check_manifest_complete refuses a manifest without it in both manifest_from_container and manifest_from_image, so a failed walk can never read as an empty /workspace. - The presence-in-base exemption for summaries now logs a warning if the base ever ships a summarised path, and an ignored live test pins that the base ships nothing under /workspace. - plain_walk() emits nothing for no roots instead of `find -xdev` (= /). - Overflow message names /var/www and /etc too. - exec_oneshot_limited takes a user and returns the exit code; the duplicate exec_oneshot_as_limited is gone. MAX_ONESHOT_OUTPUT doc fixed. - The Linux walk test now echoes a following section and asserts it parses, asserts exact sizes and the empty-scaffold case. Measured on the container that overflowed: whole manifest 0.99 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 16:07:13 -08:00
shadowdaoandClaude Opus 5.5	209ed00fdd	Stop the migration manifest overflowing on a large /workspace ... get_container_staleness failed on every start/stop of a real container with "OUTPUT_LIMIT: Command output exceeded 8388608 bytes": the manifest's flat `find /workspace` walk alone was 8.6 MB (94k entries of worktrees, node_modules, vendored trees in the writable layer). Pruning node_modules and .git by name was measured on that container and only reached 6.3 MB. - Walk /workspace two levels deep; each non-empty directory at depth 2 is one `S` entry, sized from a new ###SUMMARY section (`du -sbx`). The entry comes from find, so a failed du costs the size, never the dir. Measured on the same container: 8.6 MB -> ~25 kB. - compute_verbatim_paths: summaries count as content (not an empty tree) and are exempt from presence-in-base, since a summary hides which paths are inside; restore's --skip-old-files keeps the per-file guarantee. The verbatim set is unchanged: it already pruned to /workspace/<x>. - manifest_from_container gets its own 32 MiB budget via a new exec_oneshot_as_limited, and an OUTPUT_LIMIT is reported as an actionable sentence (describe_manifest_failure) instead of the buffer message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-02 15:53:26 -08:00