extract: verify always runs, failures remove the extract, walk fails closed

Security review of 52b1e65:
1. (HIGH) tarfile's stream could enumerate differently from GNU tar. Now
   extract.sh captures tar's exit code (`|| TAR_RC=$?`), ALWAYS runs the
   post-extract verify on whatever tar wrote, and any failure (tar,
   verify, either pre-extract scan) removes $DEST before exiting non-zero.
   The classifier also refuses (exit 3) on any stream exception and when
   its member count differs from GNU tar's own escape-quoted listing.
   Measured: a lone zero block mid-archive stops BOTH readers at the same
   member (no -i is used), so this is a guard, not a known differential.
2. (MEDIUM) os.walk swallowed unreadable directories. verify now passes
   onerror (recorded => fail) and refuses any directory that is not
   readable+searchable. 52b1e65 exited 0 on the unreadable fixture.
3. (MEDIUM) a refused tree stayed in the bind-mounted output. fail_extract
   opens up 0000 dirs (find -P -type d, never follows links), then
   rm -rf -- "$DEST".

Fixtures: cpmove-bad-{unreadable,tarfail,truncated}; a fake tar that lists
one extra member proves the count check. 14 checks fail on 52b1e65.
Docker: unreadable / tarfail runs end with status failed, failed_stage
extract and no extract-work left in /host/sanitized/<id>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
shadowdaoandClaude Opus 5.5 committed 2026-10-20 09:18:34 -07:00
1 parent 52b1e656c3
commit d4a63fba0a
4 files changed
+161 -33

No files matched your search

+36 -8
View File
@@ -36,6 +36,24 @@ log() { printf '[%s] extract: %s\n' "$(ts)" "$*"; }
[[ -f "$TARBALL" ]] || { log "tarball not found: $TARBALL"; exit 2; }
mkdir -p "$DEST"
# Any failure after this point removes the (partial) extract before exiting:
# it is the container's own scratch dir inside the bind-mounted output, and a
# half-extracted tree, or one the link check refused, must not be left
# behind for anything to read. Directories the archive made unreadable are
# opened up first (find -P never follows links; -type d skips them) so rm
# can descend.
fail_extract() {
local code="$1"; shift
log "$*"
cd /
if [[ -n "$DEST" && "$DEST" != "/" && -d "$DEST" ]]; then
find -P "$DEST" -type d ! -perm -u=rwx -exec chmod u+rwx {} \; 2>/dev/null || true
rm -rf -- "$DEST" || log "WARN: could not remove the failed extract $DEST"
log "removed the failed extract $DEST"
fi
exit "$code"
}
# --- pre-extract symlink scan ---------------------------------------------
log "scanning tarball for dangerous symlinks (cpmove vector check)"
@@ -44,9 +62,8 @@ if ! php "$SCRIPT_DIR/lib/scan-symlinks.php" \
--tarball "$TARBALL" \
--username "$USERNAME" \
--report "$SYMLINK_REPORT"; then
log "scan-symlinks.php exited with usage/IO error; aborting (this is not a sanitize-able state)"
cat "$SYMLINK_REPORT" >&2 || true
exit 3
fail_extract 3 "scan-symlinks.php exited with usage/IO error; aborting (this is not a sanitize-able state)"
fi
# --- compute exclude list from dangerous findings -------------------------
@@ -113,7 +130,7 @@ if [[ "$IMPORT_INCLUDE_MAIL" == "true" ]]; then
# classifier and by tar (control characters, non-canonical paths).
# scan-symlinks.php alone only strips DANGEROUS absolute targets.
MAIL_LINKS=$(python3 -I "$SCRIPT_DIR/lib/mail-tree-links.py" scan "$TARBALL" "$EXCLUDES_FILE" "$ACTIONS_OUT") \
|| { log "mail-tree link scan refused the archive (see above); aborting"; exit 3; }
|| fail_extract 3 "mail-tree link scan refused the archive (see above); aborting"
log "stripped $MAIL_LINKS symlink / boundary-crossing hard-link entr(y/ies) in the mail tree"
else
{
@@ -148,15 +165,26 @@ log "extracting with hardened tar flags into $DEST"
# match is effectively a literal-path skip.
# --absolute-names is NOT used — leading / in a member name is stripped.
cd "$DEST"
# tar's exit code is captured, not fatal on the spot: the mail-tree verify
# below must run on whatever tar did write, and a failed tar must still get
# its partial tree removed.
TAR_RC=0
tar --no-same-owner --no-same-permissions --no-overwrite-dir \
--exclude-from="$EXCLUDES_FILE" \
$TAR_FLAGS "$TARBALL"
$TAR_FLAGS "$TARBALL" || TAR_RC=$?
VERIFY_RC=0
if [[ "$IMPORT_INCLUDE_MAIL" == "true" ]]; then
# The authority: whatever the classifier above decided, no symlink and
# no hard link leaving the mail tree may exist in the extract.
python3 -I "$SCRIPT_DIR/lib/mail-tree-links.py" verify "$DEST" \
|| { log "extracted mail tree contains links (see above); aborting"; exit 4; }
# The authority: whatever the classifier above decided, no symlink, no
# hard link leaving the mail tree, and no uninspectable directory may
# exist in the extract. Always runs, even after a failed tar.
python3 -I "$SCRIPT_DIR/lib/mail-tree-links.py" verify "$DEST" || VERIFY_RC=$?
fi
if [[ $TAR_RC -ne 0 ]]; then
fail_extract "$TAR_RC" "tar failed (rc=$TAR_RC)$( [[ $VERIFY_RC -ne 0 ]] && echo "; mail-tree verify also failed (rc=$VERIFY_RC)" ); aborting"
fi
if [[ $VERIFY_RC -ne 0 ]]; then
fail_extract 4 "extracted mail tree failed the link/readability check (see above); aborting"
fi
log "extracted OK ($(find "$DEST" -type f | wc -l) files; $DANGEROUS_COUNT symlinks stripped)"