fix(certs): stop the test suite bricking the container; make the pairing check real

Gate findings on fix/cert-write-safety. One blocker, one latent truncation
path, a false premise in a load-bearing comment, and a set of tests that were
passing without testing anything.

THE BLOCKER - scripts/test-cert-scripts.py bricked the production container
------------------------------------------------------------------------
The openssl-availability tests build a stripped PATH out of SYMLINKS to real
system binaries (cat, grep, mktemp, mv, cp, rm, mkdir, basename, dirname, find,
date, chmod). _cleanup_tmp() then walked the temp tree calling os.chmod(p,
0o600) - and os.chmod FOLLOWS SYMLINKS. Run once as root in the real image,
which is where this file ships (COPY scripts /haproxy/scripts) and where an
operator would most plausibly run it after deploying a cert fix, it stripped
the exec bit off twelve core binaries INCLUDING chmod itself, so it could not
be undone from inside the container:

    /haproxy/scripts/cert-publish-lib.sh: line 109: /usr/bin/grep: Permission denied
    bash: /usr/bin/chmod: Permission denied

Certificate publishing stayed dead until the container was recreated. It was
invisible on a workstation because an unprivileged chmod of a root-owned file
fails EPERM straight into `except OSError: pass` - which is also why the
advertised "32 tests pass" was only ever true off-container. In the image the
shipped file measured FAILED (failures=22, skipped=1). Cleanup now skips
symlinks; the suites are green in the image and the binaries survive.

S1 - the shell half had no same-filesystem guard
------------------------------------------------
The header claimed a cross-device mv would "FAIL LOUDLY and leave the live pem
alone". GNU mv does the opposite: across filesystems it copies, so it opens and
truncates the DESTINATION and only then discovers it cannot finish - measured
as a 204800-byte partial live.pem, sentinel gone, before mv reported ENOSPC.
cert_publish() now compares stat -Lc %d of the staging and certs dirs before
writing anything, mirroring the st_dev check the Python half already had, and
the comment says what mv actually does. Latent today (both dirs share a device
on all five hosts) but both are env-overridable.

openssl is present - correct the premise, make the check mandatory
------------------------------------------------------------------
Both halves justified a fail-open with "the image does not necessarily install
the openssl CLI". It does: openssl 3.5.6 in the running container, pulled in by
ca-certificates which certbot needs, and generate_self_signed_cert() already
shells to `openssl req` with check=True during setup. The 'unavailable' branch
never fired, so the pairing check has always run - and that, not the stated
reasoning, is what made the fail-open harmless. Structural validation alone is
weak: a bundle of EMPTY pem blocks passes every structural rule and is caught
only by openssl. The check is now mandatory in both halves and a missing binary
is a loud refusal. No `cryptography` fallback: the app runs on
/usr/local/bin/python3 (3.12) where it is not importable - it belongs to
Debian's /usr/bin/python3 - and reaching for that would be a second unverified
premise.

Smaller items
-------------
* cert_bundle_valid() read the file six times; a concurrent swap between two of
  them made openssl x509 and openssl pkey judge different files and log a bogus
  "private key does not match the certificate" into the monitored error log. It
  now reads one snapshot and feeds openssl from it on stdin.
* except OSError -> except (OSError, UnicodeDecodeError): a BINARY-corrupt live
  pem made backup_existing_pem() raise out of publish_pem_bundle() entirely, so
  the republish that would have healed the host was the one thing that could
  not run. The shell half recovers fine.
* stat -c %a on a symlinked live pem reports the LINK's 0777 and produced a
  world-writable private key in the crt directory; now stat -Lc.
* The staging reaper's '*.??????' glob matched mktemp names but not the Python
  side's '<name>.<random>.tmp', so those leaked forever. Matches both now.
* renew-certificates.sh and sync-certificates.sh exited 0 even when every
  domain failed to publish, so "0 updated, 12 failed" looked identical to a
  clean run to cron, to host-renew-certificates.sh (which branches on it) and
  to monitoring - a host could silently stop publishing renewals until the
  certificates expired. They now exit 1 if any domain failed, still after
  publishing the ones that worked.

Test-quality
------------
Four TestCertPublishLibrary tests passed with cert-publish-lib.sh DELETED -
they asserted only rc != 0, and `command not found` is 127. All four now assert
the rejection REASON via assert_rejected(), and setUp() fails if the library is
missing. test_missing_openssl_still_rejects... is replaced by
test_empty_pem_blocks_are_rejected, which pins the case that makes the pairing
check necessary.

Also: the staging-containment test used startswith(certs + os.sep), so
cert_staging_dir() returning the crt directory ITSELF - the exact hazard -
still passed; test_successful_renewal_still_publishes built its "renewed" cert
with a no-op .replace() and could not tell a renewal that published nothing;
test_no_temp_file_survives_a_failed_publish failed before the staging dir
existed and asserted [] == []; FIX_ONLY was skipUnless(hasattr(hm,
'publish_pem_bundle')), so renaming that function turned 10 of 17 tests into
skips while the run still printed OK. Each is fixed and each fix is
mutation-proved: the mutation that the old assertion waved through now fails.
File mode is pinned in both suites (it was pinned nowhere), and the rename
failure is injected with a stub mv instead of chmod 0500, which root ignored -
so that test no longer skips itself precisely where it matters.

Verification
------------
IN THE BUILT IMAGE, as root (the acceptance bar):
    scripts/test-cert-scripts.py        38 tests, OK, 0 skipped
    scripts/test-cert-write-safety.py   22 tests, OK, 0 skipped
    scripts/test-config-rollback.py     17 tests, OK   (neighbour, unchanged)
Workstation: 38/38 OK for the shell suite; the Python suite needs flask.
Against the pre-fix tree (main): shell 38 failures; python failures=4, errors=1,
skipped=15 - the FIX_ONLY skips are the 14 fix-only tests plus the API guard.
The old python suite against the pre-fix tree measures skipped=10, confirming
the gate's count.
32 mutation checks, all behaving as intended: every fix breaks a test when
reverted, and every rewritten test fails under the mutation its predecessor
passed. py_compile clean, bash -n clean, shellcheck clean, no new pyflakes
warnings (same 4 pre-existing).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

haproxy_manager.py

@@ -373,8 +373,9 @@ def cert_backup_dir():
 
 # ENCRYPTED PRIVATE KEY is deliberately absent: HAProxy cannot use a
 # passphrase-protected key from a crt file, so a bundle containing one is not
-# publishable. Accepting it here would let one through on a host without the
+# publishable, and the structural layer is the only layer that names the
-# openssl CLI, where the pairing check (which would also reject it) is skipped.
+# problem ("no complete private key block") rather than reporting it as an
+# unreadable key.
 _PEM_KEY_LABELS = ('PRIVATE KEY', 'RSA PRIVATE KEY', 'EC PRIVATE KEY')
 
 
@@ -406,6 +407,11 @@ def validate_pem_structure(text):
     lack of a tool. It catches every failure mode the truncation bug produced:
     empty file, certificate without a key, key without a certificate, and a
     block cut off mid-write.
+
+    It is NOT sufficient on its own, which is why _openssl_pairing_status() is
+    mandatory rather than best-effort: a bundle of EMPTY pem blocks (a BEGIN
+    line immediately followed by its END line, no base64 between them) passes
+    every check here and is rejected only by openssl.
     """
     if not text.strip():
         return False, 'bundle is empty'
@@ -430,14 +436,34 @@ def _openssl_pairing_status(path):
     assembled bundle directly. Comparing the two public keys proves the pair.
 
     'unavailable' means the openssl BINARY is absent - a verdict about our
-    tooling, not about the bundle. The Dockerfile installs haproxy, certbot,
+    tooling, not about the bundle. validate_pem_bundle() treats it as a HARD
-    socat and curl but not the openssl CLI, so this is a real possibility.
+    FAILURE.
-    Callers treat it as a loud warning rather than a failure: the failure modes
+
-    this whole module exists to prevent (truncation, missing key, partial
+    That is a deliberate reversal. This docstring used to say "the Dockerfile
-    write) are fully covered by validate_pem_structure(), whereas refusing to
+    installs haproxy, certbot, socat and curl but not the openssl CLI, so this
-    publish whenever the checker is missing would stall renewals fleet-wide and
+    is a real possibility", and callers accepted the bundle on the structural
-    let certificates expire - a guaranteed outage traded for a hypothetical
+    checks alone. The premise is false: openssl 3.x IS in the image, as a
-    one. A mismatch, when we CAN check, is always fatal.
+    dependency of ca-certificates (which certbot requires), and
+    generate_self_signed_cert() below already runs `openssl req` with
+    check=True during first-run setup - so no container has ever reached a
+    publish without it. The 'unavailable' branch never fired, which means the
+    pairing check has in fact always run, and THAT is what made the fail-open
+    harmless - not the stated reasoning. Structural validation on its own is
+    weak: a bundle of empty pem blocks passes validate_pem_structure() and is
+    caught only here.
+
+    So an absent openssl now means the image is broken, and we say so and stop
+    instead of quietly downgrading to the weaker check. The cost is that a
+    hypothetical openssl-less image stops publishing renewals - but it does so
+    immediately and loudly, in the monitored error log, at the first renewal,
+    rather than 90 days later; and publishing an unverified bundle can take the
+    whole :443 bind, i.e. every site on the host, down at the next reload.
+
+    There is no python `cryptography` fallback on purpose: this process runs on
+    /usr/local/bin/python3 (the base image's 3.12), where cryptography is not
+    importable. It is installed for Debian's /usr/bin/python3 as a certbot
+    dependency, and reaching for that interpreter would be a second unverified
+    premise of exactly the kind this comment is correcting.
     """
     try:
         cert_pub = subprocess.run(
@@ -471,14 +497,17 @@ def _openssl_pairing_status(path):
 def validate_pem_bundle(path):
     """Validate a bundle file on disk. Returns (ok, message).
 
-    Mandatory structural validation plus a best-effort cryptographic pairing
+    Structural validation AND the cryptographic pairing check, both mandatory -
-    check - see _openssl_pairing_status() for what happens when openssl is
+    see _openssl_pairing_status() for why a missing openssl is a failure rather
-    missing (loud warning, structural verdict stands).
+    than a downgrade to structure-only.
     """
     try:
         with open(path, 'r') as fh:
             text = fh.read()
-    except OSError as e:
+    except (OSError, UnicodeDecodeError) as e:
+        # UnicodeDecodeError, not just OSError: a bundle corrupted into binary
+        # is unreadable as text but reads perfectly well as bytes, so `except
+        # OSError` let the decode error escape as an unhandled traceback.
         return False, f'cannot read assembled bundle: {e}'
 
     ok, msg = validate_pem_structure(text)
@@ -489,9 +518,13 @@ def validate_pem_bundle(path):
     if status == 'invalid':
         return False, pair_msg
     if status == 'unavailable':
-        logger.warning(
+        logger.error(
-            "Certificate key/leaf pairing check SKIPPED for %s (%s). The "
+            "Certificate key/leaf pairing check could not run for %s (%s) - "
-            "bundle passed structural validation only.", path, pair_msg)
+            "REFUSING to publish. openssl is required; structural validation "
+            "alone cannot tell a real bundle from empty pem blocks.",
+            path, pair_msg)
+        return False, (f'cert/key pairing check unavailable ({pair_msg}); '
+                       'refusing to publish on structural checks alone')
     return True, None
 
 
@@ -511,7 +544,14 @@ def backup_existing_pem(dest_path):
     try:
         with open(dest_path, 'r') as fh:
             ok, msg = validate_pem_structure(fh.read())
-    except OSError as e:
+    except (OSError, UnicodeDecodeError) as e:
+        # UnicodeDecodeError, not just OSError. A live pem corrupted into
+        # BINARY (the exact state a republish is meant to heal) raises
+        # UnicodeDecodeError here, and with only OSError caught it escaped all
+        # the way out of publish_pem_bundle() - so the one operation that would
+        # have put a working certificate back blew up on the way to taking a
+        # backup of the broken one. The shell half recovers from this fine;
+        # this is the only reason the Python half did not.
         ok, msg = False, str(e)
     backup_path = os.path.join(cert_backup_dir(), os.path.basename(dest_path))
     if not ok:
@@ -552,7 +592,9 @@ def publish_pem_bundle(dest_path, source_paths):
         try:
             with open(src, 'r') as fh:
                 data = fh.read()
-        except OSError as e:
+        except (OSError, UnicodeDecodeError) as e:
+            # Binary-corrupt source material must be a clean, reported refusal,
+            # not an unhandled UnicodeDecodeError out of the request handler.
             raise CertificatePublishError(f'cannot read {src}: {e}')
         if not data.strip():
             raise CertificatePublishError(f'source file is empty: {src}')

scripts/cert-publish-lib.sh

@@ -33,9 +33,21 @@
 #   staging: $(dirname $SSL_CERTS_DIR)/cert-staging   [$CERT_STAGING_DIR]
 #   backups: $(dirname $SSL_CERTS_DIR)/cert-backups   [$CERT_BACKUP_DIR]
 # Both siblings of the certs dir, so they are on the same filesystem and the
-# final mv is a rename(2) - atomic. If the mv ever fails (EXDEV because someone
+# final mv is a rename(2) - atomic. There is deliberately no "just write it
-# mounted the certs dir separately, permissions, ...) we FAIL LOUDLY and leave
+# directly into the certs dir" fallback path.
-# the live pem alone. There is deliberately no "just write it directly" path.
+#
+# The same-filesystem property is CHECKED, not assumed (see cert_publish step
+# (c)). An earlier revision of this comment claimed a cross-device mv would
+# "fail loudly and leave the live pem alone". It does not: GNU mv falls back to
+# copy-then-unlink across filesystems, so it OPENS THE DESTINATION FOR WRITING
+# and only then discovers it cannot finish - e.g. with a full destination
+# filesystem the live pem is already overwritten when mv reports failure. That
+# is precisely the truncation this library exists to prevent, so the device
+# numbers of the staging dir and the certs dir are compared with stat(1) before
+# anything is written, and a mismatch aborts the publish. Both directories are
+# env-overridable ($CERT_STAGING_DIR / $SSL_CERTS_DIR), so "they are siblings"
+# is not something the code can take on faith. This mirrors the explicit
+# st_dev check in publish_pem_bundle() on the Python side.
 
 # Logging: the callers define their own log_info/log_error. Only provide
 # fallbacks so this library is usable standalone (e.g. from a test or a shell).
@@ -72,23 +84,37 @@ cert_backup_dir() {
 #
 # Returns 0 if FILE is publishable as an HAProxy pem bundle.
 #
-# Layer 1 (MANDATORY, pure shell/grep, always available): structural checks.
+# Layer 1 (structure, pure shell/grep): covers the truncation / partial-write /
-#         A missing or failed structural check is a HARD FAIL. This layer is
+#         key-less failure modes this library exists to prevent.
-#         what actually covers the truncation / partial-write / key-less
+# Layer 2 (cryptographic pairing via the openssl CLI): covers what layer 1
-#         failure modes this library exists to prevent.
+#         cannot see. Structural checks are weak on their own - a bundle of
-# Layer 2 (BEST EFFORT): cryptographic pairing via the openssl CLI.
+#         EMPTY pem blocks ("-----BEGIN CERTIFICATE-----" immediately followed
-#         If openssl runs and says the cert and key do not match, that is a
+#         by "-----END CERTIFICATE-----") satisfies every grep below and is
-#         HARD FAIL. If the openssl BINARY IS ABSENT we log a loud warning and
+#         caught only by openssl.
-#         accept the bundle on the structural checks alone.
 #
-#         Rationale for not hard-failing on a missing checker: the container
+# BOTH LAYERS ARE MANDATORY. An absent openssl binary is a hard failure, not a
-#         image (see Dockerfile) installs haproxy, certbot and socat, but not
+# skip.
-#         necessarily the openssl CLI. Refusing to publish when the checker is
+#
-#         missing would stall every renewal fleet-wide and let certificates
+# The previous revision made layer 2 best-effort and justified it with "the
-#         expire - a guaranteed outage - which is strictly worse than the risk
+# container image installs haproxy, certbot and socat, but not necessarily the
-#         it prevents, since a mismatched pair can only arise from a
+# openssl CLI". That premise is false. openssl 3.x is present in the image: it
-#         mis-assembled source tree, whereas the truncation modes we are
+# is a dependency of ca-certificates, which certbot needs, and
-#         actually defending against are fully covered by layer 1.
+# generate_self_signed_cert() in haproxy_manager.py shells out to `openssl req`
+# with check=True during first-run setup, so a container that reached the point
+# of publishing a bundle has always had it. The "unavailable" branch therefore
+# never fired in production, which means the fail-open was safe only by
+# accident - and a comment that justifies a decision on a false premise is
+# worse than no comment, because the next person extends the reasoning.
+#
+# Making it mandatory does mean a hypothetical image without openssl stops
+# publishing renewals. That is the right trade: it fails immediately and
+# loudly, into the monitored error log, on the first renewal run, whereas
+# publishing an unpaired or empty-block bundle takes the whole :443 bind (i.e.
+# every site on the host) down at the next reload. There is deliberately no
+# python `cryptography` fallback: the app runs on /usr/local/bin/python3 (the
+# base image's 3.12), where cryptography is NOT importable - it is installed
+# for Debian's /usr/bin/python3 as a certbot dependency. Coding to a hardcoded
+# /usr/bin/python3 would just be a second unverified premise.
 cert_bundle_valid() {
     local file="$1"
 
@@ -100,55 +126,71 @@ cert_bundle_valid() {
         log_error "Certificate bundle $file does not exist (or is not a regular file)"
         return 1
     fi
-    if [ ! -s "$file" ]; then
+
+    # openssl is checked BEFORE any content check so a broken image is reported
+    # as a broken image rather than as a bad certificate.
+    if ! command -v openssl >/dev/null 2>&1; then
+        log_error "openssl binary not found - REFUSING to publish $file." \
+                  "The cert/key pairing check (openssl x509 -pubkey vs openssl pkey -pubout)" \
+                  "is mandatory; structural checks alone cannot tell a real bundle from" \
+                  "empty pem blocks. Install openssl in this image."
+        return 1
+    fi
+
+    # Read the file ONCE and run every check against that snapshot.
+    #
+    # This used to open $file six times (two [ ] tests, three greps, two
+    # openssl invocations). cert_bundle_valid() is called on the LIVE pem in
+    # cert_publish() step (d), where a concurrent publisher can replace it
+    # between two of those opens - each open then sees a different file. The
+    # observable symptom was a spurious "private key does not match the
+    # certificate" ERROR in the monitored error log for a pair that was fine:
+    # openssl x509 read the old bundle and openssl pkey the new one.
+    local content
+    if ! content="$(cat -- "$file" 2>/dev/null)"; then
+        log_error "Certificate bundle $file could not be read"
+        return 1
+    fi
+    if [ -z "$content" ]; then
         log_error "Certificate bundle $file is empty"
         return 1
     fi
 
     # --- layer 1: structure -------------------------------------------------
-    if ! grep -qF -- '-----BEGIN CERTIFICATE-----' "$file"; then
+    if ! grep -qF -- '-----BEGIN CERTIFICATE-----' <<< "$content"; then
         log_error "Certificate bundle $file contains no certificate block"
         return 1
     fi
-    if ! grep -qF -- '-----END CERTIFICATE-----' "$file"; then
+    if ! grep -qF -- '-----END CERTIFICATE-----' <<< "$content"; then
         log_error "Certificate bundle $file has an unterminated certificate block (truncated?)"
         return 1
     fi
 
     local key_begin key_end
-    key_begin="$(grep -m1 -oE -- '-----BEGIN (RSA |EC )?PRIVATE KEY-----' "$file")"
+    key_begin="$(grep -m1 -oE -- '-----BEGIN (RSA |EC )?PRIVATE KEY-----' <<< "$content")"
     if [ -z "$key_begin" ]; then
         log_error "Certificate bundle $file contains no private key block"
         return 1
     fi
     key_end="${key_begin/BEGIN/END}"
-    if ! grep -qF -- "$key_end" "$file"; then
+    if ! grep -qF -- "$key_end" <<< "$content"; then
         log_error "Certificate bundle $file has an unterminated private key block (truncated?)"
         return 1
     fi
 
     # --- layer 2: cert/key pairing ------------------------------------------
-    if ! command -v openssl >/dev/null 2>&1; then
+    # Fed from the same snapshot on stdin (openssl reads stdin when -in is
-        # Once per process: this fires per domain otherwise, and a renewal run
+    # omitted) rather than re-opening $file, so layer 2 judges exactly the
-        # walks every certificate on the host.
+    # bytes layer 1 judged. -passin pass: means an encrypted key fails fast
-        if [ -z "${_CERT_OPENSSL_WARNED:-}" ]; then
+    # instead of prompting - a passphrase prompt in a cron job is a hang, not
-            _CERT_OPENSSL_WARNED=1
+    # an error.
-            log_warn "openssl binary not found - SKIPPING the cert/key pairing check" \
-                     "(openssl x509 -pubkey vs openssl pkey -pubout);" \
-                     "certificate bundles are being published on structural checks alone"
-        fi
-        return 0
-    fi
-
     local cert_pub key_pub
-    # </dev/null on both: openssl pkey prompts for a passphrase on an encrypted
+    if ! cert_pub="$(openssl x509 -noout -pubkey 2>/dev/null <<< "$content")" \
-    # key, and a prompt in a cron job is a hang, not an error.
-    if ! cert_pub="$(openssl x509 -in "$file" -noout -pubkey 2>/dev/null </dev/null)" \
        || [ -z "$cert_pub" ]; then
         log_error "Certificate bundle $file: openssl could not read the certificate"
         return 1
     fi
-    if ! key_pub="$(openssl pkey -in "$file" -pubout -passin pass: 2>/dev/null </dev/null)" \
+    if ! key_pub="$(openssl pkey -pubout -passin pass: 2>/dev/null <<< "$content")" \
        || [ -z "$key_pub" ]; then
         log_error "Certificate bundle $file: openssl could not read the private key"
         return 1
@@ -174,7 +216,7 @@ cert_publish() {
     fi
 
     local cert_file="$1" key_file="$2" dest_file="$3"
-    local staging_dir backup_dir tmp base
+    local staging_dir backup_dir dest_dir tmp base
 
     # (a) sources must exist and be non-empty before we touch anything.
     if [ ! -s "$cert_file" ]; then
@@ -189,13 +231,42 @@ cert_publish() {
     # (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
     #     scans wholesale.
     staging_dir="$(cert_staging_dir)"
-    if ! mkdir -p "$staging_dir"; then
+    dest_dir="$(dirname "$dest_file")"
-        log_error "cert_publish: cannot create staging directory $staging_dir"
+    if ! mkdir -p "$staging_dir" || ! mkdir -p "$dest_dir"; then
+        log_error "cert_publish: cannot create staging directory $staging_dir or destination directory $dest_dir"
         return 1
     fi
-    # Sweep temps orphaned by a kill -9 / OOM in an earlier run. Restricted to
+
-    # the mktemp suffix shape inside our own staging dir.
+    # (c) the final swap is `mv`, which is only a rename(2) - and therefore only
-    find "$staging_dir" -maxdepth 1 -type f -name '*.??????' -mmin +1440 -delete 2>/dev/null
+    #     atomic - within one filesystem. Across filesystems GNU mv copies:
+    #     it truncates and writes the DESTINATION, then unlinks the source, so a
+    #     failure part-way through (ENOSPC is the realistic one) leaves exactly
+    #     the half-written live pem this library exists to prevent. Both paths
+    #     are env-overridable, so check instead of assuming. Same check as the
+    #     st_dev comparison in publish_pem_bundle() on the Python side.
+    local staging_dev dest_dev
+    staging_dev="$(stat -Lc '%d' "$staging_dir" 2>/dev/null)"
+    dest_dev="$(stat -Lc '%d' "$dest_dir" 2>/dev/null)"
+    if [ -z "$staging_dev" ] || [ -z "$dest_dev" ]; then
+        log_error "cert_publish: cannot stat $staging_dir and/or $dest_dir - refusing to publish $dest_file"
+        return 1
+    fi
+    if [ "$staging_dev" != "$dest_dev" ]; then
+        log_error "cert_publish: staging dir $staging_dir and destination dir $dest_dir" \
+                  "are on different filesystems, so the bundle cannot be swapped in atomically." \
+                  "Refusing to publish $dest_file (live file left untouched);" \
+                  "point CERT_STAGING_DIR at a directory on the same filesystem as $dest_dir"
+        return 1
+    fi
+
+    # Sweep temps orphaned by a kill -9 / OOM in an earlier run. Two shapes,
+    # because two publishers share this directory: mktemp's six-X suffix from
+    # this library, and `<name>.<random>.tmp` from write_config_atomically() on
+    # the Python side (tempfile.mkstemp(prefix=name + '.', suffix='.tmp')).
+    # Matching only the mktemp shape - as this did - left every Python-side
+    # temp behind forever.
+    find "$staging_dir" -maxdepth 1 -type f \
+        \( -name '*.??????' -o -name '*.tmp' \) -mmin +1440 -delete 2>/dev/null
 
     base="$(basename "$dest_file")"
     tmp="$(mktemp "${staging_dir}/${base}.XXXXXX" 2>/dev/null)"
@@ -218,14 +289,14 @@ cert_publish() {
         return 1
     fi
 
-    # (c) never promote something HAProxy would choke on.
+    # (d) never promote something HAProxy would choke on.
     if ! cert_bundle_valid "$tmp"; then
         log_error "cert_publish: assembled bundle for $dest_file failed validation (live file left untouched)"
         rm -f "$tmp"
         return 1
     fi
 
-    # (d) back up the bundle we are about to replace - but only if it is itself
+    # (e) back up the bundle we are about to replace - but only if it is itself
     #     valid. Overwriting a good backup with garbage would turn "restore the
     #     backup" into "restore a different broken file". Same semantics as
     #     create_backup(require_valid=True) in haproxy_manager.py.
@@ -252,12 +323,18 @@ cert_publish() {
     # who can read these files is not something a write-safety fix should do as
     # a side effect - and it must match write_config_atomically() on the Python
     # side, which preserves the mode the same way.
+    #
+    # -L (follow symlinks) matters: stat without it reports the mode of the
+    # SYMLINK, which is 0777 on Linux and is not a permission at all. A live
+    # pem that is a symlink therefore produced a world-WRITABLE 0777 private
+    # key sitting in the crt directory. With -L we copy the mode of the file
+    # the link points at, which is the mode an operator actually chose.
     local mode
-    mode="$(stat -c '%a' "$dest_file" 2>/dev/null)"
+    mode="$(stat -Lc '%a' "$dest_file" 2>/dev/null)"
     [ -n "$mode" ] || mode=644
     chmod "$mode" "$tmp" 2>/dev/null
 
-    # (e) atomic swap. Same filesystem by construction; if it still fails,
+    # (f) atomic swap. Same filesystem, verified in (c); if it still fails,
     #     stop - do not fall back to writing into the certs dir.
     if ! mv -f "$tmp" "$dest_file"; then
         log_error "cert_publish: failed to move $tmp into place as $dest_file" \

scripts/renew-certificates.sh

@@ -108,5 +108,18 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# A per-domain publication failure is a real failure and must be reported as
+# one. This script used to `exit 0` no matter how many domains failed, so
+# "0 updated, 12 failed" - a host that has completely stopped publishing
+# renewals - looked identical to a clean run to cron, to
+# host-renew-certificates.sh (which branches on this exit code) and to any
+# external monitoring. The first visible symptom would have been certificates
+# expiring. The loop above deliberately continues past a failed domain so the
+# others still get published; the status is reported here instead.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate renewal process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate renewal process completed"
 exit 0

scripts/sync-certificates.sh

@@ -90,5 +90,12 @@ if [ $UPDATED -gt 0 ]; then
     fi
 fi
 
+# See the matching block in renew-certificates.sh: a run in which every domain
+# failed to publish must not look like a clean run to its caller.
+if [ "$FAILED" -gt 0 ]; then
+    log_error "Certificate sync process completed with failures: $UPDATED updated, $FAILED failed"
+    exit 1
+fi
+
 log_info "Certificate sync process completed"
 exit 0

scripts/test-cert-scripts.py

@@ -41,16 +41,26 @@ binaries on PATH.
 
 The certificate material below is a real self-signed test certificate with its
 matching key (plus a second, unrelated key for the mismatch case), embedded as
-constants so the tests need no openssl to *create* material. The one test that
+constants so the tests need no openssl to *create* material. openssl IS needed
-needs openssl to *verify* pairing skips itself if the binary is absent.
+to run them: the library's cert/key pairing check is mandatory (it is the only
+layer that can reject a bundle of empty pem blocks), so without the binary
+every publish is refused by design. The image ships openssl 3.x.
+
+The acceptance bar for this file is a green run INSIDE the built image, as
+root, which is where these scripts actually execute - not on a workstation.
+Several failure modes are invisible outside the container (root ignores the
+directory permissions one test used to rely on) and one was actively
+destructive there; see _cleanup_tmp().
 """
 
 import os
 import re
 import shutil
+import stat
 import subprocess
 import tempfile
 import textwrap
+import time
 import unittest
 
 MODULE_DIR = os.path.abspath(
@@ -152,11 +162,72 @@ FV0RvuQPDYkqsx8ibqpSv7c=
 -----END PRIVATE KEY-----
 """
 
+# A SECOND, unrelated but internally consistent pair. Needed by the
+# concurrency test: two bundles that are each perfectly valid but whose keys
+# differ, so a validator that reads the certificate from one and the key from
+# the other reports a mismatch. Two bundles sharing key material - which is
+# what PREVIOUS_BUNDLE and NEW_BUNDLE are - cannot detect that at all.
+# openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
+#         -subj /CN=second.example.com
+TEST_CERT_2 = """\
+-----BEGIN CERTIFICATE-----
+MIIDGzCCAgOgAwIBAgIUKQAAvrVkden7gIg2zYMLb6dO1c8wDQYJKoZIhvcNAQEL
+BQAwHTEbMBkGA1UEAwwSc2Vjb25kLmV4YW1wbGUuY29tMB4XDTI2MDgwNjE2NTcz
+NVoXDTM2MDgwMzE2NTczNVowHTEbMBkGA1UEAwwSc2Vjb25kLmV4YW1wbGUuY29t
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv2pfTOvN2zrZJ6d+biiD
+VOczMqanuHPBCy2CnTBif+7VWf8AaywRzQ3ShfhmalVRNEFMn0MSO+GocmH71Ve1
+nl89oGiJPmvX1lpncbgM692ddhP9ez4xUeNj+QAWp9VBZhInNuM4Pawv5BPngtpj
+2MGXf3ZlBSli8Ng7jBo1fTMT3bh8GcE1rIPRvmUuQwFIt2eGnLR8jQd+xGelhAjG
+nnXtlc+ebo4r2OjljNgvtdUknBZdpiZXmjdFzyClYTeMuEen2uwMpJNc0wLbRjcU
+khVF3nw4jUnkOhWH3JYGAoWslJyEqZSANwt/eOHwXgyVuxg31bCl297iskW0IRZL
+wQIDAQABo1MwUTAdBgNVHQ4EFgQU7HH8GacJzc2j9s2UJCPwykgrIckwHwYDVR0j
+BBgwFoAU7HH8GacJzc2j9s2UJCPwykgrIckwDwYDVR0TAQH/BAUwAwEB/zANBgkq
+hkiG9w0BAQsFAAOCAQEAYE5jHX1dK091jVsFSZDdiw9AU5rrk8XpF1yuPmDisRnE
+dJ4QQq3dzWXRnp0bzZnq7fdfiEz1m39zVixov7WFp24QhenD2n5K7/wew7RpXTnA
+pAGBEdsGvBJ+3MgkRYklXCM9f9f4z21xXRNZ+BwBcM25D+gR4b+PRMQR6BhZx5R+
+y2jQsoM68cFjRApFWgmji4pBjg/eOaZMBCfVTjP+npVyqG7UtV5EyYXwPgPa/rm2
+FoP+eftzP6dszBEonkIVyyvkdscI4Wkr8hw3S0R/TP8l9lTnvN1HC3o7Es5VY53R
+swNAXWBlgm0N7A96ISLtQjgvOfeMRTCSjxW9pm0wJA==
+-----END CERTIFICATE-----
+"""
+
+TEST_KEY_2 = """\
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC/al9M683bOtkn
+p35uKINU5zMypqe4c8ELLYKdMGJ/7tVZ/wBrLBHNDdKF+GZqVVE0QUyfQxI74ahy
+YfvVV7WeXz2gaIk+a9fWWmdxuAzr3Z12E/17PjFR42P5ABan1UFmEic24zg9rC/k
+E+eC2mPYwZd/dmUFKWLw2DuMGjV9MxPduHwZwTWsg9G+ZS5DAUi3Z4actHyNB37E
+Z6WECMaede2Vz55ujivY6OWM2C+11SScFl2mJleaN0XPIKVhN4y4R6fa7Aykk1zT
+AttGNxSSFUXefDiNSeQ6FYfclgYChayUnISplIA3C3944fBeDJW7GDfVsKXb3uKy
+RbQhFkvBAgMBAAECggEABQLqy6eedRPt31sLOMFEDkzbZdFHOMpZeuqThsNmjo7z
+t9diokgeD4ZQXimbEQqsZYDAtmFGdiEp86I9JQBk/4TUiqFhHrOADBe1jQAptjfs
+iyIb51gIPnjZ1RBA5Al8zohy28T9h1+Z7+/OeCvcgLyVAixf/U9pU8D1El9cu9zv
+4q4WJPB5Tgkq+YwcmeuT8LzsKoSDmPQjFVY9v+gz6hoVUVyP6gswnlFnKjNcmfZU
+0CKP90sCAc5mKZv7RyGG920LDU4u2ggnQoK05GhXK8R3amJmoAF3i+xGeTRvNEKe
+wDC7NinTG2WDVo6y/FCvFKs0+qqlKww1u39fq166rQKBgQDtjPjvkDkSYFcWFznn
+sfsxN5R1cLpxdutLZhfvtpHIj5NkQbmOk0r3LbDAwU+UNVQ6F8jWQVHUsni9jAY6
+3RNRF4ZabC51aqg/Ssj7d5mE4kj7y6Ch/nnSXIogRxLG4bWo2rtG1M2Dt3Ef08R4
+6gjcp9ZmELyVx7H4yXFJGIaDdQKBgQDOSCHFK/ggZzzy4iC7DOq1lrbR+jyk95rh
+F5EGzyAkgJ4uYc9TCPkUDxXjWL17r61/obfbW4znaV9fHEKpz3R54osDXln3oLea
+BBWkJI3ANe8iNrGDE4FN9to4DdUMFsWX/WEiRyBPIqDy9DTyadblMLTNZjNPuLmg
+ZJOB3tRZnQKBgQCjeglya9Eq2Uv1MuSxk2VnmHU9YOed4BXLHKZKXFz1JgFr1GNL
+QAguFK536FDIkO62z9lxwR/8fRnkb7F13uBFRSg7oAlU2qKQc/nePI9UyJkrVxXj
+hYn2f6K61c6ROZFXc7e/5gDMrXhXS9gA0iZpG8PLF6eAeB39NTwV7p/bZQKBgQCV
+v5eEY58FJu0ABVhtcbsRiA+/70EHIRi2Pz1xC/vxg81RLoArb2AiR7FEEa+8kpQJ
+C4VFIPjxJXWuvf1G+Os9cFAqadw1/95JWJ29QywEVSL8W2gSF57O0l0oRCJdXEql
+Q7O4BppV2HWu6clmEZ+HUgxu77pgLWHUJi9PIExXoQKBgEq6rYBXmsSQermnPTOn
+YFx7c2ns97hsjYIbs497+gPW4/xQWwsN76t60SWqjXV4DHJCpi5Tnjo2fk5/OzK+
+HfshC9CKFDk8T0KGQWwaPwqP/OYbqOA88IlJ7xbPdSuJkjefHCtVVEtao6+HNgzm
+lniLtDMpU0MLPgB98ClQ4HDA
+-----END PRIVATE KEY-----
+"""
+
 # The bundle already on disk when a run starts. Same key material, plus a
 # trailing marker so "the live file was replaced" and "the old file was
 # archived" can be told apart byte-for-byte.
 PREVIOUS_BUNDLE = TEST_CERT + TEST_KEY + '# previous bundle\n'
 NEW_BUNDLE = TEST_CERT + TEST_KEY
+OTHER_BUNDLE = TEST_CERT_2 + TEST_KEY_2
 
 # --- stub binaries -----------------------------------------------------------
 STUB_CERTBOT = """\
@@ -216,6 +287,12 @@ class CertScriptFixture(unittest.TestCase):
     """An isolated fake /etc/haproxy + /etc/letsencrypt plus stub binaries."""
 
     def setUp(self):
+        # Without the library every "the library rejects X" assertion in this
+        # file can be satisfied by bash exiting 127, so make its absence a
+        # failure of every test rather than a silent pass of several.
+        self.assertTrue(os.path.isfile(LIB),
+                        f'{LIB} is missing - nothing below tests anything')
+
         self.tmp = tempfile.mkdtemp(prefix='haproxy-cert-test-')
         self.addCleanup(self._cleanup_tmp)
 
@@ -246,11 +323,28 @@ class CertScriptFixture(unittest.TestCase):
         self.error_log = os.path.join(self.tmp, 'haproxy-manager-errors.log')
 
     def _cleanup_tmp(self):
-        # A test may have chmod 000'd a fixture file.
+        # A test may have chmod 000'd a fixture file, which would stop rmtree.
+        #
+        # SYMLINKS ARE SKIPPED, and that is not a nicety. os.chmod() FOLLOWS
+        # symlinks, and the openssl-availability tests below build a stripped
+        # PATH directory out of symlinks to real system binaries (/usr/bin/cat,
+        # /usr/bin/chmod, ...). Walking those with chmod 0600 as root - which is
+        # how this container runs - stripped the exec bit from a dozen core
+        # binaries of the machine running the tests, chmod itself included, so
+        # it could not even be undone from inside the container: every later
+        # test failed with "/usr/bin/grep: Permission denied" and certificate
+        # publishing stayed dead until the container was recreated. It never
+        # showed up on a workstation because an unprivileged chmod of a
+        # root-owned file fails EPERM and was swallowed by `except OSError`.
+        # This file ships in the image (COPY scripts /haproxy/scripts), so
+        # running it in place is a thing an operator will do.
         for root, dirs, files in os.walk(self.tmp):
             for name in files:
+                path = os.path.join(root, name)
+                if os.path.islink(path):
+                    continue
                 try:
-                    os.chmod(os.path.join(root, name), 0o600)
+                    os.chmod(path, 0o600)
                 except OSError:
                     pass
         shutil.rmtree(self.tmp, ignore_errors=True)
@@ -310,6 +404,27 @@ class CertScriptFixture(unittest.TestCase):
             self.assertEqual(sorted(os.listdir(self.staging_dir)), [],
                              'staging file was not cleaned up')
 
+    def assert_rejected(self, result, because):
+        """The library refused, FOR THE STATED REASON.
+
+        `assertNotEqual(rc, 0)` on its own proves nothing about this library.
+        Delete cert-publish-lib.sh and `. "$1"` fails, cert_bundle_valid is
+        never defined, bash exits 127 - and a bare rc!=0 assertion passes. Four
+        tests in TestCertPublishLibrary were doing exactly that; they were
+        pinning "some bash pipeline failed", not "the bundle was rejected".
+        """
+        output = result.stdout + result.stderr
+        self.assertNotIn('command not found', output,
+                         'the shell could not find the function under test - '
+                         'this asserts nothing about the library')
+        self.assertNotEqual(127, result.returncode,
+                            f'exit 127 means "no such command", not "rejected": {output}')
+        self.assertNotEqual(0, result.returncode,
+                            f'expected a rejection, got success: {output}')
+        self.assertIn(because, output,
+                      f'rejected, but not for the expected reason '
+                      f'({because!r} not in output): {output}')
+
 
 class CertScriptBehaviour:
     """Behaviour shared by renew-certificates.sh and sync-certificates.sh.
@@ -378,8 +493,17 @@ class CertScriptBehaviour:
                          'HAProxy was reloaded even though nothing was updated')
         self.assert_certs_dir_is_clean()
         self.assert_no_staging_leftovers()
-        self.assertEqual(result.returncode, 0,
+        # This used to assert returncode == 0 with the comment "a per-domain
-                         'a per-domain failure should not change the exit code')
+        # failure should not change the exit code", codifying the script's
+        # `exit 0`. That is wrong and it is the dangerous kind of wrong: a host
+        # where EVERY domain fails to publish looked, to cron and to
+        # host-renew-certificates.sh (which branches on this exit code),
+        # exactly like a clean run. Nothing would notice until the certificates
+        # expired. Continuing past a failed domain so the others still get
+        # published is right; reporting success afterwards is not.
+        self.assertNotEqual(result.returncode, 0,
+                            'a domain that failed to publish must be reported '
+                            'in the exit code, not just in the log')
 
     def break_source_key(self):
         """Make reading the source key fail, however this environment allows.
@@ -424,8 +548,9 @@ class CertScriptBehaviour:
 
     def test_mismatched_key_is_rejected(self):
         if shutil.which('openssl') is None:
-            self.skipTest('openssl CLI not available: the cert/key pairing '
+            self.skipTest('openssl CLI not available: without it every publish '
-                          'check is best-effort and is skipped by design')
+                          'is refused, so this test could not tell a pairing '
+                          'rejection from a missing-checker rejection')
         self.seed_previous_bundle()
         before = read(self.live_pem)
         write(self.src_key, UNRELATED_KEY)
@@ -504,10 +629,10 @@ class TestSyncCertificates(CertScriptBehaviour, CertScriptFixture):
 class TestCertPublishLibrary(CertScriptFixture):
     """Unit-level checks on cert-publish-lib.sh itself."""
 
-    def call(self, snippet, *args):
+    def call(self, snippet, *args, **env_overrides):
         return subprocess.run(
             ['bash', '-c', '. "$1"; shift; ' + snippet, '_', LIB, *args],
-            env=self.env(), capture_output=True, text=True)
+            env=self.env(**env_overrides), capture_output=True, text=True)
 
     def test_valid_bundle_accepted(self):
         path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
@@ -515,14 +640,32 @@ class TestCertPublishLibrary(CertScriptFixture):
 
     def test_empty_and_missing_bundles_rejected(self):
         empty = write(os.path.join(self.tmp, 'empty.pem'), '')
-        self.assertNotEqual(self.call('cert_bundle_valid "$1"', empty).returncode, 0)
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),
+                             'is empty')
         missing = os.path.join(self.tmp, 'nope.pem')
-        self.assertNotEqual(self.call('cert_bundle_valid "$1"', missing).returncode, 0)
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', missing),
+                             'does not exist')
 
     def test_key_without_end_marker_rejected(self):
         truncated = write(os.path.join(self.tmp, 'cut.pem'),
                           TEST_CERT + '-----BEGIN PRIVATE KEY-----\nMIIEvAIB\n')
-        self.assertNotEqual(self.call('cert_bundle_valid "$1"', truncated).returncode, 0)
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', truncated),
+                             'unterminated private key block')
+
+    def test_empty_pem_blocks_are_rejected(self):
+        """Why the pairing check is mandatory rather than best-effort.
+
+        Every structural check in the library passes on this file: a complete
+        CERTIFICATE block and a complete PRIVATE KEY block, both with nothing
+        between BEGIN and END. Only openssl can tell it is not a certificate.
+        """
+        hollow = write(os.path.join(self.tmp, 'hollow.pem'),
+                       '-----BEGIN CERTIFICATE-----\n'
+                       '-----END CERTIFICATE-----\n'
+                       '-----BEGIN PRIVATE KEY-----\n'
+                       '-----END PRIVATE KEY-----\n')
+        self.assert_rejected(self.call('cert_bundle_valid "$1"', hollow),
+                             'openssl could not read the certificate')
 
     def test_a_broken_live_pem_does_not_overwrite_a_good_backup(self):
         """Mirrors create_backup(require_valid=True) in haproxy_manager.py.
@@ -544,37 +687,218 @@ class TestCertPublishLibrary(CertScriptFixture):
                          'a good backup was overwritten with an unusable pem')
 
     def test_publish_fails_loudly_when_the_rename_cannot_happen(self):
-        """No silent fallback to writing straight into the certs dir."""
+        """No silent fallback to writing straight into the certs dir.
+
+        The failure is injected with a stub `mv` that refuses, rather than by
+        chmod 0500 on the certs dir: the container these scripts run in is
+        root, root ignores directory permissions, so the chmod version skipped
+        itself exactly where it matters and only ever ran on a workstation.
+        """
         self.seed_previous_bundle()
         before = read(self.live_pem)
-        os.chmod(self.certs_dir, 0o500)          # no writes: mv will fail
+        write(os.path.join(self.bindir, 'mv'),
-        self.addCleanup(os.chmod, self.certs_dir, 0o755)
+              "#!/bin/sh\n"
-        if os.geteuid() == 0:
+              "echo \"mv: cannot move '$2': Permission denied\" >&2\n"
-            self.skipTest('root ignores directory permissions')
+              "exit 1\n", 0o755)
+        self.addCleanup(os.unlink, os.path.join(self.bindir, 'mv'))
 
         result = self.call('cert_publish "$1" "$2" "$3"',
                            self.src_cert, self.src_key, self.live_pem)
 
-        self.assertNotEqual(result.returncode, 0,
+        self.assert_rejected(result, 'NOT falling back to a direct write')
-                            'a failed rename was reported as success')
         self.assertEqual(read(self.live_pem), before,
                          'the live pem was damaged by a failed rename')
         self.assert_no_staging_leftovers()
 
+    def test_published_pem_keeps_the_mode_of_the_file_it_replaces(self):
+        """A write-safety fix must not silently re-permission private keys.
+
+        Both directions matter: mktemp stages at 0600, so without the explicit
+        chmod every publish would tighten a 0644 bundle; and the mode must not
+        be copied from a symlink (see the next test).
+        """
+        for mode in (0o644, 0o640, 0o600):
+            with self.subTest(oct(mode)):
+                write(self.live_pem, PREVIOUS_BUNDLE, mode)
+                result = self.call('cert_publish "$1" "$2" "$3"',
+                                   self.src_cert, self.src_key, self.live_pem)
+                self.assertEqual(result.returncode, 0,
+                                 result.stdout + result.stderr)
+                self.assertEqual(read(self.live_pem), NEW_BUNDLE)
+                self.assertEqual(
+                    stat.S_IMODE(os.stat(self.live_pem).st_mode), mode,
+                    'publishing changed who can read the private key')
+
+    def test_symlinked_live_pem_does_not_become_world_writable(self):
+        """`stat -c %a` on a symlink reports 0777 - the LINK's mode, not a
+        permission. Copying that onto the staged bundle put a world-writable
+        private key in the directory HAProxy serves from. -L is what makes the
+        preserved mode the mode of the file an operator actually chose.
+        """
+        target = write(os.path.join(self.tmp, 'real-bundle.pem'),
+                       PREVIOUS_BUNDLE, 0o640)
+        os.symlink(target, self.live_pem)
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+        mode = stat.S_IMODE(os.stat(self.live_pem).st_mode)
+        self.assertEqual(
+            0, mode & 0o022,
+            f'published bundle is group/world writable ({oct(mode)}) - the '
+            f'symlink mode was copied onto a real private key')
+        self.assertEqual(0o640, mode)
+
+    def test_publish_refuses_when_staging_is_on_another_filesystem(self):
+        """The header used to claim a cross-device mv "fails loudly and leaves
+        the live pem alone". GNU mv does no such thing: across filesystems it
+        copies, so it truncates and writes the DESTINATION first and only then
+        discovers it cannot finish (ENOSPC being the realistic case) - the very
+        truncation this library exists to prevent. Both directories are
+        env-overridable, so the device numbers have to be checked up front.
+        """
+        staging = os.path.join(self.a_dir_on_another_filesystem(),
+                               'cert-staging')
+        self.seed_previous_bundle()
+        before = read(self.live_pem)
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem,
+                           CERT_STAGING_DIR=staging)
+
+        self.assert_rejected(result, 'different filesystems')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was disturbed by a refused publish')
+        self.assert_certs_dir_is_clean()
+
+    def test_stale_python_side_staging_temps_are_reaped(self):
+        """The staging dir has two writers.
+
+        write_config_atomically() on the Python side stages as
+        `<name>.<random>.tmp` (tempfile.mkstemp(prefix=name + '.',
+        suffix='.tmp')). The reaper matched only mktemp's `*.??????` shape, so
+        every temp leaked by a SIGKILL on the Python side stayed there forever.
+        """
+        os.makedirs(self.staging_dir, exist_ok=True)
+        stale_py = write(os.path.join(self.staging_dir,
+                                      DOMAIN + '.pem.ab12cd34.tmp'), 'stale\n')
+        stale_sh = write(os.path.join(self.staging_dir,
+                                      DOMAIN + '.pem.AbCdEf'), 'stale\n')
+        fresh = write(os.path.join(self.staging_dir,
+                                   'recent.pem.zz99yy.tmp'), 'fresh\n')
+        old = time.time() - 3 * 24 * 3600
+        for path in (stale_py, stale_sh):
+            os.utime(path, (old, old))
+
+        result = self.call('cert_publish "$1" "$2" "$3"',
+                           self.src_cert, self.src_key, self.live_pem)
+        self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+
+        self.assertFalse(os.path.exists(stale_py),
+                         'a stale Python-side staging temp was never reaped')
+        self.assertFalse(os.path.exists(stale_sh),
+                         'a stale shell-side staging temp was never reaped')
+        self.assertTrue(os.path.exists(fresh),
+                        'the reaper deleted a temp a concurrent publisher may '
+                        'still be writing')
+
+    def test_validation_of_a_live_pem_being_republished_is_not_spurious(self):
+        """cert_bundle_valid must judge ONE snapshot of the file.
+
+        cert_publish() calls cert_bundle_valid() on the LIVE pem (step (e), to
+        decide whether it is worth backing up) while another publisher may be
+        renaming a new bundle over it. The function used to open the file six
+        times, so `openssl x509` could read the outgoing bundle and `openssl
+        pkey` the incoming one - and report "private key does not match the
+        certificate" about two files that were each perfectly fine. That ERROR
+        goes into the log monitor-errors.sh watches, which makes it a page.
+
+        The two bundles alternated below are each internally valid but carry
+        DIFFERENT key material. That matters: NEW_BUNDLE and PREVIOUS_BUNDLE
+        share a cert and a key, so alternating those two could never produce a
+        mismatch no matter how badly the reads were interleaved - the test
+        would model a world in which the bug cannot happen and pass forever.
+        """
+        write(self.live_pem, NEW_BUNDLE)
+        a = write(os.path.join(self.tmp, 'churn-a.pem'), NEW_BUNDLE)
+        b = write(os.path.join(self.tmp, 'churn-b.pem'), OTHER_BUNDLE)
+
+        # A publisher renaming over the live pem as fast as it can. Staged
+        # outside the certs dir, then renamed, exactly like cert_publish().
+        churn = subprocess.Popen(
+            ['bash', '-c',
+             'end=$((SECONDS+8)); s="$4"; while [ $SECONDS -lt $end ]; do '
+             '  cp "$1" "$s"; mv -f "$s" "$2"; '
+             '  cp "$3" "$s"; mv -f "$s" "$2"; '
+             'done', '_', a, self.live_pem, b,
+             os.path.join(self.tmp, 'churn-staged.pem')],
+            env=self.env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+        def _stop():
+            churn.kill()
+            churn.wait()
+        self.addCleanup(_stop)
+
+        result = subprocess.run(
+            ['bash', '-c',
+             '. "$1"; for i in $(seq 1 200); do cert_bundle_valid "$2" || exit 1; done',
+             '_', LIB, self.live_pem],
+            env=self.env(), capture_output=True, text=True, timeout=120)
+        _stop()
+
+        output = result.stdout + result.stderr
+        self.assertNotIn('does not match the certificate', output,
+                         'two valid bundles were reported as a mismatched pair '
+                         'because the checks read different files')
+        self.assertEqual(0, result.returncode,
+                         f'a concurrent republish made validation fail: {output}')
+
+    def a_dir_on_another_filesystem(self):
+        certs_dev = os.stat(self.certs_dir).st_dev
+        for candidate in ('/dev/shm', '/run', '/var/tmp', '/tmp', '/'):
+            try:
+                if (os.path.isdir(candidate)
+                        and os.access(candidate, os.W_OK)
+                        and os.stat(candidate).st_dev != certs_dev):
+                    path = tempfile.mkdtemp(prefix='cert-xdev-', dir=candidate)
+                    self.addCleanup(shutil.rmtree, path, True)
+                    return path
+            except OSError:
+                continue
+        self.skipTest('no writable directory on a second filesystem available')
+
     def test_haproxy_config_ok_follows_the_validator(self):
         self.assertEqual(self.call('haproxy_config_ok').returncode, 0)
         write(self.haproxy_cfg, GOOD_HAPROXY_CFG + BROKEN_TOKEN + '\n')
         self.assertNotEqual(self.call('haproxy_config_ok').returncode, 0)
 
-    def test_missing_openssl_warns_but_does_not_block(self):
+    def _openssl_free_path(self, name):
-        """Best-effort layer: a missing checker must not stall renewals."""
+        """A PATH directory with the library's tools but no openssl.
-        fake_path = os.path.join(self.tmp, 'no-openssl-bin')
+
+        Symlinks, so this directory must never be walked with a chmod that
+        follows them - see _cleanup_tmp().
+        """
+        fake_path = os.path.join(self.tmp, name)
         os.makedirs(fake_path)
         for tool in ('cat', 'grep', 'mktemp', 'mv', 'cp', 'rm', 'mkdir',
-                     'basename', 'dirname', 'find', 'date', 'chmod'):
+                     'basename', 'dirname', 'find', 'date', 'chmod', 'stat'):
             real = shutil.which(tool)
             if real:
                 os.symlink(real, os.path.join(fake_path, tool))
+        return fake_path
+
+    def test_missing_openssl_is_a_hard_failure(self):
+        """The pairing check is MANDATORY: no openssl, no publication.
+
+        This used to assert the opposite - that a missing openssl warns and
+        publishes anyway - justified by "the image does not necessarily install
+        the openssl CLI". The image does: openssl 3.x arrives with
+        ca-certificates, which certbot needs, and generate_self_signed_cert()
+        already runs `openssl req` with check=True at first-run setup. So the
+        fail-open never actually fired, and structural checks alone accept a
+        bundle of empty pem blocks (see test_empty_pem_blocks_are_rejected).
+        """
+        fake_path = self._openssl_free_path('no-openssl-bin')
         path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
 
         # bash by absolute path: the stripped PATH cannot resolve it.
@@ -583,29 +907,27 @@ class TestCertPublishLibrary(CertScriptFixture):
              '_', LIB, path],
             env=self.env(PATH=fake_path), capture_output=True, text=True)
 
-        self.assertEqual(result.returncode, 0,
+        self.assert_rejected(result, 'openssl binary not found')
-                         'a missing openssl blocked publication')
+        self.assertIn('REFUSING', result.stdout + result.stderr,
-        self.assertRegex(result.stdout + result.stderr,
+                      'a broken image must be reported as a broken image')
-                         r'(?i)warning.*openssl',
-                         'the skipped pairing check was not announced loudly')
 
-    def test_missing_openssl_still_rejects_a_structurally_broken_bundle(self):
+    def test_missing_openssl_stops_a_publish_rather_than_weakening_it(self):
-        fake_path = os.path.join(self.tmp, 'no-openssl-bin2')
+        """cert_publish must inherit the refusal, and not touch the live pem."""
-        os.makedirs(fake_path)
+        fake_path = self._openssl_free_path('no-openssl-bin2')
-        for tool in ('cat', 'grep', 'date'):
+        self.seed_previous_bundle()
-            real = shutil.which(tool)
+        before = read(self.live_pem)
-            if real:
-                os.symlink(real, os.path.join(fake_path, tool))
-        path = write(os.path.join(self.tmp, 'nokey.pem'), TEST_CERT)
 
-        # bash by absolute path: the stripped PATH cannot resolve it.
         result = subprocess.run(
-            [shutil.which('bash'), '-c', '. "$1"; cert_bundle_valid "$2"',
+            [shutil.which('bash'), '-c',
-             '_', LIB, path],
+             '. "$1"; cert_publish "$2" "$3" "$4"',
+             '_', LIB, self.src_cert, self.src_key, self.live_pem],
             env=self.env(PATH=fake_path), capture_output=True, text=True)
 
-        self.assertNotEqual(result.returncode, 0,
+        self.assert_rejected(result, 'openssl binary not found')
-                            'structural checks stopped being mandatory')
+        self.assertEqual(read(self.live_pem), before,
+                         'the live pem was disturbed by a refused publish')
+        self.assert_certs_dir_is_clean()
+        self.assert_no_staging_leftovers()
 
 
 class TestScriptsAreSane(unittest.TestCase):

scripts/test-cert-write-safety.py

@@ -50,6 +50,7 @@ import logging
 import os
 import shutil
 import sqlite3
+import stat
 import subprocess
 import sys
 import tempfile
@@ -90,8 +91,25 @@ finally:
 logging.getLogger('haproxy_manager').setLevel(logging.CRITICAL)
 
 HAS_PUBLISHER = hasattr(hm, 'publish_pem_bundle')
-FIX_ONLY = unittest.skipUnless(
+
-    HAS_PUBLISHER, 'requires the certificate publishing fix (publish_pem_bundle)')
+# FIX_ONLY marks tests that can only run against a tree that HAS the fix, so
+# that pointing HAPROXY_MANAGER_DIR at a pre-fix checkout (how the bugs were
+# reproduced) skips them instead of erroring.
+#
+# It used to be `skipUnless(HAS_PUBLISHER, ...)` unconditionally, which is
+# tautological when testing THIS tree: rename or delete publish_pem_bundle and
+# 10 of the 17 tests silently skip themselves while the run still reports OK.
+# A guard that disappears when the thing it guards disappears is not a guard.
+# So the escape hatch is now tied to the thing it exists for - testing a
+# FOREIGN tree - and a missing publisher in the repo checkout is a hard
+# failure (see TestPublisherApiIsPresent).
+_REPO_ROOT = os.path.realpath(
+    os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+TESTING_FOREIGN_TREE = os.path.realpath(MODULE_DIR) != _REPO_ROOT
+FIX_ONLY = unittest.skipIf(
+    TESTING_FOREIGN_TREE and not HAS_PUBLISHER,
+    'HAPROXY_MANAGER_DIR points at a tree without the certificate publishing '
+    'fix (publish_pem_bundle)')
 NEEDS_OPENSSL = unittest.skipUnless(
     shutil.which('openssl'), 'needs the openssl CLI')
 
@@ -186,6 +204,14 @@ k49ABlblWHBsoUF63ka1PBrMgA==
 
 GOOD_BUNDLE = LEAF_CERT + LEAF_KEY
 
+# What a bundle already on disk looks like when a test starts: the same, valid
+# material plus a trailing marker. Without it the "previous" and "new" bundles
+# are byte-identical, and a test that meant to prove a publish HAPPENED cannot
+# tell that from a publish that did nothing at all. Trailing text after the key
+# is ignored by both HAProxy and openssl, so the file stays genuinely usable.
+PREVIOUS_MARKER = '# previous bundle\n'
+PREVIOUS_BUNDLE = GOOD_BUNDLE + PREVIOUS_MARKER
+
 # Stub haproxy. Beyond the config check the parent suite's stub does, this one
 # also walks the crt directory the way HAProxy does when a `bind ... ssl crt
 # <dir>` is used: every file in there must be a loadable cert+key bundle, and
@@ -375,9 +401,9 @@ class CertPublishTestCase(unittest.TestCase):
             self.write(os.path.join(live, 'privkey.pem'), key)
         return live
 
-    def publish_live_bundle(self, domain):
+    def publish_live_bundle(self, domain, content=PREVIOUS_BUNDLE):
         """A good bundle already being served for `domain`."""
-        return self.write(os.path.join(self.certs, f'{domain}.pem'), GOOD_BUNDLE)
+        return self.write(os.path.join(self.certs, f'{domain}.pem'), content)
 
     def add_domain(self, domain, backend_name, ssl_cert_path=None):
         with sqlite3.connect(hm.DB_FILE) as conn:
@@ -478,19 +504,38 @@ class TestLivePemIsNeverTruncated(CertPublishTestCase):
         self.assertEqual('error', resp.get_json()['status'])
 
     def test_successful_renewal_still_publishes(self):
-        """The guard must not block the normal path."""
+        """The guard must not block the normal path.
+
+        The old version built its "renewed" certificate with a no-op
+        `.replace('\\n-----END CERTIFICATE-----', '\\n-----END CERTIFICATE-----')`,
+        so the renewed lineage was byte-identical to what was already being
+        served. Every assertion still passed if the renewal published nothing
+        at all - which is the failure this test is supposed to catch. The live
+        bundle now carries a marker the renewed material does not, so "the file
+        on disk changed" is an actual assertion.
+        """
         cert_path = self.publish_live_bundle('ok.example.com')
+        self.assertIn(PREVIOUS_MARKER, self.read(cert_path),
+                      'fixture precondition: the live bundle is distinguishable')
         self.add_domain('ok.example.com', 'ok_backend', ssl_cert_path=cert_path)
-        renewed = LEAF_CERT.replace('\n-----END CERTIFICATE-----',
+        self.make_lineage('ok.example.com')
-                                    '\n-----END CERTIFICATE-----')
-        self.make_lineage('ok.example.com', cert=renewed)
 
         resp = self.client.post('/api/certificates/renew')
 
         self.assertEqual(200, resp.status_code, resp.get_data(as_text=True))
-        self.assertTrue(structurally_valid(self.read(cert_path)))
+        published = self.read(cert_path)
+        self.assertEqual(GOOD_BUNDLE, published,
+                         'the renewed cert+key was not written to the live pem')
+        self.assertNotIn(PREVIOUS_MARKER, published,
+                         'the previous bundle is still on disk: the renewal '
+                         'published nothing')
+        self.assertTrue(structurally_valid(published))
         self.assertTrue(self.edge_would_start())
         self.assert_only_final_pems_in_certs_dir()
+        backup = os.path.join(hm.cert_backup_dir(), 'ok.example.com.pem')
+        self.assertTrue(os.path.exists(backup))
+        self.assertEqual(PREVIOUS_BUNDLE, self.read(backup),
+                         'the archived copy is not the bundle that was replaced')
 
 
 class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
@@ -658,11 +703,25 @@ class TestBundleValidation(CertPublishTestCase):
 
     @FIX_ONLY
     def test_staging_and_backups_live_outside_the_crt_directory(self):
-        """A temp or backup file inside the crt dir would be loaded by HAProxy."""
+        """A temp or backup file inside the crt dir would be loaded by HAProxy.
-        for path in (hm.cert_staging_dir(), hm.cert_backup_dir()):
+
+        `startswith(certs + os.sep)` alone let the worst case through: if
+        cert_staging_dir() returned the crt directory ITSELF - staging straight
+        into the directory HAProxy scans, the exact hazard this design exists
+        to prevent - the assertion passed, because the certs dir is not a
+        strict subpath of itself.
+        """
+        certs = os.path.realpath(self.certs)
+        for name, path in (('staging', hm.cert_staging_dir()),
+                           ('backup', hm.cert_backup_dir())):
+            real = os.path.realpath(path)
+            self.assertNotEqual(
+                certs, real,
+                f'the {name} directory IS the crt directory - HAProxy would '
+                f'load every temp/backup file in it')
             self.assertFalse(
-                os.path.abspath(path).startswith(os.path.abspath(self.certs) + os.sep),
+                real.startswith(certs + os.sep),
-                f'{path} must not be inside {self.certs}')
+                f'the {name} directory {path} must not be inside {self.certs}')
 
     @FIX_ONLY
     def test_previous_bundle_is_backed_up_on_publish(self):
@@ -700,18 +759,160 @@ class TestBundleValidation(CertPublishTestCase):
                          'file')
 
     @FIX_ONLY
+    @NEEDS_OPENSSL
     def test_no_temp_file_survives_a_failed_publish(self):
+        """The failure must happen AFTER something has been staged.
+
+        This used to feed publish_pem_bundle() an empty privkey, which is
+        rejected while reading the sources - before the staging directory is
+        even created. The test then asserted `[] == []` and proved nothing
+        about temp-file cleanup. A structurally perfect bundle whose key
+        belongs to a different certificate is rejected by the pairing check,
+        which runs on the fully staged file, so the temp definitely exists at
+        the moment the publish fails.
+        """
         dest = os.path.join(self.certs, 'leak.example.com.pem')
         cert = self.write(os.path.join(self.tmp, 'src4', 'fullchain.pem'),
                           LEAF_CERT)
-        empty = self.write(os.path.join(self.tmp, 'src4', 'privkey.pem'), '')
+        wrong = self.write(os.path.join(self.tmp, 'src4', 'privkey.pem'),
+                           UNRELATED_KEY)
         with self.assertRaises(hm.CertificatePublishError):
-            hm.publish_pem_bundle(dest, [cert, empty])
+            hm.publish_pem_bundle(dest, [cert, wrong])
+
+        staging = hm.cert_staging_dir()
+        self.assertTrue(os.path.isdir(staging),
+                        'the publish never got as far as staging, so this '
+                        'asserts nothing about cleanup')
+        self.assertEqual([], os.listdir(staging),
+                         'staged files must be cleaned up when a publish fails')
+        self.assertFalse(os.path.exists(dest),
+                         'a rejected bundle was published anyway')
         self.assert_only_final_pems_in_certs_dir()
-        self.assertEqual(
+
-            [], os.listdir(hm.cert_staging_dir()) if
+
-            os.path.isdir(hm.cert_staging_dir()) else [],
+    @FIX_ONLY
-            'staged files must be cleaned up when a publish fails')
+    @NEEDS_OPENSSL
+    def test_empty_pem_blocks_are_rejected(self):
+        """Why the pairing check may not be best-effort.
+
+        Structural validation is weak on its own: BEGIN/END pairs with nothing
+        between them satisfy every rule in validate_pem_structure(). Only
+        openssl can say this is not a certificate.
+        """
+        hollow = ('-----BEGIN CERTIFICATE-----\n-----END CERTIFICATE-----\n'
+                  '-----BEGIN PRIVATE KEY-----\n-----END PRIVATE KEY-----\n')
+        ok, _ = hm.validate_pem_structure(hollow)
+        self.assertTrue(ok, 'precondition: structure alone accepts this file')
+
+        path = self.write(os.path.join(self.tmp, 'hollow.pem'), hollow)
+        ok, msg = hm.validate_pem_bundle(path)
+        self.assertFalse(ok, 'a bundle of empty pem blocks was accepted')
+
+    @FIX_ONLY
+    def test_publish_refuses_when_the_pairing_check_cannot_run(self):
+        """No fail-open. 'unavailable' means the image is broken, not the cert.
+
+        The previous behaviour logged a warning and published on structural
+        checks alone, justified by "the Dockerfile does not install the openssl
+        CLI". It does - openssl 3.x comes in with ca-certificates, and
+        generate_self_signed_cert() shells out to `openssl req` with
+        check=True during setup - so this branch never fired and the fail-open
+        was safe only by accident.
+        """
+        real = hm._openssl_pairing_status
+        hm._openssl_pairing_status = lambda path: ('unavailable',
+                                                   'openssl binary not found')
+        self.addCleanup(setattr, hm, '_openssl_pairing_status', real)
+
+        dest = self.publish_live_bundle('nocheck.example.com')
+        before = self.read(dest)
+        cert = self.write(os.path.join(self.tmp, 'src5', 'fullchain.pem'),
+                          LEAF_CERT)
+        key = self.write(os.path.join(self.tmp, 'src5', 'privkey.pem'), LEAF_KEY)
+
+        with self.assertRaises(hm.CertificatePublishError):
+            hm.publish_pem_bundle(dest, [cert, key])
+
+        self.assertEqual(before, self.read(dest),
+                         'the live pem was disturbed by a refused publish')
+        self.assert_only_final_pems_in_certs_dir()
+
+    @FIX_ONLY
+    def test_binary_corrupt_live_bundle_can_still_be_republished(self):
+        """Republishing is how an operator heals a corrupt live pem.
+
+        A pem corrupted into binary (a partial write from before this fix, a
+        bad restore) is unreadable as text. backup_existing_pem() opens it in
+        text mode and caught only OSError, so the UnicodeDecodeError escaped
+        publish_pem_bundle() entirely: the single operation that would have put
+        a working certificate back blew up trying to archive the broken one.
+        The shell half recovers from this without complaint.
+        """
+        dest = os.path.join(self.certs, 'binary.example.com.pem')
+        with open(dest, 'wb') as fh:
+            fh.write(b'\x00\x01\x02\xfe\xff' * 128)
+        cert = self.write(os.path.join(self.tmp, 'src6', 'fullchain.pem'),
+                          LEAF_CERT)
+        key = self.write(os.path.join(self.tmp, 'src6', 'privkey.pem'), LEAF_KEY)
+
+        hm.publish_pem_bundle(dest, [cert, key])
+
+        self.assertEqual(GOOD_BUNDLE, self.read(dest),
+                         'a binary-corrupt live pem blocked its own repair')
+        self.assertTrue(self.edge_would_start())
+
+    @FIX_ONLY
+    def test_published_pem_file_mode(self):
+        """Publishing must not silently re-permission a private key.
+
+        The staged file is created by mkstemp at 0600; without the explicit
+        mode copy in write_config_atomically() every publish would tighten a
+        0644 bundle, and a future change in the other direction would loosen
+        one. Neither is a decision a write-safety fix gets to make as a side
+        effect, so pin it.
+        """
+        cert = self.write(os.path.join(self.tmp, 'src7', 'fullchain.pem'),
+                          LEAF_CERT)
+        key = self.write(os.path.join(self.tmp, 'src7', 'privkey.pem'), LEAF_KEY)
+
+        for mode in (0o644, 0o640, 0o600):
+            with self.subTest(oct(mode)):
+                dest = self.publish_live_bundle(f'mode{mode:o}.example.com')
+                os.chmod(dest, mode)
+                hm.publish_pem_bundle(dest, [cert, key])
+                self.assertEqual(mode,
+                                 stat.S_IMODE(os.stat(dest).st_mode),
+                                 'publishing changed who can read the key')
+
+        fresh = os.path.join(self.certs, 'fresh.example.com.pem')
+        hm.publish_pem_bundle(fresh, [cert, key])
+        fresh_mode = stat.S_IMODE(os.stat(fresh).st_mode)
+        self.assertEqual(0o644, fresh_mode,
+                         'a newly created bundle should match the 0644 that '
+                         '`cat > file` produced under the standard umask')
+        self.assertEqual(0, fresh_mode & 0o022,
+                         'a private key must never be group/world writable')
+
+
+@unittest.skipIf(TESTING_FOREIGN_TREE,
+                 'HAPROXY_MANAGER_DIR points at another tree')
+class TestPublisherApiIsPresent(unittest.TestCase):
+    """FIX_ONLY must never be able to hide the fix going missing.
+
+    With `FIX_ONLY = skipUnless(hasattr(hm, 'publish_pem_bundle'))`, renaming
+    that one function turned 10 of these 17 tests into skips and the run still
+    printed OK. This class fails loudly instead.
+    """
+
+    def test_publisher_api_is_present(self):
+        for name in ('publish_pem_bundle', 'validate_pem_bundle',
+                     'validate_pem_structure', 'backup_existing_pem',
+                     'cert_staging_dir', 'cert_backup_dir',
+                     'CertificatePublishError', '_openssl_pairing_status'):
+            self.assertTrue(
+                hasattr(hm, name),
+                f'haproxy_manager.{name} is gone - the tests that exercise it '
+                f'would otherwise skip themselves and report OK')
 
 
 class TestClusterSecretSelfHeal(CertPublishTestCase):