cloud-hosting-platform/haproxy-manager-base
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(certs): stop the test suite bricking the container; make the pairing check real

Browse Source 
Gate findings on fix/cert-write-safety. One blocker, one latent truncation
path, a false premise in a load-bearing comment, and a set of tests that were
passing without testing anything.

THE BLOCKER - scripts/test-cert-scripts.py bricked the production container
------------------------------------------------------------------------
The openssl-availability tests build a stripped PATH out of SYMLINKS to real
system binaries (cat, grep, mktemp, mv, cp, rm, mkdir, basename, dirname, find,
date, chmod). _cleanup_tmp() then walked the temp tree calling os.chmod(p,
0o600) - and os.chmod FOLLOWS SYMLINKS. Run once as root in the real image,
which is where this file ships (COPY scripts /haproxy/scripts) and where an
operator would most plausibly run it after deploying a cert fix, it stripped
the exec bit off twelve core binaries INCLUDING chmod itself, so it could not
be undone from inside the container:

    /haproxy/scripts/cert-publish-lib.sh: line 109: /usr/bin/grep: Permission denied
    bash: /usr/bin/chmod: Permission denied

Certificate publishing stayed dead until the container was recreated. It was
invisible on a workstation because an unprivileged chmod of a root-owned file
fails EPERM straight into `except OSError: pass` - which is also why the
advertised "32 tests pass" was only ever true off-container. In the image the
shipped file measured FAILED (failures=22, skipped=1). Cleanup now skips
symlinks; the suites are green in the image and the binaries survive.

S1 - the shell half had no same-filesystem guard
------------------------------------------------
The header claimed a cross-device mv would "FAIL LOUDLY and leave the live pem
alone". GNU mv does the opposite: across filesystems it copies, so it opens and
truncates the DESTINATION and only then discovers it cannot finish - measured
as a 204800-byte partial live.pem, sentinel gone, before mv reported ENOSPC.
cert_publish() now compares stat -Lc %d of the staging and certs dirs before
writing anything, mirroring the st_dev check the Python half already had, and
the comment says what mv actually does. Latent today (both dirs share a device
on all five hosts) but both are env-overridable.

openssl is present - correct the premise, make the check mandatory
------------------------------------------------------------------
Both halves justified a fail-open with "the image does not necessarily install
the openssl CLI". It does: openssl 3.5.6 in the running container, pulled in by
ca-certificates which certbot needs, and generate_self_signed_cert() already
shells to `openssl req` with check=True during setup. The 'unavailable' branch
never fired, so the pairing check has always run - and that, not the stated
reasoning, is what made the fail-open harmless. Structural validation alone is
weak: a bundle of EMPTY pem blocks passes every structural rule and is caught
only by openssl. The check is now mandatory in both halves and a missing binary
is a loud refusal. No `cryptography` fallback: the app runs on
/usr/local/bin/python3 (3.12) where it is not importable - it belongs to
Debian's /usr/bin/python3 - and reaching for that would be a second unverified
premise.

Smaller items
-------------
* cert_bundle_valid() read the file six times; a concurrent swap between two of
  them made openssl x509 and openssl pkey judge different files and log a bogus
  "private key does not match the certificate" into the monitored error log. It
  now reads one snapshot and feeds openssl from it on stdin.
* except OSError -> except (OSError, UnicodeDecodeError): a BINARY-corrupt live
  pem made backup_existing_pem() raise out of publish_pem_bundle() entirely, so
  the republish that would have healed the host was the one thing that could
  not run. The shell half recovers fine.
* stat -c %a on a symlinked live pem reports the LINK's 0777 and produced a
  world-writable private key in the crt directory; now stat -Lc.
* The staging reaper's '*.??????' glob matched mktemp names but not the Python
  side's '<name>.<random>.tmp', so those leaked forever. Matches both now.
* renew-certificates.sh and sync-certificates.sh exited 0 even when every
  domain failed to publish, so "0 updated, 12 failed" looked identical to a
  clean run to cron, to host-renew-certificates.sh (which branches on it) and
  to monitoring - a host could silently stop publishing renewals until the
  certificates expired. They now exit 1 if any domain failed, still after
  publishing the ones that worked.

Test-quality
------------
Four TestCertPublishLibrary tests passed with cert-publish-lib.sh DELETED -
they asserted only rc != 0, and `command not found` is 127. All four now assert
the rejection REASON via assert_rejected(), and setUp() fails if the library is
missing. test_missing_openssl_still_rejects... is replaced by
test_empty_pem_blocks_are_rejected, which pins the case that makes the pairing
check necessary.

Also: the staging-containment test used startswith(certs + os.sep), so
cert_staging_dir() returning the crt directory ITSELF - the exact hazard -
still passed; test_successful_renewal_still_publishes built its "renewed" cert
with a no-op .replace() and could not tell a renewal that published nothing;
test_no_temp_file_survives_a_failed_publish failed before the staging dir
existed and asserted [] == []; FIX_ONLY was skipUnless(hasattr(hm,
'publish_pem_bundle')), so renaming that function turned 10 of 17 tests into
skips while the run still printed OK. Each is fixed and each fix is
mutation-proved: the mutation that the old assertion waved through now fails.
File mode is pinned in both suites (it was pinned nowhere), and the rename
failure is injected with a stub mv instead of chmod 0500, which root ignored -
so that test no longer skips itself precisely where it matters.

Verification
------------
IN THE BUILT IMAGE, as root (the acceptance bar):
    scripts/test-cert-scripts.py        38 tests, OK, 0 skipped
    scripts/test-cert-write-safety.py   22 tests, OK, 0 skipped
    scripts/test-config-rollback.py     17 tests, OK   (neighbour, unchanged)
Workstation: 38/38 OK for the shell suite; the Python suite needs flask.
Against the pre-fix tree (main): shell 38 failures; python failures=4, errors=1,
skipped=15 - the FIX_ONLY skips are the 14 fix-only tests plus the API guard.
The old python suite against the pre-fix tree measures skipped=10, confirming
the gate's count.
32 mutation checks, all behaving as intended: every fix breaks a test when
reverted, and every rewritten test fails under the mutation its predecessor
passed. py_compile clean, bash -n clean, shellcheck clean, no new pyflakes
warnings (same 4 pre-existing).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
shadowdao
2026-08-06 10:03:37 -07:00
co-authored by Claude Opus 5
parent 22dab685d0
commit 4f0949d534
6 changed files with 792 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files
haproxy_manager.py
+61 -19
View File
@@ -373,8 +373,9 @@ def cert_backup_dir():
# ENCRYPTED PRIVATE KEY is deliberately absent: HAProxy cannot use a
# passphrase-protected key from a crt file, so a bundle containing one is not
# publishable. Accepting it here would let one through on a host without the
# openssl CLI, where the pairing check (which would also reject it) is skipped.
# publishable, and the structural layer is the only layer that names the
# problem ("no complete private key block") rather than reporting it as an
# unreadable key.
_PEM_KEY_LABELS = ('PRIVATE KEY', 'RSA PRIVATE KEY', 'EC PRIVATE KEY')
@@ -406,6 +407,11 @@ def validate_pem_structure(text):
lack of a tool. It catches every failure mode the truncation bug produced:
empty file, certificate without a key, key without a certificate, and a
block cut off mid-write.
It is NOT sufficient on its own, which is why _openssl_pairing_status() is
mandatory rather than best-effort: a bundle of EMPTY pem blocks (a BEGIN
line immediately followed by its END line, no base64 between them) passes
every check here and is rejected only by openssl.
"""
if not text.strip():
return False, 'bundle is empty'
@@ -430,14 +436,34 @@ def _openssl_pairing_status(path):
assembled bundle directly. Comparing the two public keys proves the pair.
'unavailable' means the openssl BINARY is absent - a verdict about our
tooling, not about the bundle. The Dockerfile installs haproxy, certbot,
socat and curl but not the openssl CLI, so this is a real possibility.
Callers treat it as a loud warning rather than a failure: the failure modes
this whole module exists to prevent (truncation, missing key, partial
write) are fully covered by validate_pem_structure(), whereas refusing to
publish whenever the checker is missing would stall renewals fleet-wide and
let certificates expire - a guaranteed outage traded for a hypothetical
one. A mismatch, when we CAN check, is always fatal.
tooling, not about the bundle. validate_pem_bundle() treats it as a HARD
FAILURE.
That is a deliberate reversal. This docstring used to say "the Dockerfile
installs haproxy, certbot, socat and curl but not the openssl CLI, so this
is a real possibility", and callers accepted the bundle on the structural
checks alone. The premise is false: openssl 3.x IS in the image, as a
dependency of ca-certificates (which certbot requires), and
generate_self_signed_cert() below already runs `openssl req` with
check=True during first-run setup - so no container has ever reached a
publish without it. The 'unavailable' branch never fired, which means the
pairing check has in fact always run, and THAT is what made the fail-open
harmless - not the stated reasoning. Structural validation on its own is
weak: a bundle of empty pem blocks passes validate_pem_structure() and is
caught only here.
So an absent openssl now means the image is broken, and we say so and stop
instead of quietly downgrading to the weaker check. The cost is that a
hypothetical openssl-less image stops publishing renewals - but it does so
immediately and loudly, in the monitored error log, at the first renewal,
rather than 90 days later; and publishing an unverified bundle can take the
whole :443 bind, i.e. every site on the host, down at the next reload.
There is no python `cryptography` fallback on purpose: this process runs on
/usr/local/bin/python3 (the base image's 3.12), where cryptography is not
importable. It is installed for Debian's /usr/bin/python3 as a certbot
dependency, and reaching for that interpreter would be a second unverified
premise of exactly the kind this comment is correcting.
"""
try:
cert_pub = subprocess.run(
@@ -471,14 +497,17 @@ def _openssl_pairing_status(path):
def validate_pem_bundle(path):
"""Validate a bundle file on disk. Returns (ok, message).
Mandatory structural validation plus a best-effort cryptographic pairing
check - see _openssl_pairing_status() for what happens when openssl is
missing (loud warning, structural verdict stands).
Structural validation AND the cryptographic pairing check, both mandatory -
see _openssl_pairing_status() for why a missing openssl is a failure rather
than a downgrade to structure-only.
"""
try:
with open(path, 'r') as fh:
text = fh.read()
except OSError as e:
except (OSError, UnicodeDecodeError) as e:
# UnicodeDecodeError, not just OSError: a bundle corrupted into binary
# is unreadable as text but reads perfectly well as bytes, so `except
# OSError` let the decode error escape as an unhandled traceback.
return False, f'cannot read assembled bundle: {e}'
ok, msg = validate_pem_structure(text)
@@ -489,9 +518,13 @@ def validate_pem_bundle(path):
if status == 'invalid':
return False, pair_msg
if status == 'unavailable':
logger.warning(
"Certificate key/leaf pairing check SKIPPED for %s (%s). The "
"bundle passed structural validation only.", path, pair_msg)
logger.error(
"Certificate key/leaf pairing check could not run for %s (%s) - "
"REFUSING to publish. openssl is required; structural validation "
"alone cannot tell a real bundle from empty pem blocks.",
path, pair_msg)
return False, (f'cert/key pairing check unavailable ({pair_msg}); '
'refusing to publish on structural checks alone')
return True, None
@@ -511,7 +544,14 @@ def backup_existing_pem(dest_path):
try:
with open(dest_path, 'r') as fh:
ok, msg = validate_pem_structure(fh.read())
except OSError as e:
except (OSError, UnicodeDecodeError) as e:
# UnicodeDecodeError, not just OSError. A live pem corrupted into
# BINARY (the exact state a republish is meant to heal) raises
# UnicodeDecodeError here, and with only OSError caught it escaped all
# the way out of publish_pem_bundle() - so the one operation that would
# have put a working certificate back blew up on the way to taking a
# backup of the broken one. The shell half recovers from this fine;
# this is the only reason the Python half did not.
ok, msg = False, str(e)
backup_path = os.path.join(cert_backup_dir(), os.path.basename(dest_path))
if not ok:
@@ -552,7 +592,9 @@ def publish_pem_bundle(dest_path, source_paths):
try:
with open(src, 'r') as fh:
data = fh.read()
except OSError as e:
except (OSError, UnicodeDecodeError) as e:
# Binary-corrupt source material must be a clean, reported refusal,
# not an unhandled UnicodeDecodeError out of the request handler.
raise CertificatePublishError(f'cannot read {src}: {e}')
if not data.strip():
raise CertificatePublishError(f'source file is empty: {src}')