DELETE /api/domain ran, unconditionally, for any ssl_enabled row:
os.remove(ssl_cert_path)
certbot delete --cert-name <domain> --non-interactive
`domains.domain` is UNIQUE. `domains.ssl_cert_path` is not, and nothing
anywhere guarded against two rows naming the same file. Sharing is not an
edge case -- it is the normal shape of the table, because
request_ssl_bundle() deliberately creates it: one SAN certificate is issued
as `--cert-name <primary>`, published once to
/etc/haproxy/certs/<primary>.pem, and then EVERY included name's row is
pointed at that same path ("Mark every name in the bundle as ssl_enabled,
all pointing at the same combined .pem").
Measured read-only on the live SQLite in the haproxy-manager containers on
2026-08-23:
* whp01: 157 domain rows, 150 ssl_enabled, 71 distinct cert paths.
39 of those paths are referenced by MORE THAN ONE row, covering 118 of
the 150 SSL-enabled rows. Worst cases: brain-jar.com.pem and
arclightcourt.com.pem with 10 domains each, hackerpublicradio.org.pem
with 6, anhonesthost.com.pem with 5.
* whp02: 33 rows, 31 ssl_enabled, 15 distinct paths, 11 shared across 27
rows -- including threeworldsoneheart.org.pem, referenced by the apex,
its www, and mail.threeworldsoneheart.org. A production cleanup of that
mail.* row was stopped short precisely because removing it would have
unlinked the PEM the serving site is using.
So removing one domain unlinked a file up to nine other configured domains
were being served from. HAProxy binds the crt directory
(`bind ... ssl crt /etc/haproxy/certs`), so the loss is not noticed until
the next reload or restart, at which point the listener refuses to come up
or those names fall back to the wrong certificate.
`certbot delete` is the worse half. It destroys the lineage's archive, live
symlinks and renewal config; recovery is a fresh, rate-limited ACME order.
The old code passed `--cert-name <domain>`, which is also simply the wrong
lineage for a SAN member: 81 of whp01's 150 SSL-enabled rows have a cert
path whose basename is not their own domain, so for those the call was a
silent no-op -- while for a bundle PRIMARY it deleted the one lineage still
renewing the certificate every other name in the bundle is served with.
The fix refcounts, after the row is deleted so the query answers "who else
still needs this":
* lineage_name_for_cert_path() -- the lineage is the published bundle's
basename minus .pem, the same derivation _quarantine_superseded_certs()
already uses, not the domain being removed.
* domains_referencing_cert_path() / domains_referencing_lineage() -- the
remaining rows that name that file, and that lineage.
* remove_domain() unlinks only when the list is empty, `certbot delete`s
only when the list is empty, logs the retained names explicitly when it
skips, and reports them as certificate_retained_for /
lineage_retained_for in the API response.
ssl_enabled is deliberately not filtered on in the refcount: the two
mistakes are not symmetric. A stale PEM left in the crt directory costs a
few kilobytes; an unlinked live one is HTTPS down for every name it serves.
Cleanup is deferred, not cancelled -- removing the last name on a bundle
still unlinks the file and deletes the lineage.
No row on either production host has ssl_enabled=1 with an empty
ssl_cert_path, so the "no path, no attributable lineage" branch changes
nothing on the current fleet.
Tests (scripts/test-cert-write-safety.py, +9, suite now 31, all offline):
last reference -> file unlinked and lineage deleted; shared file survives
removal of a SAN member AND of the bundle primary, byte-for-byte, with the
edge still starting; shared lineage is not certbot-deleted; the production
mail.* shape; removing every name eventually cleans up; an unrelated
bundle is never collateral damage. Assertions are on os.path.exists, file
contents and the recorded certbot argv, never on which branch ran.
Mutation-tested, all five mutants killed:
1. guard absent entirely (suite run with HAPROXY_MANAGER_DIR pointed at
main) -> 6 failures, incl. "example.com is still configured and still
served from this file".
2. refcount taken before the row is deleted -> 5 failures, incl. "the
last reference is gone - now it may be removed".
3. certbot guard removed, file guard kept -> 3 failures, incl.
[] != ['delete --cert-name example.com --non-interactive'].
4. lineage taken from the domain name instead of the cert path -> 3
failures, incl. 'delete --cert-name example.com' != 'delete
--cert-name www.example.com'.
5. file-unlink guard removed, certbot guard kept -> 4 failures, incl.
"two sites are still served from this bundle".
Other suites unchanged and green: test-config-rollback, test-cert-scripts,
test-stick-table-contract, test-runtime-map-contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shadowdaoandClaude Opus 5
b072786192