fix(ssl): retire SAN-overlapping certs, never publish expired lineages, report no-op renewals

brainjar.net served an expired certificate for two days (2026-10-07..09):
the old brain-jar.com bundle (CN brain-jar.com, SANs incl. all eight
brainjar names) stayed in /etc/haproxy/certs and won the SNI match over the
valid brainjar.net bundle, because superseded-cert cleanup matched on CN
only, and renew-certificates.sh re-published the expired lineage every 12h.

- Retirement now matches on ANY shared name (CN or SAN). The overlapping pem
  is moved to cert-retired/ and its lineage's live/ dir and renewal conf to
  live-retired/ and renewal-retired/ (after HAProxy loads the new bundle).
  Nothing is deleted any more: `certbot delete` is gone from this path.
- Shared-PEM safety: a candidate is kept when it is still the only valid
  certificate for a name in the domains table (retirement_blockers()). Expired
  certs never block; names the caller lists in the new optional
  `unreachable_names` body field do not block.
- /api/ssl/bundle passes --keep-until-expiring and reports `renewed`
  (fingerprint before vs after), `certbot_not_due` and the certificate's
  `not_after`, so WHP can stop recording certbot no-ops as renewals.
- Never publish an expired certificate: renew-certificates.sh and
  sync-certificates.sh skip expired lineages (ERROR-logged, counted
  separately), and cert_publish / publish_pem_bundle refuse expired leaves.
- VERSION 2026.10.3.

Tests: test-cert-write-safety.py 31 -> 47, test-cert-scripts.py 38 -> 44,
each new group verified to fail against a reverted fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
shadowdaoandClaude Opus 5.5 committed 2026-10-09 11:05:07 -07:00
1 parent e134b2b234
commit e58996d6c9
18 files changed
+1006 -112

No files matched your search

+1
View File
@@ -10,6 +10,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - moved to `cert-retired/`, `live-retired/`, `renewal-retired/`, never deleted - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
### Reading stick tables (and why it is easy to get silently wrong)
+1 -1
View File
@@ -1 +1 @@
2026.09.1
2026.10.3
+388 -101
View File
@@ -8,7 +8,7 @@ import socket
import psutil
import functools
import logging
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
import json
import ipaddress
import shutil
@@ -570,6 +570,19 @@ def backup_existing_pem(dest_path):
return None
def leaf_is_expired(path):
"""True only when openssl positively says the first certificate in `path`
has expired. Unreadable/missing openssl returns False: those cases are
rejected, with a better message, by validate_pem_bundle()."""
try:
proc = subprocess.run(
['openssl', 'x509', '-checkend', '0', '-noout', '-in', path],
capture_output=True, text=True, stdin=subprocess.DEVNULL)
except Exception:
return False
return proc.returncode == 1 and 'will expire' in (proc.stdout or '')
def publish_pem_bundle(dest_path, source_paths):
"""Publish cert+key as a combined PEM at dest_path, atomically.
@@ -611,6 +624,15 @@ def publish_pem_bundle(dest_path, source_paths):
raise CertificatePublishError(
f'assembled bundle for {dest_path} is not usable: {msg}')
# Never publish an expired certificate. HAProxy loads the crt directory
# wholesale and serves the first file that claims an SNI name, so an
# expired file in there can only ever shadow a valid one (brain-jar.com.pem
# did exactly that to brainjar.net on 2026-10-07).
if leaf_is_expired(source_paths[0]):
raise CertificatePublishError(
f'refusing to publish {dest_path}: the certificate in '
f'{source_paths[0]} has expired')
dest_dir = os.path.dirname(dest_path) or '.'
try:
os.makedirs(dest_dir, exist_ok=True)
@@ -972,132 +994,336 @@ def request_ssl():
log_operation('request_ssl', False, str(e))
return jsonify({'status': 'error', 'message': str(e)}), 500
def _quarantine_superseded_certs(keep_path, keep_lineage, bundle_names):
"""Move aside cert files that the just-issued bundle supersedes.
# ---------------------------------------------------------------------------
# Superseded-certificate retirement (incident 2026-10-07, brainjar.net)
# ---------------------------------------------------------------------------
# HAProxy loads /etc/haproxy/certs as a DIRECTORY and, when two files claim the
# same SNI name, serves whichever it loaded first. So a new bundle has no effect
# while an older file that also lists one of its names is still in there.
#
# brainjar.net served an EXPIRED certificate for two days for exactly that
# reason. The orchestrator re-bundled brainjar.net and its seven siblings
# without the dead brain-jar.com, but the cleanup that was meant to move the old
# file aside matched on the old certificate's CN only. Its CN was brain-jar.com,
# which was not in the new bundle, so brain-jar.com.pem (which also listed all
# eight brainjar names) stayed put and kept winning the SNI match. On top of
# that, renew-certificates.sh re-published the old lineage from live/ every 12
# hours, so moving the file alone would not have stuck.
#
# What retirement does now:
# * any OTHER pem/lineage that shares ANY name with the new bundle (CN or
# SAN) is a candidate, not only one whose CN is in the bundle;
# * a candidate is moved aside only if no name that this edge still serves
# would lose its last valid certificate (the shared-PEM hazard: one bundle
# routinely serves several sites' names, see domains_referencing_cert_path);
# * nothing is deleted. The pem goes to the cert-retired sibling directory and
# the lineage's live/<name> and renewal/<name>.conf go to live-retired/ and
# renewal-retired/, so certbot stops renewing it and renew-certificates.sh
# stops re-publishing it. Every move is logged and can be reversed with mv.
# (This used to `certbot delete` the lineage, which is irreversible.)
A `.pem` in /etc/haproxy/certs/ is "superseded" iff its certificate's CN
is one of the bundle's names AND the file isn't the bundle's own combined
file. We don't look at SANs of the OLD certs — being the CN is enough,
since that's what HAProxy SNI-matches against and what the file
convention names it after.
# Where certbot keeps its lineages. A module global so the tests can repoint
# it; find_certbot_live_dir() predates it and still hardcodes the live path.
LETSENCRYPT_DIR = '/etc/letsencrypt'
This used to os.remove() the file and immediately `certbot delete` the
lineage, both BEFORE anything had checked that the replacement bundle was
usable — destroying the only two copies of a working certificate on the
strength of a `cat` whose exit status nobody read. Now:
* the superseded file is MOVED to the cert backup directory instead of
unlinked, so an operator can put it back by hand;
* the certbot lineage is left alone here. Deleting it is irreversible
(archive, live and renewal config all go) and recovering means fresh,
rate-limited ACME orders, so it happens only in
_delete_superseded_lineages(), after HAProxy has actually loaded the
replacement.
def cert_retired_dir():
"""Where retired .pem files are moved. Sibling of the crt dir, never inside."""
return _cert_sibling_dir('cert-retired')
Moving still solves the problem the removal existed for: the old file no
longer sits in the crt directory shadowing the new bundle's SNI match.
Returns a summary dict for logging / API response. Entries in 'removed'
carry the lineage name for the deletion phase.
def parse_cert_text(text):
"""Parse `openssl x509 -noout -subject -nameopt multiline -enddate
-fingerprint -sha256 -ext subjectAltName` output.
Returns {'cn', 'names', 'not_after', 'fingerprint'}; names is the lower-cased
set of the CN plus every DNS SAN. not_after is a UTC epoch or None.
"""
summary = {'removed': [], 'errors': [], 'skipped': []}
cn = None
names = set()
not_after = None
fingerprint = None
in_san = False
for raw in text.splitlines():
line = raw.strip()
if line.startswith('commonName') and '=' in line and cn is None:
cn = line.split('=', 1)[1].strip().lower()
continue
if line.startswith('notAfter='):
try:
not_after = int(datetime.strptime(
' '.join(line.split('=', 1)[1].split()),
'%b %d %H:%M:%S %Y %Z').replace(
tzinfo=timezone.utc).timestamp())
except ValueError:
not_after = None
continue
if '=' in line and line.lower().split('=', 1)[0].endswith('fingerprint'):
fingerprint = line.split('=', 1)[1].strip().upper()
continue
if line.startswith('X509v3 Subject Alternative Name'):
in_san = True
continue
if in_san:
for part in line.split(','):
part = part.strip()
if part.startswith('DNS:'):
names.add(part[4:].strip().lower())
in_san = False
if cn:
names.add(cn)
return {'cn': cn, 'names': names, 'not_after': not_after,
'fingerprint': fingerprint}
def read_cert_info(path):
"""parse_cert_text() for the first certificate in `path`, or None."""
try:
proc = subprocess.run(
['openssl', 'x509', '-in', path, '-noout', '-subject',
'-nameopt', 'multiline', '-enddate', '-fingerprint', '-sha256',
'-ext', 'subjectAltName'],
capture_output=True, text=True, stdin=subprocess.DEVNULL)
except Exception:
return None
if proc.returncode != 0:
return None
info = parse_cert_text(proc.stdout)
return info if info['names'] else None
def cert_name_covers(cert_name, host):
"""Does certificate name `cert_name` (maybe `*.x`) cover hostname `host`?"""
cert_name = (cert_name or '').lower()
host = (host or '').lower()
if cert_name == host:
return True
if cert_name.startswith('*.') and not host.startswith('*.'):
suffix = cert_name[1:] # ".example.com"
head = host[:-len(suffix)] if host.endswith(suffix) else ''
return bool(head) and '.' not in head
return False
def _cert_covers(names, host):
return any(cert_name_covers(n, host) for n in names)
def _served_names():
"""Every hostname this edge has a route for (the SQLite domains table)."""
try:
with sqlite3.connect(DB_FILE) as conn:
return {row[0].lower() for row in
conn.execute('SELECT domain FROM domains').fetchall()
if row[0]}
except Exception as e:
# Unknown served set -> treat every name as served, i.e. never retire a
# still-valid certificate on the strength of a failed query.
logger.error("Could not read the domains table for retirement checks: %s", e)
return None
def retirement_blockers(candidate, others, bundle_names, served_names,
unreachable_names, now=None):
"""Names that would lose their LAST valid certificate if `candidate` went.
candidate/others are read_cert_info() dicts. `others` are the certificates
that would remain in the crt directory. Pure, so it is unit-testable.
* An expired candidate blocks nothing: it is not a valid certificate for
anyone, and while it sits in the crt dir it can only shadow a good one.
* A name the caller asserts is not served here (unreachable_names - DNS
points elsewhere or nowhere) blocks nothing.
* A name with no route on this edge blocks nothing.
* served_names=None (could not read the DB) means "assume everything is
served", so a valid candidate is kept unless the bundle covers it all.
"""
now = time.time() if now is None else now
if candidate.get('not_after') is None:
return ['<unreadable expiry>']
if candidate['not_after'] <= now:
return []
unreachable = {n.lower() for n in (unreachable_names or ())}
bundle = {n.lower() for n in bundle_names}
if served_names is None:
candidates_to_check = set(candidate['names'])
else:
candidates_to_check = {s for s in served_names
if _cert_covers(candidate['names'], s)}
blockers = []
for name in sorted(candidates_to_check):
if name in unreachable or _cert_covers(bundle, name):
continue
covered_elsewhere = any(
o is not candidate
and o.get('fingerprint') != candidate.get('fingerprint')
and o.get('not_after') is not None and o['not_after'] > now
and _cert_covers(o['names'], name)
for o in others)
if not covered_elsewhere:
blockers.append(name)
return blockers
def _overlaps_bundle(info, bundle_names):
return any(_cert_covers(info['names'], b) for b in bundle_names) or \
any(n in {b.lower() for b in bundle_names} for n in info['names'])
def _unique_dest(directory, name):
dest = os.path.join(directory, name)
if not os.path.exists(dest):
return dest
stamp = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')
dest = os.path.join(directory, f'{name}.{stamp}')
n = 1
while os.path.exists(dest):
dest = os.path.join(directory, f'{name}.{stamp}.{n}')
n += 1
return dest
def _quarantine_superseded_certs(keep_path, keep_lineage, bundle_names,
unreachable_names=()):
"""Move aside .pem files in the crt dir that overlap the new bundle.
Runs after the new bundle is published and before HAProxy reloads, so the
replacement is never the only thing we have. See the block comment above
for the rules. Returns {'removed', 'kept', 'skipped', 'errors'}.
"""
summary = {'removed': [], 'kept': [], 'skipped': [], 'errors': []}
if not os.path.isdir(SSL_CERTS_DIR):
return summary
keep_basename = os.path.basename(keep_path)
served = _served_names()
# Inventory once; `remaining` shrinks as files are retired so the coverage
# check for the next candidate sees what will actually be left.
inventory = {}
for fname in sorted(os.listdir(SSL_CERTS_DIR)):
if not fname.endswith('.pem'):
continue
info = read_cert_info(os.path.join(SSL_CERTS_DIR, fname))
if info is None:
summary['skipped'].append({'file': fname, 'reason': 'openssl could not read it'})
continue
inventory[fname] = info
remaining = dict(inventory)
for fname, info in inventory.items():
if fname == keep_basename:
continue
fpath = os.path.join(SSL_CERTS_DIR, fname)
try:
cn_proc = subprocess.run(
['openssl', 'x509', '-in', fpath, '-noout', '-subject', '-nameopt', 'multiline'],
capture_output=True, text=True
)
if cn_proc.returncode != 0:
summary['skipped'].append({'file': fname, 'reason': 'openssl read failed'})
continue
# `-nameopt multiline` lays out the subject one RDN per line; CN is
# the row matching `commonName`. Robust against unusual subject orderings.
cn = None
for line in cn_proc.stdout.splitlines():
line = line.strip()
if line.startswith('commonName'):
# format: "commonName = example.com"
parts = line.split('=', 1)
if len(parts) == 2:
cn = parts[1].strip()
break
if not cn:
summary['skipped'].append({'file': fname, 'reason': 'no CN found'})
continue
except Exception as e:
summary['skipped'].append({'file': fname, 'reason': f'inspect failed: {e}'})
if not _overlaps_bundle(info, bundle_names):
continue
if cn not in bundle_names:
continue # not superseded — different domain group
# This file's CN is now part of our new bundle — supersede it.
lineage_name = fname[:-len('.pem')]
if lineage_name == keep_lineage:
# Defensive: shouldn't happen because of keep_basename check, but
# don't accidentally drop the lineage we just wrote.
others = [i for f, i in remaining.items() if f != fname]
blockers = retirement_blockers(info, others, bundle_names, served,
unreachable_names)
if blockers:
logger.warning(
"Not retiring %s although it overlaps the new %s bundle: it is "
"still the only valid certificate for %s", fname, keep_lineage,
', '.join(blockers))
summary['kept'].append({'file': fname, 'blockers': blockers})
continue
try:
os.makedirs(cert_backup_dir(), exist_ok=True)
quarantine_path = os.path.join(cert_backup_dir(), fname)
# Move, not unlink: out of the crt directory (so it stops shadowing
# the new bundle) but still on disk for manual recovery.
shutil.move(fpath, quarantine_path)
os.makedirs(cert_retired_dir(), exist_ok=True)
dest = _unique_dest(cert_retired_dir(), fname)
shutil.move(os.path.join(SSL_CERTS_DIR, fname), dest)
remaining.pop(fname, None)
logger.warning(
"Retired superseded certificate %s (names: %s; expires %s) -> %s "
"because it overlaps the new %s bundle",
fname, ', '.join(sorted(info['names'])),
datetime.fromtimestamp(info['not_after'], timezone.utc).isoformat()
if info['not_after'] else 'unknown', dest, keep_lineage)
summary['removed'].append({
'file': fname,
'cn': cn,
'lineage': lineage_name,
'moved_to': quarantine_path,
'lineage_deleted': False,
'cn': info['cn'],
'names': sorted(info['names']),
'expired': bool(info['not_after'] and info['not_after'] <= time.time()),
'moved_to': dest,
})
except Exception as e:
summary['errors'].append({'file': fname, 'error': str(e)})
return summary
def _delete_superseded_lineages(summary):
"""`certbot delete` the lineages quarantined by _quarantine_superseded_certs().
def _retire_superseded_lineages(keep_lineage, bundle_names, unreachable_names=(),
keep_fingerprint=None):
"""Move overlapping certbot lineages out of live/ and renewal/.
IRREVERSIBLE: certbot removes the lineage's archive, live symlinks and
renewal config. If it turns out we needed that certificate, the only way
back is a fresh ACME order, which Let's Encrypt rate-limits — an outage
measured in hours. So this is gated hardest of anything in this module: it
runs only after the replacement bundle has been assembled, validated,
published, AND loaded by a HAProxy that reloaded successfully.
Mutates `summary` in place. Best-effort: some files have no corresponding
lineage (e.g. self-signed dev certs) and a failure here is harmless — a
dead lineage merely wastes a renewal attempt on the next 12h cron tick.
Same overlap and safety rules as _quarantine_superseded_certs(), evaluated
against the pem files that are left in the crt dir. Runs only after HAProxy
has loaded the new bundle. Moving live/<x> to live-retired/<x> (same depth)
keeps its relative ../../archive symlinks valid, so `mv` back restores it.
"""
for entry in summary.get('removed', []):
lineage_name = entry.get('lineage')
if not lineage_name:
summary = {'retired': [], 'kept': [], 'errors': []}
live_root = os.path.join(LETSENCRYPT_DIR, 'live')
if not os.path.isdir(live_root):
return summary
served = _served_names()
remaining_pems = []
if os.path.isdir(SSL_CERTS_DIR):
for fname in sorted(os.listdir(SSL_CERTS_DIR)):
if fname.endswith('.pem'):
info = read_cert_info(os.path.join(SSL_CERTS_DIR, fname))
if info is not None:
info['_file'] = fname
remaining_pems.append(info)
for lineage in sorted(os.listdir(live_root)):
lineage_dir = os.path.join(live_root, lineage)
if lineage == keep_lineage or not os.path.isdir(lineage_dir):
continue
cert_file = os.path.join(lineage_dir, 'cert.pem')
if not os.path.exists(cert_file):
cert_file = os.path.join(lineage_dir, 'fullchain.pem')
info = read_cert_info(cert_file)
if info is None or not _overlaps_bundle(info, bundle_names):
continue
if keep_fingerprint and info.get('fingerprint') == keep_fingerprint:
continue
# The lineage's own published pem is not "another" certificate, even if
# its fingerprint drifted from the lineage (renewed, not republished).
others = [p for p in remaining_pems if p['_file'] != f'{lineage}.pem']
blockers = retirement_blockers(info, others, bundle_names, served,
unreachable_names)
if blockers:
logger.warning(
"Not retiring certbot lineage %s although it overlaps the new %s "
"bundle: it is still the only valid certificate for %s",
lineage, keep_lineage, ', '.join(blockers))
summary['kept'].append({'lineage': lineage, 'blockers': blockers})
continue
entry = {'lineage': lineage, 'names': sorted(info['names'])}
try:
cb_proc = subprocess.run(
['certbot', 'delete', '--cert-name', lineage_name, '-n'],
capture_output=True, text=True
)
entry['lineage_deleted'] = (cb_proc.returncode == 0)
if cb_proc.returncode != 0:
entry['certbot_stderr'] = (cb_proc.stderr or '').strip()[:200]
live_dest_root = os.path.join(LETSENCRYPT_DIR, 'live-retired')
os.makedirs(live_dest_root, exist_ok=True)
live_dest = _unique_dest(live_dest_root, lineage)
shutil.move(lineage_dir, live_dest)
entry['live_moved_to'] = live_dest
conf = os.path.join(LETSENCRYPT_DIR, 'renewal', f'{lineage}.conf')
if os.path.exists(conf):
conf_dest_root = os.path.join(LETSENCRYPT_DIR, 'renewal-retired')
os.makedirs(conf_dest_root, exist_ok=True)
conf_dest = _unique_dest(conf_dest_root, f'{lineage}.conf')
shutil.move(conf, conf_dest)
entry['renewal_moved_to'] = conf_dest
logger.warning(
"Retired certbot lineage %s (names: %s) because it overlaps the "
"new %s bundle: live -> %s, renewal conf -> %s. Archive left in "
"place; reverse with mv to restore.", lineage,
', '.join(entry['names']), keep_lineage, entry['live_moved_to'],
entry.get('renewal_moved_to', '(none)'))
summary['retired'].append(entry)
except Exception as e:
entry['certbot_error'] = str(e)
entry['error'] = str(e)
summary['errors'].append(entry)
return summary
@app.route('/api/ssl/bundle', methods=['POST'])
@require_api_key
def request_ssl_bundle():
@@ -1108,7 +1334,18 @@ def request_ssl_bundle():
/api/ssl calls when a site has multiple domains.
Body:
{"primary": "example.com", "sans": ["www.example.com", ...]}
{"primary": "example.com", "sans": ["www.example.com", ...],
"unreachable_names": ["dead.example.com", ...]} # optional
unreachable_names: names the caller (WHP's orchestrator) has established do
not resolve to this host. They are left out of the bundle by the caller;
here they only tell superseded-certificate retirement that an older cert's
coverage of them is not worth keeping (see retirement_blockers()).
The response says whether certbot actually issued ("renewed") or found the
existing lineage not yet due and did nothing, and carries the published
certificate's not_after, so the caller can schedule from the real expiry
instead of recording a no-op as a renewal.
The cert lineage uses --cert-name <primary>, so renewal under the same
name doesn't proliferate -0001/-0002 dirs (the issue we hit with the
@@ -1146,6 +1383,13 @@ def request_ssl_bundle():
return jsonify({'status': 'error', 'message': f'invalid SAN: {s!r}'}), 400
cleaned_sans.append(s)
unreachable_names = data.get('unreachable_names') or []
if not isinstance(unreachable_names, list) or not all(
isinstance(n, str) for n in unreachable_names):
return jsonify({'status': 'error',
'message': '"unreachable_names" must be a list of strings'}), 400
unreachable_names = [n.strip().lower() for n in unreachable_names if n.strip()]
seen = {primary}
names = [primary]
for s in cleaned_sans:
@@ -1160,8 +1404,11 @@ def request_ssl_bundle():
'message': f'Too many SANs ({len(names)}); Let\'s Encrypt limit is 100',
}), 400
# --keep-until-expiring is what -n does anyway when the lineage exists with
# the same names; it is spelled out so the no-op case is deliberate, and it
# is detected below rather than reported as an issuance.
cmd = [
'certbot', 'certonly', '-n', '--standalone',
'certbot', 'certonly', '-n', '--standalone', '--keep-until-expiring',
'--preferred-challenges', 'http', '--http-01-port=8688',
'--cert-name', primary,
]
@@ -1172,6 +1419,11 @@ def request_ssl_bundle():
# Defensive: clear any stale lock left by a SIGKILLed prior run.
clear_stale_certbot_locks()
# fullchain.pem starts with the leaf, so it identifies the certificate.
before_dir = find_certbot_live_dir(primary)
before_info = read_cert_info(os.path.join(before_dir, 'fullchain.pem')) \
if before_dir else None
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
stderr_excerpt = (result.stderr or '').strip()[:800]
@@ -1196,6 +1448,20 @@ def request_ssl_bundle():
key_path = os.path.join(live_dir, 'privkey.pem')
combined_path = f'{SSL_CERTS_DIR}/{primary}.pem'
# Did certbot issue, or did it find the lineage not yet due? Measured by
# the certificate itself (fingerprint before vs after), with certbot's
# own wording as a second signal.
new_info = read_cert_info(cert_path)
output = (result.stdout or '') + (result.stderr or '')
certbot_not_due = ('not yet due for renewal' in output.lower()
or 'no action taken' in output.lower())
if before_info is None:
renewed = new_info is not None
elif new_info is None:
renewed = not certbot_not_due
else:
renewed = new_info.get('fingerprint') != before_info.get('fingerprint')
os.makedirs(SSL_CERTS_DIR, exist_ok=True)
try:
publish_pem_bundle(combined_path, [cert_path, key_path])
@@ -1235,8 +1501,9 @@ def request_ssl_bundle():
# only copy we have.
cleanup_summary = _quarantine_superseded_certs(
keep_path=combined_path,
keep_lineage=primary,
keep_lineage=os.path.basename(live_dir),
bundle_names=set(names),
unreachable_names=unreachable_names,
)
# Raises if the config does not validate or HAProxy does not reload;
@@ -1249,26 +1516,46 @@ def request_ssl_bundle():
logger.critical(
"HAProxy did not reload after publishing the bundle for "
"%s. The superseded certificate files were moved to %s and "
"their certbot lineages were NOT deleted, so they can be "
"their certbot lineages were NOT retired, so they can be "
"restored by hand: %s",
primary, cert_backup_dir(),
primary, cert_retired_dir(),
[e['file'] for e in cleanup_summary['removed']])
raise
_delete_superseded_lineages(cleanup_summary)
# Only now that HAProxy has loaded the new bundle: stop certbot renewing
# (and renew-certificates.sh re-publishing) overlapping old lineages.
lineage_summary = _retire_superseded_lineages(
keep_lineage=os.path.basename(live_dir),
bundle_names=set(names),
unreachable_names=unreachable_names,
keep_fingerprint=new_info.get('fingerprint') if new_info else None,
)
cleanup_summary['lineages'] = lineage_summary
not_after = new_info.get('not_after') if new_info else None
log_operation(
'request_ssl_bundle', True,
f'SSL bundle issued for {primary} covering {len(names)} names; '
f'cleaned up {len(cleanup_summary["removed"])} superseded lineage(s)'
f'SSL bundle {"issued" if renewed else "not yet due (existing certificate kept)"} '
f'for {primary} covering {len(names)} names; retired '
f'{len(cleanup_summary["removed"])} superseded pem(s) and '
f'{len(lineage_summary["retired"])} lineage(s)'
)
return jsonify({
'status': 'success',
'primary': primary,
'names': names,
'cert_path': combined_path,
'renewed': bool(renewed),
'certbot_not_due': bool(certbot_not_due),
'not_after': not_after,
'not_after_iso': datetime.fromtimestamp(not_after, timezone.utc).isoformat()
if not_after else None,
'fingerprint': new_info.get('fingerprint') if new_info else None,
'cleanup': cleanup_summary,
'message': f'Bundled certificate obtained for {len(names)} names',
'message': (f'Bundled certificate obtained for {len(names)} names'
if renewed else
f'Existing certificate for {len(names)} names is not yet '
'due for renewal; nothing was issued'),
})
except Exception as e:
log_operation('request_ssl_bundle', False, str(e))
+30
View File
@@ -203,6 +203,31 @@ cert_bundle_valid() {
return 0
}
# cert_is_expired CERT_FILE
#
# Returns 0 only when openssl positively reports that the first certificate in
# CERT_FILE has expired. Anything else (not expired, unreadable, no openssl)
# returns 1; unreadable material is rejected with a better message by
# cert_bundle_valid().
#
# Why: HAProxy loads the crt directory wholesale and serves the FIRST file that
# claims an SNI name. An expired bundle in there is never useful and can only
# shadow a valid one - on 2026-10-07 renew-certificates.sh kept re-publishing
# the expired brain-jar.com lineage every 12 hours, and HAProxy served it for
# brainjar.net instead of the valid brainjar.net bundle next to it.
cert_is_expired() {
local file="$1" out
[ -f "$file" ] || return 1
command -v openssl >/dev/null 2>&1 || return 1
out="$(openssl x509 -checkend 0 -noout -in "$file" 2>/dev/null </dev/null)"
[ $? -eq 1 ] && [[ "$out" == *"will expire"* ]]
}
# cert_not_after CERT_FILE - the notAfter date string, or empty.
cert_not_after() {
openssl x509 -enddate -noout -in "$1" 2>/dev/null </dev/null | sed 's/^notAfter=//'
}
# cert_publish CERT_FILE KEY_FILE DEST_FILE
#
# Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
@@ -227,6 +252,11 @@ cert_publish() {
log_error "cert_publish: private key $key_file is missing or empty"
return 1
fi
# (a2) never publish an expired certificate (see cert_is_expired).
if cert_is_expired "$cert_file"; then
log_error "cert_publish: certificate $cert_file expired ($(cert_not_after "$cert_file")) - refusing to publish $dest_file"
return 1
fi
# (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
# scans wholesale.
+11 -2
View File
@@ -65,13 +65,22 @@ fi
# Copy certificates for each domain
UPDATED=0
FAILED=0
SKIPPED_EXPIRED=0
while read -r domain; do
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
# Never publish an expired lineage. It cannot help any client and,
# because HAProxy serves the first file claiming an SNI name, it can
# shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
# Counted separately from FAILED: a dead lineage that keeps expiring is
# reported on every run (ERROR log) without making every run "fail".
log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
# Assemble in a staging dir and rename into place. NEVER redirect into
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -89,7 +98,7 @@ while read -r domain; do
fi
done <<< "$DOMAINS"
log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
log_info "Certificate update completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped"
# Reload HAProxy if any certificates were updated
if [ $UPDATED -gt 0 ]; then
+11 -2
View File
@@ -47,13 +47,22 @@ fi
# Copy certificates for each domain
UPDATED=0
FAILED=0
SKIPPED_EXPIRED=0
while read -r domain; do
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
# Never publish an expired lineage. It cannot help any client and,
# because HAProxy serves the first file claiming an SNI name, it can
# shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
# Counted separately from FAILED: a dead lineage that keeps expiring is
# reported on every run (ERROR log) without making every run "fail".
log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
# Assemble in a staging dir and rename into place. NEVER redirect into
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
@@ -71,7 +80,7 @@ while read -r domain; do
fi
done <<< "$DOMAINS"
log_info "Certificate sync completed: $UPDATED updated, $FAILED failed"
log_info "Certificate sync completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped"
# Reload HAProxy if any certificates were updated
if [ $UPDATED -gt 0 ]; then
+72
View File
@@ -73,6 +73,15 @@ LIB = os.path.join(SCRIPTS_DIR, 'cert-publish-lib.sh')
BROKEN_TOKEN = '__BROKEN__'
DOMAIN = 'test.example.com'
# Real SAN certificates (see scripts/test-fixtures/san-certs/). expired_old is
# the brain-jar.com shape: a multi-SAN certificate that expired in 2020.
SAN_FIXTURES = os.path.join(SCRIPTS_DIR, 'test-fixtures', 'san-certs')
def san_fixture(name):
with open(os.path.join(SAN_FIXTURES, name)) as fh:
return fh.read()
# --- test key material -------------------------------------------------------
# openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \
# -subj /CN=test.example.com
@@ -618,6 +627,42 @@ class CertScriptBehaviour:
self.assertFalse(self.reload_attempted())
# -- never publish an expired certificate (brainjar.net, 2026-10-07) ----
def test_expired_lineage_is_skipped_and_reported(self):
"""renew-certificates.sh used to re-publish EVERY live/ lineage every 12h,
expired or not; the expired brain-jar.com bundle then shadowed the valid
brainjar.net bundle for the names they shared."""
dead = os.path.join(self.le_live, 'brain-jar.com')
os.makedirs(dead)
write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
result = self.run_it()
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertFalse(
os.path.exists(os.path.join(self.certs_dir, 'brain-jar.com.pem')),
'an expired lineage was published into the crt directory')
# The healthy lineage next to it is still published.
self.assertEqual(read(self.live_pem), NEW_BUNDLE)
self.assertIn('Skipping EXPIRED certificate lineage brain-jar.com', self.logs())
self.assertIn('1 updated, 0 failed, 1 expired lineage(s) skipped', self.logs())
self.assertIn('brain-jar.com', read(self.error_log),
'the skip must land in the monitored error log')
def test_expired_lineage_does_not_replace_an_existing_pem(self):
dead = os.path.join(self.le_live, 'brain-jar.com')
os.makedirs(dead)
write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
existing = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'), OTHER_BUNDLE)
self.run_it()
self.assertEqual(read(existing), OTHER_BUNDLE,
'an expired lineage overwrote the file in the crt dir')
class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture):
SCRIPT = 'renew-certificates.sh'
@@ -638,6 +683,33 @@ class TestCertPublishLibrary(CertScriptFixture):
path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0)
def test_cert_is_expired_reports_both_directions(self):
expired = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
valid = write(os.path.join(self.tmp, 'valid.crt'), san_fixture('new_bundle.crt'))
self.assertEqual(0, self.call('cert_is_expired "$1"', expired).returncode,
'an expired certificate was not recognised as expired')
self.assertEqual(1, self.call('cert_is_expired "$1"', valid).returncode,
'a valid certificate was reported as expired')
self.assertEqual(1, self.call('cert_is_expired "$1"',
os.path.join(self.tmp, 'missing')).returncode)
def test_cert_publish_refuses_an_expired_certificate(self):
cert = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
key = write(os.path.join(self.tmp, 'expired.key'), san_fixture('expired_old.key'))
dest = os.path.join(self.certs_dir, 'brain-jar.com.pem')
result = self.call('cert_publish "$1" "$2" "$3"', cert, key, dest)
self.assert_rejected(result, because='expired')
self.assertFalse(os.path.exists(dest), 'expired certificate was published')
# And the same material with a valid certificate IS publishable, so the
# refusal above is about expiry, not about the fixture.
good_cert = write(os.path.join(self.tmp, 'good.crt'), san_fixture('new_bundle.crt'))
good_key = write(os.path.join(self.tmp, 'good.key'), san_fixture('new_bundle.key'))
ok = self.call('cert_publish "$1" "$2" "$3"', good_cert, good_key, dest)
self.assertEqual(0, ok.returncode, ok.stdout + ok.stderr)
self.assertTrue(os.path.exists(dest))
def test_empty_and_missing_bundles_rejected(self):
empty = write(os.path.join(self.tmp, 'empty.pem'), '')
self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),
+260 -6
View File
@@ -254,6 +254,9 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
# certbot stub: succeeds, announces a renewal, and records every invocation so
# tests can assert that `certbot delete` did or did not run.
# TEST_CERTBOT_INSTALL_SRC/_DIR: on certonly, copy fullchain.pem + privkey.pem
# from SRC into DIR (a real issuance changes the lineage's certificate).
# TEST_CERTBOT_NOT_DUE: on certonly, print certbot's "not yet due" wording.
FAKE_CERTBOT = textwrap.dedent("""\
#!/bin/sh
if [ -n "$TEST_CERTBOT_LOG" ]; then
@@ -262,6 +265,14 @@ FAKE_CERTBOT = textwrap.dedent("""\
case "$1" in
renew) echo "Congratulations, all renewals succeeded" ;;
delete) [ -n "$TEST_CERTBOT_DELETE_FAILS" ] && exit 1 ;;
certonly)
if [ -n "$TEST_CERTBOT_INSTALL_SRC" ]; then
mkdir -p "$TEST_CERTBOT_INSTALL_DIR"
cp "$TEST_CERTBOT_INSTALL_SRC/fullchain.pem" "$TEST_CERTBOT_INSTALL_SRC/privkey.pem" "$TEST_CERTBOT_INSTALL_DIR/"
fi
if [ -n "$TEST_CERTBOT_NOT_DUE" ]; then
echo "Certificate not yet due for renewal; no action taken."
fi ;;
esac
exit 0
""")
@@ -331,6 +342,7 @@ class CertPublishTestCase(unittest.TestCase):
'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'),
'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'),
'SSL_CERTS_DIR': self.certs,
'LETSENCRYPT_DIR': os.path.join(self.tmp, 'letsencrypt'),
'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'),
'API_KEY': None,
}
@@ -574,7 +586,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
@FIX_ONLY
@NEEDS_OPENSSL
def test_lineage_is_deleted_only_after_haproxy_loads_the_bundle(self):
def test_lineage_is_untouched_when_haproxy_does_not_load_the_bundle(self):
"""A reload failure must leave the lineage intact and the file recoverable."""
old_path = self._setup_supersede(broken=False)
# Make generate_config() produce a config the validator rejects, so the
@@ -591,7 +603,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
[], self.certbot_deletes(),
'the lineage must not be deleted when HAProxy did not reload')
self.assertFalse(os.path.exists(old_path))
quarantined = os.path.join(hm.cert_backup_dir(),
quarantined = os.path.join(hm.cert_retired_dir(),
os.path.basename(old_path))
self.assertTrue(os.path.exists(quarantined),
'the superseded file must be recoverable by hand')
@@ -599,8 +611,9 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
@FIX_ONLY
@NEEDS_OPENSSL
def test_successful_bundle_still_supersedes_and_deletes(self):
"""The cleanup must still do its job on the happy path."""
def test_successful_bundle_still_supersedes_without_deleting(self):
"""The cleanup must still do its job on the happy path - by moving, never
by `certbot delete` (irreversible; recovery is a rate-limited ACME order)."""
old_path = self._setup_supersede(broken=False)
resp = self.client.post('/api/ssl/bundle', json={
@@ -612,12 +625,253 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
self.assertFalse(os.path.exists(old_path),
'the superseded file must leave the crt directory or '
'it keeps shadowing the new bundle')
self.assertEqual(['delete --cert-name www.test.example.com -n'],
self.certbot_deletes())
self.assertTrue(os.path.exists(os.path.join(
hm.cert_retired_dir(), os.path.basename(old_path))),
'the superseded file must be moved aside, not destroyed')
self.assertEqual([], self.certbot_deletes(),
'retirement must never `certbot delete` anything')
self.assertTrue(self.edge_would_start())
self.assert_only_final_pems_in_certs_dir()
# --- SAN-overlap retirement (brainjar.net, 2026-10-07) -----------------------
SAN_FIXTURES = os.path.join(MODULE_DIR, 'scripts', 'test-fixtures', 'san-certs')
def san(name):
with open(os.path.join(SAN_FIXTURES, name)) as fh:
return fh.read()
FIX_ONLY_RETIRE = unittest.skipIf(
TESTING_FOREIGN_TREE and not hasattr(hm, 'retirement_blockers'),
'HAPROXY_MANAGER_DIR points at a tree without SAN-overlap retirement')
class TestRetirementRules(unittest.TestCase):
"""retirement_blockers() is the shared-PEM safety rule, tested directly."""
NOW = 2_000_000_000
def cert(self, names, days_left, fp):
return {'names': set(names), 'fingerprint': fp,
'not_after': self.NOW + days_left * 86400, 'cn': sorted(names)[0]}
@FIX_ONLY_RETIRE
def test_expired_candidate_never_blocks(self):
old = self.cert(['brain-jar.com', 'brainjar.net'], -2, 'A')
self.assertEqual([], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'brain-jar.com', 'brainjar.net'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_valid_candidate_that_is_last_cert_for_a_served_name_is_kept(self):
old = self.cert(['shared.example', 'brainjar.net', 'other.example'], 40, 'A')
self.assertEqual(['other.example'], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'brainjar.net', 'other.example'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_unreachable_names_do_not_block(self):
old = self.cert(['dead.example', 'brainjar.net'], 40, 'A')
self.assertEqual([], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'brainjar.net', 'dead.example'},
['dead.example'], now=self.NOW))
@FIX_ONLY_RETIRE
def test_name_covered_by_another_valid_cert_does_not_block(self):
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
other = self.cert(['other.example'], 60, 'B')
self.assertEqual([], hm.retirement_blockers(
old, [other], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_cover_by_an_expired_or_identical_cert_does_not_count(self):
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
expired = self.cert(['other.example'], -1, 'B')
same = self.cert(['other.example'], 60, 'A')
self.assertEqual(['other.example'], hm.retirement_blockers(
old, [expired, same], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_unknown_served_set_keeps_every_valid_name(self):
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
self.assertEqual(['other.example'], hm.retirement_blockers(
old, [], {'brainjar.net'}, None, [], now=self.NOW))
@FIX_ONLY_RETIRE
def test_wildcard_coverage(self):
self.assertTrue(hm.cert_name_covers('*.example.com', 'a.example.com'))
self.assertFalse(hm.cert_name_covers('*.example.com', 'a.b.example.com'))
self.assertFalse(hm.cert_name_covers('*.example.com', 'example.com'))
old = self.cert(['*.example.com', 'brainjar.net'], 40, 'A')
self.assertEqual(['shop.example.com'], hm.retirement_blockers(
old, [], {'brainjar.net'}, {'shop.example.com'}, [], now=self.NOW))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_read_cert_info_reads_sans_and_expiry(self):
tmp = tempfile.mkdtemp(prefix='cert-info-')
self.addCleanup(shutil.rmtree, tmp, True)
path = os.path.join(tmp, 'c.pem')
with open(path, 'w') as fh:
fh.write(san('expired_old.crt'))
info = hm.read_cert_info(path)
self.assertEqual({'brain-jar.com', 'www.brain-jar.com', 'brainjar.net',
'www.brainjar.net'}, info['names'])
self.assertEqual('brain-jar.com', info['cn'])
self.assertEqual(1585699200, info['not_after']) # 2020-04-01T00:00:00Z
self.assertTrue(info['fingerprint'])
class TestSanOverlapRetirement(CertPublishTestCase):
"""The new bundle must not stay shadowed by an older cert sharing a SAN."""
def setUp(self):
super().setUp()
self.le = os.path.join(self.tmp, 'letsencrypt')
self.new_src = os.path.join(self.tmp, 'issued', 'brainjar.net')
self.write(os.path.join(self.new_src, 'fullchain.pem'), san('new_bundle.crt'))
self.write(os.path.join(self.new_src, 'privkey.pem'), san('new_bundle.key'))
# certbot "issues" brainjar.net into live/ when called.
os.environ['TEST_CERTBOT_INSTALL_SRC'] = self.new_src
os.environ['TEST_CERTBOT_INSTALL_DIR'] = os.path.join(self.le_live, 'brainjar.net')
for name in ('brainjar.net', 'www.brainjar.net'):
self.add_domain(name, name.replace('.', '_'))
def old_lineage(self, lineage, fixture):
live = os.path.join(self.le_live, lineage)
self.write(os.path.join(live, 'fullchain.pem'), san(f'{fixture}.crt'))
self.write(os.path.join(live, 'cert.pem'), san(f'{fixture}.crt'))
self.write(os.path.join(live, 'privkey.pem'), san(f'{fixture}.key'))
conf = self.write(os.path.join(self.le, 'renewal', f'{lineage}.conf'),
f'# renewal conf for {lineage}\n')
pem = self.write(os.path.join(self.certs, f'{lineage}.pem'),
san(f'{fixture}.crt') + san(f'{fixture}.key'))
return live, conf, pem
def bundle(self, **extra):
body = {'primary': 'brainjar.net', 'sans': ['www.brainjar.net']}
body.update(extra)
resp = self.client.post('/api/ssl/bundle', json=body)
self.assertEqual(200, resp.status_code, resp.get_data(as_text=True))
return resp.get_json()
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_expired_overlapping_cert_with_a_different_cn_is_retired(self):
"""The incident: CN brain-jar.com, SANs include brainjar.net."""
self.add_domain('brain-jar.com', 'bjc')
self.add_domain('www.brain-jar.com', 'wbjc')
live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
data = self.bundle(unreachable_names=['brain-jar.com', 'www.brain-jar.com'])
self.assertFalse(os.path.exists(pem),
'the expired SAN-overlapping pem is still shadowing the bundle')
retired = [e['file'] for e in data['cleanup']['removed']]
self.assertEqual(['brain-jar.com.pem'], retired)
self.assertTrue(os.path.exists(os.path.join(hm.cert_retired_dir(), 'brain-jar.com.pem')))
# Lineage retired, not deleted: renew-certificates.sh can no longer
# re-publish it and certbot can no longer try to renew it.
self.assertFalse(os.path.exists(live))
self.assertFalse(os.path.exists(conf))
self.assertTrue(os.path.isdir(os.path.join(self.le, 'live-retired', 'brain-jar.com')))
self.assertTrue(os.path.exists(os.path.join(self.le, 'renewal-retired', 'brain-jar.com.conf')))
self.assertEqual([], self.certbot_deletes())
self.assertTrue(os.path.exists(os.path.join(self.certs, 'brainjar.net.pem')))
self.assertTrue(self.edge_would_start())
self.assert_only_final_pems_in_certs_dir()
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_expired_overlap_is_retired_even_without_the_unreachable_hint(self):
self.add_domain('brain-jar.com', 'bjc')
live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
self.bundle()
self.assertFalse(os.path.exists(pem))
self.assertFalse(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_shared_pem_still_serving_another_site_is_kept(self):
"""The shared-PEM hazard: never retire the only valid cert for a served name."""
self.add_domain('other-site.example', 'other_site')
live, conf, pem = self.old_lineage('shared-old.example', 'shared_old')
data = self.bundle()
self.assertTrue(os.path.exists(pem),
'retired the only valid certificate for other-site.example')
self.assertTrue(os.path.exists(live))
self.assertTrue(os.path.exists(conf))
kept = {e['file']: e['blockers'] for e in data['cleanup']['kept']}
self.assertEqual({'shared-old.example.pem': ['other-site.example']}, kept)
self.assertEqual(['other-site.example'],
data['cleanup']['lineages']['kept'][0]['blockers'])
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_valid_overlap_whose_other_names_are_unreachable_is_retired(self):
self.add_domain('dead.example', 'dead')
live, conf, pem = self.old_lineage('dead.example', 'dead_old')
self.bundle(unreachable_names=['dead.example'])
self.assertFalse(os.path.exists(pem))
self.assertFalse(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_valid_overlap_still_served_without_the_hint_is_kept(self):
"""Same files as above, but nobody said dead.example is gone: keep it."""
self.add_domain('dead.example', 'dead')
live, conf, pem = self.old_lineage('dead.example', 'dead_old')
self.bundle()
self.assertTrue(os.path.exists(pem))
self.assertTrue(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_non_overlapping_certificates_are_untouched(self):
live, conf, pem = self.old_lineage('unrelated.example', 'unrelated')
self.bundle()
self.assertTrue(os.path.exists(pem), 'retired a cert sharing no name with the bundle')
self.assertTrue(os.path.exists(live))
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_response_distinguishes_issuance_from_not_yet_due(self):
issued = self.bundle()
self.assertTrue(issued['renewed'], 'a fresh issuance was not reported as one')
self.assertEqual(4922899200, issued['not_after']) # 2126-01-01T00:00:00Z
# Lineage now exists with the same names; certbot keeps it.
del os.environ['TEST_CERTBOT_INSTALL_SRC']
os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
noop = self.bundle()
self.assertFalse(noop['renewed'], 'a certbot no-op was reported as a renewal')
self.assertTrue(noop['certbot_not_due'])
self.assertEqual(4922899200, noop['not_after'])
self.assertIn('--keep-until-expiring', self.certbot_invocations()[-1])
class TestExpiredCertificateIsNeverPublished(CertPublishTestCase):
@FIX_ONLY_RETIRE
@NEEDS_OPENSSL
def test_publisher_refuses_an_expired_leaf(self):
src = os.path.join(self.tmp, 'expired')
cert = self.write(os.path.join(src, 'fullchain.pem'), san('expired_old.crt'))
key = self.write(os.path.join(src, 'privkey.pem'), san('expired_old.key'))
live = self.publish_live_bundle('brain-jar.com')
with self.assertRaises(hm.CertificatePublishError) as ctx:
hm.publish_pem_bundle(live, [cert, key])
self.assertIn('expired', str(ctx.exception))
self.assertEqual(PREVIOUS_BUNDLE, self.read(live))
class TestBundleValidation(CertPublishTestCase):
"""What may and may not be published."""
@@ -0,0 +1,18 @@
-----BEGIN CERTIFICATE-----
MIIC9DCCAdygAwIBAgICEAMwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMZGVh
ZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
EwYDVQQDDAxkZWFkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCumL3T27a3D6fZbIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnO
abvnQNxEVqyYtTJOn9taitVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BU
SpW96rK40KVftzzDEUnH+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0
seOIdyP/n3T2cX1wsCa/F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9b
PBd2cBaBwTe/3TICu0RnvvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2
wzG4zlKU31fdLvQOzMGH1W5H91sBAgMBAAGjSDBGMCUGA1UdEQQeMByCDGRlYWQu
ZXhhbXBsZYIMYnJhaW5qYXIubmV0MB0GA1UdDgQWBBSPDJiv0Z2YpuloexOfvpIQ
UrWufTANBgkqhkiG9w0BAQsFAAOCAQEAJwBtN3Qw41XvYuSa8Mjn869QOTKEgkaR
EuNQ8WXVBOTRJdxaLHDunGO4ExjHHswCWzXkefYOoIClAU9qLnk9M667BSNxOcz0
VcG/DUMd0tO8X4Zm0zOMjt4VC7whgj7MdYtYVg2+Jhb4sMSx6OqmCUAD4X6ULSTn
lZK2holEC/H3bKt8Q6cecgeubgWojP1bWmWK7jQL5NRIzxQ0KzFxgAf3t2Z05nX8
loABkBiR7vDthkQseUW6RQMayDyzhVqORLt+gZ9D5T3hT1M7VaDc4Twr7UbWPFcR
GuQ7RwKplPIIbPe7+c91pm+agdSDjnV5HmKcAMCwf7bWGQ9/Xc0BFg==
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCumL3T27a3D6fZ
bIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnOabvnQNxEVqyYtTJOn9ta
itVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BUSpW96rK40KVftzzDEUnH
+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0seOIdyP/n3T2cX1wsCa/
F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9bPBd2cBaBwTe/3TICu0Rn
vvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2wzG4zlKU31fdLvQOzMGH
1W5H91sBAgMBAAECggEADUvYBFLn6gPWSZj55nxOqR+2Lmtwh04omgbu2fpWhPTE
ujgKd/23srnnMZXnR6qZQQIPGwJiOPnJuRjiu/PMZSj8j54bbwfA6XmbH9SCIzm0
la50UGBSkw0T50jBI76vICMm6PBkf0rT1oKN9VOd/qlkJoCUiFcGCh177Fbt07oY
R3iAKu7/+yEANX0iEcxbO330TPW3oI0ah5OryfKVoNvxpF8NUfV9pWY66APkOYMJ
zsYvIMvzV/KIgfrtyxJjSjgd7Ud8uictYG1BrZBXpIuGxuxyKUJ7OXOvMMVNnEME
2uzk/aL75UjpurJJU0LCywdUOGwRqS2q4PYQiTeDLQKBgQDjdywH7RODoYeH05Rd
sa2MkNx0Day1i3A4IBds55mzn8jxpT55HlqvHWhNA30kEw01H8+6sB7iMX04xQtS
u3N2opR1VlCvMJhGyRUYUtfu4NnAi/HSu7mCOFGAwaLclI7BAI2or2UrclrlUhGb
CYIBKfEr6Tn6b6AffWaOruCbtQKBgQDEf70R922fnJK+//QuiYoaESwJd3/k10yJ
nH0ZAgbnFcNLT9YVFxHvlt4RHGIotaBjfN+FKMn045jeekgeVdX+uF8+7FwEazAH
zOfxswOeKd8EOooqCD7n7cW3Ma6ev8oWmXwJsz2coRW7Z5zCGQWSzCEDB4yFU9Mt
0z6aN9SJnQKBgQCvBZSIqL+6KgD53zArA7sRDKnEVn1M13TKis7bij02/DyGyuoO
EfAfDBakI4oXxt0Dwm15LWkmDIRYuS0kRH3si96FhpvgoRiaZsDnjV7/PF3wjP3O
c4mjDCiSVOw2o8xP71gQspy62GveRc6sJ52zvY0Hh5aH/3oKoNfX8/F5fQKBgALQ
vbnDLMvrp2KbMjl0eQkUk+fy0V9LAmq6VkSEderbsfejRjL5KKWhKL9gA7wHliu2
v1EeG36pwCf34o6ktjM3Gw5AvrTFgGOsOOqKoROkD/D0kcQ2aAhUrM+KI7FcmB2o
x1EwQ0K6QCB2RJW3XBAVYqezdPz4DX+LlksJX781AoGBAISGOer3s4+oRjO/2VuT
hnMQrET3A0GTol8cgYL2N9wfx7jy5bhSfiJIX/dkh5IrPGYoBdWM87gcUgNQ8AZ7
il5dVeG80oc7KFU/QltZJu9VS32vwXnNsnrXupQ5Ee0ztTLciTMWU5jxvB0ZJHtK
StgYepr4IPDUaObqrPx4Jvfm
-----END PRIVATE KEY-----
@@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDGjCCAgKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwGDEWMBQGA1UEAwwNYnJh
aW4tamFyLmNvbTAeFw0yMDAxMDEwMDAwMDBaFw0yMDA0MDEwMDAwMDBaMBgxFjAU
BgNVBAMMDWJyYWluLWphci5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC9SJUMsK5HhVEATu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9s
i5KzgYp0K2z899wQAKuLzZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ
7jax2QWFMQVhyy6M5QTCHUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyN
N8eolukhvFxQNU3E7oo1jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSy
UEc+Gje6dJLZ9CdJtSQSNGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8
FHC7atJOe5EoaekaAXw5Tq+yB+ODAgMBAAGjbjBsMEsGA1UdEQREMEKCDWJyYWlu
LWphci5jb22CEXd3dy5icmFpbi1qYXIuY29tggxicmFpbmphci5uZXSCEHd3dy5i
cmFpbmphci5uZXQwHQYDVR0OBBYEFAWqbBh2PDUvqQzZoyh960cpuhcOMA0GCSqG
SIb3DQEBCwUAA4IBAQCaovtrXl56YcFlXWMrW/XZBe2/fXI14860qyiR5rb07lS6
CbKOsnRyQdxZ3cdfRwTN1UMmz5IPOZ6fhdo3IPaS2Wae1OdeupRaRWr1SAJtpVZ3
WWugDaumsWV5eWVxVTf93yDEkMERWtTGMviCKBeeHjzHIFiD/EjZznE87V5DYwuf
guOd1F4wXCMRFu8q7sBdPXjQ8rZMlQT+dQXzMST9Ee5PXiInmCAA5nT8vMInAwd2
rP6FxuHJB75ekYzs57CJ82BKKboOCESjUbm54wmVV5pBL305/EjxGf+VovNlRktg
t1znolntIIofE25UCbVNPdrtuDkZgKT3Rv8tCdTP
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC9SJUMsK5HhVEA
Tu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9si5KzgYp0K2z899wQAKuL
zZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ7jax2QWFMQVhyy6M5QTC
HUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyNN8eolukhvFxQNU3E7oo1
jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSyUEc+Gje6dJLZ9CdJtSQS
NGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8FHC7atJOe5EoaekaAXw5
Tq+yB+ODAgMBAAECggEABAeOCL71eYH4jKRMOBbXu98483u1CxnTEa8klnw9ZVES
JctJjY96B/EMwv+g/V+gxhgAkiK9VWfg6bEplOVDWgR9fRBoP57mVkOv3QPn5fcR
xlt9LQQZ+pULQRZoUAFEkVEkV1sfIBJmDLMufOnbAHCKa9+hVlrlCqu0lJgM8put
2bkpSAqwroHVfmD0SmKFAZggBhFwXFJzwmUf3WL+N25f4RZpJsX17FOJ216mlhQ5
edje6dcDpmko2VQtGJXYr8Ge02ZYeRyKGcTsgUZlarB4xUZq1pcnUUSy0Ez7LVbk
tKjPcUpZLlnAA/a2pDgHmwN1+fv/X+Wwv/vZaQMBTQKBgQDuZFyfDRSHCssHtQ4+
Ezxg4WEjwXkq58WxL0zTtLTcquZfQZ1VXgLwAylVwver0I4/UgCfW/AYY+2eLrba
FhuYOosd/3rsUQqf9y+TkSkhe1uVS2/1zjzC1G/5coleLddjzd+YBpEtVxfxDRbA
rVePY3GxkJiTrjAg5OETk7GYpwKBgQDLQ6bPhaocJPQmDi3hKyW8zPJ8wAJNyI0c
RWt9gjeidjmMvwCKtJbBDcV5V0oQ5SB2M5E49mT58B62totAD7FS18c4a4133haV
a1pLBX0U524eQgDzMJ6YA6N9bHdXOH4FZd+Pw90bHetbT1HcKpOdXv4SF/XITju5
Ro5xiymdxQKBgBB4fq2+iZyK8lwl9kXzrkhDSiLeSzDbspz7V6OI6LYWWLRcJDKx
vCI5FbIHNnV4g1t8nu5ju8LzpIc5UwjeXim2YcZA0I1jTAihW3gmn2Dg1QMLo0Co
bypG4zWo5DTNlgQfRUOLsJwhWFYfQWMuls4hYMmT4/bQIiMjrSldGvghAoGALpaB
7DyZobvCjdpgH6TREHX0+izUGlnmJd2vZmPrNnxccmrwx9neV789TLN6EWdCynTF
S0YGHNcxYvVx8yvlm7RDQCY+lYO8r2CcoGdQpPLqZVsBdrvlDbh/x2aN79rN35Ta
6cudqPJxmEsU2x6TGNXF7AW72r47qBqI2XNpBXUCgYAFQdHRRxa2E5rgV0WkL7PF
vXok3isAzpnKAsb5oxPZbhIdLo5EPUGCslsTE1Na8NBdUSoyjhgkMELEDx/ceGL7
IIlNrc/4j71+IahljzaX/D5b5qaW+RnoTVAlbMRR7L/ecbkoovhQR9d37QdoiD/t
B9si92+XB4HYDAjXFxTK3w==
-----END PRIVATE KEY-----
@@ -0,0 +1,18 @@
-----BEGIN CERTIFICATE-----
MIIC+DCCAeCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMYnJh
aW5qYXIubmV0MCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
EwYDVQQDDAxicmFpbmphci5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCyhL5YcFhPEekc/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdb
l4twhlkcyvyaEbcHIgLEgz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgU
QcAAvdAyP77jDF6lX11oCjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zz
XRVp4l2WyqEUJmKxcpppAnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU
0v78QfI+ueCw261UWZ+iXwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQ
dFOfKjH0QGInyByji9eAaXlmvCKtAgMBAAGjTDBKMCkGA1UdEQQiMCCCDGJyYWlu
amFyLm5ldIIQd3d3LmJyYWluamFyLm5ldDAdBgNVHQ4EFgQUdDxfZKX96j61mW7O
NiQpN4T+sZ4wDQYJKoZIhvcNAQELBQADggEBAJE1lP62caByZT7GkKtJLfOj84Ih
MeP751w2gdU8ZfiO8GxXXiR3ZjYzbJgsEi9LiNT9jqHeBf3pv/mJLjPVZ+U557ws
Gq3V5A3uKlRKGcejXlHaWnExhWqLyIaLTeLKstbJyiB/ja4yeTL2TYcOvvlu6weh
ERqwx0p6r7LSQ3nRfTdjEhNq3FVUmqQ8neBP/vXLp2hKr2ACZndGyQOiv0oZWzud
part6RMd8gRNq1Tctrl0YqRYoLRIddBrnTtUtac4FLqxLSBNc03/I9ZgF56Yf+Sl
M1ikmyDJ2ZYJFU6v42Og4/8o3fpq0ttP6pKPi8fInGaFQ+KXO1NWFf1ogD0=
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCyhL5YcFhPEekc
/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdbl4twhlkcyvyaEbcHIgLE
gz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgUQcAAvdAyP77jDF6lX11o
CjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zzXRVp4l2WyqEUJmKxcppp
AnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU0v78QfI+ueCw261UWZ+i
XwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQdFOfKjH0QGInyByji9eA
aXlmvCKtAgMBAAECggEAEQi8lvjFXGojmco3H7PVD7SZftLOFidVh9Fo8/JAe+2Z
LQM/hIjZ1i7sNverbQXMzAQMfXESS2Jx7Clv0mZ0aRgE0hq7wXrabyHfFUPOtqDa
6cPHsz8YvwAOmcQ6W3fBPYI6pzSfdV03rv1GGeCC7ETI3mIupwgUbANDuiDcVY1q
+VuK/hDzPJ5KslDHUNR42MgWacIftq99fV9jN0zQQw1EcyO8QwpXzTSEbdyiAkT+
n/tR9He2MYhgRhjyWPMaxmY6XQpv1h5ASq1yT1kI2oNTaRT3xvd40sLkbfwpZnxi
HnqzCkPptEpWeCLrme+r71+Hxu/+Bph7tPtcuK7jgQKBgQD8XCoPRiWbgCl6kM2G
E3GeM6fftTieTIs7sFVte7wutIKlsec4vizc94ipGaoT7yGEG0T8g9+5hAMbebc2
zzhxvJMB2s18Z/He+Ujv1L1AhBBNVY3zCAxfdq8/n5MGXppEdVRo9QOQndoBfuAV
Qw2Vp4bcy728n4T7sBugW0Ad7QKBgQC1F+utuiF8XI2yx0mX0+Fbty6o+odyM0eb
G/QQDlcGpY0K7V3WAFMESNfytig8OhidzRfWn8lNoqhd80SJfCuiYjiGJ8tjBzcp
+Vqrd/KAxJySJ3NJ2MwyAp7W3l3HAEZLGVYVCankoa4O3qR8YqaIZwc2nk5+8KnG
nBYFlSF/wQKBgQCqaRYh0E2O7wGTeXq/ohukJ9+hqf88rhWLTeyyDM8rfsuGVlb8
MzeaUzEed20pHt60FbIFDowh+yoaAJwez9mevCdNJlKCQrLKze1Zvn+5tDRJ9I7x
YSrxv/Rhc0FWGMX1moFQirtIib9wKRvqX6gpucHREWfjmpizWN2e8F+glQKBgAOp
/jy/nVnUzZEmcxi9Urv2Ut1cZE0tOG+s+x0wDRa3Vsk30EQ6MXPOvX/Nq7a8Pi9D
LQ5MoVgOyHVOLL7TYlagwiAEwhq41KbmwqzhemickcZrQevmlQsV/ny+dwtFuCZg
1tsKHQPhOZJXcuCJZnEwFHdIcLIk0GqSUee/ArSBAoGALOD+Opk+0j4ObNcqtrpP
UcEYqhOuAscbxLoUv53wnOvc7XzMG8JCK1y5LthB2Xanr1JSlheMRrvGxC+WlDmc
5v0Nz71cq5QQ1p8AXnosLRjuy/+edMvGIxxwFMEJBO2ra2Jizpdwd34bjpQ42+CQ
6pVwFQX0oet8SI5xWUWx7NU=
-----END PRIVATE KEY-----
@@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDGjCCAgKgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwHTEbMBkGA1UEAwwSc2hh
cmVkLW9sZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAw
WjAdMRswGQYDVQQDDBJzaGFyZWQtb2xkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQCBg7iYuvjhIG7nlFeja5Lyv2t2fdtNvxzUib7ygejC
+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpjo5/ym2KFZPwiSZoqKUugi9lCoKuL
2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMHYFwURjARRAku3a/GJUHQR0C572mW
mt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64fGeYSOaYOHEOp/VqIRlzsy7eGQh3
xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2JbqFxvijCLnBz5WsrS/S/VWOELNEfGr
84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLfWuSgp5hRAgMBAAGjYjBgMD8GA1Ud
EQQ4MDaCEnNoYXJlZC1vbGQuZXhhbXBsZYIMYnJhaW5qYXIubmV0ghJvdGhlci1z
aXRlLmV4YW1wbGUwHQYDVR0OBBYEFP+AdZ9MkBmFjAQTgLnWZxbJtnvkMA0GCSqG
SIb3DQEBCwUAA4IBAQA32Xl/1IrOyESd2UQcWp/fuHvkivLgamJrheKvKP0Y0vMV
atRu0D0GhqMOr79OT60H/H1FFW7bWQ+UPcZpnuXec40/PcrHxwaSoiLoIUVWhB4t
BOhtTDReX29gEjnB7OqBCJu4ub3HnpFTflUZd7n5A4EcetBsowc5EZ9UAtfvkgGD
nK9ONvNVGurk/V8xgtMJ877ZK1fn+UtgvIWp/F1Gc/taLhe+oCZOPkOqvvqUnS8q
QyIhqmMWRUkBP+jwRA4TP4pj7OH38VY/Ae408Xe4Y97GF7NbOjAL0bkX/bHkLnSG
WzoGO0TK7pCemeFQ+Ghic8Yl0eDynoP7rZIhCKS0
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCBg7iYuvjhIG7n
lFeja5Lyv2t2fdtNvxzUib7ygejC+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpj
o5/ym2KFZPwiSZoqKUugi9lCoKuL2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMH
YFwURjARRAku3a/GJUHQR0C572mWmt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64
fGeYSOaYOHEOp/VqIRlzsy7eGQh3xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2Jbq
FxvijCLnBz5WsrS/S/VWOELNEfGr84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLf
WuSgp5hRAgMBAAECggEACB1tvTwciKvp5wliGF45feDv5c5q+vXE462h0OYXjyfl
1sNuMuoLjhYyFvdcQO2rlTBjB8Ald1hXoeidsVYlF9XvqmJRjjS6MAazMpEzTnba
66V6U1j5gOQ6pi3SjfBJGFYo8L+cy/NqUIBWz0+6hp/BsjW3uW2aMGer2Y/zs2Oi
s0kUNqqUWQpGpEDxbDnUAzto4OOcTOO4A5NB5Oe2ifzFiXCWtTNqFIC0ae9tiqGy
m0sggF2hzmlH1WC54rbv7xujYT6tOAEi1OKp/Cpj8XVLbjaqju5avRl1t2nDRQri
9vnOI9P4bnJKbIYzwJVyy8B1M5iK+W2K7wUQ4ZMetQKBgQC2zwtywQ4jUduLWf4f
6HLUIvmcHdJI0YBHS1+4fKc7tsakRo/0FHxNdUlBhqkYTyp57h6/R1T2gG70Kla1
MarGIgyPIlUXsRojlUcNyKXqSVs+kejftIClb+VDXWrpKtGtyxuESsMjaJBsBBAN
hg3yUx2NL2ct4nnlpl1iq3pH9wKBgQC1XkrjntuzjVYlFeg2ih5va3PUD2qmH9Ej
leNXTGhzZL1eQWz4GaRCgzJH8q41pXBxcXQ8hMXZCWdW5GHBYfVqjTRKMUPdI6FI
ypwMbbTBfQfw1kz5VHfLH3romxikNQXeorZkIZqv7/aLPsxLY78XdW8nSDIsT9ZP
mh7It+Tf9wKBgHT7VSYf6D4t2cH42qfshd6+auC+LJywg9gboatbCWEq+z6QcPoO
SeT/Utt9oqONtW7FNCPPL812LopMtaPJnrCfhU3q+GzqxgFTO17UsWeVfbAyisTs
qw4y5pEPLKT846TFPlyPwwPbm9+jUep1iGMzUMvSM8KnFL4ikcvcU6nNAoGBAJQI
MVHV0HadzokYd19EnpDMgE6+goqy/21vIo6XY82DlPjAp+NccnCDSt/kmQJaVQuS
0Y+vwxhhj4IhiBRiEwBz3hEMYtq1ASg7juCqDlW66mdNUqhSpDAbMqN2K8ejK3ZL
EllqACrHzlBa4j5KXKcWB7JlP+ZhB4gTQDsNtXaDAoGBAJCktIdyM3b6UtM5OofB
lzn6/2uT5u46ECBRybk/J+Y2gfcS2cFzkH179SUB4WgfR0h21FlLmjns5ZvHVk0p
/U4aY3/J42JU6lCYrsyV5BBH+O0stSmrUG/4JdKRe6Mq/pB321RsWD06LVJrBDpi
6NUPW03tNLc+Qe+cnQu0c7y+
-----END PRIVATE KEY-----
@@ -0,0 +1,18 @@
-----BEGIN CERTIFICATE-----
MIIC8zCCAdugAwIBAgICEAQwDQYJKoZIhvcNAQELBQAwHDEaMBgGA1UEAwwRdW5y
ZWxhdGVkLmV4YW1wbGUwHhcNMjAwMTAxMDAwMDAwWhcNMjAwNDAxMDAwMDAwWjAc
MRowGAYDVQQDDBF1bnJlbGF0ZWQuZXhhbXBsZTCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBALEpfqdRPBSrbzixAxD2WhsO1AOTUd/Gguf3zjDZ3TUmWJKb
OhlFthtU6PJVhMdAqOk+hiz9Coo/UngWjykyNMOj1YIpq3M4/P8wBTNxkJd/z2x0
iAQBKvWlB6hxf9c1kenu/ea1iMIThA/uudtX3g8k39Asj/37+o070cdBJExWGnbG
lz103FCO6kBzlJPGIVQwzKH6OKthTzi7Uxy5Fxc/ymS9gc43F0tyY6892Wyl8Y62
a3r1PpZoWR0GDFxPIyCS7NP44T15DSL/2RBi4YnWXOC8QMvPnHbSKjoQ6GLQtGkO
1NO4Uy3XbwDFanYXv6Wxv0oUmZMuY7CRTOMdxCMCAwEAAaM/MD0wHAYDVR0RBBUw
E4IRdW5yZWxhdGVkLmV4YW1wbGUwHQYDVR0OBBYEFBh0L/5U5f94TylibBCgS9pD
/VXkMA0GCSqGSIb3DQEBCwUAA4IBAQB5MLqNjTqIdyztjJsKPdYUkA27NpfdqqY6
Yprv5N7U0slsGUqFIDrSuJ1mMMwTpt02IAFyCOWMUtze+/0PqM9k9TqxJH2cTODp
0ET34Qh8NTo0B534EQUWhny+EzgNAzWK02wbpgm88+Mp/t//sHzhWzOm7uk1dlkq
+5ty/8OveWwAtqAWlDtcsRhkEjzynZKUCAF1extwUSuapFSm20mJ4ivt7Ff5MkTI
fq8P+3XAxWTaBW8DzBIzYd1mwFHK2pcklLMRRTW3hukPxEcTivBgJdwsHMskM3UO
/ejy5EHem5ohCMbwMybvUqLS4xdoqXV36Sj2I1zv0IjgO0wgdK1r
-----END CERTIFICATE-----
@@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCxKX6nUTwUq284
sQMQ9lobDtQDk1HfxoLn984w2d01JliSmzoZRbYbVOjyVYTHQKjpPoYs/QqKP1J4
Fo8pMjTDo9WCKatzOPz/MAUzcZCXf89sdIgEASr1pQeocX/XNZHp7v3mtYjCE4QP
7rnbV94PJN/QLI/9+/qNO9HHQSRMVhp2xpc9dNxQjupAc5STxiFUMMyh+jirYU84
u1McuRcXP8pkvYHONxdLcmOvPdlspfGOtmt69T6WaFkdBgxcTyMgkuzT+OE9eQ0i
/9kQYuGJ1lzgvEDLz5x20io6EOhi0LRpDtTTuFMt128AxWp2F7+lsb9KFJmTLmOw
kUzjHcQjAgMBAAECggEAJjBKo+iH6fC2beMCRTLFY5XDr+2etX0GFNi+8prT6zQT
3a5mTLJROf0Ipcw3bfgLMRQQOjxlfD2AGnDKdVfkNdPoH0G6OoErUBra1CcDwxSC
8t7/vtU9nGvxhabJputeQ64HvyacjzFDw2dzWsvLCzqooCgq9dXOQYmGg8wP9gAx
pxfKnd84h41C3pXERdufi6nJHPjZPRW+dKsQ1NMLmtgO5YQnTt3LfVOkZaRa7Izq
ZFWZCBfdsNHHuXCYj3FtqYrqSxELuZAxJ62hxpizrfQdKixguGTObHhC4muIo9fR
eIMCvrAzrs4Fs8nM/K/IPyDU9wPwINK6Z+sxJkhM+QKBgQDaL3Yt3T/LE6bTYyb0
EjVcXoybnuAjFQAenCTIsBRJv21Yv2ddqjq6rOTD64hFuFWPMEOi1TJxIPsFFDMw
NeAwF41fiSrtU8awXs0hTKRQeHjElSBckumOl1DrHu2oKuwWXLniViD62vMsWmpZ
89KT2rgFQuMPvsD3pps8l6W2HwKBgQDP3eTwdVJTdRYlZonqd0+LHG7h715TT1OE
QJOdN9vK5g/9GOCOGHoUxB3IxJVmvz57PrfWaD2rSZ6JMs4TcAY5a0PbJEpgqoJY
kIm/EDnSrFxu2L8DNsAzuThfzXrwWc7PzajYJctYtEfvxCyfo/GdZahg2xaEU0Le
dp+7fxtJfQKBgGVdLWtLYarH+3Yzl1JG71F1f5+A5SKHTCHsas5tt083xuY3Lh9+
yYAWeMVotEunwuAeDR4mLCQQrW4Dn3BjZz8QghaDVFUj2afwsx4oTCIyhfDWU02O
10DunaJOIR1Cq6IL/DPhzbqdY+IUM1qNAyS7tUiEdZbICUBEv9Qbo9oDAoGAORnY
cSGZaumoGsAPNMRvqmQ2JYSO4c3RwGvI4RBeJV+LlthFn2Mj7h5tnCA8dkwkytBQ
yeo6vpnrnf7B018bppxLuKBAekt8oP2biUvYxoGopp6N4kEhDVRdIxwcIHhbMoRp
VG8VbBmiBUBbjcMIdVdeqLSme9UvclMdz/C7uzUCgYEAzD2dvoJSu9zfo3fACw1X
te53Rvr6Kq6DTw0SS4UrAnxlhX9TwD9gymIrR9rxnC+UPwfMwWMdpYqY+Da4sF1G
uwd+nyZ4iFNq+lbLii9rmJ58h03+uo4lUXWTrAik6qOF1ztIfYKWFrOyBQmxjWwo
HnUinudalENFJYZRwSUAtXo=
-----END PRIVATE KEY-----