Every path that refreshed a combined certificate did
with open(combined_path, 'w') as combined: # TRUNCATES
subprocess.run(['cat', cert, key], stdout=combined) # rc ignored
where combined_path is the bundle HAProxy is currently serving. The live file
was emptied before any source material had been read and the cat status was
never checked, so an unreadable source, a zero-length privkey, a full disk or a
killed container left a truncated or key-less PEM in place. HAProxy loads
/etc/haproxy/certs as a directory, so one unusable file there fails the whole
ssl bind - HTTPS down for every site on the host, and unlike a bad haproxy.cfg
it is not recoverable by config rollback.
The bundle endpoint compounded it: superseded .pem files were unlinked and
their lineages `certbot delete`d before anything had checked the replacement,
destroying both copies of a working certificate. Recovery there means fresh,
rate-limited ACME orders.
Bundles are now assembled in a staging directory beside the crt directory (never
inside it - HAProxy would try to load a temp file), validated there, and swapped
in with os.replace(). Validation is mandatory structural checks in pure Python
plus a best-effort openssl key/leaf pairing check; a missing openssl warns
loudly and does not silently pass. The previous bundle is copied to
/etc/haproxy/cert-backups first. Superseded certs are moved aside rather than
deleted, and `certbot delete` runs only after HAProxy has reloaded onto the
replacement. The same guarantees are implemented for the cron/renewal shell
path in scripts/cert-publish-lib.sh, which also gates the reload on `haproxy -c`.
Reuses write_config_atomically() from the config-rollback fix (extended with
staging_dir/validate) rather than adding a second atomic writer.
Also heals a zero-byte QUIC cluster-secret file, which previously made
get_or_create_cluster_secret() return '' forever.
Tests: scripts/test-cert-write-safety.py (17) and scripts/test-cert-scripts.py
(32) in the existing stdlib-unittest, stub-binary convention. Both bug classes
reproduce against the pre-fix tree via HAPROXY_MANAGER_DIR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
113 lines
3.9 KiB
Bash
113 lines
3.9 KiB
Bash
#!/usr/bin/env bash
|
|
|
|
# Certificate Renewal Script for HAProxy Manager
|
|
# This script runs certbot renew and copies certificates to HAProxy format
|
|
|
|
# Configuration
|
|
LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
|
|
ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
|
|
DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
|
|
SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
|
|
LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
|
|
|
|
# Logging functions
|
|
log_info() {
|
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [INFO] $*" | tee -a "$LOG_FILE"
|
|
}
|
|
|
|
log_error() {
|
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
|
|
}
|
|
|
|
# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
|
|
# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
|
|
# uses this script's logging rather than its own fallbacks.
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck source=cert-publish-lib.sh
|
|
if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
|
|
. "${SCRIPT_DIR}/cert-publish-lib.sh"
|
|
else
|
|
log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
|
|
exit 1
|
|
fi
|
|
|
|
log_info "Starting certificate renewal process"
|
|
|
|
# Run certbot renewal — don't exit on failure, some certs may have
|
|
# renewed successfully even if others failed (e.g., domain no longer
|
|
# pointed here). Continue to copy/combine whatever succeeded.
|
|
CERTBOT_OUTPUT=$(certbot renew --no-random-sleep-on-renew 2>&1)
|
|
CERTBOT_EXIT=$?
|
|
|
|
if [ $CERTBOT_EXIT -eq 0 ]; then
|
|
log_info "Certbot renewal completed successfully"
|
|
else
|
|
log_error "Certbot renewal had failures (exit code $CERTBOT_EXIT):"
|
|
# Log the specific failures
|
|
echo "$CERTBOT_OUTPUT" | grep -E "Failed to renew|failure" | while read -r line; do
|
|
log_error " $line"
|
|
done
|
|
log_info "Continuing to process successfully renewed certificates..."
|
|
fi
|
|
|
|
# Copy all certificates to HAProxy format
|
|
# Ensure SSL certs directory exists
|
|
mkdir -p "$SSL_CERTS_DIR"
|
|
|
|
# Get all SSL-enabled domains from database
|
|
DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
|
|
|
|
if [ -z "$DOMAINS" ]; then
|
|
log_info "No SSL-enabled domains found"
|
|
exit 0
|
|
fi
|
|
|
|
# Copy certificates for each domain
|
|
UPDATED=0
|
|
FAILED=0
|
|
|
|
while read -r domain; do
|
|
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
|
|
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
|
|
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
|
|
|
|
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
|
|
# Assemble in a staging dir and rename into place. NEVER redirect into
|
|
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and
|
|
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
|
|
# takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
|
|
if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
|
|
log_info "Updated certificate for $domain"
|
|
UPDATED=$((UPDATED + 1))
|
|
else
|
|
log_error "Failed to combine certificate for $domain"
|
|
FAILED=$((FAILED + 1))
|
|
fi
|
|
else
|
|
log_error "Certificate files not found for $domain"
|
|
FAILED=$((FAILED + 1))
|
|
fi
|
|
done <<< "$DOMAINS"
|
|
|
|
log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
|
|
|
|
# Reload HAProxy if any certificates were updated
|
|
if [ $UPDATED -gt 0 ]; then
|
|
# Never reload onto unvalidated material: a reload that fails to load the
|
|
# certs directory drops HTTPS for every site on this host.
|
|
if ! haproxy_config_ok; then
|
|
log_error "HAProxy configuration does not validate - refusing to reload after certificate renewal"
|
|
exit 1
|
|
fi
|
|
|
|
if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
|
|
log_info "HAProxy reloaded successfully"
|
|
else
|
|
log_error "Failed to reload HAProxy"
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
log_info "Certificate renewal process completed"
|
|
exit 0
|