Files
haproxy-manager-base/scripts/renew-certificates.sh
T
shadowdaoandClaude Opus 5 22dab685d0 fix(certs): never truncate a live PEM; validate before publishing
Every path that refreshed a combined certificate did

    with open(combined_path, 'w') as combined:             # TRUNCATES
        subprocess.run(['cat', cert, key], stdout=combined) # rc ignored

where combined_path is the bundle HAProxy is currently serving. The live file
was emptied before any source material had been read and the cat status was
never checked, so an unreadable source, a zero-length privkey, a full disk or a
killed container left a truncated or key-less PEM in place. HAProxy loads
/etc/haproxy/certs as a directory, so one unusable file there fails the whole
ssl bind - HTTPS down for every site on the host, and unlike a bad haproxy.cfg
it is not recoverable by config rollback.

The bundle endpoint compounded it: superseded .pem files were unlinked and
their lineages `certbot delete`d before anything had checked the replacement,
destroying both copies of a working certificate. Recovery there means fresh,
rate-limited ACME orders.

Bundles are now assembled in a staging directory beside the crt directory (never
inside it - HAProxy would try to load a temp file), validated there, and swapped
in with os.replace(). Validation is mandatory structural checks in pure Python
plus a best-effort openssl key/leaf pairing check; a missing openssl warns
loudly and does not silently pass. The previous bundle is copied to
/etc/haproxy/cert-backups first. Superseded certs are moved aside rather than
deleted, and `certbot delete` runs only after HAProxy has reloaded onto the
replacement. The same guarantees are implemented for the cron/renewal shell
path in scripts/cert-publish-lib.sh, which also gates the reload on `haproxy -c`.

Reuses write_config_atomically() from the config-rollback fix (extended with
staging_dir/validate) rather than adding a second atomic writer.

Also heals a zero-byte QUIC cluster-secret file, which previously made
get_or_create_cluster_secret() return '' forever.

Tests: scripts/test-cert-write-safety.py (17) and scripts/test-cert-scripts.py
(32) in the existing stdlib-unittest, stub-binary convention. Both bug classes
reproduce against the pre-fix tree via HAPROXY_MANAGER_DIR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:03:37 -07:00

113 lines
3.9 KiB
Bash

#!/usr/bin/env bash
# Certificate Renewal Script for HAProxy Manager
# This script runs certbot renew and copies certificates to HAProxy format
# Configuration
LOG_FILE="${LOG_FILE:-/var/log/haproxy-manager.log}"
ERROR_LOG_FILE="${ERROR_LOG_FILE:-/var/log/haproxy-manager-errors.log}"
DB_FILE="${DB_FILE:-/etc/haproxy/haproxy_config.db}"
SSL_CERTS_DIR="${SSL_CERTS_DIR:-/etc/haproxy/certs}"
LETSENCRYPT_LIVE_DIR="${LETSENCRYPT_LIVE_DIR:-/etc/letsencrypt/live}"
# Logging functions
log_info() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [INFO] $*" | tee -a "$LOG_FILE"
}
log_error() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >> "$ERROR_LOG_FILE"
}
# Safe certificate publication helpers (cert_publish / cert_bundle_valid /
# haproxy_config_ok). Sourced AFTER the log_* functions above so the library
# uses this script's logging rather than its own fallbacks.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=cert-publish-lib.sh
if [ -r "${SCRIPT_DIR}/cert-publish-lib.sh" ]; then
. "${SCRIPT_DIR}/cert-publish-lib.sh"
else
log_error "Missing ${SCRIPT_DIR}/cert-publish-lib.sh - refusing to touch live certificates"
exit 1
fi
log_info "Starting certificate renewal process"
# Run certbot renewal — don't exit on failure, some certs may have
# renewed successfully even if others failed (e.g., domain no longer
# pointed here). Continue to copy/combine whatever succeeded.
CERTBOT_OUTPUT=$(certbot renew --no-random-sleep-on-renew 2>&1)
CERTBOT_EXIT=$?
if [ $CERTBOT_EXIT -eq 0 ]; then
log_info "Certbot renewal completed successfully"
else
log_error "Certbot renewal had failures (exit code $CERTBOT_EXIT):"
# Log the specific failures
echo "$CERTBOT_OUTPUT" | grep -E "Failed to renew|failure" | while read -r line; do
log_error " $line"
done
log_info "Continuing to process successfully renewed certificates..."
fi
# Copy all certificates to HAProxy format
# Ensure SSL certs directory exists
mkdir -p "$SSL_CERTS_DIR"
# Get all SSL-enabled domains from database
DOMAINS=$(find "$LETSENCRYPT_LIVE_DIR/" -mindepth 1 -maxdepth 1 -type d -printf '%f\n')
if [ -z "$DOMAINS" ]; then
log_info "No SSL-enabled domains found"
exit 0
fi
# Copy certificates for each domain
UPDATED=0
FAILED=0
while read -r domain; do
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
# Assemble in a staging dir and rename into place. NEVER redirect into
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
# takes down the whole ssl bind. See scripts/cert-publish-lib.sh.
if cert_publish "$CERT_FILE" "$KEY_FILE" "$COMBINED_FILE"; then
log_info "Updated certificate for $domain"
UPDATED=$((UPDATED + 1))
else
log_error "Failed to combine certificate for $domain"
FAILED=$((FAILED + 1))
fi
else
log_error "Certificate files not found for $domain"
FAILED=$((FAILED + 1))
fi
done <<< "$DOMAINS"
log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
# Reload HAProxy if any certificates were updated
if [ $UPDATED -gt 0 ]; then
# Never reload onto unvalidated material: a reload that fails to load the
# certs directory drops HTTPS for every site on this host.
if ! haproxy_config_ok; then
log_error "HAProxy configuration does not validate - refusing to reload after certificate renewal"
exit 1
fi
if echo "reload" | socat stdio /tmp/haproxy-cli 2>/dev/null; then
log_info "HAProxy reloaded successfully"
else
log_error "Failed to reload HAProxy"
exit 1
fi
fi
log_info "Certificate renewal process completed"
exit 0