cloud-hosting-platform/haproxy-manager-base
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fix/remove-domain-shared-cert-refcount
haproxy-manager-base/CLAUDE.md
T
shadowdaoandClaude Opus 5 b072786192 fix(domain-removal): stop deleting certificates other live sites are served from 
DELETE /api/domain ran, unconditionally, for any ssl_enabled row:

    os.remove(ssl_cert_path)
    certbot delete --cert-name <domain> --non-interactive

`domains.domain` is UNIQUE. `domains.ssl_cert_path` is not, and nothing
anywhere guarded against two rows naming the same file. Sharing is not an
edge case -- it is the normal shape of the table, because
request_ssl_bundle() deliberately creates it: one SAN certificate is issued
as `--cert-name <primary>`, published once to
/etc/haproxy/certs/<primary>.pem, and then EVERY included name's row is
pointed at that same path ("Mark every name in the bundle as ssl_enabled,
all pointing at the same combined .pem").

Measured read-only on the live SQLite in the haproxy-manager containers on
2026-08-23:

  * whp01: 157 domain rows, 150 ssl_enabled, 71 distinct cert paths.
    39 of those paths are referenced by MORE THAN ONE row, covering 118 of
    the 150 SSL-enabled rows. Worst cases: brain-jar.com.pem and
    arclightcourt.com.pem with 10 domains each, hackerpublicradio.org.pem
    with 6, anhonesthost.com.pem with 5.
  * whp02: 33 rows, 31 ssl_enabled, 15 distinct paths, 11 shared across 27
    rows -- including threeworldsoneheart.org.pem, referenced by the apex,
    its www, and mail.threeworldsoneheart.org. A production cleanup of that
    mail.* row was stopped short precisely because removing it would have
    unlinked the PEM the serving site is using.

So removing one domain unlinked a file up to nine other configured domains
were being served from. HAProxy binds the crt directory
(`bind ... ssl crt /etc/haproxy/certs`), so the loss is not noticed until
the next reload or restart, at which point the listener refuses to come up
or those names fall back to the wrong certificate.

`certbot delete` is the worse half. It destroys the lineage's archive, live
symlinks and renewal config; recovery is a fresh, rate-limited ACME order.
The old code passed `--cert-name <domain>`, which is also simply the wrong
lineage for a SAN member: 81 of whp01's 150 SSL-enabled rows have a cert
path whose basename is not their own domain, so for those the call was a
silent no-op -- while for a bundle PRIMARY it deleted the one lineage still
renewing the certificate every other name in the bundle is served with.

The fix refcounts, after the row is deleted so the query answers "who else
still needs this":

  * lineage_name_for_cert_path() -- the lineage is the published bundle's
    basename minus .pem, the same derivation _quarantine_superseded_certs()
    already uses, not the domain being removed.
  * domains_referencing_cert_path() / domains_referencing_lineage() -- the
    remaining rows that name that file, and that lineage.
  * remove_domain() unlinks only when the list is empty, `certbot delete`s
    only when the list is empty, logs the retained names explicitly when it
    skips, and reports them as certificate_retained_for /
    lineage_retained_for in the API response.

ssl_enabled is deliberately not filtered on in the refcount: the two
mistakes are not symmetric. A stale PEM left in the crt directory costs a
few kilobytes; an unlinked live one is HTTPS down for every name it serves.
Cleanup is deferred, not cancelled -- removing the last name on a bundle
still unlinks the file and deletes the lineage.

No row on either production host has ssl_enabled=1 with an empty
ssl_cert_path, so the "no path, no attributable lineage" branch changes
nothing on the current fleet.

Tests (scripts/test-cert-write-safety.py, +9, suite now 31, all offline):
last reference -> file unlinked and lineage deleted; shared file survives
removal of a SAN member AND of the bundle primary, byte-for-byte, with the
edge still starting; shared lineage is not certbot-deleted; the production
mail.* shape; removing every name eventually cleans up; an unrelated
bundle is never collateral damage. Assertions are on os.path.exists, file
contents and the recorded certbot argv, never on which branch ran.

Mutation-tested, all five mutants killed:
  1. guard absent entirely (suite run with HAPROXY_MANAGER_DIR pointed at
     main) -> 6 failures, incl. "example.com is still configured and still
     served from this file".
  2. refcount taken before the row is deleted -> 5 failures, incl. "the
     last reference is gone - now it may be removed".
  3. certbot guard removed, file guard kept -> 3 failures, incl.
     [] != ['delete --cert-name example.com --non-interactive'].
  4. lineage taken from the domain name instead of the cert path -> 3
     failures, incl. 'delete --cert-name example.com' != 'delete
     --cert-name www.example.com'.
  5. file-unlink guard removed, certbot guard kept -> 4 failures, incl.
     "two sites are still served from this bundle".

Other suites unchanged and green: test-config-rollback, test-cert-scripts,
test-stick-table-contract, test-runtime-map-contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-23 04:47:26 -07:00

9.4 KiB
Raw Permalink Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Development Commands

Testing

  • API Testing: ./scripts/test-api.sh - Tests all API endpoints with optional authentication
  • Certificate Request Testing: ./scripts/test-certificate-request.sh - Tests certificate generation endpoints
  • Stick-table contract: python3 scripts/test-stick-table-contract.py - offline; holds the templates' store clauses, STICK_TABLE_FIELD_CONTRACT, and every consumer to each other. Run it after touching any stick-table line.
  • Runtime-map contract: python3 scripts/test-runtime-map-contract.py - offline; asserts the runtime map commands are @1-prefixed, reference the map by FILE PATH (never #<id>), carry the value 1, and that every captured rejection is classified as a failure. Run it after touching any add map/del map/clear map path.
  • Certificate destruction safety: python3 scripts/test-cert-write-safety.py - offline; asserts a live .pem is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so ssl_cert_path is routinely shared). Run it after touching any os.remove/certbot delete/PEM-write path.
  • Manual Testing: Run curl commands against http://localhost:8000 endpoints as shown in README.md

Reading stick tables (and why it is easy to get silently wrong)

/tmp/haproxy-cli is HAProxy's master CLI socket. Worker commands (show table, show map, add map, ...) need an @1 prefix. Without it HAProxy answers Unknown command: 'show', ... and socat still exits 0 — so an exit-status check passes and the help text gets parsed as data. Always use haproxy_cli(cmd, worker=True) in Python, which inspects the response body.

Stick-table entries are name=value / name(window_ms)=value pairs, not fixed columns; the first token is an allocation pointer (0x...:), not the key. Parse by NAME, and treat a missing field as an ERROR — never default it to 0. The web table stores only conn_cur, conn_rate, http_req_rate, http_err_rate; it holds no history and no counter of past blocks. What was actually denied/tarpitted is in the edge access log on the host at /var/log/haproxy.log (shipped 2026.08.8), not in any stick table.

This is written down because /api/security/stats and show-tarpit-ips.sh reported "Scan Count"/"BLOCKED" figures parsed from gpc0/gpc1 — fields no stick table has ever stored — for their entire existence. See the header of haproxy_tarpit_config.txt and the contract test.

Changing a runtime map (add map / del map)

Same socket, two more ways to fail silently — and both were live in add_ip_to_runtime_map()/remove_ip_from_runtime_map() for their whole existence:

  • Reference the map by FILE PATH, never #<id>. Ids are assigned at config-parse time and move on every config regeneration (on whp01 blocked_ips.map is 37, trusted_ips.map is 10 — there is no id 0). Use add map /etc/haproxy/blocked_ips.map <ip> 1.
  • A mutation answers NOTHING on success, so an empty body is the only success — any output at all is a rejection. Worse, @1 add map #0 <ip> 1 also answers nothing and adds nothing, so the body cannot prove an add worked. Read it back with @1 get map <path> <key>.
  • Entries must carry the value 1; haproxy.cfg matches with map_ip(...,0) -m int gt 0, so a valueless entry does not block.

In Python use haproxy_cli(cmd, worker=True, expect_empty=True) for mutations and runtime_map_lookup() / runtime_map_keys() to verify. The runtime map is only a fast path: /etc/haproxy/blocked_ips.map is authoritative and HAProxy re-reads it on reload, so a failed runtime command must degrade to "enforced on reload" and be reported, never swallowed.

Running the Application

  • Docker Build: docker build -t haproxy-manager .
  • Local Development: python haproxy_manager.py (requires HAProxy, certbot, and dependencies installed)
  • Container Run: See README.md for various docker run configurations

Monitoring and Debugging

  • Error Monitoring: ./scripts/monitor-errors.sh - Monitor application error logs
  • External Monitoring: ./scripts/monitor-errors-external.sh - External monitoring script
  • Health Check: curl http://localhost:8000/health
  • Log Files:
    • /var/log/haproxy-manager.log - General application logs
    • /var/log/haproxy-manager-errors.log - Error logs for alerting

Architecture Overview

Core Components

  1. haproxy_manager.py - Main Flask application providing:

    • RESTful API for HAProxy configuration management
    • SQLite database integration for domain/backend storage
    • Let's Encrypt certificate automation
    • HAProxy configuration generation from Jinja2 templates
    • Optional API key authentication via HAPROXY_API_KEY environment variable

  2. Database Schema - SQLite database with three main tables:

    • domains - Domain configurations with SSL settings
    • backends - Backend service definitions linked to domains
    • backend_servers - Individual servers within backend groups

  3. Template System - Jinja2 templates for HAProxy configuration generation:

    • hap_header.tpl - Global HAProxy settings, defaults, and HTTP/2 tuning
    • hap_backend.tpl - Backend server definitions
    • hap_listener.tpl - Frontend listener configurations with rate limiting
    • hap_letsencrypt.tpl - SSL certificate configurations
    • hap_security_tables.tpl - Stats frontend and security stick tables
    • Template override support for custom backend configurations

  4. Certificate Management - Automated SSL certificate handling:

    • Let's Encrypt integration with certbot
    • Self-signed certificate fallback for development
    • Certificate renewal automation via cron
    • Certificate download endpoints for external services

Configuration Flow

  1. Domain added via /api/domain endpoint → Database updated
  2. generate_config() function → Reads database, renders Jinja2 templates → Writes /etc/haproxy/haproxy.cfg
  3. HAProxy reload via socket API (/tmp/haproxy-cli) or process restart
  4. SSL certificate generation via Let's Encrypt or self-signed fallback

Key Design Patterns

  • Template-driven configuration: HAProxy config generated from modular Jinja2 templates
  • Database-backed state: All configuration persisted in SQLite for reliability
  • API-first design: All operations exposed via REST endpoints
  • Process monitoring: Health checks and automatic HAProxy restart capabilities
  • Comprehensive logging: Operation logging with error alerting support

Authentication & Security

  • Optional API key authentication controlled by HAPROXY_API_KEY environment variable
  • All API endpoints (except /health and /) require Bearer token when API key is set
  • Certificate private keys combined with certificates in HAProxy-compatible format
  • Default backend page for unmatched domains instead of exposing HAProxy errors

Rate Limiting & Connection Limits (hap_listener.tpl)

  • Stick table: type ip size 200k expire 10m tracking conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
  • Tracks real client IP via var(txn.real_ip) to work correctly behind Cloudflare/proxies
  • Rate limit thresholds:
    • Tarpit at 3000 req/10s (300 req/s)
    • Hard block (deny) at 5000 req/10s (500 req/s)
    • Connection rate limit: 500/10s
    • Concurrent connection limit: 500
    • Error rate limit: 100/30s
  • Whitelist bypasses (exempt from rate limits):
    • is_local — RFC1918 private address ranges
    • is_trusted_ip — source IPs listed in trusted_ips.list
    • is_whitelisted — real IPs (from proxy headers) matched in trusted_ips.map

Trusted IP Whitelist Files

  • trusted_ips.list — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
  • trusted_ips.map — Real IP whitelist for proxy-header matching (format: <IP> 1)
  • Both files are baked into the Docker image via COPY in the Dockerfile
  • Ship as comment-only templates (no real IPs). Add trusted IPs locally and do not commit them — this repo is mirrored publicly. Entries persist in the /etc/haproxy named volume across recreates

Timeout Hardening (hap_header.tpl)

  • timeout http-request: 300s -> 30s (slowloris protection)
  • timeout connect: 120s -> 10s
  • timeout client: 10m -> 5m
  • timeout http-keep-alive: 120s -> 30s

HTTP/2 Protection (hap_header.tpl)

  • tune.h2.fe.max-total-streams 2000 — limits total streams per HTTP/2 connection
  • tune.h2.fe.glitches-threshold 50 — CVE-2023-44487 Rapid Reset protection

Stats Frontend (hap_security_tables.tpl)

  • HAProxy stats page bound to 127.0.0.1:8404 (localhost only, accessible inside container)
  • Template: templates/hap_security_tables.tpl

Deployment Context

  • Designed to run as Docker container with persistent volumes for certificates and configurations
  • Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
  • Stats page on port 8404 (localhost only inside container)
  • Management interface on port 8000 should be firewall-protected in production
  • Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with start-period=60s and timeout=10s
  • Supports deployment on servers with git directory at /root/whp and web file sync via rsync to /docker/whp/web/
  • HAProxy is version 3.0.11
Reference in New Issue View Git Blame Copy Permalink