Files
haproxy-manager-base/CLAUDE.md
shadowdaoandClaude Sonnet 5.5 aebb68fe9d ci: make tag verification independent of buildx version
buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'`
and prints the whole inspect text, so the verify step rejected every ref as a
"bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh:
the index digest is the sha256 of `inspect --raw`, cross-checked against the
anchored `Digest:` line and a bare --format output when present; platforms are
read from the raw JSON. All other checks (every tag in every registry, platform
match, one digest across registries, any miss fails) are unchanged. Adds an
offline test with a stub emulating 0.28 and 0.38 output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:30:21 -08:00

17 KiB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Development Commands

Testing

  • API Testing: ./scripts/test-api.sh - Tests all API endpoints with optional authentication
  • Certificate Request Testing: ./scripts/test-certificate-request.sh - Tests certificate generation endpoints
  • Stick-table contract: python3 scripts/test-stick-table-contract.py - offline; holds the templates' store clauses, STICK_TABLE_FIELD_CONTRACT, and every consumer to each other. Run it after touching any stick-table line.
  • Runtime-map contract: python3 scripts/test-runtime-map-contract.py - offline; asserts the runtime map commands are @1-prefixed, reference the map by FILE PATH (never #<id>), carry the value 1, and that every captured rejection is classified as a failure. Run it after touching any add map/del map/clear map path.
  • Certificate destruction safety: python3 scripts/test-cert-write-safety.py - offline; asserts a live .pem is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so ssl_cert_path is routinely shared). Run it after touching any os.remove/certbot delete/PEM-write path.
  • API thread-pool safety: python3 scripts/test-gunicorn-socket-timeouts.py - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, HAPROXY_MGR_CLIENT_TIMEOUT), 2050 never-sending connections are reaped (HAPROXY_MGR_FIRST_BYTE_TIMEOUT), slowloris header drips are cut (HAPROXY_MGR_HEADER_DEADLINE, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (HAPROXY_MGR_API_ALLOWED_PEERS) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching haproxy_gunicorn_worker.py, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: enqueue_req, accept, on_client_socket_readable, murder_keepalived, handle, handle_request).
  • WAF body inspection: python3 scripts/test-waf-body-inspection.py - offline (runs in the image build); pins option http-buffer-request, the X-WHP-WAF-Body complete/partial stamp (strip -> stamp -> SPOE -> strip), tune.bufsize against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in coraza-spoa/config.yaml. Run it after touching the Coraza block, tune.bufsize, or the SPOE engine.
  • Superseded-cert retirement: covered by the same file (TestRetirementRules, TestSanOverlapRetirement). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to cert-retired/, the lineage's live+archive+renewal moved TOGETHER to /etc/letsencrypt/retired/<stamp>/ (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own --cert-name lineage is live/<primary> exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (retirement_blockers()). GET /api/capabilities advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to cert-retired/ by cert_retire_expired_pems (never the last pem, never default_self_signed_cert.pem). Expired lineages are never published (cert_is_expired in cert-publish-lib.sh, leaf_is_expired() in Python). Fixtures: scripts/test-fixtures/san-certs/.
  • Multi-arch index assembly (coraza-spoa): python3 scripts/test-assemble-multiarch-index.py - offline (NOT run in the image build; it also reads .gitea/workflows/build-push-coraza.yaml); scripts/assemble-multiarch-index.sh turns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome is success (failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing. scripts/e2e-coraza-multiarch.sh rehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch").
  • Publish verify gate: python3 scripts/test-verify-published-tags.py - offline (needs jq; NOT run in the image build); scripts/verify-published-tags.sh is the last build-push.yaml step: every tag in every registry must resolve, carry the assembled platforms, and share ONE index digest. The digest is the sha256 of imagetools inspect --raw cross-checked against the text Digest: line / bare --format output (buildx 0.28 ignores --format and prints full text; CI run 5381).
  • Default splash: python3 scripts/test-default-splash.py - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by http-request return ... file /haproxy/errors/default-splash.html BEFORE the Coraza SPOE call, never for a routed host (routed_hosts.list/routed_wildcards.list, written by generate_config() from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in generate_config(). See README "Default Splash".
  • gunicorn access logging: python3 scripts/test-gunicorn-access-logging.py - real gunicorn; :8080 has no access log, :8000 access goes to /var/log/haproxy/api-access.log (rotated by scripts/gunicorn-access-rotate.py), nothing per-request on stdout (journald rate-limit/Graylog flood), gunicorn errors stay on stderr. See docs/logging.md. Run after touching start-up.sh's gunicorn lines.
  • Access log receiver: python3 scripts/test-access-log-receiver.py - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching haproxy-log-receiver.py or the log line in hap_header.tpl.
  • Manual Testing: Run curl commands against http://localhost:8000 endpoints as shown in README.md

Reading stick tables (and why it is easy to get silently wrong)

/tmp/haproxy-cli is HAProxy's master CLI socket. Worker commands (show table, show map, add map, ...) need an @1 prefix. Without it HAProxy answers Unknown command: 'show', ... and socat still exits 0 — so an exit-status check passes and the help text gets parsed as data. Always use haproxy_cli(cmd, worker=True) in Python, which inspects the response body.

Stick-table entries are name=value / name(window_ms)=value pairs, not fixed columns; the first token is an allocation pointer (0x...:), not the key. Parse by NAME, and treat a missing field as an ERROR — never default it to 0. The web table stores only conn_cur, conn_rate, http_req_rate, http_err_rate; it holds no history and no counter of past blocks. What was actually denied/tarpitted is in the edge access log, since 2026.11.2 INSIDE the container at /var/log/haproxy/access.log (in-container receiver, volume haproxy-logs; README "Access Log"), not in any stick table. (2026.08.8-2026.11.1 sent it to host rsyslog on 172.18.0.1:514, which had no listener on the fleet.)

This is written down because /api/security/stats and show-tarpit-ips.sh reported "Scan Count"/"BLOCKED" figures parsed from gpc0/gpc1 — fields no stick table has ever stored — for their entire existence. See the header of haproxy_tarpit_config.txt and the contract test.

Changing a runtime map (add map / del map)

Same socket, two more ways to fail silently — and both were live in add_ip_to_runtime_map()/remove_ip_from_runtime_map() for their whole existence:

  • Reference the map by FILE PATH, never #<id>. Ids are assigned at config-parse time and move on every config regeneration (on whp01 blocked_ips.map is 37, trusted_ips.map is 10 — there is no id 0). Use add map /etc/haproxy/blocked_ips.map <ip> 1.
  • A mutation answers NOTHING on success, so an empty body is the only success — any output at all is a rejection. Worse, @1 add map #0 <ip> 1 also answers nothing and adds nothing, so the body cannot prove an add worked. Read it back with @1 get map <path> <key>.
  • Entries must carry the value 1; haproxy.cfg matches with map_ip(...,0) -m int gt 0, so a valueless entry does not block.

In Python use haproxy_cli(cmd, worker=True, expect_empty=True) for mutations and runtime_map_lookup() / runtime_map_keys() to verify. The runtime map is only a fast path: /etc/haproxy/blocked_ips.map is authoritative and HAProxy re-reads it on reload, so a failed runtime command must degrade to "enforced on reload" and be reported, never swallowed.

Running the Application

  • Docker Build: docker build -t haproxy-manager .
  • Local Development: python haproxy_manager.py (requires HAProxy, certbot, and dependencies installed)
  • Container Run: See README.md for various docker run configurations

Monitoring and Debugging

  • Error Monitoring: ./scripts/monitor-errors.sh - Monitor application error logs
  • External Monitoring: ./scripts/monitor-errors-external.sh - External monitoring script
  • Health Check: curl http://localhost:8000/health
  • Log Files:
    • /var/log/haproxy-manager.log - General application logs
    • /var/log/haproxy-manager-errors.log - Error logs for alerting

Architecture Overview

Core Components

  1. haproxy_manager.py - Main Flask application providing:

    • RESTful API for HAProxy configuration management
    • SQLite database integration for domain/backend storage
    • Let's Encrypt certificate automation
    • HAProxy configuration generation from Jinja2 templates
    • Optional API key authentication via HAPROXY_API_KEY environment variable
  2. Database Schema - SQLite database with three main tables:

    • domains - Domain configurations with SSL settings
    • backends - Backend service definitions linked to domains
    • backend_servers - Individual servers within backend groups
  3. Template System - Jinja2 templates for HAProxy configuration generation:

    • hap_header.tpl - Global HAProxy settings, defaults, and HTTP/2 tuning
    • hap_backend.tpl - Backend server definitions
    • hap_listener.tpl - Frontend listener configurations with rate limiting
    • hap_letsencrypt.tpl - SSL certificate configurations
    • hap_security_tables.tpl - Stats frontend and security stick tables
    • Template override support for custom backend configurations
  4. Certificate Management - Automated SSL certificate handling:

    • Let's Encrypt integration with certbot
    • Self-signed certificate fallback for development
    • Certificate renewal automation via cron
    • Certificate download endpoints for external services

Configuration Flow

  1. Domain added via /api/domain endpoint → Database updated
  2. generate_config() function → Reads database, renders Jinja2 templates → Writes /etc/haproxy/haproxy.cfg
  3. HAProxy reload via socket API (/tmp/haproxy-cli) or process restart
  4. SSL certificate generation via Let's Encrypt or self-signed fallback

Key Design Patterns

  • Template-driven configuration: HAProxy config generated from modular Jinja2 templates
  • Database-backed state: All configuration persisted in SQLite for reliability
  • API-first design: All operations exposed via REST endpoints
  • Process monitoring: Health checks and automatic HAProxy restart capabilities
  • Comprehensive logging: Operation logging with error alerting support

Authentication & Security

  • Optional API key authentication controlled by HAPROXY_API_KEY environment variable
  • All API endpoints (except /health and /) require Bearer token when API key is set
  • Certificate private keys combined with certificates in HAProxy-compatible format
  • Default backend page for unmatched domains instead of exposing HAProxy errors

Rate Limiting & Connection Limits (hap_listener.tpl)

  • Stick table: type ip size 200k expire 10m tracking conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
  • Tracks real client IP via var(txn.real_ip) to work correctly behind Cloudflare/proxies
  • Rate limit thresholds:
    • Tarpit at 3000 req/10s (300 req/s)
    • Hard block (deny) at 5000 req/10s (500 req/s)
    • Connection rate limit: 500/10s
    • Concurrent connection limit: 500
    • Error rate limit: 100/30s
  • Whitelist bypasses (exempt from rate limits):
    • is_local — RFC1918 private address ranges
    • is_trusted_ip — source IPs listed in trusted_ips.list
    • is_whitelisted — real IPs (from proxy headers) matched in trusted_ips.map

Trusted IP Whitelist Files

  • trusted_ips.list — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
  • trusted_ips.map — Real IP whitelist for proxy-header matching (format: <IP> 1)
  • Both files are baked into the Docker image via COPY in the Dockerfile
  • Ship as comment-only templates (no real IPs). Add trusted IPs locally and do not commit them — this repo is mirrored publicly. Entries persist in the /etc/haproxy named volume across recreates

Timeout Hardening (hap_header.tpl)

  • timeout http-request: 300s -> 30s (slowloris protection)
  • timeout connect: 120s -> 10s
  • timeout client: 10m -> 5m
  • timeout http-keep-alive: 120s -> 30s

HTTP/2 Protection (hap_header.tpl)

  • tune.h2.fe.max-total-streams 2000 — limits total streams per HTTP/2 connection
  • tune.h2.fe.glitches-threshold 50 — CVE-2023-44487 Rapid Reset protection

Stats Frontend (hap_security_tables.tpl)

  • HAProxy stats page bound to 127.0.0.1:8404 (localhost only, accessible inside container)
  • Template: templates/hap_security_tables.tpl

Deployment Context

  • Designed to run as Docker container with persistent volumes for certificates and configurations
  • Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
  • Stats page on port 8404 (localhost only inside container)
  • Management interface on port 8000 should be firewall-protected in production
  • Dockerfile HEALTHCHECK runs scripts/healthcheck.sh: port 8000 /health (4s max) AND port 80 (HAProxy). WHP's lib-edge-container.sh passes its own --health-cmd, which REPLACES the image's; it must call /haproxy/scripts/healthcheck.sh or the fleet never probes :8000
  • The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway (HAPROXY_MGR_API_ALLOWED_PEERS, default 127.0.0.1,::1,gateway; ports via HAPROXY_MGR_PEER_FILTER_PORTS, default 8000). haproxy-manager shares client-net with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP -p 127.0.0.1:8000:8000 publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
  • gunicorn uses haproxy_gunicorn_worker.TimeoutThreadWorker (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; --timeout only watches the main thread's heartbeat
  • Supports deployment on servers with git directory at /root/whp and web file sync via rsync to /docker/whp/web/
  • HAProxy is the 3.2 LTS series from haproxy.debian.net (trixie-backports-3.2, native OpenSSL 3.5 QUIC); exact build in /haproxy/HAPROXY_VERSION. Debian trixie's 3.0.11 had the DNS-resolver watchdog abort (haproxy #3101)
  • HAProxy runs as haproxy -W -db (foreground master, started by haproxy_manager.spawn_haproxy_master()), NOT daemonised: daemon re-points the master's stderr at /dev/null and loses ALERTs and the ha_panic/watchdog thread dump. Its stderr is appended to /var/log/haproxy/haproxy-stderr.log (haproxy-logs volume) and scripts/haproxy-stderr-relay.py re-emits it on the container stderr (haproxy-stderr: prefix). The in-container supervisor probes with pidof every HAPROXY_SUPERVISOR_INTERVAL (default 2 s) and only runs python when HAProxy is gone; Crash evidence (dump blocks opened only by the anchored Thread N is about to kill header, [ALERT] (PID) :, Current worker (N) exited with code N; ANCHORED patterns, so keywords echoed from URLs/Host/server names never match) bypasses the bucket within a budget (1000 lines / 10 min, 60 s cooldown after a bound closes a block) and goes to STDERR (journald PRIORITY 3); routine reload noise is collapsed into one [relay] reload/start ok: N routine lines collapsed line per reload and everything benign goes to STDOUT (PRIORITY 6; docker's journald driver does not parse <N> prefixes), with a per-minute [relay] suppressed N lines summary. python3 scripts/test-haproxy-stderr-relay.py (offline) covers the relay