haproxy-manager-base/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Development Commands

Testing

• API Testing: ./scripts/test-api.sh - Tests all API endpoints with optional authentication
• Certificate Request Testing: ./scripts/test-certificate-request.sh - Tests certificate generation endpoints
• Stick-table contract: python3 scripts/test-stick-table-contract.py - offline; holds the templates' store clauses, STICK_TABLE_FIELD_CONTRACT, and every consumer to each other. Run it after touching any stick-table line.
• Runtime-map contract: python3 scripts/test-runtime-map-contract.py - offline; asserts the runtime map commands are @1-prefixed, reference the map by FILE PATH (never #<id>), carry the value 1, and that every captured rejection is classified as a failure. Run it after touching any add map/del map/clear map path.
• Certificate destruction safety: python3 scripts/test-cert-write-safety.py - offline; asserts a live .pem is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so ssl_cert_path is routinely shared). Run it after touching any os.remove/certbot delete/PEM-write path.
• API thread-pool safety: python3 scripts/test-gunicorn-socket-timeouts.py - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, HAPROXY_MGR_CLIENT_TIMEOUT), 2050 never-sending connections are reaped (HAPROXY_MGR_FIRST_BYTE_TIMEOUT), slowloris header drips are cut (HAPROXY_MGR_HEADER_DEADLINE, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (HAPROXY_MGR_API_ALLOWED_PEERS) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching haproxy_gunicorn_worker.py, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: enqueue_req, accept, on_client_socket_readable, murder_keepalived, handle, handle_request).
• WAF body inspection: python3 scripts/test-waf-body-inspection.py - offline (runs in the image build); pins option http-buffer-request, the X-WHP-WAF-Body complete/partial stamp (strip -> stamp -> SPOE -> strip), tune.bufsize against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in coraza-spoa/config.yaml. Run it after touching the Coraza block, tune.bufsize, or the SPOE engine.
• Superseded-cert retirement: covered by the same file (TestRetirementRules, TestSanOverlapRetirement). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to cert-retired/, the lineage's live+archive+renewal moved TOGETHER to /etc/letsencrypt/retired/<stamp>/ (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own --cert-name lineage is live/<primary> exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (retirement_blockers()). GET /api/capabilities advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to cert-retired/ by cert_retire_expired_pems (never the last pem, never default_self_signed_cert.pem). Expired lineages are never published (cert_is_expired in cert-publish-lib.sh, leaf_is_expired() in Python). Fixtures: scripts/test-fixtures/san-certs/.
• Default splash: python3 scripts/test-default-splash.py - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by http-request return ... file /haproxy/errors/default-splash.html BEFORE the Coraza SPOE call, never for a routed host (routed_hosts.list/routed_wildcards.list, written by generate_config() from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in generate_config(). See README "Default Splash".
• Access log receiver: python3 scripts/test-access-log-receiver.py - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching haproxy-log-receiver.py or the log line in hap_header.tpl.
• Manual Testing: Run curl commands against http://localhost:8000 endpoints as shown in README.md

Reading stick tables (and why it is easy to get silently wrong)

/tmp/haproxy-cli is HAProxy's master CLI socket. Worker commands (show table, show map, add map, ...) need an @1 prefix. Without it HAProxy answers Unknown command: 'show', ... and socat still exits 0 — so an exit-status check passes and the help text gets parsed as data. Always use haproxy_cli(cmd, worker=True) in Python, which inspects the response body.

Stick-table entries are name=value / name(window_ms)=value pairs, not fixed columns; the first token is an allocation pointer (0x...:), not the key. Parse by NAME, and treat a missing field as an ERROR — never default it to 0. The web table stores only conn_cur, conn_rate, http_req_rate, http_err_rate; it holds no history and no counter of past blocks. What was actually denied/tarpitted is in the edge access log, since 2026.11.2 INSIDE the container at /var/log/haproxy/access.log (in-container receiver, volume haproxy-logs; README "Access Log"), not in any stick table. (2026.08.8-2026.11.1 sent it to host rsyslog on 172.18.0.1:514, which had no listener on the fleet.)

This is written down because /api/security/stats and show-tarpit-ips.sh reported "Scan Count"/"BLOCKED" figures parsed from gpc0/gpc1 — fields no stick table has ever stored — for their entire existence. See the header of haproxy_tarpit_config.txt and the contract test.

Changing a runtime map (add map / del map)

Same socket, two more ways to fail silently — and both were live in add_ip_to_runtime_map()/remove_ip_from_runtime_map() for their whole existence:

• Reference the map by FILE PATH, never #<id>. Ids are assigned at config-parse time and move on every config regeneration (on whp01 blocked_ips.map is 37, trusted_ips.map is 10 — there is no id 0). Use add map /etc/haproxy/blocked_ips.map <ip> 1.
• A mutation answers NOTHING on success, so an empty body is the only success — any output at all is a rejection. Worse, @1 add map #0 <ip> 1 also answers nothing and adds nothing, so the body cannot prove an add worked. Read it back with @1 get map <path> <key>.
• Entries must carry the value 1; haproxy.cfg matches with map_ip(...,0) -m int gt 0, so a valueless entry does not block.

In Python use haproxy_cli(cmd, worker=True, expect_empty=True) for mutations and runtime_map_lookup() / runtime_map_keys() to verify. The runtime map is only a fast path: /etc/haproxy/blocked_ips.map is authoritative and HAProxy re-reads it on reload, so a failed runtime command must degrade to "enforced on reload" and be reported, never swallowed.

Running the Application

• Docker Build: docker build -t haproxy-manager .
• Local Development: python haproxy_manager.py (requires HAProxy, certbot, and dependencies installed)
• Container Run: See README.md for various docker run configurations

Monitoring and Debugging

• Error Monitoring: ./scripts/monitor-errors.sh - Monitor application error logs
• External Monitoring: ./scripts/monitor-errors-external.sh - External monitoring script
• Health Check: curl http://localhost:8000/health
• Log Files:
  • /var/log/haproxy-manager.log - General application logs
  • /var/log/haproxy-manager-errors.log - Error logs for alerting

Architecture Overview

Core Components

1. haproxy_manager.py - Main Flask application providing:

   • RESTful API for HAProxy configuration management
   • SQLite database integration for domain/backend storage
   • Let's Encrypt certificate automation
   • HAProxy configuration generation from Jinja2 templates
   • Optional API key authentication via HAPROXY_API_KEY environment variable

2. Database Schema - SQLite database with three main tables:

   • domains - Domain configurations with SSL settings
   • backends - Backend service definitions linked to domains
   • backend_servers - Individual servers within backend groups

3. Template System - Jinja2 templates for HAProxy configuration generation:

   • hap_header.tpl - Global HAProxy settings, defaults, and HTTP/2 tuning
   • hap_backend.tpl - Backend server definitions
   • hap_listener.tpl - Frontend listener configurations with rate limiting
   • hap_letsencrypt.tpl - SSL certificate configurations
   • hap_security_tables.tpl - Stats frontend and security stick tables
   • Template override support for custom backend configurations

4. Certificate Management - Automated SSL certificate handling:

   • Let's Encrypt integration with certbot
   • Self-signed certificate fallback for development
   • Certificate renewal automation via cron
   • Certificate download endpoints for external services

Configuration Flow

1. Domain added via /api/domain endpoint → Database updated
2. generate_config() function → Reads database, renders Jinja2 templates → Writes /etc/haproxy/haproxy.cfg
3. HAProxy reload via socket API (/tmp/haproxy-cli) or process restart
4. SSL certificate generation via Let's Encrypt or self-signed fallback

Key Design Patterns

• Template-driven configuration: HAProxy config generated from modular Jinja2 templates
• Database-backed state: All configuration persisted in SQLite for reliability
• API-first design: All operations exposed via REST endpoints
• Process monitoring: Health checks and automatic HAProxy restart capabilities
• Comprehensive logging: Operation logging with error alerting support

Authentication & Security

• Optional API key authentication controlled by HAPROXY_API_KEY environment variable
• All API endpoints (except /health and /) require Bearer token when API key is set
• Certificate private keys combined with certificates in HAProxy-compatible format
• Default backend page for unmatched domains instead of exposing HAProxy errors

Rate Limiting & Connection Limits (hap_listener.tpl)

• Stick table: type ip size 200k expire 10m tracking conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
• Tracks real client IP via var(txn.real_ip) to work correctly behind Cloudflare/proxies
• Rate limit thresholds:
  • Tarpit at 3000 req/10s (300 req/s)
  • Hard block (deny) at 5000 req/10s (500 req/s)
  • Connection rate limit: 500/10s
  • Concurrent connection limit: 500
  • Error rate limit: 100/30s
• Whitelist bypasses (exempt from rate limits):
  • is_local — RFC1918 private address ranges
  • is_trusted_ip — source IPs listed in trusted_ips.list
  • is_whitelisted — real IPs (from proxy headers) matched in trusted_ips.map

Trusted IP Whitelist Files

• trusted_ips.list — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
• trusted_ips.map — Real IP whitelist for proxy-header matching (format: <IP> 1)
• Both files are baked into the Docker image via COPY in the Dockerfile
• Ship as comment-only templates (no real IPs). Add trusted IPs locally and do not commit them — this repo is mirrored publicly. Entries persist in the /etc/haproxy named volume across recreates

Timeout Hardening (hap_header.tpl)

• timeout http-request: 300s -> 30s (slowloris protection)
• timeout connect: 120s -> 10s
• timeout client: 10m -> 5m
• timeout http-keep-alive: 120s -> 30s

HTTP/2 Protection (hap_header.tpl)

• tune.h2.fe.max-total-streams 2000 — limits total streams per HTTP/2 connection
• tune.h2.fe.glitches-threshold 50 — CVE-2023-44487 Rapid Reset protection

Stats Frontend (hap_security_tables.tpl)

• HAProxy stats page bound to 127.0.0.1:8404 (localhost only, accessible inside container)
• Template: templates/hap_security_tables.tpl

Deployment Context

• Designed to run as Docker container with persistent volumes for certificates and configurations
• Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
• Stats page on port 8404 (localhost only inside container)
• Management interface on port 8000 should be firewall-protected in production
• Dockerfile HEALTHCHECK runs scripts/healthcheck.sh: port 8000 /health (4s max) AND port 80 (HAProxy). WHP's lib-edge-container.sh passes its own --health-cmd, which REPLACES the image's; it must call /haproxy/scripts/healthcheck.sh or the fleet never probes :8000
• The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway (HAPROXY_MGR_API_ALLOWED_PEERS, default 127.0.0.1,::1,gateway; ports via HAPROXY_MGR_PEER_FILTER_PORTS, default 8000). haproxy-manager shares client-net with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP -p 127.0.0.1:8000:8000 publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
• gunicorn uses haproxy_gunicorn_worker.TimeoutThreadWorker (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; --timeout only watches the main thread's heartbeat
• Supports deployment on servers with git directory at /root/whp and web file sync via rsync to /docker/whp/web/
• HAProxy is version 3.0.11