buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'`
and prints the whole inspect text, so the verify step rejected every ref as a
"bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh:
the index digest is the sha256 of `inspect --raw`, cross-checked against the
anchored `Digest:` line and a bare --format output when present; platforms are
read from the raw JSON. All other checks (every tag in every registry, platform
match, one digest across registries, any miss fails) are unchanged. Adds an
offline test with a stub emulating 0.28 and 0.38 output.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
17 KiB
CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Development Commands
Testing
- API Testing:
./scripts/test-api.sh- Tests all API endpoints with optional authentication - Certificate Request Testing:
./scripts/test-certificate-request.sh- Tests certificate generation endpoints - Stick-table contract:
python3 scripts/test-stick-table-contract.py- offline; holds the templates'storeclauses,STICK_TABLE_FIELD_CONTRACT, and every consumer to each other. Run it after touching anystick-tableline. - Runtime-map contract:
python3 scripts/test-runtime-map-contract.py- offline; asserts the runtime map commands are@1-prefixed, reference the map by FILE PATH (never#<id>), carry the value1, and that every captured rejection is classified as a failure. Run it after touching anyadd map/del map/clear mappath. - Certificate destruction safety:
python3 scripts/test-cert-write-safety.py- offline; asserts a live.pemis never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, sossl_cert_pathis routinely shared). Run it after touching anyos.remove/certbot delete/PEM-write path. - API thread-pool safety:
python3 scripts/test-gunicorn-socket-timeouts.py- runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout,HAPROXY_MGR_CLIENT_TIMEOUT), 2050 never-sending connections are reaped (HAPROXY_MGR_FIRST_BYTE_TIMEOUT), slowloris header drips are cut (HAPROXY_MGR_HEADER_DEADLINE, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (HAPROXY_MGR_API_ALLOWED_PEERS) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touchinghaproxy_gunicorn_worker.py, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals:enqueue_req,accept,on_client_socket_readable,murder_keepalived,handle,handle_request). - WAF body inspection:
python3 scripts/test-waf-body-inspection.py- offline (runs in the image build); pinsoption http-buffer-request, theX-WHP-WAF-Bodycomplete/partial stamp (strip -> stamp -> SPOE -> strip),tune.bufsizeagainst the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits incoraza-spoa/config.yaml. Run it after touching the Coraza block,tune.bufsize, or the SPOE engine. - Superseded-cert retirement: covered by the same file (
TestRetirementRules,TestSanOverlapRetirement). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved tocert-retired/, the lineage's live+archive+renewal moved TOGETHER to/etc/letsencrypt/retired/<stamp>/(leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own--cert-namelineage islive/<primary>exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (retirement_blockers()).GET /api/capabilitiesadvertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved tocert-retired/bycert_retire_expired_pems(never the last pem, neverdefault_self_signed_cert.pem). Expired lineages are never published (cert_is_expiredincert-publish-lib.sh,leaf_is_expired()in Python). Fixtures:scripts/test-fixtures/san-certs/. - Multi-arch index assembly (coraza-spoa):
python3 scripts/test-assemble-multiarch-index.py- offline (NOT run in the image build; it also reads.gitea/workflows/build-push-coraza.yaml);scripts/assemble-multiarch-index.shturns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome issuccess(failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing.scripts/e2e-coraza-multiarch.shrehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch"). - Publish verify gate:
python3 scripts/test-verify-published-tags.py- offline (needsjq; NOT run in the image build);scripts/verify-published-tags.shis the last build-push.yaml step: every tag in every registry must resolve, carry the assembled platforms, and share ONE index digest. The digest is the sha256 ofimagetools inspect --rawcross-checked against the textDigest:line / bare--formatoutput (buildx 0.28 ignores--formatand prints full text; CI run 5381). - Default splash:
python3 scripts/test-default-splash.py- offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered byhttp-request return ... file /haproxy/errors/default-splash.htmlBEFORE the Coraza SPOE call, never for a routed host (routed_hosts.list/routed_wildcards.list, written bygenerate_config()from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops ingenerate_config(). See README "Default Splash". - gunicorn access logging:
python3 scripts/test-gunicorn-access-logging.py- real gunicorn; :8080 has no access log, :8000 access goes to/var/log/haproxy/api-access.log(rotated byscripts/gunicorn-access-rotate.py), nothing per-request on stdout (journald rate-limit/Graylog flood), gunicorn errors stay on stderr. See docs/logging.md. Run after touching start-up.sh's gunicorn lines. - Access log receiver:
python3 scripts/test-access-log-receiver.py- offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touchinghaproxy-log-receiver.pyor thelogline inhap_header.tpl. - Manual Testing: Run
curlcommands againsthttp://localhost:8000endpoints as shown in README.md
Reading stick tables (and why it is easy to get silently wrong)
/tmp/haproxy-cli is HAProxy's master CLI socket. Worker commands
(show table, show map, add map, ...) need an @1 prefix. Without it
HAProxy answers Unknown command: 'show', ... and socat still exits 0 — so
an exit-status check passes and the help text gets parsed as data. Always use
haproxy_cli(cmd, worker=True) in Python, which inspects the response body.
Stick-table entries are name=value / name(window_ms)=value pairs, not fixed
columns; the first token is an allocation pointer (0x...:), not the key. Parse
by NAME, and treat a missing field as an ERROR — never default it to 0. The
web table stores only conn_cur, conn_rate, http_req_rate,
http_err_rate; it holds no history and no counter of past blocks. What was
actually denied/tarpitted is in the edge access log, since 2026.11.2 INSIDE the
container at /var/log/haproxy/access.log (in-container receiver, volume
haproxy-logs; README "Access Log"), not in any stick table. (2026.08.8-2026.11.1
sent it to host rsyslog on 172.18.0.1:514, which had no listener on the fleet.)
This is written down because /api/security/stats and show-tarpit-ips.sh
reported "Scan Count"/"BLOCKED" figures parsed from gpc0/gpc1 — fields no
stick table has ever stored — for their entire existence. See the header of
haproxy_tarpit_config.txt and the contract test.
Changing a runtime map (add map / del map)
Same socket, two more ways to fail silently — and both were live in
add_ip_to_runtime_map()/remove_ip_from_runtime_map() for their whole
existence:
- Reference the map by FILE PATH, never
#<id>. Ids are assigned at config-parse time and move on every config regeneration (on whp01blocked_ips.mapis 37,trusted_ips.mapis 10 — there is no id 0). Useadd map /etc/haproxy/blocked_ips.map <ip> 1. - A mutation answers NOTHING on success, so an empty body is the only
success — any output at all is a rejection. Worse,
@1 add map #0 <ip> 1also answers nothing and adds nothing, so the body cannot prove an add worked. Read it back with@1 get map <path> <key>. - Entries must carry the value
1; haproxy.cfg matches withmap_ip(...,0) -m int gt 0, so a valueless entry does not block.
In Python use haproxy_cli(cmd, worker=True, expect_empty=True) for mutations
and runtime_map_lookup() / runtime_map_keys() to verify. The runtime map is
only a fast path: /etc/haproxy/blocked_ips.map is authoritative and HAProxy
re-reads it on reload, so a failed runtime command must degrade to
"enforced on reload" and be reported, never swallowed.
Running the Application
- Docker Build:
docker build -t haproxy-manager . - Local Development:
python haproxy_manager.py(requires HAProxy, certbot, and dependencies installed) - Container Run: See README.md for various docker run configurations
Monitoring and Debugging
- Error Monitoring:
./scripts/monitor-errors.sh- Monitor application error logs - External Monitoring:
./scripts/monitor-errors-external.sh- External monitoring script - Health Check:
curl http://localhost:8000/health - Log Files:
/var/log/haproxy-manager.log- General application logs/var/log/haproxy-manager-errors.log- Error logs for alerting
Architecture Overview
Core Components
-
haproxy_manager.py - Main Flask application providing:
- RESTful API for HAProxy configuration management
- SQLite database integration for domain/backend storage
- Let's Encrypt certificate automation
- HAProxy configuration generation from Jinja2 templates
- Optional API key authentication via
HAPROXY_API_KEYenvironment variable
-
Database Schema - SQLite database with three main tables:
domains- Domain configurations with SSL settingsbackends- Backend service definitions linked to domainsbackend_servers- Individual servers within backend groups
-
Template System - Jinja2 templates for HAProxy configuration generation:
hap_header.tpl- Global HAProxy settings, defaults, and HTTP/2 tuninghap_backend.tpl- Backend server definitionshap_listener.tpl- Frontend listener configurations with rate limitinghap_letsencrypt.tpl- SSL certificate configurationshap_security_tables.tpl- Stats frontend and security stick tables- Template override support for custom backend configurations
-
Certificate Management - Automated SSL certificate handling:
- Let's Encrypt integration with certbot
- Self-signed certificate fallback for development
- Certificate renewal automation via cron
- Certificate download endpoints for external services
Configuration Flow
- Domain added via
/api/domainendpoint → Database updated generate_config()function → Reads database, renders Jinja2 templates → Writes/etc/haproxy/haproxy.cfg- HAProxy reload via socket API (
/tmp/haproxy-cli) or process restart - SSL certificate generation via Let's Encrypt or self-signed fallback
Key Design Patterns
- Template-driven configuration: HAProxy config generated from modular Jinja2 templates
- Database-backed state: All configuration persisted in SQLite for reliability
- API-first design: All operations exposed via REST endpoints
- Process monitoring: Health checks and automatic HAProxy restart capabilities
- Comprehensive logging: Operation logging with error alerting support
Authentication & Security
- Optional API key authentication controlled by
HAPROXY_API_KEYenvironment variable - All API endpoints (except
/healthand/) require Bearer token when API key is set - Certificate private keys combined with certificates in HAProxy-compatible format
- Default backend page for unmatched domains instead of exposing HAProxy errors
Rate Limiting & Connection Limits (hap_listener.tpl)
- Stick table:
type ip size 200k expire 10mtrackingconn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(30s) - Tracks real client IP via
var(txn.real_ip)to work correctly behind Cloudflare/proxies - Rate limit thresholds:
- Tarpit at 3000 req/10s (300 req/s)
- Hard block (deny) at 5000 req/10s (500 req/s)
- Connection rate limit: 500/10s
- Concurrent connection limit: 500
- Error rate limit: 100/30s
- Whitelist bypasses (exempt from rate limits):
is_local— RFC1918 private address rangesis_trusted_ip— source IPs listed intrusted_ips.listis_whitelisted— real IPs (from proxy headers) matched intrusted_ips.map
Trusted IP Whitelist Files
trusted_ips.list— Source IP whitelist for rate limit bypass (one CIDR/IP per line)trusted_ips.map— Real IP whitelist for proxy-header matching (format:<IP> 1)- Both files are baked into the Docker image via
COPYin the Dockerfile - Ship as comment-only templates (no real IPs). Add trusted IPs locally and do not commit them — this repo is mirrored publicly. Entries persist in the
/etc/haproxynamed volume across recreates
Timeout Hardening (hap_header.tpl)
timeout http-request: 300s -> 30s (slowloris protection)timeout connect: 120s -> 10stimeout client: 10m -> 5mtimeout http-keep-alive: 120s -> 30s
HTTP/2 Protection (hap_header.tpl)
tune.h2.fe.max-total-streams 2000— limits total streams per HTTP/2 connectiontune.h2.fe.glitches-threshold 50— CVE-2023-44487 Rapid Reset protection
Stats Frontend (hap_security_tables.tpl)
- HAProxy stats page bound to
127.0.0.1:8404(localhost only, accessible inside container) - Template:
templates/hap_security_tables.tpl
Deployment Context
- Designed to run as Docker container with persistent volumes for certificates and configurations
- Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
- Stats page on port 8404 (localhost only inside container)
- Management interface on port 8000 should be firewall-protected in production
- Dockerfile HEALTHCHECK runs
scripts/healthcheck.sh: port 8000/health(4s max) AND port 80 (HAProxy). WHP'slib-edge-container.shpasses its own--health-cmd, which REPLACES the image's; it must call/haproxy/scripts/healthcheck.shor the fleet never probes :8000 - The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway (
HAPROXY_MGR_API_ALLOWED_PEERS, default127.0.0.1,::1,gateway; ports viaHAPROXY_MGR_PEER_FILTER_PORTS, default8000). haproxy-manager sharesclient-netwith every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP-p 127.0.0.1:8000:8000publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added - gunicorn uses
haproxy_gunicorn_worker.TimeoutThreadWorker(gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout;--timeoutonly watches the main thread's heartbeat - Supports deployment on servers with git directory at
/root/whpand web file sync via rsync to/docker/whp/web/ - HAProxy is the 3.2 LTS series from haproxy.debian.net (
trixie-backports-3.2, native OpenSSL 3.5 QUIC); exact build in/haproxy/HAPROXY_VERSION. Debian trixie's 3.0.11 had the DNS-resolver watchdog abort (haproxy #3101) - HAProxy runs as
haproxy -W -db(foreground master, started byhaproxy_manager.spawn_haproxy_master()), NOT daemonised:daemonre-points the master's stderr at /dev/null and loses ALERTs and the ha_panic/watchdog thread dump. Its stderr is appended to/var/log/haproxy/haproxy-stderr.log(haproxy-logs volume) andscripts/haproxy-stderr-relay.pyre-emits it on the container stderr (haproxy-stderr:prefix). The in-container supervisor probes withpidofeveryHAPROXY_SUPERVISOR_INTERVAL(default 2 s) and only runs python when HAProxy is gone; Crash evidence (dump blocks opened only by the anchoredThread N is about to killheader,[ALERT] (PID) :,Current worker (N) exited with code N; ANCHORED patterns, so keywords echoed from URLs/Host/server names never match) bypasses the bucket within a budget (1000 lines / 10 min, 60 s cooldown after a bound closes a block) and goes to STDERR (journald PRIORITY 3); routine reload noise is collapsed into one[relay] reload/start ok: N routine lines collapsedline per reload and everything benign goes to STDOUT (PRIORITY 6; docker's journald driver does not parse<N>prefixes), with a per-minute[relay] suppressed N linessummary.python3 scripts/test-haproxy-stderr-relay.py(offline) covers the relay