Files
haproxy-manager-base/CLAUDE.md
shadowdaoandClaude Sonnet 5.5 aebb68fe9d ci: make tag verification independent of buildx version
buildx v0.28.0 ignores `imagetools inspect --format '{{.Manifest.Digest}}'`
and prints the whole inspect text, so the verify step rejected every ref as a
"bad digest" (CI run 5381). Move the gate into scripts/verify-published-tags.sh:
the index digest is the sha256 of `inspect --raw`, cross-checked against the
anchored `Digest:` line and a bare --format output when present; platforms are
read from the raw JSON. All other checks (every tag in every registry, platform
match, one digest across registries, any miss fails) are unchanged. Adds an
offline test with a stub emulating 0.28 and 0.38 output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:30:21 -08:00

184 lines
17 KiB
Markdown

# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Development Commands
### Testing
- **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
- **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
- **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine.
- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
- **Multi-arch index assembly (coraza-spoa)**: `python3 scripts/test-assemble-multiarch-index.py` - offline (NOT run in the image build; it also reads `.gitea/workflows/build-push-coraza.yaml`); `scripts/assemble-multiarch-index.sh` turns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome is `success` (failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing. `scripts/e2e-coraza-multiarch.sh` rehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch").
- **Publish verify gate**: `python3 scripts/test-verify-published-tags.py` - offline (needs `jq`; NOT run in the image build); `scripts/verify-published-tags.sh` is the last build-push.yaml step: every tag in every registry must resolve, carry the assembled platforms, and share ONE index digest. The digest is the sha256 of `imagetools inspect --raw` cross-checked against the text `Digest:` line / bare `--format` output (buildx 0.28 ignores `--format` and prints full text; CI run 5381).
- **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash".
- **gunicorn access logging**: `python3 scripts/test-gunicorn-access-logging.py` - real gunicorn; :8080 has no access log, :8000 access goes to `/var/log/haproxy/api-access.log` (rotated by `scripts/gunicorn-access-rotate.py`), nothing per-request on stdout (journald rate-limit/Graylog flood), gunicorn errors stay on stderr. See docs/logging.md. Run after touching start-up.sh's gunicorn lines.
- **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`.
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
### Reading stick tables (and why it is easy to get silently wrong)
`/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Worker commands
(`show table`, `show map`, `add map`, ...) need an `@1` prefix. Without it
HAProxy answers `Unknown command: 'show', ...` **and socat still exits 0** — so
an exit-status check passes and the help text gets parsed as data. Always use
`haproxy_cli(cmd, worker=True)` in Python, which inspects the response body.
Stick-table entries are `name=value` / `name(window_ms)=value` pairs, not fixed
columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
by NAME, and treat a missing field as an ERROR — never default it to `0`. The
`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
`http_err_rate`; it holds **no history and no counter of past blocks**. What was
actually denied/tarpitted is in the edge access log, since 2026.11.2 INSIDE the
container at `/var/log/haproxy/access.log` (in-container receiver, volume
`haproxy-logs`; README "Access Log"), not in any stick table. (2026.08.8-2026.11.1
sent it to host rsyslog on 172.18.0.1:514, which had no listener on the fleet.)
This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
stick table has ever stored — for their entire existence. See the header of
`haproxy_tarpit_config.txt` and the contract test.
### Changing a runtime map (`add map` / `del map`)
Same socket, two more ways to fail silently — and both were live in
`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
existence:
* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
config-parse time and move on every config regeneration (on whp01
`blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
`add map /etc/haproxy/blocked_ips.map <ip> 1`.
* **A mutation answers NOTHING on success**, so an empty body is the only
success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
*also* answers nothing and adds nothing, so the body cannot prove an add
worked. **Read it back** with `@1 get map <path> <key>`.
* Entries must carry the value `1`; haproxy.cfg matches with
`map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
re-reads it on reload, so a failed runtime command must degrade to
"enforced on reload" and be reported, never swallowed.
### Running the Application
- **Docker Build**: `docker build -t haproxy-manager .`
- **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)
- **Container Run**: See README.md for various docker run configurations
### Monitoring and Debugging
- **Error Monitoring**: `./scripts/monitor-errors.sh` - Monitor application error logs
- **External Monitoring**: `./scripts/monitor-errors-external.sh` - External monitoring script
- **Health Check**: `curl http://localhost:8000/health`
- **Log Files**:
- `/var/log/haproxy-manager.log` - General application logs
- `/var/log/haproxy-manager-errors.log` - Error logs for alerting
## Architecture Overview
### Core Components
1. **haproxy_manager.py** - Main Flask application providing:
- RESTful API for HAProxy configuration management
- SQLite database integration for domain/backend storage
- Let's Encrypt certificate automation
- HAProxy configuration generation from Jinja2 templates
- Optional API key authentication via `HAPROXY_API_KEY` environment variable
2. **Database Schema** - SQLite database with three main tables:
- `domains` - Domain configurations with SSL settings
- `backends` - Backend service definitions linked to domains
- `backend_servers` - Individual servers within backend groups
3. **Template System** - Jinja2 templates for HAProxy configuration generation:
- `hap_header.tpl` - Global HAProxy settings, defaults, and HTTP/2 tuning
- `hap_backend.tpl` - Backend server definitions
- `hap_listener.tpl` - Frontend listener configurations with rate limiting
- `hap_letsencrypt.tpl` - SSL certificate configurations
- `hap_security_tables.tpl` - Stats frontend and security stick tables
- Template override support for custom backend configurations
4. **Certificate Management** - Automated SSL certificate handling:
- Let's Encrypt integration with certbot
- Self-signed certificate fallback for development
- Certificate renewal automation via cron
- Certificate download endpoints for external services
### Configuration Flow
1. Domain added via `/api/domain` endpoint → Database updated
2. `generate_config()` function → Reads database, renders Jinja2 templates → Writes `/etc/haproxy/haproxy.cfg`
3. HAProxy reload via socket API (`/tmp/haproxy-cli`) or process restart
4. SSL certificate generation via Let's Encrypt or self-signed fallback
### Key Design Patterns
- **Template-driven configuration**: HAProxy config generated from modular Jinja2 templates
- **Database-backed state**: All configuration persisted in SQLite for reliability
- **API-first design**: All operations exposed via REST endpoints
- **Process monitoring**: Health checks and automatic HAProxy restart capabilities
- **Comprehensive logging**: Operation logging with error alerting support
### Authentication & Security
- Optional API key authentication controlled by `HAPROXY_API_KEY` environment variable
- All API endpoints (except `/health` and `/`) require Bearer token when API key is set
- Certificate private keys combined with certificates in HAProxy-compatible format
- Default backend page for unmatched domains instead of exposing HAProxy errors
### Rate Limiting & Connection Limits (hap_listener.tpl)
- **Stick table**: `type ip size 200k expire 10m` tracking `conn_cur`, `conn_rate(10s)`, `http_req_rate(10s)`, `http_err_rate(30s)`
- Tracks real client IP via `var(txn.real_ip)` to work correctly behind Cloudflare/proxies
- **Rate limit thresholds**:
- Tarpit at 3000 req/10s (300 req/s)
- Hard block (deny) at 5000 req/10s (500 req/s)
- Connection rate limit: 500/10s
- Concurrent connection limit: 500
- Error rate limit: 100/30s
- **Whitelist bypasses** (exempt from rate limits):
- `is_local` — RFC1918 private address ranges
- `is_trusted_ip` — source IPs listed in `trusted_ips.list`
- `is_whitelisted` — real IPs (from proxy headers) matched in `trusted_ips.map`
### Trusted IP Whitelist Files
- `trusted_ips.list` — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
- `trusted_ips.map` — Real IP whitelist for proxy-header matching (format: `<IP> 1`)
- Both files are baked into the Docker image via `COPY` in the Dockerfile
- Ship as comment-only templates (no real IPs). Add trusted IPs locally and do **not** commit them — this repo is mirrored publicly. Entries persist in the `/etc/haproxy` named volume across recreates
### Timeout Hardening (hap_header.tpl)
- `timeout http-request`: 300s -> 30s (slowloris protection)
- `timeout connect`: 120s -> 10s
- `timeout client`: 10m -> 5m
- `timeout http-keep-alive`: 120s -> 30s
### HTTP/2 Protection (hap_header.tpl)
- `tune.h2.fe.max-total-streams 2000` — limits total streams per HTTP/2 connection
- `tune.h2.fe.glitches-threshold 50` — CVE-2023-44487 Rapid Reset protection
### Stats Frontend (hap_security_tables.tpl)
- HAProxy stats page bound to `127.0.0.1:8404` (localhost only, accessible inside container)
- Template: `templates/hap_security_tables.tpl`
### Deployment Context
- Designed to run as Docker container with persistent volumes for certificates and configurations
- Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
- Stats page on port 8404 (localhost only inside container)
- Management interface on port 8000 should be firewall-protected in production
- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
- HAProxy is the 3.2 LTS series from haproxy.debian.net (`trixie-backports-3.2`, native OpenSSL 3.5 QUIC); exact build in `/haproxy/HAPROXY_VERSION`. Debian trixie's 3.0.11 had the DNS-resolver watchdog abort (haproxy #3101)
- HAProxy runs as `haproxy -W -db` (foreground master, started by `haproxy_manager.spawn_haproxy_master()`), NOT daemonised: `daemon` re-points the master's stderr at /dev/null and loses ALERTs and the ha_panic/watchdog thread dump. Its stderr is appended to `/var/log/haproxy/haproxy-stderr.log` (haproxy-logs volume) and `scripts/haproxy-stderr-relay.py` re-emits it on the container stderr (`haproxy-stderr: ` prefix). The in-container supervisor probes with `pidof` every `HAPROXY_SUPERVISOR_INTERVAL` (default 2 s) and only runs python when HAProxy is gone; Crash evidence (dump blocks opened only by the anchored `Thread N is about to kill` header, `[ALERT] (PID) :`, `Current worker (N) exited with code N`; ANCHORED patterns, so keywords echoed from URLs/Host/server names never match) bypasses the bucket within a budget (1000 lines / 10 min, 60 s cooldown after a bound closes a block) and goes to STDERR (journald PRIORITY 3); routine reload noise is collapsed into one `[relay] reload/start ok: N routine lines collapsed` line per reload and everything benign goes to STDOUT (PRIORITY 6; docker's journald driver does not parse `<N>` prefixes), with a per-minute `[relay] suppressed N lines` summary. `python3 scripts/test-haproxy-stderr-relay.py` (offline) covers the relay