shadowdaoandClaude Opus 5.5
0ce144d2fb
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.
The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).
- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
* per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
re-applied on keep-alive reuse; bounds socket I/O, not handling time
* watchdog: if every pool thread has been in the same request for more
than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
all thread stacks and exit so the master respawns the worker
* client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
60s) as defence in depth for any future in-process client; /health now
reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
added to the Docker build gate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
173 lines
10 KiB
Markdown
173 lines
10 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Development Commands
|
|
|
|
### Testing
|
|
- **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
|
|
- **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
|
|
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
|
|
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
|
|
- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
|
|
- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~30s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides `ThreadWorker.enqueue_req`).
|
|
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
|
|
|
|
### Reading stick tables (and why it is easy to get silently wrong)
|
|
|
|
`/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Worker commands
|
|
(`show table`, `show map`, `add map`, ...) need an `@1` prefix. Without it
|
|
HAProxy answers `Unknown command: 'show', ...` **and socat still exits 0** — so
|
|
an exit-status check passes and the help text gets parsed as data. Always use
|
|
`haproxy_cli(cmd, worker=True)` in Python, which inspects the response body.
|
|
|
|
Stick-table entries are `name=value` / `name(window_ms)=value` pairs, not fixed
|
|
columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
|
|
by NAME, and treat a missing field as an ERROR — never default it to `0`. The
|
|
`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
|
|
`http_err_rate`; it holds **no history and no counter of past blocks**. What was
|
|
actually denied/tarpitted is in the edge access log on the **host** at
|
|
`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table.
|
|
|
|
This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
|
|
reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
|
|
stick table has ever stored — for their entire existence. See the header of
|
|
`haproxy_tarpit_config.txt` and the contract test.
|
|
|
|
### Changing a runtime map (`add map` / `del map`)
|
|
|
|
Same socket, two more ways to fail silently — and both were live in
|
|
`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
|
|
existence:
|
|
|
|
* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
|
|
config-parse time and move on every config regeneration (on whp01
|
|
`blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
|
|
`add map /etc/haproxy/blocked_ips.map <ip> 1`.
|
|
* **A mutation answers NOTHING on success**, so an empty body is the only
|
|
success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
|
|
*also* answers nothing and adds nothing, so the body cannot prove an add
|
|
worked. **Read it back** with `@1 get map <path> <key>`.
|
|
* Entries must carry the value `1`; haproxy.cfg matches with
|
|
`map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
|
|
|
|
In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
|
|
and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
|
|
only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
|
|
re-reads it on reload, so a failed runtime command must degrade to
|
|
"enforced on reload" and be reported, never swallowed.
|
|
|
|
### Running the Application
|
|
- **Docker Build**: `docker build -t haproxy-manager .`
|
|
- **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)
|
|
- **Container Run**: See README.md for various docker run configurations
|
|
|
|
### Monitoring and Debugging
|
|
- **Error Monitoring**: `./scripts/monitor-errors.sh` - Monitor application error logs
|
|
- **External Monitoring**: `./scripts/monitor-errors-external.sh` - External monitoring script
|
|
- **Health Check**: `curl http://localhost:8000/health`
|
|
- **Log Files**:
|
|
- `/var/log/haproxy-manager.log` - General application logs
|
|
- `/var/log/haproxy-manager-errors.log` - Error logs for alerting
|
|
|
|
## Architecture Overview
|
|
|
|
### Core Components
|
|
|
|
1. **haproxy_manager.py** - Main Flask application providing:
|
|
- RESTful API for HAProxy configuration management
|
|
- SQLite database integration for domain/backend storage
|
|
- Let's Encrypt certificate automation
|
|
- HAProxy configuration generation from Jinja2 templates
|
|
- Optional API key authentication via `HAPROXY_API_KEY` environment variable
|
|
|
|
2. **Database Schema** - SQLite database with three main tables:
|
|
- `domains` - Domain configurations with SSL settings
|
|
- `backends` - Backend service definitions linked to domains
|
|
- `backend_servers` - Individual servers within backend groups
|
|
|
|
3. **Template System** - Jinja2 templates for HAProxy configuration generation:
|
|
- `hap_header.tpl` - Global HAProxy settings, defaults, and HTTP/2 tuning
|
|
- `hap_backend.tpl` - Backend server definitions
|
|
- `hap_listener.tpl` - Frontend listener configurations with rate limiting
|
|
- `hap_letsencrypt.tpl` - SSL certificate configurations
|
|
- `hap_security_tables.tpl` - Stats frontend and security stick tables
|
|
- Template override support for custom backend configurations
|
|
|
|
4. **Certificate Management** - Automated SSL certificate handling:
|
|
- Let's Encrypt integration with certbot
|
|
- Self-signed certificate fallback for development
|
|
- Certificate renewal automation via cron
|
|
- Certificate download endpoints for external services
|
|
|
|
### Configuration Flow
|
|
|
|
1. Domain added via `/api/domain` endpoint → Database updated
|
|
2. `generate_config()` function → Reads database, renders Jinja2 templates → Writes `/etc/haproxy/haproxy.cfg`
|
|
3. HAProxy reload via socket API (`/tmp/haproxy-cli`) or process restart
|
|
4. SSL certificate generation via Let's Encrypt or self-signed fallback
|
|
|
|
### Key Design Patterns
|
|
|
|
- **Template-driven configuration**: HAProxy config generated from modular Jinja2 templates
|
|
- **Database-backed state**: All configuration persisted in SQLite for reliability
|
|
- **API-first design**: All operations exposed via REST endpoints
|
|
- **Process monitoring**: Health checks and automatic HAProxy restart capabilities
|
|
- **Comprehensive logging**: Operation logging with error alerting support
|
|
|
|
### Authentication & Security
|
|
|
|
- Optional API key authentication controlled by `HAPROXY_API_KEY` environment variable
|
|
- All API endpoints (except `/health` and `/`) require Bearer token when API key is set
|
|
- Certificate private keys combined with certificates in HAProxy-compatible format
|
|
- Default backend page for unmatched domains instead of exposing HAProxy errors
|
|
|
|
### Rate Limiting & Connection Limits (hap_listener.tpl)
|
|
|
|
- **Stick table**: `type ip size 200k expire 10m` tracking `conn_cur`, `conn_rate(10s)`, `http_req_rate(10s)`, `http_err_rate(30s)`
|
|
- Tracks real client IP via `var(txn.real_ip)` to work correctly behind Cloudflare/proxies
|
|
- **Rate limit thresholds**:
|
|
- Tarpit at 3000 req/10s (300 req/s)
|
|
- Hard block (deny) at 5000 req/10s (500 req/s)
|
|
- Connection rate limit: 500/10s
|
|
- Concurrent connection limit: 500
|
|
- Error rate limit: 100/30s
|
|
- **Whitelist bypasses** (exempt from rate limits):
|
|
- `is_local` — RFC1918 private address ranges
|
|
- `is_trusted_ip` — source IPs listed in `trusted_ips.list`
|
|
- `is_whitelisted` — real IPs (from proxy headers) matched in `trusted_ips.map`
|
|
|
|
### Trusted IP Whitelist Files
|
|
|
|
- `trusted_ips.list` — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
|
|
- `trusted_ips.map` — Real IP whitelist for proxy-header matching (format: `<IP> 1`)
|
|
- Both files are baked into the Docker image via `COPY` in the Dockerfile
|
|
- Ship as comment-only templates (no real IPs). Add trusted IPs locally and do **not** commit them — this repo is mirrored publicly. Entries persist in the `/etc/haproxy` named volume across recreates
|
|
|
|
### Timeout Hardening (hap_header.tpl)
|
|
|
|
- `timeout http-request`: 300s -> 30s (slowloris protection)
|
|
- `timeout connect`: 120s -> 10s
|
|
- `timeout client`: 10m -> 5m
|
|
- `timeout http-keep-alive`: 120s -> 30s
|
|
|
|
### HTTP/2 Protection (hap_header.tpl)
|
|
|
|
- `tune.h2.fe.max-total-streams 2000` — limits total streams per HTTP/2 connection
|
|
- `tune.h2.fe.glitches-threshold 50` — CVE-2023-44487 Rapid Reset protection
|
|
|
|
### Stats Frontend (hap_security_tables.tpl)
|
|
|
|
- HAProxy stats page bound to `127.0.0.1:8404` (localhost only, accessible inside container)
|
|
- Template: `templates/hap_security_tables.tpl`
|
|
|
|
### Deployment Context
|
|
|
|
- Designed to run as Docker container with persistent volumes for certificates and configurations
|
|
- Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
|
|
- Stats page on port 8404 (localhost only inside container)
|
|
- Management interface on port 8000 should be firewall-protected in production
|
|
- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
|
|
- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
|
|
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
|
|
- HAProxy is version 3.0.11 |