cloud-hosting-platform/haproxy-manager-base
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a00431854d9a4cc225ada4b45c425838be21a8fc
haproxy-manager-base/CLAUDE.md
T
shadowdaoandClaude Opus 5 c8d16b6990 fix(ip-blocking): the runtime map fast path has never once run 
add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent
`add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True
whenever socat exited 0. Neither command has ever worked, on any deployment,
for the entire life of the feature -- while logging "Added IP x to runtime map"
every single time. Two independent defects:

  * NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map
    commands are worker commands. Captured verbatim on whp01:

        $ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli
        Unknown command: 'add', but maybe one of the following ones is a better match:
          @!<pid>   : send a command to the <pid> process
          ...
        $ echo $?
        0

    socat exits 0 on the rejection, so `result.returncode == 0` was true. Same
    silence PR #7 fixed on the `show table` path.
  * `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move
    on every config regeneration -- `@1 show map` on whp01 reports
    blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0.
    Hardcoding any number is wrong; the map is referenced by FILE PATH, which
    is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0).

And a third silence, which is why a response-body check alone is not enough
here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to
any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer
`Unknown map identifier.`. On the add path the reply is byte-for-byte identical
to success. Only reading the entry back can tell them apart.

IP blocking itself was never broken: update_blocked_ips_map() rewrites
/etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it.
That path is untouched and stays authoritative. What was broken is the
no-reload fast path, plus every report that it had worked.

  * haproxy_manager.py: both functions send `@1 add|del map
    /etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with
    `get map` before returning True. runtime_map_lookup()/runtime_map_keys()
    are the read-back primitives. `sync_blocked_ips` loses `clear map #0`
    (which the master socket rejected just as loudly and just as invisibly) and
    verifies the whole set with one `show map` instead of counting commands
    that did not visibly complain; it answers 207 + `runtime_map_synced: false`
    when the runtime map does not match the database.
  * haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy
    answers those with nothing on success, so an empty body is the success and
    ANY non-empty body is a rejection. That is stricter than the marker list on
    purpose -- markers only recognise rejections someone has already seen, and
    it catches `'add map' expects three parameters ...`, which matches nothing.
    HaproxyCliError carries `.responses` so `del map` answering `Key not found.`
    (the requested end state) is told apart from a real failure without regex.
  * The four callers capture the boolean instead of discarding it and report
    `runtime_map_updated` / `runtime_map_failures` in the API response and the
    operation log. A runtime failure degrades to "enforced on the reload that
    already happens two lines later" -- never to an unblocked IP, never to a
    500.
  * scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes
    on the wire (`@1` first, map by path, value `1`), classifies every captured
    response, and scans the repo's Python string literals and shell/template
    code lines for `#<id>` map references -- comments may describe the old
    form, code may not use it. Verified to fail on each defect reintroduced
    separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the-
    reply (1).
  * The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md
    and the comment in templates/hap_listener.tpl -- where every copy of it
    additionally omitted the `1`, which `-m int gt 0` needs to match.

The only template change is a comment; `haproxy -c` on the live rendered config
with it applied is clean (HAProxy 3.0.11, warnings unchanged).

Verified on whp01 against the running container (docker cp + SIGHUP, no
recreate). Before: both functions returned True and logged success while
`@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed
add lands with value "1" and the remove takes it out again; the old command
form is now classified as a failure; a `#0` map reference returns False via the
read-back. End to end through the API, `runtime_map_updated: true`, and
/api/blocked-ips/sync -- which used to be a no-op reporting a full sync --
reports 264/264 verified present.

The runtime path was isolated from the reload that normally follows it: with
NO map-file write and NO reload (same haproxy worker pid throughout), adding
100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not
in the is_local ranges) to the runtime map alone flipped a live site from
HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path
working for the first time. All test IPs were removed afterwards: 0 rows in
blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer
sites, the panel /health and `haproxy -c` are byte-identical to the baseline
taken before the change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 11:15:01 -07:00

170 lines
9.0 KiB
Markdown
Raw Blame History

# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Development Commands
### Testing
- **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
- **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
### Reading stick tables (and why it is easy to get silently wrong)
`/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Worker commands
(`show table`, `show map`, `add map`, ...) need an `@1` prefix. Without it
HAProxy answers `Unknown command: 'show', ...` **and socat still exits 0** — so
an exit-status check passes and the help text gets parsed as data. Always use
`haproxy_cli(cmd, worker=True)` in Python, which inspects the response body.
Stick-table entries are `name=value` / `name(window_ms)=value` pairs, not fixed
columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
by NAME, and treat a missing field as an ERROR — never default it to `0`. The
`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
`http_err_rate`; it holds **no history and no counter of past blocks**. What was
actually denied/tarpitted is in the edge access log on the **host** at
`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table.
This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
stick table has ever stored — for their entire existence. See the header of
`haproxy_tarpit_config.txt` and the contract test.
### Changing a runtime map (`add map` / `del map`)
Same socket, two more ways to fail silently — and both were live in
`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
existence:
* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
config-parse time and move on every config regeneration (on whp01
`blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
`add map /etc/haproxy/blocked_ips.map <ip> 1`.
* **A mutation answers NOTHING on success**, so an empty body is the only
success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
*also* answers nothing and adds nothing, so the body cannot prove an add
worked. **Read it back** with `@1 get map <path> <key>`.
* Entries must carry the value `1`; haproxy.cfg matches with
`map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
re-reads it on reload, so a failed runtime command must degrade to
"enforced on reload" and be reported, never swallowed.
### Running the Application
- **Docker Build**: `docker build -t haproxy-manager .`
- **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)
- **Container Run**: See README.md for various docker run configurations
### Monitoring and Debugging
- **Error Monitoring**: `./scripts/monitor-errors.sh` - Monitor application error logs
- **External Monitoring**: `./scripts/monitor-errors-external.sh` - External monitoring script
- **Health Check**: `curl http://localhost:8000/health`
- **Log Files**:
- `/var/log/haproxy-manager.log` - General application logs
- `/var/log/haproxy-manager-errors.log` - Error logs for alerting
## Architecture Overview
### Core Components
1. **haproxy_manager.py** - Main Flask application providing:
- RESTful API for HAProxy configuration management
- SQLite database integration for domain/backend storage
- Let's Encrypt certificate automation
- HAProxy configuration generation from Jinja2 templates
- Optional API key authentication via `HAPROXY_API_KEY` environment variable
2. **Database Schema** - SQLite database with three main tables:
- `domains` - Domain configurations with SSL settings
- `backends` - Backend service definitions linked to domains
- `backend_servers` - Individual servers within backend groups
3. **Template System** - Jinja2 templates for HAProxy configuration generation:
- `hap_header.tpl` - Global HAProxy settings, defaults, and HTTP/2 tuning
- `hap_backend.tpl` - Backend server definitions
- `hap_listener.tpl` - Frontend listener configurations with rate limiting
- `hap_letsencrypt.tpl` - SSL certificate configurations
- `hap_security_tables.tpl` - Stats frontend and security stick tables
- Template override support for custom backend configurations
4. **Certificate Management** - Automated SSL certificate handling:
- Let's Encrypt integration with certbot
- Self-signed certificate fallback for development
- Certificate renewal automation via cron
- Certificate download endpoints for external services
### Configuration Flow
1. Domain added via `/api/domain` endpoint → Database updated
2. `generate_config()` function → Reads database, renders Jinja2 templates → Writes `/etc/haproxy/haproxy.cfg`
3. HAProxy reload via socket API (`/tmp/haproxy-cli`) or process restart
4. SSL certificate generation via Let's Encrypt or self-signed fallback
### Key Design Patterns
- **Template-driven configuration**: HAProxy config generated from modular Jinja2 templates
- **Database-backed state**: All configuration persisted in SQLite for reliability
- **API-first design**: All operations exposed via REST endpoints
- **Process monitoring**: Health checks and automatic HAProxy restart capabilities
- **Comprehensive logging**: Operation logging with error alerting support
### Authentication & Security
- Optional API key authentication controlled by `HAPROXY_API_KEY` environment variable
- All API endpoints (except `/health` and `/`) require Bearer token when API key is set
- Certificate private keys combined with certificates in HAProxy-compatible format
- Default backend page for unmatched domains instead of exposing HAProxy errors
### Rate Limiting & Connection Limits (hap_listener.tpl)
- **Stick table**: `type ip size 200k expire 10m` tracking `conn_cur`, `conn_rate(10s)`, `http_req_rate(10s)`, `http_err_rate(30s)`
- Tracks real client IP via `var(txn.real_ip)` to work correctly behind Cloudflare/proxies
- **Rate limit thresholds**:
- Tarpit at 3000 req/10s (300 req/s)
- Hard block (deny) at 5000 req/10s (500 req/s)
- Connection rate limit: 500/10s
- Concurrent connection limit: 500
- Error rate limit: 100/30s
- **Whitelist bypasses** (exempt from rate limits):
- `is_local` — RFC1918 private address ranges
- `is_trusted_ip` — source IPs listed in `trusted_ips.list`
- `is_whitelisted` — real IPs (from proxy headers) matched in `trusted_ips.map`
### Trusted IP Whitelist Files
- `trusted_ips.list` — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
- `trusted_ips.map` — Real IP whitelist for proxy-header matching (format: `<IP> 1`)
- Both files are baked into the Docker image via `COPY` in the Dockerfile
- Ship as comment-only templates (no real IPs). Add trusted IPs locally and do **not** commit them — this repo is mirrored publicly. Entries persist in the `/etc/haproxy` named volume across recreates
### Timeout Hardening (hap_header.tpl)
- `timeout http-request`: 300s -> 30s (slowloris protection)
- `timeout connect`: 120s -> 10s
- `timeout client`: 10m -> 5m
- `timeout http-keep-alive`: 120s -> 30s
### HTTP/2 Protection (hap_header.tpl)
- `tune.h2.fe.max-total-streams 2000` — limits total streams per HTTP/2 connection
- `tune.h2.fe.glitches-threshold 50` — CVE-2023-44487 Rapid Reset protection
### Stats Frontend (hap_security_tables.tpl)
- HAProxy stats page bound to `127.0.0.1:8404` (localhost only, accessible inside container)
- Template: `templates/hap_security_tables.tpl`
### Deployment Context
- Designed to run as Docker container with persistent volumes for certificates and configurations
- Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
- Stats page on port 8404 (localhost only inside container)
- Management interface on port 8000 should be firewall-protected in production
- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s`
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
- HAProxy is version 3.0.11
Reference in New Issue View Git Blame Copy Permalink