cloud-hosting-platform/haproxy-manager-base

Files

T

shadowdaoandClaude Opus 5 c8d16b6990 fix(ip-blocking): the runtime map fast path has never once run

add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent
`add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True
whenever socat exited 0. Neither command has ever worked, on any deployment,
for the entire life of the feature -- while logging "Added IP x to runtime map"
every single time. Two independent defects:

  * NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map
    commands are worker commands. Captured verbatim on whp01:

        $ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli
        Unknown command: 'add', but maybe one of the following ones is a better match:
          @!<pid>   : send a command to the <pid> process
          ...
        $ echo $?
        0

    socat exits 0 on the rejection, so `result.returncode == 0` was true. Same
    silence PR #7 fixed on the `show table` path.
  * `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move
    on every config regeneration -- `@1 show map` on whp01 reports
    blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0.
    Hardcoding any number is wrong; the map is referenced by FILE PATH, which
    is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0).

And a third silence, which is why a response-body check alone is not enough
here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to
any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer
`Unknown map identifier.`. On the add path the reply is byte-for-byte identical
to success. Only reading the entry back can tell them apart.

IP blocking itself was never broken: update_blocked_ips_map() rewrites
/etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it.
That path is untouched and stays authoritative. What was broken is the
no-reload fast path, plus every report that it had worked.

  * haproxy_manager.py: both functions send `@1 add|del map
    /etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with
    `get map` before returning True. runtime_map_lookup()/runtime_map_keys()
    are the read-back primitives. `sync_blocked_ips` loses `clear map #0`
    (which the master socket rejected just as loudly and just as invisibly) and
    verifies the whole set with one `show map` instead of counting commands
    that did not visibly complain; it answers 207 + `runtime_map_synced: false`
    when the runtime map does not match the database.
  * haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy
    answers those with nothing on success, so an empty body is the success and
    ANY non-empty body is a rejection. That is stricter than the marker list on
    purpose -- markers only recognise rejections someone has already seen, and
    it catches `'add map' expects three parameters ...`, which matches nothing.
    HaproxyCliError carries `.responses` so `del map` answering `Key not found.`
    (the requested end state) is told apart from a real failure without regex.
  * The four callers capture the boolean instead of discarding it and report
    `runtime_map_updated` / `runtime_map_failures` in the API response and the
    operation log. A runtime failure degrades to "enforced on the reload that
    already happens two lines later" -- never to an unblocked IP, never to a
    500.
  * scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes
    on the wire (`@1` first, map by path, value `1`), classifies every captured
    response, and scans the repo's Python string literals and shell/template
    code lines for `#<id>` map references -- comments may describe the old
    form, code may not use it. Verified to fail on each defect reintroduced
    separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the-
    reply (1).
  * The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md
    and the comment in templates/hap_listener.tpl -- where every copy of it
    additionally omitted the `1`, which `-m int gt 0` needs to match.

The only template change is a comment; `haproxy -c` on the live rendered config
with it applied is clean (HAProxy 3.0.11, warnings unchanged).

Verified on whp01 against the running container (docker cp + SIGHUP, no
recreate). Before: both functions returned True and logged success while
`@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed
add lands with value "1" and the remove takes it out again; the old command
form is now classified as a failure; a `#0` map reference returns False via the
read-back. End to end through the API, `runtime_map_updated: true`, and
/api/blocked-ips/sync -- which used to be a no-op reporting a full sync --
reports 264/264 verified present.

The runtime path was isolated from the reload that normally follows it: with
NO map-file write and NO reload (same haproxy worker pid throughout), adding
100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not
in the is_local ranges) to the runtime map alone flipped a live site from
HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path
working for the first time. All test IPs were removed afterwards: 0 rows in
blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer
sites, the panel /health and `haproxy -c` are byte-identical to the baseline
taken before the change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-22 11:15:01 -07:00

9.0 KiB

Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Development Commands

Testing

API Testing: ./scripts/test-api.sh - Tests all API endpoints with optional authentication
Certificate Request Testing: ./scripts/test-certificate-request.sh - Tests certificate generation endpoints
Stick-table contract: python3 scripts/test-stick-table-contract.py - offline; holds the templates' store clauses, STICK_TABLE_FIELD_CONTRACT, and every consumer to each other. Run it after touching any stick-table line.
Runtime-map contract: python3 scripts/test-runtime-map-contract.py - offline; asserts the runtime map commands are @1-prefixed, reference the map by FILE PATH (never #<id>), carry the value 1, and that every captured rejection is classified as a failure. Run it after touching any add map/del map/clear map path.
Manual Testing: Run curl commands against http://localhost:8000 endpoints as shown in README.md

Reading stick tables (and why it is easy to get silently wrong)

/tmp/haproxy-cli is HAProxy's master CLI socket. Worker commands (show table, show map, add map, ...) need an @1 prefix. Without it HAProxy answers Unknown command: 'show', ... and socat still exits 0 — so an exit-status check passes and the help text gets parsed as data. Always use haproxy_cli(cmd, worker=True) in Python, which inspects the response body.

Stick-table entries are name=value / name(window_ms)=value pairs, not fixed columns; the first token is an allocation pointer (0x...:), not the key. Parse by NAME, and treat a missing field as an ERROR — never default it to 0. The web table stores only conn_cur, conn_rate, http_req_rate, http_err_rate; it holds no history and no counter of past blocks. What was actually denied/tarpitted is in the edge access log on the host at /var/log/haproxy.log (shipped 2026.08.8), not in any stick table.

This is written down because /api/security/stats and show-tarpit-ips.sh reported "Scan Count"/"BLOCKED" figures parsed from gpc0/gpc1 — fields no stick table has ever stored — for their entire existence. See the header of haproxy_tarpit_config.txt and the contract test.

Changing a runtime map (`add map` / `del map`)

Same socket, two more ways to fail silently — and both were live in add_ip_to_runtime_map()/remove_ip_from_runtime_map() for their whole existence:

Reference the map by FILE PATH, never #<id>. Ids are assigned at config-parse time and move on every config regeneration (on whp01 blocked_ips.map is 37, trusted_ips.map is 10 — there is no id 0). Use add map /etc/haproxy/blocked_ips.map <ip> 1.
A mutation answers NOTHING on success, so an empty body is the only success — any output at all is a rejection. Worse, @1 add map #0 <ip> 1 also answers nothing and adds nothing, so the body cannot prove an add worked. Read it back with @1 get map <path> <key>.
Entries must carry the value 1; haproxy.cfg matches with map_ip(...,0) -m int gt 0, so a valueless entry does not block.

In Python use haproxy_cli(cmd, worker=True, expect_empty=True) for mutations and runtime_map_lookup() / runtime_map_keys() to verify. The runtime map is only a fast path: /etc/haproxy/blocked_ips.map is authoritative and HAProxy re-reads it on reload, so a failed runtime command must degrade to "enforced on reload" and be reported, never swallowed.

Running the Application

Docker Build: docker build -t haproxy-manager .
Local Development: python haproxy_manager.py (requires HAProxy, certbot, and dependencies installed)
Container Run: See README.md for various docker run configurations

Monitoring and Debugging

Error Monitoring: ./scripts/monitor-errors.sh - Monitor application error logs
External Monitoring: ./scripts/monitor-errors-external.sh - External monitoring script
Health Check: curl http://localhost:8000/health
Log Files:
- /var/log/haproxy-manager.log - General application logs
- /var/log/haproxy-manager-errors.log - Error logs for alerting

Architecture Overview

Core Components

haproxy_manager.py - Main Flask application providing:
- RESTful API for HAProxy configuration management
- SQLite database integration for domain/backend storage
- Let's Encrypt certificate automation
- HAProxy configuration generation from Jinja2 templates
- Optional API key authentication via HAPROXY_API_KEY environment variable
Database Schema - SQLite database with three main tables:
- domains - Domain configurations with SSL settings
- backends - Backend service definitions linked to domains
- backend_servers - Individual servers within backend groups
Template System - Jinja2 templates for HAProxy configuration generation:
- hap_header.tpl - Global HAProxy settings, defaults, and HTTP/2 tuning
- hap_backend.tpl - Backend server definitions
- hap_listener.tpl - Frontend listener configurations with rate limiting
- hap_letsencrypt.tpl - SSL certificate configurations
- hap_security_tables.tpl - Stats frontend and security stick tables
- Template override support for custom backend configurations
Certificate Management - Automated SSL certificate handling:
- Let's Encrypt integration with certbot
- Self-signed certificate fallback for development
- Certificate renewal automation via cron
- Certificate download endpoints for external services

Configuration Flow

Domain added via /api/domain endpoint → Database updated
generate_config() function → Reads database, renders Jinja2 templates → Writes /etc/haproxy/haproxy.cfg
HAProxy reload via socket API (/tmp/haproxy-cli) or process restart
SSL certificate generation via Let's Encrypt or self-signed fallback

Key Design Patterns

Template-driven configuration: HAProxy config generated from modular Jinja2 templates
Database-backed state: All configuration persisted in SQLite for reliability
API-first design: All operations exposed via REST endpoints
Process monitoring: Health checks and automatic HAProxy restart capabilities
Comprehensive logging: Operation logging with error alerting support

Authentication & Security

Optional API key authentication controlled by HAPROXY_API_KEY environment variable
All API endpoints (except /health and /) require Bearer token when API key is set
Certificate private keys combined with certificates in HAProxy-compatible format
Default backend page for unmatched domains instead of exposing HAProxy errors

Rate Limiting & Connection Limits (hap_listener.tpl)

Stick table: type ip size 200k expire 10m tracking conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
Tracks real client IP via var(txn.real_ip) to work correctly behind Cloudflare/proxies
Rate limit thresholds:
- Tarpit at 3000 req/10s (300 req/s)
- Hard block (deny) at 5000 req/10s (500 req/s)
- Connection rate limit: 500/10s
- Concurrent connection limit: 500
- Error rate limit: 100/30s
Whitelist bypasses (exempt from rate limits):
- is_local — RFC1918 private address ranges
- is_trusted_ip — source IPs listed in trusted_ips.list
- is_whitelisted — real IPs (from proxy headers) matched in trusted_ips.map

Trusted IP Whitelist Files

trusted_ips.list — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
trusted_ips.map — Real IP whitelist for proxy-header matching (format: <IP> 1)
Both files are baked into the Docker image via COPY in the Dockerfile
Ship as comment-only templates (no real IPs). Add trusted IPs locally and do not commit them — this repo is mirrored publicly. Entries persist in the /etc/haproxy named volume across recreates

Timeout Hardening (hap_header.tpl)

timeout http-request: 300s -> 30s (slowloris protection)
timeout connect: 120s -> 10s
timeout client: 10m -> 5m
timeout http-keep-alive: 120s -> 30s

HTTP/2 Protection (hap_header.tpl)

tune.h2.fe.max-total-streams 2000 — limits total streams per HTTP/2 connection
tune.h2.fe.glitches-threshold 50 — CVE-2023-44487 Rapid Reset protection

Stats Frontend (hap_security_tables.tpl)

HAProxy stats page bound to 127.0.0.1:8404 (localhost only, accessible inside container)
Template: templates/hap_security_tables.tpl

Deployment Context

Designed to run as Docker container with persistent volumes for certificates and configurations
Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
Stats page on port 8404 (localhost only inside container)
Management interface on port 8000 should be firewall-protected in production
Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with start-period=60s and timeout=10s
Supports deployment on servers with git directory at /root/whp and web file sync via rsync to /docker/whp/web/
HAProxy is version 3.0.11

9.0 KiB Raw Blame History