cloud-hosting-platform/haproxy-manager-base
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b6a62e7f9f748b171985e03d937204b66b0f123a
haproxy-manager-base/CLAUDE.md
T
shadowdaoandClaude Opus 5 b6a62e7f9f fix(security-stats): stop reporting counters the stick tables never stored 
/api/security/stats and scripts/show-tarpit-ips.sh reported "Scan Count",
"offense count" and BLOCKED/TARPITTED status parsed from gpc0/gpc1. No stick
table in this repo has ever stored a general-purpose counter -- the `web` table
stores conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s), and
the two brute-force tables store http_req_rate(60s). Every one of those figures
was fabricated, and an operator was making decisions on them.

Three independent silences kept it alive:

  * `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
    and the loop `continue`d -- so the endpoint always answered
    `active_threats: 0` with an empty list. Live on whp01 it also reported
    parts[0], the `0x...:` allocation pointer, as the source IP.
  * The command was sent to /tmp/haproxy-cli WITHOUT the `@1` worker prefix.
    That is the MASTER CLI socket, which answers "Unknown command: 'show' ..."
    -- and socat still exits 0, so the `returncode != 0` guard never fired.
    `total_tracked_ips` was the line count of that help text (8) while the real
    table held 388 entries.
  * The shell consumers wrote `gpc0=${gpc0:-0}`, rendering a field that does
    not exist as a confident zero.

Report what the tables actually store, rather than adding gpc counters to make
the old semantics real. Adding them would mean editing hap_listener.tpl -- the
one change here with a silent-total-outage failure mode -- to rebuild
enforcement history that the edge access log (shipped 2026.08.8, on the host at
/var/log/haproxy.log) already records per request, with status codes,
termination states and request references the stick table could never hold.

  * haproxy_manager.py: STICK_TABLE_FIELD_CONTRACT names what each table
    stores. haproxy_cli() sends worker commands with `@1`, falls back to the
    bare form for a plain stats socket, and inspects the RESPONSE BODY because
    socat's exit status is worthless here. parse_stick_table_entry() reads
    name=value / name(window_ms)=value pairs by NAME, never by position.
    read_stick_table() RAISES -- naming the field -- when a row is missing a
    contract field, instead of defaulting it to 0.
  * /api/security/stats returns the four real counters with their windows, the
    true `used:` count, and no invented threat_level/blocked/offense_count.
    Fewer numbers, all of them real.
  * scripts/show-edge-ip-rates.sh replaces the fabricated report; the four
    expected fields are declared once as EXPECTED_FIELDS and drive the parser.
    show-tarpit-ips.sh becomes a shim that explains why its numbers are gone
    and points at where tarpit events actually live.
  * monitor-attacks.sh loses fourteen fabricated "threat" categories and a
    composite threat score, all permanently zero; its access-log section now
    says the log is on the host instead of silently printing nothing.
  * haproxy_tarpit_config.txt -- the never-shipped design sketch these counters
    were copied from -- gets a NOT IMPLEMENTED banner.
  * scripts/test-stick-table-contract.py (offline, 21 tests) holds the
    templates' `store` clauses, STICK_TABLE_FIELD_CONTRACT and every consumer
    to each other, and asserts each loud-failure path against the real captured
    responses. Template and consumers can no longer drift apart quietly.

No template is touched, so haproxy.cfg is unchanged.

Verified on whp01: total_tracked_ips now tracks `used:` exactly (511 vs the
table's 511, was 8 vs 388), and per-IP values match `show table web key <ip>`
field for field. haproxy PIDs unmoved, `haproxy -c` warnings unchanged, five
customer sites HTTP 200.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 10:48:49 -07:00

7.5 KiB
Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Development Commands

Testing

  • API Testing: ./scripts/test-api.sh - Tests all API endpoints with optional authentication
  • Certificate Request Testing: ./scripts/test-certificate-request.sh - Tests certificate generation endpoints
  • Stick-table contract: python3 scripts/test-stick-table-contract.py - offline; holds the templates' store clauses, STICK_TABLE_FIELD_CONTRACT, and every consumer to each other. Run it after touching any stick-table line.
  • Manual Testing: Run curl commands against http://localhost:8000 endpoints as shown in README.md

Reading stick tables (and why it is easy to get silently wrong)

/tmp/haproxy-cli is HAProxy's master CLI socket. Worker commands (show table, show map, add map, ...) need an @1 prefix. Without it HAProxy answers Unknown command: 'show', ... and socat still exits 0 — so an exit-status check passes and the help text gets parsed as data. Always use haproxy_cli(cmd, worker=True) in Python, which inspects the response body.

Stick-table entries are name=value / name(window_ms)=value pairs, not fixed columns; the first token is an allocation pointer (0x...:), not the key. Parse by NAME, and treat a missing field as an ERROR — never default it to 0. The web table stores only conn_cur, conn_rate, http_req_rate, http_err_rate; it holds no history and no counter of past blocks. What was actually denied/tarpitted is in the edge access log on the host at /var/log/haproxy.log (shipped 2026.08.8), not in any stick table.

This is written down because /api/security/stats and show-tarpit-ips.sh reported "Scan Count"/"BLOCKED" figures parsed from gpc0/gpc1 — fields no stick table has ever stored — for their entire existence. See the header of haproxy_tarpit_config.txt and the contract test.

Running the Application

  • Docker Build: docker build -t haproxy-manager .
  • Local Development: python haproxy_manager.py (requires HAProxy, certbot, and dependencies installed)
  • Container Run: See README.md for various docker run configurations

Monitoring and Debugging

  • Error Monitoring: ./scripts/monitor-errors.sh - Monitor application error logs
  • External Monitoring: ./scripts/monitor-errors-external.sh - External monitoring script
  • Health Check: curl http://localhost:8000/health
  • Log Files:
    • /var/log/haproxy-manager.log - General application logs
    • /var/log/haproxy-manager-errors.log - Error logs for alerting

Architecture Overview

Core Components

  1. haproxy_manager.py - Main Flask application providing:

    • RESTful API for HAProxy configuration management
    • SQLite database integration for domain/backend storage
    • Let's Encrypt certificate automation
    • HAProxy configuration generation from Jinja2 templates
    • Optional API key authentication via HAPROXY_API_KEY environment variable

  2. Database Schema - SQLite database with three main tables:

    • domains - Domain configurations with SSL settings
    • backends - Backend service definitions linked to domains
    • backend_servers - Individual servers within backend groups

  3. Template System - Jinja2 templates for HAProxy configuration generation:

    • hap_header.tpl - Global HAProxy settings, defaults, and HTTP/2 tuning
    • hap_backend.tpl - Backend server definitions
    • hap_listener.tpl - Frontend listener configurations with rate limiting
    • hap_letsencrypt.tpl - SSL certificate configurations
    • hap_security_tables.tpl - Stats frontend and security stick tables
    • Template override support for custom backend configurations

  4. Certificate Management - Automated SSL certificate handling:

    • Let's Encrypt integration with certbot
    • Self-signed certificate fallback for development
    • Certificate renewal automation via cron
    • Certificate download endpoints for external services

Configuration Flow

  1. Domain added via /api/domain endpoint → Database updated
  2. generate_config() function → Reads database, renders Jinja2 templates → Writes /etc/haproxy/haproxy.cfg
  3. HAProxy reload via socket API (/tmp/haproxy-cli) or process restart
  4. SSL certificate generation via Let's Encrypt or self-signed fallback

Key Design Patterns

  • Template-driven configuration: HAProxy config generated from modular Jinja2 templates
  • Database-backed state: All configuration persisted in SQLite for reliability
  • API-first design: All operations exposed via REST endpoints
  • Process monitoring: Health checks and automatic HAProxy restart capabilities
  • Comprehensive logging: Operation logging with error alerting support

Authentication & Security

  • Optional API key authentication controlled by HAPROXY_API_KEY environment variable
  • All API endpoints (except /health and /) require Bearer token when API key is set
  • Certificate private keys combined with certificates in HAProxy-compatible format
  • Default backend page for unmatched domains instead of exposing HAProxy errors

Rate Limiting & Connection Limits (hap_listener.tpl)

  • Stick table: type ip size 200k expire 10m tracking conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
  • Tracks real client IP via var(txn.real_ip) to work correctly behind Cloudflare/proxies
  • Rate limit thresholds:
    • Tarpit at 3000 req/10s (300 req/s)
    • Hard block (deny) at 5000 req/10s (500 req/s)
    • Connection rate limit: 500/10s
    • Concurrent connection limit: 500
    • Error rate limit: 100/30s
  • Whitelist bypasses (exempt from rate limits):
    • is_local — RFC1918 private address ranges
    • is_trusted_ip — source IPs listed in trusted_ips.list
    • is_whitelisted — real IPs (from proxy headers) matched in trusted_ips.map

Trusted IP Whitelist Files

  • trusted_ips.list — Source IP whitelist for rate limit bypass (one CIDR/IP per line)
  • trusted_ips.map — Real IP whitelist for proxy-header matching (format: <IP> 1)
  • Both files are baked into the Docker image via COPY in the Dockerfile
  • Ship as comment-only templates (no real IPs). Add trusted IPs locally and do not commit them — this repo is mirrored publicly. Entries persist in the /etc/haproxy named volume across recreates

Timeout Hardening (hap_header.tpl)

  • timeout http-request: 300s -> 30s (slowloris protection)
  • timeout connect: 120s -> 10s
  • timeout client: 10m -> 5m
  • timeout http-keep-alive: 120s -> 30s

HTTP/2 Protection (hap_header.tpl)

  • tune.h2.fe.max-total-streams 2000 — limits total streams per HTTP/2 connection
  • tune.h2.fe.glitches-threshold 50 — CVE-2023-44487 Rapid Reset protection

Stats Frontend (hap_security_tables.tpl)

  • HAProxy stats page bound to 127.0.0.1:8404 (localhost only, accessible inside container)
  • Template: templates/hap_security_tables.tpl

Deployment Context

  • Designed to run as Docker container with persistent volumes for certificates and configurations
  • Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
  • Stats page on port 8404 (localhost only inside container)
  • Management interface on port 8000 should be firewall-protected in production
  • Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with start-period=60s and timeout=10s
  • Supports deployment on servers with git directory at /root/whp and web file sync via rsync to /docker/whp/web/
  • HAProxy is version 3.0.11
Reference in New Issue View Git Blame Copy Permalink