haproxy-manager-base/README.md

HAProxy Manager

A Flask-based API service for managing HAProxy configurations, domains, and SSL certificates. A Flask-based API service for managing HAProxy configurations with dynamic SSL certificate management and health monitoring.

To run the container:

# Without API key authentication (default)
docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

# With API key authentication (recommended for production)
docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

Features

• RESTful API for HAProxy configuration management
• Database-backed configuration storage using SQLite
• Automatic HAProxy configuration generation from templates
• Let's Encrypt SSL certificate integration with auto-renewal
• Health monitoring endpoint
• Dynamic backend server management
• Template override support for custom backend configurations
• Process monitoring and auto-restart capabilities
• Socket-based HAProxy runtime API integration
• NEW: API key authentication for secure access
• NEW: Certificate renewal API endpoint
• NEW: Certificate download endpoints for other services
• NEW: Comprehensive error logging and alerting system
• NEW: Certificate status monitoring with expiration dates
• NEW: Default backend page for unmatched domains

Security

API Key Authentication

When the HAPROXY_API_KEY environment variable is set, all API endpoints (except /health and /) require authentication using a Bearer token:

# Example API call with authentication
curl -H "Authorization: Bearer your-secure-api-key-here" \
     http://localhost:8000/api/domains

If no API key is set, the service runs without authentication (useful for development).

Requirements

• HAProxy
• Python 3.x
• Flask
• SQLite3
• Certbot (for Let's Encrypt certificates)
• OpenSSL (for self-signed start-up certificate)

Web UI Interface

The HAProxy Manager includes a web-based user interface accessible at port 8000, providing:

• Domain and backend server management interface
• SSL certificate status monitoring

Do Not Expose port 8000 to the open internet If you need to have it exposed to the internet, restrict it to an IP Address via IPTABLES or other firewalls.

# Allow access from the specific IP address (replace 192.168.1.100 with your IP)
iptables -A INPUT -p tcp --dport 8000 -s {YOUR_PUBLIC_IP} -j ACCEPT

# Drop all other connections to port 8000
iptables -A INPUT -p tcp --dport 8000 -j DROP

If you need to be able to access the web interface from multiple locations, I recommend putting it behind an authenticated Proxy like Authentik

API Endpoints

Health Check

Check the status of the HAProxy Manager service.

GET /health

# Response
{
    "status": "healthy",
    "haproxy_status": "running",
    "database": "connected"
}

Get Domains

Retrieve all configured domains and their backend information.

GET /api/domains
Authorization: Bearer your-api-key

# Response
[
    {
        "id": 1,
        "domain": "example.com",
        "ssl_enabled": 1,
        "ssl_cert_path": "/etc/haproxy/certs/example.com.pem",
        "template_override": null,
        "backend_name": "example_backend"
    }
]

Add Domain

Add a new domain with backend servers configuration.

POST /api/domain
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domain": "example.com",
    "backend_name": "example_backend",
    "template_override": null,
    "servers": [
        {
            "name": "server1",
            "address": "10.0.0.1",
            "port": 8080,
            "options": "check"
        },
        {
            "name": "server2",
            "address": "10.0.0.2",
            "port": 8080,
            "options": "check backup"
        }
    ]
}

# Response
{
    "status": "success",
    "domain_id": 1
}

Enable SSL

Request and configure SSL certificate for a domain using Let's Encrypt.

POST /api/ssl
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domain": "example.com"
}

# Response
{
    "status": "success"
}

Remove Domain

Remove a domain and its associated backend configuration.

DELETE /api/domain
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domain": "example.com"
}

# Response
{
    "status": "success",
    "message": "Domain configuration removed"
}

Regenerate Configuration

Regenerate HAProxy configuration from database.

GET /api/regenerate
Authorization: Bearer your-api-key

# Response
{
    "status": "success"
}

Reload HAProxy

Reload HAProxy configuration without restart.

GET /api/reload
Authorization: Bearer your-api-key

# Response
{
    "status": "success"
}

New Certificate Management Endpoints

Request Certificate Generation

Request certificate generation for one or more domains.

POST /api/certificates/request
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domains": ["example.com", "api.example.com"],
    "force_renewal": false,
    "include_www": true
}

# Response
{
    "status": "completed",
    "summary": {
        "total": 2,
        "successful": 2,
        "failed": 0
    },
    "results": [
        {
            "domain": "example.com",
            "status": "success",
            "message": "Certificate obtained successfully",
            "cert_path": "/etc/haproxy/certs/example.com.pem",
            "domains_covered": ["example.com", "www.example.com"]
        },
        {
            "domain": "api.example.com",
            "status": "success",
            "message": "Certificate obtained successfully",
            "cert_path": "/etc/haproxy/certs/api.example.com.pem",
            "domains_covered": ["api.example.com"]
        }
    ]
}

Renew All Certificates

Trigger renewal of all Let's Encrypt certificates and reload HAProxy.

POST /api/certificates/renew
Authorization: Bearer your-api-key

# Response
{
    "status": "success",
    "message": "Certificates renewed and HAProxy reloaded"
}

Get Certificate Status

Get status of all certificates including expiration dates.

GET /api/certificates/status
Authorization: Bearer your-api-key

# Response
{
    "certificates": [
        {
            "domain": "example.com",
            "ssl_enabled": true,
            "cert_path": "/etc/haproxy/certs/example.com.pem",
            "expires": "2024-12-31T23:59:59",
            "days_until_expiry": 45
        }
    ]
}

Download Certificate Files

Download certificate files for use by other services.

# Download combined certificate (cert + key)
GET /api/certificates/example.com/download
Authorization: Bearer your-api-key

# Download private key only
GET /api/certificates/example.com/key
Authorization: Bearer your-api-key

# Download certificate only (no private key)
GET /api/certificates/example.com/cert
Authorization: Bearer your-api-key

Certificate Renewal

The HAProxy Manager includes automatic certificate renewal with multiple scheduling options:

Automatic Renewal (Container-based)

By default, a cron job runs inside the container every 12 hours to check and renew certificates:

• Runs at minute 0 of every 12th hour (12:00 AM, 12:00 PM)
• Automatically reloads HAProxy if certificates are renewed
• Logs all renewal attempts to /var/log/haproxy-manager.log
• Errors logged to /var/log/haproxy-manager-errors.log

Manual Renewal via API

Trigger certificate renewal manually using the API:

curl -X POST http://localhost:8000/api/certificates/renew \
  -H "Authorization: Bearer your-api-key"

Host-side Renewal (Recommended for Production)

For more control over scheduling, run renewals from the host machine using the provided script:

# Make the script executable
chmod +x scripts/host-renew-certificates.sh

# Add to host crontab (edit with: crontab -e)
0 */12 * * * /path/to/haproxy-manager-base/scripts/host-renew-certificates.sh

# Or run manually
./scripts/host-renew-certificates.sh

The host-side script:

• Executes the renewal process inside the running container
• Maintains separate host-side logs at /var/log/haproxy-manager-host-renewal.log
• Automatically detects if the container is running
• Supports custom container names via CONTAINER_NAME environment variable

See scripts/host-crontab-example.txt for more crontab configuration examples.

Renewal Script Features

The renewal script (scripts/renew-certificates.sh) includes:

• Comprehensive logging with timestamps
• Retry logic for HAProxy reload (3 attempts with 5-second delays)
• HAProxy socket health checks before reload
• Proper error handling and exit codes
• Detection of whether certificates actually needed renewal

Logging and Monitoring

The HAProxy Manager includes comprehensive logging and error tracking:

Log Files

• /var/log/haproxy-manager.log - General application logs
• /var/log/haproxy-manager-errors.log - Error logs for alerting
• /var/log/haproxy-manager-host-renewal.log - Host-side renewal logs (when using host script)

Logged Operations

All API operations are logged with timestamps and success/failure status:

• Domain management (add/remove)
• SSL certificate operations
• Configuration generation
• HAProxy reload/restart operations
• Certificate renewals

Error Alerting

Failed operations are logged to the error log file. You can monitor this file for alerting:

# Monitor error log for alerting
tail -f /var/log/haproxy-manager-errors.log

Environment Variables

Variable	Description	Default
HAPROXY_API_KEY	API key for authentication (optional)	None (no auth)
HAPROXY_DEFAULT_PAGE_TITLE	Title for the default page	Site Not Configured
HAPROXY_DEFAULT_MAIN_MESSAGE	Main message on the default page	This domain has not been configured yet. Please contact your system administrator to set up this website.
HAPROXY_DEFAULT_SECONDARY_MESSAGE	Secondary message on the default page	If you believe this is an error, please check the domain name and try again.
HAPROXY_DEFAULT_SPLASH	off (or 0/false/no) disables the default splash for unrouted hosts and restores the default-backend page below	on

Default Splash (unrouted hosts)

Since 2026.11.1 (404-everywhere + robots since 2026.11.2), any request whose Host is not routed to a site -- the server's own hostname (registered cert-only, no backend), a bare IP, a stale or unknown name, a request with no Host at all -- is answered by HAProxy itself with the static page errors/default-splash.html (http-request return ... file): 404 on EVERY path (including /) so search engines never index the server's IP or hostname as a site, with cache-control: no-store, x-robots-tag: noindex, nofollow, noarchive and x-content-type-options: nosniff; /robots.txt is 200 text/plain User-agent: * / Disallow: / (errors/robots-unrouted.txt). Monitors pointed at the bare server hostname on 80/443 must accept 404 (WHP's own probes -- healthcheck.sh, the watchdog -- accept any status). The page names no version, echoes nothing from the request (it is loaded with file, not lf-file), and links nowhere -- in particular not to the control panel.

Why. use_backend default-backend does not short-circuit HAProxy's http-request rules, so the old catch-all was inspected by Coraza with the full server-wide rule set like customer traffic. A browser on https://<server-hostname>/ also sends the control panel's cookies (cookies are not port-scoped, so :8443 cookies go to :443), and anything Coraza matched there showed the WAF block page for a host with no site behind it. The splash rules sit before the SPOE call, so unrouted requests never reach the WAF; there is nothing behind them to protect.

What still applies / is kept out: rate limits, the IP block (blocked IPs still get the blocked-ip page), the wp-* gates, and the multipart guard run first; /.well-known/acme-challenge/ still reaches certbot (the server hostname's own certificate is issued through it); hosts in suspended_domains.list still get the suspended page. Configured sites, wildcards and their WAF inspection are unchanged. Plain HTTP on :80 gets the splash too, not a redirect: an unknown name usually has no valid certificate, so a redirect would trade a clean page for a TLS warning. Over TLS the presented certificate is whatever SNI selects -- the hostname's own cert if one is issued, otherwise the default bundle.

How "routed" is decided. generate_config() writes /etc/haproxy/routed_hosts.list (exact names) and /etc/haproxy/routed_wildcards.list (.base suffixes) from the same domain rows, in the same call, as the routing ACLs, and both are in the rollback set next to haproxy.cfg. The splash ACLs compare the Host port-stripped and lowercased, a superset of the routing ACLs (which compare the raw Host), so a host that routes is never splashed. Never hand-edit the lists; they are rewritten on every regeneration (also when the splash is off, so turning it on never meets a missing -f file).

Deploying to an existing host. The page ships at /haproxy/errors/default-splash.html and the template under /haproxy/templates/ -- both in the image, OUTSIDE the /etc/haproxy named volume, so the volume cannot shadow them. The lists live in the volume but are generated, not copied. So an image bump plus a recreate of the container is the whole deployment: on start generate_config() renders the new listener and writes both lists, then HAProxy loads them. Nothing to copy into the volume, no WHP-side change. Verify after the recreate:

docker exec haproxy-manager grep -c default-splash.html /etc/haproxy/haproxy.cfg   # 2
docker exec haproxy-manager cat /etc/haproxy/routed_hosts.list                     # every site
curl -skI https://<server-hostname>/                       # 404 + x-robots-tag noindex
curl -sk https://<server-hostname>/robots.txt              # User-agent: * / Disallow: /
curl -sk -o /dev/null -w '%{http_code}\n' https://<a-customer-site>/                 # unchanged

Rollback: recreate on the previous image tag, or set HAPROXY_DEFAULT_SPLASH=off and regenerate.

Covered by scripts/test-default-splash.py (template order/conditions, the page, list generation) and the splash-off scenario of scripts/validate-rendered-config.py.

Access Log (in-container, since 2026.11.2)

HAProxy logs (log /log.sock ... format rfc5424, hap_header.tpl) to a UNIX datagram socket inside its chroot (/var/lib/haproxy/log.sock). scripts/haproxy-log-receiver.py, started and supervised by start-up.sh (a restart loop: back within ~2 s), writes one line per request to /var/log/haproxy/access.log, rotated at 40 MB with 4 gzip generations (worst case 200 MB; HAPROXY_ACCESS_LOG_MAX_BYTES, HAPROXY_ACCESS_LOG_BACKUPS). No network listener anywhere. Fields: client/real IP (cip=), Host, status, termination state, request id, UA, SNI, HTTP version, WAF body stamp and the Coraza wrule=/wrules=/wscore= fields (unchanged log-format).

• Never blocks HAProxy. Sends are non-blocking: a dead or stalled receiver, or a full disk, only drops log lines (counted as DroppedLogs in echo "@1 show info" | socat stdio /tmp/haproxy-cli). Lab: with the receiver SIGSTOPped, 2500/2500 requests still answered 200 in ~1 s; 1025 lines queued, 1475 dropped and counted.
• Not in docker logs. The receiver prints no log lines (WHP's AI log monitor reads docker logs haproxy-manager; a per-request firehose there would storm it). Only its start line and rate-limited errors go to stderr.
• Persistence. Mount a named volume at /var/log/haproxy (WHP: -v haproxy-logs:/var/log/haproxy) or the log resets on every recreate. Give the container --sysctl net.unix.max_dgram_qlen=1024 (a netns starts at 10) for burst headroom.
• HAPROXY_SYSLOG_TARGET=<ip>:<port> still overrides the target for a remote collector. The old default 172.18.0.1:514 (host rsyslog) is gone.

Reading it:

docker exec haproxy-manager tail -f /var/log/haproxy/access.log
docker exec haproxy-manager grep ' 403 ' /var/log/haproxy/access.log | tail
docker exec haproxy-manager sh -c 'zcat /var/log/haproxy/access.log.*.gz' | grep <request-id>
docker cp haproxy-manager:/var/log/haproxy/ ./edge-logs/    # or the volume's _data dir on the host

Rollout: image bump + recreate with the volume and sysctl (WHP's lib-edge-container.sh adds both). The receiver, its script and the template are in the image, outside the /etc/haproxy named volume, so nothing can be shadowed; haproxy.cfg is regenerated on start with the new log line.

Default Backend Configuration

With the default splash on (above), unrouted hosts no longer reach this page; it still serves the blocked-IP and suspended pages, and serves unrouted hosts again when HAPROXY_DEFAULT_SPLASH=off.

When a domain is accessed that hasn't been configured in HAProxy, the system will serve a default page instead of showing an error. This default page:

• Informs visitors that the site is not configured
• Displays the domain name and current timestamp
• Is fully customizable through environment variables

Customizing the Default Page

You can customize the default page by setting environment variables:

docker run -d \
  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
  -v lets-encrypt:/etc/letsencrypt \
  -v haproxy:/etc/haproxy \
  -e HAPROXY_API_KEY=your-secure-api-key-here \
  -e HAPROXY_DEFAULT_PAGE_TITLE="Website Coming Soon" \
  -e HAPROXY_DEFAULT_MAIN_MESSAGE="This website is currently under construction and will be available soon." \
  -e HAPROXY_DEFAULT_SECONDARY_MESSAGE="Please check back later or contact us for more information." \
  --name haproxy-manager \
  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

Example Usage

Setting up with API key authentication:

# Start container with API key
docker run -d \
  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
  -v lets-encrypt:/etc/letsencrypt \
  -v haproxy:/etc/haproxy \
  -e HAPROXY_API_KEY=your-secure-api-key-here \
  --name haproxy-manager \
  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

# Add a domain
curl -X POST http://localhost:8000/api/domain \
  -H "Authorization: Bearer your-secure-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "backend_name": "example_backend",
    "servers": [
      {"name": "server1", "address": "10.0.0.1", "port": 8080, "options": "check"}
    ]
  }'

# Request SSL certificate
curl -X POST http://localhost:8000/api/ssl \
  -H "Authorization: Bearer your-secure-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"domain": "example.com"}'

# Renew certificates
curl -X POST http://localhost:8000/api/certificates/renew \
  -H "Authorization: Bearer your-secure-api-key-here"

# Request certificate generation for another service
curl -X POST http://localhost:8000/api/certificates/request \
  -H "Authorization: Bearer your-secure-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "domains": ["api.example.com"],
    "force_renewal": false,
    "include_www": false
  }'

# Download certificate for another service
curl -H "Authorization: Bearer your-secure-api-key-here" \
  http://localhost:8000/api/certificates/example.com/download \
  -o example.com.pem