add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent `add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True whenever socat exited 0. Neither command has ever worked, on any deployment, for the entire life of the feature -- while logging "Added IP x to runtime map" every single time. Two independent defects: * NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map commands are worker commands. Captured verbatim on whp01: $ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli Unknown command: 'add', but maybe one of the following ones is a better match: @!<pid> : send a command to the <pid> process ... $ echo $? 0 socat exits 0 on the rejection, so `result.returncode == 0` was true. Same silence PR #7 fixed on the `show table` path. * `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move on every config regeneration -- `@1 show map` on whp01 reports blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0. Hardcoding any number is wrong; the map is referenced by FILE PATH, which is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0). And a third silence, which is why a response-body check alone is not enough here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer `Unknown map identifier.`. On the add path the reply is byte-for-byte identical to success. Only reading the entry back can tell them apart. IP blocking itself was never broken: update_blocked_ips_map() rewrites /etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it. That path is untouched and stays authoritative. What was broken is the no-reload fast path, plus every report that it had worked. * haproxy_manager.py: both functions send `@1 add|del map /etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with `get map` before returning True. runtime_map_lookup()/runtime_map_keys() are the read-back primitives. `sync_blocked_ips` loses `clear map #0` (which the master socket rejected just as loudly and just as invisibly) and verifies the whole set with one `show map` instead of counting commands that did not visibly complain; it answers 207 + `runtime_map_synced: false` when the runtime map does not match the database. * haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy answers those with nothing on success, so an empty body is the success and ANY non-empty body is a rejection. That is stricter than the marker list on purpose -- markers only recognise rejections someone has already seen, and it catches `'add map' expects three parameters ...`, which matches nothing. HaproxyCliError carries `.responses` so `del map` answering `Key not found.` (the requested end state) is told apart from a real failure without regex. * The four callers capture the boolean instead of discarding it and report `runtime_map_updated` / `runtime_map_failures` in the API response and the operation log. A runtime failure degrades to "enforced on the reload that already happens two lines later" -- never to an unblocked IP, never to a 500. * scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes on the wire (`@1` first, map by path, value `1`), classifies every captured response, and scans the repo's Python string literals and shell/template code lines for `#<id>` map references -- comments may describe the old form, code may not use it. Verified to fail on each defect reintroduced separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the- reply (1). * The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md and the comment in templates/hap_listener.tpl -- where every copy of it additionally omitted the `1`, which `-m int gt 0` needs to match. The only template change is a comment; `haproxy -c` on the live rendered config with it applied is clean (HAProxy 3.0.11, warnings unchanged). Verified on whp01 against the running container (docker cp + SIGHUP, no recreate). Before: both functions returned True and logged success while `@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed add lands with value "1" and the remove takes it out again; the old command form is now classified as a failure; a `#0` map reference returns False via the read-back. End to end through the API, `runtime_map_updated: true`, and /api/blocked-ips/sync -- which used to be a no-op reporting a full sync -- reports 264/264 verified present. The runtime path was isolated from the reload that normally follows it: with NO map-file write and NO reload (same haproxy worker pid throughout), adding 100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not in the is_local ranges) to the runtime map alone flipped a live site from HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path working for the first time. All test IPs were removed afterwards: 0 rows in blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer sites, the panel /health and `haproxy -c` are byte-identical to the baseline taken before the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HAProxy Manager
A Flask-based API service for managing HAProxy configurations, domains, and SSL certificates. A Flask-based API service for managing HAProxy configurations with dynamic SSL certificate management and health monitoring.
To run the container:
# Without API key authentication (default)
docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
# With API key authentication (recommended for production)
docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
Features
- RESTful API for HAProxy configuration management
- Database-backed configuration storage using SQLite
- Automatic HAProxy configuration generation from templates
- Let's Encrypt SSL certificate integration with auto-renewal
- Health monitoring endpoint
- Dynamic backend server management
- Template override support for custom backend configurations
- Process monitoring and auto-restart capabilities
- Socket-based HAProxy runtime API integration
- NEW: API key authentication for secure access
- NEW: Certificate renewal API endpoint
- NEW: Certificate download endpoints for other services
- NEW: Comprehensive error logging and alerting system
- NEW: Certificate status monitoring with expiration dates
- NEW: Default backend page for unmatched domains
Security
API Key Authentication
When the HAPROXY_API_KEY environment variable is set, all API endpoints (except /health and /) require authentication using a Bearer token:
# Example API call with authentication
curl -H "Authorization: Bearer your-secure-api-key-here" \
http://localhost:8000/api/domains
If no API key is set, the service runs without authentication (useful for development).
Requirements
- HAProxy
- Python 3.x
- Flask
- SQLite3
- Certbot (for Let's Encrypt certificates)
- OpenSSL (for self-signed start-up certificate)
Web UI Interface
The HAProxy Manager includes a web-based user interface accessible at port 8000, providing:
- Domain and backend server management interface
- SSL certificate status monitoring
Do Not Expose port 8000 to the open internet If you need to have it exposed to the internet, restrict it to an IP Address via IPTABLES or other firewalls.
# Allow access from the specific IP address (replace 192.168.1.100 with your IP)
iptables -A INPUT -p tcp --dport 8000 -s {YOUR_PUBLIC_IP} -j ACCEPT
# Drop all other connections to port 8000
iptables -A INPUT -p tcp --dport 8000 -j DROP
If you need to be able to access the web interface from multiple locations, I recommend putting it behind an authenticated Proxy like Authentik
API Endpoints
Health Check
Check the status of the HAProxy Manager service.
GET /health
# Response
{
"status": "healthy",
"haproxy_status": "running",
"database": "connected"
}
Get Domains
Retrieve all configured domains and their backend information.
GET /api/domains
Authorization: Bearer your-api-key
# Response
[
{
"id": 1,
"domain": "example.com",
"ssl_enabled": 1,
"ssl_cert_path": "/etc/haproxy/certs/example.com.pem",
"template_override": null,
"backend_name": "example_backend"
}
]
Add Domain
Add a new domain with backend servers configuration.
POST /api/domain
Authorization: Bearer your-api-key
Content-Type: application/json
{
"domain": "example.com",
"backend_name": "example_backend",
"template_override": null,
"servers": [
{
"name": "server1",
"address": "10.0.0.1",
"port": 8080,
"options": "check"
},
{
"name": "server2",
"address": "10.0.0.2",
"port": 8080,
"options": "check backup"
}
]
}
# Response
{
"status": "success",
"domain_id": 1
}
Enable SSL
Request and configure SSL certificate for a domain using Let's Encrypt.
POST /api/ssl
Authorization: Bearer your-api-key
Content-Type: application/json
{
"domain": "example.com"
}
# Response
{
"status": "success"
}
Remove Domain
Remove a domain and its associated backend configuration.
DELETE /api/domain
Authorization: Bearer your-api-key
Content-Type: application/json
{
"domain": "example.com"
}
# Response
{
"status": "success",
"message": "Domain configuration removed"
}
Regenerate Configuration
Regenerate HAProxy configuration from database.
GET /api/regenerate
Authorization: Bearer your-api-key
# Response
{
"status": "success"
}
Reload HAProxy
Reload HAProxy configuration without restart.
GET /api/reload
Authorization: Bearer your-api-key
# Response
{
"status": "success"
}
New Certificate Management Endpoints
Request Certificate Generation
Request certificate generation for one or more domains.
POST /api/certificates/request
Authorization: Bearer your-api-key
Content-Type: application/json
{
"domains": ["example.com", "api.example.com"],
"force_renewal": false,
"include_www": true
}
# Response
{
"status": "completed",
"summary": {
"total": 2,
"successful": 2,
"failed": 0
},
"results": [
{
"domain": "example.com",
"status": "success",
"message": "Certificate obtained successfully",
"cert_path": "/etc/haproxy/certs/example.com.pem",
"domains_covered": ["example.com", "www.example.com"]
},
{
"domain": "api.example.com",
"status": "success",
"message": "Certificate obtained successfully",
"cert_path": "/etc/haproxy/certs/api.example.com.pem",
"domains_covered": ["api.example.com"]
}
]
}
Renew All Certificates
Trigger renewal of all Let's Encrypt certificates and reload HAProxy.
POST /api/certificates/renew
Authorization: Bearer your-api-key
# Response
{
"status": "success",
"message": "Certificates renewed and HAProxy reloaded"
}
Get Certificate Status
Get status of all certificates including expiration dates.
GET /api/certificates/status
Authorization: Bearer your-api-key
# Response
{
"certificates": [
{
"domain": "example.com",
"ssl_enabled": true,
"cert_path": "/etc/haproxy/certs/example.com.pem",
"expires": "2024-12-31T23:59:59",
"days_until_expiry": 45
}
]
}
Download Certificate Files
Download certificate files for use by other services.
# Download combined certificate (cert + key)
GET /api/certificates/example.com/download
Authorization: Bearer your-api-key
# Download private key only
GET /api/certificates/example.com/key
Authorization: Bearer your-api-key
# Download certificate only (no private key)
GET /api/certificates/example.com/cert
Authorization: Bearer your-api-key
Certificate Renewal
The HAProxy Manager includes automatic certificate renewal with multiple scheduling options:
Automatic Renewal (Container-based)
By default, a cron job runs inside the container every 12 hours to check and renew certificates:
- Runs at minute 0 of every 12th hour (12:00 AM, 12:00 PM)
- Automatically reloads HAProxy if certificates are renewed
- Logs all renewal attempts to
/var/log/haproxy-manager.log - Errors logged to
/var/log/haproxy-manager-errors.log
Manual Renewal via API
Trigger certificate renewal manually using the API:
curl -X POST http://localhost:8000/api/certificates/renew \
-H "Authorization: Bearer your-api-key"
Host-side Renewal (Recommended for Production)
For more control over scheduling, run renewals from the host machine using the provided script:
# Make the script executable
chmod +x scripts/host-renew-certificates.sh
# Add to host crontab (edit with: crontab -e)
0 */12 * * * /path/to/haproxy-manager-base/scripts/host-renew-certificates.sh
# Or run manually
./scripts/host-renew-certificates.sh
The host-side script:
- Executes the renewal process inside the running container
- Maintains separate host-side logs at
/var/log/haproxy-manager-host-renewal.log - Automatically detects if the container is running
- Supports custom container names via
CONTAINER_NAMEenvironment variable
See scripts/host-crontab-example.txt for more crontab configuration examples.
Renewal Script Features
The renewal script (scripts/renew-certificates.sh) includes:
- Comprehensive logging with timestamps
- Retry logic for HAProxy reload (3 attempts with 5-second delays)
- HAProxy socket health checks before reload
- Proper error handling and exit codes
- Detection of whether certificates actually needed renewal
Logging and Monitoring
The HAProxy Manager includes comprehensive logging and error tracking:
Log Files
/var/log/haproxy-manager.log- General application logs/var/log/haproxy-manager-errors.log- Error logs for alerting/var/log/haproxy-manager-host-renewal.log- Host-side renewal logs (when using host script)
Logged Operations
All API operations are logged with timestamps and success/failure status:
- Domain management (add/remove)
- SSL certificate operations
- Configuration generation
- HAProxy reload/restart operations
- Certificate renewals
Error Alerting
Failed operations are logged to the error log file. You can monitor this file for alerting:
# Monitor error log for alerting
tail -f /var/log/haproxy-manager-errors.log
Environment Variables
| Variable | Description | Default |
|---|---|---|
HAPROXY_API_KEY |
API key for authentication (optional) | None (no auth) |
HAPROXY_DEFAULT_PAGE_TITLE |
Title for the default page | Site Not Configured |
HAPROXY_DEFAULT_MAIN_MESSAGE |
Main message on the default page | This domain has not been configured yet. Please contact your system administrator to set up this website. |
HAPROXY_DEFAULT_SECONDARY_MESSAGE |
Secondary message on the default page | If you believe this is an error, please check the domain name and try again. |
Default Backend Configuration
When a domain is accessed that hasn't been configured in HAProxy, the system will serve a default page instead of showing an error. This default page:
- Informs visitors that the site is not configured
- Displays the domain name and current timestamp
- Is fully customizable through environment variables
Customizing the Default Page
You can customize the default page by setting environment variables:
docker run -d \
-p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
-v lets-encrypt:/etc/letsencrypt \
-v haproxy:/etc/haproxy \
-e HAPROXY_API_KEY=your-secure-api-key-here \
-e HAPROXY_DEFAULT_PAGE_TITLE="Website Coming Soon" \
-e HAPROXY_DEFAULT_MAIN_MESSAGE="This website is currently under construction and will be available soon." \
-e HAPROXY_DEFAULT_SECONDARY_MESSAGE="Please check back later or contact us for more information." \
--name haproxy-manager \
your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
Example Usage
Setting up with API key authentication:
# Start container with API key
docker run -d \
-p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
-v lets-encrypt:/etc/letsencrypt \
-v haproxy:/etc/haproxy \
-e HAPROXY_API_KEY=your-secure-api-key-here \
--name haproxy-manager \
your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
# Add a domain
curl -X POST http://localhost:8000/api/domain \
-H "Authorization: Bearer your-secure-api-key-here" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"backend_name": "example_backend",
"servers": [
{"name": "server1", "address": "10.0.0.1", "port": 8080, "options": "check"}
]
}'
# Request SSL certificate
curl -X POST http://localhost:8000/api/ssl \
-H "Authorization: Bearer your-secure-api-key-here" \
-H "Content-Type: application/json" \
-d '{"domain": "example.com"}'
# Renew certificates
curl -X POST http://localhost:8000/api/certificates/renew \
-H "Authorization: Bearer your-secure-api-key-here"
# Request certificate generation for another service
curl -X POST http://localhost:8000/api/certificates/request \
-H "Authorization: Bearer your-secure-api-key-here" \
-H "Content-Type: application/json" \
-d '{
"domains": ["api.example.com"],
"force_renewal": false,
"include_www": false
}'
# Download certificate for another service
curl -H "Authorization: Bearer your-secure-api-key-here" \
http://localhost:8000/api/certificates/example.com/download \
-o example.com.pem