haproxy-manager-base/README.md

# HAProxy Manager

A Flask-based API service for managing HAProxy configurations, domains, and SSL certificates.
A Flask-based API service for managing HAProxy configurations with dynamic SSL certificate management and health monitoring.

To run the container:
```bash
# Without API key authentication (default)
docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

# With API key authentication (recommended for production)
docker run -d -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 -v lets-encrypt:/etc/letsencrypt -v haproxy:/etc/haproxy -e HAPROXY_API_KEY=your-secure-api-key-here --name haproxy-manager your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
```

## Features

- RESTful API for HAProxy configuration management
- Database-backed configuration storage using SQLite
- Automatic HAProxy configuration generation from templates
- Let's Encrypt SSL certificate integration with auto-renewal
- Health monitoring endpoint
- Dynamic backend server management
- Template override support for custom backend configurations
- Process monitoring and auto-restart capabilities
- Socket-based HAProxy runtime API integration
- **NEW**: API key authentication for secure access
- **NEW**: Certificate renewal API endpoint
- **NEW**: Certificate download endpoints for other services
- **NEW**: Comprehensive error logging and alerting system
- **NEW**: Certificate status monitoring with expiration dates
- **NEW**: Default backend page for unmatched domains

## Security

### API Key Authentication

When the `HAPROXY_API_KEY` environment variable is set, all API endpoints (except `/health` and `/`) require authentication using a Bearer token:

```bash
# Example API call with authentication
curl -H "Authorization: Bearer your-secure-api-key-here" \
     http://localhost:8000/api/domains
```

If no API key is set, the service runs without authentication (useful for development).

## Requirements

- HAProxy
- Python 3.x
- Flask
- SQLite3
- Certbot (for Let's Encrypt certificates)
- OpenSSL (for self-signed start-up certificate)

## Web UI Interface

The HAProxy Manager includes a web-based user interface accessible at port 8000, providing:
- Domain and backend server management interface
- SSL certificate status monitoring

__Do Not Expose port 8000 to the open internet__
If you need to have it exposed to the internet, restrict it to an IP Address via IPTABLES or other firewalls.
```bash
# Allow access from the specific IP address (replace 192.168.1.100 with your IP)
iptables -A INPUT -p tcp --dport 8000 -s {YOUR_PUBLIC_IP} -j ACCEPT

# Drop all other connections to port 8000
iptables -A INPUT -p tcp --dport 8000 -j DROP
```
If you need to be able to access the web interface from multiple locations, I recommend putting it behind an authenticated Proxy like Authentik

## API Endpoints

### Health Check
Check the status of the HAProxy Manager service.

```bash
GET /health

# Response
{
    "status": "healthy",
    "haproxy_status": "running",
    "database": "connected"
}
```

### Get Domains
Retrieve all configured domains and their backend information.

```bash
GET /api/domains
Authorization: Bearer your-api-key

# Response
[
    {
        "id": 1,
        "domain": "example.com",
        "ssl_enabled": 1,
        "ssl_cert_path": "/etc/haproxy/certs/example.com.pem",
        "template_override": null,
        "backend_name": "example_backend"
    }
]
```

### Add Domain
Add a new domain with backend servers configuration.

```bash
POST /api/domain
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domain": "example.com",
    "backend_name": "example_backend",
    "template_override": null,
    "servers": [
        {
            "name": "server1",
            "address": "10.0.0.1",
            "port": 8080,
            "options": "check"
        },
        {
            "name": "server2",
            "address": "10.0.0.2",
            "port": 8080,
            "options": "check backup"
        }
    ]
}

# Response
{
    "status": "success",
    "domain_id": 1
}
```

### Enable SSL
Request and configure SSL certificate for a domain using Let's Encrypt.

```bash
POST /api/ssl
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domain": "example.com"
}

# Response
{
    "status": "success"
}
```

### Remove Domain
Remove a domain and its associated backend configuration.

```bash
DELETE /api/domain
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domain": "example.com"
}

# Response
{
    "status": "success",
    "message": "Domain configuration removed"
}
```

### Regenerate Configuration
Regenerate HAProxy configuration from database.

```bash
GET /api/regenerate
Authorization: Bearer your-api-key

# Response
{
    "status": "success"
}
```

### Reload HAProxy
Reload HAProxy configuration without restart.

```bash
GET /api/reload
Authorization: Bearer your-api-key

# Response
{
    "status": "success"
}
```

## New Certificate Management Endpoints

### Request Certificate Generation
Request certificate generation for one or more domains.

```bash
POST /api/certificates/request
Authorization: Bearer your-api-key
Content-Type: application/json

{
    "domains": ["example.com", "api.example.com"],
    "force_renewal": false,
    "include_www": true
}

# Response
{
    "status": "completed",
    "summary": {
        "total": 2,
        "successful": 2,
        "failed": 0
    },
    "results": [
        {
            "domain": "example.com",
            "status": "success",
            "message": "Certificate obtained successfully",
            "cert_path": "/etc/haproxy/certs/example.com.pem",
            "domains_covered": ["example.com", "www.example.com"]
        },
        {
            "domain": "api.example.com",
            "status": "success",
            "message": "Certificate obtained successfully",
            "cert_path": "/etc/haproxy/certs/api.example.com.pem",
            "domains_covered": ["api.example.com"]
        }
    ]
}
```

### Renew All Certificates
Trigger renewal of all Let's Encrypt certificates and reload HAProxy.

```bash
POST /api/certificates/renew
Authorization: Bearer your-api-key

# Response
{
    "status": "success",
    "message": "Certificates renewed and HAProxy reloaded"
}
```

### Get Certificate Status
Get status of all certificates including expiration dates.

```bash
GET /api/certificates/status
Authorization: Bearer your-api-key

# Response
{
    "certificates": [
        {
            "domain": "example.com",
            "ssl_enabled": true,
            "cert_path": "/etc/haproxy/certs/example.com.pem",
            "expires": "2024-12-31T23:59:59",
            "days_until_expiry": 45
        }
    ]
}
```

### Download Certificate Files
Download certificate files for use by other services.

```bash
# Download combined certificate (cert + key)
GET /api/certificates/example.com/download
Authorization: Bearer your-api-key

# Download private key only
GET /api/certificates/example.com/key
Authorization: Bearer your-api-key

# Download certificate only (no private key)
GET /api/certificates/example.com/cert
Authorization: Bearer your-api-key
```

## Certificate Renewal

The HAProxy Manager includes automatic certificate renewal with multiple scheduling options:

### Automatic Renewal (Container-based)
By default, a cron job runs inside the container every 12 hours to check and renew certificates:
- Runs at minute 0 of every 12th hour (12:00 AM, 12:00 PM)
- Automatically reloads HAProxy if certificates are renewed
- Logs all renewal attempts to `/var/log/haproxy-manager.log`
- Errors logged to `/var/log/haproxy-manager-errors.log`

### Manual Renewal via API
Trigger certificate renewal manually using the API:
```bash
curl -X POST http://localhost:8000/api/certificates/renew \
  -H "Authorization: Bearer your-api-key"
```

### Host-side Renewal (Recommended for Production)
For more control over scheduling, run renewals from the host machine using the provided script:

```bash
# Make the script executable
chmod +x scripts/host-renew-certificates.sh

# Add to host crontab (edit with: crontab -e)
0 */12 * * * /path/to/haproxy-manager-base/scripts/host-renew-certificates.sh

# Or run manually
./scripts/host-renew-certificates.sh
```

The host-side script:
- Executes the renewal process inside the running container
- Maintains separate host-side logs at `/var/log/haproxy-manager-host-renewal.log`
- Automatically detects if the container is running
- Supports custom container names via `CONTAINER_NAME` environment variable

See [scripts/host-crontab-example.txt](scripts/host-crontab-example.txt) for more crontab configuration examples.

### Renewal Script Features
The renewal script ([scripts/renew-certificates.sh](scripts/renew-certificates.sh)) includes:
- Comprehensive logging with timestamps
- Retry logic for HAProxy reload (3 attempts with 5-second delays)
- HAProxy socket health checks before reload
- Proper error handling and exit codes
- Detection of whether certificates actually needed renewal

## Logging and Monitoring

The HAProxy Manager includes comprehensive logging and error tracking:

### Log Files
- `/var/log/haproxy-manager.log` - General application logs
- `/var/log/haproxy-manager-errors.log` - Error logs for alerting
- `/var/log/haproxy-manager-host-renewal.log` - Host-side renewal logs (when using host script)

### Logged Operations
All API operations are logged with timestamps and success/failure status:
- Domain management (add/remove)
- SSL certificate operations
- Configuration generation
- HAProxy reload/restart operations
- Certificate renewals

### Error Alerting
Failed operations are logged to the error log file. You can monitor this file for alerting:
```bash
# Monitor error log for alerting
tail -f /var/log/haproxy-manager-errors.log
```

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `HAPROXY_API_KEY` | API key for authentication (optional) | None (no auth) |
| `HAPROXY_DEFAULT_PAGE_TITLE` | Title for the default page | Site Not Configured |
| `HAPROXY_DEFAULT_MAIN_MESSAGE` | Main message on the default page | This domain has not been configured yet. Please contact your system administrator to set up this website. |
| `HAPROXY_DEFAULT_SECONDARY_MESSAGE` | Secondary message on the default page | If you believe this is an error, please check the domain name and try again. |
| `HAPROXY_DEFAULT_SPLASH` | `off` (or `0`/`false`/`no`) disables the default splash for unrouted hosts and restores the default-backend page below | on |

## Default Splash (unrouted hosts)

Since 2026.11.1 (404-everywhere + robots since 2026.11.2), any request whose Host is **not routed to a site** -- the
server's own hostname (registered cert-only, no backend), a bare IP, a stale
or unknown name, a request with no Host at all -- is answered by HAProxy
itself with the static page `errors/default-splash.html` (`http-request
return ... file`): `404` on EVERY path (including `/`) so search engines never
index the server's IP or hostname as a site, with `cache-control: no-store`,
`x-robots-tag: noindex, nofollow, noarchive` and `x-content-type-options:
nosniff`; `/robots.txt` is `200 text/plain` `User-agent: *` / `Disallow: /`
(`errors/robots-unrouted.txt`). Monitors pointed at the bare server hostname
on 80/443 must accept 404 (WHP's own probes -- healthcheck.sh, the watchdog --
accept any status). The page
names no version, echoes nothing from the request (it is loaded with `file`,
not `lf-file`), and links nowhere -- in particular not to the control panel.

**Why.** `use_backend default-backend` does not short-circuit HAProxy's
`http-request` rules, so the old catch-all was inspected by Coraza with the
full server-wide rule set like customer traffic. A browser on
`https://<server-hostname>/` also sends the control panel's cookies (cookies
are not port-scoped, so `:8443` cookies go to `:443`), and anything Coraza
matched there showed the WAF block page for a host with no site behind it.
The splash rules sit before the SPOE call, so unrouted requests never reach
the WAF; there is nothing behind them to protect.

**What still applies / is kept out:** rate limits, the IP block (blocked IPs
still get the blocked-ip page), the wp-* gates, and the multipart guard run
first; `/.well-known/acme-challenge/` still reaches certbot (the server
hostname's own certificate is issued through it); hosts in
`suspended_domains.list` still get the suspended page. Configured sites,
wildcards and their WAF inspection are unchanged. Plain HTTP on `:80` gets
the splash too, not a redirect: an unknown name usually has no valid
certificate, so a redirect would trade a clean page for a TLS warning.
Over TLS the presented certificate is whatever SNI selects -- the
hostname's own cert if one is issued, otherwise the default bundle.

**How "routed" is decided.** `generate_config()` writes
`/etc/haproxy/routed_hosts.list` (exact names) and
`/etc/haproxy/routed_wildcards.list` (`.base` suffixes) from the same domain
rows, in the same call, as the routing ACLs, and both are in the rollback set
next to `haproxy.cfg`. The splash ACLs compare the Host port-stripped and
lowercased, a superset of the routing ACLs (which compare the raw Host), so a
host that routes is never splashed. Never hand-edit the lists; they are
rewritten on every regeneration (also when the splash is off, so turning it
on never meets a missing `-f` file).

**Deploying to an existing host.** The page ships at
`/haproxy/errors/default-splash.html` and the template under
`/haproxy/templates/` -- both in the image, OUTSIDE the `/etc/haproxy` named
volume, so the volume cannot shadow them. The lists live in the volume but are
generated, not copied. So an image bump plus a recreate of the container is
the whole deployment: on start `generate_config()` renders the new listener
and writes both lists, then HAProxy loads them. Nothing to copy into the
volume, no WHP-side change. Verify after the recreate:

```bash
docker exec haproxy-manager grep -c default-splash.html /etc/haproxy/haproxy.cfg   # 2
docker exec haproxy-manager cat /etc/haproxy/routed_hosts.list                     # every site
curl -skI https://<server-hostname>/                       # 404 + x-robots-tag noindex
curl -sk https://<server-hostname>/robots.txt              # User-agent: * / Disallow: /
curl -sk -o /dev/null -w '%{http_code}\n' https://<a-customer-site>/                 # unchanged
```

Rollback: recreate on the previous image tag, or set
`HAPROXY_DEFAULT_SPLASH=off` and regenerate.

Covered by `scripts/test-default-splash.py` (template order/conditions, the
page, list generation) and the `splash-off` scenario of
`scripts/validate-rendered-config.py`.

## Access Log (in-container, since 2026.11.2)

HAProxy logs (`log /log.sock ... format rfc5424`, `hap_header.tpl`) to a UNIX
datagram socket inside its chroot (`/var/lib/haproxy/log.sock`).
`scripts/haproxy-log-receiver.py`, started and supervised by `start-up.sh` (a
restart loop: back within ~2 s), writes one line per request to
`/var/log/haproxy/access.log`, rotated at 40 MB with 4 gzip generations
(worst case 200 MB; `HAPROXY_ACCESS_LOG_MAX_BYTES`, `HAPROXY_ACCESS_LOG_BACKUPS`).
No network listener anywhere. Fields: client/real IP (`cip=`), Host, status,
termination state, request id, UA, SNI, HTTP version, WAF body stamp and the
Coraza `wrule=/wrules=/wscore=` fields (unchanged `log-format`).

* **Never blocks HAProxy.** Sends are non-blocking: a dead or stalled receiver,
  or a full disk, only drops log lines (counted as `DroppedLogs` in
  `echo "@1 show info" | socat stdio /tmp/haproxy-cli`). Lab: with the
  receiver SIGSTOPped, 2500/2500 requests still answered 200 in ~1 s;
  1025 lines queued, 1475 dropped and counted.
* **Not in `docker logs`.** The receiver prints no log lines (WHP's AI log
  monitor reads `docker logs haproxy-manager`; a per-request firehose there
  would storm it). Only its start line and rate-limited errors go to stderr.
* **Persistence.** Mount a named volume at `/var/log/haproxy` (WHP:
  `-v haproxy-logs:/var/log/haproxy`) or the log resets on every recreate.
  Give the container `--sysctl net.unix.max_dgram_qlen=1024` (a netns starts
  at 10) for burst headroom.
* `HAPROXY_SYSLOG_TARGET=<ip>:<port>` still overrides the target for a remote
  collector. The old default `172.18.0.1:514` (host rsyslog) is gone.

Reading it:

```bash
docker exec haproxy-manager tail -f /var/log/haproxy/access.log
docker exec haproxy-manager grep ' 403 ' /var/log/haproxy/access.log | tail
docker exec haproxy-manager sh -c 'zcat /var/log/haproxy/access.log.*.gz' | grep <request-id>
docker cp haproxy-manager:/var/log/haproxy/ ./edge-logs/    # or the volume's _data dir on the host
```

Rollout: image bump + recreate with the volume and sysctl (WHP's
`lib-edge-container.sh` adds both). The receiver, its script and the template
are in the image, outside the `/etc/haproxy` named volume, so nothing can be
shadowed; `haproxy.cfg` is regenerated on start with the new `log` line.

## Default Backend Configuration

With the default splash on (above), unrouted hosts no longer reach this page; it still serves the blocked-IP and suspended pages, and serves unrouted hosts again when `HAPROXY_DEFAULT_SPLASH=off`.

When a domain is accessed that hasn't been configured in HAProxy, the system will serve a default page instead of showing an error. This default page:

- Informs visitors that the site is not configured
- Displays the domain name and current timestamp
- Is fully customizable through environment variables

### Customizing the Default Page

You can customize the default page by setting environment variables:

```bash
docker run -d \
  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
  -v lets-encrypt:/etc/letsencrypt \
  -v haproxy:/etc/haproxy \
  -e HAPROXY_API_KEY=your-secure-api-key-here \
  -e HAPROXY_DEFAULT_PAGE_TITLE="Website Coming Soon" \
  -e HAPROXY_DEFAULT_MAIN_MESSAGE="This website is currently under construction and will be available soon." \
  -e HAPROXY_DEFAULT_SECONDARY_MESSAGE="Please check back later or contact us for more information." \
  --name haproxy-manager \
  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest
```

## Example Usage

### Setting up with API key authentication:
```bash
# Start container with API key
docker run -d \
  -p 80:80 -p 443:443 -p 443:443/udp -p 8000:8000 \
  -v lets-encrypt:/etc/letsencrypt \
  -v haproxy:/etc/haproxy \
  -e HAPROXY_API_KEY=your-secure-api-key-here \
  --name haproxy-manager \
  your-registry.example.com/cloud-hosting-platform/haproxy-manager-base:latest

# Add a domain
curl -X POST http://localhost:8000/api/domain \
  -H "Authorization: Bearer your-secure-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "backend_name": "example_backend",
    "servers": [
      {"name": "server1", "address": "10.0.0.1", "port": 8080, "options": "check"}
    ]
  }'

# Request SSL certificate
curl -X POST http://localhost:8000/api/ssl \
  -H "Authorization: Bearer your-secure-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"domain": "example.com"}'

# Renew certificates
curl -X POST http://localhost:8000/api/certificates/renew \
  -H "Authorization: Bearer your-secure-api-key-here"

# Request certificate generation for another service
curl -X POST http://localhost:8000/api/certificates/request \
  -H "Authorization: Bearer your-secure-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "domains": ["api.example.com"],
    "force_renewal": false,
    "include_www": false
  }'

# Download certificate for another service
curl -H "Authorization: Bearer your-secure-api-key-here" \
  http://localhost:8000/api/certificates/example.com/download \
  -o example.com.pem
```