cloud-hosting-platform/haproxy-manager-base

Files

HAProxy Manager Build and Push / Build-and-Push (push) Successful in 51s

Details

CRITICAL FIX: Migrate HAProxy IP blocking from ACL to map files

**Problem Solved:**
- HAProxy ACL 64-word limit caused config parsing failures
- "too many words, truncating after word 64" error
- Complete service outage when >64 IPs were blocked
- Error: "no such ACL : 'is_blocked'" broke all traffic routing

**Solution: HAProxy Map Files (v1.6+)**
- ✅ Unlimited IP addresses (no word limits)
- ✅ Runtime updates without config reloads
- ✅ Better performance (hash table vs linear search)
- ✅ Safer config management with validation & rollback

**Technical Implementation:**

**Map File Integration:**
- `/etc/haproxy/blocked_ips.map` stores all blocked IPs
- `http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }`
- Runtime updates: `echo "add map #0 IP" | socat stdio /var/run/haproxy.sock`

**Safety Features Added:**
- `create_backup()` - Automatic config/map backups before changes
- `validate_haproxy_config()` - Config validation before applying
- `restore_backup()` - Automatic rollback on failures
- `reload_haproxy_safely()` - Safe reload with validation pipeline

**Runtime Management:**
- `update_blocked_ips_map()` - Sync database to map file
- `add_ip_to_runtime_map()` - Immediate IP blocking without reload
- `remove_ip_from_runtime_map()` - Immediate IP unblocking

**New API Endpoints:**
- `POST /api/config/reload` - Safe config reload with rollback
- `POST /api/blocked-ips/sync` - Sync database to runtime map

**Template Changes:**
- Replaced ACL method: `acl is_blocked src IP1 IP2...` (64 limit)
- With map method: `http-request deny if { src -f blocked_ips.map }` (unlimited)

**Backwards Compatibility:**
- Existing API endpoints unchanged (GET/POST/DELETE /api/blocked-ips)
- Database schema unchanged
- Automatic migration on first config generation

**Performance Improvements:**
- O(1) hash table lookups vs O(n) linear ACL search
- No config reloads needed for IP changes
- Supports millions of IPs if needed
- Memory efficient external file storage

**Documentation:**
- Complete migration guide in MIGRATION_GUIDE.md
- Updated API documentation with new endpoints
- Runtime management examples
- Troubleshooting guide

**Production Safety:**
- All changes include automatic backup/restore
- Config validation prevents bad deployments
- Runtime updates avoid service interruption
- Comprehensive error logging and monitoring

This fixes the critical production outage caused by ACL word limits
while providing a more scalable and performant IP blocking solution.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-08-22 08:31:17 -07:00

4.7 KiB

Raw Blame History

HAProxy Manager Migration Guide: ACL to Map Files

Critical Issue Fixed

HAProxy has a 64 word limit per ACL line, which caused the following error when too many IPs were blocked:

[ALERT] (1485) : config : parsing [/etc/haproxy/haproxy.cfg:58]: too many words, truncating after word 64, position 880: <197.5.145.73>.
[ALERT] (1485) : config : parsing [/etc/haproxy/haproxy.cfg:61] : error detected while parsing an 'http-request set-path' condition : no such ACL : 'is_blocked'.

This caused HAProxy to drop traffic for ALL sites, creating a critical outage.

Solution: Map Files

We've migrated from ACL-based IP blocking to HAProxy map files which:

✅ No word limits - handle millions of IPs
✅ Runtime updates - no config reloads needed
✅ Better performance - hash table lookups instead of linear search
✅ Config validation - automatic rollback on failures
✅ Backup/restore - automatic backup before any changes

What Changed

Before (Problematic ACL Method)

# In haproxy.cfg template
acl is_blocked src 192.168.1.1 192.168.1.2 ... (64 word limit!)
http-request set-path /blocked-ip if is_blocked

After (Map File Method)

# In haproxy.cfg
http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }

# In /etc/haproxy/blocked_ips.map
192.168.1.1
192.168.1.2
64.235.37.112

New Features

1. Safe Configuration Management

Automatic backups before any changes
Configuration validation before applying
Automatic rollback if validation fails
Graceful error handling

2. Runtime IP Management

# Add IP without reload (immediate effect)
echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock

# Remove IP without reload
echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock

3. New API Endpoints

Safe Config Reload

curl -X POST http://localhost:8000/api/config/reload \
  -H "Authorization: Bearer your-api-key"

Sync Runtime Maps

curl -X POST http://localhost:8000/api/blocked-ips/sync \
  -H "Authorization: Bearer your-api-key"

Migration Process

Automatic Migration

The system automatically:

Creates /etc/haproxy/blocked_ips.map from database
Updates HAProxy config to use map files
Validates new configuration
Creates backups before applying changes

Manual Migration (if needed)

# 1. Stop HAProxy manager
systemctl stop haproxy-manager

# 2. Backup current config
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.backup

# 3. Update HAProxy manager code
git pull origin main

# 4. Start HAProxy manager
systemctl start haproxy-manager

# 5. Trigger config regeneration
curl -X POST http://localhost:8000/api/config/reload \
  -H "Authorization: Bearer your-api-key"

Rollback Plan

If issues occur, the system automatically:

Restores backup configuration
Reloads HAProxy with known-good config
Logs all errors for debugging

Manual rollback if needed:

# Restore backup
cp /etc/haproxy/haproxy.cfg.backup /etc/haproxy/haproxy.cfg
systemctl reload haproxy

Performance Benefits

Feature	Old ACL Method	New Map Method
IP Limit	64 IPs max	Unlimited
Updates	Full reload required	Runtime updates
Lookup Speed	O(n) linear	O(1) hash table
Memory Usage	High (all in config)	Low (external file)
Restart Required	Yes	No

Monitoring

Check HAProxy manager logs for any issues:

tail -f /var/log/haproxy-manager.log

Key log entries to watch for:

Configuration validation passed/failed
Backup created/restored
Runtime map updated
Safe reload completed

Troubleshooting

Map File Not Found

# Check if map file exists
ls -la /etc/haproxy/blocked_ips.map

# Manually create if missing
curl -X POST http://localhost:8000/api/blocked-ips/sync \
  -H "Authorization: Bearer your-api-key"

Runtime Updates Not Working

# Check HAProxy stats socket
ls -la /var/run/haproxy.sock /tmp/haproxy-cli

# Test socket connection
echo "show info" | socat stdio /var/run/haproxy.sock

Config Validation Failures