cloud-hosting-platform/haproxy-manager-base

Files

T

shadowdaoandClaude Opus 5 c8d16b6990 fix(ip-blocking): the runtime map fast path has never once run

add_ip_to_runtime_map() and remove_ip_from_runtime_map() sent
`add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True
whenever socat exited 0. Neither command has ever worked, on any deployment,
for the entire life of the feature -- while logging "Added IP x to runtime map"
every single time. Two independent defects:

  * NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket; map
    commands are worker commands. Captured verbatim on whp01:

        $ echo "add map #0 192.0.2.77 1" | socat stdio /tmp/haproxy-cli
        Unknown command: 'add', but maybe one of the following ones is a better match:
          @!<pid>   : send a command to the <pid> process
          ...
        $ echo $?
        0

    socat exits 0 on the rejection, so `result.returncode == 0` was true. Same
    silence PR #7 fixed on the `show table` path.
  * `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and move
    on every config regeneration -- `@1 show map` on whp01 reports
    blocked_ips.map as 37 and trusted_ips.map as 10. There is no id 0.
    Hardcoding any number is wrong; the map is referenced by FILE PATH, which
    is what haproxy.cfg itself names in map_ip(/etc/haproxy/blocked_ips.map,0).

And a third silence, which is why a response-body check alone is not enough
here: `@1 add map #0 <ip> 1` returns an EMPTY body, exit 0, and adds nothing to
any map -- while `@1 del map #0 <ip>` and `@1 show map #0` both answer
`Unknown map identifier.`. On the add path the reply is byte-for-byte identical
to success. Only reading the entry back can tell them apart.

IP blocking itself was never broken: update_blocked_ips_map() rewrites
/etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads it.
That path is untouched and stays authoritative. What was broken is the
no-reload fast path, plus every report that it had worked.

  * haproxy_manager.py: both functions send `@1 add|del map
    /etc/haproxy/blocked_ips.map <ip> [1]` and READ THE ENTRY BACK with
    `get map` before returning True. runtime_map_lookup()/runtime_map_keys()
    are the read-back primitives. `sync_blocked_ips` loses `clear map #0`
    (which the master socket rejected just as loudly and just as invisibly) and
    verifies the whole set with one `show map` instead of counting commands
    that did not visibly complain; it answers 207 + `runtime_map_synced: false`
    when the runtime map does not match the database.
  * haproxy_cli() grows `expect_empty=True` for MUTATING commands: HAProxy
    answers those with nothing on success, so an empty body is the success and
    ANY non-empty body is a rejection. That is stricter than the marker list on
    purpose -- markers only recognise rejections someone has already seen, and
    it catches `'add map' expects three parameters ...`, which matches nothing.
    HaproxyCliError carries `.responses` so `del map` answering `Key not found.`
    (the requested end state) is told apart from a real failure without regex.
  * The four callers capture the boolean instead of discarding it and report
    `runtime_map_updated` / `runtime_map_failures` in the API response and the
    operation log. A runtime failure degrades to "enforced on the reload that
    already happens two lines later" -- never to an unblocked IP, never to a
    500.
  * scripts/test-runtime-map-contract.py (offline, 26 tests) asserts the bytes
    on the wire (`@1` first, map by path, value `1`), classifies every captured
    response, and scans the repo's Python string literals and shell/template
    code lines for `#<id>` map references -- comments may describe the old
    form, code may not use it. Verified to fail on each defect reintroduced
    separately: no `@1` (3 failures), `#0` (4), no read-back (2), trust-the-
    reply (1).
  * The `#0` form is also corrected in IP_BLOCKING_API.md, MIGRATION_GUIDE.md
    and the comment in templates/hap_listener.tpl -- where every copy of it
    additionally omitted the `1`, which `-m int gt 0` needs to match.

The only template change is a comment; `haproxy -c` on the live rendered config
with it applied is clean (HAProxy 3.0.11, warnings unchanged).

Verified on whp01 against the running container (docker cp + SIGHUP, no
recreate). Before: both functions returned True and logged success while
`@1 get map` answered `found=no` and entry_cnt stayed at 263. After: the fixed
add lands with value "1" and the remove takes it out again; the old command
form is now classified as a failure; a `#0` map reference returns False via the
read-back. End to end through the API, `runtime_map_updated: true`, and
/api/blocked-ips/sync -- which used to be a no-op reporting a full sync --
reports 264/264 verified present.

The runtime path was isolated from the reload that normally follows it: with
NO map-file write and NO reload (same haproxy worker pid throughout), adding
100.123.171.78 (whp01's own netbird overlay address -- not a customer IP, not
in the is_local ranges) to the runtime map alone flipped a live site from
HTTP 200 to 403, and removing it flipped it back to 200. That is the fast path
working for the first time. All test IPs were removed afterwards: 0 rows in
blocked_ips, 0 lines in the map file, entry_cnt back to 263. Six customer
sites, the panel /health and `haproxy -c` are byte-identical to the baseline
taken before the change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-22 11:15:01 -07:00

5.2 KiB

Raw Blame History

HAProxy Manager Migration Guide: ACL to Map Files

Critical Issue Fixed

HAProxy has a 64 word limit per ACL line, which caused the following error when too many IPs were blocked:

[ALERT] (1485) : config : parsing [/etc/haproxy/haproxy.cfg:58]: too many words, truncating after word 64, position 880: <197.5.145.73>.
[ALERT] (1485) : config : parsing [/etc/haproxy/haproxy.cfg:61] : error detected while parsing an 'http-request set-path' condition : no such ACL : 'is_blocked'.

This caused HAProxy to drop traffic for ALL sites, creating a critical outage.

Solution: Map Files

We've migrated from ACL-based IP blocking to HAProxy map files which:

✅ No word limits - handle millions of IPs
✅ Runtime updates - no config reloads needed
✅ Better performance - hash table lookups instead of linear search
✅ Config validation - automatic rollback on failures
✅ Backup/restore - automatic backup before any changes

What Changed

Before (Problematic ACL Method)

# In haproxy.cfg template
acl is_blocked src 192.168.1.1 192.168.1.2 ... (64 word limit!)
http-request set-path /blocked-ip if is_blocked

After (Map File Method)

# In haproxy.cfg
http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }

# In /etc/haproxy/blocked_ips.map
192.168.1.1
192.168.1.2
64.235.37.112

New Features

1. Safe Configuration Management

Automatic backups before any changes
Configuration validation before applying
Automatic rollback if validation fails
Graceful error handling

2. Runtime IP Management

Map commands go to a worker (@1), reference the map by file path (ids move between config regenerations, and #0 silently adds nothing), and carry the value 1 that map_ip(...,0) -m int gt 0 matches on:

# Add IP without reload (immediate effect)
echo "@1 add map /etc/haproxy/blocked_ips.map 192.168.1.100 1" | socat stdio /tmp/haproxy-cli

# Remove IP without reload
echo "@1 del map /etc/haproxy/blocked_ips.map 192.168.1.100" | socat stdio /tmp/haproxy-cli

socat exits 0 even when HAProxy rejects the command, so read the response body (or read the entry back with @1 get map ...) rather than the exit status. See IP_BLOCKING_API.md for the full set.

3. New API Endpoints

Safe Config Reload

curl -X POST http://localhost:8000/api/config/reload \
  -H "Authorization: Bearer your-api-key"

Sync Runtime Maps

curl -X POST http://localhost:8000/api/blocked-ips/sync \
  -H "Authorization: Bearer your-api-key"

Migration Process

Automatic Migration

The system automatically:

Creates /etc/haproxy/blocked_ips.map from database
Updates HAProxy config to use map files
Validates new configuration
Creates backups before applying changes

Manual Migration (if needed)

# 1. Stop HAProxy manager
systemctl stop haproxy-manager

# 2. Backup current config
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.backup

# 3. Update HAProxy manager code
git pull origin main

# 4. Start HAProxy manager
systemctl start haproxy-manager

# 5. Trigger config regeneration
curl -X POST http://localhost:8000/api/config/reload \
  -H "Authorization: Bearer your-api-key"

Rollback Plan

If issues occur, the system automatically:

Restores backup configuration
Reloads HAProxy with known-good config
Logs all errors for debugging

Manual rollback if needed:

# Restore backup
cp /etc/haproxy/haproxy.cfg.backup /etc/haproxy/haproxy.cfg
systemctl reload haproxy

Performance Benefits

Feature	Old ACL Method	New Map Method
IP Limit	64 IPs max	Unlimited
Updates	Full reload required	Runtime updates
Lookup Speed	O(n) linear	O(1) hash table
Memory Usage	High (all in config)	Low (external file)
Restart Required	Yes	No

Monitoring

Check HAProxy manager logs for any issues:

tail -f /var/log/haproxy-manager.log

Key log entries to watch for:

Configuration validation passed/failed
Backup created/restored
Runtime map updated
Safe reload completed

Troubleshooting

Map File Not Found

# Check if map file exists
ls -la /etc/haproxy/blocked_ips.map

# Manually create if missing
curl -X POST http://localhost:8000/api/blocked-ips/sync \
  -H "Authorization: Bearer your-api-key"

Runtime Updates Not Working

# Check HAProxy stats socket
ls -la /var/run/haproxy.sock /tmp/haproxy-cli

# Test socket connection
echo "show info" | socat stdio /var/run/haproxy.sock

Config Validation Failures