fix(domain-removal): stop deleting certificates other live sites are served from

DELETE /api/domain ran, unconditionally, for any ssl_enabled row:

    os.remove(ssl_cert_path)
    certbot delete --cert-name <domain> --non-interactive

`domains.domain` is UNIQUE. `domains.ssl_cert_path` is not, and nothing
anywhere guarded against two rows naming the same file. Sharing is not an
edge case -- it is the normal shape of the table, because
request_ssl_bundle() deliberately creates it: one SAN certificate is issued
as `--cert-name <primary>`, published once to
/etc/haproxy/certs/<primary>.pem, and then EVERY included name's row is
pointed at that same path ("Mark every name in the bundle as ssl_enabled,
all pointing at the same combined .pem").

Measured read-only on the live SQLite in the haproxy-manager containers on
2026-08-23:

  * whp01: 157 domain rows, 150 ssl_enabled, 71 distinct cert paths.
    39 of those paths are referenced by MORE THAN ONE row, covering 118 of
    the 150 SSL-enabled rows. Worst cases: brain-jar.com.pem and
    arclightcourt.com.pem with 10 domains each, hackerpublicradio.org.pem
    with 6, anhonesthost.com.pem with 5.
  * whp02: 33 rows, 31 ssl_enabled, 15 distinct paths, 11 shared across 27
    rows -- including threeworldsoneheart.org.pem, referenced by the apex,
    its www, and mail.threeworldsoneheart.org. A production cleanup of that
    mail.* row was stopped short precisely because removing it would have
    unlinked the PEM the serving site is using.

So removing one domain unlinked a file up to nine other configured domains
were being served from. HAProxy binds the crt directory
(`bind ... ssl crt /etc/haproxy/certs`), so the loss is not noticed until
the next reload or restart, at which point the listener refuses to come up
or those names fall back to the wrong certificate.

`certbot delete` is the worse half. It destroys the lineage's archive, live
symlinks and renewal config; recovery is a fresh, rate-limited ACME order.
The old code passed `--cert-name <domain>`, which is also simply the wrong
lineage for a SAN member: 81 of whp01's 150 SSL-enabled rows have a cert
path whose basename is not their own domain, so for those the call was a
silent no-op -- while for a bundle PRIMARY it deleted the one lineage still
renewing the certificate every other name in the bundle is served with.

The fix refcounts, after the row is deleted so the query answers "who else
still needs this":

  * lineage_name_for_cert_path() -- the lineage is the published bundle's
    basename minus .pem, the same derivation _quarantine_superseded_certs()
    already uses, not the domain being removed.
  * domains_referencing_cert_path() / domains_referencing_lineage() -- the
    remaining rows that name that file, and that lineage.
  * remove_domain() unlinks only when the list is empty, `certbot delete`s
    only when the list is empty, logs the retained names explicitly when it
    skips, and reports them as certificate_retained_for /
    lineage_retained_for in the API response.

ssl_enabled is deliberately not filtered on in the refcount: the two
mistakes are not symmetric. A stale PEM left in the crt directory costs a
few kilobytes; an unlinked live one is HTTPS down for every name it serves.
Cleanup is deferred, not cancelled -- removing the last name on a bundle
still unlinks the file and deletes the lineage.

No row on either production host has ssl_enabled=1 with an empty
ssl_cert_path, so the "no path, no attributable lineage" branch changes
nothing on the current fleet.

Tests (scripts/test-cert-write-safety.py, +9, suite now 31, all offline):
last reference -> file unlinked and lineage deleted; shared file survives
removal of a SAN member AND of the bundle primary, byte-for-byte, with the
edge still starting; shared lineage is not certbot-deleted; the production
mail.* shape; removing every name eventually cleans up; an unrelated
bundle is never collateral damage. Assertions are on os.path.exists, file
contents and the recorded certbot argv, never on which branch ran.

Mutation-tested, all five mutants killed:
  1. guard absent entirely (suite run with HAPROXY_MANAGER_DIR pointed at
     main) -> 6 failures, incl. "example.com is still configured and still
     served from this file".
  2. refcount taken before the row is deleted -> 5 failures, incl. "the
     last reference is gone - now it may be removed".
  3. certbot guard removed, file guard kept -> 3 failures, incl.
     [] != ['delete --cert-name example.com --non-interactive'].
  4. lineage taken from the domain name instead of the cert path -> 3
     failures, incl. 'delete --cert-name example.com' != 'delete
     --cert-name www.example.com'.
  5. file-unlink guard removed, certbot guard kept -> 4 failures, incl.
     "two sites are still served from this bundle".

Other suites unchanged and green: test-config-rollback, test-cert-scripts,
test-stick-table-contract, test-runtime-map-contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -7,8 +7,55 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ### Testing
 - **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
 - **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
+- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
+- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
+- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
 - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
 
+### Reading stick tables (and why it is easy to get silently wrong)
+
+`/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Worker commands
+(`show table`, `show map`, `add map`, ...) need an `@1` prefix. Without it
+HAProxy answers `Unknown command: 'show', ...` **and socat still exits 0** — so
+an exit-status check passes and the help text gets parsed as data. Always use
+`haproxy_cli(cmd, worker=True)` in Python, which inspects the response body.
+
+Stick-table entries are `name=value` / `name(window_ms)=value` pairs, not fixed
+columns; the first token is an allocation pointer (`0x...:`), not the key. Parse
+by NAME, and treat a missing field as an ERROR — never default it to `0`. The
+`web` table stores only `conn_cur`, `conn_rate`, `http_req_rate`,
+`http_err_rate`; it holds **no history and no counter of past blocks**. What was
+actually denied/tarpitted is in the edge access log on the **host** at
+`/var/log/haproxy.log` (shipped 2026.08.8), not in any stick table.
+
+This is written down because `/api/security/stats` and `show-tarpit-ips.sh`
+reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
+stick table has ever stored — for their entire existence. See the header of
+`haproxy_tarpit_config.txt` and the contract test.
+
+### Changing a runtime map (`add map` / `del map`)
+
+Same socket, two more ways to fail silently — and both were live in
+`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
+existence:
+
+* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
+  config-parse time and move on every config regeneration (on whp01
+  `blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
+  `add map /etc/haproxy/blocked_ips.map <ip> 1`.
+* **A mutation answers NOTHING on success**, so an empty body is the only
+  success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
+  *also* answers nothing and adds nothing, so the body cannot prove an add
+  worked. **Read it back** with `@1 get map <path> <key>`.
+* Entries must carry the value `1`; haproxy.cfg matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
+
+In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
+and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
+only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
+re-reads it on reload, so a failed runtime command must degrade to
+"enforced on reload" and be reported, never swallowed.
+
 ### Running the Application
 - **Docker Build**: `docker build -t haproxy-manager .`
 - **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)

IP_BLOCKING_API.md

@@ -508,20 +508,45 @@ curl -X POST http://localhost:8000/api/blocked-ips/sync \
 
 For advanced users, you can interact directly with HAProxy's runtime API:
 
+Three things about these commands are easy to get wrong, and each one fails
+**silently** (socat exits 0 either way — the rejection, if any, is only in the
+response body):
+
+* `/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Map commands are
+  worker commands and need the `@1` prefix. Without it the reply is
+  `Unknown command: 'add', ...`.
+* Reference the map by its **file path**, never by `#<id>`. Ids are assigned at
+  config-parse time and move on every config regeneration (on a live edge,
+  `blocked_ips.map` is id 37, `trusted_ips.map` is 10 — there is no id 0).
+  Worse, `@1 add map #0 <ip> 1` returns an **empty** reply and adds nothing.
+* Entries must carry the value `1`. `haproxy.cfg` matches with
+  `map_ip(...,0) -m int gt 0`, so a valueless entry does not block. (`add map`
+  with no value is rejected: `'add map' expects three parameters ...`.)
+
 ```bash
+MAP=/etc/haproxy/blocked_ips.map
+
 # Add IP to runtime (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map $MAP 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP from runtime
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
+
+# Confirm what actually happened (do not trust the exit status)
+echo "@1 get map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
 
 # Clear all blocked IPs from runtime
-echo "clear map #0" | socat stdio /var/run/haproxy.sock
+echo "@1 clear map $MAP" | socat stdio /tmp/haproxy-cli
 
-# Show all runtime map entries
+# Show all runtime map entries, and the map ids currently in use
-echo "show map #0" | socat stdio /var/run/haproxy.sock
+echo "@1 show map $MAP" | socat stdio /tmp/haproxy-cli
+echo "@1 show map" | socat stdio /tmp/haproxy-cli
 ```
 
+The runtime map is a **fast path only**. `/etc/haproxy/blocked_ips.map` is
+authoritative: HAProxy re-reads it on reload, so a failed runtime command
+delays a block until the next reload rather than losing it.
+
 ## Migration from ACL Method
 
 If you're upgrading from the old ACL-based method:

MIGRATION_GUIDE.md

@@ -50,14 +50,22 @@ http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }
 - **Graceful error handling**
 
 ### 2. Runtime IP Management
+Map commands go to a **worker** (`@1`), reference the map by **file path**
+(ids move between config regenerations, and `#0` silently adds nothing), and
+carry the value `1` that `map_ip(...,0) -m int gt 0` matches on:
+
 ```bash
 # Add IP without reload (immediate effect)
-echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 add map /etc/haproxy/blocked_ips.map 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
 
 # Remove IP without reload
-echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
+echo "@1 del map /etc/haproxy/blocked_ips.map 192.168.1.100" | socat stdio /tmp/haproxy-cli
 ```
 
+socat exits 0 even when HAProxy rejects the command, so read the response body
+(or read the entry back with `@1 get map ...`) rather than the exit status.
+See IP_BLOCKING_API.md for the full set.
+
 ### 3. New API Endpoints
 
 #### Safe Config Reload

VERSION

@@ -1 +1 @@
-2026.08.8
+2026.08.11

haproxy_manager.py

@@ -1624,6 +1624,61 @@ def request_certificates():
     else:
         return jsonify(response), 500  # All failed
 
+def lineage_name_for_cert_path(cert_path):
+    """The certbot lineage name implied by a published bundle path.
+
+    Every writer in this module publishes to ``{SSL_CERTS_DIR}/<name>.pem`` and
+    issues that lineage with ``--cert-name <name>`` (see request_ssl() and
+    request_ssl_bundle()); _quarantine_superseded_certs() derives the lineage
+    from the filename the same way. So the basename minus ``.pem`` IS the
+    lineage, and it is NOT necessarily the domain being removed: a bundle
+    issued as ``--cert-name example.com`` also serves www.example.com and any
+    other SAN, all of whose DB rows point at ``/etc/haproxy/certs/example.com.pem``.
+    """
+    if not cert_path:
+        return None
+    base = os.path.basename(cert_path)
+    return base[:-len('.pem')] if base.endswith('.pem') else base
+
+
+def domains_referencing_cert_path(cursor, cert_path):
+    """Domains (still in the table) whose ssl_cert_path is exactly `cert_path`.
+
+    Call this AFTER the row being removed has been deleted, so the answer is
+    "who else still needs this file".
+
+    ssl_enabled is deliberately NOT filtered on. HAProxy binds the whole crt
+    directory, so a file is load-bearing for any row that names it; and the
+    asymmetry of the two mistakes is total - keeping a stale PEM costs nothing,
+    unlinking a live one is HTTPS down for every name it serves.
+    """
+    if not cert_path:
+        return []
+    cursor.execute(
+        'SELECT domain FROM domains WHERE ssl_cert_path = ? ORDER BY domain',
+        (cert_path,))
+    return [row[0] for row in cursor.fetchall()]
+
+
+def domains_referencing_lineage(cursor, lineage):
+    """Domains (still in the table) served by certbot lineage `lineage`.
+
+    Same shape as domains_referencing_cert_path(), one level further back:
+    `certbot delete --cert-name X` destroys the archive, live symlinks and
+    renewal config for X. If any remaining domain is served by a bundle
+    published from that lineage, deleting it means the next renewal silently
+    stops happening for all of them and the material cannot be recovered
+    without a fresh, rate-limited ACME order.
+    """
+    if not lineage:
+        return []
+    cursor.execute(
+        "SELECT domain, ssl_cert_path FROM domains "
+        "WHERE ssl_cert_path IS NOT NULL AND ssl_cert_path != ''")
+    return sorted({domain for domain, path in cursor.fetchall()
+                   if lineage_name_for_cert_path(path) == lineage})
+
+
 @app.route('/api/domain', methods=['DELETE'])
 @require_api_key
 def remove_domain():
@@ -1662,33 +1717,74 @@ def remove_domain():
             # Delete domain
             cursor.execute('DELETE FROM domains WHERE id = ?', (domain_id,))
 
+            # Refcount the certificate BEFORE anything is unlinked or deleted,
+            # with the row above already gone so the query answers "who else
+            # still needs this". One .pem serves many names: request_ssl_bundle()
+            # issues a single SAN cert and points EVERY included domain's row at
+            # the same /etc/haproxy/certs/<primary>.pem. Removing one of those
+            # names used to os.remove() that file and `certbot delete` its
+            # lineage unconditionally - taking HTTPS down for every other name
+            # in the bundle, and destroying the only recoverable copy with it.
+            cert_path_users = domains_referencing_cert_path(cursor, ssl_cert_path)
+            lineage = lineage_name_for_cert_path(ssl_cert_path) if ssl_cert_path else None
+            # The lineage to delete is the one this domain's bundle came from,
+            # not the domain's own name - for a SAN member those differ.
+            lineage_users = domains_referencing_lineage(cursor, lineage)
+
+        cert_retained_for = []
+        lineage_retained_for = []
+
         # Delete SSL certificate from HAProxy certs directory
         if ssl_enabled and ssl_cert_path:
-            try:
+            if cert_path_users:
-                os.remove(ssl_cert_path)
+                cert_retained_for = cert_path_users
-                logger.info(f"Removed HAProxy certificate file: {ssl_cert_path}")
+                logger.info(
-            except OSError as e:
+                    "Kept HAProxy certificate file %s while removing %s: still "
-                logger.warning(f"Failed to remove certificate file {ssl_cert_path}: {e}")
+                    "referenced by %d other domain(s): %s",
+                    ssl_cert_path, domain, len(cert_path_users),
+                    ', '.join(cert_path_users))
+            else:
+                try:
+                    os.remove(ssl_cert_path)
+                    logger.info(f"Removed HAProxy certificate file: {ssl_cert_path}")
+                except OSError as e:
+                    logger.warning(f"Failed to remove certificate file {ssl_cert_path}: {e}")
 
         # Remove certificate from certbot
         if ssl_enabled:
-            try:
+            if not lineage:
-                result = subprocess.run(
+                logger.info(
-                    ['certbot', 'delete', '--cert-name', domain, '--non-interactive'],
+                    "Skipping certbot delete for %s: no certificate path on the "
-                    capture_output=True, text=True
+                    "removed row, so no lineage can be attributed to it", domain)
-                )
+            elif lineage_users:
-                if result.returncode == 0:
+                lineage_retained_for = lineage_users
-                    logger.info(f"Removed Let's Encrypt certificate for {domain}")
+                logger.info(
-                else:
+                    "Kept Let's Encrypt lineage %s while removing %s: still "
-                    logger.warning(f"Failed to remove Let's Encrypt certificate for {domain}: {result.stderr}")
+                    "serving %d other domain(s): %s",
-            except Exception as e:
+                    lineage, domain, len(lineage_users), ', '.join(lineage_users))
-                logger.warning(f"Error removing Let's Encrypt certificate for {domain}: {e}")
+            else:
+                try:
+                    result = subprocess.run(
+                        ['certbot', 'delete', '--cert-name', lineage, '--non-interactive'],
+                        capture_output=True, text=True
+                    )
+                    if result.returncode == 0:
+                        logger.info(f"Removed Let's Encrypt certificate {lineage} for {domain}")
+                    else:
+                        logger.warning(f"Failed to remove Let's Encrypt certificate {lineage} for {domain}: {result.stderr}")
+                except Exception as e:
+                    logger.warning(f"Error removing Let's Encrypt certificate {lineage} for {domain}: {e}")
 
         # Regenerate HAProxy config
         generate_config()
 
         log_operation('remove_domain', True, f'Domain {domain} removed successfully')
-        return jsonify({'status': 'success', 'message': 'Domain configuration removed'})
+        return jsonify({
+            'status': 'success',
+            'message': 'Domain configuration removed',
+            'certificate_retained_for': cert_retained_for,
+            'lineage_retained_for': lineage_retained_for,
+        })
 
     except Exception as e:
         log_operation('remove_domain', False, str(e))
@@ -1735,8 +1831,10 @@ def add_blocked_ip():
             log_operation('add_blocked_ip', False, f'Failed to update map file for {ip_address}')
             return jsonify({'status': 'error', 'message': 'Failed to update blocked IPs map file'}), 500
         
-        # Add to runtime map for immediate effect
+        # Add to runtime map for immediate effect. The map FILE above is what
-        add_ip_to_runtime_map(ip_address)
+        # actually enforces the block once HAProxy re-reads it; this is only
+        # the fast path, so a False here is reported, not fatal.
+        runtime_ok = add_ip_to_runtime_map(ip_address)
         
         # Reload HAProxy to ensure consistency
         try:
@@ -1753,8 +1851,18 @@ def add_blocked_ip():
         except Exception as e:
             logger.warning(f"Error reloading HAProxy after blocking IP {ip_address}: {e}")
         
-        log_operation('add_blocked_ip', True, f'IP {ip_address} blocked successfully')
+        log_operation('add_blocked_ip', True,
-        return jsonify({'status': 'success', 'blocked_ip_id': blocked_ip_id, 'message': f'IP {ip_address} has been blocked'})
+                      f'IP {ip_address} blocked successfully '
+                      f'(runtime map fast path: {"ok" if runtime_ok else "FAILED, enforced on reload"})')
+        return jsonify({
+            'status': 'success',
+            'blocked_ip_id': blocked_ip_id,
+            'message': f'IP {ip_address} has been blocked',
+            # False = the block is enforced from the map file at reload rather
+            # than instantly. The caller can tell the difference; it used to be
+            # reported as instant unconditionally.
+            'runtime_map_updated': runtime_ok
+        })
     except sqlite3.IntegrityError:
         log_operation('add_blocked_ip', False, f'IP {ip_address} is already blocked')
         return jsonify({'status': 'error', 'message': 'IP address is already blocked'}), 409
@@ -1790,8 +1898,9 @@ def remove_blocked_ip():
             log_operation('remove_blocked_ip', False, f'Failed to update map file for {ip_address}')
             return jsonify({'status': 'error', 'message': 'Failed to update blocked IPs map file'}), 500
         
-        # Remove from runtime map for immediate effect
+        # Remove from runtime map for immediate effect. As with blocking, the
-        remove_ip_from_runtime_map(ip_address)
+        # map file is authoritative and the reload below picks it up.
+        runtime_ok = remove_ip_from_runtime_map(ip_address)
         
         # Reload HAProxy to ensure consistency
         try:
@@ -1808,8 +1917,14 @@ def remove_blocked_ip():
         except Exception as e:
             logger.warning(f"Error reloading HAProxy after unblocking IP {ip_address}: {e}")
         
-        log_operation('remove_blocked_ip', True, f'IP {ip_address} unblocked successfully')
+        log_operation('remove_blocked_ip', True,
-        return jsonify({'status': 'success', 'message': f'IP {ip_address} has been unblocked'})
+                      f'IP {ip_address} unblocked successfully '
+                      f'(runtime map fast path: {"ok" if runtime_ok else "FAILED, applied on reload"})')
+        return jsonify({
+            'status': 'success',
+            'message': f'IP {ip_address} has been unblocked',
+            'runtime_map_updated': runtime_ok
+        })
     except Exception as e:
         log_operation('remove_blocked_ip', False, str(e))
         return jsonify({'status': 'error', 'message': str(e)}), 500
@@ -1843,103 +1958,375 @@ def sync_blocked_ips():
             cursor.execute('SELECT ip_address FROM blocked_ips ORDER BY ip_address')
             blocked_ips = [row[0] for row in cursor.fetchall()]
         
-        # Try to clear all entries from runtime map (might fail if empty, that's ok)
+        # Clear the runtime map and re-add every blocked IP. The map is
+        # referenced by FILE PATH: `clear map #0` (what this used to send, with
+        # no `@1` either) was answered by the master socket with "Unknown
+        # command: 'clear'" and socat exited 0, so this whole block was a no-op
+        # that reported a full sync.
         try:
-            if os.path.exists(HAPROXY_SOCKET_PATH):
+            haproxy_cli('clear map %s' % BLOCKED_IPS_MAP_PATH,
-                socket_path = HAPROXY_SOCKET_PATH
+                        worker=True, expect_empty=True)
-            else:
+        except HaproxyCliError as e:
-                socket_path = '/tmp/haproxy-cli'
+            log_operation('sync_blocked_ips', False, f'Failed to clear runtime map: {e}')
+            logger.warning("Failed to clear runtime map: %s. The map file is "
+                           "still authoritative and correct.", e)
+            return jsonify({
+                'status': 'error',
+                'message': f'Failed to clear runtime map: {e}',
+                'map_file_updated': True,
+                'runtime_map_synced': False,
+                'total_ips': len(blocked_ips)
+            }), 500
 
-            subprocess.run(f'echo "clear map #0" | socat stdio {socket_path}', 
+        # verify=False here: one `show map` read-back below costs a single
-                         shell=True, capture_output=True)
+        # round trip instead of one per IP, and answers the same question for
-        except:
+        # the whole set.
-            pass  # Clear might fail if map is empty
+        accepted = 0
-        
-        # Add all IPs to runtime map
-        success_count = 0
         for ip in blocked_ips:
-            if add_ip_to_runtime_map(ip):
+            if add_ip_to_runtime_map(ip, verify=False):
-                success_count += 1
+                accepted += 1
 
-        log_operation('sync_blocked_ips', True, f'Synced {success_count}/{len(blocked_ips)} IPs to runtime map')
+        # Ground truth, not a count of commands that did not visibly complain.
+        try:
+            present = runtime_map_keys(BLOCKED_IPS_MAP_PATH)
+        except HaproxyCliError as e:
+            log_operation('sync_blocked_ips', False, f'Could not read back runtime map: {e}')
+            return jsonify({
+                'status': 'error',
+                'message': f'Could not read back the runtime map to verify the sync: {e}',
+                'map_file_updated': True,
+                'runtime_map_synced': False,
+                'total_ips': len(blocked_ips)
+            }), 500
+
+        missing = [ip for ip in blocked_ips if ip not in present]
+        synced = len(blocked_ips) - len(missing)
+        ok = not missing
+        if missing:
+            logger.warning(
+                "Runtime map sync incomplete: %d/%d IPs are not in %s (first "
+                "few: %s). They remain blocked via the map file on reload.",
+                len(missing), len(blocked_ips), BLOCKED_IPS_MAP_PATH, missing[:5])
+
+        log_operation('sync_blocked_ips', ok,
+                      f'Verified {synced}/{len(blocked_ips)} IPs present in the runtime map')
         return jsonify({
-            'status': 'success', 
+            'status': 'success' if ok else 'partial',
-            'message': f'Synced {success_count}/{len(blocked_ips)} IPs to runtime map',
+            'message': f'Verified {synced}/{len(blocked_ips)} IPs present in the runtime map',
             'total_ips': len(blocked_ips),
-            'synced_ips': success_count
+            'synced_ips': synced,
-        })
+            'accepted_commands': accepted,
+            'missing_ips': missing[:50],
+            'runtime_map_synced': ok
+        }), (200 if ok else 207)
     except Exception as e:
         log_operation('sync_blocked_ips', False, str(e))
         return jsonify({'status': 'error', 'message': str(e)}), 500
 
+# ---------------------------------------------------------------------------
+# HAProxy runtime API (stick tables)
+#
+# WHY THIS SECTION IS SO DEFENSIVE
+# --------------------------------
+# The previous /api/security/stats read `gpc0` and `gpc1` out of `show table
+# web` and reported them as "scan count" / "offense count" / "blocked". No
+# stick table in this repo has EVER stored a general-purpose counter — see
+# STICK_TABLE_FIELD_CONTRACT below and the `store` clauses in
+# templates/hap_listener.tpl and templates/hap_security_tables.tpl. Three
+# separate silences let that survive:
+#
+#   1. `int(parts[3])` on a positional split raised ValueError on `exp=368842`
+#      and the loop just `continue`d, so every row was skipped and the endpoint
+#      always answered `active_threats: 0` with an empty list. An operator
+#      reading that saw "no threats" and could not tell it apart from "the
+#      parser is broken".
+#   2. The command was sent WITHOUT a worker prefix. /tmp/haproxy-cli is the
+#      MASTER socket; `show table web` there is answered with "Unknown command:
+#      'show', but maybe one of the following ones is a better match: ..." --
+#      and socat still exits 0, so `result.returncode != 0` never fired. The
+#      reported `total_tracked_ips` was literally the number of lines in that
+#      help text minus one (8), while the real table held 388 entries.
+#   3. The shell consumers defaulted every missing field to 0 (`${gpc0:-0}`),
+#      so a field that does not exist rendered as a confident zero.
+#
+# Rules for anything added here, all three aimed at the same failure mode:
+#   * Read the CONTRACT, not positions. Stick-table output is `name=value` /
+#     `name(window)=value` pairs whose order and presence follow the template's
+#     `store` clause. Positional indexing silently reads the wrong column the
+#     moment that clause changes.
+#   * NEVER default a missing field to a number. A field the table does not
+#     store must surface as an error naming the field, not as 0.
+#   * NEVER trust socat's exit status. HAProxy reports command errors in the
+#     response BODY and the socket still closes cleanly. Use haproxy_cli().
+#
+# scripts/test-stick-table-contract.py holds STICK_TABLE_FIELD_CONTRACT, the
+# rendered templates and the shell consumers to each other, and fails if any
+# one of them drifts.
+# ---------------------------------------------------------------------------
+
+# What each stick table ACTUALLY stores, per its `store` clause. The single
+# source of truth for every consumer in this repo. Keep in sync with the
+# templates -- the contract test enforces that, in both directions.
+STICK_TABLE_FIELD_CONTRACT = {
+    'web': ('conn_cur', 'conn_rate', 'http_req_rate', 'http_err_rate'),
+    'wp_bruteforce': ('http_req_rate',),
+    'xmlrpc_bruteforce': ('http_req_rate',),
+}
+
+# Metadata every stick-table entry carries regardless of the `store` clause.
+STICK_TABLE_ENTRY_META = ('key', 'use', 'exp', 'shard')
+
+# HAProxy answers a rejected runtime command in the response body and the
+# socket still closes 0. These are the prefixes it uses.
+_HAPROXY_CLI_ERROR_MARKERS = (
+    'Unknown command',
+    'No such table',
+    # `add map #0 ...` / `get map /etc/haproxy/nope.map ...` -- captured
+    # verbatim from HAProxy 3.0.11 on the live edge.
+    'Unknown map identifier',
+    'Key not found',
+    'Permission denied',
+    "Can't find the specified process",
+    'unknown process',
+    'Missing ',
+)
+
+_STICK_TABLE_TOKEN_RE = re.compile(r'^([a-z_][a-z0-9_]*)(?:\(([^)]*)\))?=(.*)$')
+_STICK_TABLE_HEADER_RE = re.compile(
+    r'^#\s*table:\s*(?P<name>[^,]+),\s*type:\s*(?P<type>[^,]+),\s*'
+    r'size:\s*(?P<size>\d+),\s*used:\s*(?P<used>\d+)')
+
+
+class HaproxyCliError(RuntimeError):
+    """A runtime-API command was rejected, timed out, or answered nothing.
+
+    Exists so a rejected command cannot be mistaken for an empty result. That
+    distinction is the whole point of this module's stick-table code.
+
+    `.responses` holds the raw, stripped response body of every attempt, so a
+    caller can tell one rejection apart from another (`del map` answering
+    "Key not found." is a no-op, not a failure) without regex-matching the
+    formatted message.
+    """
+
+    responses = ()
+
+
+def _haproxy_socket_path():
+    return HAPROXY_SOCKET_PATH if os.path.exists(HAPROXY_SOCKET_PATH) else '/tmp/haproxy-cli'
+
+
+def _cli_response_is_error(text):
+    if text is None:
+        return True
+    head = text.lstrip()
+    return any(head.startswith(marker) for marker in _HAPROXY_CLI_ERROR_MARKERS)
+
+
+def _cli_send(command, socket_path, timeout):
+    proc = subprocess.run(
+        ['socat', 'stdio', socket_path],
+        input=command + '\n', capture_output=True, text=True, timeout=timeout)
+    if proc.returncode != 0:
+        raise HaproxyCliError(
+            'socat failed talking to %s (exit %d): %s'
+            % (socket_path, proc.returncode, (proc.stderr or '').strip()))
+    return proc.stdout
+
+
+def haproxy_cli(command, worker=False, timeout=None, expect_empty=False):
+    """Send one runtime-API command and return its response, or raise.
+
+    `worker=True` marks a command that only the WORKER answers (show table,
+    show map, add map, ...). On this deployment the socket is HAProxy's MASTER
+    CLI, where those need an `@1` prefix; on a plain stats socket they must NOT
+    have one. Rather than guessing from configuration that can change under us,
+    try the prefixed form and fall back -- and raise if BOTH are rejected,
+    instead of returning HAProxy's help text as if it were data.
+
+    `expect_empty=True` is for MUTATING commands (add/del/clear map, set map).
+    HAProxy answers those with nothing at all on success, so the rule inverts:
+    an empty body is the success, and ANY non-empty body is a rejection. That
+    is deliberately stricter than matching _HAPROXY_CLI_ERROR_MARKERS -- the
+    marker list can only ever recognise the rejections someone has already
+    seen, and a mutation that prints anything has not done what was asked. Two
+    real examples this catches that the marker list did not:
+    `'add map' expects three parameters ...` and `Unknown map identifier.`
+    """
+    socket_path = _haproxy_socket_path()
+    timeout = timeout if timeout is not None else DEFAULT_SUBPROCESS_TIMEOUT
+    attempts = (['@1 ' + command, command] if worker else [command])
+    failures = []
+    bodies = []
+    for attempt in attempts:
+        try:
+            out = _cli_send(attempt, socket_path, timeout)
+        except subprocess.TimeoutExpired:
+            raise HaproxyCliError('timed out after %ss running %r on %s'
+                                  % (timeout, attempt, socket_path))
+        body = out.strip()
+        if expect_empty:
+            if not body:
+                return out
+        elif body and not _cli_response_is_error(out):
+            return out
+        bodies.append(body)
+        failures.append('%r -> %r' % (attempt, body[:200] or '<empty response>'))
+    error = HaproxyCliError(
+        'HAProxy rejected %r on %s (socat exited 0 -- the rejection is in the '
+        'response body, which is exactly why this is checked): %s'
+        % (command, socket_path, '; '.join(failures)))
+    error.responses = tuple(bodies)
+    raise error
+
+
+def parse_stick_table_entry(line):
+    """{field: {'value': str, 'window_ms': int|None}} for one `show table` row.
+
+    Parses `name=value` / `name(window)=value` pairs by NAME. The leading
+    `0x...:` allocation pointer is skipped -- reading it as the key is how the
+    old code came to report memory addresses as IP addresses.
+    """
+    fields = {}
+    for token in line.split():
+        m = _STICK_TABLE_TOKEN_RE.match(token)
+        if not m:
+            continue  # the 0x...: pointer, or anything else unnamed
+        name, window, value = m.group(1), m.group(2), m.group(3)
+        fields[name] = {
+            'value': value,
+            'window_ms': int(window) if window and window.isdigit() else None,
+        }
+    return fields
+
+
+def read_stick_table(table):
+    """(header dict, [(raw line, parsed fields)]) for a stick table.
+
+    Raises HaproxyCliError if the response is not a stick-table dump, or if any
+    row is missing a field the contract says the table stores. A field the
+    table does not carry is an ERROR here, never a zero.
+    """
+    expected = STICK_TABLE_FIELD_CONTRACT.get(table)
+    if expected is None:
+        raise HaproxyCliError(
+            'no field contract for stick table %r; add it to '
+            'STICK_TABLE_FIELD_CONTRACT (and to the templates) first' % table)
+
+    raw = haproxy_cli('show table %s' % table, worker=True)
+    lines = raw.strip().split('\n')
+    header = _STICK_TABLE_HEADER_RE.match(lines[0]) if lines else None
+    if not header:
+        raise HaproxyCliError(
+            'response to `show table %s` is not a stick-table dump; first line '
+            'was %r' % (table, lines[0][:200] if lines else ''))
+
+    entries = []
+    for line in lines[1:]:
+        if not line.strip() or line.lstrip().startswith('#'):
+            continue
+        fields = parse_stick_table_entry(line)
+        if 'key' not in fields:
+            raise HaproxyCliError(
+                'stick-table row for %r has no key= field: %r' % (table, line[:200]))
+        missing = [f for f in expected if f not in fields]
+        if missing:
+            raise HaproxyCliError(
+                'stick table %r no longer stores %s -- STICK_TABLE_FIELD_CONTRACT '
+                'and the `store` clause in templates/hap_listener.tpl have drifted '
+                'apart. Present: %s. Offending row: %r'
+                % (table, ', '.join(missing),
+                   ', '.join(sorted(fields)), line[:200]))
+        entries.append((line, fields))
+
+    return {
+        'name': header.group('name'),
+        'type': header.group('type'),
+        'size': int(header.group('size')),
+        'used': int(header.group('used')),
+    }, entries
+
+
 @app.route('/api/security/stats', methods=['GET'])
 @require_api_key
 def get_security_stats():
-    """Get current security statistics from HAProxy stick table"""
+    """Per-source connection and request rates from the `web` stick table.
+
+    Reports ONLY what the table stores: conn_cur, conn_rate, http_req_rate and
+    http_err_rate, each with the window HAProxy is actually counting over. It
+    deliberately does NOT classify a "threat level" or report a "blocked" flag:
+    the thresholds live in templates/hap_listener.tpl and a copy here would be
+    a second source of truth free to drift, which is the class of bug this
+    endpoint used to be. Sources at or over a limit are visible from the rates
+    themselves, and the enforcement that actually happened -- 429s, tarpits
+    (termination state PT), WAF denials -- is in the edge access log on the
+    HOST at /var/log/haproxy.log, which records per-request outcomes the stick
+    table never held.
+
+    Query params:
+      limit         max sources returned (default 50)
+      min_req_rate  only sources at or above this http_req_rate (default 1,
+                    i.e. sources with current activity; pass 0 for all)
+    """
     try:
-        if os.path.exists(HAPROXY_SOCKET_PATH):
+        limit = max(1, min(int(request.args.get('limit', 50)), 1000))
-            socket_path = HAPROXY_SOCKET_PATH
+        min_req_rate = max(0, int(request.args.get('min_req_rate', 1)))
-        else:
+    except ValueError:
-            socket_path = '/tmp/haproxy-cli'
+        return jsonify({'status': 'error',
+                        'message': 'limit and min_req_rate must be integers'}), 400
 
-        # Get stick table data
+    try:
-        cmd = f'echo "show table web" | socat stdio {socket_path}'
+        header, entries = read_stick_table('web')
-        result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
+    except HaproxyCliError as e:
-
+        log_operation('get_security_stats', False, str(e))
-        if result.returncode != 0:
+        return jsonify({'status': 'error', 'message': str(e)}), 502
-            return jsonify({'status': 'error', 'message': 'Failed to get stick table data'}), 500
-
-        # Parse stick table output
-        lines = result.stdout.strip().split('\n')
-        threats = []
-
-        for line in lines[1:]:  # Skip header
-            parts = line.split()
-            if len(parts) >= 8:
-                ip = parts[0]
-                try:
-                    gpc0 = int(parts[3]) if len(parts) > 3 else 0
-                    gpc1 = int(parts[4]) if len(parts) > 4 else 0
-                    req_rate = int(parts[5]) if len(parts) > 5 else 0
-                    err_rate = int(parts[6]) if len(parts) > 6 else 0
-                    conn_rate = int(parts[7]) if len(parts) > 7 else 0
-
-                    # Only include IPs with significant activity
-                    if gpc0 > 0 or gpc1 > 0 or req_rate > 30 or err_rate > 5 or conn_rate > 10:
-                        threat_level = 'low'
-                        if gpc1 > 2:
-                            threat_level = 'critical'
-                        elif gpc0 > 0 or err_rate > 10:
-                            threat_level = 'high'
-                        elif req_rate > 40 or conn_rate > 15:
-                            threat_level = 'medium'
-
-                        threats.append({
-                            'ip': ip,
-                            'blocked': gpc0 > 0,
-                            'repeat_offender': gpc1 > 2,
-                            'offense_count': gpc1,
-                            'request_rate': req_rate,
-                            'error_rate': err_rate,
-                            'connection_rate': conn_rate,
-                            'threat_level': threat_level
-                        })
-                except (ValueError, IndexError):
-                    continue
-
-        # Sort by threat level
-        threats.sort(key=lambda x: (x['offense_count'], x['error_rate'], x['request_rate']), reverse=True)
-
-        return jsonify({
-            'status': 'success',
-            'total_tracked_ips': len(lines) - 1,
-            'active_threats': len(threats),
-            'threats': threats[:50]  # Limit to top 50
-        })
     except Exception as e:
         log_operation('get_security_stats', False, str(e))
         return jsonify({'status': 'error', 'message': str(e)}), 500
 
+    counters = STICK_TABLE_FIELD_CONTRACT['web']
+    windows = {}
+    sources = []
+    active = 0
+    for line, fields in entries:
+        row = {'ip': fields['key']['value']}
+        for name in counters:
+            # A non-numeric counter means HAProxy's output format changed under
+            # us. Say so; do not coerce it to 0 and report it as a measurement.
+            try:
+                row[name] = int(fields[name]['value'])
+            except ValueError:
+                msg = ('stick table web reported a non-numeric %s=%r; the '
+                       '`show table` output format has changed. Row: %r'
+                       % (name, fields[name]['value'], line[:200]))
+                log_operation('get_security_stats', False, msg)
+                return jsonify({'status': 'error', 'message': msg}), 502
+            if fields[name]['window_ms'] is not None:
+                windows.setdefault(name, fields[name]['window_ms'])
+        if any(row[name] > 0 for name in counters):
+            active += 1
+        if row['http_req_rate'] >= min_req_rate:
+            sources.append(row)
+
+    sources.sort(key=lambda r: (r['http_req_rate'], r['http_err_rate'],
+                                r['conn_rate'], r['conn_cur']), reverse=True)
+
+    return jsonify({
+        'status': 'success',
+        'table': header['name'],
+        'table_size': header['size'],
+        'total_tracked_ips': header['used'],
+        'sources_with_activity': active,
+        'counters': list(counters),
+        'counter_windows_ms': windows,
+        'returned': len(sources[:limit]),
+        'min_req_rate': min_req_rate,
+        'sources': sources[:limit],
+        'note': ('Current per-source rates only. The stick table stores no '
+                 'history and no counter of past blocks; enforcement events '
+                 'are in the edge access log on the host at /var/log/haproxy.log.'),
+    })
+
 @app.route('/api/security/temporary-block', methods=['POST'])
 @require_api_key
 def temporary_block():
@@ -1977,13 +2364,16 @@ def temporary_block():
         if not update_blocked_ips_map():
             return jsonify({'status': 'error', 'message': 'Failed to update map file'}), 500
 
-        add_ip_to_runtime_map(ip_address)
+        runtime_ok = add_ip_to_runtime_map(ip_address)
 
-        log_operation('temporary_block', True, f'Temporarily blocked {ip_address} for {duration_minutes} minutes')
+        log_operation('temporary_block', True,
+                      f'Temporarily blocked {ip_address} for {duration_minutes} minutes '
+                      f'(runtime map fast path: {"ok" if runtime_ok else "FAILED, enforced on reload"})')
         return jsonify({
             'status': 'success',
             'message': f'IP {ip_address} temporarily blocked for {duration_minutes} minutes',
-            'expires_at': expiry_time.isoformat()
+            'expires_at': expiry_time.isoformat(),
+            'runtime_map_updated': runtime_ok
         })
     except Exception as e:
         log_operation('temporary_block', False, str(e))
@@ -1996,6 +2386,10 @@ def clear_expired_blocks():
     try:
         current_time = datetime.now()
         expired_ips = []
+        # IPs the runtime fast path could not drop. They still come out of the
+        # map file below, so they unblock on the next reload -- but silently
+        # reporting them as cleared is what this whole change is about.
+        runtime_failures = []
 
         with sqlite3.connect(DB_FILE) as conn:
             cursor = conn.cursor()
@@ -2016,17 +2410,21 @@ def clear_expired_blocks():
                 # Remove expired IPs
                 for ip in expired_ips:
                     cursor.execute('DELETE FROM blocked_ips WHERE ip_address = ?', (ip,))
-                    remove_ip_from_runtime_map(ip)
+                    if not remove_ip_from_runtime_map(ip):
+                        runtime_failures.append(ip)
 
         # Update map file if any IPs were removed
         if expired_ips:
             update_blocked_ips_map()
 
-        log_operation('clear_expired_blocks', True, f'Cleared {len(expired_ips)} expired IP blocks')
+        log_operation('clear_expired_blocks', not runtime_failures,
+                      f'Cleared {len(expired_ips)} expired IP blocks '
+                      f'({len(runtime_failures)} not removed from the runtime map)')
         return jsonify({
-            'status': 'success',
+            'status': 'success' if not runtime_failures else 'partial',
             'message': f'Cleared {len(expired_ips)} expired IP blocks',
-            'cleared_ips': expired_ips
+            'cleared_ips': expired_ips,
+            'runtime_map_failures': runtime_failures
         })
     except Exception as e:
         log_operation('clear_expired_blocks', False, str(e))
@@ -3065,50 +3463,167 @@ def update_blocked_ips_map(promote_backup=True):
         logger.error(f"Failed to update blocked IPs map: {e}")
         return False
 
-def add_ip_to_runtime_map(ip_address):
+# ---------------------------------------------------------------------------
-    """Add IP to HAProxy runtime map without reload"""
+# Runtime map fast path (blocked IPs)
+#
+# WHAT WAS WRONG, AND WHY NOBODY NOTICED FOR ITS ENTIRE EXISTENCE
+# ---------------------------------------------------------------
+# add_ip_to_runtime_map()/remove_ip_from_runtime_map() sent
+# `add map #0 <ip> 1` / `del map #0 <ip>` to /tmp/haproxy-cli and returned True
+# whenever socat exited 0. Two independent defects, three silences:
+#
+#   1. NO `@1` PREFIX. /tmp/haproxy-cli is HAProxy's MASTER CLI socket. Map
+#      commands are worker commands. The master answers
+#      `Unknown command: 'add', but maybe one of the following ones is a better
+#      match: ...` -- and socat still exits 0, so `result.returncode == 0` was
+#      true and the function logged "Added IP x to runtime map".
+#   2. `#0` IS NOT A VALID MAP ID. Ids are assigned at config-parse time and
+#      move whenever the config is regenerated; on whp01 blocked_ips.map is
+#      id 37 and trusted_ips.map is 10. There is no id 0. Hardcoding ANY number
+#      is wrong -- reference the map by its FILE PATH, which is stable because
+#      it is what haproxy.cfg names in `map_ip(/etc/haproxy/blocked_ips.map,0)`.
+#   3. `add map #0 ...` fails SILENTLY EVEN WITH `@1`. Captured on HAProxy
+#      3.0.11: `@1 add map #0 192.0.2.88 1` returns an EMPTY body, exit 0, and
+#      adds nothing to any map -- while `@1 del map #0 <ip>` and
+#      `@1 show map #0` both answer `Unknown map identifier.`. So a
+#      response-body check alone cannot catch defect 2 on the add path. That is
+#      why every mutation here is READ BACK with `get map` instead of trusting
+#      either the exit status or the (empty) reply.
+#
+# The blocking itself never depended on this: update_blocked_ips_map() rewrites
+# /etc/haproxy/blocked_ips.map and the callers reload HAProxy, which re-reads
+# the file. The FILE IS AUTHORITATIVE; this is only the no-reload fast path.
+# Every function below therefore returns a bool the caller can report, and
+# never raises into a request handler -- a runtime-map failure must degrade to
+# "enforced on reload", not to "not blocked" and not to a 500.
+#
+# scripts/test-runtime-map-contract.py holds the command strings and the
+# classification of every captured response to these rules.
+# ---------------------------------------------------------------------------
+
+# The value every blocked_ips.map entry must carry. haproxy.cfg matches with
+# `map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0`, so a keyed entry with
+# no value evaluates to 0 and is NOT blocked. Runtime map and file must agree.
+BLOCKED_IPS_MAP_VALUE = '1'
+
+_GET_MAP_FOUND_RE = re.compile(r'\bfound=(yes|no)\b')
+_GET_MAP_VALUE_RE = re.compile(r'\bvalue="([^"]*)"')
+
+
+def runtime_map_lookup(map_path, key):
+    """(found, value) for one key in a runtime map, or raise HaproxyCliError.
+
+    Reads back what `add map`/`del map` actually did. `get map` answers
+    `type=ip, case=sensitive, found=yes, idx=tree, key="1.2.3.4", value="1",
+    type="str"` or `type=ip, case=sensitive, found=no`; an unusable map
+    reference answers `Unknown map identifier.`, which haproxy_cli() rejects.
+    """
+    out = haproxy_cli('get map %s %s' % (map_path, key), worker=True)
+    match = _GET_MAP_FOUND_RE.search(out)
+    if not match:
+        raise HaproxyCliError(
+            'unparseable `get map %s %s` response (no found=yes/no): %r'
+            % (map_path, key, out.strip()[:200]))
+    if match.group(1) == 'no':
+        return (False, None)
+    value = _GET_MAP_VALUE_RE.search(out)
+    return (True, value.group(1) if value else None)
+
+
+def runtime_map_keys(map_path):
+    """The set of keys currently in a runtime map, or raise HaproxyCliError.
+
+    `show map <file>` emits `<0x-pointer> <key> <value>` per line. An empty map
+    legitimately emits nothing, which is why this does not go through the
+    non-empty check.
+    """
     try:
-        if os.path.exists(HAPROXY_SOCKET_PATH):
+        out = haproxy_cli('show map %s' % map_path, worker=True)
-            socket_path = HAPROXY_SOCKET_PATH
+    except HaproxyCliError as e:
-        else:
+        # An empty map is a real, distinguishable state -- not a rejection.
-            socket_path = '/tmp/haproxy-cli'
+        if e.responses and all(body == '' for body in e.responses):
+            return set()
+        raise
+    keys = set()
+    for line in out.splitlines():
+        parts = line.split()
+        if len(parts) >= 2 and parts[0].startswith('0x'):
+            keys.add(parts[1])
+    return keys
 
-        # Add to runtime map (map file ID 0 for blocked IPs)
-        # Format: add map #<id> <key> <value>
-        # For IP blocking, value is always "1"
-        cmd = f'echo "add map #0 {ip_address} 1" | socat stdio {socket_path}'
-        result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
 
-        if result.returncode == 0:
+def add_ip_to_runtime_map(ip_address, verify=True):
-            logger.info(f"Added IP {ip_address} to runtime map")
+    """Add IP to the running HAProxy's blocked map without a reload.
-            return True
+
-        else:
+    Returns True only when the entry is verifiably present with the value the
-            logger.warning(f"Failed to add IP to runtime map: {result.stderr}")
+    config matches on. False means the FILE + reload path is what will enforce
-            return False
+    this block -- which it does regardless; see the section comment above.
+    """
+    try:
+        haproxy_cli(
+            'add map %s %s %s' % (BLOCKED_IPS_MAP_PATH, ip_address, BLOCKED_IPS_MAP_VALUE),
+            worker=True, expect_empty=True)
+        if verify:
+            found, value = runtime_map_lookup(BLOCKED_IPS_MAP_PATH, ip_address)
+            if not found:
+                raise HaproxyCliError(
+                    '`add map` was accepted but %s is not in %s afterwards -- '
+                    'the command did nothing (this is exactly how `#0` failed)'
+                    % (ip_address, BLOCKED_IPS_MAP_PATH))
+            if value != BLOCKED_IPS_MAP_VALUE:
+                raise HaproxyCliError(
+                    '%s is in %s with value %r, not %r -- haproxy.cfg matches '
+                    'with `-m int gt 0`, so this entry does NOT block'
+                    % (ip_address, BLOCKED_IPS_MAP_PATH, value, BLOCKED_IPS_MAP_VALUE))
+        logger.info(f"Added IP {ip_address} to runtime map (verified={verify})")
+        return True
+    except HaproxyCliError as e:
+        logger.warning(
+            "Runtime map fast path FAILED for %s: %s. The block is NOT lost -- "
+            "%s was rewritten and HAProxy re-reads it on reload -- but it does "
+            "not take effect until that reload completes.",
+            ip_address, e, BLOCKED_IPS_MAP_PATH)
+        return False
     except Exception as e:
-        logger.error(f"Error adding IP to runtime map: {e}")
+        logger.error(f"Error adding IP {ip_address} to runtime map: {e}")
         return False
 
-def remove_ip_from_runtime_map(ip_address):
+
-    """Remove IP from HAProxy runtime map without reload"""
+def remove_ip_from_runtime_map(ip_address, verify=True):
+    """Remove IP from the running HAProxy's blocked map without a reload.
+
+    Returns True only when the key is verifiably gone. `Key not found.` means
+    the runtime map never had it, which is the requested end state, so that is
+    a success -- but it is logged, because it also means the runtime map and
+    the file had drifted apart.
+    """
     try:
-        if os.path.exists(HAPROXY_SOCKET_PATH):
+        try:
-            socket_path = HAPROXY_SOCKET_PATH
+            haproxy_cli('del map %s %s' % (BLOCKED_IPS_MAP_PATH, ip_address),
-        else:
+                        worker=True, expect_empty=True)
-            socket_path = '/tmp/haproxy-cli'
+        except HaproxyCliError as e:
-        
+            if not any(body.startswith('Key not found') for body in e.responses):
-        # Remove from runtime map (map file ID 0 for blocked IPs)
+                raise
-        cmd = f'echo "del map #0 {ip_address}" | socat stdio {socket_path}'
+            logger.info(
-        result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
+                "Runtime map had no entry for %s to remove (`Key not found.`); "
-        
+                "the file and the runtime map had drifted", ip_address)
-        if result.returncode == 0:
+        if verify:
-            logger.info(f"Removed IP {ip_address} from runtime map")
+            found, _ = runtime_map_lookup(BLOCKED_IPS_MAP_PATH, ip_address)
-            return True
+            if found:
-        else:
+                raise HaproxyCliError(
-            logger.warning(f"Failed to remove IP from runtime map: {result.stderr}")
+                    '`del map` was accepted but %s is STILL in %s'
-            return False
+                    % (ip_address, BLOCKED_IPS_MAP_PATH))
+        logger.info(f"Removed IP {ip_address} from runtime map (verified={verify})")
+        return True
+    except HaproxyCliError as e:
+        logger.warning(
+            "Runtime map fast path FAILED for %s: %s. The unblock is NOT lost -- "
+            "%s was rewritten and HAProxy re-reads it on reload -- but the IP "
+            "stays blocked until that reload completes.",
+            ip_address, e, BLOCKED_IPS_MAP_PATH)
+        return False
     except Exception as e:
-        logger.error(f"Error removing IP from runtime map: {e}")
+        logger.error(f"Error removing IP {ip_address} from runtime map: {e}")
         return False
 
 def start_haproxy():

haproxy_tarpit_config.txt

@@ -1,3 +1,37 @@
+# =============================================================================
+# NOT IMPLEMENTED. THIS FILE IS A DESIGN SKETCH THAT WAS NEVER SHIPPED.
+# =============================================================================
+#
+# Nothing here is deployed, has ever been deployed, or is rendered into
+# haproxy.cfg. The real edge config is generated from templates/*.tpl. Compare:
+#
+#   THIS FILE proposes:  store gpc0,gpc1,gpc2,http_err_rate(30s),...
+#                        plus sc-inc-gpc0/1/2 scan-escalation rules
+#   templates/hap_listener.tpl ACTUALLY has:
+#                        store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(30s)
+#
+# There is no gpc0, no gpc1, no gpc2, and no scan-escalation state anywhere on
+# the edge, and never has been.
+#
+# WHY THE BANNER. This file was mistaken for the shipped config. Two consumers
+# -- /api/security/stats in haproxy_manager.py and scripts/show-tarpit-ips.sh --
+# were written to parse gpc0/gpc1 out of `show table web`, defaulting missing
+# fields to 0. The result was a "Scan Count" column and BLOCKED/TARPITTED
+# statuses that were pure fabrication, presented to an operator as fact, for
+# the entire life of both tools. Fixed 2026-08-22; scripts/test-stick-table-
+# contract.py now fails if any consumer's field expectations and the templates'
+# `store` clauses ever drift apart again.
+#
+# IF YOU WANT TO REVIVE ANY OF THIS: the counters must be added to a template
+# `store` clause and to STICK_TABLE_FIELD_CONTRACT in haproxy_manager.py first.
+# Adding them to a consumer alone produces confident zeros, not data. Note also
+# that sc0/sc1/sc2 are all in use and HAProxy's tune.stick-counters defaults to
+# 3, and that since 2026.08.8 the edge has real per-request access logging on
+# the host at /var/log/haproxy.log -- which records what was actually denied,
+# tarpitted and rate-limited, with request references. That log is a better
+# source for most of what this sketch was reaching for.
+# =============================================================================
+
 global
     daemon
     log stdout local0 info

scripts/monitor-attacks.sh

@@ -1,136 +1,148 @@
-#!/bin/bash
+#!/usr/bin/env bash
+#
+# monitor-attacks.sh — HAProxy edge activity monitor.
+#
+# Two sections, both fed from real data:
+#   1. Current per-IP rates, from the `web` stick table (delegated to
+#      show-edge-ip-rates.sh — there is exactly one stick-table parser).
+#   2. Recent enforcement events, from the HAProxy access log.
+#
+# HISTORY / WHY THIS IS SHORTER THAN IT USED TO BE
+#   The previous version printed a "Threat Intelligence Dashboard" with
+#   fourteen categories (auth_fail, authz_fail, scanner, sql_inj, traversal,
+#   wp_brute, admin_scan, shell_att, repeat_off, manual_bl, auto_bl,
+#   glitch_rate, ...) and a composite "threat score", all parsed out of
+#   gpc(0), gpc(1), gpc(3), gpc(12), gpc(13) and glitch_rate(300s). NONE of
+#   those fields exist: the `web` table stores only conn_cur, conn_rate,
+#   http_req_rate and http_err_rate. Every category was permanently 0 and the
+#   whole dashboard printed nothing while implying it was watching. All of it
+#   has been deleted rather than "fixed" — there was no data source to fix it
+#   against.
+#
+# Usage: monitor-attacks.sh [live]
+# Env:   LOG_FILE=<path>  access log to read (default /var/log/haproxy.log)
+#        LOG_LINES=<n>    how many trailing log lines to scan (default 500)
 
-# Real-time attack monitoring for HAProxy
+set -uo pipefail
-# Shows blocked requests and suspicious activity
 
-LOG_FILE="/var/log/haproxy.log"
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-SOCKET="/tmp/haproxy-cli"
+LOG_FILE="${LOG_FILE:-/var/log/haproxy.log}"
+LOG_LINES="${LOG_LINES:-500}"
 
-echo "==================================================="
+# --- Section 1: current rates (real stick-table data) -----------------------
-echo "HAProxy Security Monitor - Real-time Attack Detection"
+show_rates() {
-echo "==================================================="
+    "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
-echo ""
-
-# Function to show current threats with HAProxy 3.0.11 metrics
-show_threats() {
-    echo "HAProxy 3.0.11 Threat Intelligence Dashboard:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 {
-            # Parse the stick table output for array-based GPC values
-            ip = $1
-            # Look for GPC array values in the data
-            auth_fail = 0
-            authz_fail = 0
-            rate_viol = 0
-            scanner = 0
-            sql_inj = 0
-            traversal = 0
-            wp_brute = 0
-            admin_scan = 0
-            shell_att = 0
-            repeat_off = 0
-            manual_bl = 0
-            auto_bl = 0
-            glitch_rate = 0
-            threat_score = 0
-
-            # Extract relevant metrics (simplified parsing)
-            if ($0 ~ /gpc\(0\)=([0-9]+)/) {
-                match($0, /gpc\(0\)=([0-9]+)/, arr); auth_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(1\)=([0-9]+)/) {
-                match($0, /gpc\(1\)=([0-9]+)/, arr); authz_fail = arr[1]
-            }
-            if ($0 ~ /gpc\(3\)=([0-9]+)/) {
-                match($0, /gpc\(3\)=([0-9]+)/, arr); scanner = arr[1]
-            }
-            if ($0 ~ /gpc\(12\)=([0-9]+)/) {
-                match($0, /gpc\(12\)=([0-9]+)/, arr); repeat_off = arr[1]
-            }
-            if ($0 ~ /gpc\(13\)=([0-9]+)/) {
-                match($0, /gpc\(13\)=([0-9]+)/, arr); manual_bl = arr[1]
-            }
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr); glitch_rate = arr[1]
-            }
-
-            # Calculate composite threat score (simplified)
-            threat_score = auth_fail*10 + authz_fail*8 + scanner*12 + repeat_off*25 + manual_bl*100
-
-            # Only show IPs with significant threat indicators
-            if (auth_fail > 0 || authz_fail > 0 || scanner > 0 || repeat_off > 0 || manual_bl > 0 || glitch_rate > 0) {
-                threat_level = "LOW"
-                if (threat_score >= 100) threat_level = "CRITICAL"
-                else if (threat_score >= 50) threat_level = "HIGH"
-                else if (threat_score >= 20) threat_level = "MEDIUM"
-
-                printf "%-15s [%8s] Score:%-3d Auth:%-2d Authz:%-2d Scanner:%-1d Repeat:%-1d Glitch:%-2d\n",
-                       ip, threat_level, threat_score, auth_fail, authz_fail, scanner, repeat_off, glitch_rate
-            }
-        }' | head -15
-
-    echo ""
-    echo "Top HTTP/2 Protocol Violators:"
-    echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
-        awk 'NR>1 && $0 ~ /glitch/ {
-            if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
-                match($0, /glitch_rate\(300s\)=([0-9]+)/, arr)
-                if (arr[1] > 2) {
-                    printf "%-15s glitch_rate:%-3s\n", $1, arr[1]
-                }
-            }
-        }' | head -5
-    echo "---------------------------------------------------"
 }
 
-# Function to show recent blocks
+# --- Section 2: recent enforcement events (real access-log data) ------------
 show_recent_blocks() {
-    echo "Recent Blocked Requests:"
+    echo "Recent enforcement events (last $LOG_LINES log lines):"
-    tail -100 "$LOG_FILE" 2>/dev/null | \
+    echo
-        grep -E "(bot_scanner|scan_admin|scan_shells|sql_injection|directory_traversal|rate_abuse|tarpit|denied|403)" | \
+
-        tail -10 | \
+    if [ ! -f "$LOG_FILE" ] || [ ! -r "$LOG_FILE" ]; then
-        awk '{
+        cat <<MSG
-            if (match($0, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+/)) {
+  Access log not readable at: $LOG_FILE
-                ip = substr($0, RSTART, RLENGTH)
+
-                gsub(/:.*/, "", ip)
+  This is expected INSIDE the haproxy-manager container: HAProxy logs to
-                reason = ""
+  syslog on the DOCKER HOST, and the file lives on the host, not in here.
-                if ($0 ~ /bot_scanner/) reason = "BOT_SCANNER"
+  Read it from the host instead:
-                else if ($0 ~ /scan_admin/) reason = "ADMIN_SCAN"
+
-                else if ($0 ~ /scan_shells/) reason = "SHELL_SCAN"
+    grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
-                else if ($0 ~ /sql_injection/) reason = "SQL_INJECTION"
+    grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limited / blocked
-                else if ($0 ~ /directory_traversal/) reason = "DIR_TRAVERSAL"
+    grep -a 'cip=<IP>'  /var/log/haproxy.log | tail -50     # one client IP
-                else if ($0 ~ /rate_abuse/) reason = "RATE_ABUSE"
+    grep -a 'id=<uuid>' /var/log/haproxy.log                # one request reference
-                else if ($0 ~ /tarpit/) reason = "TARPIT"
+                                                            # (the UUID on the block page)
-                else if ($0 ~ /denied/) reason = "DENIED"
+    tail -f /var/log/haproxy.log | grep -aE ' (PT|PR)--'    # live
-                else if ($0 ~ /403/) reason = "BLOCKED"
+
-                printf "[%s] %-15s %s\n", strftime("%H:%M:%S"), ip, reason
+  Or point this script at a copy:  LOG_FILE=/path/to/haproxy.log $0
+MSG
+        echo
+        return 0
+    fi
+
+    printf "%-15s %-16s %-4s %-5s %-28s %s\n" "TIME" "CLIENT IP" "CODE" "TERM" "HOST" "REQUEST / REQUEST-ID"
+    printf "%s\n" "-----------------------------------------------------------------------------------------------------"
+
+    local found
+    found=$(tail -n "$LOG_LINES" "$LOG_FILE" 2>/dev/null | awk '
+    {
+        status = ""; term = ""; cip = ""; host = ""; id = ""; ts = ""; req = ""
+
+        # %tr is bracketed: [22/Aug/2026:10:11:12.345] -> keep HH:MM:SS
+        # No {n} interval expressions here: not every awk in a slim Debian
+        # image supports them. Spelled out instead.
+        if (match($0, /\[[0-9][0-9]\/[A-Za-z][A-Za-z][A-Za-z]\/[0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/)) {
+            ts = substr($0, RSTART + 13, 8)
+        }
+
+        # Anchor on the %TR/%Tw/%Tc/%Tr/%Ta timers block: %ST follows it, and
+        # the termination state (%tsc) is 4 fields further on (%B %CC %CS %tsc).
+        for (i = 1; i <= NF; i++) {
+            if ($i ~ /^[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+$/) {
+                status = $(i + 1)
+                term = $(i + 5)
+                break
             }
-        }'
+        }
-    echo ""
+
+        # Only enforcement outcomes: tarpit (PT--), deny (PR--), 403, 429.
+        if (!(term ~ /^PT/ || term ~ /^PR/ || status == "403" || status == "429")) next
+
+        for (i = 1; i <= NF; i++) {
+            if (substr($i, 1, 4) == "cip=")  cip  = substr($i, 5)
+            if (substr($i, 1, 5) == "host=") host = substr($i, 6)
+            if (substr($i, 1, 3) == "id=")   id   = substr($i, 4)
+        }
+
+        if (match($0, /"[A-Z]+ [^"]*"/)) {
+            req = substr($0, RSTART + 1, RLENGTH - 2)
+            if (length(req) > 42) req = substr(req, 1, 41) "..."
+        }
+
+        if (cip == "") cip = "-"
+        if (host == "") host = "-"
+        if (term == "") term = "-"
+        if (status == "") status = "-"
+        printf "%-15s %-16s %-4s %-5s %-28s %s\n", ts, cip, status, term, host, req
+        if (id != "" && id != "-") printf "%-15s %s\n", "", "  id=" id
+        n++
+    }
+    END { if (n == 0) print "(no tarpit/deny/403/429 events in the scanned window)" }
+    ')
+    printf '%s\n' "$found"
+    echo
+    echo "TERM = HAProxy termination state: PT-- tarpit, PR-- deny (incl. WAF/rate limit)."
+    echo "id=  = request reference; it is printed on the block page and is how a"
+    echo "       customer support ticket correlates to an exact request here."
 }
 
-# Monitor mode selection
+banner() {
-if [ "$1" == "live" ]; then
+    echo "==================================================="
-    echo "Live monitoring mode - Press Ctrl+C to exit"
+    echo "HAProxy Edge Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
-    echo ""
+    echo "==================================================="
+    echo
+}
 
+if [ "${1:-}" = "live" ]; then
+    echo "Live monitoring mode - Press Ctrl+C to exit"
     while true; do
         clear
-        echo "==================================================="
+        banner
-        echo "HAProxy Security Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
+        show_rates || true
-        echo "==================================================="
+        echo
-        echo ""
-        show_threats
-        echo ""
         show_recent_blocks
         sleep 5
     done
 else
-    # Single run mode
+    banner
-    show_threats
+    rc=0
-    echo ""
+    show_rates || rc=$?
+    echo
     show_recent_blocks
-    echo ""
+    echo
-    echo "Tip: Run with 'live' parameter for continuous monitoring"
+    echo "Tip: run with 'live' for a refreshing view."
     echo "Usage: $0 [live]"
+    # Propagate a stick-table read failure: if the rates section could not be
+    # produced, this run did NOT report what it claims to report.
+    exit "$rc"
 fi

scripts/show-edge-ip-rates.sh

@@ -0,0 +1,314 @@
+#!/usr/bin/env bash
+#
+# show-edge-ip-rates.sh — real, current per-IP rate counters from the HAProxy
+# `web` stick table.
+#
+# WHAT THIS CAN TELL YOU
+#   The `web` stick table (templates/hap_listener.tpl) stores exactly four
+#   counters per client IP:
+#       conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s)
+#   Those are INSTANTANEOUS values — the current concurrency and the current
+#   sliding-window rates. This script prints them, and nothing else.
+#
+# WHAT THIS CANNOT TELL YOU
+#   * Who has been tarpitted, denied, or rate-limited. The stick table stores
+#     NO history and NO counter of past enforcement actions. It has no gpc0 /
+#     gpc1 / gpc(N) / gpc_rate / glitch_rate columns at all — any tool that
+#     claims to read them from this table is fabricating numbers.
+#   * Anything about an IP that has gone quiet: entries expire after 10m.
+#
+#   Real enforcement events live in the HAProxy ACCESS LOG, which is on the
+#   DOCKER HOST at /var/log/haproxy.log (it does NOT exist inside this
+#   container). The log-format carries the HAProxy termination state plus
+#   cip= (real client IP), host=, ua= and id= (the request UUID shown on the
+#   block page, which correlates with customer support tickets).
+#
+#   Ready to run ON THE HOST:
+#       # last 20 tarpitted (PT--) or denied (PR--) requests
+#       grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+#       # everything HAProxy answered 429/403 to, newest last
+#       grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20
+#       # everything for one client IP
+#       grep -a 'cip=203.0.113.7' /var/log/haproxy.log | tail -50
+#       # look up one request reference from a support ticket
+#       grep -a 'id=<uuid-from-the-block-page>' /var/log/haproxy.log
+#
+# USAGE
+#   show-edge-ip-rates.sh [-a|--all]
+#     -a, --all   also show rows whose counters are all zero (off by default:
+#                 a table with hundreds of idle entries is pure noise)
+#
+# ENVIRONMENT
+#   SHOW_ALL=1               same as --all
+#   HAPROXY_SOCKET=<path>    override the CLI socket (default /tmp/haproxy-cli)
+#   HAPROXY_TABLE_DUMP=<file>
+#                            parse a previously captured `show table web` dump
+#                            from a file instead of talking to the socket.
+#                            Supported seam for offline analysis of a captured
+#                            support bundle, and for testing this parser.
+#
+# NOTE ON THE SOCKET
+#   /tmp/haproxy-cli is HAProxy's MASTER CLI socket, so worker commands need an
+#   `@1` prefix. Without it HAProxy answers "Unknown command: 'show' ..." AND
+#   socat still exits 0 — so exit status is worthless here and this script
+#   inspects the RESPONSE BODY instead.
+
+set -euo pipefail
+
+SOCKET="${HAPROXY_SOCKET:-/tmp/haproxy-cli}"
+TABLE="web"
+
+# Fields this script expects the `web` stick table to store. Keep on ONE line
+# in this exact NAME=(a b c d) shape — the contract test greps for it, and the
+# parser below is driven entirely by it.
+EXPECTED_FIELDS=(conn_cur conn_rate http_req_rate http_err_rate)
+
+# Which of EXPECTED_FIELDS to sort on (descending). Falls back to the first
+# field if this name is not in the list.
+SORT_FIELD="http_req_rate"
+
+SHOW_ALL="${SHOW_ALL:-0}"
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -a|--all) SHOW_ALL=1 ;;
+        -h|--help) sed -n '2,60p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
+        *) echo "Unknown argument: $1" >&2; echo "Usage: $0 [-a|--all]" >&2; exit 2 ;;
+    esac
+    shift
+done
+
+die() { echo "ERROR: $*" >&2; exit 1; }
+
+# First non-blank line of a blob.
+#
+# Deliberately NOT `printf ... | sed -n '/./{p;q;}'`. sed quits after the first
+# match and closes the pipe; on a real 550-entry table dump printf is still
+# writing and takes SIGPIPE, so under `set -o pipefail` the whole command
+# substitution returns 141 and `set -e` kills the script -- silently, with no
+# output at all. That is the same class of failure this script exists to stop
+# hiding, so it does not get to happen here. A plain read loop has no pipeline
+# and no early close.
+first_nonblank() {
+    local line
+    while IFS= read -r line || [ -n "$line" ]; do
+        case "$line" in
+            *[![:space:]]*) printf '%s\n' "$line"; return 0 ;;
+        esac
+    done <<EOF
+$1
+EOF
+    return 0
+}
+
+# Return 0 if the CLI response body is a rejection rather than table data.
+# Checked on the body because socat's exit status is 0 either way.
+body_is_rejected() {
+    local first
+    first=$(first_nonblank "$1")
+    case "$first" in
+        "Unknown command"*|"No such table"*|"Permission denied"*) return 0 ;;
+        *) return 1 ;;
+    esac
+}
+
+send_cmd() {
+    printf '%s\n' "$1" | socat stdio "$SOCKET" 2>/dev/null
+}
+
+# ---------------------------------------------------------------- fetch data
+BODY=""
+SOURCE=""
+if [ -n "${HAPROXY_TABLE_DUMP:-}" ]; then
+    [ -r "$HAPROXY_TABLE_DUMP" ] || die "HAPROXY_TABLE_DUMP is set but '$HAPROXY_TABLE_DUMP' is not readable."
+    BODY=$(cat "$HAPROXY_TABLE_DUMP")
+    SOURCE="file $HAPROXY_TABLE_DUMP"
+else
+    [ -S "$SOCKET" ] || die "HAProxy CLI socket not found at $SOCKET (is HAProxy running, and are you inside the haproxy-manager container?)"
+    command -v socat >/dev/null 2>&1 || die "socat is not installed; cannot talk to $SOCKET"
+
+    # Master socket form first, then the plain stats-socket form.
+    BODY=$(send_cmd "@1 show table $TABLE" || true)
+    SOURCE="socket $SOCKET (@1 show table $TABLE)"
+    if [ -z "${BODY//[[:space:]]/}" ] || body_is_rejected "$BODY"; then
+        FALLBACK=$(send_cmd "show table $TABLE" || true)
+        if [ -n "${FALLBACK//[[:space:]]/}" ] && ! body_is_rejected "$FALLBACK"; then
+            BODY="$FALLBACK"
+            SOURCE="socket $SOCKET (show table $TABLE)"
+        else
+            echo "ERROR: HAProxy rejected BOTH '@1 show table $TABLE' and 'show table $TABLE'." >&2
+            echo "  @1 response : $(first_nonblank "$BODY")" >&2
+            echo "  bare response: $(first_nonblank "$FALLBACK")" >&2
+            echo "  Check the socket is HAProxy's CLI and that the table '$TABLE' exists" >&2
+            echo "  (a config reload without the frontend would drop it)." >&2
+            exit 1
+        fi
+    fi
+fi
+
+# ------------------------------------------------------------- header checks
+HEADER=$(first_nonblank "$BODY")
+case "$HEADER" in
+    "# table: $TABLE,"*) : ;;
+    *)
+        echo "ERROR: unexpected first line from '$SOURCE'." >&2
+        echo "  expected it to start with: # table: $TABLE," >&2
+        echo "  got                      : $HEADER" >&2
+        exit 1
+        ;;
+esac
+
+TBL_SIZE=$(printf '%s\n' "$HEADER" | sed -n 's/.*size:\([0-9]*\).*/\1/p')
+TBL_USED=$(printf '%s\n' "$HEADER" | sed -n 's/.*used:\([0-9]*\).*/\1/p')
+[ -n "$TBL_SIZE" ] || TBL_SIZE="?"
+[ -n "$TBL_USED" ] || TBL_USED="?"
+
+# ------------------------------------------------------------------- parsing
+# awk emits:
+#   W \t <field>:<window-seconds-or-dash> ...       (one line, from first row)
+#   R \t <sortkey> \t <ip> \t <value per EXPECTED_FIELDS in order>
+# and exits 1 after reporting any row missing an expected field.
+PARSED=""
+if ! PARSED=$(printf '%s\n' "$BODY" | awk -v fieldlist="${EXPECTED_FIELDS[*]}" -v sortfield="$SORT_FIELD" '
+BEGIN {
+    nf = split(fieldlist, F, " ")
+    sortidx = 1
+    for (i = 1; i <= nf; i++) if (F[i] == sortfield) sortidx = i
+    wprinted = 0
+}
+/^#/ { next }
+!/key=/ { next }
+{
+    split("", val, " "); split("", win, " ")
+    for (i = 1; i <= NF; i++) {
+        tok = $i
+        p = index(tok, "=")
+        if (p == 0) continue
+        lhs = substr(tok, 1, p - 1)
+        rhs = substr(tok, p + 1)
+        b = index(lhs, "(")
+        if (b > 0) {
+            nm = substr(lhs, 1, b - 1)
+            win[nm] = substr(lhs, b + 1, length(lhs) - b - 1)
+        } else {
+            nm = lhs
+            win[nm] = ""
+        }
+        val[nm] = rhs
+    }
+
+    missing = ""
+    for (i = 1; i <= nf; i++) if (!(F[i] in val)) missing = missing (missing == "" ? "" : ", ") F[i]
+    if (missing != "") {
+        printf "ERROR: stick table row is missing expected field(s): %s\n", missing > "/dev/stderr"
+        printf "  offending row: %s\n", $0 > "/dev/stderr"
+        printf "  this script expects the web table to store: %s\n", fieldlist > "/dev/stderr"
+        print  "  Those expectations and the templates/hap_listener.tpl `store` clause have DRIFTED." > "/dev/stderr"
+        print  "  Fix one or the other; refusing to print 0 for a counter HAProxy never reported." > "/dev/stderr"
+        exit 1
+    }
+    if (!("key" in val)) {
+        printf "ERROR: stick table row has no key= field: %s\n", $0 > "/dev/stderr"
+        exit 1
+    }
+
+    if (!wprinted) {
+        line = "W"
+        for (i = 1; i <= nf; i++) {
+            w = win[F[i]]
+            if (w ~ /^[0-9]+$/) w = sprintf("%g", w / 1000); else w = "-"
+            line = line "\t" F[i] ":" w
+        }
+        print line
+        wprinted = 1
+    }
+
+    nonzero = 0
+    row = ""
+    for (i = 1; i <= nf; i++) {
+        v = val[F[i]]
+        if (v + 0 != 0) nonzero = 1
+        row = row "\t" v
+    }
+    printf "R\t%s\t%s\t%d%s\n", val[F[sortidx]] + 0, val["key"], nonzero, row
+}
+'); then
+    exit 1
+fi
+
+# ------------------------------------------------------------------ printing
+WINSPEC=$(printf '%s\n' "$PARSED" | sed -n 's/^W\t//p' || true)
+
+echo "==================================================================="
+echo "  HAProxy edge IP rates — table '$TABLE' (current values only)"
+echo "==================================================================="
+echo "Source     : $SOURCE"
+echo "Tracked    : ${TBL_USED} of ${TBL_SIZE} slots in use"
+if [ "$SHOW_ALL" = "1" ]; then
+    echo "Filter     : showing ALL tracked IPs"
+else
+    echo "Filter     : showing only IPs with a non-zero counter (use --all for every row)"
+fi
+echo
+
+# Column headers, with each counter's window rendered in SECONDS (HAProxy
+# reports the window in milliseconds, e.g. conn_rate(10000) = 10s).
+HDR=$(printf "%-18s" "IP Address")
+i=0
+for f in "${EXPECTED_FIELDS[@]}"; do
+    w=$(printf '%s\n' "$WINSPEC" | tr '\t' '\n' | sed -n "s/^${f}://p")
+    if [ -n "$w" ] && [ "$w" != "-" ]; then
+        label="${f}/${w}s"
+    else
+        label="$f"
+    fi
+    HDR="$HDR $(printf '%18s' "$label")"
+    i=$((i + 1))
+done
+echo "$HDR"
+printf '%s\n' "$HDR" | sed 's/./-/g'
+
+ROWS=$(printf '%s\n' "$PARSED" | sed -n 's/^R\t//p' || true)
+shown=0
+if [ -n "$ROWS" ]; then
+    while IFS=$'\t' read -r sortkey ip nonzero rest; do
+        [ -n "${ip:-}" ] || continue
+        if [ "$SHOW_ALL" != "1" ] && [ "$nonzero" = "0" ]; then
+            continue
+        fi
+        line=$(printf "%-18s" "$ip")
+        oldifs="$IFS"; IFS=$'\t'
+        # shellcheck disable=SC2086
+        set -- $rest
+        IFS="$oldifs"
+        for v in "$@"; do
+            line="$line $(printf '%18s' "$v")"
+        done
+        echo "$line"
+        shown=$((shown + 1))
+    done < <(printf '%s\n' "$ROWS" | sort -t"$(printf '\t')" -k1,1nr)
+fi
+
+if [ "$shown" -eq 0 ]; then
+    if [ "$SHOW_ALL" = "1" ]; then
+        echo "(no IPs currently tracked)"
+    else
+        echo "(no IP currently has a non-zero counter — re-run with --all to list idle entries)"
+    fi
+fi
+
+echo
+echo "==================================================================="
+echo "These are CURRENT values. The table keeps no history and no record of"
+echo "past tarpits/denials. For actual enforcement events, read the access"
+echo "log ON THE DOCKER HOST (it does not exist in this container):"
+echo "  grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20     # tarpit / deny"
+echo "  grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20    # rate-limit / block"
+echo "  grep -a 'cip=<IP>' /var/log/haproxy.log | tail -50        # one client"
+echo "  grep -a 'id=<uuid>' /var/log/haproxy.log                  # one request reference"
+echo
+echo "Operator actions (via the MASTER CLI socket — the @1 prefix is required):"
+echo "  printf '@1 show table $TABLE key <IP>\\n'  | socat stdio $SOCKET"
+echo "  printf '@1 set table $TABLE key <IP> data.http_req_rate 0\\n' | socat stdio $SOCKET"
+echo "  printf '@1 clear table $TABLE key <IP>\\n' | socat stdio $SOCKET   # drop one entry"
+echo "  printf '@1 clear table $TABLE\\n'          | socat stdio $SOCKET   # drop ALL entries"
+echo "==================================================================="

scripts/show-tarpit-ips.sh

@@ -1,123 +1,35 @@
-#!/bin/bash
+#!/usr/bin/env bash
-
-# Script to display IPs that have been tarpitted by HAProxy 3.0
-# Uses HAProxy stats socket to query stick-table data
 #
-# Usage in Docker container:
+# DEPRECATED SHIM — kept so existing docs/runbooks/muscle memory keep working.
-#   docker exec -it haproxy-manager /haproxy/scripts/show-tarpit-ips.sh
+#
+# This script used to print a "Tarpitted IPs Report" with a "Scan Count" and a
+# BLOCKED / SILENT-DROP / TARPIT status per IP, all derived from gpc0 and gpc1
+# stick-table columns. Those columns DO NOT EXIST: the `web` table
+# (templates/hap_listener.tpl) stores only conn_cur, conn_rate, http_req_rate
+# and http_err_rate. The old parser defaulted every missing field to 0, so the
+# whole report was fabricated — every IP showed "Scan Count 0 / Normal"
+# regardless of what it was actually doing.
+#
+# The stick table also keeps NO history, so nothing in it can identify who was
+# tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST
+# at /var/log/haproxy.log (it does not exist inside this container):
+#   grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20   # tarpit / deny
+#   grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20  # rate-limit / block
+#
+# What IS knowable from the stick table — the current per-IP rates — is printed
+# by show-edge-ip-rates.sh, which this shim now runs.
 
-SOCKET="/tmp/haproxy-cli"
+set -euo pipefail
 
-# Check if socket exists
+cat >&2 <<'NOTE'
-if [ ! -S "$SOCKET" ]; then
+NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits.
-    echo "Error: HAProxy socket not found at $SOCKET"
+      The HAProxy stick table stores no history and no gpc0/gpc1 counters, so
-    echo "Make sure HAProxy is running with stats socket enabled"
+      the old "Scan Count"/"BLOCKED" columns were fabricated numbers.
-    exit 1
+      Actual tarpit/deny events are in /var/log/haproxy.log ON THE HOST:
-fi
+        grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
+      Running show-edge-ip-rates.sh instead (current rates, real values):
 
-echo "==================================================================="
+NOTE
-echo "                    HAProxy Tarpitted IPs Report                   "
-echo "==================================================================="
-echo
-echo "Showing IPs tracked in the stick-table with scan detection counters:"
-echo "(gpc0 = total scan attempts, gpc1 = escalation level)"
-echo
 
-# In HAProxy 3.0, we need to use the proper process prefix
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-# The web frontend table is in the worker process, not master
+exec "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
-# First check which process has the table
-# Note: grep for actual worker line, not the header
-PROCESS_ID=$(echo "show proc" | socat stdio "$SOCKET" 2>/dev/null | grep -E '^[0-9]+.*worker' | awk '{print $1}' | head -1)
-
-if [ -z "$PROCESS_ID" ]; then
-    echo "Error: Could not find HAProxy worker process"
-    echo "Try: echo 'show proc' | socat stdio $SOCKET"
-    exit 1
-fi
-
-# Show stick-table entries from the web frontend using the worker process
-# Use printf to avoid bash history expansion issues with !
-printf "@!%s show table web\n" "${PROCESS_ID}" | socat stdio "$SOCKET" 2>/dev/null | {
-    # Skip the header line
-    read header
-    
-    # Check if we got an error or empty response
-    if echo "$header" | grep -q "No such table"; then
-        echo "Error: Table 'web' not found. HAProxy may need to be reloaded."
-        exit 1
-    fi
-    
-    has_data=false
-    echo "IP Address           | Scan Count | Level | HTTP Err Rate | Status"
-    echo "---------------------|------------|-------|---------------|------------------"
-    
-    # Process each line
-    while IFS= read -r line; do
-        # Skip empty lines and comments
-        if [ -z "$line" ] || echo "$line" | grep -q "^#"; then
-            continue
-        fi
-        
-        # HAProxy 3.0 format: 0x... key=<ip> use=... exp=... gpc0=... gpc1=... http_err_rate(10s)=...
-        if echo "$line" | grep -q "key="; then
-            has_data=true
-            
-            # Extract IP and counters
-            ip=$(echo "$line" | grep -o 'key=[^ ]*' | cut -d'=' -f2)
-            gpc0=$(echo "$line" | grep -o 'gpc0=[0-9]*' | cut -d'=' -f2)
-            gpc1=$(echo "$line" | grep -o 'gpc1=[0-9]*' | cut -d'=' -f2)
-            err_rate=$(echo "$line" | grep -o 'http_err_rate([^)]*=[0-9]*' | grep -o '[0-9]*$')
-            
-            # Set defaults if values are empty
-            gpc0=${gpc0:-0}
-            gpc1=${gpc1:-0}
-            err_rate=${err_rate:-0}
-            
-            # Determine status based on scan count and escalation
-            status=""
-            if [ "$gpc0" -ge 100 ]; then
-                status="BLOCKED (429)"
-            elif [ "$gpc0" -ge 60 ]; then
-                status="SILENT-DROP"
-            elif [ "$gpc0" -ge 40 ]; then
-                if [ "$gpc1" -ge 2 ]; then
-                    status="SILENT-DROP (repeat)"
-                else
-                    status="TARPIT 10s"
-                fi
-            elif [ "$gpc0" -ge 25 ]; then
-                status="TARPIT 10s"
-            else
-                status="Normal"
-            fi
-            
-            # Format output
-            printf "%-20s | %10s | %5s | %13s | %s\n" "$ip" "$gpc0" "$gpc1" "$err_rate/10s" "$status"
-        fi
-    done
-    
-    if [ "$has_data" = false ]; then
-        echo "(No IPs currently tracked - table is empty)"
-    fi
-}
-
-echo
-echo "==================================================================="
-echo "Legend:"
-echo "  - Scan Count 25-39: Low scanner → TARPIT 10s delay"
-echo "  - Scan Count 40-59: Medium scanner → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "  - Scan Count 60-99: High scanner → SILENT-DROP (immediate disconnect)"
-echo "  - Scan Count 100+:  Critical scanner → BLOCKED (429 response)"
-echo "  - Burst (5+ in 10s): → TARPIT 10s (1st), SILENT-DROP (repeat)"
-echo "==================================================================="
-echo "Note: Only counts suspicious scripts/configs, NOT missing images/fonts/CSS"
-echo "Note: IPs are tracked for 1 hour since last activity"
-echo
-echo "To clear a specific IP from the table:"
-echo "  printf '@!${PROCESS_ID} del table web key <IP>\\n' | socat stdio $SOCKET"
-echo
-echo "To clear all entries:"
-echo "  printf '@!${PROCESS_ID} clear table web\\n' | socat stdio $SOCKET"
-echo
-echo "Debug: Worker PID is ${PROCESS_ID}"
-echo

scripts/test-cert-write-safety.py

@@ -27,6 +27,8 @@ These tests pin the invariants:
   * nothing is published that is not a complete, validated cert+key pair;
   * no old certificate file is removed and no lineage deleted until the
     replacement is validated, in place, and actually loaded by HAProxy;
+  * removing ONE domain never unlinks a .pem, or deletes a certbot lineage,
+    that other still-configured domains are being served from;
   * only final .pem files ever exist in the crt directory.
 
 Running
@@ -894,6 +896,184 @@ class TestBundleValidation(CertPublishTestCase):
                          'a private key must never be group/world writable')
 
 
+class TestSharedCertificateSurvivesDomainRemoval(CertPublishTestCase):
+    """Bug 5: DELETE /api/domain unlinked a PEM other live sites were served from.
+
+    `domains.domain` is UNIQUE; `domains.ssl_cert_path` is not, and nothing
+    ever made it so. request_ssl_bundle() issues one SAN certificate and points
+    every included name's row at the same /etc/haproxy/certs/<primary>.pem, so
+    sharing is not an edge case - it is the normal shape of the table. Measured
+    on production the day this was written: 39 of 71 distinct cert paths on one
+    host were referenced by more than one domain row (118 of 150 SSL-enabled
+    rows), and one shared path was
+    /etc/haproxy/certs/threeworldsoneheart.org.pem, referenced by the live
+    apex, its www, and a mail.* alias.
+
+    remove_domain() did, unconditionally:
+
+        os.remove(ssl_cert_path)
+        certbot delete --cert-name <domain>
+
+    Removing the mail.* alias would therefore have deleted the PEM the apex was
+    serving on, and (had the alias been the bundle primary) the lineage behind
+    it - HTTPS down for every other name in the bundle, with no local copy and
+    only a fresh, rate-limited ACME order to recover from.
+
+    These tests assert file-system and certbot-invocation outcomes, not which
+    branch was taken.
+    """
+
+    BUNDLE = '/etc/haproxy/certs'  # documentation only; SSL_CERTS_DIR is stubbed
+
+    def _cert_for(self, primary):
+        """Publish a bundle for `primary` and return its path."""
+        return self.publish_live_bundle(primary)
+
+    def remove(self, domain):
+        return self.client.delete('/api/domain', json={'domain': domain})
+
+    # -- last reference: the cleanup must still happen --------------------
+
+    def test_last_reference_removal_unlinks_the_pem(self):
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_example', ssl_cert_path=cert)
+
+        resp = self.remove('example.com')
+
+        self.assertEqual(200, resp.status_code, resp.data)
+        self.assertFalse(
+            os.path.exists(cert),
+            'nothing else referenced this bundle - it must be cleaned up, or '
+            'the crt directory accumulates certs for domains that are gone')
+
+    def test_last_reference_removal_deletes_the_lineage(self):
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_example', ssl_cert_path=cert)
+
+        self.remove('example.com')
+
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes(),
+            'the last name on a lineage went away - the lineage should go too')
+
+    def test_lineage_deleted_is_the_bundles_not_the_domains_own_name(self):
+        """Removing a SAN member must target the lineage that issued the file."""
+        cert = self._cert_for('example.com')
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('www.example.com')
+
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes(),
+            'the lineage is named after the bundle primary (--cert-name), not '
+            'after whichever SAN happened to be removed last')
+
+    # -- shared reference: nothing may be destroyed -----------------------
+
+    def test_shared_pem_survives_removal_of_one_name(self):
+        cert = self._cert_for('example.com')
+        before = self.read(cert)
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        resp = self.remove('www.example.com')
+
+        self.assertEqual(200, resp.status_code, resp.data)
+        self.assertTrue(
+            os.path.exists(cert),
+            'example.com is still configured and still served from this file')
+        self.assertEqual(before, self.read(cert),
+                         'the surviving bundle must be byte-for-byte intact')
+        self.assertTrue(self.edge_would_start(),
+                        'the edge must still load the crt directory')
+
+    def test_shared_pem_survives_removal_of_the_bundle_primary(self):
+        """The worst shape: the name being removed IS the lineage/file name."""
+        cert = self._cert_for('example.com')
+        before = self.read(cert)
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('example.com')
+
+        self.assertTrue(
+            os.path.exists(cert),
+            'www.example.com is still configured and is served from this exact '
+            'file - removing the apex must not unlink it')
+        self.assertEqual(before, self.read(cert))
+        self.assertTrue(self.edge_would_start())
+
+    def test_shared_lineage_is_not_certbot_deleted(self):
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('example.com')
+
+        self.assertEqual(
+            [], self.certbot_deletes(),
+            'certbot delete destroys archive, live and renewal config for a '
+            'lineage that is still renewing the certificate www.example.com '
+            'is served with')
+
+    def test_production_shape_mail_alias_removal(self):
+        """The exact row that stopped a production cleanup.
+
+        mail.threeworldsoneheart.org carried
+        ssl_cert_path=/etc/haproxy/certs/threeworldsoneheart.org.pem - the live
+        PEM of a different, serving site.
+        """
+        cert = self._cert_for('threeworldsoneheart.org')
+        before = self.read(cert)
+        self.add_domain('threeworldsoneheart.org', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.threeworldsoneheart.org', 'be_www', ssl_cert_path=cert)
+        self.add_domain('mail.threeworldsoneheart.org', 'be_mail', ssl_cert_path=cert)
+
+        self.remove('mail.threeworldsoneheart.org')
+
+        self.assertTrue(os.path.exists(cert),
+                        'two sites are still served from this bundle')
+        self.assertEqual(before, self.read(cert))
+        self.assertEqual([], self.certbot_deletes())
+        self.assertTrue(self.edge_would_start())
+
+    def test_removing_every_name_eventually_cleans_up(self):
+        """The guard defers cleanup, it does not cancel it."""
+        cert = self._cert_for('example.com')
+        self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
+        self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
+
+        self.remove('www.example.com')
+        self.assertTrue(os.path.exists(cert), 'apex still needs it')
+
+        self.remove('example.com')
+        self.assertFalse(os.path.exists(cert),
+                         'the last reference is gone - now it may be removed')
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes())
+
+    def test_unrelated_domains_certificate_is_untouched(self):
+        """Sharing is by exact path; different bundles stay independent."""
+        mine = self._cert_for('example.com')
+        theirs = self._cert_for('other.example')
+        theirs_before = self.read(theirs)
+        self.add_domain('example.com', 'be_mine', ssl_cert_path=mine)
+        self.add_domain('other.example', 'be_theirs', ssl_cert_path=theirs)
+
+        self.remove('example.com')
+
+        self.assertFalse(os.path.exists(mine))
+        self.assertTrue(os.path.exists(theirs),
+                        'a different bundle must not be collateral damage')
+        self.assertEqual(theirs_before, self.read(theirs))
+        self.assertEqual(
+            ['delete --cert-name example.com --non-interactive'],
+            self.certbot_deletes())
+
+
 @unittest.skipIf(TESTING_FOREIGN_TREE,
                  'HAPROXY_MANAGER_DIR points at another tree')
 class TestPublisherApiIsPresent(unittest.TestCase):

scripts/test-runtime-map-contract.py

@@ -0,0 +1,397 @@
+#!/usr/bin/env python3
+"""Contract test: the runtime-map fast path (blocked IPs).
+
+Why this file exists
+--------------------
+`add_ip_to_runtime_map()` and `remove_ip_from_runtime_map()` spent their whole
+existence sending
+
+    add map #0 <ip> 1
+    del map #0 <ip>
+
+to `/tmp/haproxy-cli` and returning True whenever socat exited 0. Neither
+command has ever worked. Two independent defects:
+
+  * **No `@1` prefix.** `/tmp/haproxy-cli` is HAProxy's MASTER CLI socket; map
+    commands are worker commands. The master answers `Unknown command: 'add',
+    but maybe one of the following ones is a better match: ...` -- and **socat
+    still exits 0**, so `result.returncode == 0` was true and the function
+    logged "Added IP x to runtime map".
+  * **`#0` is not a valid map id.** Ids are assigned at config-parse time and
+    move on every config regeneration; on the live edge `blocked_ips.map` is
+    id 37 and `trusted_ips.map` is 10. There is no id 0. Any hardcoded number
+    is wrong -- the map must be referenced by its FILE PATH, which is what
+    haproxy.cfg itself names in `map_ip(/etc/haproxy/blocked_ips.map,0)`.
+
+And a third silence that makes a response-body check alone insufficient:
+`@1 add map #0 <ip> 1` returns an **empty body**, exit 0, and adds nothing
+anywhere -- while `@1 del map #0 <ip>` answers `Unknown map identifier.`. The
+add path can therefore only be trusted after reading the entry back.
+
+IP blocking still worked, because `update_blocked_ips_map()` rewrites
+`/etc/haproxy/blocked_ips.map` and the callers reload HAProxy, which re-reads
+it. The FILE is authoritative; this is the no-reload fast path, and it has
+never once run while reporting that it did.
+
+What it enforces
+----------------
+  1. The command strings actually sent: `@1` prefix first, map referenced by
+     PATH and never by `#<id>`, and the value `1` that
+     `map_ip(...,0) -m int gt 0` requires.
+  2. Every captured rejection is classified as FAILURE (returns False), not
+     success -- including the two that carry no error text at all.
+  3. Success is only reported when the entry reads back in the state asked
+     for. Not the exit status, not an empty reply.
+  4. No source in this repo builds a map command with a `#<id>` reference.
+     Comments may describe the old form; code may not use it.
+
+Runs fully offline: `_cli_send()` is replaced, so no socket, no socat, no
+HAProxy, no network.
+
+Running
+-------
+    python3 scripts/test-runtime-map-contract.py
+"""
+
+import ast
+import io
+import os
+import re
+import sys
+import glob
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+logging.getLogger().setLevel(logging.CRITICAL)
+
+MAP = haproxy_manager.BLOCKED_IPS_MAP_PATH
+IP = '192.0.2.77'
+
+# ---------------------------------------------------------------------------
+# Responses captured VERBATIM from the haproxy-manager container on the live
+# edge (HAProxy 3.0.11, 2026-08-22). socat exited 0 for every single one of
+# them, which is the entire reason none of this is decided on exit status.
+# ---------------------------------------------------------------------------
+
+# What the MASTER socket answers to an unprefixed `add map ...` -- i.e. the
+# reply the old code read as success.
+MASTER_REJECTS_ADD = """\
+Unknown command: 'add', but maybe one of the following ones is a better match:
+  @!<pid>                                 : send a command to the <pid> process
+  @master                                 : send a command to the master process
+  hard-reload                             : achieve a hard-reload (-st) of haproxy
+  reload                                  : achieve a soft-reload (-sf) of haproxy
+  user                                    : lower the level of the current CLI session to user
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+MASTER_REJECTS_DEL = MASTER_REJECTS_ADD.replace("'add'", "'del'")
+
+# `@1 del map #0 <ip>` / `@1 get map /etc/haproxy/nope.map <ip>`.
+UNKNOWN_MAP_IDENTIFIER = 'Unknown map identifier. Please use #<id> or <file>.\n'
+
+# `@1 add map <map> <ip>` with the value omitted (as the old docs showed).
+ADD_MAP_MISSING_VALUE = (
+    "'add map' expects three parameters (map identifier, key and value) or one "
+    "parameter (map identifier) and a payload\n")
+
+# `@1 del map <map> <ip>` for a key the runtime map does not hold.
+KEY_NOT_FOUND = 'Key not found.\n'
+
+# A successful mutation. THIS IS THE WHOLE PROBLEM: it is byte-for-byte what
+# `@1 add map #0 <ip> 1` also returns while adding nothing at all.
+MUTATION_OK = ''
+
+GET_FOUND = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+             'value="1", type="str"\n' % IP)
+GET_NOT_FOUND = 'type=ip, case=sensitive, found=no\n'
+# `@1 get map #0 <ip>` -- "found", but with no value. haproxy.cfg matches with
+# `-m int gt 0`, so an entry like this does NOT block.
+GET_FOUND_NO_VALUE = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
+                      'value=none\n' % IP)
+
+SHOW_MAP_SAMPLE = (
+    '0x7f6e5a788700 101.36.109.130 1\n'
+    '0x7f6e5a788780 101.47.140.218 1\n'
+    '0x7f6e5a1fd500 %s 1\n' % IP)
+
+
+class FakeSocket(object):
+    """Replaces _cli_send(). Records every command, answers from a script.
+
+    `script` is a list of (substring, response) pairs, consulted in order; the
+    first whose substring appears in the command wins. Anything unmatched is
+    an explicit test bug, not a silent default.
+    """
+
+    def __init__(self, script):
+        self.script = script
+        self.sent = []
+
+    def __call__(self, command, socket_path, timeout):
+        self.sent.append(command)
+        for needle, response in self.script:
+            if needle in command:
+                return response
+        raise AssertionError('test script has no response for %r' % command)
+
+
+def with_socket(script):
+    """Install a FakeSocket for the duration of a `with` block."""
+    class _Ctx(object):
+        def __enter__(self):
+            self.fake = FakeSocket(script)
+            self._real = haproxy_manager._cli_send
+            haproxy_manager._cli_send = self.fake
+            return self.fake
+
+        def __exit__(self, *exc):
+            haproxy_manager._cli_send = self._real
+            return False
+    return _Ctx()
+
+
+# Every command the happy path needs, in the order the code issues them.
+HAPPY_ADD = [('get map', GET_FOUND), ('add map', MUTATION_OK)]
+HAPPY_DEL = [('get map', GET_NOT_FOUND), ('del map', MUTATION_OK)]
+
+
+class CommandsAreWellFormed(unittest.TestCase):
+    """Guard 1: the exact bytes on the wire.
+
+    Both original defects are visible here and nowhere else -- a `#0` map
+    reference and a missing `@1` are perfectly ordinary-looking Python.
+    """
+
+    def test_add_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_ADD) as fake:
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 add map %s %s 1' % (MAP, IP))
+
+    def test_del_sends_worker_prefixed_path_referenced_command(self):
+        with with_socket(HAPPY_DEL) as fake:
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+        self.assertEqual(fake.sent[0], '@1 del map %s %s' % (MAP, IP))
+
+    def test_every_command_is_tried_on_the_worker_first(self):
+        """`/tmp/haproxy-cli` is the MASTER socket; bare map commands 404."""
+        for run in (lambda: haproxy_manager.add_ip_to_runtime_map(IP),
+                    lambda: haproxy_manager.remove_ip_from_runtime_map(IP)):
+            with with_socket(HAPPY_ADD + HAPPY_DEL) as fake:
+                run()
+            for command in fake.sent:
+                self.assertTrue(command.startswith('@1 '),
+                                '%r is missing the @1 worker prefix' % command)
+
+    def test_no_command_references_a_map_by_id(self):
+        """Map ids move on every config regeneration. Path, always."""
+        script = HAPPY_ADD + HAPPY_DEL + [('show map', SHOW_MAP_SAMPLE)]
+        with with_socket(script) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+            haproxy_manager.remove_ip_from_runtime_map(IP)
+            haproxy_manager.runtime_map_keys(MAP)
+        for command in fake.sent:
+            self.assertNotRegex(
+                command, r'\bmap\s+#',
+                '%r references a map by id; ids are not stable' % command)
+            self.assertIn(MAP, command,
+                          '%r does not name the map file' % command)
+
+    def test_add_carries_the_value_the_config_matches_on(self):
+        """`map_ip(...,0) -m int gt 0`: a valueless entry does not block."""
+        self.assertEqual(haproxy_manager.BLOCKED_IPS_MAP_VALUE, '1')
+        with with_socket(HAPPY_ADD) as fake:
+            haproxy_manager.add_ip_to_runtime_map(IP)
+        self.assertTrue(fake.sent[0].endswith(' %s 1' % IP),
+                        '%r has no value; HAProxy rejects it' % fake.sent[0])
+
+
+class RejectionIsFailure(unittest.TestCase):
+    """Guard 2+3: nothing may report success unless the map really changed.
+
+    Every response below was returned by the live socket with **exit code 0**.
+    The old code returned True for all of them.
+    """
+
+    def test_master_socket_rejection_of_add_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_master_socket_rejection_of_del_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MASTER_REJECTS_DEL)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_unknown_map_identifier_is_failure(self):
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket([('get map', GET_FOUND), ('del map', UNKNOWN_MAP_IDENTIFIER)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_missing_value_rejection_is_failure(self):
+        """Carries no marker word at all -- caught by 'a mutation says nothing'."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', ADD_MAP_MISSING_VALUE)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_silent_noop_add_is_failure(self):
+        """The `#0` failure mode: accepted, empty reply, nothing added.
+
+        Nothing in the response distinguishes this from success. Only the
+        read-back does -- which is why the read-back is not optional.
+        """
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_add_that_lands_without_a_value_is_failure(self):
+        with with_socket([('get map', GET_FOUND_NO_VALUE), ('add map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+
+    def test_del_that_leaves_the_key_behind_is_failure(self):
+        with with_socket([('get map', GET_FOUND), ('del map', MUTATION_OK)]):
+            self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_key_not_found_on_del_is_the_requested_end_state(self):
+        """Not a failure: the runtime map already lacks the key."""
+        with with_socket([('get map', GET_NOT_FOUND), ('del map', KEY_NOT_FOUND)]):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_verified_success_is_reported_as_success(self):
+        with with_socket(HAPPY_ADD):
+            self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
+        with with_socket(HAPPY_DEL):
+            self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
+
+    def test_a_runtime_failure_never_raises_into_the_request_handler(self):
+        """The map file + reload still enforces the block; degrade, don't 500."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
+            self.assertIs(haproxy_manager.add_ip_to_runtime_map(IP), False)
+
+    def test_mutations_are_checked_for_an_empty_body_not_a_marker_list(self):
+        """_HAPROXY_CLI_ERROR_MARKERS can only know rejections already seen."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            with with_socket([('add map', 'something nobody has ever seen\n')]):
+                haproxy_manager.haproxy_cli('add map %s x 1' % MAP,
+                                            worker=True, expect_empty=True)
+
+    def test_the_new_error_markers_are_recognised(self):
+        for response in (UNKNOWN_MAP_IDENTIFIER, KEY_NOT_FOUND,
+                         MASTER_REJECTS_ADD):
+            self.assertTrue(haproxy_manager._cli_response_is_error(response),
+                            '%r must be classified as an error' % response[:40])
+        for response in (GET_FOUND, GET_NOT_FOUND, SHOW_MAP_SAMPLE):
+            self.assertFalse(haproxy_manager._cli_response_is_error(response),
+                             '%r is data, not an error' % response[:40])
+
+    def test_socat_exit_zero_carries_no_information(self):
+        """FakeSocket never signals failure any other way, and neither did socat."""
+        with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]) as fake:
+            self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
+        self.assertTrue(fake.sent, 'the command was sent and "succeeded" at the '
+                                   'process level; only the body says otherwise')
+
+
+class ReadBack(unittest.TestCase):
+    """The read-back primitives the guarantees above rest on."""
+
+    def test_lookup_reports_found_with_value(self):
+        with with_socket([('get map', GET_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (True, '1'))
+
+    def test_lookup_reports_not_found(self):
+        with with_socket([('get map', GET_NOT_FOUND)]):
+            self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
+                             (False, None))
+
+    def test_lookup_raises_on_a_rejected_reference(self):
+        with with_socket([('get map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_lookup(MAP, IP)
+
+    def test_keys_parses_show_map_output(self):
+        with with_socket([('show map', SHOW_MAP_SAMPLE)]):
+            self.assertEqual(
+                haproxy_manager.runtime_map_keys(MAP),
+                {'101.36.109.130', '101.47.140.218', IP})
+
+    def test_empty_map_is_not_a_rejection(self):
+        with with_socket([('show map', '')]):
+            self.assertEqual(haproxy_manager.runtime_map_keys(MAP), set())
+
+    def test_rejected_show_map_still_raises(self):
+        with with_socket([('show map', UNKNOWN_MAP_IDENTIFIER)]):
+            with self.assertRaises(haproxy_manager.HaproxyCliError):
+                haproxy_manager.runtime_map_keys(MAP)
+
+
+MAP_BY_ID_RE = re.compile(r'\b(?:add|del|clear|show|get)\s+map\s+#')
+
+
+class NoSourceBuildsAMapIdCommand(unittest.TestCase):
+    """Guard 4: `map #<id>` may be described in comments, never executed.
+
+    Scanning string literals rather than raw text is deliberate -- the whole
+    reason this bug is documented at length in the source is so the next reader
+    does not reintroduce it, and a plain grep would fail on those comments.
+    """
+
+    def test_no_python_string_literal_builds_a_map_id_command(self):
+        tree = ast.parse(io.open('haproxy_manager.py', encoding='utf-8').read())
+        offenders = [
+            node.value for node in ast.walk(tree)
+            if isinstance(node, ast.Constant) and isinstance(node.value, str)
+            and MAP_BY_ID_RE.search(node.value)
+            and not (ast.get_docstring(tree) == node.value)
+        ]
+        # Docstrings are string literals too; exclude any literal that is a
+        # docstring of a module/class/function.
+        docstrings = set()
+        for node in ast.walk(tree):
+            if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef,
+                                 ast.AsyncFunctionDef)):
+                doc = ast.get_docstring(node, clean=False)
+                if doc:
+                    docstrings.add(doc)
+        offenders = [o for o in offenders if o not in docstrings]
+        self.assertEqual(offenders, [],
+                         'these string literals build a map command with an '
+                         'unstable #<id> reference')
+
+    def test_no_shell_or_template_code_line_uses_a_map_id(self):
+        targets = (glob.glob('scripts/*.sh') + glob.glob('templates/*.tpl')
+                   + ['Dockerfile'])
+        offenders = []
+        for path in targets:
+            if not os.path.exists(path):
+                continue
+            for lineno, line in enumerate(
+                    io.open(path, encoding='utf-8').read().splitlines(), 1):
+                if line.lstrip().startswith('#'):
+                    continue  # a comment describing the old form is fine
+                if MAP_BY_ID_RE.search(line):
+                    offenders.append('%s:%d: %s' % (path, lineno, line.strip()))
+        self.assertEqual(offenders, [],
+                         'map ids are assigned at config-parse time and move; '
+                         'reference the map file by path')
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

scripts/test-stick-table-contract.py

@@ -0,0 +1,475 @@
+#!/usr/bin/env python3
+"""Contract test: what the stick tables STORE vs what the consumers READ.
+
+Why this file exists
+--------------------
+`/api/security/stats` and `scripts/show-tarpit-ips.sh` spent their whole
+existence reporting "Scan Count", "offense count" and "BLOCKED" figures parsed
+out of `gpc0` and `gpc1`. **No stick table in this repo has ever stored a
+general-purpose counter.** Every one of those numbers was fabricated, and an
+operator was making decisions on them.
+
+Nothing caught it, because each layer failed silently in a different way:
+
+  * `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
+    and the loop `continue`d -- so the endpoint answered `active_threats: 0`
+    with an empty list. "No threats" and "the parser is broken" looked
+    identical.
+  * The command went to the MASTER CLI socket without the `@1` worker prefix.
+    HAProxy answered `Unknown command: 'show', but maybe one of the following
+    ones is a better match: ...` and **socat still exited 0**, so the
+    `returncode != 0` guard never fired. The reported `total_tracked_ips` was
+    the line count of that help text (8) while the real table held 388 entries.
+  * The shell consumers wrote `gpc0=${gpc0:-0}`, so a field that does not exist
+    rendered as a confident zero.
+
+The durable fix is not "parse better" -- it is making the template and its
+consumers unable to drift apart without something going red. That is this file.
+
+What it enforces
+----------------
+  1. `STICK_TABLE_FIELD_CONTRACT` in haproxy_manager.py equals, exactly and in
+     both directions, the `store` clauses in the rendered templates.
+  2. Every shell consumer's `EXPECTED_FIELDS=(...)` array equals the contract.
+  3. A captured sample of REAL `show table` output from the live edge parses to
+     exactly the contract's fields plus the entry metadata -- so the contract
+     describes reality, not just itself.
+  4. The loud-failure behaviour: `read_stick_table()` RAISES on a rejected
+     command, on a non-table response, and on a row missing a contract field.
+     It must never answer zeros. Guard 4 is the one that would have caught the
+     original bug on day one.
+  5. No `store` clause names a general-purpose counter, and no template tracks
+     one -- the state this repo is actually in, asserted rather than assumed.
+
+Assertions about the TEMPLATES go through `rule_lines()`, which strips comments
+before matching. These templates quote their own rules in prose at length; a
+bare `assertIn` over the rendered text passes just as happily against a rule
+that has been commented out. Same lesson, and same helper, as
+scripts/test-wpadmin-gate.py.
+
+Runs fully offline -- no HAProxy, no socket, no network.
+
+Running
+-------
+    python3 scripts/test-stick-table-contract.py
+"""
+
+import os
+import re
+import sys
+import logging
+import tempfile
+import unittest
+
+MODULE_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
+)
+os.chdir(MODULE_DIR)
+sys.path.insert(0, MODULE_DIR)
+
+_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
+_real_file_handler = logging.FileHandler
+logging.FileHandler = (
+    lambda filename, *a, **kw: _real_file_handler(
+        os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
+)
+
+import haproxy_manager  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# A captured sample of REAL output, so this test can assert against reality
+# without a live socket.
+#
+# Provenance: `echo "@1 show table web" | socat stdio /tmp/haproxy-cli` inside
+# the haproxy-manager container on whp01, 2026-08-22, HAProxy 3.0.11. Rows
+# trimmed for length; format byte-for-byte as emitted.
+#
+# Note what is and is NOT here: no gpc0, no gpc1, no gpc(N), no gpc_rate, no
+# glitch_rate. Note also that field windows come back in MILLISECONDS
+# (`conn_rate(10000)`), not the `10s` the template writes -- a consumer that
+# labels the raw number "10s" is off by a factor of 1000.
+# ---------------------------------------------------------------------------
+LIVE_TABLE_SAMPLE = """\
+# table: web, type: ip, size:204800, used:388
+0x7f6e5447e728: key=17.58.57.102 use=0 exp=368842 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e541cf7c8: key=43.173.182.9 use=0 exp=171404 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=0
+0x7f6e5547f528: key=95.129.255.180 use=0 exp=590639 shard=0 conn_rate(10000)=1 conn_cur=0 http_req_rate(10000)=1 http_err_rate(30000)=0
+0x7f6e5447e488: key=5.9.105.254 use=0 exp=556277 shard=0 conn_rate(10000)=0 conn_cur=0 http_req_rate(10000)=0 http_err_rate(30000)=1
+"""
+
+# The verbatim reply the MASTER CLI socket gives to an unprefixed worker
+# command. Captured the same way. socat exits 0 on this -- which is the entire
+# reason haproxy_cli() inspects the body.
+MASTER_SOCKET_REJECTION = """\
+Unknown command: 'show', but maybe one of the following ones is a better match:
+  show cli level                          : display the level of the current CLI session
+  show cli sockets                        : dump list of cli sockets
+  show proc                               : show processes status
+  show startup-logs                       : report logs emitted during HAProxy startup
+  show version                            : show version of the current process
+  help [<command>]                        : list matching or all commands
+  prompt [timed]                          : toggle interactive mode with prompt
+  quit                                    : disconnect
+"""
+
+# Anything matching these in a `store` clause is a general-purpose counter.
+GPC_PATTERN = re.compile(r'\bgpc|glitch')
+
+# Shell consumers that must declare their field expectations as a single
+# EXPECTED_FIELDS array, and the table each one reads.
+SHELL_CONSUMERS = {
+    'scripts/show-edge-ip-rates.sh': 'web',
+}
+
+EXPECTED_FIELDS_RE = re.compile(r'^\s*EXPECTED_FIELDS=\(([^)]*)\)\s*$', re.M)
+
+
+def rule_lines(cfg, needle):
+    """Comment-stripped config lines containing `needle`.
+
+    A line that is entirely a comment is dropped; a line mixing config with a
+    trailing comment is truncated at the first ' #' before matching. Without
+    this, every assertion below would pass against a rule that had been
+    commented out but whose text survived in the surrounding prose -- and these
+    templates quote their own rules in prose constantly. See
+    scripts/test-wpadmin-gate.py, where a mutation audit proved the point.
+    """
+    out = []
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if stripped and not stripped.startswith('#'):
+            code = stripped.split(' #', 1)[0].rstrip()
+            if code and needle in code:
+                out.append(code)
+    return out
+
+
+def render_listener():
+    return haproxy_manager.template_env.get_template('hap_listener.tpl').render(
+        crt_path='/etc/haproxy/certs',
+        suspension_enabled=False,
+        coraza_spoe_backend=None,
+    )
+
+
+def render_security_tables():
+    return haproxy_manager.template_env.get_template('hap_security_tables.tpl').render()
+
+
+def stick_tables_from(cfg):
+    """{table name: (field, ...)} for every stick-table declared in `cfg`.
+
+    A stick-table takes the name of the frontend/backend/listen section that
+    declares it -- that name is what `show table <name>` wants, so the section
+    header is part of the contract, not incidental. Section headers and
+    stick-table lines are both read comment-stripped.
+    """
+    tables = {}
+    section = None
+    for raw in cfg.split('\n'):
+        stripped = raw.strip()
+        if not stripped or stripped.startswith('#'):
+            continue
+        code = stripped.split(' #', 1)[0].rstrip()
+        header = re.match(r'^(frontend|backend|listen)\s+(\S+)', code)
+        if header:
+            section = header.group(2)
+            continue
+        if code.startswith('stick-table'):
+            store = re.search(r'\bstore\s+(\S+)', code)
+            if not store:
+                raise AssertionError(
+                    'stick-table in section %r has no `store` clause: %r'
+                    % (section, code))
+            if section is None:
+                raise AssertionError(
+                    'stick-table declared outside any section: %r' % code)
+            # store is a comma-separated list; each item is name or name(window)
+            fields = tuple(item.split('(')[0]
+                           for item in store.group(1).split(','))
+            tables[section] = fields
+    return tables
+
+
+def all_template_stick_tables():
+    tables = {}
+    for cfg in (render_listener(), render_security_tables()):
+        for name, fields in stick_tables_from(cfg).items():
+            if name in tables:
+                raise AssertionError('stick table %r declared twice' % name)
+            tables[name] = fields
+    return tables
+
+
+class StickTableContract(unittest.TestCase):
+    """Guard 1 + 5: the templates and the Python contract, held together."""
+
+    def setUp(self):
+        self.templates = all_template_stick_tables()
+        self.contract = haproxy_manager.STICK_TABLE_FIELD_CONTRACT
+
+    def test_templates_actually_declare_stick_tables(self):
+        """Guard the guard: an empty parse would make every other check vacuous."""
+        self.assertTrue(self.templates,
+                        'parsed no stick tables out of the templates at all -- '
+                        'stick_tables_from() is broken, not the templates')
+
+    def test_same_table_names(self):
+        self.assertEqual(
+            sorted(self.templates), sorted(self.contract),
+            'STICK_TABLE_FIELD_CONTRACT and the templates disagree on WHICH '
+            'stick tables exist. Add/remove the table in both places.')
+
+    def test_same_fields_per_table(self):
+        for table in sorted(self.templates):
+            with self.subTest(table=table):
+                self.assertEqual(
+                    sorted(self.templates[table]),
+                    sorted(self.contract.get(table, ())),
+                    "stick table %r stores %s but STICK_TABLE_FIELD_CONTRACT "
+                    "claims %s. Whichever is wrong, a consumer is about to read "
+                    "a field that is never populated -- which is the bug this "
+                    "test exists for." % (table,
+                                          list(self.templates[table]),
+                                          list(self.contract.get(table, ()))))
+
+    def test_web_table_is_the_one_the_api_reads(self):
+        self.assertIn('web', self.contract)
+        self.assertIn('web', self.templates)
+
+    def test_no_general_purpose_counters_are_stored(self):
+        """The state of the world today, asserted rather than assumed.
+
+        If a gpc/glitch counter is ever genuinely added to a template, this
+        test is the place to update -- and updating it forces whoever does so
+        to also add the field to STICK_TABLE_FIELD_CONTRACT (test_same_fields_
+        per_table) and to the shell consumers (test_shell_consumers_match_
+        contract). That chain is the point: a counter cannot appear in a
+        consumer without existing in the table, and cannot appear in the table
+        without the consumers being updated.
+        """
+        for table, fields in self.templates.items():
+            for field in fields:
+                self.assertIsNone(
+                    GPC_PATTERN.search(field),
+                    'stick table %r now stores %r. Update this test, '
+                    'STICK_TABLE_FIELD_CONTRACT, and every consumer.'
+                    % (table, field))
+
+    def test_track_sc_counters_have_a_table_each(self):
+        """Every `track-scN ... table X` names a table that really exists.
+
+        A typo here is invisible to `haproxy -c` only in the sense that it is
+        NOT -- but it is invisible to the consumers, which would query a table
+        that is never written.
+        """
+        cfg = render_listener()
+        for line in rule_lines(cfg, 'track-sc'):
+            named = re.search(r'\btable\s+(\S+)', line)
+            if named:
+                self.assertIn(
+                    named.group(1), self.templates,
+                    'track-sc rule references undeclared table %r: %r'
+                    % (named.group(1), line))
+
+    def test_sc_counter_indices_fit_haproxys_limit(self):
+        """sc0/sc1/sc2 are all HAProxy gives us by default.
+
+        `tune.stick-counters` defaults to 3. A `track-sc3` without raising it
+        is a config-time failure, and the templates' own comments assume the
+        limit -- so assert it rather than leaving it as folklore.
+        """
+        cfg = render_listener() + '\n' + render_security_tables()
+        raised = rule_lines(cfg, 'tune.stick-counters')
+        limit = 3
+        if raised:
+            limit = int(re.search(r'(\d+)', raised[-1]).group(1))
+        for line in rule_lines(cfg, 'track-sc'):
+            idx = int(re.search(r'track-sc(\d+)', line).group(1))
+            self.assertLess(
+                idx, limit,
+                'track-sc%d exceeds tune.stick-counters (%d): %r'
+                % (idx, limit, line))
+
+
+class ShellConsumerContract(unittest.TestCase):
+    """Guard 2: the shell consumers cannot drift from the contract."""
+
+    def test_shell_consumers_match_contract(self):
+        for path, table in sorted(SHELL_CONSUMERS.items()):
+            with self.subTest(script=path):
+                full = os.path.join(MODULE_DIR, path)
+                self.assertTrue(os.path.exists(full),
+                                '%s is missing; it is a declared consumer of '
+                                'stick table %r' % (path, table))
+                with open(full) as fh:
+                    src = fh.read()
+                m = EXPECTED_FIELDS_RE.search(src)
+                self.assertIsNotNone(
+                    m, '%s must declare its field expectations once as a '
+                       'single-line `EXPECTED_FIELDS=(a b c)` array so this '
+                       'test can hold it to the template' % path)
+                declared = sorted(m.group(1).split())
+                self.assertEqual(
+                    declared,
+                    sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table]),
+                    '%s reads %s but stick table %r stores %s'
+                    % (path, declared, table,
+                       sorted(haproxy_manager.STICK_TABLE_FIELD_CONTRACT[table])))
+
+    def test_retired_script_no_longer_parses_phantom_counters(self):
+        """show-tarpit-ips.sh may explain gpc0/gpc1; it may not extract them.
+
+        The shim is allowed -- encouraged -- to name the fields in prose so an
+        operator who runs it learns why its numbers went away. What it must not
+        do is go back to pulling values out of them.
+        """
+        path = os.path.join(MODULE_DIR, 'scripts/show-tarpit-ips.sh')
+        if not os.path.exists(path):
+            self.skipTest('show-tarpit-ips.sh has been removed outright')
+        with open(path) as fh:
+            lines = fh.readlines()
+        for raw in lines:
+            stripped = raw.strip()
+            if not stripped or stripped.startswith('#'):
+                continue
+            code = stripped.split(' #', 1)[0]
+            self.assertIsNone(
+                re.search(r"grep -o ['\"]?gpc|gpc[0-9]*=\$|sc_get_gpc|sc-inc-gpc", code),
+                'show-tarpit-ips.sh is extracting a general-purpose counter '
+                'again: %r' % stripped)
+
+
+class LiveSampleParses(unittest.TestCase):
+    """Guard 3: the contract describes real HAProxy output, not just itself."""
+
+    def test_header_parses(self):
+        header, entries = self._read()
+        self.assertEqual(header['name'], 'web')
+        self.assertEqual(header['type'], 'ip')
+        self.assertEqual(header['size'], 204800)
+        self.assertEqual(header['used'], 388)
+        self.assertEqual(len(entries), 4)
+
+    def test_sample_fields_are_exactly_contract_plus_metadata(self):
+        expected = set(haproxy_manager.STICK_TABLE_FIELD_CONTRACT['web'])
+        expected |= set(haproxy_manager.STICK_TABLE_ENTRY_META)
+        _, entries = self._read()
+        for line, fields in entries:
+            self.assertEqual(
+                set(fields), expected,
+                'real `show table web` output carries %s, contract+metadata '
+                'expects %s. Row: %r'
+                % (sorted(fields), sorted(expected), line))
+
+    def test_key_is_the_ip_not_the_allocation_pointer(self):
+        """The original bug read parts[0] -- the `0x...:` pointer -- as the IP."""
+        _, entries = self._read()
+        ips = [f['key']['value'] for _, f in entries]
+        self.assertIn('95.129.255.180', ips)
+        for ip in ips:
+            self.assertFalse(ip.startswith('0x'),
+                             'parsed a memory address as an IP: %r' % ip)
+
+    def test_windows_are_milliseconds(self):
+        """HAProxy reports `conn_rate(10000)` for a `conn_rate(10s)` store.
+
+        Asserted because labelling that raw 10000 as "10s" (or as seconds) is
+        an easy and completely silent way to be wrong by 1000x in the panel.
+        """
+        _, entries = self._read()
+        _, fields = entries[0]
+        self.assertEqual(fields['conn_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_req_rate']['window_ms'], 10000)
+        self.assertEqual(fields['http_err_rate']['window_ms'], 30000)
+        self.assertIsNone(fields['conn_cur']['window_ms'],
+                          'conn_cur is a gauge, not a rate; it has no window')
+
+    def test_values_are_the_real_ones(self):
+        _, entries = self._read()
+        by_ip = {f['key']['value']: f for _, f in entries}
+        self.assertEqual(by_ip['95.129.255.180']['http_req_rate']['value'], '1')
+        self.assertEqual(by_ip['5.9.105.254']['http_err_rate']['value'], '1')
+        self.assertEqual(by_ip['17.58.57.102']['http_req_rate']['value'], '0')
+
+    def _read(self):
+        return _read_table_from(LIVE_TABLE_SAMPLE)
+
+
+class FailsLoudly(unittest.TestCase):
+    """Guard 4: every way this can go wrong must raise, never return zeros.
+
+    This is the guard that would have caught the original bug immediately. Each
+    case below is a real response the old code accepted silently.
+    """
+
+    def test_master_socket_rejection_is_not_data(self):
+        """The exact reply that used to be reported as `total_tracked_ips: 8`."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(MASTER_SOCKET_REJECTION)
+        self.assertIn('not a stick-table dump', str(ctx.exception))
+
+    def test_socat_exit_zero_does_not_mean_success(self):
+        """haproxy_cli() must reject on the BODY, not the exit status.
+
+        socat returns 0 for every response above -- the rejection is only ever
+        visible in the text.
+        """
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error(MASTER_SOCKET_REJECTION))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('No such table\n'))
+        self.assertTrue(
+            haproxy_manager._cli_response_is_error('Permission denied\n'))
+        self.assertFalse(
+            haproxy_manager._cli_response_is_error(LIVE_TABLE_SAMPLE))
+
+    def test_missing_contract_field_raises_and_names_it(self):
+        """A field the table stopped storing must not silently become 0."""
+        degraded = LIVE_TABLE_SAMPLE.replace(' http_err_rate(30000)=0', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(degraded)
+        msg = str(ctx.exception)
+        self.assertIn('http_err_rate', msg,
+                      'the error must name the missing field')
+        self.assertIn('drifted', msg,
+                      'the error must say what actually went wrong')
+
+    def test_empty_response_raises(self):
+        with self.assertRaises(haproxy_manager.HaproxyCliError):
+            _read_table_from('')
+
+    def test_row_without_key_raises(self):
+        broken = LIVE_TABLE_SAMPLE.replace('key=17.58.57.102 ', '')
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            _read_table_from(broken)
+        self.assertIn('no key=', str(ctx.exception))
+
+    def test_unknown_table_raises_before_touching_the_socket(self):
+        """Querying a table with no contract is a programming error, not a 0."""
+        with self.assertRaises(haproxy_manager.HaproxyCliError) as ctx:
+            haproxy_manager.read_stick_table('does_not_exist')
+        self.assertIn('no field contract', str(ctx.exception))
+
+    def test_empty_table_is_not_an_error(self):
+        """A table with zero entries is a legitimate, distinguishable result."""
+        header, entries = _read_table_from(
+            '# table: web, type: ip, size:204800, used:0\n')
+        self.assertEqual(header['used'], 0)
+        self.assertEqual(entries, [])
+
+
+def _read_table_from(response, table='web'):
+    """Run read_stick_table() against a canned response instead of a socket."""
+    real = haproxy_manager.haproxy_cli
+    haproxy_manager.haproxy_cli = lambda cmd, worker=False, timeout=None: response
+    try:
+        return haproxy_manager.read_stick_table(table)
+    finally:
+        haproxy_manager.haproxy_cli = real
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)

templates/hap_listener.tpl

@@ -582,7 +582,9 @@ frontend web
 
     # IP blocking using map file (manual blocks only)
     # Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
-    # Runtime updates: echo "add map #0 IP_ADDRESS 1" | socat stdio /var/run/haproxy.sock
+    # Runtime updates (worker command, map referenced by PATH -- "#<id>" ids
+    # move on every config regeneration and "#0" silently adds nothing):
+    #   echo "@1 add map /etc/haproxy/blocked_ips.map IP_ADDRESS 1" | socat stdio /tmp/haproxy-cli
     # Checks the real client IP (from headers if present, otherwise src)
     # map_ip() converter supports both single IPs and CIDR ranges (e.g., 192.168.1.0/24)
     acl is_blocked_ip var(txn.real_ip),map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0