fix(security-stats): stop reporting counters the stick tables never stored #7
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: cloud-hosting-platform/haproxy-manager-base#7
Reference in New Issue
Block a user
The problem
/api/security/statsandscripts/show-tarpit-ips.shreported Scan Count, offense count and BLOCKED / TARPITTED status parsed fromgpc0/gpc1. No stick table in this repo has ever stored a general-purpose counter.Live on whp01:
Three independent silences kept it alive:
int(parts[3])on a positional split hitexp=368842, raisedValueError, and the loopcontinued — so the endpoint always answeredactive_threats: 0with an empty list. It also readparts[0], the0x...:allocation pointer, as the source IP./tmp/haproxy-cliwithout the@1worker prefix. That is the MASTER CLI socket; it answersUnknown command: 'show', ...— and socat still exits 0, soreturncode != 0never fired.total_tracked_ipswas the line count of that help text (8) while the real table held 388.gpc0=${gpc0:-0}, rendering a nonexistent field as a confident zero. The in-container report listed 518 IPs, every oneScan Count 0 / Normal.The fix, and why this one
The alternative was adding gpc counters to
hap_listener.tplso the old semantics become real. Rejected: it is the one change here with a silent-total-outage failure mode (bad haproxy.cfg = container stays Up, ports never bound,/healthstill 200), and it would rebuild enforcement history the edge access log already records — shipped 2026.08.8, on the host at/var/log/haproxy.log, with status codes, termination states (PT--tarpit,PR--deny) and per-request UUIDs the stick table could never hold.sc0/sc1/sc2are also all in use withtune.stick-countersdefaulting to 3.No template is touched.
haproxy.cfgis unchanged.STICK_TABLE_FIELD_CONTRACTnames what each table stores — one source of truth.haproxy_cli()sends worker commands with@1, falls back to the bare form for a plain stats socket, and inspects the response body because socat's exit status is worthless here.parse_stick_table_entry()readsname=value/name(window_ms)=valuepairs by name, never by position.read_stick_table()raises, naming the field, when a row is missing a contract field — never defaults it to0./api/security/statsreturns the four real counters with their windows and the trueused:count. No inventedthreat_level/blocked/offense_count. Fewer numbers, all of them real.scripts/show-edge-ip-rates.shreplaces the fabricated report;show-tarpit-ips.shbecomes a shim that explains why its numbers are gone.monitor-attacks.shloses fourteen fabricated categories and a composite threat score, all permanently zero.haproxy_tarpit_config.txt— the never-shipped sketch these counters were copied from — gets a NOT IMPLEMENTED banner.The durable win
scripts/test-stick-table-contract.py(offline, 21 tests) holds the templates'storeclauses,STICK_TABLE_FIELD_CONTRACT, and every consumer's declared field list to each other, and asserts each loud-failure path against real captured responses. Template and consumers can no longer drift apart quietly. It caught a real bug during review: the new shell parser exited 141 with no output at all on a 550-row table —printf | sed -n '/./{p;q;}'underset -o pipefailtakes SIGPIPE. Small samples passed; only real-size input showed it.Verification on whp01 (live-patched via
docker cp+SIGHUPto gunicorn; container NOT recreated)total_tracked_ips8(lines of an error message)511— exactly the table'sused:511active_threats/ sources0, empty listScan Count 0 / NormalAPI output matches
@1 show table web key <ip>field for field. HAProxy PIDs 29/30 unmoved,haproxy -cwarnings unchanged, ports 80/443/8000/8080/8404 bound, five customer sites HTTP 200.The top source the new endpoint surfaced,
213.111.144.104, is a live wp-login brute-forcer the access log shows being tarpitted (429 PT--) and hitting the wp2shell block (403 PR--,ua=wp2shell). The old panel classified itScan Count 0 / Normal.Deploy note
The live fix on whp01 is a container-local patch and will be lost on the next
haproxy-managerrecreate until this merges and CI builds the image (VERSION bumped to2026.08.9).Separate bug found, NOT fixed here
add_ip_to_runtime_map()/remove_ip_from_runtime_map()have the same root cause and are also silently inert:add map #0 <ip> 1is sent unprefixed to the master socket (rejected), and#0is not a valid map id here anyway —show mapreports blocked_ips.map as id 37. Both log success and returnTrue. Enforcement is unaffected because the map file is rewritten and haproxy reloaded, but the runtime fast path has never worked. Left alone deliberately: it touches IP blocking on a prod edge, which is a different blast radius from a statistics endpoint.🤖 Generated with Claude Code
/api/security/stats and scripts/show-tarpit-ips.sh reported "Scan Count", "offense count" and BLOCKED/TARPITTED status parsed from gpc0/gpc1. No stick table in this repo has ever stored a general-purpose counter -- the `web` table stores conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s), and the two brute-force tables store http_req_rate(60s). Every one of those figures was fabricated, and an operator was making decisions on them. Three independent silences kept it alive: * `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError, and the loop `continue`d -- so the endpoint always answered `active_threats: 0` with an empty list. Live on whp01 it also reported parts[0], the `0x...:` allocation pointer, as the source IP. * The command was sent to /tmp/haproxy-cli WITHOUT the `@1` worker prefix. That is the MASTER CLI socket, which answers "Unknown command: 'show' ..." -- and socat still exits 0, so the `returncode != 0` guard never fired. `total_tracked_ips` was the line count of that help text (8) while the real table held 388 entries. * The shell consumers wrote `gpc0=${gpc0:-0}`, rendering a field that does not exist as a confident zero. Report what the tables actually store, rather than adding gpc counters to make the old semantics real. Adding them would mean editing hap_listener.tpl -- the one change here with a silent-total-outage failure mode -- to rebuild enforcement history that the edge access log (shipped 2026.08.8, on the host at /var/log/haproxy.log) already records per request, with status codes, termination states and request references the stick table could never hold. * haproxy_manager.py: STICK_TABLE_FIELD_CONTRACT names what each table stores. haproxy_cli() sends worker commands with `@1`, falls back to the bare form for a plain stats socket, and inspects the RESPONSE BODY because socat's exit status is worthless here. parse_stick_table_entry() reads name=value / name(window_ms)=value pairs by NAME, never by position. read_stick_table() RAISES -- naming the field -- when a row is missing a contract field, instead of defaulting it to 0. * /api/security/stats returns the four real counters with their windows, the true `used:` count, and no invented threat_level/blocked/offense_count. Fewer numbers, all of them real. * scripts/show-edge-ip-rates.sh replaces the fabricated report; the four expected fields are declared once as EXPECTED_FIELDS and drive the parser. show-tarpit-ips.sh becomes a shim that explains why its numbers are gone and points at where tarpit events actually live. * monitor-attacks.sh loses fourteen fabricated "threat" categories and a composite threat score, all permanently zero; its access-log section now says the log is on the host instead of silently printing nothing. * haproxy_tarpit_config.txt -- the never-shipped design sketch these counters were copied from -- gets a NOT IMPLEMENTED banner. * scripts/test-stick-table-contract.py (offline, 21 tests) holds the templates' `store` clauses, STICK_TABLE_FIELD_CONTRACT and every consumer to each other, and asserts each loud-failure path against the real captured responses. Template and consumers can no longer drift apart quietly. No template is touched, so haproxy.cfg is unchanged. Verified on whp01: total_tracked_ips now tracks `used:` exactly (511 vs the table's 511, was 8 vs 388), and per-IP values match `show table web key <ip>` field for field. haproxy PIDs unmoved, `haproxy -c` warnings unchanged, five customer sites HTTP 200. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>