fix(security-stats): stop reporting counters the stick tables never stored
/api/security/stats and scripts/show-tarpit-ips.sh reported "Scan Count",
"offense count" and BLOCKED/TARPITTED status parsed from gpc0/gpc1. No stick
table in this repo has ever stored a general-purpose counter -- the `web` table
stores conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s), and
the two brute-force tables store http_req_rate(60s). Every one of those figures
was fabricated, and an operator was making decisions on them.
Three independent silences kept it alive:
* `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
and the loop `continue`d -- so the endpoint always answered
`active_threats: 0` with an empty list. Live on whp01 it also reported
parts[0], the `0x...:` allocation pointer, as the source IP.
* The command was sent to /tmp/haproxy-cli WITHOUT the `@1` worker prefix.
That is the MASTER CLI socket, which answers "Unknown command: 'show' ..."
-- and socat still exits 0, so the `returncode != 0` guard never fired.
`total_tracked_ips` was the line count of that help text (8) while the real
table held 388 entries.
* The shell consumers wrote `gpc0=${gpc0:-0}`, rendering a field that does
not exist as a confident zero.
Report what the tables actually store, rather than adding gpc counters to make
the old semantics real. Adding them would mean editing hap_listener.tpl -- the
one change here with a silent-total-outage failure mode -- to rebuild
enforcement history that the edge access log (shipped 2026.08.8, on the host at
/var/log/haproxy.log) already records per request, with status codes,
termination states and request references the stick table could never hold.
* haproxy_manager.py: STICK_TABLE_FIELD_CONTRACT names what each table
stores. haproxy_cli() sends worker commands with `@1`, falls back to the
bare form for a plain stats socket, and inspects the RESPONSE BODY because
socat's exit status is worthless here. parse_stick_table_entry() reads
name=value / name(window_ms)=value pairs by NAME, never by position.
read_stick_table() RAISES -- naming the field -- when a row is missing a
contract field, instead of defaulting it to 0.
* /api/security/stats returns the four real counters with their windows, the
true `used:` count, and no invented threat_level/blocked/offense_count.
Fewer numbers, all of them real.
* scripts/show-edge-ip-rates.sh replaces the fabricated report; the four
expected fields are declared once as EXPECTED_FIELDS and drive the parser.
show-tarpit-ips.sh becomes a shim that explains why its numbers are gone
and points at where tarpit events actually live.
* monitor-attacks.sh loses fourteen fabricated "threat" categories and a
composite threat score, all permanently zero; its access-log section now
says the log is on the host instead of silently printing nothing.
* haproxy_tarpit_config.txt -- the never-shipped design sketch these counters
were copied from -- gets a NOT IMPLEMENTED banner.
* scripts/test-stick-table-contract.py (offline, 21 tests) holds the
templates' `store` clauses, STICK_TABLE_FIELD_CONTRACT and every consumer
to each other, and asserts each loud-failure path against the real captured
responses. Template and consumers can no longer drift apart quietly.
No template is touched, so haproxy.cfg is unchanged.
Verified on whp01: total_tracked_ips now tracks `used:` exactly (511 vs the
table's 511, was 8 vs 388), and per-IP values match `show table web key <ip>`
field for field. haproxy PIDs unmoved, `haproxy -c` warnings unchanged, five
customer sites HTTP 200.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+30
-118
@@ -1,123 +1,35 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Script to display IPs that have been tarpitted by HAProxy 3.0
|
||||
# Uses HAProxy stats socket to query stick-table data
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Usage in Docker container:
|
||||
# docker exec -it haproxy-manager /haproxy/scripts/show-tarpit-ips.sh
|
||||
# DEPRECATED SHIM — kept so existing docs/runbooks/muscle memory keep working.
|
||||
#
|
||||
# This script used to print a "Tarpitted IPs Report" with a "Scan Count" and a
|
||||
# BLOCKED / SILENT-DROP / TARPIT status per IP, all derived from gpc0 and gpc1
|
||||
# stick-table columns. Those columns DO NOT EXIST: the `web` table
|
||||
# (templates/hap_listener.tpl) stores only conn_cur, conn_rate, http_req_rate
|
||||
# and http_err_rate. The old parser defaulted every missing field to 0, so the
|
||||
# whole report was fabricated — every IP showed "Scan Count 0 / Normal"
|
||||
# regardless of what it was actually doing.
|
||||
#
|
||||
# The stick table also keeps NO history, so nothing in it can identify who was
|
||||
# tarpitted. Real enforcement events live in the access log ON THE DOCKER HOST
|
||||
# at /var/log/haproxy.log (it does not exist inside this container):
|
||||
# grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20 # tarpit / deny
|
||||
# grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20 # rate-limit / block
|
||||
#
|
||||
# What IS knowable from the stick table — the current per-IP rates — is printed
|
||||
# by show-edge-ip-rates.sh, which this shim now runs.
|
||||
|
||||
SOCKET="/tmp/haproxy-cli"
|
||||
set -euo pipefail
|
||||
|
||||
# Check if socket exists
|
||||
if [ ! -S "$SOCKET" ]; then
|
||||
echo "Error: HAProxy socket not found at $SOCKET"
|
||||
echo "Make sure HAProxy is running with stats socket enabled"
|
||||
exit 1
|
||||
fi
|
||||
cat >&2 <<'NOTE'
|
||||
NOTE: show-tarpit-ips.sh is deprecated and cannot report tarpits.
|
||||
The HAProxy stick table stores no history and no gpc0/gpc1 counters, so
|
||||
the old "Scan Count"/"BLOCKED" columns were fabricated numbers.
|
||||
Actual tarpit/deny events are in /var/log/haproxy.log ON THE HOST:
|
||||
grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20
|
||||
Running show-edge-ip-rates.sh instead (current rates, real values):
|
||||
|
||||
echo "==================================================================="
|
||||
echo " HAProxy Tarpitted IPs Report "
|
||||
echo "==================================================================="
|
||||
echo
|
||||
echo "Showing IPs tracked in the stick-table with scan detection counters:"
|
||||
echo "(gpc0 = total scan attempts, gpc1 = escalation level)"
|
||||
echo
|
||||
NOTE
|
||||
|
||||
# In HAProxy 3.0, we need to use the proper process prefix
|
||||
# The web frontend table is in the worker process, not master
|
||||
# First check which process has the table
|
||||
# Note: grep for actual worker line, not the header
|
||||
PROCESS_ID=$(echo "show proc" | socat stdio "$SOCKET" 2>/dev/null | grep -E '^[0-9]+.*worker' | awk '{print $1}' | head -1)
|
||||
|
||||
if [ -z "$PROCESS_ID" ]; then
|
||||
echo "Error: Could not find HAProxy worker process"
|
||||
echo "Try: echo 'show proc' | socat stdio $SOCKET"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Show stick-table entries from the web frontend using the worker process
|
||||
# Use printf to avoid bash history expansion issues with !
|
||||
printf "@!%s show table web\n" "${PROCESS_ID}" | socat stdio "$SOCKET" 2>/dev/null | {
|
||||
# Skip the header line
|
||||
read header
|
||||
|
||||
# Check if we got an error or empty response
|
||||
if echo "$header" | grep -q "No such table"; then
|
||||
echo "Error: Table 'web' not found. HAProxy may need to be reloaded."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
has_data=false
|
||||
echo "IP Address | Scan Count | Level | HTTP Err Rate | Status"
|
||||
echo "---------------------|------------|-------|---------------|------------------"
|
||||
|
||||
# Process each line
|
||||
while IFS= read -r line; do
|
||||
# Skip empty lines and comments
|
||||
if [ -z "$line" ] || echo "$line" | grep -q "^#"; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# HAProxy 3.0 format: 0x... key=<ip> use=... exp=... gpc0=... gpc1=... http_err_rate(10s)=...
|
||||
if echo "$line" | grep -q "key="; then
|
||||
has_data=true
|
||||
|
||||
# Extract IP and counters
|
||||
ip=$(echo "$line" | grep -o 'key=[^ ]*' | cut -d'=' -f2)
|
||||
gpc0=$(echo "$line" | grep -o 'gpc0=[0-9]*' | cut -d'=' -f2)
|
||||
gpc1=$(echo "$line" | grep -o 'gpc1=[0-9]*' | cut -d'=' -f2)
|
||||
err_rate=$(echo "$line" | grep -o 'http_err_rate([^)]*=[0-9]*' | grep -o '[0-9]*$')
|
||||
|
||||
# Set defaults if values are empty
|
||||
gpc0=${gpc0:-0}
|
||||
gpc1=${gpc1:-0}
|
||||
err_rate=${err_rate:-0}
|
||||
|
||||
# Determine status based on scan count and escalation
|
||||
status=""
|
||||
if [ "$gpc0" -ge 100 ]; then
|
||||
status="BLOCKED (429)"
|
||||
elif [ "$gpc0" -ge 60 ]; then
|
||||
status="SILENT-DROP"
|
||||
elif [ "$gpc0" -ge 40 ]; then
|
||||
if [ "$gpc1" -ge 2 ]; then
|
||||
status="SILENT-DROP (repeat)"
|
||||
else
|
||||
status="TARPIT 10s"
|
||||
fi
|
||||
elif [ "$gpc0" -ge 25 ]; then
|
||||
status="TARPIT 10s"
|
||||
else
|
||||
status="Normal"
|
||||
fi
|
||||
|
||||
# Format output
|
||||
printf "%-20s | %10s | %5s | %13s | %s\n" "$ip" "$gpc0" "$gpc1" "$err_rate/10s" "$status"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$has_data" = false ]; then
|
||||
echo "(No IPs currently tracked - table is empty)"
|
||||
fi
|
||||
}
|
||||
|
||||
echo
|
||||
echo "==================================================================="
|
||||
echo "Legend:"
|
||||
echo " - Scan Count 25-39: Low scanner → TARPIT 10s delay"
|
||||
echo " - Scan Count 40-59: Medium scanner → TARPIT 10s (1st), SILENT-DROP (repeat)"
|
||||
echo " - Scan Count 60-99: High scanner → SILENT-DROP (immediate disconnect)"
|
||||
echo " - Scan Count 100+: Critical scanner → BLOCKED (429 response)"
|
||||
echo " - Burst (5+ in 10s): → TARPIT 10s (1st), SILENT-DROP (repeat)"
|
||||
echo "==================================================================="
|
||||
echo "Note: Only counts suspicious scripts/configs, NOT missing images/fonts/CSS"
|
||||
echo "Note: IPs are tracked for 1 hour since last activity"
|
||||
echo
|
||||
echo "To clear a specific IP from the table:"
|
||||
echo " printf '@!${PROCESS_ID} del table web key <IP>\\n' | socat stdio $SOCKET"
|
||||
echo
|
||||
echo "To clear all entries:"
|
||||
echo " printf '@!${PROCESS_ID} clear table web\\n' | socat stdio $SOCKET"
|
||||
echo
|
||||
echo "Debug: Worker PID is ${PROCESS_ID}"
|
||||
echo
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
exec "$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
|
||||
|
||||
Reference in New Issue
Block a user