fix(security-stats): stop reporting counters the stick tables never stored
/api/security/stats and scripts/show-tarpit-ips.sh reported "Scan Count",
"offense count" and BLOCKED/TARPITTED status parsed from gpc0/gpc1. No stick
table in this repo has ever stored a general-purpose counter -- the `web` table
stores conn_cur, conn_rate(10s), http_req_rate(10s), http_err_rate(30s), and
the two brute-force tables store http_req_rate(60s). Every one of those figures
was fabricated, and an operator was making decisions on them.
Three independent silences kept it alive:
* `int(parts[3])` on a positional split hit `exp=368842`, raised ValueError,
and the loop `continue`d -- so the endpoint always answered
`active_threats: 0` with an empty list. Live on whp01 it also reported
parts[0], the `0x...:` allocation pointer, as the source IP.
* The command was sent to /tmp/haproxy-cli WITHOUT the `@1` worker prefix.
That is the MASTER CLI socket, which answers "Unknown command: 'show' ..."
-- and socat still exits 0, so the `returncode != 0` guard never fired.
`total_tracked_ips` was the line count of that help text (8) while the real
table held 388 entries.
* The shell consumers wrote `gpc0=${gpc0:-0}`, rendering a field that does
not exist as a confident zero.
Report what the tables actually store, rather than adding gpc counters to make
the old semantics real. Adding them would mean editing hap_listener.tpl -- the
one change here with a silent-total-outage failure mode -- to rebuild
enforcement history that the edge access log (shipped 2026.08.8, on the host at
/var/log/haproxy.log) already records per request, with status codes,
termination states and request references the stick table could never hold.
* haproxy_manager.py: STICK_TABLE_FIELD_CONTRACT names what each table
stores. haproxy_cli() sends worker commands with `@1`, falls back to the
bare form for a plain stats socket, and inspects the RESPONSE BODY because
socat's exit status is worthless here. parse_stick_table_entry() reads
name=value / name(window_ms)=value pairs by NAME, never by position.
read_stick_table() RAISES -- naming the field -- when a row is missing a
contract field, instead of defaulting it to 0.
* /api/security/stats returns the four real counters with their windows, the
true `used:` count, and no invented threat_level/blocked/offense_count.
Fewer numbers, all of them real.
* scripts/show-edge-ip-rates.sh replaces the fabricated report; the four
expected fields are declared once as EXPECTED_FIELDS and drive the parser.
show-tarpit-ips.sh becomes a shim that explains why its numbers are gone
and points at where tarpit events actually live.
* monitor-attacks.sh loses fourteen fabricated "threat" categories and a
composite threat score, all permanently zero; its access-log section now
says the log is on the host instead of silently printing nothing.
* haproxy_tarpit_config.txt -- the never-shipped design sketch these counters
were copied from -- gets a NOT IMPLEMENTED banner.
* scripts/test-stick-table-contract.py (offline, 21 tests) holds the
templates' `store` clauses, STICK_TABLE_FIELD_CONTRACT and every consumer
to each other, and asserts each loud-failure path against the real captured
responses. Template and consumers can no longer drift apart quietly.
No template is touched, so haproxy.cfg is unchanged.
Verified on whp01: total_tracked_ips now tracks `used:` exactly (511 vs the
table's 511, was 8 vs 388), and per-IP values match `show table web key <ip>`
field for field. haproxy PIDs unmoved, `haproxy -c` warnings unchanged, five
customer sites HTTP 200.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+130
-118
@@ -1,136 +1,148 @@
|
||||
#!/bin/bash
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# monitor-attacks.sh — HAProxy edge activity monitor.
|
||||
#
|
||||
# Two sections, both fed from real data:
|
||||
# 1. Current per-IP rates, from the `web` stick table (delegated to
|
||||
# show-edge-ip-rates.sh — there is exactly one stick-table parser).
|
||||
# 2. Recent enforcement events, from the HAProxy access log.
|
||||
#
|
||||
# HISTORY / WHY THIS IS SHORTER THAN IT USED TO BE
|
||||
# The previous version printed a "Threat Intelligence Dashboard" with
|
||||
# fourteen categories (auth_fail, authz_fail, scanner, sql_inj, traversal,
|
||||
# wp_brute, admin_scan, shell_att, repeat_off, manual_bl, auto_bl,
|
||||
# glitch_rate, ...) and a composite "threat score", all parsed out of
|
||||
# gpc(0), gpc(1), gpc(3), gpc(12), gpc(13) and glitch_rate(300s). NONE of
|
||||
# those fields exist: the `web` table stores only conn_cur, conn_rate,
|
||||
# http_req_rate and http_err_rate. Every category was permanently 0 and the
|
||||
# whole dashboard printed nothing while implying it was watching. All of it
|
||||
# has been deleted rather than "fixed" — there was no data source to fix it
|
||||
# against.
|
||||
#
|
||||
# Usage: monitor-attacks.sh [live]
|
||||
# Env: LOG_FILE=<path> access log to read (default /var/log/haproxy.log)
|
||||
# LOG_LINES=<n> how many trailing log lines to scan (default 500)
|
||||
|
||||
# Real-time attack monitoring for HAProxy
|
||||
# Shows blocked requests and suspicious activity
|
||||
set -uo pipefail
|
||||
|
||||
LOG_FILE="/var/log/haproxy.log"
|
||||
SOCKET="/tmp/haproxy-cli"
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
LOG_FILE="${LOG_FILE:-/var/log/haproxy.log}"
|
||||
LOG_LINES="${LOG_LINES:-500}"
|
||||
|
||||
echo "==================================================="
|
||||
echo "HAProxy Security Monitor - Real-time Attack Detection"
|
||||
echo "==================================================="
|
||||
echo ""
|
||||
|
||||
# Function to show current threats with HAProxy 3.0.11 metrics
|
||||
show_threats() {
|
||||
echo "HAProxy 3.0.11 Threat Intelligence Dashboard:"
|
||||
echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
|
||||
awk 'NR>1 {
|
||||
# Parse the stick table output for array-based GPC values
|
||||
ip = $1
|
||||
# Look for GPC array values in the data
|
||||
auth_fail = 0
|
||||
authz_fail = 0
|
||||
rate_viol = 0
|
||||
scanner = 0
|
||||
sql_inj = 0
|
||||
traversal = 0
|
||||
wp_brute = 0
|
||||
admin_scan = 0
|
||||
shell_att = 0
|
||||
repeat_off = 0
|
||||
manual_bl = 0
|
||||
auto_bl = 0
|
||||
glitch_rate = 0
|
||||
threat_score = 0
|
||||
|
||||
# Extract relevant metrics (simplified parsing)
|
||||
if ($0 ~ /gpc\(0\)=([0-9]+)/) {
|
||||
match($0, /gpc\(0\)=([0-9]+)/, arr); auth_fail = arr[1]
|
||||
}
|
||||
if ($0 ~ /gpc\(1\)=([0-9]+)/) {
|
||||
match($0, /gpc\(1\)=([0-9]+)/, arr); authz_fail = arr[1]
|
||||
}
|
||||
if ($0 ~ /gpc\(3\)=([0-9]+)/) {
|
||||
match($0, /gpc\(3\)=([0-9]+)/, arr); scanner = arr[1]
|
||||
}
|
||||
if ($0 ~ /gpc\(12\)=([0-9]+)/) {
|
||||
match($0, /gpc\(12\)=([0-9]+)/, arr); repeat_off = arr[1]
|
||||
}
|
||||
if ($0 ~ /gpc\(13\)=([0-9]+)/) {
|
||||
match($0, /gpc\(13\)=([0-9]+)/, arr); manual_bl = arr[1]
|
||||
}
|
||||
if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
|
||||
match($0, /glitch_rate\(300s\)=([0-9]+)/, arr); glitch_rate = arr[1]
|
||||
}
|
||||
|
||||
# Calculate composite threat score (simplified)
|
||||
threat_score = auth_fail*10 + authz_fail*8 + scanner*12 + repeat_off*25 + manual_bl*100
|
||||
|
||||
# Only show IPs with significant threat indicators
|
||||
if (auth_fail > 0 || authz_fail > 0 || scanner > 0 || repeat_off > 0 || manual_bl > 0 || glitch_rate > 0) {
|
||||
threat_level = "LOW"
|
||||
if (threat_score >= 100) threat_level = "CRITICAL"
|
||||
else if (threat_score >= 50) threat_level = "HIGH"
|
||||
else if (threat_score >= 20) threat_level = "MEDIUM"
|
||||
|
||||
printf "%-15s [%8s] Score:%-3d Auth:%-2d Authz:%-2d Scanner:%-1d Repeat:%-1d Glitch:%-2d\n",
|
||||
ip, threat_level, threat_score, auth_fail, authz_fail, scanner, repeat_off, glitch_rate
|
||||
}
|
||||
}' | head -15
|
||||
|
||||
echo ""
|
||||
echo "Top HTTP/2 Protocol Violators:"
|
||||
echo "show table web" | socat stdio "$SOCKET" 2>/dev/null | \
|
||||
awk 'NR>1 && $0 ~ /glitch/ {
|
||||
if ($0 ~ /glitch_rate\(300s\)=([0-9]+)/) {
|
||||
match($0, /glitch_rate\(300s\)=([0-9]+)/, arr)
|
||||
if (arr[1] > 2) {
|
||||
printf "%-15s glitch_rate:%-3s\n", $1, arr[1]
|
||||
}
|
||||
}
|
||||
}' | head -5
|
||||
echo "---------------------------------------------------"
|
||||
# --- Section 1: current rates (real stick-table data) -----------------------
|
||||
show_rates() {
|
||||
"$SCRIPT_DIR/show-edge-ip-rates.sh" "$@"
|
||||
}
|
||||
|
||||
# Function to show recent blocks
|
||||
# --- Section 2: recent enforcement events (real access-log data) ------------
|
||||
show_recent_blocks() {
|
||||
echo "Recent Blocked Requests:"
|
||||
tail -100 "$LOG_FILE" 2>/dev/null | \
|
||||
grep -E "(bot_scanner|scan_admin|scan_shells|sql_injection|directory_traversal|rate_abuse|tarpit|denied|403)" | \
|
||||
tail -10 | \
|
||||
awk '{
|
||||
if (match($0, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+/)) {
|
||||
ip = substr($0, RSTART, RLENGTH)
|
||||
gsub(/:.*/, "", ip)
|
||||
reason = ""
|
||||
if ($0 ~ /bot_scanner/) reason = "BOT_SCANNER"
|
||||
else if ($0 ~ /scan_admin/) reason = "ADMIN_SCAN"
|
||||
else if ($0 ~ /scan_shells/) reason = "SHELL_SCAN"
|
||||
else if ($0 ~ /sql_injection/) reason = "SQL_INJECTION"
|
||||
else if ($0 ~ /directory_traversal/) reason = "DIR_TRAVERSAL"
|
||||
else if ($0 ~ /rate_abuse/) reason = "RATE_ABUSE"
|
||||
else if ($0 ~ /tarpit/) reason = "TARPIT"
|
||||
else if ($0 ~ /denied/) reason = "DENIED"
|
||||
else if ($0 ~ /403/) reason = "BLOCKED"
|
||||
printf "[%s] %-15s %s\n", strftime("%H:%M:%S"), ip, reason
|
||||
echo "Recent enforcement events (last $LOG_LINES log lines):"
|
||||
echo
|
||||
|
||||
if [ ! -f "$LOG_FILE" ] || [ ! -r "$LOG_FILE" ]; then
|
||||
cat <<MSG
|
||||
Access log not readable at: $LOG_FILE
|
||||
|
||||
This is expected INSIDE the haproxy-manager container: HAProxy logs to
|
||||
syslog on the DOCKER HOST, and the file lives on the host, not in here.
|
||||
Read it from the host instead:
|
||||
|
||||
grep -aE ' (PT|PR)--' /var/log/haproxy.log | tail -20 # tarpit / deny
|
||||
grep -aE ' (429|403) ' /var/log/haproxy.log | tail -20 # rate-limited / blocked
|
||||
grep -a 'cip=<IP>' /var/log/haproxy.log | tail -50 # one client IP
|
||||
grep -a 'id=<uuid>' /var/log/haproxy.log # one request reference
|
||||
# (the UUID on the block page)
|
||||
tail -f /var/log/haproxy.log | grep -aE ' (PT|PR)--' # live
|
||||
|
||||
Or point this script at a copy: LOG_FILE=/path/to/haproxy.log $0
|
||||
MSG
|
||||
echo
|
||||
return 0
|
||||
fi
|
||||
|
||||
printf "%-15s %-16s %-4s %-5s %-28s %s\n" "TIME" "CLIENT IP" "CODE" "TERM" "HOST" "REQUEST / REQUEST-ID"
|
||||
printf "%s\n" "-----------------------------------------------------------------------------------------------------"
|
||||
|
||||
local found
|
||||
found=$(tail -n "$LOG_LINES" "$LOG_FILE" 2>/dev/null | awk '
|
||||
{
|
||||
status = ""; term = ""; cip = ""; host = ""; id = ""; ts = ""; req = ""
|
||||
|
||||
# %tr is bracketed: [22/Aug/2026:10:11:12.345] -> keep HH:MM:SS
|
||||
# No {n} interval expressions here: not every awk in a slim Debian
|
||||
# image supports them. Spelled out instead.
|
||||
if (match($0, /\[[0-9][0-9]\/[A-Za-z][A-Za-z][A-Za-z]\/[0-9][0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/)) {
|
||||
ts = substr($0, RSTART + 13, 8)
|
||||
}
|
||||
|
||||
# Anchor on the %TR/%Tw/%Tc/%Tr/%Ta timers block: %ST follows it, and
|
||||
# the termination state (%tsc) is 4 fields further on (%B %CC %CS %tsc).
|
||||
for (i = 1; i <= NF; i++) {
|
||||
if ($i ~ /^[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+\/[+-]?[0-9]+$/) {
|
||||
status = $(i + 1)
|
||||
term = $(i + 5)
|
||||
break
|
||||
}
|
||||
}'
|
||||
echo ""
|
||||
}
|
||||
|
||||
# Only enforcement outcomes: tarpit (PT--), deny (PR--), 403, 429.
|
||||
if (!(term ~ /^PT/ || term ~ /^PR/ || status == "403" || status == "429")) next
|
||||
|
||||
for (i = 1; i <= NF; i++) {
|
||||
if (substr($i, 1, 4) == "cip=") cip = substr($i, 5)
|
||||
if (substr($i, 1, 5) == "host=") host = substr($i, 6)
|
||||
if (substr($i, 1, 3) == "id=") id = substr($i, 4)
|
||||
}
|
||||
|
||||
if (match($0, /"[A-Z]+ [^"]*"/)) {
|
||||
req = substr($0, RSTART + 1, RLENGTH - 2)
|
||||
if (length(req) > 42) req = substr(req, 1, 41) "..."
|
||||
}
|
||||
|
||||
if (cip == "") cip = "-"
|
||||
if (host == "") host = "-"
|
||||
if (term == "") term = "-"
|
||||
if (status == "") status = "-"
|
||||
printf "%-15s %-16s %-4s %-5s %-28s %s\n", ts, cip, status, term, host, req
|
||||
if (id != "" && id != "-") printf "%-15s %s\n", "", " id=" id
|
||||
n++
|
||||
}
|
||||
END { if (n == 0) print "(no tarpit/deny/403/429 events in the scanned window)" }
|
||||
')
|
||||
printf '%s\n' "$found"
|
||||
echo
|
||||
echo "TERM = HAProxy termination state: PT-- tarpit, PR-- deny (incl. WAF/rate limit)."
|
||||
echo "id= = request reference; it is printed on the block page and is how a"
|
||||
echo " customer support ticket correlates to an exact request here."
|
||||
}
|
||||
|
||||
# Monitor mode selection
|
||||
if [ "$1" == "live" ]; then
|
||||
echo "Live monitoring mode - Press Ctrl+C to exit"
|
||||
echo ""
|
||||
banner() {
|
||||
echo "==================================================="
|
||||
echo "HAProxy Edge Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
|
||||
echo "==================================================="
|
||||
echo
|
||||
}
|
||||
|
||||
if [ "${1:-}" = "live" ]; then
|
||||
echo "Live monitoring mode - Press Ctrl+C to exit"
|
||||
while true; do
|
||||
clear
|
||||
echo "==================================================="
|
||||
echo "HAProxy Security Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
|
||||
echo "==================================================="
|
||||
echo ""
|
||||
show_threats
|
||||
echo ""
|
||||
banner
|
||||
show_rates || true
|
||||
echo
|
||||
show_recent_blocks
|
||||
sleep 5
|
||||
done
|
||||
else
|
||||
# Single run mode
|
||||
show_threats
|
||||
echo ""
|
||||
banner
|
||||
rc=0
|
||||
show_rates || rc=$?
|
||||
echo
|
||||
show_recent_blocks
|
||||
echo ""
|
||||
echo "Tip: Run with 'live' parameter for continuous monitoring"
|
||||
echo
|
||||
echo "Tip: run with 'live' for a refreshing view."
|
||||
echo "Usage: $0 [live]"
|
||||
fi
|
||||
# Propagate a stick-table read failure: if the rates section could not be
|
||||
# produced, this run did NOT report what it claims to report.
|
||||
exit "$rc"
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user