fix(logging): stop silently discarding every access log line
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m46s
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m46s
haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.
The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.
Changes:
* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
/var/log/messages or the Graylog forwarder, bound to the bridge IP rather
than 0.0.0.0.
* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
standalone/home deployments on a different bridge subnet can retarget it.
* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
not sufficient for incident response -- it omits %ID entirely (verified
against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
for CF-fronted sites. The new format keeps the first 16 fields byte-identical
to the httplog default (so existing parsers still work) and appends
cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
Adds a User-Agent capture in slot 1 to feed it.
* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
It does not, which made the documented support-correlation workflow
(X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
supported when it could never have worked.
Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".
UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.
Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:
<150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
[22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+41
-11
@@ -2,20 +2,46 @@
|
||||
# Global settings
|
||||
#---------------------------------------------------------------------
|
||||
global
|
||||
# to have these messages end up in /var/log/haproxy.log you will
|
||||
# need to:
|
||||
# ACCESS LOG DESTINATION.
|
||||
#
|
||||
# 1) configure syslog to accept network log events. This is done
|
||||
# by adding the '-r' option to the SYSLOGD_OPTIONS in
|
||||
# /etc/sysconfig/syslog
|
||||
# This used to be `log 127.0.0.1 local2`, which was a silent black hole:
|
||||
# 127.0.0.1 is the CONTAINER's own loopback, nothing has ever listened on
|
||||
# udp/514 in the container netns, and there is no /dev/log in the image.
|
||||
# Every access log line -- ~1.5M/day across the whole edge -- was written
|
||||
# to a socket with no receiver and dropped. Nothing errored, nothing
|
||||
# warned, and `haproxy -c` was perfectly happy. The cost only shows up
|
||||
# during an incident: per-IP 429s, tarpits, wp-admin gate redirects,
|
||||
# WAF 403s and `silent-drop`s left no record anywhere, so the edge could
|
||||
# not be asked what it had rejected. Only aggregate stick-table counters
|
||||
# survived.
|
||||
#
|
||||
# 2) configure local2 events to go to the /var/log/haproxy.log
|
||||
# file. A line like the following can be added to
|
||||
# /etc/sysconfig/syslog
|
||||
# Now points at the DOCKER BRIDGE GATEWAY, where the host's rsyslog has an
|
||||
# imudp listener bound (installed idempotently by WHP's
|
||||
# setup-haproxy-logrotate.sh, which also writes the logrotate stanza).
|
||||
# The host writes local2 to /var/log/haproxy.log and stops it there, so it
|
||||
# does not also flood /var/log/messages or the Graylog forwarder.
|
||||
#
|
||||
# local2.* /var/log/haproxy.log
|
||||
# WHY NOT `log stdout format raw local0`: it is INCOMPATIBLE with the
|
||||
# `daemon` keyword below, and incompatible SILENTLY. Verified on the
|
||||
# pinned 3.0.11 binary: with `daemon` set, a `log stdout` config serves
|
||||
# traffic normally and emits ZERO log lines, and `haproxy -c` returns 0
|
||||
# with no error and no warning -- so the CI config gate
|
||||
# (scripts/validate-rendered-config.py) cannot catch it either. Making it
|
||||
# work means dropping `daemon` / adding -db so haproxy stays in the
|
||||
# foreground, which in turn breaks the three synchronous
|
||||
# `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites in
|
||||
# haproxy_manager.py (they would block until the 180s timeout and then be
|
||||
# killed). That is a change to the exact code path whose failure mode is
|
||||
# "container Up, ports 80/443 never bound, every site down, /health still
|
||||
# 200". Not worth it for a logging change.
|
||||
#
|
||||
log 127.0.0.1 local2
|
||||
# UDP means a dead listener degrades to dropped log lines, never to a
|
||||
# stalled or failing request path -- the correct failure direction for an
|
||||
# edge fronting ~60 customer sites.
|
||||
#
|
||||
# len 2048 accommodates the enriched log-format in hap_listener.tpl
|
||||
# (URL + User-Agent + UUID); the default 1024 would truncate long ones.
|
||||
log {{ syslog_target }} len 2048 format rfc5424 local2 info
|
||||
|
||||
chroot /var/lib/haproxy
|
||||
pidfile /var/run/haproxy.pid
|
||||
@@ -122,7 +148,11 @@ defaults
|
||||
maxconn 3000
|
||||
|
||||
# Per-request unique reference, used:
|
||||
# - in the log line (httplog includes %ID)
|
||||
# - in the access log line, as the `id=` field of the custom log-format
|
||||
# in hap_listener.tpl. NOTE: `option httplog` does NOT include %ID
|
||||
# (verified against haproxy 3.0.11) -- this comment used to claim it
|
||||
# did, which made the support workflow below look supported when it
|
||||
# was not. The explicit log-format is what actually carries it.
|
||||
# - echoed to clients in the X-Request-Reference response header on
|
||||
# WAF blocks so a customer can quote it when opening a support ticket
|
||||
# - embedded in /etc/haproxy/errors/403-waf.html so a blocked visitor
|
||||
|
||||
Reference in New Issue
Block a user