fix(logging): stop silently discarding every access log line
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m46s

haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.

The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.

Changes:

* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
  the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
  WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
  host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
  /var/log/messages or the Graylog forwarder, bound to the bridge IP rather
  than 0.0.0.0.

* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
  standalone/home deployments on a different bridge subnet can retarget it.

* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
  not sufficient for incident response -- it omits %ID entirely (verified
  against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
  for CF-fronted sites. The new format keeps the first 16 fields byte-identical
  to the httplog default (so existing parsers still work) and appends
  cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
  Adds a User-Agent capture in slot 1 to feed it.

* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
  It does not, which made the documented support-correlation workflow
  (X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
  supported when it could never have worked.

Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".

UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.

Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:

  <150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
  [22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
  1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
  HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
  host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude
2026-08-22 10:11:30 -07:00
parent 67837f59cb
commit 711c670319
4 changed files with 89 additions and 13 deletions
+41 -11
View File
@@ -2,20 +2,46 @@
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
# ACCESS LOG DESTINATION.
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
# This used to be `log 127.0.0.1 local2`, which was a silent black hole:
# 127.0.0.1 is the CONTAINER's own loopback, nothing has ever listened on
# udp/514 in the container netns, and there is no /dev/log in the image.
# Every access log line -- ~1.5M/day across the whole edge -- was written
# to a socket with no receiver and dropped. Nothing errored, nothing
# warned, and `haproxy -c` was perfectly happy. The cost only shows up
# during an incident: per-IP 429s, tarpits, wp-admin gate redirects,
# WAF 403s and `silent-drop`s left no record anywhere, so the edge could
# not be asked what it had rejected. Only aggregate stick-table counters
# survived.
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
# Now points at the DOCKER BRIDGE GATEWAY, where the host's rsyslog has an
# imudp listener bound (installed idempotently by WHP's
# setup-haproxy-logrotate.sh, which also writes the logrotate stanza).
# The host writes local2 to /var/log/haproxy.log and stops it there, so it
# does not also flood /var/log/messages or the Graylog forwarder.
#
# local2.* /var/log/haproxy.log
# WHY NOT `log stdout format raw local0`: it is INCOMPATIBLE with the
# `daemon` keyword below, and incompatible SILENTLY. Verified on the
# pinned 3.0.11 binary: with `daemon` set, a `log stdout` config serves
# traffic normally and emits ZERO log lines, and `haproxy -c` returns 0
# with no error and no warning -- so the CI config gate
# (scripts/validate-rendered-config.py) cannot catch it either. Making it
# work means dropping `daemon` / adding -db so haproxy stays in the
# foreground, which in turn breaks the three synchronous
# `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites in
# haproxy_manager.py (they would block until the 180s timeout and then be
# killed). That is a change to the exact code path whose failure mode is
# "container Up, ports 80/443 never bound, every site down, /health still
# 200". Not worth it for a logging change.
#
log 127.0.0.1 local2
# UDP means a dead listener degrades to dropped log lines, never to a
# stalled or failing request path -- the correct failure direction for an
# edge fronting ~60 customer sites.
#
# len 2048 accommodates the enriched log-format in hap_listener.tpl
# (URL + User-Agent + UUID); the default 1024 would truncate long ones.
log {{ syslog_target }} len 2048 format rfc5424 local2 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
@@ -122,7 +148,11 @@ defaults
maxconn 3000
# Per-request unique reference, used:
# - in the log line (httplog includes %ID)
# - in the access log line, as the `id=` field of the custom log-format
# in hap_listener.tpl. NOTE: `option httplog` does NOT include %ID
# (verified against haproxy 3.0.11) -- this comment used to claim it
# did, which made the support workflow below look supported when it
# was not. The explicit log-format is what actually carries it.
# - echoed to clients in the X-Request-Reference response header on
# WAF blocks so a customer can quote it when opening a support ticket
# - embedded in /etc/haproxy/errors/403-waf.html so a blocked visitor