fix(logging): stop silently discarding every access log line
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m46s

haproxy.cfg's global section has had `log 127.0.0.1 local2` since day one.
That is the CONTAINER's own loopback: nothing has ever listened on udp/514 in
the container netns and there is no /dev/log in the image. Every access log
line -- ~1.5M/day across ~60 customer sites -- was written to a socket with no
receiver and dropped. Nothing errored, nothing warned, and `haproxy -c` was
perfectly happy, so this survived unnoticed.

The cost only shows up during an incident. Per-IP 429s, tarpits, wp-admin gate
redirects, WAF 403s and `silent-drop`s left no record anywhere, so the edge
could not be asked what it had actually rejected -- only aggregate stick-table
counters survived. That blind spot applies to every WHP host.

Changes:

* hap_header.tpl: point `log` at {{ syslog_target }} (default 172.18.0.1:514,
  the client-net bridge gateway) with `len 2048 format rfc5424 local2 info`.
  WHP's setup-haproxy-syslog.sh installs the matching rsyslog receiver on the
  host, in a dedicated ruleset ending in stop() so 1.5M lines/day cannot flood
  /var/log/messages or the Graylog forwarder, bound to the bridge IP rather
  than 0.0.0.0.

* haproxy_manager.py: render that target from HAPROXY_SYSLOG_TARGET so
  standalone/home deployments on a different bridge subnet can retarget it.

* hap_listener.tpl: add a frontend-scoped `log-format`. `option httplog` is
  not sufficient for incident response -- it omits %ID entirely (verified
  against 3.0.11), and its %ci is the Cloudflare edge rather than the visitor
  for CF-fronted sites. The new format keeps the first 16 fields byte-identical
  to the httplog default (so existing parsers still work) and appends
  cip=<real client, from var(txn.real_ip)>, id=<uuid>, host=, ua=, sni=, hv=.
  Adds a User-Agent capture in slot 1 to feed it.

* hap_header.tpl: correct the comment claiming `option httplog` includes %ID.
  It does not, which made the documented support-correlation workflow
  (X-Request-Reference -> access log -> coraza audit.log -> rule_id) look
  supported when it could never have worked.

Deliberately NOT using `log stdout format raw local0`: it is incompatible with
the `daemon` keyword, and incompatible SILENTLY. Verified on the pinned 3.0.11
binary -- with `daemon` set, a `log stdout` config serves traffic normally and
emits zero log lines, while `haproxy -c` returns 0 with no error and no
warning, so scripts/validate-rendered-config.py could not catch it either.
Making it work would mean dropping `daemon`, which breaks the three
synchronous `subprocess.run(['haproxy', '-W', ...], check=True)` launch sites
in haproxy_manager.py -- the exact code path whose failure mode is "container
Up, ports 80/443 never bound, every site down, /health still 200".

UDP was chosen so a dead listener degrades to dropped log lines rather than a
stalled request path.

Verified: scripts/validate-rendered-config.py passes `haproxy -c` on both the
"default" and "full" scenarios against the real 3.0.11 binary; and a live
haproxy running WITH `daemon` (as production does) was confirmed to emit real
lines carrying the true client IP from CF-Connecting-IP:

  <150>1 2026-08-22T17:05:21+00:00 - haproxy 109 - - 127.0.0.1:51194
  [22/Aug/2026:17:05:21.217] t t/<NOSRV> 0/-1/-1/-1/0 200 73 - - LR--
  1/1/0/0/0 0/0 {cf-site.example|Mozilla/5.0 RealVisitor} "GET /checkout/
  HTTP/1.1" cip=203.0.113.77 id=dfe94fa9-8d95-4126-81e1-821578f22872
  host=cf-site.example ua=Mozilla/5.0 RealVisitor sni=- hv=1

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude
2026-08-22 10:11:30 -07:00
parent 67837f59cb
commit 711c670319
4 changed files with 89 additions and 13 deletions
+11
View File
@@ -2351,9 +2351,20 @@ def generate_config():
except Exception as e:
logger.error(f"Failed to create {suspended_list_path}: {e}")
# Access-log destination for the `log` line in the global section.
# Default 172.18.0.1:514 is the docker bridge gateway for WHP's
# `client-net`, i.e. the host, where rsyslog's imudp listener is bound
# by setup-haproxy-logrotate.sh. Overridable so this image stays usable
# on standalone/home deployments with a different bridge subnet or a
# remote log collector -- set HAPROXY_SYSLOG_TARGET to `<ip>:<port>`.
# UDP, so an absent listener drops log lines and never affects request
# handling.
syslog_target = os.environ.get('HAPROXY_SYSLOG_TARGET', '172.18.0.1:514').strip()
# Add Haproxy Default Headers
default_headers = template_env.get_template('hap_header.tpl').render(
cluster_secret = get_or_create_cluster_secret(),
syslog_target = syslog_target,
)
config_parts.append(default_headers)