Files
haproxy-manager-base/docs/logging.md
shadowdaoandClaude Sonnet 5.5 fa8fec0b4f fix(logging): keep gunicorn access logs off container stdout
Both gunicorns ran with --access-logfile -, so every :8080 default-page /
blocked-IP / suspended-site hit and every :8000 API call became a docker
stdout line. With the journald log driver (WHP PR #260) that is one journal
entry per request: journald's default rate limit (10000 msgs / 30 s per
service) could suppress the relay's crash-dump lines, and Graylog would be
flooded (the edge access log must stay out of it).

* :8080 default-page gunicorn: no access log.
* :8000 API gunicorn: access log to /var/log/haproxy/api-access.log (mode
  0640, haproxy-logs volume), bounded by scripts/gunicorn-access-rotate.py
  (copy-truncate, 20 MiB x 3, 30 s poll). 401/403 are in the file only.
* gunicorn ERROR logging stays on stderr; supervisor/relay/startup lines
  unchanged on stdout/stderr.
* New build-gate test scripts/test-gunicorn-access-logging.py; docs/logging.md.

VERSION unchanged (2026.11.5): a NEW rc image must be built from this branch
for the change to ship.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
2026-11-16 22:12:53 -08:00

2.2 KiB

Container logging policy (what reaches docker logs)

docker logs haproxy-manager (and the docker logging driver: json-file today, journald -> Graylog under WHP PR #260) must carry only low-volume, high-value lines. Per-request logs stay out of it.

Source Destination Volume
HAProxy edge access log (1.5M lines/day) /var/log/haproxy/access.log via haproxy-log-receiver.py (20 MiB cap, gzip generations) NOT in docker logs
HAProxy master stderr (ALERT, crash dumps) haproxy-stderr.log -> haproxy-stderr-relay.py -> stdout (benign, rate limited) / stderr (crash evidence) rare
Supervisor / relay / init lines stdout / stderr rare
gunicorn :8080 default-page / blocked-IP / suspended-site NO access log none
gunicorn :8000 management API access log /var/log/haproxy/api-access.log (haproxy-logs volume, mode 0640), rotated by scripts/gunicorn-access-rotate.py (copy-truncate, 20 MiB x 3) NOT in docker logs
gunicorn error log (tracebacks, worker timeouts) stderr (--error-logfile -) rare

Why

Before this change both gunicorns ran with --access-logfile -, so each :8080 hit and each :8000 API call was one docker stdout line. With the journald driver that is one journal entry per request: journald's default rate limit (10000 messages / 30 s per sending service) can then suppress the very crash-dump lines the relay emits, and Graylog would be flooded.

Knobs

HAPROXY_MGR_API_ACCESS_LOG (path), HAPROXY_API_ACCESS_LOG_MAX_BYTES (20971520), HAPROXY_API_ACCESS_LOG_BACKUPS (3), HAPROXY_API_ACCESS_LOG_POLL (30 s; the live file can exceed the cap by at most one interval of API traffic).

Not done

Failed-auth (401/403) lines are NOT mirrored to stdout; they are in the file (grep the status column). Add a rate-limited mirror later if wanted.

Tests

python3 scripts/test-gunicorn-access-logging.py (runs in the image build): real gunicorn, a burst of default-page requests produces zero stdout lines, API requests land in the file not stdout, handler exceptions still reach stderr, and rotation is bounded.

Deploying this needs a NEW rc image built from feat/haproxy-upgrade (VERSION stays 2026.11.5 until then).