fix(logging): keep gunicorn access logs off container stdout

Both gunicorns ran with --access-logfile -, so every :8080 default-page /
blocked-IP / suspended-site hit and every :8000 API call became a docker
stdout line. With the journald log driver (WHP PR #260) that is one journal
entry per request: journald's default rate limit (10000 msgs / 30 s per
service) could suppress the relay's crash-dump lines, and Graylog would be
flooded (the edge access log must stay out of it).

* :8080 default-page gunicorn: no access log.
* :8000 API gunicorn: access log to /var/log/haproxy/api-access.log (mode
  0640, haproxy-logs volume), bounded by scripts/gunicorn-access-rotate.py
  (copy-truncate, 20 MiB x 3, 30 s poll). 401/403 are in the file only.
* gunicorn ERROR logging stays on stderr; supervisor/relay/startup lines
  unchanged on stdout/stderr.
* New build-gate test scripts/test-gunicorn-access-logging.py; docs/logging.md.

VERSION unchanged (2026.11.5): a NEW rc image must be built from this branch
for the change to ship.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
This commit is contained in:
shadowdaoandClaude Sonnet 5.5 committed 2026-11-16 22:12:53 -08:00
1 parent 68976142f5
commit fa8fec0b4f
7 files changed
+324 -2

No files matched your search

+1
View File
@@ -15,6 +15,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
- **Multi-arch index assembly (coraza-spoa)**: `python3 scripts/test-assemble-multiarch-index.py` - offline (NOT run in the image build; it also reads `.gitea/workflows/build-push-coraza.yaml`); `scripts/assemble-multiarch-index.sh` turns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome is `success` (failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing. `scripts/e2e-coraza-multiarch.sh` rehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch").
- **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash".
- **gunicorn access logging**: `python3 scripts/test-gunicorn-access-logging.py` - real gunicorn; :8080 has no access log, :8000 access goes to `/var/log/haproxy/api-access.log` (rotated by `scripts/gunicorn-access-rotate.py`), nothing per-request on stdout (journald rate-limit/Graylog flood), gunicorn errors stay on stderr. See docs/logging.md. Run after touching start-up.sh's gunicorn lines.
- **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`.
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
+1
View File
@@ -158,6 +158,7 @@ RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
&& python3 /haproxy/scripts/test-cert-scripts.py \
&& python3 /haproxy/scripts/test-waf-body-inspection.py \
&& python3 /haproxy/scripts/test-gunicorn-socket-timeouts.py \
&& python3 /haproxy/scripts/test-gunicorn-access-logging.py \
&& python3 /haproxy/scripts/validate-rendered-config.py
# Create log directories
RUN mkdir -p /var/log/haproxy && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
+2
View File
@@ -451,6 +451,8 @@ Covered by `scripts/test-default-splash.py` (template order/conditions, the
page, list generation) and the `splash-off` scenario of
`scripts/validate-rendered-config.py`.
> gunicorn access logs (:8080 none, :8000 bounded file, not stdout): see docs/logging.md.
## Access Log (in-container, since 2026.11.2)
HAProxy logs (`log /log.sock ... format rfc5424`, `hap_header.tpl`) to a UNIX
+43
View File
@@ -0,0 +1,43 @@
# Container logging policy (what reaches `docker logs`)
`docker logs haproxy-manager` (and the docker logging driver: json-file today,
journald -> Graylog under WHP PR #260) must carry only low-volume, high-value
lines. Per-request logs stay out of it.
| Source | Destination | Volume |
|---|---|---|
| HAProxy edge access log (1.5M lines/day) | `/var/log/haproxy/access.log` via `haproxy-log-receiver.py` (20 MiB cap, gzip generations) | NOT in `docker logs` |
| HAProxy master stderr (ALERT, crash dumps) | `haproxy-stderr.log` -> `haproxy-stderr-relay.py` -> stdout (benign, rate limited) / stderr (crash evidence) | rare |
| Supervisor / relay / init lines | stdout / stderr | rare |
| gunicorn :8080 default-page / blocked-IP / suspended-site | NO access log | none |
| gunicorn :8000 management API access log | `/var/log/haproxy/api-access.log` (haproxy-logs volume, mode 0640), rotated by `scripts/gunicorn-access-rotate.py` (copy-truncate, 20 MiB x 3) | NOT in `docker logs` |
| gunicorn error log (tracebacks, worker timeouts) | stderr (`--error-logfile -`) | rare |
## Why
Before this change both gunicorns ran with `--access-logfile -`, so each :8080
hit and each :8000 API call was one docker stdout line. With the journald
driver that is one journal entry per request: journald's default rate limit
(10000 messages / 30 s per sending service) can then suppress the very
crash-dump lines the relay emits, and Graylog would be flooded.
## Knobs
`HAPROXY_MGR_API_ACCESS_LOG` (path), `HAPROXY_API_ACCESS_LOG_MAX_BYTES`
(20971520), `HAPROXY_API_ACCESS_LOG_BACKUPS` (3), `HAPROXY_API_ACCESS_LOG_POLL`
(30 s; the live file can exceed the cap by at most one interval of API traffic).
## Not done
Failed-auth (401/403) lines are NOT mirrored to stdout; they are in the file
(grep the status column). Add a rate-limited mirror later if wanted.
## Tests
`python3 scripts/test-gunicorn-access-logging.py` (runs in the image build):
real gunicorn, a burst of default-page requests produces zero stdout lines,
API requests land in the file not stdout, handler exceptions still reach
stderr, and rotation is bounded.
Deploying this needs a NEW rc image built from `feat/haproxy-upgrade`
(VERSION stays 2026.11.5 until then).
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
"""Bound the management-API (:8000) gunicorn access log file.
Why
---
Gunicorn's access log used to go to stdout (`--access-logfile -`), i.e. every
:8080 default-page / blocked-IP / suspended-site hit and every :8000 API call
became a `docker logs` line. With the journald docker log driver that is one
journal entry per request (journald's default rate limit, 10000 msgs / 30 s per
sending service, can then SUPPRESS the crash-dump lines the relay emits) and it
floods Graylog. Now:
* :8080 (default page) has NO access log at all;
* :8000 (API) writes its access log to API_ACCESS_LOG (default
/var/log/haproxy/api-access.log, in the haproxy-logs volume), NOT stdout;
* gunicorn ERROR logging (tracebacks, worker timeouts) stays on stderr.
This process keeps that file bounded: copy-truncate rotation at
HAPROXY_API_ACCESS_LOG_MAX_BYTES (default 20 MiB), HAPROXY_API_ACCESS_LOG_BACKUPS
generations (default 3), rotated copies mode 0640 (same policy as
haproxy-stderr.log). Truncating in place is safe because gunicorn opens the
file in append mode. Several gunicorn processes share the file, so renaming it
is not an option.
Bound: size is checked every HAPROXY_API_ACCESS_LOG_POLL seconds (default 30),
so the live file can exceed the limit only by what the API writes in one
interval (a few hundred bytes per call).
"""
import os
import shutil
import sys
import time
def _env_num(name, default, cast=int):
try:
v = cast(os.environ.get(name, default))
return v if v > 0 else default
except (TypeError, ValueError):
return default
def rotate_if_needed(path, max_bytes, backups):
"""Copy-truncate rotate. Returns True if it rotated."""
try:
if os.path.getsize(path) <= max_bytes:
return False
for i in range(backups, 1, -1):
src = '%s.%d' % (path, i - 1)
if os.path.exists(src):
os.replace(src, '%s.%d' % (path, i))
shutil.copyfile(path, path + '.1')
os.chmod(path + '.1', 0o640)
with open(path, 'r+b') as fh:
fh.truncate(0)
return True
except FileNotFoundError:
return False
except OSError as exc:
sys.stderr.write('gunicorn-access-rotate: rotate failed: %s\n' % exc)
return False
def main():
path = os.environ.get('HAPROXY_MGR_API_ACCESS_LOG', '/var/log/haproxy/api-access.log')
max_bytes = _env_num('HAPROXY_API_ACCESS_LOG_MAX_BYTES', 20 * 1024 * 1024)
backups = _env_num('HAPROXY_API_ACCESS_LOG_BACKUPS', 3)
poll = _env_num('HAPROXY_API_ACCESS_LOG_POLL', 30.0, float)
while True:
try:
rotate_if_needed(path, max_bytes, backups)
except Exception as exc: # never die; the supervisor loop restarts us anyway
sys.stderr.write('gunicorn-access-rotate: %r\n' % exc)
time.sleep(poll)
if __name__ == '__main__':
main()
+24 -2
View File
@@ -89,6 +89,28 @@ cron &
done
) &
# gunicorn access logs (incident-prevention for the journald log driver).
# gunicorn used to log every request to stdout (`--access-logfile -`), so every
# :8080 default-page / blocked-IP hit and every :8000 API call became a docker
# log line -> journal entry -> Graylog line, and could trip journald's rate
# limit and suppress the relay's crash-dump lines. Now:
# :8080 no access log at all;
# :8000 access log to a size-bounded FILE in the haproxy-logs volume (mode
# 0640), rotated by scripts/gunicorn-access-rotate.py (copy-truncate,
# 20 MiB x 3). Set HAPROXY_MGR_API_ACCESS_LOG to relocate it.
# Gunicorn ERROR logging (--error-logfile -) stays on stderr -> `docker logs`.
API_ACCESS_LOG="${HAPROXY_MGR_API_ACCESS_LOG:-/var/log/haproxy/api-access.log}"
export HAPROXY_MGR_API_ACCESS_LOG="${API_ACCESS_LOG}"
mkdir -p "$(dirname "${API_ACCESS_LOG}")"
[ -e "${API_ACCESS_LOG}" ] || : > "${API_ACCESS_LOG}"
chmod 0640 "${API_ACCESS_LOG}" 2>/dev/null || true
(
while true; do
python /haproxy/scripts/gunicorn-access-rotate.py || true
sleep 2
done
) &
# Phase 1: container init
python /haproxy/scripts/init.py
@@ -135,7 +157,7 @@ gunicorn \
--workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
--max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
--timeout 30 \
--access-logfile - --error-logfile - --log-level info \
--error-logfile - --log-level info \
--pythonpath /haproxy \
'haproxy_manager:default_app' &
@@ -147,6 +169,6 @@ exec gunicorn \
--workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
--max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
--timeout "${API_TIMEOUT}" \
--access-logfile - --error-logfile - --log-level info \
--access-logfile "${API_ACCESS_LOG}" --error-logfile - --log-level info \
--pythonpath /haproxy \
'haproxy_manager:app'
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env python3
"""Tests: gunicorn access logging must not flood the container's stdout.
Context: with the journald docker log driver every stdout line is a journal
entry (default rate limit 10000 msgs/30 s per service could suppress crash-dump
lines) and, via Graylog, a stored message. start-up.sh therefore gives :8080 no
access log and sends :8000's to a bounded FILE; gunicorn errors stay on stderr.
Pins (static, from start-up.sh):
* no gunicorn invocation uses `--access-logfile -`;
* the :8080 invocation has no --access-logfile at all;
* the :8000 invocation logs access to ${API_ACCESS_LOG}, errors to `-`.
Pins (live, REAL gunicorn, same flag shapes):
* a burst of default-page requests (no access log) writes ZERO lines to
stdout and none to stderr;
* a burst of API requests writes ZERO stdout lines, the access file receives
them, and a handler exception still reaches STDERR (error logging intact);
* scripts/gunicorn-access-rotate.py keeps the file bounded (copy-truncate,
backups capped, rotated copy mode 0640).
Runs in the Docker build gate.
"""
import importlib.util
import os
import re
import socket
import stat
import subprocess
import sys
import tempfile
import textwrap
import time
import unittest
import urllib.request
HERE = os.path.dirname(os.path.abspath(__file__))
try:
import gunicorn # noqa: F401
except ImportError:
sys.stderr.write('SKIP: gunicorn not installed\n')
raise SystemExit(77)
spec = importlib.util.spec_from_file_location('rot', os.path.join(HERE, 'gunicorn-access-rotate.py'))
rot = importlib.util.module_from_spec(spec)
spec.loader.exec_module(rot)
APP = textwrap.dedent('''\
def app(environ, start_response):
if environ['PATH_INFO'] == '/boom':
raise RuntimeError('handler-exploded')
start_response('200 OK', [('Content-Type', 'text/plain'), ('Content-Length', '2')])
return [b'ok']
''')
def free_port():
s = socket.socket()
s.bind(('127.0.0.1', 0))
p = s.getsockname()[1]
s.close()
return p
def start_script():
with open(os.path.join(HERE, 'start-up.sh')) as fh:
return fh.read()
class StaticFlags(unittest.TestCase):
def setUp(self):
self.src = start_script()
# Each invocation = from "gunicorn \" to its app target line.
self.inv = re.findall(r'^(?:exec )?gunicorn \\\n(?:.*\\\n)*?.*haproxy_manager:\w+.*$', self.src, re.M)
def test_two_invocations(self):
self.assertEqual(len(self.inv), 2, self.inv)
def test_no_stdout_access_log(self):
for i in self.inv:
self.assertNotRegex(i, r'--access-logfile\s+-(\s|$)')
def test_default_page_has_no_access_log(self):
d = [i for i in self.inv if '0.0.0.0:8080' in i]
self.assertEqual(len(d), 1)
self.assertNotIn('--access-logfile', d[0])
self.assertIn('--error-logfile -', d[0])
def test_api_access_log_is_a_file(self):
a = [i for i in self.inv if '0.0.0.0:8000' in i]
self.assertEqual(len(a), 1)
self.assertIn('--access-logfile "${API_ACCESS_LOG}"', a[0])
self.assertIn('--error-logfile -', a[0])
def test_rotator_started(self):
self.assertIn('gunicorn-access-rotate.py', self.src)
class Live(unittest.TestCase):
N_DEFAULT = 300
N_API = 100
def setUp(self):
self.dir = tempfile.mkdtemp(prefix='gunicorn-access-')
with open(os.path.join(self.dir, 'tapp.py'), 'w') as fh:
fh.write(APP)
def run_gunicorn(self, extra, n, boom=False):
port = free_port()
out = open(os.path.join(self.dir, 'out-%d' % port), 'w+')
err = open(os.path.join(self.dir, 'err-%d' % port), 'w+')
p = subprocess.Popen(
[sys.executable, '-m', 'gunicorn', '--bind', '127.0.0.1:%d' % port,
'--workers', '1', '--threads', '4', '--error-logfile', '-', '--log-level', 'info',
'--pythonpath', self.dir] + extra + ['tapp:app'],
stdout=out, stderr=err)
try:
for _ in range(100):
try:
urllib.request.urlopen('http://127.0.0.1:%d/' % port, timeout=2).read()
break
except Exception:
time.sleep(0.2)
for _ in range(n):
urllib.request.urlopen('http://127.0.0.1:%d/x' % port, timeout=5).read()
if boom:
try:
urllib.request.urlopen('http://127.0.0.1:%d/boom' % port, timeout=5).read()
except Exception:
pass
time.sleep(0.5)
finally:
p.terminate()
p.wait(10)
out.seek(0)
err.seek(0)
o, e = out.read(), err.read()
out.close()
err.close()
return o, e
def test_default_page_burst_is_silent(self):
out, err = self.run_gunicorn([], self.N_DEFAULT)
self.assertEqual(out, '', 'access logging leaked to stdout: %r' % out[:200])
self.assertNotIn('GET /x', err)
def test_api_burst_goes_to_file_not_stdout_and_errors_stay_on_stderr(self):
path = os.path.join(self.dir, 'api-access.log')
open(path, 'w').close()
out, err = self.run_gunicorn(['--access-logfile', path], self.N_API, boom=True)
self.assertEqual(out, '', 'access logging leaked to stdout: %r' % out[:200])
self.assertNotIn('GET /x', err)
with open(path) as fh:
lines = fh.read().splitlines()
self.assertGreaterEqual(len([l for l in lines if 'GET /x' in l]), self.N_API)
self.assertIn('handler-exploded', err) # error logging intact
class Rotation(unittest.TestCase):
def test_bounded_and_mode(self):
d = tempfile.mkdtemp(prefix='gunicorn-rot-')
path = os.path.join(d, 'api-access.log')
for _ in range(12):
with open(path, 'ab') as fh:
fh.write(b'x' * 1000 + b'\n')
rot.rotate_if_needed(path, 4000, 3)
files = sorted(os.listdir(d))
self.assertLessEqual(len(files), 4, files) # live + 3 backups
self.assertLessEqual(os.path.getsize(path), 4000 + 1001)
self.assertEqual(stat.S_IMODE(os.stat(path + '.1').st_mode), 0o640)
self.assertFalse(os.path.exists(path + '.4'))
def test_missing_file_is_not_fatal(self):
self.assertFalse(rot.rotate_if_needed('/nonexistent/api-access.log', 10, 3))
if __name__ == '__main__':
unittest.main()