fix(api): reap never-readable conns, bound total header time, harden watchdog
Review round 1 on 0ce144d (FIX-FIRST):
F1: stock gthread never reaps a connection that never sends a byte; it
holds a worker_connections slot forever and at 1000 per worker the run
loop stops polling, wedging the worker permanently (survives the
attacker disconnecting). accept/on_client_socket_readable/
murder_keepalived now track such conns in a FIFO with a deadline
(HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s) and close expired ones on every
loop pass, including the at-capacity branch.
F2: slowloris (1 header byte per <20s) never tripped the per-recv
timeout. A total request-line+headers deadline
(HAPROXY_MGR_HEADER_DEADLINE, 10s) is armed in handle() and cleared in
handle_request(); a 1s monitor loop shutdown(SHUT_RDWR)s expired
sockets. Armed per thread, not at accept, so legit requests queued
behind busy threads are never cut for waiting.
F4: monitor loop wrapped in try/except with logging; wedge threshold
1200s -> 3600s (multi-domain cert requests run 180s per domain).
F5: docstring now says the timeout bounds each whole sendall().
F6: warn when gunicorn != 23.0.0 (worker overrides gthread internals).
API key compared with hmac.compare_digest.
Tests: +3 production-shape (2x4) tests: 2050 idle conns reaped and
/health recovers while still held; 8 drip conns cut within the deadline
with /health responsive; slow-but-complete headers still served.
15 tests, in the Docker build gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0ce144d2fb
commit
fd06b30d78
5 files changed
+343
-53
No files matched your search
+9
-5
@@ -94,11 +94,15 @@ MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
|
||||
# Worker class (incident 2026-10-09): stock gthread makes every client socket
|
||||
# fully blocking with NO timeout, so any peer that sends a few bytes and goes
|
||||
# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
|
||||
# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. This subclass
|
||||
# sets a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, default
|
||||
# 20s) and runs a watchdog that exits a worker whose threads have ALL been
|
||||
# stuck for HAPROXY_MGR_WEDGE_SECONDS (default 1200) so the master respawns it.
|
||||
# See haproxy_gunicorn_worker.py. --timeout below does NOT cover either case:
|
||||
# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. Stock gthread
|
||||
# also never reaps connections that send nothing (2x1000 of them wedge it
|
||||
# permanently) and has no total header-read bound (slowloris). This subclass
|
||||
# adds: a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, 20s), a
|
||||
# first-byte deadline (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s), a total
|
||||
# request-line+headers deadline (HAPROXY_MGR_HEADER_DEADLINE, 10s), and a
|
||||
# watchdog that exits a worker whose threads have ALL been stuck for
|
||||
# HAPROXY_MGR_WEDGE_SECONDS (3600) so the master respawns it.
|
||||
# See haproxy_gunicorn_worker.py. --timeout below does NOT cover any of these:
|
||||
# for gthread it only watches the main thread's heartbeat.
|
||||
WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
|
||||
|
||||
|
||||
Reference in new issue
Block a user