fix(api): stop silent clients from wedging the gunicorn thread pool

On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.

The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).

- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
  * per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
    re-applied on keep-alive reuse; bounds socket I/O, not handling time
  * watchdog: if every pool thread has been in the same request for more
    than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
    all thread stacks and exit so the master respawns the worker
  * client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
  60s) as defence in depth for any future in-process client; /health now
  reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
  HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
  added to the Docker build gate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

scripts/start-up.sh

@@ -77,7 +77,7 @@ HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
 ) &
 
 # Phase 2: WSGI servers
-# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
+# Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
 # (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
 # 1000 — worker recycle frequency).
 #
@@ -91,12 +91,23 @@ API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
 MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
 MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
 
+# Worker class (incident 2026-10-09): stock gthread makes every client socket
+# fully blocking with NO timeout, so any peer that sends a few bytes and goes
+# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
+# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. This subclass
+# sets a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, default
+# 20s) and runs a watchdog that exits a worker whose threads have ALL been
+# stuck for HAPROXY_MGR_WEDGE_SECONDS (default 1200) so the master respawns it.
+# See haproxy_gunicorn_worker.py. --timeout below does NOT cover either case:
+# for gthread it only watches the main thread's heartbeat.
+WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
+
 # Default page server on :8080. Stays in the background.
 # --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
 # without forking. --max-requests recycles the worker to bound memory drift.
 gunicorn \
     --bind 0.0.0.0:8080 \
-    --workers 1 --threads 4 --worker-class gthread \
+    --workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout 30 \
     --access-logfile - --error-logfile - --log-level info \
@@ -108,7 +119,7 @@ gunicorn \
 # up). Longer --timeout because cert issuance hits ACME and can take a while.
 exec gunicorn \
     --bind 0.0.0.0:8000 \
-    --workers "${API_WORKERS}" --threads 4 --worker-class gthread \
+    --workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout "${API_TIMEOUT}" \
     --access-logfile - --error-logfile - --log-level info \