fix(api): stop silent clients from wedging the gunicorn thread pool

On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.

The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).

- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
  * per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
    re-applied on keep-alive reuse; bounds socket I/O, not handling time
  * watchdog: if every pool thread has been in the same request for more
    than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
    all thread stacks and exit so the master respawns the worker
  * client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
  60s) as defence in depth for any future in-process client; /health now
  reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
  HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
  added to the Docker build gate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

haproxy_manager.py

@@ -51,6 +51,28 @@ def _bounded_subprocess_run(*args, **kwargs):
 
 subprocess.run = _bounded_subprocess_run
 
+# ---------------------------------------------------------------------------
+# Default socket timeout (incident 2026-10-09) -- defence in depth only
+# ---------------------------------------------------------------------------
+# This module makes no outbound network calls today (ACME, HAProxy's admin
+# socket etc. all go through bounded subprocesses above). This default makes
+# sure any socket created in-process later -- a urllib/http.client call added
+# to some handler -- cannot block a worker thread forever by omission.
+#
+# It does NOT protect the API's CLIENT connections: gunicorn's gthread worker
+# calls sock.setblocking(True) on each one, which resets the timeout to None
+# and overrides this default. That is what actually wedged the pool on
+# 2026-10-09; it is fixed by haproxy_gunicorn_worker.TimeoutThreadWorker.
+DEFAULT_SOCKET_TIMEOUT = float(os.environ.get('HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT', '60'))
+if DEFAULT_SOCKET_TIMEOUT > 0:
+    socket.setdefaulttimeout(DEFAULT_SOCKET_TIMEOUT)
+
+try:
+    from haproxy_gunicorn_worker import pool_status as _gunicorn_pool_status
+except Exception:  # dev server / tests without gunicorn
+    def _gunicorn_pool_status():
+        return None
+
 app = Flask(__name__)
 
 # Default page server (port 8080) — served to HAProxy clients whose request hit
@@ -741,11 +763,19 @@ def health_check():
             cursor.execute('SELECT 1')
             cursor.fetchone()
 
-        return jsonify({
+        body = {
             'status': 'healthy',
             'haproxy_status': 'running' if haproxy_running else 'stopped',
             'database': 'connected'
-        }), 200
+        }
+        # Pool saturation of the worker that answered ('busy' includes this
+        # probe). Leaking threads show as busy/oldest_busy_seconds climbing;
+        # a fully wedged worker cannot answer at all, which is what the
+        # healthcheck's short --max-time catches.
+        pool = _gunicorn_pool_status()
+        if pool is not None:
+            body['worker'] = pool
+        return jsonify(body), 200
     except Exception as e:
         return jsonify({
             'status': 'unhealthy',