fix(api): stop silent clients from wedging the gunicorn thread pool

On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.

The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).

- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
  * per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
    re-applied on keep-alive reuse; bounds socket I/O, not handling time
  * watchdog: if every pool thread has been in the same request for more
    than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
    all thread stacks and exit so the master respawns the worker
  * client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
  60s) as defence in depth for any future in-process client; /health now
  reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
  HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
  added to the Docker build gate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

scripts/healthcheck.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Container liveness probe: BOTH planes must answer quickly.
+#
+#   1. Management API (:8000) -- GET /health with a short --max-time. On
+#      2026-10-09 (sdbees) the API workers' thread pools were wedged for long
+#      enough that /health hung 40s+, while the container reported "healthy"
+#      because the deployed --health-cmd only probed :80. A hung API is a dead
+#      control plane (every WHP site update times out), so it must turn the
+#      container unhealthy.
+#   2. HAProxy data plane (:80) -- any HTTP answer at all (the default page,
+#      a redirect, a 4xx/5xx) proves HAProxy accepted and answered. Only a
+#      connect/timeout failure fails it, which is why there is no -f here.
+#
+# 127.0.0.1, not localhost: the API binds IPv4 only, and resolving localhost
+# to ::1 first costs a failed connect on every probe.
+#
+# WHP's lib-edge-container.sh overrides the image HEALTHCHECK with its own
+# --health-cmd; point that at this script so the fleet gets the same probe.
+set -u
+API_TIMEOUT="${HAPROXY_MGR_HEALTH_API_TIMEOUT:-4}"
+EDGE_TIMEOUT="${HAPROXY_MGR_HEALTH_EDGE_TIMEOUT:-5}"
+
+if ! curl -sf -o /dev/null --max-time "$API_TIMEOUT" http://127.0.0.1:8000/health; then
+    echo "management API :8000 /health failed or took longer than ${API_TIMEOUT}s"
+    exit 1
+fi
+if ! curl -s -o /dev/null --max-time "$EDGE_TIMEOUT" http://127.0.0.1/; then
+    echo "HAProxy :80 did not answer within ${EDGE_TIMEOUT}s"
+    exit 1
+fi
+exit 0

scripts/start-up.sh

@@ -77,7 +77,7 @@ HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
 ) &
 
 # Phase 2: WSGI servers
-# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
+# Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
 # (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
 # 1000 — worker recycle frequency).
 #
@@ -91,12 +91,23 @@ API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
 MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
 MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
 
+# Worker class (incident 2026-10-09): stock gthread makes every client socket
+# fully blocking with NO timeout, so any peer that sends a few bytes and goes
+# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
+# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. This subclass
+# sets a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, default
+# 20s) and runs a watchdog that exits a worker whose threads have ALL been
+# stuck for HAPROXY_MGR_WEDGE_SECONDS (default 1200) so the master respawns it.
+# See haproxy_gunicorn_worker.py. --timeout below does NOT cover either case:
+# for gthread it only watches the main thread's heartbeat.
+WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
+
 # Default page server on :8080. Stays in the background.
 # --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
 # without forking. --max-requests recycles the worker to bound memory drift.
 gunicorn \
     --bind 0.0.0.0:8080 \
-    --workers 1 --threads 4 --worker-class gthread \
+    --workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout 30 \
     --access-logfile - --error-logfile - --log-level info \
@@ -108,7 +119,7 @@ gunicorn \
 # up). Longer --timeout because cert issuance hits ACME and can take a while.
 exec gunicorn \
     --bind 0.0.0.0:8000 \
-    --workers "${API_WORKERS}" --threads 4 --worker-class gthread \
+    --workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
     --max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
     --timeout "${API_TIMEOUT}" \
     --access-logfile - --error-logfile - --log-level info \

scripts/test-gunicorn-socket-timeouts.py

@@ -0,0 +1,356 @@
+#!/usr/bin/env python3
+"""Regression tests: a silent peer must not be able to wedge the API's threads.
+
+Why this file exists
+--------------------
+On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool threads
+blocked in the kernel in tcp_recvmsg -> sk_wait_data, /health hung 40s+, and the
+container stayed "healthy". The app makes no in-process outbound socket reads;
+the blocked reads were gunicorn gthread reading the CLIENT socket, which it
+makes fully blocking with no timeout (TConn.init -> setblocking(True)).
+
+These tests run REAL gunicorn processes (no mocks) against tiny WSGI apps and
+pin:
+
+  * the failure mode itself, on the stock gthread worker (control);
+  * that haproxy_gunicorn_worker.TimeoutThreadWorker frees the thread when a
+    client stalls before / during the request line, mid-body, or on a reused
+    keep-alive connection;
+  * that a legitimately SLOW request (handling time > client timeout) is not
+    cut off -- the timeout bounds socket I/O, not request duration;
+  * the watchdog: a worker whose every thread is stuck is replaced;
+  * the timeout is logged as one line, not a traceback per scanner;
+  * haproxy_manager's socket.setdefaulttimeout: an in-process client call to a
+    stalled upstream (accepts, never replies) times out instead of hanging;
+  * why that default alone is not enough (setblocking(True) clears it).
+
+Running
+-------
+    python3 scripts/test-gunicorn-socket-timeouts.py
+
+stdlib unittest, like the other scripts/test-*.py; needs only the app's own
+requirements (gunicorn, Flask...). Runs in the Docker build gate. ~25s.
+"""
+
+import os
+import socket
+import subprocess
+import sys
+import tempfile
+import textwrap
+import threading
+import time
+import unittest
+import urllib.request
+
+REPO_DIR = os.path.abspath(
+    os.environ.get('HAPROXY_MANAGER_DIR',
+                   os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')))
+
+try:
+    import gunicorn  # noqa: F401
+except ImportError:
+    sys.stderr.write('SKIP: gunicorn not installed (pip install -r requirements.txt)\n')
+    raise SystemExit(77)
+
+WSGI_APP = textwrap.dedent('''\
+    import os, time
+
+    def app(environ, start_response):
+        path = environ.get('PATH_INFO', '/')
+        if path == '/sleep':
+            time.sleep(float(environ.get('QUERY_STRING') or 3))
+        elif path == '/hang':
+            time.sleep(3600)
+        elif path == '/body':
+            n = int(environ.get('CONTENT_LENGTH') or 0)
+            environ['wsgi.input'].read(n)
+        body = ('pid=%d' % os.getpid()).encode()
+        start_response('200 OK', [('Content-Type', 'text/plain'),
+                                  ('Content-Length', str(len(body)))])
+        return [body]
+''')
+
+
+def free_port():
+    s = socket.socket()
+    s.bind(('127.0.0.1', 0))
+    port = s.getsockname()[1]
+    s.close()
+    return port
+
+
+def http_get(port, path='/', timeout=5):
+    with urllib.request.urlopen('http://127.0.0.1:%d%s' % (port, path),
+                                timeout=timeout) as r:
+        return r.read().decode()
+
+
+class Gunicorn:
+    """A real gunicorn master on a free port, with its output captured."""
+
+    def __init__(self, worker_class, threads=1, env=None):
+        self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
+        with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
+            fh.write(WSGI_APP)
+        self.port = free_port()
+        self.log_path = os.path.join(self.tmp, 'gunicorn.log')
+        e = dict(os.environ)
+        e.update(env or {})
+        self.log = open(self.log_path, 'w')
+        self.proc = subprocess.Popen(
+            [sys.executable, '-m', 'gunicorn',
+             '--bind', '127.0.0.1:%d' % self.port,
+             '--workers', '1', '--threads', str(threads),
+             '--worker-class', worker_class,
+             '--timeout', '120', '--graceful-timeout', '2',
+             '--log-level', 'info', '--error-logfile', '-',
+             '--pythonpath', '%s,%s' % (REPO_DIR, self.tmp),
+             'tapp:app'],
+            stdout=self.log, stderr=subprocess.STDOUT, env=e)
+        deadline = time.time() + 15
+        while time.time() < deadline:
+            try:
+                http_get(self.port, '/', timeout=1)
+                return
+            except Exception:
+                time.sleep(0.2)
+        self.stop()
+        raise RuntimeError('gunicorn did not come up:\n' + self.output())
+
+    def output(self):
+        self.log.flush()
+        with open(self.log_path) as fh:
+            return fh.read()
+
+    def stop(self):
+        if self.proc.poll() is None:
+            self.proc.terminate()
+            try:
+                self.proc.wait(5)
+            except subprocess.TimeoutExpired:
+                self.proc.kill()
+                self.proc.wait()
+        self.log.close()
+
+
+def stall_client(port, payload=b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03'):
+    """Connect and send a few bytes that are not a complete HTTP request (by
+    default the start of a TLS ClientHello -- what an HTTPS scanner sends to
+    this plain-HTTP port), then go silent and keep the connection open."""
+    s = socket.create_connection(('127.0.0.1', port))
+    if payload:
+        s.sendall(payload)
+    return s
+
+
+def wait_closed_by_server(sock, within):
+    """True if the server closed (EOF/RST) the connection within `within` s."""
+    sock.settimeout(within)
+    try:
+        while True:
+            data = sock.recv(4096)
+            if not data:
+                return True
+    except (ConnectionResetError, BrokenPipeError):
+        return True
+    except socket.timeout:
+        return False
+
+
+STOCK = 'gthread'
+FIXED = 'haproxy_gunicorn_worker.TimeoutThreadWorker'
+FAST = {'HAPROXY_MGR_CLIENT_TIMEOUT': '1'}
+
+
+class StockGthreadControl(unittest.TestCase):
+    """Proves the failure mode is real, so the fixed-worker tests mean something."""
+
+    def test_stock_gthread_is_wedged_by_one_silent_client(self):
+        g = Gunicorn(STOCK, threads=1)
+        self.addCleanup(g.stop)
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        time.sleep(0.3)
+        with self.assertRaises(Exception, msg='stock gthread should be wedged'):
+            http_get(g.port, '/', timeout=3)
+
+
+class TimeoutWorkerTests(unittest.TestCase):
+
+    def start(self, threads=1, env=None):
+        g = Gunicorn(FIXED, threads=threads, env=env or FAST)
+        self.addCleanup(g.stop)
+        return g
+
+    def test_silent_client_is_dropped_and_thread_freed(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        t0 = time.time()
+        # Only one thread: this request can only be served once the stalled
+        # connection has released it.
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertLess(time.time() - t0, 4)
+        self.assertTrue(wait_closed_by_server(s, 3),
+                        'server should have closed the stalled connection')
+
+    def test_connect_and_send_nothing_does_not_hold_a_thread(self):
+        # Never readable -> never handed to a pool thread (main loop only).
+        g = self.start()
+        s = stall_client(g.port, payload=b'')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
+
+    def test_half_written_headers(self):
+        g = self.start()
+        s = stall_client(g.port, payload=b'GET /health HTTP/1.1\r\nHost: x\r\n')
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_body_shorter_than_content_length(self):
+        g = self.start()
+        s = stall_client(g.port, payload=(
+            b'POST /body HTTP/1.1\r\nHost: x\r\nContent-Length: 100\r\n\r\n0123456789'))
+        self.addCleanup(s.close)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+
+    def test_timeout_reapplied_on_keepalive_reuse(self):
+        g = self.start()
+        s = socket.create_connection(('127.0.0.1', g.port))
+        self.addCleanup(s.close)
+        s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
+        s.settimeout(3)
+        self.assertIn(b'200 OK', s.recv(4096))
+        # Same connection, now stall mid-request. gthread calls conn.init()
+        # (setblocking(True)) again for the reused connection.
+        s.sendall(b'GET / HTT')
+        time.sleep(0.3)
+        self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
+        self.assertTrue(wait_closed_by_server(s, 3))
+
+    def test_slow_legit_request_is_not_cut_off(self):
+        # Handling takes 2.5s, client timeout is 1s: must still succeed.
+        g = self.start()
+        self.assertTrue(http_get(g.port, '/sleep?2.5', timeout=10).startswith('pid='))
+
+    def test_timeout_logged_as_one_line_not_traceback(self):
+        g = self.start()
+        s = stall_client(g.port)
+        self.addCleanup(s.close)
+        http_get(g.port, '/', timeout=5)
+        time.sleep(0.3)
+        out = g.output()
+        self.assertIn('Closed client connection idle', out)
+        self.assertNotIn('Traceback', out)
+
+    def test_watchdog_replaces_fully_wedged_worker(self):
+        g = self.start(env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                            'HAPROXY_MGR_WEDGE_SECONDS': '2',
+                            'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.5'})
+        pid_before = http_get(g.port, '/')
+        # Wedge the only thread inside the app (not a socket read).
+        hung = threading.Thread(target=lambda: self._swallow(g.port), daemon=True)
+        hung.start()
+        time.sleep(0.5)
+        deadline = time.time() + 12
+        pid_after = None
+        while time.time() < deadline:
+            try:
+                pid_after = http_get(g.port, '/', timeout=2)
+                if pid_after != pid_before:
+                    break
+            except Exception:
+                pass
+            time.sleep(0.3)
+        self.assertIsNotNone(pid_after)
+        self.assertNotEqual(pid_before, pid_after, 'worker was not replaced')
+        out = g.output()
+        self.assertIn('WEDGED WORKER', out)
+        self.assertIn('tapp.py', out)  # the stuck thread's stack was dumped
+        self.assertIsNone(g.proc.poll(), 'master must keep running')
+
+    def test_watchdog_leaves_partially_busy_pool_alone(self):
+        # 2 threads, one hung forever: the pool still serves, so no restart.
+        g = self.start(threads=2, env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
+                                       'HAPROXY_MGR_WEDGE_SECONDS': '1',
+                                       'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.3'})
+        pid_before = http_get(g.port, '/')
+        threading.Thread(target=lambda: self._swallow(g.port), daemon=True).start()
+        time.sleep(3)
+        self.assertEqual(pid_before, http_get(g.port, '/', timeout=3))
+        self.assertNotIn('WEDGED WORKER', g.output())
+
+    @staticmethod
+    def _swallow(port):
+        try:
+            http_get(port, '/hang', timeout=60)
+        except Exception:
+            pass
+
+
+class StalledUpstreamDefaultTimeout(unittest.TestCase):
+    """haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
+    client call to an upstream that accepts and never replies must time out."""
+
+    def test_in_process_client_to_stalled_upstream_times_out(self):
+        srv = socket.socket()
+        srv.bind(('127.0.0.1', 0))
+        srv.listen(5)
+        self.addCleanup(srv.close)
+        port = srv.getsockname()[1]
+        accepted = []
+        threading.Thread(target=lambda: accepted.append(srv.accept()),
+                         daemon=True).start()
+        # Import the real module in a child (it changes process-global socket
+        # state and opens its log files at import), then make a client call
+        # with NO explicit timeout -- what a careless future handler would do.
+        probe = textwrap.dedent('''\
+            import logging, os, sys, tempfile, time, urllib.request
+            d = tempfile.mkdtemp()
+            real = logging.FileHandler
+            logging.FileHandler = lambda fn, *a, **k: real(os.path.join(d, os.path.basename(fn)), *a, **k)
+            os.chdir(sys.argv[1]); sys.path.insert(0, sys.argv[1])
+            import haproxy_manager
+            logging.FileHandler = real
+            import socket
+            print('default', socket.getdefaulttimeout())
+            t0 = time.time()
+            try:
+                urllib.request.urlopen('http://127.0.0.1:%s/' % sys.argv[2]).read()
+                print('result returned')
+            except Exception as e:
+                print('result', type(e).__name__, round(time.time() - t0, 1))
+        ''')
+        env = dict(os.environ, HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT='1.5')
+        r = subprocess.run([sys.executable, '-c', probe, REPO_DIR, str(port)],
+                           capture_output=True, text=True, timeout=30, env=env)
+        if 'No module named' in r.stderr and 'flask' in r.stderr.lower():
+            self.skipTest('Flask not installed')
+        self.assertIn('default 1.5', r.stdout, r.stderr)
+        line = [l for l in r.stdout.splitlines() if l.startswith('result')][0]
+        self.assertTrue(line.startswith('result TimeoutError '), line)
+        elapsed = float(line.split()[-1])
+        self.assertLess(elapsed, 5)
+        self.assertTrue(accepted, 'upstream never saw the connection')
+        accepted[0][0].close()
+
+    def test_default_timeout_alone_does_not_cover_gthread_client_sockets(self):
+        # Documents why the custom worker exists: gthread's setblocking(True)
+        # resets any timeout -- including the process default -- to None.
+        old = socket.getdefaulttimeout()
+        socket.setdefaulttimeout(60)
+        try:
+            a, b = socket.socketpair()
+        finally:
+            socket.setdefaulttimeout(old)
+        self.addCleanup(a.close)
+        self.addCleanup(b.close)
+        self.assertEqual(a.gettimeout(), 60)
+        a.setblocking(True)
+        self.assertIsNone(a.gettimeout())
+
+
+if __name__ == '__main__':
+    unittest.main(verbosity=2)