fix(api): stop silent clients from wedging the gunicorn thread pool

On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.

The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).

- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
  * per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
    re-applied on keep-alive reuse; bounds socket I/O, not handling time
  * watchdog: if every pool thread has been in the same request for more
    than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
    all thread stacks and exit so the master respawns the worker
  * client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
  60s) as defence in depth for any future in-process client; /health now
  reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
  HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
  added to the Docker build gate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Dockerfile

@@ -44,6 +44,7 @@ WORKDIR /haproxy
 COPY ./templates /haproxy/templates
 COPY requirements.txt /haproxy/
 COPY haproxy_manager.py /haproxy/
+COPY haproxy_gunicorn_worker.py /haproxy/
 COPY scripts /haproxy/scripts
 COPY trusted_ips.list /etc/haproxy/trusted_ips.list
 COPY trusted_ips.map /etc/haproxy/trusted_ips.map
@@ -92,7 +93,8 @@ RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
  && python3 /haproxy/scripts/test-config-rollback.py \
  && python3 /haproxy/scripts/test-cert-write-safety.py \
  && python3 /haproxy/scripts/test-cert-scripts.py \
- && python3 /haproxy/scripts/validate-rendered-config.py
+ && python3 /haproxy/scripts/validate-rendered-config.py \
+ && python3 /haproxy/scripts/test-gunicorn-socket-timeouts.py
 # Create log directories
 RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
 RUN chmod 755 /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
@@ -105,7 +107,10 @@ RUN mkdir -p /var/spool/cron/crontabs && \
 # 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
 # must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
 EXPOSE 80 443 443/udp 8000
-# Add health check
+# Health check: management API (:8000, short timeout) AND HAProxy (:80).
+# See scripts/healthcheck.sh. NOTE: WHP's lib-edge-container.sh passes its own
+# --health-cmd, which REPLACES this one on deployed hosts; it must be pointed
+# at /haproxy/scripts/healthcheck.sh for the fleet to get the :8000 probe.
 HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
-    CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1
+    CMD /haproxy/scripts/healthcheck.sh
 CMD ["/haproxy/scripts/start-up.sh"]