shadowdaoandClaude Opus 5.5
0ce144d2fb
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool
threads blocked in tcp_recvmsg -> sk_wait_data; /health and
/api/blocked-ips hung 40s+ while the container reported healthy.
The app makes no in-process outbound socket reads (every ACME/HAProxy
interaction is an already-bounded subprocess). The blocked reads were
gunicorn gthread reading the CLIENT socket: TConn.init() calls
setblocking(True), i.e. timeout None, before parsing the request. Any
peer that sends a few bytes and goes quiet (a TLS ClientHello on this
plain-HTTP port, scanners; :8000 is published on 0.0.0.0) pins a pool
thread forever. gunicorn --timeout does not catch it (gthread only
watches the main thread's heartbeat), --max-requests never triggers on
a worker that finishes nothing, and socket.setdefaulttimeout() is reset
by setblocking(True).
- haproxy_gunicorn_worker.TimeoutThreadWorker (used for :8000 and :8080):
* per-connection socket timeout HAPROXY_MGR_CLIENT_TIMEOUT (20s),
re-applied on keep-alive reuse; bounds socket I/O, not handling time
* watchdog: if every pool thread has been in the same request for more
than HAPROXY_MGR_WEDGE_SECONDS (1200s, > certbot renew's 900s), dump
all thread stacks and exit so the master respawns the worker
* client timeouts logged as one INFO line, not a traceback
- haproxy_manager: socket.setdefaulttimeout (HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT,
60s) as defence in depth for any future in-process client; /health now
reports the answering worker's busy/total threads and oldest request age
- scripts/healthcheck.sh: probes :8000/health (4s) AND :80; image
HEALTHCHECK uses it (WHP's --health-cmd override must too)
- scripts/test-gunicorn-socket-timeouts.py (12 tests, real gunicorn),
added to the Docker build gate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
32 lines
1.4 KiB
Bash
Executable File
32 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Container liveness probe: BOTH planes must answer quickly.
|
|
#
|
|
# 1. Management API (:8000) -- GET /health with a short --max-time. On
|
|
# 2026-10-09 (sdbees) the API workers' thread pools were wedged for long
|
|
# enough that /health hung 40s+, while the container reported "healthy"
|
|
# because the deployed --health-cmd only probed :80. A hung API is a dead
|
|
# control plane (every WHP site update times out), so it must turn the
|
|
# container unhealthy.
|
|
# 2. HAProxy data plane (:80) -- any HTTP answer at all (the default page,
|
|
# a redirect, a 4xx/5xx) proves HAProxy accepted and answered. Only a
|
|
# connect/timeout failure fails it, which is why there is no -f here.
|
|
#
|
|
# 127.0.0.1, not localhost: the API binds IPv4 only, and resolving localhost
|
|
# to ::1 first costs a failed connect on every probe.
|
|
#
|
|
# WHP's lib-edge-container.sh overrides the image HEALTHCHECK with its own
|
|
# --health-cmd; point that at this script so the fleet gets the same probe.
|
|
set -u
|
|
API_TIMEOUT="${HAPROXY_MGR_HEALTH_API_TIMEOUT:-4}"
|
|
EDGE_TIMEOUT="${HAPROXY_MGR_HEALTH_EDGE_TIMEOUT:-5}"
|
|
|
|
if ! curl -sf -o /dev/null --max-time "$API_TIMEOUT" http://127.0.0.1:8000/health; then
|
|
echo "management API :8000 /health failed or took longer than ${API_TIMEOUT}s"
|
|
exit 1
|
|
fi
|
|
if ! curl -s -o /dev/null --max-time "$EDGE_TIMEOUT" http://127.0.0.1/; then
|
|
echo "HAProxy :80 did not answer within ${EDGE_TIMEOUT}s"
|
|
exit 1
|
|
fi
|
|
exit 0
|