fix(api): bound all subprocess calls + run 2 workers to prevent API stall
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m9s

The management API wedged on whp01 2026-07-07: every panel call to the
manager (config regenerate during a WHP site update, SSL, even /health)
timed out at 30s while customer sites stayed up. Root cause: all four
gunicorn gthread worker threads were permanently blocked in socket reads
inside untimed subprocess.run() calls (certbot ACME / socat reloads). A
stalled external command holds its worker thread forever; gunicorn
--timeout can't rescue it (gthread only kills a worker whose main thread
stops heart-beating, and ours kept polling). Stalled calls accumulated
until the 4-thread pool was exhausted and the whole API went dark.

- Wrap subprocess.run with a default timeout (HAPROXY_MGR_SUBPROCESS_TIMEOUT,
  180s) so every external command is bounded and releases its thread on
  expiry via the existing per-endpoint try/except. Bounding by default
  covers all ~30 call sites and any future one.
- certbot renew keeps an explicit 900s timeout (walks every lineage).
- API_WORKERS default 1 -> 2: a single worker made a thread-pool wedge a
  total outage; a second worker keeps the API answering while one recycles.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-06 17:31:44 -07:00
parent 7732e2a2ff
commit 2a2b9739fc
3 changed files with 44 additions and 4 deletions
+7 -1
View File
@@ -47,7 +47,13 @@ HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
# (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
# 1000 — worker recycle frequency).
API_WORKERS="${HAPROXY_MGR_API_WORKERS:-1}"
#
# API_WORKERS default is 2 (was 1). A single worker is a single point of
# failure: if its gthread pool ever wedges (see the 2026-07-07 subprocess-hang
# incident — now bounded by DEFAULT_SUBPROCESS_TIMEOUT in haproxy_manager.py),
# the entire management API goes dark. A second worker keeps the API answering
# (config regenerate, health, SSL) while the other recycles via --max-requests.
API_WORKERS="${HAPROXY_MGR_API_WORKERS:-2}"
API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"