shadowdao
2a2b9739fc
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 1m9s
The management API wedged on whp01 2026-07-07: every panel call to the manager (config regenerate during a WHP site update, SSL, even /health) timed out at 30s while customer sites stayed up. Root cause: all four gunicorn gthread worker threads were permanently blocked in socket reads inside untimed subprocess.run() calls (certbot ACME / socat reloads). A stalled external command holds its worker thread forever; gunicorn --timeout can't rescue it (gthread only kills a worker whose main thread stops heart-beating, and ours kept polling). Stalled calls accumulated until the 4-thread pool was exhausted and the whole API went dark. - Wrap subprocess.run with a default timeout (HAPROXY_MGR_SUBPROCESS_TIMEOUT, 180s) so every external command is bounded and releases its thread on expiry via the existing per-endpoint try/except. Bounding by default covers all ~30 call sites and any future one. - certbot renew keeps an explicit 900s timeout (walks every lineage). - API_WORKERS default 1 -> 2: a single worker made a thread-pool wedge a total outage; a second worker keeps the API answering while one recycles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 lines
10 B
Plaintext
2 lines
10 B
Plaintext
2026.07.3
|