The management API wedged on whp01 2026-07-07: every panel call to the
manager (config regenerate during a WHP site update, SSL, even /health)
timed out at 30s while customer sites stayed up. Root cause: all four
gunicorn gthread worker threads were permanently blocked in socket reads
inside untimed subprocess.run() calls (certbot ACME / socat reloads). A
stalled external command holds its worker thread forever; gunicorn
--timeout can't rescue it (gthread only kills a worker whose main thread
stops heart-beating, and ours kept polling). Stalled calls accumulated
until the 4-thread pool was exhausted and the whole API went dark.
- Wrap subprocess.run with a default timeout (HAPROXY_MGR_SUBPROCESS_TIMEOUT,
180s) so every external command is bounded and releases its thread on
expiry via the existing per-endpoint try/except. Bounding by default
covers all ~30 call sites and any future one.
- certbot renew keeps an explicit 900s timeout (walks every lineage).
- API_WORKERS default 1 -> 2: a single worker made a thread-pool wedge a
total outage; a second worker keeps the API answering while one recycles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
haproxy runs as a background child of PID 1 (gunicorn) with nothing
watching it after init. If the haproxy master dies mid-life (observed
2026-07-01 on whp01: SIGABRT -> exit 134, reaped by gunicorn and logged
as "Worker (pid:22) exited"), the container stays "up", Docker's
--restart never fires, and haproxy is down until the external host
watchdog full-restarts the whole container minutes later (dropping every
connection).
Add an in-container supervisor loop in start-up.sh (Phase 1.5) that runs
scripts/ensure_haproxy.py every HAPROXY_SUPERVISOR_INTERVAL (default 15s).
ensure_haproxy.py calls the existing, idempotent start_haproxy() only when
haproxy isn't running (psutil guard), reviving it in place within one
interval with no container restart. Same entrypoint-supervision pattern
shipped for cac-litespeed.
Validated locally: killing haproxy -> revived with new PIDs in ~one
interval, container stayed healthy, no spurious restarts while healthy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related fixes for the issues the AI Monitor surfaced on whp01 on
2026-05-12 (haproxy-manager going "healthy but stalled" after long
uptime, and noise from POST /blocked-ip returning 405):
1. Production WSGI server. The Flask app was running on werkzeug's
built-in dev server (the one that prints "WARNING: This is a
development server" on every startup). werkzeug is single-threaded
and accumulates worker state over long uptimes; after ~24h on whp01
the health endpoint stops responding while the container still
reports "healthy" because Docker's HEALTHCHECK uses an HTTP probe
from inside the same werkzeug process that's stalled.
Replace with gunicorn (gthread worker class, --max-requests=1000
with jitter so workers recycle periodically). Two gunicorn instances,
one per Flask app — port 8000 for the management API, port 8080 for
the default/blocked-ip page server. Both lift their app objects from
the haproxy_manager module so gunicorn can import them.
Required structural change: default_app was created INSIDE the
__name__ == '__main__' block at module bottom, where gunicorn could
never reach it. Moved to module level. The __main__ block now stays
only for `python haproxy_manager.py` local-dev workflow.
Container init (init_db, certbot register, generate_config,
start_haproxy) extracted into a do_initial_setup() function called
from a new scripts/init.py. start-up.sh runs init.py to completion
before either gunicorn binds, which keeps HAProxy startup off the
WSGI workers' fork paths (no race between workers all trying to
start_haproxy() at once).
2. /blocked-ip and / accept ALL methods. HAProxy proxies blocked-IP
traffic to default_app preserving the original verb, so a blocked
POST request used to hit Flask's GET-only route and get a 405 +
the AI Monitor flagged the noise. Adding the full method list lets
the 403 page render regardless of verb.
Gunicorn settings tunable via env (workers, timeout, max-requests).
API gets --timeout 120 because ACME cert issuance can be slow; the
default page server stays on the gunicorn default 30s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Volume-mounted /etc/haproxy can shadow the image-baked
trusted_ips.list/trusted_ips.map, causing HAProxy to fail
config validation with "failed to open pattern file" on
non-WHP deployments. Touch empty files if they don't exist
so the ACLs always parse.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shadowdao