fix(shared-ols): unmapped Host gets 421, not a 200 that hides a dead site

The shared-OLS catch-all (`map _health *`) served html/index.html --
HTTP 200, 11 bytes, "shared-ols" -- to any Host no customer vhost claimed.
Three live customer sites (joshuaknapp.net, streamers.channel,
blog.anti-social.online) sat in exactly that state for ~2 months on whp01
and no monitor noticed, because every uptime check asks "is it 200?" and
it was. A tier-wide catch-all that answers 200 makes a missing vhost
indistinguishable from a working site.

An unmapped Host now gets 421 Misdirected Request with a short generic
body. 421 is semantically exact (the server cannot produce a response for
the requested authority) and, unlike 404, cannot be confused with a normal
answer from a real site.

The discriminator is the request path plus the client address, NOT the
Host -- the vhost is selected by the listener map, so by the time these
rules run the Host is no longer available to branch on:

  * `/healthz` from an internal client address (loopback, RFC1918) -> 200 "ok"
  * everything else, every path, every Host, both listeners -> 421

The 421 for `/` is UNCONDITIONAL: no header, source address or Host talks
this vhost into a 200 there, so the property the change exists to
guarantee does not rest on anything spoofable. The address gate only
hardens /healthz, and X-Forwarded-For cannot be used against it because
HAProxy replaces that header with the real client IP.

Health probes keep passing unchanged. Both forms were run against a
container carrying this change and both exit 0 with "ok":
  curl -fsSk https://127.0.0.1/healthz   (Dockerfile.shared-ols HEALTHCHECK)
  curl -sfk  https://localhost/healthz   (WHP setup-shared-ols.sh --health-cmd)
`docker inspect` reported healthy with failingStreak=0, on a container with
a customer site and on a zero-site container.

Measured on the lab VM against OLS 1.8.4 (the production base image):
  unmapped Host, `/`, :443 and :80   -> 421, 356 bytes, identical for every
                                        unmapped Host (no enumeration signal)
  unmapped Host, any deeper path     -> the same 421
  configured site, both names, :443/:80 -> 200, served normally
  litespeed -t                        -> 0 [ERROR] lines (warnings only, and
                                        only about the lab fixture's uid/gid)

Two OLS behaviours were measured rather than assumed, and both shaped the
implementation -- see the comment block in entrypoint-shared-ols.sh:
`context / { type redirect statusCode 421 }` silently degrades to a 302
with an unexpanded Location, and the `errorpage 421` body is fetched as a
fresh request through the same rewrite rules (so it needs a %{THE_REQUEST}
guard, since %{IS_SUBREQ} and %{ENV:REDIRECT_STATUS} are not populated).

The old index.html is removed, not just bypassed: if these rules ever
stopped applying, `context /` would fall back to the docRoot index, and
with no index.html that is a 403 -- wrong-but-loud, rather than a 200 that
is wrong-and-silent.

Known consumer to land alongside this: whp-monitoring's
probe_shared_ols_catchall() currently detects the catch-all by matching
`200` + body `shared-ols`, a signature this change deletes. It must also
accept 421, or the detector silently stops detecting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Dockerfile.shared-ols

@@ -24,9 +24,13 @@ FROM litespeedtech/openlitespeed:${OLS_VERSION}-lsphp${PHPVER}
 ##   - gettext-base: envsubst for render-shared-ols-config.sh
 ##   - openssl: self-signed cert for the :443 listener (HAProxy verifies none)
 ##   - curl/ca-certificates: HEALTHCHECK
+##   - procps: provides pgrep, which entrypoint-shared-ols.sh's ols_running()
+##     liveness check depends on. Only transitively present via the base image
+##     today (Ubuntu 24.04 pulls it in) — pin it explicitly so it can't be
+##     pruned as "unused" and silently break the supervisor's crash detection.
 RUN apt-get update && \
     DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
-        inotify-tools gettext-base openssl ca-certificates curl && \
+        inotify-tools gettext-base openssl ca-certificates curl procps && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*
 
@@ -51,6 +55,13 @@ EXPOSE 80 443
 
 ## Health: the entrypoint renders a catch-all _health vhost serving /healthz, so
 ## this passes from boot (zero customer sites) onward. Self-signed :443.
+##
+## MUST stay on /healthz, and must stay a LOOPBACK request. That vhost answers
+## 421 for every other path/Host so an unmapped customer hostname can never look
+## "up" to a monitor; /healthz answers 200 only for an internal client address
+## (loopback here). Probing `/` instead would fail the healthcheck and restart
+## the whole shared tier. WHP's setup-shared-ols.sh overrides this with the
+## equivalent `curl -sfk https://localhost/healthz`; keep the two in step.
 HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
   CMD curl -fsSk https://127.0.0.1/healthz || exit 1
 

scripts/entrypoint-shared-ols.sh

@@ -32,18 +32,127 @@ if [ ! -f "$CERT_FILE" ]; then
     -keyout "$KEY_FILE" -out "$CERT_FILE" -subj "/CN=shared-ols" 2>/dev/null
 fi
 
-## ---- health vhost (catch-all): valid server with zero customer sites +
-## answers HAProxy health checks that hit by IP / unknown Host with a 200 ----
+## ---- health vhost (catch-all) ----
+## This vhost is mapped `map _health *` by render-shared-ols-config.sh, so it
+## answers EVERY Host that no customer vhost claims. It exists so the server is
+## valid with zero customer sites and so local/edge health probes get a 200.
+##
+## IT MUST NOT ANSWER 200 FOR AN UNMAPPED CUSTOMER HOST.
+## It used to serve html/index.html ("shared-ols", 11 bytes) with HTTP 200 to
+## anything that fell through. Measured 2026-08: three live customer sites
+## (their vhost had silently stopped being rendered) served that 200 for ~2
+## months and no monitor noticed, because every uptime check asks "is it 200?"
+## and the answer was yes. A hostname this server cannot serve now gets
+## 421 Misdirected Request -- semantically exact (RFC 7540 s9.1.2: the server is
+## not able to produce a response for the combination of scheme and authority in
+## the request URI) and unambiguous to monitoring in a way 404 is not, since a
+## 404 is a perfectly normal answer from a real, working site.
+##
+## THE DISCRIMINATOR: request path /healthz AND an INTERNAL client address.
+##   * Path alone is not enough -- anyone can request /healthz.
+##   * REMOTE_ADDR is the half an outside caller cannot choose, BECAUSE of
+##     `useIpInProxyHeader 1` in httpd_config_base.tpl: OLS resolves the client
+##     IP from X-Forwarded-For, and HAProxy -- the only thing that can reach
+##     this tier, which has no host-published ports and sits on client-net --
+##     SETS (not appends) that header:
+##     `http-request set-header X-Forwarded-For %[var(txn.real_ip)]` in
+##     haproxy-manager-base/templates/hap_backend.tpl, which DISCARDS whatever
+##     the client sent. So a request arriving from outside carries the real
+##     public client IP. Verified on the lab: `-H 'X-Forwarded-For: 8.8.8.8'`
+##     on /healthz returns 421.
+##   * MEASURED LIMIT OF THE IP GATE, stated plainly rather than assumed away:
+##     OLS takes the FIRST element of a multi-value X-Forwarded-For as
+##     REMOTE_ADDR. `X-Forwarded-For: 10.0.0.1, 8.8.8.8` returns 200 on /healthz
+##     here, and anchoring the pattern ^...$ does NOT change that (tested both
+##     ways) -- because by the time the rule sees REMOTE_ADDR it is already the
+##     single token `10.0.0.1`. The anchors are kept because they are correct
+##     and free, not because they close that hole. What closes it is HAProxy:
+##     `http-request set-header X-Forwarded-For %[var(txn.real_ip)]` REPLACES
+##     whatever the client sent with one value.
+##   * AND THE GATE IS NOT LOAD-BEARING ANYWAY. It only guards /healthz. `/`,
+##     and every other path, is 421 UNCONDITIONALLY -- no header, source
+##     address or Host can talk this vhost into a 200 there. So even a total
+##     bypass of the IP gate buys an attacker a 3-byte `ok` on /healthz, never
+##     a "the site is up" answer on the URL a monitor actually requests. That
+##     is the property this change exists to guarantee, and it does not rest on
+##     anything spoofable.
+##   * The probes that MUST keep passing all originate inside: the Docker
+##     HEALTHCHECK (`curl -sfk https://127.0.0.1/healthz` in Dockerfile.shared-ols,
+##     overridden by WHP's setup-shared-ols.sh to `https://localhost/healthz`)
+##     connects over loopback and sends no X-Forwarded-For, so REMOTE_ADDR falls
+##     back to the peer, 127.0.0.1. An edge/host probe of the container IP comes
+##     from the docker gateway (172.16/12), also allowed.
+##
+## `/` is 421 for EVERY client, internal ones included -- there is deliberately
+## no "internal clients still get the old 200 page" escape hatch, because that
+## is exactly the response that hid the outage. Anything probing this tier for
+## liveness must ask for /healthz.
+##
+## WHY REWRITE AND NOT A REDIRECT CONTEXT: `context / { type redirect
+## statusCode 421 }` was measured on this image (OLS 1.8.4) and does NOT work --
+## 421 is not in OLS's accepted status-code list, so it silently degrades to a
+## 302 with a literal, unexpanded `Location: $DOC_ROOT/?`. A rewrite `[R=421,L]`
+## does emit a real 421.
+##
+## WHY THE THE_REQUEST GUARD ON THE ERROR PAGE: a bare [R=421] has no body, and
+## a bare 421 with no explanation is a support ticket. `errorpage 421` supplies
+## the body, but OLS fetches that URL as a fresh internal request that runs
+## through these same rules -- without an exception it is itself 421'd and the
+## body comes back empty (measured: content-length 0). %{IS_SUBREQ} and
+## %{ENV:REDIRECT_STATUS} are NOT populated by OLS's rewrite engine (both
+## measured, both no-ops), but %{THE_REQUEST} keeps the ORIGINAL request line
+## across the internal fetch. So: serve misdirected.html when the client did not
+## itself ask for it, which lets the error page render while a direct external
+## GET /misdirected.html still gets 421 -- no path on this catch-all answers 200
+## to an outside caller.
+##
+## The body is deliberately generic: no branding, no customer names, nothing
+## that reveals which hostnames this server does serve. Every unmapped Host and
+## every path gets the byte-identical 421, so the response cannot be used to
+## enumerate configured vs unconfigured hostnames.
 cat > "$HEALTH_DIR/vhconf.conf" <<'EOF'
 docRoot                   $VH_ROOT/html
 enableScript              0
+
+errorpage 421 {
+  url                     /misdirected.html
+}
+
+rewrite {
+  enable                  1
+  rules                   <<<END_rules
+  RewriteCond %{THE_REQUEST} !\s/+misdirected\.html
+  RewriteRule ^/?misdirected\.html$ - [L]
+  RewriteCond %{REMOTE_ADDR} ^(127\.0\.0\.1|::1|10\.[0-9.]+|192\.168\.[0-9.]+|172\.(1[6-9]|2[0-9]|3[01])\.[0-9.]+)$
+  RewriteRule ^/?healthz$ - [L]
+  RewriteRule .* - [R=421,L]
+  END_rules
+}
+
 context / {
   allowBrowse             1
   location                $DOC_ROOT/
 }
 EOF
 printf 'ok\n' > "$HEALTH_DIR/html/healthz"
-printf 'shared-ols\n' > "$HEALTH_DIR/html/index.html"
+cat > "$HEALTH_DIR/html/misdirected.html" <<'EOF'
+<!DOCTYPE html>
+<html lang="en">
+<head><meta charset="utf-8"><title>421 Misdirected Request</title></head>
+<body>
+<h1>421 Misdirected Request</h1>
+<p>This hostname is not configured on this server.</p>
+<p>If you own this domain, check that its DNS points to the correct server and
+that the site is active in your hosting control panel.</p>
+</body>
+</html>
+EOF
+## The old catch-all index.html ("shared-ols") is gone on purpose, and actively
+## removed so an in-place upgrade of a long-lived container cannot leave it
+## behind. If these rewrite rules were ever to stop applying, `context /` would
+## fall back to serving the docRoot index -- with no index.html that is a 403,
+## which is wrong-but-loud, instead of a 200 that is wrong-and-silent.
+rm -f "$HEALTH_DIR/html/index.html"
 
 ## ---- ownership: OLS reads conf/ as lsadm. chown the base conf dir + health dir
 ## NON-recursively (the per-site files under conf/shared-sites are written by the
@@ -51,7 +160,7 @@ printf 'shared-ols\n' > "$HEALTH_DIR/html/index.html"
 ## every container (re)start, delaying first-listen after a crash). The render
 ## script chowns the httpd_config.conf it produces. ----
 chown lsadm:nogroup "$LSWS_CONF" "$HEALTH_DIR" "$HEALTH_DIR/html" 2>/dev/null || true
-chown lsadm:nogroup "$HEALTH_DIR/vhconf.conf" "$HEALTH_DIR/html/healthz" "$HEALTH_DIR/html/index.html" 2>/dev/null || true
+chown lsadm:nogroup "$HEALTH_DIR/vhconf.conf" "$HEALTH_DIR/html/healthz" "$HEALTH_DIR/html/misdirected.html" 2>/dev/null || true
 
 ## ---- assemble httpd_config.conf from the panel's per-site files ----
 /scripts/render-shared-ols-config.sh
@@ -75,19 +184,47 @@ term_handler() {
 }
 trap term_handler TERM INT
 
-## Variable + here-string, not a pipe into `grep -qi` — see the long note on the
-## identical function in entrypoint-litespeed.sh: `grep -q` closing the pipe on
-## a match can leave the writer dying 141, and `set -o pipefail` (line 14) turns
-## that into "OLS is down" *because* the running line matched. The reason is
-## structural (a pipefail script must not pipe into an early-exit reader), not
-## that this particular output is small; and the here-string is safe here for
-## the separate reason that `lswsctrl status` is far below the size at which
-## bash spills a here-string to a temp file. A non-zero `lswsctrl` still counts
-## as not running, as pipefail made it count before.
+## NOT `lswsctrl status` (unlike the otherwise-identical function in
+## entrypoint-litespeed.sh). `lswsctrl` appends a timestamped line to
+## logs/lsrestart.log on EVERY invocation it makes, including `status` — and
+## this loop polls every 3s forever. Measured on whp01: lsrestart.log is 96 MB,
+## holding 1,819,286 `status` lines against 2,429 real `restart` lines; at one
+## poll per 3s that's ~63 days of continuous polling, which is exactly the
+## file's age, and it isn't rotated on any host (whp01/whp02/sdbees all growing
+## at ~1.5 MB/day). So: check liveness directly instead of shelling out to a
+## tool whose logging is a side effect we don't want on a fixed timer.
+##
+## Verified (docker run litespeedtech/openlitespeed:1.8.4-lsphp83, the exact
+## base this image is built FROM — see Dockerfile.shared-ols): the running main
+## process shows in `ps` as `openlitespeed (lshttpd - main)`, one PID, always
+## present while OLS is up and absent the instant it is killed (checked via
+## `ps aux` immediately after `kill -9` on the main PID). `pgrep -f` matches
+## against the full command line, and no other process on this image's `ps`
+## output contains that string, so this cannot cross-match an unrelated
+## process. It also cannot self-match: pgrep excludes its own PID by default,
+## and the invoking process here is bash executing this script file, whose own
+## argv never contains the pattern text (only the *source lines* of this script
+## do, which `pgrep -f` never sees).
+##
+## Deliberately NOT the pidfile (/tmp/lshttpd/lshttpd.pid, confirmed present in
+## the same probe): pidfiles are known to go stale across a crash (verified —
+## after `kill -9` the file still held the dead PID), and treating a stale PID
+## as "alive" if the kernel ever reuses that number is a false positive this
+## supervisor cannot afford (see below). `pgrep -f` reads the live process
+## table, so there is no staleness window to reason about.
+##
+## Conservative on both failure directions, which matters because this is a
+## supervisor predicate, not a metric: a false negative makes start_ols() run
+## `lswsctrl start` against an already-running OLS — verified against the same
+## probe base image, that is NOT a no-op, it sends SIGUSR1 to the live main
+## process, i.e. the same graceful self-restart QUIC.cloud IP refreshes trigger
+## (see entrypoint-litespeed.sh's note on that handoff) — a brief, zero-
+## downtime blip at worst. A false positive is worse: it leaves a genuinely
+## dead OLS un-revived until some later poll happens to notice. So if this
+## predicate is ever in doubt it should err toward reporting "not running", not
+## "running".
 ols_running() {
-  local st
-  st=$(/usr/local/lsws/bin/lswsctrl status 2>/dev/null) || return 1
-  grep -qi 'running with pid' <<<"$st"
+  pgrep -f 'lshttpd - main' >/dev/null 2>&1
 }
 
 MAX_STARTS=5

scripts/ols-htaccess-watcher.sh

@@ -15,6 +15,20 @@
 ## runs it and the panel monitors it (check-ols-htaccess-watcher.php).
 set -uo pipefail
 
+## WATCH_ROOT is deliberately left as the host-wide /mnt/users, not narrowed to
+## the shared-OLS tenant set, even though that set IS derivable in-container
+## (render-shared-ols-config.sh's $SITES_ROOT/*/site.meta VHROOT= is exactly
+## that list). Narrowing it would mean handing inotifywait a fixed argv list of
+## VHROOT dirs at process start — and inotifywait cannot be told to watch a NEW
+## directory once running. The panel provisions sites onto this container live,
+## between renders; a site added after the watcher started would then sit
+## outside every watch until the next container restart, i.e. exactly the
+## silent-failure mode (spec 7) this script exists to prevent, now for brand
+## new tenants instead of none. Doing this safely needs a reload path (SIGHUP
+## re-exec off the current site.meta list, coordinated with
+## render-shared-ols-config.sh) that does not exist yet and is its own change.
+## So: WATCH_ROOT stays broad, and correctness comes entirely from the path
+## match below, which is sufficient on its own.
 WATCH_ROOT="${OLS_WATCH_ROOT:-/mnt/users}"
 DEBOUNCE="${OLS_HTACCESS_DEBOUNCE:-15}"   # coalesce window (s)
 FLOOR="${OLS_HTACCESS_FLOOR:-60}"         # min seconds between restarts
@@ -24,16 +38,17 @@ last_restart=0
 log() { echo "ols-htaccess-watcher: $*" >&2; }
 
 do_restart() {
+  path="$1"
   now=$(date +%s)
   if [ $((now - last_restart)) -lt "$FLOOR" ]; then
-    log "within ${FLOOR}s floor — coalescing, skipping restart"
+    log "within ${FLOOR}s floor — coalescing, skipping restart ($path)"
     return
   fi
   if "$LSWSCTRL" restart >/dev/null 2>&1; then
     last_restart=$now
-    log "graceful restart issued (.htaccess change)"
+    log "graceful restart issued — $path changed"
   else
-    log "WARNING: lswsctrl restart failed"
+    log "WARNING: lswsctrl restart failed ($path)"
   fi
 }
 
@@ -41,18 +56,34 @@ if ! command -v inotifywait >/dev/null 2>&1; then
   log "FATAL: inotifywait not installed (inotify-tools)"; exit 1
 fi
 mkdir -p "$WATCH_ROOT"
-log "watching $WATCH_ROOT for .htaccess changes (debounce=${DEBOUNCE}s floor=${FLOOR}s)"
+log "watching $WATCH_ROOT for docroot (public_html) .htaccess changes (debounce=${DEBOUNCE}s floor=${FLOOR}s)"
 
-## -m monitor, -r recursive. We filter to .htaccess in the read loop rather than
-## --include so this works on older inotify-tools too. modify/create/delete/move
-## all matter (delete of .htaccess also changes rewrite behavior).
-inotifywait -m -r -e modify,create,delete,move "$WATCH_ROOT" --format '%f' 2>/dev/null |
-while read -r fname; do
-  case "$fname" in
-    .htaccess) ;;
+## -m monitor, -r recursive. We filter in the read loop rather than --include
+## so this works on older inotify-tools too. modify/create/delete/move all
+## matter (delete of .htaccess also changes rewrite behavior).
+##
+## --format '%w%f' (full path), NOT '%f' (basename only). OLS reads .htaccess
+## (RewriteFile) only from a vhost's DOCROOT — VHROOT, i.e.
+## /mnt/users/<user>/<domain>/public_html (see render-shared-ols-config.sh /
+## entrypoint-lsphp.sh) — never anything below it. A basename-only match fires
+## for ANY .htaccess anywhere under a tenant, at any depth, and WordPress
+## plugins write plenty of those that OLS never opens: measured on whp01 over
+## 24h, this watcher fired 63 restarts, of which the docroot .htaccess actually
+## changed in 0. All 28 distinct files behind those 63 were plugin guard files
+## — Wordfence self-healing waf/views/vendor/tmp/models/lib/.htaccess, W3 Total
+## Cache writing one per cached URL under wp-content/cache/page_enhanced/, plus
+## WPForms/Gravity Forms/UpdraftPlus/Groundhogg/WP Staging upload guards — and
+## most of those tenants are on the shared Apache tier (cac-fpm), not this OLS
+## tier at all, so their cache churn was restarting the OLS serving 15 unrelated
+## tenants for no reason. Matching the full path down to /public_html/.htaccess
+## is what actually ties a change to something OLS will reread.
+inotifywait -m -r -e modify,create,delete,move "$WATCH_ROOT" --format '%w%f' 2>/dev/null |
+while read -r path; do
+  case "$path" in
+    */public_html/.htaccess) ;;
     *) continue ;;
   esac
-  ## A tenant .htaccess changed. Coalesce the save-burst, then restart ONCE.
+  ## A tenant DOCROOT .htaccess changed. Coalesce the save-burst, then restart ONCE.
   ##
   ## The coalesce is HARD-BOUNDED to DEBOUNCE seconds: a previous version blocked
   ## on `read -t DEBOUNCE` which, on a busy multi-tenant server, never timed out
@@ -69,5 +100,5 @@ while read -r fname; do
       break      # ~2s of total quiet — the burst has settled
     fi
   done
-  do_restart
+  do_restart "$path"
 done

scripts/render-shared-ols-config.sh

@@ -156,8 +156,25 @@ for meta in "$SITES_ROOT"/*/site.meta; do
 done
 
 ## --- 5. ALWAYS add a health vhost mapped to the catch-all so the server is
-## valid with zero customer sites and HAProxy health checks (which hit by IP /
-## unknown Host) get a 200. Exact-domain maps above win over this '*'. ---
+## valid with zero customer sites. Exact-domain maps above win over this '*'.
+##
+## THIS MAP IS WHY AN UNMAPPED HOST GETS AN ANSWER AT ALL. Anything the loop
+## above did not emit a `map` for -- a customer domain whose site dir went
+## missing, a stale DNS record, a scanner probing by IP -- lands here. It used
+## to answer 200 with an 11-byte "shared-ols" body, which is how three live
+## customer sites stayed silently broken for ~2 months: every uptime monitor
+## asks "is it 200?" and it was.
+##
+## The health vhost (its vhconf.conf is written by entrypoint-shared-ols.sh,
+## which carries the full rationale) now answers 421 Misdirected Request with a
+## short generic body for any Host it cannot serve, and keeps 200 ONLY for
+## GET /healthz from an internal client address -- the Docker HEALTHCHECK and
+## edge liveness probes. Do NOT reintroduce a 200 here for `/`: probe /healthz.
+##
+## The listener `map` itself is unchanged, deliberately. Dropping the catch-all
+## instead would make OLS answer an unmapped Host from whichever vhost it
+## considers first, which is worse: an unmapped Host would be served SOMEONE
+## ELSE'S SITE. ---
 {
   echo ""
   echo "virtualhost _health {"