cloud-hosting-platform/cloud-apache-container
4
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(shared-ols): unmapped Host gets 421, not a 200 that hides a dead site #22

Open
jknapp wants to merge 1 commits from fix/shared-ols-catchall-421 into trunk
pull from: fix/shared-ols-catchall-421
merge into: :trunk
Conversation 1 Commits 1 Files Changed 3
+139 -6
jknapp commented 2026-08-23 02:45:08 +00:00
Owner
Copy Link
Copy Source

What

The shared-OpenLiteSpeed catch-all (map _health *) answered HTTP 200 with an 11-byte shared-ols body for any Host no customer vhost claimed. An unmapped Host now gets 421 Misdirected Request with a short generic page; /healthz from an internal client address still gets 200 ok.

Three real customer sites (joshuaknapp.net, streamers.channel, blog.anti-social.online) sat in exactly that state on whp01 for roughly two months without a single alert, because every uptime monitor asks "did it return 200?" and the answer was yes.

The discriminator, and why it cannot be bypassed

The vhost is selected by the listener map, so by the time these rules run the Host is gone as a discriminator. The rules branch on request path plus client address instead:

request result
/healthz from loopback / RFC1918 200 ok
everything else — any path, any Host, :80 and :443 421 + body

/ is 421 unconditionally. No header, source address or Host talks this vhost into a 200 there, so the property this change exists to guarantee does not rest on anything spoofable. The address gate only hardens /healthz, and it holds because useIpInProxyHeader 1 makes OLS read the client IP from X-Forwarded-For, which HAProxy replaces (http-request set-header X-Forwarded-For %[var(txn.real_ip)]) with the real client IP. Measured: -H 'X-Forwarded-For: 8.8.8.8' on /healthz returns 421.

One limit is documented in the code rather than assumed away: OLS takes the first element of a multi-value X-Forwarded-For, so 10.0.0.1, 8.8.8.8 does reach 200 on /healthz (anchoring the pattern does not change this — tested both ways). It cannot arrive through the edge because HAProxy sets a single value, and even a total bypass buys a 3-byte ok on /healthz, never a "site is up" answer on /.

Health checks still pass — run, not reasoned about

Both forms, executed against a container carrying this change:

$ docker exec <c> sh -c 'curl -fsSk https://127.0.0.1/healthz || exit 1'   # Dockerfile HEALTHCHECK
ok
   exit=0
$ docker exec <c> sh -c 'curl -sfk https://localhost/healthz || exit 1'    # WHP setup-shared-ols.sh --health-cmd
ok
   exit=0

docker health: healthy failingStreak=0
docker health (zero-site container): healthy failingStreak=0
  exit=0 out='ok\n'
  exit=0 out='ok\n'
  exit=0 out='ok\n'

Verification on the lab VM (OLS 1.8.4, the production base image)

unmapped /                 -> code=421 size=356 ct=text/html
unmapped /wp-login.php     -> code=421 size=356
unmapped :80 /             -> code=421 size=356
configured / (https)       -> code=200 body=CONFIGURED-SITE-OK
configured www / (https)   -> code=200 body=CONFIGURED-SITE-OK
configured / (http :80)    -> code=200 body=CONFIGURED-SITE-OK
zero-site container, unmapped / -> code=421 size=356

litespeed -t: 0 [ERROR] lines   (WARNs only, and only about the lab fixture's uid/gid)

Enumeration check — every unmapped Host returns the byte-identical response, so it cannot be used to tell configured from unconfigured hostnames:

aaa.example       code=421 size=356 type=text/html
zzz.example       code=421 size=356 type=text/html
joshuaknapp.net   code=421 size=356 type=text/html
192.0.2.1         code=421 size=356 type=text/html

The 421 body verbatim:

HTTP/2 421
content-type: text/html
content-length: 356

<!DOCTYPE html>
<html lang="en">
<head><meta charset="utf-8"><title>421 Misdirected Request</title></head>
<body>
<h1>421 Misdirected Request</h1>
<p>This hostname is not configured on this server.</p>
<p>If you own this domain, check that its DNS points to the correct server and
that the site is active in your hosting control panel.</p>
</body>
</html>

No branding, no customer names, nothing that reveals which hostnames this server does serve.

Two OLS behaviours measured, not assumed

  • context / { type redirect statusCode 421 } does not work — 421 is not in OLS's accepted status-code list, so it silently degrades to a 302 with a literal, unexpanded Location: $DOC_ROOT/?. A rewrite [R=421,L] emits a real 421.
  • The errorpage 421 body is fetched as a fresh request through the same rewrite rules, so without an exception it is itself 421'd and the body comes back empty (content-length 0). %{IS_SUBREQ} and %{ENV:REDIRECT_STATUS} are not populated by OLS's rewrite engine (both tested, both no-ops); %{THE_REQUEST} does survive the internal fetch, so the guard keys on that. A direct external GET /misdirected.html therefore still gets 421 — no path on this catch-all answers 200 to an outside caller.

The old index.html is removed, not merely bypassed: if these rules ever stopped applying, context / would fall back to the docRoot index, and with no index.html that is a 403 — wrong-but-loud, rather than a 200 that is wrong-and-silent.

Consumers of the catch-all, and what happens to each

Audited across whp, haproxy-manager-base and whp-monitoring.

Unaffected

  • HAProxy backend health check — the highest-risk consumer, and it turns out not to be one. The shared_ols server line is server shared-ols-01 shared-ols-01:443 check ssl verify none … with no option httpchk anywhere in the assembled config, so HAProxy does a TCP+TLS connect and never sends an HTTP request. A 421 is invisible to it. (whp/web-files/libs/haproxy_manager.php, haproxy-manager-base/templates/hap_backend.tpl, hap_header.tpl.)
  • Docker healthcheck (both forms) — verified above.
  • ACME http-01 — HAProxy's letsencrypt-acl is rendered before the per-domain ACLs and always wins; challenges never reach this tier.
  • haproxy-manager / edge / watchdog health probes — they hit the HAProxy frontend, not OLS.
  • shared-httpd /ping, site PHP-wedge probe, boot orchestrator, verify-host.sh, log shipping, the render/reload path, the .htaccess watcher — different tier, or docker exec/filesystem only, no HTTP.
  • In-container dotfile/backup deny probes — they use a real customer Host and a non-/ path; they already got 404 when the vhost was missing and now get 421, and report "could not verify" either way.
  • WP auto-rollback gate — classifyBrokenResponse() only trips on >=500, code 0, or 200-with-empty-body. No rollback storm.

Newly loud — and correctly so, since these sites really are broken

  • WpRollbackManager.php post-restore verification — the http_check step now fails instead of passing on a catch-all 200.
  • wp-campaign-followup.php / WpCampaignFollowup.php — remediation is marked recreate_failed instead of silently passing.
  • process-log-review.php — its $httpOk = code >= 200 && code < 400 short-circuit no longer fires for these sites, so the AI review actually runs. Directionally right; budget for the tokens.

Needs a companion change — please do not merge this alone

  • whp-monitoring/scripts/whp-monitor-poll.py, probe_shared_ols_catchall() detects the catch-all by matching 200 + body shared-ols — the exact signature this PR deletes. It must also accept 421 (and scripts/tests/test_shared_ols_catchall.py plus its fixture encode the same assumption). The status is 421 as specified, so that check needs no redefinition, only the added branch.
  • scheduled-site-checks.php / site_health_helpers.php treat any code >= 200 && < 500 as up, so WHP's own site monitor still will not flag these sites after this change. Raising the 421 is necessary but not sufficient; that threshold is a separate decision.

Divergence worth a deliberate call

The shared-httpd twin (whp-wt-default-vhost/web-files/configs/shared-httpd-default-vhost.conf) gives its catch-all a Require all denied — 403 — with RewriteRule "^/ping/?$" - [R=204,L] for its healthcheck. After this PR the two shared tiers disagree on what an unmapped Host gets (403 vs 421). 421 is the better answer of the two and this PR implements what was asked for; flagging it so the divergence is chosen rather than drifted into.

Testing

Lab only — a throwaway container built from shared-ols:latest on the internal test VM, with the modified scripts copied in and the container restarted so the real entrypoint code path ran. Both lab containers were removed afterwards. Nothing on any production host was touched.

🤖 Generated with Claude Code

## What The shared-OpenLiteSpeed catch-all (`map _health *`) answered **HTTP 200** with an 11-byte `shared-ols` body for any Host no customer vhost claimed. An unmapped Host now gets **421 Misdirected Request** with a short generic page; `/healthz` from an internal client address still gets `200 ok`. Three real customer sites (`joshuaknapp.net`, `streamers.channel`, `blog.anti-social.online`) sat in exactly that state on whp01 for roughly two months without a single alert, because every uptime monitor asks "did it return 200?" and the answer was yes. ## The discriminator, and why it cannot be bypassed The vhost is selected by the listener `map`, so by the time these rules run **the Host is gone as a discriminator**. The rules branch on request path plus client address instead: | request | result | |---|---| | `/healthz` from loopback / RFC1918 | `200 ok` | | everything else — any path, any Host, `:80` and `:443` | `421` + body | `/` is **421 unconditionally**. No header, source address or Host talks this vhost into a 200 there, so the property this change exists to guarantee does not rest on anything spoofable. The address gate only hardens `/healthz`, and it holds because `useIpInProxyHeader 1` makes OLS read the client IP from `X-Forwarded-For`, which HAProxy **replaces** (`http-request set-header X-Forwarded-For %[var(txn.real_ip)]`) with the real client IP. Measured: `-H 'X-Forwarded-For: 8.8.8.8'` on `/healthz` returns 421. One limit is documented in the code rather than assumed away: OLS takes the **first** element of a multi-value `X-Forwarded-For`, so `10.0.0.1, 8.8.8.8` does reach `200` on `/healthz` (anchoring the pattern does not change this — tested both ways). It cannot arrive through the edge because HAProxy sets a single value, and even a total bypass buys a 3-byte `ok` on `/healthz`, never a "site is up" answer on `/`. ## Health checks still pass — run, not reasoned about Both forms, executed against a container carrying this change: ``` $ docker exec <c> sh -c 'curl -fsSk https://127.0.0.1/healthz || exit 1' # Dockerfile HEALTHCHECK ok exit=0 $ docker exec <c> sh -c 'curl -sfk https://localhost/healthz || exit 1' # WHP setup-shared-ols.sh --health-cmd ok exit=0 docker health: healthy failingStreak=0 docker health (zero-site container): healthy failingStreak=0 exit=0 out='ok\n' exit=0 out='ok\n' exit=0 out='ok\n' ``` ## Verification on the lab VM (OLS 1.8.4, the production base image) ``` unmapped / -> code=421 size=356 ct=text/html unmapped /wp-login.php -> code=421 size=356 unmapped :80 / -> code=421 size=356 configured / (https) -> code=200 body=CONFIGURED-SITE-OK configured www / (https) -> code=200 body=CONFIGURED-SITE-OK configured / (http :80) -> code=200 body=CONFIGURED-SITE-OK zero-site container, unmapped / -> code=421 size=356 litespeed -t: 0 [ERROR] lines (WARNs only, and only about the lab fixture's uid/gid) ``` Enumeration check — every unmapped Host returns the byte-identical response, so it cannot be used to tell configured from unconfigured hostnames: ``` aaa.example code=421 size=356 type=text/html zzz.example code=421 size=356 type=text/html joshuaknapp.net code=421 size=356 type=text/html 192.0.2.1 code=421 size=356 type=text/html ``` The 421 body verbatim: ``` HTTP/2 421 content-type: text/html content-length: 356 <!DOCTYPE html> <html lang="en"> <head><meta charset="utf-8"><title>421 Misdirected Request</title></head> <body> <h1>421 Misdirected Request</h1> <p>This hostname is not configured on this server.</p> <p>If you own this domain, check that its DNS points to the correct server and that the site is active in your hosting control panel.</p> </body> </html> ``` No branding, no customer names, nothing that reveals which hostnames this server does serve. ## Two OLS behaviours measured, not assumed * `context / { type redirect statusCode 421 }` **does not work** — 421 is not in OLS's accepted status-code list, so it silently degrades to a 302 with a literal, unexpanded `Location: $DOC_ROOT/?`. A rewrite `[R=421,L]` emits a real 421. * The `errorpage 421` body is fetched as a **fresh request through the same rewrite rules**, so without an exception it is itself 421'd and the body comes back empty (content-length 0). `%{IS_SUBREQ}` and `%{ENV:REDIRECT_STATUS}` are not populated by OLS's rewrite engine (both tested, both no-ops); `%{THE_REQUEST}` does survive the internal fetch, so the guard keys on that. A direct external `GET /misdirected.html` therefore still gets 421 — no path on this catch-all answers 200 to an outside caller. The old `index.html` is **removed**, not merely bypassed: if these rules ever stopped applying, `context /` would fall back to the docRoot index, and with no index.html that is a 403 — wrong-but-loud, rather than a 200 that is wrong-and-silent. ## Consumers of the catch-all, and what happens to each Audited across `whp`, `haproxy-manager-base` and `whp-monitoring`. **Unaffected** * **HAProxy backend health check** — the highest-risk consumer, and it turns out not to be one. The shared_ols server line is `server shared-ols-01 shared-ols-01:443 check ssl verify none …` with **no `option httpchk`** anywhere in the assembled config, so HAProxy does a TCP+TLS connect and never sends an HTTP request. A 421 is invisible to it. (`whp/web-files/libs/haproxy_manager.php`, `haproxy-manager-base/templates/hap_backend.tpl`, `hap_header.tpl`.) * **Docker healthcheck** (both forms) — verified above. * **ACME http-01** — HAProxy's `letsencrypt-acl` is rendered before the per-domain ACLs and always wins; challenges never reach this tier. * **haproxy-manager / edge / watchdog health probes** — they hit the HAProxy frontend, not OLS. * **shared-httpd `/ping`, site PHP-wedge probe, boot orchestrator, `verify-host.sh`, log shipping, the render/reload path, the `.htaccess` watcher** — different tier, or `docker exec`/filesystem only, no HTTP. * **In-container dotfile/backup deny probes** — they use a real customer Host and a non-`/` path; they already got 404 when the vhost was missing and now get 421, and report "could not verify" either way. * **WP auto-rollback gate** — `classifyBrokenResponse()` only trips on >=500, code 0, or 200-with-empty-body. No rollback storm. **Newly loud — and correctly so, since these sites really are broken** * `WpRollbackManager.php` post-restore verification — the `http_check` step now fails instead of passing on a catch-all 200. * `wp-campaign-followup.php` / `WpCampaignFollowup.php` — remediation is marked `recreate_failed` instead of silently passing. * `process-log-review.php` — its `$httpOk = code >= 200 && code < 400` short-circuit no longer fires for these sites, so the AI review actually runs. Directionally right; budget for the tokens. **Needs a companion change — please do not merge this alone** * `whp-monitoring/scripts/whp-monitor-poll.py`, `probe_shared_ols_catchall()` detects the catch-all by matching `200` + body `shared-ols` — **the exact signature this PR deletes**. It must also accept 421 (and `scripts/tests/test_shared_ols_catchall.py` plus its fixture encode the same assumption). The status is 421 as specified, so that check needs no redefinition, only the added branch. * `scheduled-site-checks.php` / `site_health_helpers.php` treat any code `>= 200 && < 500` as up, so **WHP's own site monitor still will not flag these sites** after this change. Raising the 421 is necessary but not sufficient; that threshold is a separate decision. ## Divergence worth a deliberate call The shared-httpd twin (`whp-wt-default-vhost/web-files/configs/shared-httpd-default-vhost.conf`) gives its catch-all a `Require all denied` — **403** — with `RewriteRule "^/ping/?$" - [R=204,L]` for its healthcheck. After this PR the two shared tiers disagree on what an unmapped Host gets (403 vs 421). 421 is the better answer of the two and this PR implements what was asked for; flagging it so the divergence is chosen rather than drifted into. ## Testing Lab only — a throwaway container built from `shared-ols:latest` on the internal test VM, with the modified scripts copied in and the container restarted so the real entrypoint code path ran. Both lab containers were removed afterwards. Nothing on any production host was touched. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
jknapp added 1 commit 2026-08-23 02:45:08 +00:00
fix(shared-ols): unmapped Host gets 421, not a 200 that hides a dead site 11c02d94ec 
The shared-OLS catch-all (`map _health *`) served html/index.html --
HTTP 200, 11 bytes, "shared-ols" -- to any Host no customer vhost claimed.
Three live customer sites (joshuaknapp.net, streamers.channel,
blog.anti-social.online) sat in exactly that state for ~2 months on whp01
and no monitor noticed, because every uptime check asks "is it 200?" and
it was. A tier-wide catch-all that answers 200 makes a missing vhost
indistinguishable from a working site.

An unmapped Host now gets 421 Misdirected Request with a short generic
body. 421 is semantically exact (the server cannot produce a response for
the requested authority) and, unlike 404, cannot be confused with a normal
answer from a real site.

The discriminator is the request path plus the client address, NOT the
Host -- the vhost is selected by the listener map, so by the time these
rules run the Host is no longer available to branch on:

  * `/healthz` from an internal client address (loopback, RFC1918) -> 200 "ok"
  * everything else, every path, every Host, both listeners -> 421

The 421 for `/` is UNCONDITIONAL: no header, source address or Host talks
this vhost into a 200 there, so the property the change exists to
guarantee does not rest on anything spoofable. The address gate only
hardens /healthz, and X-Forwarded-For cannot be used against it because
HAProxy replaces that header with the real client IP.

Health probes keep passing unchanged. Both forms were run against a
container carrying this change and both exit 0 with "ok":
  curl -fsSk https://127.0.0.1/healthz   (Dockerfile.shared-ols HEALTHCHECK)
  curl -sfk  https://localhost/healthz   (WHP setup-shared-ols.sh --health-cmd)
`docker inspect` reported healthy with failingStreak=0, on a container with
a customer site and on a zero-site container.

Measured on the lab VM against OLS 1.8.4 (the production base image):
  unmapped Host, `/`, :443 and :80   -> 421, 356 bytes, identical for every
                                        unmapped Host (no enumeration signal)
  unmapped Host, any deeper path     -> the same 421
  configured site, both names, :443/:80 -> 200, served normally
  litespeed -t                        -> 0 [ERROR] lines (warnings only, and
                                        only about the lab fixture's uid/gid)

Two OLS behaviours were measured rather than assumed, and both shaped the
implementation -- see the comment block in entrypoint-shared-ols.sh:
`context / { type redirect statusCode 421 }` silently degrades to a 302
with an unexpanded Location, and the `errorpage 421` body is fetched as a
fresh request through the same rewrite rules (so it needs a %{THE_REQUEST}
guard, since %{IS_SUBREQ} and %{ENV:REDIRECT_STATUS} are not populated).

The old index.html is removed, not just bypassed: if these rules ever
stopped applying, `context /` would fall back to the docRoot index, and
with no index.html that is a 403 -- wrong-but-loud, rather than a 200 that
is wrong-and-silent.

Known consumer to land alongside this: whp-monitoring's
probe_shared_ols_catchall() currently detects the catch-all by matching
`200` + body `shared-ols`, a signature this change deletes. It must also
accept 421, or the detector silently stops detecting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jknapp commented 2026-08-23 03:06:20 +00:00
Author
Owner
Copy Link
Copy Source

Shelved, not rejected — deliberately left open.

A per-site page monitor (customer-supplied path + expected text, checked hourly) is being built instead. It asserts on content the customer actually cares about, which detects this failure class more directly than a status code, and it needs no fleet-wide image change.

What this PR still uniquely buys, if it is ever revived:

  • Unmonitored sites. The page monitor is subscription-scoped. A customer running their own external uptime check would see a 421 and know; with a 200 they never find out.
  • Correctness. Returning 200 for a Host the server cannot serve is simply wrong.

Costs that argued against shipping it now: it is a shared-image change every host picks up on its next recreate; WHP's own site checks treat >= 200 && < 500 as up so 421 would read as healthy without a second change; it leaves the two shared tiers disagreeing (shared-httpd returns 403); and the fleet drift audit found zero current instances for it to catch.

The work is complete and measured — discriminator, healthcheck proof, catch-all consumer survey and the multi-value XFF limit are all documented above. Revive if the tier divergence gets settled or unmonitored-site coverage becomes a priority.

**Shelved, not rejected** — deliberately left open. A per-site page monitor (customer-supplied path + expected text, checked hourly) is being built instead. It asserts on content the customer actually cares about, which detects this failure class more directly than a status code, and it needs no fleet-wide image change. What this PR still uniquely buys, if it is ever revived: - **Unmonitored sites.** The page monitor is subscription-scoped. A customer running their own external uptime check would see a 421 and know; with a 200 they never find out. - **Correctness.** Returning 200 for a Host the server cannot serve is simply wrong. Costs that argued against shipping it now: it is a shared-image change every host picks up on its next recreate; WHP's own site checks treat `>= 200 && < 500` as up so 421 would read as healthy without a second change; it leaves the two shared tiers disagreeing (shared-httpd returns 403); and the fleet drift audit found **zero** current instances for it to catch. The work is complete and measured — discriminator, healthcheck proof, catch-all consumer survey and the multi-value XFF limit are all documented above. Revive if the tier divergence gets settled or unmonitored-site coverage becomes a priority.
You are not authorized to merge this pull request.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/shared-ols-catchall-421:fix/shared-ols-catchall-421
git checkout fix/shared-ols-catchall-421
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jknapp whmcs-sam whp-cicd-user whp-deploy
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cloud-hosting-platform/cloud-apache-container#22