shadowdaoandClaude Opus 5
cf6936e225
ols-htaccess-watcher.sh matched .htaccess by basename only, so ANY .htaccess under a tenant (WordPress plugin guard files, not just the docroot OLS reads) triggered a full graceful restart. Measured on whp01 over 24h: 63 restarts, 0 of them from a docroot .htaccess actually changing — all from Wordfence/W3TC/ WPForms/etc. self-healing files, mostly on tenants that aren't even on this tier. Now matches the full path (%w%f) against */public_html/.htaccess, the only .htaccess OLS ever reads, and logs which path triggered each restart. entrypoint-shared-ols.sh's 3s supervisor poll called `lswsctrl status`, which appends a line to lsrestart.log on every invocation. Measured on whp01: 1,819,286 status lines vs 2,429 real restarts in a 96 MB, never-rotated log. ols_running() now checks the process table directly (pgrep -f 'lshttpd - main', verified against the litespeedtech/openlitespeed base image) instead of shelling out to a logging tool on a fixed timer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
105 lines
5.2 KiB
Bash
105 lines
5.2 KiB
Bash
#!/usr/bin/env bash
|
|
## ols-htaccess-watcher.sh — graceful-restart the shared OLS when any tenant's
|
|
## .htaccess changes. OLS reads .htaccess (RewriteFile) at (re)start, NOT per
|
|
## request, so without this a WordPress permalink/LiteSpeed-Cache change would
|
|
## silently not take effect. Required by spec 5.3.
|
|
##
|
|
## Watches all docroots for .htaccess writes, COALESCES bursts (a WP plugin save
|
|
## touches the file several times) within a debounce window, and RATE-LIMITS to
|
|
## a floor (one restart per FLOOR seconds) so many tenants saving at once can't
|
|
## trigger a restart storm. Debounce/floor are env-tunable (panel discloses the
|
|
## resulting "~60s" window to customers).
|
|
##
|
|
## Failure of THIS process is the silent-ticket failure mode (spec 7): if it
|
|
## dies, tenants' rewrite changes stop applying with no error. The entrypoint
|
|
## runs it and the panel monitors it (check-ols-htaccess-watcher.php).
|
|
set -uo pipefail
|
|
|
|
## WATCH_ROOT is deliberately left as the host-wide /mnt/users, not narrowed to
|
|
## the shared-OLS tenant set, even though that set IS derivable in-container
|
|
## (render-shared-ols-config.sh's $SITES_ROOT/*/site.meta VHROOT= is exactly
|
|
## that list). Narrowing it would mean handing inotifywait a fixed argv list of
|
|
## VHROOT dirs at process start — and inotifywait cannot be told to watch a NEW
|
|
## directory once running. The panel provisions sites onto this container live,
|
|
## between renders; a site added after the watcher started would then sit
|
|
## outside every watch until the next container restart, i.e. exactly the
|
|
## silent-failure mode (spec 7) this script exists to prevent, now for brand
|
|
## new tenants instead of none. Doing this safely needs a reload path (SIGHUP
|
|
## re-exec off the current site.meta list, coordinated with
|
|
## render-shared-ols-config.sh) that does not exist yet and is its own change.
|
|
## So: WATCH_ROOT stays broad, and correctness comes entirely from the path
|
|
## match below, which is sufficient on its own.
|
|
WATCH_ROOT="${OLS_WATCH_ROOT:-/mnt/users}"
|
|
DEBOUNCE="${OLS_HTACCESS_DEBOUNCE:-15}" # coalesce window (s)
|
|
FLOOR="${OLS_HTACCESS_FLOOR:-60}" # min seconds between restarts
|
|
LSWSCTRL=/usr/local/lsws/bin/lswsctrl
|
|
last_restart=0
|
|
|
|
log() { echo "ols-htaccess-watcher: $*" >&2; }
|
|
|
|
do_restart() {
|
|
path="$1"
|
|
now=$(date +%s)
|
|
if [ $((now - last_restart)) -lt "$FLOOR" ]; then
|
|
log "within ${FLOOR}s floor — coalescing, skipping restart ($path)"
|
|
return
|
|
fi
|
|
if "$LSWSCTRL" restart >/dev/null 2>&1; then
|
|
last_restart=$now
|
|
log "graceful restart issued — $path changed"
|
|
else
|
|
log "WARNING: lswsctrl restart failed ($path)"
|
|
fi
|
|
}
|
|
|
|
if ! command -v inotifywait >/dev/null 2>&1; then
|
|
log "FATAL: inotifywait not installed (inotify-tools)"; exit 1
|
|
fi
|
|
mkdir -p "$WATCH_ROOT"
|
|
log "watching $WATCH_ROOT for docroot (public_html) .htaccess changes (debounce=${DEBOUNCE}s floor=${FLOOR}s)"
|
|
|
|
## -m monitor, -r recursive. We filter in the read loop rather than --include
|
|
## so this works on older inotify-tools too. modify/create/delete/move all
|
|
## matter (delete of .htaccess also changes rewrite behavior).
|
|
##
|
|
## --format '%w%f' (full path), NOT '%f' (basename only). OLS reads .htaccess
|
|
## (RewriteFile) only from a vhost's DOCROOT — VHROOT, i.e.
|
|
## /mnt/users/<user>/<domain>/public_html (see render-shared-ols-config.sh /
|
|
## entrypoint-lsphp.sh) — never anything below it. A basename-only match fires
|
|
## for ANY .htaccess anywhere under a tenant, at any depth, and WordPress
|
|
## plugins write plenty of those that OLS never opens: measured on whp01 over
|
|
## 24h, this watcher fired 63 restarts, of which the docroot .htaccess actually
|
|
## changed in 0. All 28 distinct files behind those 63 were plugin guard files
|
|
## — Wordfence self-healing waf/views/vendor/tmp/models/lib/.htaccess, W3 Total
|
|
## Cache writing one per cached URL under wp-content/cache/page_enhanced/, plus
|
|
## WPForms/Gravity Forms/UpdraftPlus/Groundhogg/WP Staging upload guards — and
|
|
## most of those tenants are on the shared Apache tier (cac-fpm), not this OLS
|
|
## tier at all, so their cache churn was restarting the OLS serving 15 unrelated
|
|
## tenants for no reason. Matching the full path down to /public_html/.htaccess
|
|
## is what actually ties a change to something OLS will reread.
|
|
inotifywait -m -r -e modify,create,delete,move "$WATCH_ROOT" --format '%w%f' 2>/dev/null |
|
|
while read -r path; do
|
|
case "$path" in
|
|
*/public_html/.htaccess) ;;
|
|
*) continue ;;
|
|
esac
|
|
## A tenant DOCROOT .htaccess changed. Coalesce the save-burst, then restart ONCE.
|
|
##
|
|
## The coalesce is HARD-BOUNDED to DEBOUNCE seconds: a previous version blocked
|
|
## on `read -t DEBOUNCE` which, on a busy multi-tenant server, never timed out
|
|
## (unrelated file writes under $WATCH_ROOT kept resetting it) — so the restart
|
|
## was starved and rewrite changes silently never applied. Here we read further
|
|
## events only until the deadline OR ~2s of total quiet, whichever comes first,
|
|
## so continuous activity can delay us by at most DEBOUNCE. do_restart's FLOOR
|
|
## then rate-limits across consecutive bursts.
|
|
deadline=$(( $(date +%s) + DEBOUNCE ))
|
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
|
if read -r -t 2 _; then
|
|
continue # more activity — keep coalescing toward the deadline
|
|
else
|
|
break # ~2s of total quiet — the burst has settled
|
|
fi
|
|
done
|
|
do_restart "$path"
|
|
done
|