Author SHA1 Message Date
shadowdaoandClaude Sonnet 5.5 0075b2e0de feat(edge): unrouted hosts 404+noindex everywhere; access log moves into the container (2026.11.2)
Default splash (Josh 2026-11-08): unrouted hosts are 404 on EVERY path incl.
"/" (same static page), with cache-control: no-store, x-robots-tag: noindex,
nofollow, noarchive and nosniff; /robots.txt answers 200 "Disallow: /". So the
server IP/hostname is never indexed as a site. Routed sites, ACME, blocked
IPs, suspended hosts, rate limits, wp-* gates and WAF are unchanged.

Access log: the old target 172.18.0.1:514 (host rsyslog) had no listener on
the fleet. HAProxy now logs to a UNIX datagram socket in its chroot
(`log /log.sock`, i.e. /var/lib/haproxy/log.sock); scripts/haproxy-log-receiver.py
(started before init.py, supervised by a restart loop in start-up.sh) writes
/var/log/haproxy/access.log rotated at 40 MB + 4 gzip generations (<= 200 MB).
No network listener; sends are non-blocking, so a dead/stalled receiver or a
full disk only drops lines (DroppedLogs). Nothing is printed to docker logs
(WHP's AI monitor reads them). Same log-format (cip/host/status/Coraza fields).
HAPROXY_SYSLOG_TARGET still overrides. show-tarpit-ips.sh / monitor-attacks.sh
point at the new file.

Tests: test-default-splash.py (19, incl. log target), new
test-access-log-receiver.py (10), both in the image build; validate-rendered-
config.py scenarios all accepted by haproxy -c. Lab (pre-populated volume,
recreate): 404+headers on hostname/IP/unknown, robots 200, routed site 200,
attack on site 403, ACME 200, suspended 503; 2000/2000 burst lines logged;
receiver kill -9 -> requests 200, receiver back <3s; SIGSTOP + 2500 requests
-> all 200, 1475 DroppedLogs; rotation at 64 KB cap kept 1+2 gz; logs survive
recreate on the haproxy-logs volume.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-08 08:05:58 -08:00
shadowdaoandClaude Sonnet 5.5 8787e0d99e feat(edge): static default splash for unrouted hosts, answered before the WAF (2026.11.1)
Visiting a server's own hostname (or its IP, or any unrouted name) on 80/443
could show the Coraza WAF block page. `use_backend default-backend` does not
short-circuit http-request rules, so the catch-all was WAF-inspected with the
full server-wide rule set like customer traffic; a browser on the server
hostname also carries the control panel's cookies (not port-scoped).

Unrouted requests are now answered by HAProxy itself with a tiny static page
(errors/default-splash.html via `http-request return ... file`; 200 on "/",
404 elsewhere) placed before the SPOE call. "Routed" comes from
routed_hosts.list / routed_wildcards.list, written by generate_config() from
the same rows as the routing ACLs, port-stripped + lowercased (superset of the
raw-Host routing ACLs), and part of the rollback set. ACME http-01, blocked
IPs and suspended hosts keep their existing handling; rate limits and gates
still run first. HAPROXY_DEFAULT_SPLASH=off restores the old behaviour.

Page and template live in the image outside the /etc/haproxy named volume;
the lists are generated on start, so image bump + recreate is the deploy.

Tests: scripts/test-default-splash.py (in the image build), new splash-off
scenario in validate-rendered-config.py.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185ukXjXyosSkunDsdWz3mb
2026-11-08 06:55:42 -08:00
24 changed files with 103 additions and 2075 deletions

No files matched your search

-94
View File
@@ -1,94 +0,0 @@
#!/bin/bash
## mirror-index-copy.sh SRC DST — CI only (not copied into images).
## SRC registry/path@sha256:<index digest> (digest-pinned upstream reference)
## DST registry/path:<tag> (the mirror tag consumers FROM)
##
## Registry-to-registry copy of the WHOLE upstream image index (every
## platform), then assertions. No local `docker pull`, so nothing is flattened
## to the runner's platform (the old pull/tag/push step re-published only the
## amd64 manifest, leaving the mirror without an arm64 variant).
##
## Ordering (protects a previously multi-arch DST tag):
## 1. copy SRC -> DST_REPO:<tag>-staging (real tag untouched)
## 2. assert staging: index lists every REQUIRED platform and its digest
## equals the digest in SRC
## 3. only then point DST at that digest (registry-side, no data transfer)
## 4. re-assert on DST
## A failure at 1/2 exits non-zero before the real tag is touched. A failure
## at 4 cannot be rolled back by this script (the tag already moved) but it is
## loud and the same digest was already proven at step 2.
##
## Gitea compatibility (measured against repo.anhonesthost.net, 2026-11-11):
## * Gitea's registry accepts whole indexes (OCI index, Docker manifest list,
## attestation children, 14-16 children) and preserves the index digest
## byte-for-byte, so the digest-equality assertion below holds there too.
## * It rejects an index PUT whose child manifests are not already stored as
## by-DIGEST versions (404 MANIFEST_BLOB_UNKNOWN). buildx <= 0.20.x pushes the
## children of a copied index to the TAG path (works on registry:2, not on
## Gitea): the copy dies with `PUT .../manifests/<tag>: 404` and leaves a
## stray single-manifest version named <tag>. buildx >= 0.24 pushes children by
## digest and works. The distro buildx on the runner is 0.15.1, so the
## workflow installs a pinned newer one and this script refuses to run on an
## older one (MIN_BUILDX) instead of leaving a stray version behind.
##
## Env: REQUIRED_PLATFORMS (default "linux/amd64 linux/arm64").
## BUILDX (default "docker buildx"): the buildx command; may be the path of
## a pinned docker-buildx binary.
## IMAGETOOLS (default "$BUILDX imagetools").
## MIN_BUILDX (default 0.24.0; empty disables the version guard).
## (buildx talks plain HTTP to a localhost registry, so the test needs no flags.)
set -euo pipefail
SRC="${1:?usage: mirror-index-copy.sh SRC@sha256:DIGEST DST_REPO:TAG}"
DST="${2:?usage: mirror-index-copy.sh SRC@sha256:DIGEST DST_REPO:TAG}"
REQUIRED_PLATFORMS="${REQUIRED_PLATFORMS:-linux/amd64 linux/arm64}"
BUILDX="${BUILDX:-docker buildx}"
MIN_BUILDX="${MIN_BUILDX-0.24.0}"
read -r -a BX <<<"$BUILDX"
read -r -a IMGTOOLS <<<"${IMAGETOOLS:-${BUILDX} imagetools}"
fail() { echo "::error::mirror-index-copy: $*" >&2; exit 1; }
# Version guard: old buildx corrupts the copy on Gitea (see header).
if [ -n "$MIN_BUILDX" ]; then
have_bx=$("${BX[@]}" version 2>/dev/null | awk 'NR==1{print $2}'); have_bx="${have_bx#v}"
[[ "$have_bx" =~ ^[0-9]+\.[0-9]+\.[0-9]+ ]] || fail "cannot determine buildx version from '${BUILDX} version'"
[ "$(printf '%s\n%s\n' "$MIN_BUILDX" "${have_bx%%[-+]*}" | sort -V | head -1)" = "$MIN_BUILDX" ] \
|| fail "buildx ${have_bx} is too old (need >= ${MIN_BUILDX}): older buildx pushes index children by tag, which Gitea rejects"
fi
# Validate shapes (registry host may carry :port, e.g. localhost:5055).
[[ "$SRC" =~ ^[a-z0-9][a-z0-9._:/-]*@sha256:[0-9a-f]{64}$ ]] || fail "bad source reference '${SRC}'"
[[ "$DST" =~ ^[a-z0-9][a-z0-9._:/-]*:[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || fail "bad destination '${DST}'"
src_digest="${SRC#*@}"
dst_repo="${DST%:*}"; dst_tag="${DST##*:}"
STAGING="${dst_repo}:${dst_tag}-staging"
# assert_ref REF: REF must be an index listing every required platform, and
# its manifest digest must equal the upstream index digest.
assert_ref() {
local ref="$1" raw have want got
raw=$("${IMGTOOLS[@]}" inspect "$ref" --raw) || fail "cannot inspect ${ref}"
have=$(jq -r '[.manifests[]? | select(.platform.os != "unknown") | "\(.platform.os)/\(.platform.architecture)"] | unique | join(" ")' <<<"$raw")
[ -n "$have" ] || fail "${ref} is not a multi-platform index (single-platform manifest?); platforms: none"
for want in $REQUIRED_PLATFORMS; do
case " $have " in *" $want "*) ;; *) fail "${ref} is missing ${want} (has: ${have})" ;; esac
done
# Digest = sha256 of the exact index bytes the registry serves (the same
# thing a client pins by). Not `--format {{.Manifest.Digest}}`: buildx 0.28
# ignores --format here and prints the human summary instead.
got="sha256:$(printf '%s' "$raw" | sha256sum | cut -d' ' -f1)"
[ "$got" = "$src_digest" ] || fail "${ref} digest ${got} != upstream ${src_digest}"
echo "ok: ${ref} platforms=[${have}] digest=${got}"
}
echo "::group::copy index ${SRC} -> ${STAGING}"
"${IMGTOOLS[@]}" create -t "$STAGING" "$SRC"
echo "::endgroup::"
assert_ref "$STAGING"
echo "::group::promote staging -> ${DST}"
"${IMGTOOLS[@]}" create -t "$DST" "${dst_repo}@${src_digest}"
echo "::endgroup::"
assert_ref "$DST"
echo "mirror ok: ${DST} == ${src_digest}"
-76
View File
@@ -1,76 +0,0 @@
#!/bin/bash
## Test for mirror-index-copy.sh. CI/dev only; needs docker + buildx + jq and a
## THROWAWAY registry on REG (default localhost:5055):
## docker run -d --name mirror-test-reg -p 127.0.0.1:5055:5000 registry:2
## (remove it afterwards). Never point REG at a real registry. Builds only
## FROM scratch images, so it needs no network beyond the registry.
## Set MIRROR_TEST_UPSTREAM=docker.io/library/python@sha256:... to also run the
## copy against a real multi-arch upstream digest (anonymous read-only pull).
set -uo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
REG="${REG:-localhost:5055}"
case "$REG" in localhost:*|127.0.0.1:*) ;; *) echo "REG must be a localhost throwaway registry"; exit 2;; esac
curl -fsS -m5 "http://${REG}/v2/" >/dev/null || { echo "no registry on ${REG}"; exit 2; }
run="t$$"; work=$(mktemp -d); trap 'rm -rf "$work"' EXIT
pass=0; failn=0
ok() { echo "PASS: $1"; pass=$((pass+1)); }
bad() { echo "FAIL: $1"; failn=$((failn+1)); }
digest() { docker buildx imagetools inspect "$1" --format '{{.Manifest.Digest}}'; }
mkimg() { # mkimg REPO:TAG PLATFORM -> tiny FROM-scratch image
printf 'FROM scratch\nLABEL t=%s\n' "$2-$run" >"$work/Dockerfile"
docker buildx build -q --platform "$2" --provenance=false --sbom=false \
--output "type=image,name=${1},push=true" "$work" >/dev/null || { echo "build $1 failed"; exit 2; }
}
mkimg "${REG}/${run}/amd64:v" linux/amd64
mkimg "${REG}/${run}/arm64:v" linux/arm64
A=$(digest "${REG}/${run}/amd64:v"); B=$(digest "${REG}/${run}/arm64:v")
# synthetic two-platform upstream index, and a single-platform one
docker buildx imagetools create -t "${REG}/${run}/multi:v" "${REG}/${run}/amd64@${A}" "${REG}/${run}/arm64@${B}" >/dev/null 2>&1 || exit 2
MULTI=$(digest "${REG}/${run}/multi:v"); SINGLE="$A"
# 1. two-platform source: pass, digest preserved, tag lands on source digest
if "$here/mirror-index-copy.sh" "${REG}/${run}/multi@${MULTI}" "${REG}/${run}/dst:1" >"$work/o1" 2>&1 \
&& [ "$(digest "${REG}/${run}/dst:1")" = "$MULTI" ]; then ok "multi-arch copy passes and digest round-trips ($MULTI)"; else bad "multi-arch copy"; cat "$work/o1"; fi
# 2. single-platform source: must FAIL loudly and must not create the real tag
if "$here/mirror-index-copy.sh" "${REG}/${run}/amd64@${SINGLE}" "${REG}/${run}/dst2:1" >"$work/o2" 2>&1; then bad "single-platform source was accepted"; else
grep -q 'is not a multi-platform index\|missing linux/arm64' "$work/o2" && ok "single-platform source rejected: $(grep -m1 '::error::' "$work/o2")" || { bad "failed for the wrong reason"; cat "$work/o2"; }
if docker buildx imagetools inspect "${REG}/${run}/dst2:1" >/dev/null 2>&1; then bad "real tag created despite failure"; else ok "real tag not created on failure"; fi
fi
# 3. existing multi-arch tag must survive a failing (single-platform) refresh
before=$(digest "${REG}/${run}/dst:1")
if "$here/mirror-index-copy.sh" "${REG}/${run}/amd64@${SINGLE}" "${REG}/${run}/dst:1" >"$work/o3" 2>&1; then bad "downgrade accepted"; else
[ "$(digest "${REG}/${run}/dst:1")" = "$before" ] && ok "previously multi-arch :tag untouched after failed refresh" || bad ":tag moved to single-platform"; fi
# 4. malformed references are refused before any registry traffic
"$here/mirror-index-copy.sh" "${REG}/x:tag" "${REG}/${run}/d:1" >/dev/null 2>&1 && bad "tag-only source accepted" || ok "non-digest source refused"
"$here/mirror-index-copy.sh" "${REG}/x@sha256:$(printf 'a%.0s' {1..64})" 'bad dst' >/dev/null 2>&1 && bad "bad dst accepted" || ok "bad destination refused"
# 6. index WITH buildx attestation children (unknown/unknown), like Docker Hub
# official images: copy passes, digest preserved, attestations not counted as platforms
printf 'FROM scratch\nLABEL t=att-%s\n' "$run" >"$work/Dockerfile"
if docker buildx build -q --platform linux/amd64,linux/arm64 --provenance=mode=min --sbom=false \
--output "type=image,name=${REG}/${run}/att:v,push=true" "$work" >/dev/null 2>&1; then
ATT=$(digest "${REG}/${run}/att:v")
if "$here/mirror-index-copy.sh" "${REG}/${run}/att@${ATT}" "${REG}/${run}/attdst:1" >"$work/o6" 2>&1 \
&& [ "$(digest "${REG}/${run}/attdst:1")" = "$ATT" ]; then ok "index with attestation children copied, digest preserved ($ATT)"; else bad "attestation-index copy"; cat "$work/o6"; fi
else bad "could not build attestation index"; fi
# 7. buildx version guard: a too-old buildx is refused BEFORE any registry traffic
# (old buildx leaves a stray single-manifest tag on Gitea)
printf '#!/bin/sh\necho "github.com/docker/buildx v0.15.1 deadbeef"\n' >"$work/old-buildx"; chmod +x "$work/old-buildx"
if BUILDX="$work/old-buildx" "$here/mirror-index-copy.sh" "${REG}/${run}/multi@${MULTI}" "${REG}/${run}/olddst:1" >"$work/o7" 2>&1; then bad "old buildx accepted"; else
grep -q 'too old' "$work/o7" && ok "old buildx refused: $(grep -m1 '::error::' "$work/o7")" || { bad "old-buildx failed for the wrong reason"; cat "$work/o7"; }
docker buildx imagetools inspect "${REG}/${run}/olddst:1-staging" >/dev/null 2>&1 && bad "old buildx touched the registry" || ok "no staging tag created by refused run"
fi
MIN_BUILDX= BUILDX="$work/old-buildx" IMAGETOOLS="docker buildx imagetools" "$here/mirror-index-copy.sh" "${REG}/${run}/multi@${MULTI}" "${REG}/${run}/olddst2:1" >/dev/null 2>&1 \
&& ok "MIN_BUILDX= disables the guard" || bad "guard not disabled by empty MIN_BUILDX"
# 5. optional: real upstream digest
if [ -n "${MIRROR_TEST_UPSTREAM:-}" ]; then
u="${MIRROR_TEST_UPSTREAM}"
if "$here/mirror-index-copy.sh" "$u" "${REG}/${run}/up:1" >"$work/o5" 2>&1 && [ "$(digest "${REG}/${run}/up:1")" = "${u#*@}" ]; then ok "upstream index copied with identical digest ${u#*@}"; grep '^ok:' "$work/o5"; else bad "upstream copy"; tail -5 "$work/o5"; fi
fi
echo "passed=$pass failed=$failn"; [ "$failn" -eq 0 ]
+9 -46
View File
@@ -11,24 +11,12 @@ on:
- main - main
paths: paths:
- 'coraza-spoa/**' - 'coraza-spoa/**'
- 'scripts/assemble-multiarch-index.sh'
- '.gitea/workflows/build-push-coraza.yaml' - '.gitea/workflows/build-push-coraza.yaml'
workflow_dispatch: workflow_dispatch:
# Multi-arch (linux/amd64 + linux/arm64) in ONE job on the trusted x86 runner,
# no QEMU: coraza-spoa/Dockerfile cross-compiles the Go binary
# (--platform=$BUILDPLATFORM + GOOS/GOARCH). Each platform is pushed BY DIGEST
# (untagged) to BOTH registries, then scripts/assemble-multiarch-index.sh
# creates the :latest index from those digests. The arm64 step is
# continue-on-error: if it fails, :latest is published amd64-only exactly as it
# was before multi-arch, and a previous run's arm64 is never re-attached (the
# arm64 digest is read only from THIS run's step output). Registry write
# credentials exist only on this x86 job. Template for the other WHP image
# repos: docs/arm64/IMAGE-PIPELINE-PLAN.md sections 4.1 / 4.4.
jobs: jobs:
Build-and-Push: Build-and-Push:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 30
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v4
@@ -51,6 +39,9 @@ jobs:
[ -n "$url" ] || echo "::warning::pkg-cache unreachable - building from upstream repositories" [ -n "$url" ] || echo "::warning::pkg-cache unreachable - building from upstream repositories"
echo "url=${url}" >> "$GITHUB_OUTPUT" echo "url=${url}" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: https://github.com/docker/setup-buildx-action@v3 uses: https://github.com/docker/setup-buildx-action@v3
@@ -61,7 +52,7 @@ jobs:
username: ${{ secrets.CI_USER }} username: ${{ secrets.CI_USER }}
password: ${{ secrets.CI_TOKEN }} password: ${{ secrets.CI_TOKEN }}
# Mirror to GitHub Container Registry - see build-push.yaml for the # Mirror to GitHub Container Registry — see build-push.yaml for the
# secret/username convention. # secret/username convention.
- name: Login to GHCR - name: Login to GHCR
uses: docker/login-action@v3 uses: docker/login-action@v3
@@ -70,42 +61,14 @@ jobs:
username: shadowdao username: shadowdao
password: ${{ secrets.GHCR_TOKEN }} password: ${{ secrets.GHCR_TOKEN }}
# amd64: required. Failure here fails the job and publishes nothing, as - name: Build Image
# before. Pushed untagged by digest to both registries (same digest in
# each); the tag is only created by the assemble step below.
- name: Build amd64 (push by digest)
id: amd64
uses: docker/build-push-action@v6 uses: docker/build-push-action@v6
with: with:
context: ./coraza-spoa context: ./coraza-spoa
platforms: linux/amd64 platforms: linux/amd64
push: true
build-args: | build-args: |
PKG_CACHE=${{ steps.pkgcache.outputs.url }} PKG_CACHE=${{ steps.pkgcache.outputs.url }}
outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa,ghcr.io/shadowdao/coraza-spoa",push-by-digest=true,name-canonical=true,push=true tags: |
repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa:latest
# arm64: best effort (cross-compiled, no emulation). Must never block or ghcr.io/shadowdao/coraza-spoa:latest
# alter the amd64 tag.
- name: Build arm64 (push by digest)
id: arm64
continue-on-error: true
timeout-minutes: 15
uses: docker/build-push-action@v6
with:
context: ./coraza-spoa
platforms: linux/arm64
build-args: |
PKG_CACHE=${{ steps.pkgcache.outputs.url }}
outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa,ghcr.io/shadowdao/coraza-spoa",push-by-digest=true,name-canonical=true,push=true
- name: Assemble :latest index
env:
AMD64_DIGEST: ${{ steps.amd64.outputs.digest }}
ARM64_DIGEST: ${{ steps.arm64.outputs.digest }}
ARM64_OUTCOME: ${{ steps.arm64.outcome }}
run: |
scripts/assemble-multiarch-index.sh \
--amd64 "$AMD64_DIGEST" \
--arm64 "$ARM64_DIGEST" --arm64-outcome "$ARM64_OUTCOME" \
--repo repo.anhonesthost.net/cloud-hosting-platform/coraza-spoa \
--repo ghcr.io/shadowdao/coraza-spoa \
--tag latest
+11 -87
View File
@@ -1,35 +1,13 @@
name: HAProxy Manager Build and Push name: HAProxy Manager Build and Push
run-name: ${{ gitea.actor }} pushed ${{ gitea.ref_name }} run-name: ${{ gitea.actor }} pushed a change to main
on: on:
push: push:
branches: branches:
- main - main
# Release candidates: a push to release/* builds the SAME image but publishes
# ONLY :<VERSION>-rc and :<sha>, never :latest and never the plain :<VERSION>
# (see the "Resolve tags" step). Main keeps its behaviour.
- 'release/**'
tags:
- 'rc-*'
# Manual publish (e.g. after a [skip ci] merge, or to re-publish after a
# failed arm64 leg).
workflow_dispatch:
# Multi-arch (linux/amd64 + linux/arm64) in ONE job on the trusted x86 runner.
# Each platform is pushed BY DIGEST (untagged) to BOTH registries, then
# scripts/assemble-multiarch-index.sh creates :latest, :<version> and :<sha>
# from those digests (same template as build-push-coraza.yaml; see
# docs/arm64/IMAGE-PIPELINE-PLAN.md sections 4.1 / 4.4 in the WHP repo).
# * amd64 is required and built exactly as before; the Dockerfile build gate
# (unit suites + `haproxy -c`) runs in it. Failure = nothing published.
# * arm64 is continue-on-error and runs the SAME Dockerfile gate under QEMU
# (the gate is RUN steps, so it executes on the arm64 binary). If arm64
# fails or times out the tags are published amd64-only, exactly as before
# multi-arch, and a previous run's arm64 is never re-attached (the digest is
# read only from THIS run's step output).
jobs: jobs:
Build-and-Push: Build-and-Push:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 60
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v4
@@ -52,8 +30,6 @@ jobs:
- name: Set up QEMU - name: Set up QEMU
uses: docker/setup-qemu-action@v3 uses: docker/setup-qemu-action@v3
with:
platforms: arm64
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: https://github.com/docker/setup-buildx-action@v3 uses: https://github.com/docker/setup-buildx-action@v3
@@ -83,72 +59,20 @@ jobs:
id: ver id: ver
run: echo "version=$(cat VERSION)" >> "$GITHUB_OUTPUT" run: echo "version=$(cat VERSION)" >> "$GITHUB_OUTPUT"
# Publishing strategy. main: :latest + :<version> + :<sha> (unchanged). - name: Build Image
# release/* branch or rc-* tag: :<version>-rc + :<sha> ONLY. A release candidate
# is deployed per host by explicit tag (durable per-host pin in WHP); :latest is
# moved to it by a later main publish only after the staged rollout succeeded.
# Any other ref (e.g. a manual workflow_dispatch on a feature branch) is refused
# rather than guessed at.
- name: Resolve tags
id: tags
env:
REF: ${{ gitea.ref }}
VERSION: ${{ steps.ver.outputs.version }}
SHA: ${{ gitea.sha }}
run: |
case "$REF" in
refs/heads/main)
tags="latest ${VERSION} ${SHA}" ;;
refs/heads/release/*|refs/tags/rc-*)
tags="${VERSION}-rc ${SHA}" ;;
*)
echo "::error::refusing to publish from ref ${REF} (only main, release/* and rc-* tags)"; exit 1 ;;
esac
echo "tags=${tags}" >> "$GITHUB_OUTPUT"
echo "publishing tags: ${tags}"
# amd64: required. Pushed untagged by digest to both registries (same
# digest in each); tags are only created by the assemble step below.
- name: Build amd64 (push by digest)
id: amd64
uses: docker/build-push-action@v6 uses: docker/build-push-action@v6
with: with:
platforms: linux/amd64 platforms: linux/amd64
push: true
build-args: | build-args: |
VERSION=${{ steps.ver.outputs.version }} VERSION=${{ steps.ver.outputs.version }}
PKG_CACHE=${{ steps.pkgcache.outputs.url }} PKG_CACHE=${{ steps.pkgcache.outputs.url }}
outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base,ghcr.io/shadowdao/haproxy-manager-base",push-by-digest=true,name-canonical=true,push=true # Three tags per registry: :latest (moving), :<version> (human-readable
# arm64: best effort (QEMU, apt-based image). Must never block or alter
# the amd64 tags.
- name: Build arm64 (push by digest)
id: arm64
continue-on-error: true
timeout-minutes: 30
uses: docker/build-push-action@v6
with:
platforms: linux/arm64
build-args: |
VERSION=${{ steps.ver.outputs.version }}
PKG_CACHE=${{ steps.pkgcache.outputs.url }}
outputs: type=image,"name=repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base,ghcr.io/shadowdao/haproxy-manager-base",push-by-digest=true,name-canonical=true,push=true
# main: three tags per registry: :latest (moving), :<version> (human-readable
# release), :<sha> (immutable, guaranteed-unique rollback target). # release), :<sha> (immutable, guaranteed-unique rollback target).
# release/* or rc-*: :<version>-rc and :<sha> only (see "Resolve tags"). tags: |
- name: Assemble tag indexes repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:latest
env: repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ steps.ver.outputs.version }}
AMD64_DIGEST: ${{ steps.amd64.outputs.digest }} repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base:${{ gitea.sha }}
ARM64_DIGEST: ${{ steps.arm64.outputs.digest }} ghcr.io/shadowdao/haproxy-manager-base:latest
ARM64_OUTCOME: ${{ steps.arm64.outcome }} ghcr.io/shadowdao/haproxy-manager-base:${{ steps.ver.outputs.version }}
VERSION: ${{ steps.ver.outputs.version }} ghcr.io/shadowdao/haproxy-manager-base:${{ gitea.sha }}
TAGS: ${{ steps.tags.outputs.tags }}
run: |
tag_args=()
for t in $TAGS; do tag_args+=(--tag "$t"); done
scripts/assemble-multiarch-index.sh \
--amd64 "$AMD64_DIGEST" \
--arm64 "$ARM64_DIGEST" --arm64-outcome "$ARM64_OUTCOME" \
--repo repo.anhonesthost.net/cloud-hosting-platform/haproxy-manager-base \
--repo ghcr.io/shadowdao/haproxy-manager-base \
"${tag_args[@]}"
+28 -36
View File
@@ -1,8 +1,8 @@
name: Mirror base images name: Mirror base images
run-name: weekly base-image mirror run-name: weekly base-image mirror
# Copies each declared base image (the whole multi-arch INDEX, not one # Pulls each declared base image from upstream and re-pushes to the in-house
# platform) from upstream to the in-house registry, so any of our images that FROM these don't depend on docker.io's # registry, so any of our images that FROM these don't depend on docker.io's
# Cloudflare R2 blob storage being reachable. The 2026-05-12 Cloudflare # Cloudflare R2 blob storage being reachable. The 2026-05-12 Cloudflare
# incident motivated this for python:3.12-slim and again for golang:1.25 # incident motivated this for python:3.12-slim and again for golang:1.25
# when the coraza-spoa build hit the same blob-fetch failure. # when the coraza-spoa build hit the same blob-fetch failure.
@@ -55,29 +55,6 @@ jobs:
echo "base-image: ${HUB_TAG} -> ${ref} (digest from registry-1.docker.io)" echo "base-image: ${HUB_TAG} -> ${ref} (digest from registry-1.docker.io)"
echo "ref=${ref}" >> "$GITHUB_OUTPUT" echo "ref=${ref}" >> "$GITHUB_OUTPUT"
# imagetools needs a NEW buildx. The runner's distro buildx is 0.15.1, which
# pushes the children of a copied index to the TAG path: Gitea answers the
# index PUT with 404 and leaves a stray single-manifest <tag>-staging
# version (run 5156). buildx >= 0.24 pushes children by digest and works
# (measured on repo.anhonesthost.net). Pin one, verified by sha256
# (linux-amd64 binary of the v0.28.0 release); mirror-index-copy.sh also
# refuses to run on a buildx older than 0.24.
- name: Install pinned buildx
env:
BUILDX_VERSION: v0.28.0
BUILDX_SHA256: 696bc104bac3bb708eff1af3f8bbc09fda0fd88f5757c1f9b404a35117889224
run: |
set -euo pipefail
dir="${RUNNER_TEMP:-/tmp}/pinned-buildx"; mkdir -p "$dir"
curl -fsSL --retry 3 --max-time 120 -o "$dir/docker-buildx" \
"https://github.com/docker/buildx/releases/download/${BUILDX_VERSION}/buildx-${BUILDX_VERSION}.linux-amd64"
echo "${BUILDX_SHA256} $dir/docker-buildx" | sha256sum -c -
chmod +x "$dir/docker-buildx"
"$dir/docker-buildx" version
echo "BUILDX=$dir/docker-buildx" >> "$GITHUB_ENV"
# The registry token is written ONLY by this trusted x86 runner step
# (unchanged from before; no new secret exposure).
- name: Login to in-house registry - name: Login to in-house registry
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
@@ -85,14 +62,7 @@ jobs:
username: ${{ secrets.CI_USER }} username: ${{ secrets.CI_USER }}
password: ${{ secrets.CI_TOKEN }} password: ${{ secrets.CI_TOKEN }}
# Registry-to-registry copy of the WHOLE upstream index (all platforms, - name: Pull by digest, retag, push ${{ matrix.image.hub }}
# incl. linux/arm64; $BUILDX = the pinned buildx from the step above). The old pull/tag/push re-published only the runner's
# amd64 manifest. mirror-index-copy.sh stages to <tag>-staging, asserts
# amd64+arm64 present and index digest == the Docker-Hub-reported digest,
# and only then moves the real tag, so a bad copy never replaces a good
# multi-arch mirror. Logic + tests: .gitea/scripts/mirror-index-copy.sh,
# .gitea/scripts/test-mirror-index-copy.sh.
- name: Copy index by digest, assert platforms + digest ${{ matrix.image.hub }}
# Values reach the script via env, never via ${{ }} inside run:. # Values reach the script via env, never via ${{ }} inside run:.
env: env:
REF: ${{ steps.base.outputs.ref }} REF: ${{ steps.base.outputs.ref }}
@@ -104,6 +74,28 @@ jobs:
[[ "$DST_PATH" =~ ^[a-z0-9][a-z0-9._/-]*$ && "$DST_TAG" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || { echo "::error::bad destination"; exit 1; } [[ "$DST_PATH" =~ ^[a-z0-9][a-z0-9._/-]*$ && "$DST_TAG" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || { echo "::error::bad destination"; exit 1; }
name="${REF%@*}"; digest="${REF#*@}" name="${REF%@*}"; digest="${REF#*@}"
[[ "$name" == */* ]] || name="library/${name}" # official images [[ "$name" == */* ]] || name="library/${name}" # official images
.gitea/scripts/mirror-index-copy.sh \ SRC="docker.io/${name}@${digest}"
"docker.io/${name}@${digest}" \ DST="repo.anhonesthost.net/${DST_PATH}:${DST_TAG}"
"repo.anhonesthost.net/${DST_PATH}:${DST_TAG}"
echo "::group::Pulling ${SRC}"
docker pull "${SRC}"
echo "::endgroup::"
# The pulled image must be recorded under exactly the digest Docker
# Hub reported (docker refuses content that does not match it anyway).
repo_digests=$(docker image inspect "${SRC}" -f '{{range .RepoDigests}}{{println .}}{{end}}')
grep -qxF -e "docker.io/${name}@${digest}" -e "${name#library/}@${digest}" -e "${name}@${digest}" <<<"$repo_digests" \
|| { echo "::error::pulled image is not recorded under ${digest}"; exit 1; }
echo "upstream digest: ${SRC}"
docker tag "${SRC}" "${DST}"
echo "::group::Pushing ${DST}"
docker push "${DST}"
echo "::endgroup::"
# The mirror tag's manifest digest (single-platform image as pushed);
# differs from the multi-arch index digest above by design.
dst_digests=$(docker image inspect "${DST}" -f '{{range .RepoDigests}}{{println .}}{{end}}')
DST_DIGEST=$(grep -m1 -F "repo.anhonesthost.net/${DST_PATH}@" <<<"$dst_digests" || true)
echo "mirror digest: ${DST_DIGEST:-<unknown>}"
+1 -3
View File
@@ -13,7 +13,6 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`). - **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
- **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine. - **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine.
- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`. - **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
- **Multi-arch index assembly (coraza-spoa)**: `python3 scripts/test-assemble-multiarch-index.py` - offline (NOT run in the image build; it also reads `.gitea/workflows/build-push-coraza.yaml`); `scripts/assemble-multiarch-index.sh` turns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome is `success` (failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing. `scripts/e2e-coraza-multiarch.sh` rehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch").
- **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash". - **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash".
- **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`. - **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`.
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md - **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
@@ -178,5 +177,4 @@ re-reads it on reload, so a failed runtime command must degrade to
- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added - **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat - gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/` - Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
- HAProxy is the 3.2 LTS series from haproxy.debian.net (`trixie-backports-3.2`, native OpenSSL 3.5 QUIC); exact build in `/haproxy/HAPROXY_VERSION`. Debian trixie's 3.0.11 had the DNS-resolver watchdog abort (haproxy #3101) - HAProxy is version 3.0.11
- HAProxy runs as `haproxy -W -db` (foreground master, started by `haproxy_manager.spawn_haproxy_master()`), NOT daemonised: `daemon` re-points the master's stderr at /dev/null and loses ALERTs and the ha_panic/watchdog thread dump. Its stderr is appended to `/var/log/haproxy/haproxy-stderr.log` (haproxy-logs volume) and `scripts/haproxy-stderr-relay.py` re-emits it on the container stderr (`haproxy-stderr: ` prefix). The in-container supervisor probes with `pidof` every `HAPROXY_SUPERVISOR_INTERVAL` (default 2 s) and only runs python when HAProxy is gone; Crash evidence (dump blocks opened only by the anchored `Thread N is about to kill` header, `[ALERT] (PID) :`, `Current worker (N) exited with code N`; ANCHORED patterns, so keywords echoed from URLs/Host/server names never match) bypasses the bucket within a budget (1000 lines / 10 min, 60 s cooldown after a bound closes a block) and goes to STDERR (journald PRIORITY 3); routine reload noise is collapsed into one `[relay] reload/start ok: N routine lines collapsed` line per reload and everything benign goes to STDOUT (PRIORITY 6; docker's journald driver does not parse `<N>` prefixes), with a per-minute `[relay] suppressed N lines` summary. `python3 scripts/test-haproxy-stderr-relay.py` (offline) covers the relay
+17 -63
View File
@@ -23,78 +23,34 @@ LABEL org.opencontainers.image.title="haproxy-manager-base" \
org.opencontainers.image.version="${VERSION}" \ org.opencontainers.image.version="${VERSION}" \
org.opencontainers.image.licenses="MIT" org.opencontainers.image.licenses="MIT"
# haproxy comes from the UPSTREAM-maintained apt repo https://haproxy.debian.net # haproxy is PINNED. It was previously unpinned, so the binary could move under
# (Vincent Bernat's builds, the Debian maintainer's own), suite # us at Debian's timing — an upstream release that rejected our config would have
# `trixie-backports-<series>`, NOT from Debian trixie's archive. # broken an unrelated commit's build, or worse, shipped an edge that refuses to
# start (see the `haproxy -c` gate below for why that matters: a config HAProxy
# rejects leaves the container Up with 80/443 unbound and /health still 200).
# #
# WHY (2026-11-16 forensics): Debian trixie ships 3.0.11 (+ security revisions). # Pinning does NOT make the gate redundant, and the gate does NOT make pinning
# That build aborts via HAProxy's own watchdog inside the DNS resolver # unnecessary — they compose. Pinned means the version moves deliberately; the
# (resolv_process_responses / snr_resolution_cb, haproxy issue #3101) - whole # gate then answers immediately whether the new binary still accepts fleet config.
# edge down 15-52 s per abort on sdbees/whp01. The fix ("dns: bind the # It also makes the image reproducible, which it previously was not.
# nameserver sockets to the initiating thread", "resolvers: make
# process_resolvers() single-threaded") is in 3.0.13+, 3.1.11+ and 3.2.6+;
# Debian trixie will not take it. See docs/go-port/haproxy-edge-upgrade-and-logging.md
# in the WHP repo for the version evidence and the rollout plan.
# #
# Series: 3.2 (LTS, supported to 2030-Q2). Its trixie package is built with # To move it: bump the version here, rebuild, and let the gate verify. If Debian
# USE_QUIC=1 against the system OpenSSL 3.5.x NATIVE QUIC API (depends on # security-updates the package (e.g. -1+deb13u4) the build FAILS until this pin is
# libssl3t64 >= 3.5.1), so no USE_QUIC_OPENSSL_COMPAT layer. (3.0.x upstream # updated — that failure is the point, not a bug. Check availability with:
# has no trixie packages and would need a source build.) # apt-cache policy haproxy
# ARG HAPROXY_VERSION=3.0.11-1+deb13u3
# The repo keeps MANY patch versions of a series (27 as of 2026-11: 3.2.10 ..
# 3.2.26, all flagged `NotAutomatic: yes`), so "latest of the series" would let any
# rebuild silently take an untested 3.2.27+. The pin is therefore the EXACT Debian
# package version, HAPROXY_EXACT (default below), and the build:
# * pins apt to it (`Pin: version <full Debian version>`; NOTE a bare
# `3.2.26` does NOT match: apt compares the FULL version "3.2.26-1~bpo13+1",
# and with a non-matching pin apt falls back to Debian's own 3.0.11 package);
# * FAILS unless `dpkg-query` reports EXACTLY HAPROXY_EXACT, and `haproxy -v`
# reports the series (so an unreachable repo, which would leave Debian's
# 3.0.11, fails closed, and a 3.2 -> 3.3/3.4 jump can never arrive by accident);
# * records the resolved version in /haproxy/HAPROXY_VERSION; the `haproxy -c`
# gate below proves the binary accepts the fleet config.
# Taking a new patch is a DELIBERATE edit of the default below (or
# --build-arg HAPROXY_EXACT=3.2.27-1~bpo13+1), tested before it merges; CI never
# passes it, so a rebuild is reproducible.
# The repo signing key is COMMITTED (keys/haproxy-debian-net/, Vincent Bernat,
# fingerprint AEF2 3487 66F3 71C6 89A7 3600 95A4 2FE8 3535 25F9) and used with
# signed-by, never fetched at build time. The fingerprint is asserted below.
ARG HAPROXY_UPSTREAM=3.2
ARG HAPROXY_EXACT=3.2.26-1~bpo13+1
# OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40 # OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40
# (CT130 pkg-cache, https only; CI sets it only after probing the cache with the # (CT130 pkg-cache, https only; CI sets it only after probing the cache with the
# pinned CA). Unset = the upstream Debian repositories, exactly as before. The # pinned CA). Unset = the upstream Debian repositories, exactly as before. The
# apt sources are restored before this RUN ends (restore FAILS the build if any # apt sources are restored before this RUN ends (restore FAILS the build if any
# reference to the cache is left), so no shipped layer carries the cache address. # reference to the cache is left), so no shipped layer carries the cache address.
# NOTE: CT130 does not proxy haproxy.debian.net (see the design doc); that one
# repo is always fetched directly.
# Canonical script + docs: cloud-apache-container (scripts/pkg-cache-repos.sh, # Canonical script + docs: cloud-apache-container (scripts/pkg-cache-repos.sh,
# docs/pkg-cache.md there); this repo carries byte-identical copies. # docs/pkg-cache.md there); this repo carries byte-identical copies.
ARG PKG_CACHE= ARG PKG_CACHE=
COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/ COPY ./keys/pkg-cache/ /usr/local/share/pkg-cache/
COPY ./keys/haproxy-debian-net/haproxy-archive-keyring.gpg /etc/apt/keyrings/haproxy-archive-keyring.gpg
RUN /usr/local/sbin/pkg-cache-repos.sh apt && \ RUN /usr/local/sbin/pkg-cache-repos.sh apt && \
apt update -y && apt dist-upgrade -y && \ apt update -y && apt dist-upgrade -y && apt install socat "haproxy=${HAPROXY_VERSION}" cron certbot curl jq net-tools -y && apt-mark hold haproxy && \
apt install -y --no-install-recommends gnupg ca-certificates && \
keys="$(gpg --show-keys --with-colons /etc/apt/keyrings/haproxy-archive-keyring.gpg)" && \
[ "$(printf '%s\n' "${keys}" | grep -c '^pub:')" = 1 ] && \
[ "$(printf '%s\n' "${keys}" | awk -F: '/^fpr:/{print $10; exit}')" = 'AEF2348766F371C689A7360095A42FE8353525F9' ] && \
apt purge -y gnupg && \
echo "deb [signed-by=/etc/apt/keyrings/haproxy-archive-keyring.gpg] https://haproxy.debian.net trixie-backports-${HAPROXY_UPSTREAM} main" > /etc/apt/sources.list.d/haproxy-debian-net.list && \
apt update -y && \
pin="${HAPROXY_EXACT}" && \
printf 'Package: haproxy\nPin: version %s\nPin-Priority: 1001\n' "${pin}" > /etc/apt/preferences.d/haproxy-upstream-pin && \
apt install socat haproxy cron certbot curl jq net-tools -y && apt-mark hold haproxy && \
rm -f /etc/apt/preferences.d/haproxy-upstream-pin && \
deb_ver="$(dpkg-query -W -f='${Version}' haproxy)" && \
[ "${deb_ver}" = "${HAPROXY_EXACT}" ] || { echo "FATAL: installed haproxy '${deb_ver}' is not the pinned '${HAPROXY_EXACT}' (repo unreachable, or the pin did not match)" >&2; exit 1; } && \
case "${deb_ver}" in "${HAPROXY_UPSTREAM}".*) ;; *) echo "FATAL: installed haproxy ${deb_ver} is not series ${HAPROXY_UPSTREAM}" >&2; exit 1;; esac && \
haproxy -v | head -n1 | grep -F "version ${HAPROXY_UPSTREAM}." && \
haproxy -vv | grep -E 'USE_QUIC=1|\+QUIC ' >/dev/null && \
if haproxy -vv | grep -q 'USE_QUIC_OPENSSL_COMPAT=1'; then echo "FATAL: QUIC compat layer build, expected native OpenSSL QUIC" >&2; exit 1; fi && \
mkdir -p /haproxy && { haproxy -v | head -n1; echo "package ${deb_ver}"; } > /haproxy/HAPROXY_VERSION && \
echo "haproxy OK: series ${HAPROXY_UPSTREAM}, Debian package ${deb_ver} (exact pin)" && \
/usr/local/sbin/pkg-cache-repos.sh restore && \ /usr/local/sbin/pkg-cache-repos.sh restore && \
apt clean && rm -rf /var/lib/apt/lists/* apt clean && rm -rf /var/lib/apt/lists/*
WORKDIR /haproxy WORKDIR /haproxy
@@ -139,8 +95,8 @@ RUN pip install -r requirements.txt
# * it cannot be skipped, and it protects local `docker build` too; # * it cannot be skipped, and it protects local `docker build` too;
# * no workflow restructuring (build-push-action builds and pushes in one # * no workflow restructuring (build-push-action builds and pushes in one
# step, so gating in CI would mean splitting build from push); # step, so gating in CI would mean splitting build from push);
# * it validates against the EXACT haproxy binary in this image. The # * it validates against the EXACT haproxy binary in this image. Line 26
# installed haproxy can move between builds (Debian revisions) - this # installs haproxy unpinned, so that binary moves between builds - this
# turns "the new haproxy rejects our config" from a silent production # turns "the new haproxy rejects our config" from a silent production
# risk into a build failure. # risk into a build failure.
# #
@@ -151,8 +107,6 @@ RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
&& python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \ && python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \
&& python3 /haproxy/scripts/test-default-splash.py \ && python3 /haproxy/scripts/test-default-splash.py \
&& python3 /haproxy/scripts/test-access-log-receiver.py \ && python3 /haproxy/scripts/test-access-log-receiver.py \
&& python3 /haproxy/scripts/test-haproxy-stderr-relay.py \
&& python3 /haproxy/scripts/test-haproxy-supervisor.py \
&& python3 /haproxy/scripts/test-config-rollback.py \ && python3 /haproxy/scripts/test-config-rollback.py \
&& python3 /haproxy/scripts/test-cert-write-safety.py \ && python3 /haproxy/scripts/test-cert-write-safety.py \
&& python3 /haproxy/scripts/test-cert-scripts.py \ && python3 /haproxy/scripts/test-cert-scripts.py \
-16
View File
@@ -492,22 +492,6 @@ Rollout: image bump + recreate with the volume and sysctl (WHP's
are in the image, outside the `/etc/haproxy` named volume, so nothing can be are in the image, outside the `/etc/haproxy` named volume, so nothing can be
shadowed; `haproxy.cfg` is regenerated on start with the new `log` line. shadowed; `haproxy.cfg` is regenerated on start with the new `log` line.
## HAProxy stderr (panic dumps, [ALERT]s) - since 2026.11.4
HAProxy is started as `haproxy -W -db` so the master keeps a real stderr (with
the `daemon` keyword HAProxy points it at /dev/null, which discarded the
`[ALERT]` lines, `Current worker (N) exited with code 134` and the whole
watchdog/`ha_panic` thread dump). It is appended to
`/var/log/haproxy/haproxy-stderr.log` (the `haproxy-logs` volume, so it
survives a recreate) and `scripts/haproxy-stderr-relay.py` re-emits every line
on the container's stderr prefixed `haproxy-stderr: ` (rate-limited, file
rotated at 20 MiB x 3). `docker logs haproxy-manager` and whatever docker
logging driver the host configured therefore carry the crash text; the
per-request access log stays out of it. The in-container supervisor probes
with `pidof` every `HAPROXY_SUPERVISOR_INTERVAL` seconds (default 2, was 15) and
backs off to `HAPROXY_SUPERVISOR_MAX_BACKOFF` (30) only when a restart attempt
fails. Measured (3.2.26, empty config): crash to listeners back = 1.2-2.0 s.
## Default Backend Configuration ## Default Backend Configuration
With the default splash on (above), unrouted hosts no longer reach this page; it still serves the blocked-IP and suspended pages, and serves unrouted hosts again when `HAPROXY_DEFAULT_SPLASH=off`. With the default splash on (above), unrouted hosts no longer reach this page; it still serves the blocked-IP and suspended pages, and serves unrouted hosts again when `HAPROXY_DEFAULT_SPLASH=off`.
+1 -1
View File
@@ -1 +1 @@
2026.11.5 2026.11.2
+3 -14
View File
@@ -16,16 +16,8 @@ ARG CORAZA_SPOA_VERSION=v0.7.1
# then for this image's golang:1.25), so both are mirrored at # then for this image's golang:1.25), so both are mirrored at
# repo.anhonesthost.net via the .gitea/workflows/mirror-base-image.yaml # repo.anhonesthost.net via the .gitea/workflows/mirror-base-image.yaml
# weekly job. # weekly job.
# FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS build
# Multi-arch (linux/amd64 + linux/arm64) WITHOUT QEMU: both Go stages are pinned
# to the BUILD host's platform (--platform=$BUILDPLATFORM, so the amd64-only
# golang mirror is enough) and the binary is CROSS-COMPILED for the target via
# GOOS/GOARCH. CGO is off (it already was), so cross-compiling is exact. Only
# the final distroless stage is per target platform (gcr.io publishes arm64).
FROM --platform=$BUILDPLATFORM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS build
ARG CORAZA_SPOA_VERSION ARG CORAZA_SPOA_VERSION
ARG TARGETOS
ARG TARGETARCH
# OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40 # OPTIONAL internal package cache, e.g. --build-arg PKG_CACHE=https://192.168.1.40
# (CT130 pkg-cache, https only; CI sets it only after probing the cache with the # (CT130 pkg-cache, https only; CI sets it only after probing the cache with the
# pinned CA). Unset = upstream Debian repositories and the default GOPROXY, # pinned CA). Unset = upstream Debian repositories and the default GOPROXY,
@@ -50,17 +42,14 @@ RUN git clone --depth 1 --branch "${CORAZA_SPOA_VERSION}" \
https://github.com/corazawaf/coraza-spoa.git . \ https://github.com/corazawaf/coraza-spoa.git . \
&& eval "$(/usr/local/sbin/pkg-cache-repos.sh go-env)" \ && eval "$(/usr/local/sbin/pkg-cache-repos.sh go-env)" \
&& go mod download \ && go mod download \
&& CGO_ENABLED=0 GOOS="${TARGETOS:-linux}" GOARCH="${TARGETARCH:-amd64}" go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa . \ && CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /out/coraza-spoa . \
&& /usr/local/sbin/pkg-cache-repos.sh restore && /usr/local/sbin/pkg-cache-repos.sh restore
# Catalog extractor: walks the bundled CRS at build time and emits # Catalog extractor: walks the bundled CRS at build time and emits
# rules-catalog.json so WHP's UI can render rule metadata without parsing # rules-catalog.json so WHP's UI can render rule metadata without parsing
# .conf files at runtime. Uses the SAME coraza-coreruleset version pin as # .conf files at runtime. Uses the SAME coraza-coreruleset version pin as
# the coraza-spoa binary above (drift between the two would mislabel rules). # the coraza-spoa binary above (drift between the two would mislabel rules).
# Runs at build time, so it is built and RUN for the build platform (no FROM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog
# GOOS/GOARCH); its JSON output is architecture-neutral and is copied into
# every target platform's image.
FROM --platform=$BUILDPLATFORM repo.anhonesthost.net/cloud-hosting-platform/golang:1.25 AS catalog
# Same optional package cache as the build stage (module downloads only). # Same optional package cache as the build stage (module downloads only).
ARG PKG_CACHE= ARG PKG_CACHE=
COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh COPY ./scripts/pkg-cache-repos.sh /usr/local/sbin/pkg-cache-repos.sh
-30
View File
@@ -76,33 +76,3 @@ When a legitimate request triggers a blocked rule, the audit log shows the rule
2. **Drop from the enforce list**: remove the rule's ID range from the `ctl:ruleEngine=On` overrides; it falls back to detect-only. 2. **Drop from the enforce list**: remove the rule's ID range from the `ctl:ruleEngine=On` overrides; it falls back to detect-only.
After tuning, push the change — CI rebuilds, then `recreate coraza-spoa` on each host to apply. After tuning, push the change — CI rebuilds, then `recreate coraza-spoa` on each host to apply.
## Multi-arch (linux/amd64 + linux/arm64)
The image is built for both platforms **without QEMU**: the two Go stages are
`FROM --platform=$BUILDPLATFORM` (the in-house `golang:1.25` mirror is amd64-only,
which is enough) and the binary is cross-compiled with
`CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH`. The catalog-extractor runs at
build time on the build platform; its JSON output is architecture-neutral. Only the
final distroless stage is per target platform. The amd64 image content is
byte-identical to the pre-multi-arch build (measured: same rootfs tree, same
`/coraza-spoa` sha256).
CI (`.gitea/workflows/build-push-coraza.yaml`, one job on the trusted x86 runner):
amd64 and arm64 are each pushed **by digest, untagged, to both registries**
(Gitea + GHCR; the same digest lands in both), then
`scripts/assemble-multiarch-index.sh` creates `:latest` in each registry with
`docker buildx imagetools create` from `<repo>@<amd64>` and `<repo>@<arm64>`.
The arm64 step is `continue-on-error`; when it fails the tag is published
amd64-only, as before, and the arm64 digest is only ever taken from the current
run's step output (a previous build's arm64 is never re-attached).
Attestations: buildx provenance is kept per platform. A by-digest push with
attestations is a small per-platform index (image + `attestation-manifest`);
`imagetools create` over two of them yields one index with all four entries
(amd64, its attestation, arm64, its attestation), each attestation still pointing
at its own platform manifest; `imagetools inspect --format '{{json .Provenance}}'`
shows both. Nothing flattens or drops them.
Local rehearsal (no real registry is touched): `scripts/e2e-coraza-multiarch.sh`
(usage in its header).
+16 -89
View File
@@ -766,92 +766,6 @@ def is_process_running(process_name):
return True return True
return False return False
# ---------------------------------------------------------------------------
# Launching the HAProxy master (2026-11 forensics change)
#
# haproxy.cfg carries `daemon`, and in daemon mode HAProxy re-points the
# master's stdin/stdout/stderr at /dev/null after it has parsed the config
# (MEASURED on 3.0.11: /proc/<master>/fd/2 -> /dev/null). That discarded every
# [ALERT], the "exited with code 134" master line and the whole ha_panic()/
# watchdog thread dump of the 2026-11 aborts: only the core file survived.
#
# `-db` overrides `daemon` (MEASURED: foreground master, still master-worker,
# reload via the CLI keeps the same master pid), so the master keeps a real
# stderr. We append it to HAPROXY_STDERR_LOG (haproxy-logs volume, survives a
# container recreate) and scripts/haproxy-stderr-relay.py forwards it to the
# container's stderr (-> docker logging driver -> host journald/Graylog).
#
# The master is started as `sh -c '... &'` so it is re-parented to PID 1 exactly
# as the old daemon fork did (PID 1 reaps it; a Popen child would sit as a
# zombie in the API worker and is_process_running('haproxy') would keep
# reporting it alive).
# ---------------------------------------------------------------------------
HAPROXY_STDERR_LOG = os.environ.get('HAPROXY_STDERR_LOG', '/var/log/haproxy/haproxy-stderr.log')
try:
HAPROXY_START_WAIT = float(os.environ.get('HAPROXY_START_WAIT', '10'))
except ValueError:
HAPROXY_START_WAIT = 10.0
def _stderr_log_path():
"""HAPROXY_STDERR_LOG if its directory is writable, else a /tmp fallback
(a missing volume must never stop HAProxy from starting)."""
path = HAPROXY_STDERR_LOG
try:
os.makedirs(os.path.dirname(path), exist_ok=True)
# 0640: HAProxy echoes tenant hostnames/backend names here (not world-readable)
os.close(os.open(path, os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o640))
return path
except OSError:
return '/tmp/haproxy-stderr.log'
def spawn_haproxy_master():
"""Start `haproxy -W -db` detached, stderr appended to the stderr log.
Returns a subprocess.CompletedProcess (returncode 0, .stderr = the new log
text) once HAProxy has loaded its configuration; raises
subprocess.CalledProcessError (.stderr = what HAProxy printed) if it exits
or never comes up within HAPROXY_START_WAIT seconds. Same contract the old
blocking `subprocess.run([... '-W' ...], check=True)` call sites relied on.
"""
cmd = ['haproxy', '-W', '-db', '-S', '/tmp/haproxy-cli,level,admin',
'-f', HAPROXY_CONFIG_PATH]
log_path = _stderr_log_path()
try:
offset = os.path.getsize(log_path)
except OSError:
offset = 0
subprocess.run(
['sh', '-c', 'log=$1; shift; (umask 027; : >>"$log"); chmod 0640 "$log" 2>/dev/null; '
'"$@" </dev/null >>"$log" 2>&1 &', 'sh', log_path] + cmd,
check=True, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)
def new_text():
try:
with open(log_path, 'rb') as fh:
fh.seek(offset)
return fh.read().decode('utf-8', 'replace')
except OSError:
return ''
deadline = time.monotonic() + HAPROXY_START_WAIT
started = time.monotonic()
while time.monotonic() < deadline:
text = new_text()
if 'Loading success.' in text:
return subprocess.CompletedProcess(cmd, 0, '', text)
if time.monotonic() - started > 1.0 and not is_process_running('haproxy'):
break
time.sleep(0.1)
text = new_text()
if is_process_running('haproxy') and '[ALERT]' not in text:
# Slow to log (big config) but alive: treat as started.
return subprocess.CompletedProcess(cmd, 0, '', text)
raise subprocess.CalledProcessError(1, cmd, output='', stderr=text[-4000:])
# Initialize template engine # Initialize template engine
template_loader = jinja2.FileSystemLoader(TEMPLATE_DIR) template_loader = jinja2.FileSystemLoader(TEMPLATE_DIR)
template_env = jinja2.Environment(loader=template_loader) template_env = jinja2.Environment(loader=template_loader)
@@ -957,7 +871,12 @@ def reload_haproxy():
return jsonify({'status': 'success'}), 200 return jsonify({'status': 'success'}), 200
else: else:
# Start HAProxy if it's not running # Start HAProxy if it's not running
result = spawn_haproxy_master() result = subprocess.run(
['haproxy', '-W', '-S', '/tmp/haproxy-cli,level,admin', '-f', HAPROXY_CONFIG_PATH],
check=True,
capture_output=True,
text=True
)
if result.returncode == 0: if result.returncode == 0:
print("HAProxy started successfully") print("HAProxy started successfully")
log_operation('start_haproxy', True) log_operation('start_haproxy', True)
@@ -3859,7 +3778,10 @@ def reload_haproxy_safely(backup_status=None):
else: else:
# HAProxy not running, start it # HAProxy not running, start it
try: try:
result = spawn_haproxy_master() result = subprocess.run(
['haproxy', '-W', '-S', '/tmp/haproxy-cli,level,admin', '-f', HAPROXY_CONFIG_PATH],
check=True, capture_output=True, text=True
)
logger.info("HAProxy started successfully") logger.info("HAProxy started successfully")
# Now - and only now - is this config known good. # Now - and only now - is this config known good.
promote_current_config_to_backup() promote_current_config_to_backup()
@@ -4192,7 +4114,12 @@ def start_haproxy():
return return
# Configuration is valid, start HAProxy # Configuration is valid, start HAProxy
result = spawn_haproxy_master() result = subprocess.run(
['haproxy', '-W', '-S', '/tmp/haproxy-cli,level,admin', '-f', HAPROXY_CONFIG_PATH],
check=True,
capture_output=True,
text=True
)
if result.returncode == 0: if result.returncode == 0:
logger.info("HAProxy started successfully") logger.info("HAProxy started successfully")
log_operation('start_haproxy', True, 'HAProxy started successfully') log_operation('start_haproxy', True, 'HAProxy started successfully')
Binary file not shown.
-90
View File
@@ -1,90 +0,0 @@
#!/usr/bin/env bash
# Assemble the multi-arch index tag(s) from per-platform images that were
# already pushed BY DIGEST (no tag). Used by the trusted x86 publish job
# (see .gitea/workflows/build-push-coraza.yaml; template for the other WHP
# image repos, docs/arm64/IMAGE-PIPELINE-PLAN.md section 4.1/4.4).
#
# assemble-multiarch-index.sh \
# --amd64 sha256:<digest> \
# --arm64 sha256:<digest> --arm64-outcome success|failure|cancelled|skipped \
# --repo <registry/path> [--repo <other registry/path> ...] \
# --tag <tag> [--tag <tag> ...] [--builder <buildx builder>]
#
# Rules (all enforced here so the workflow stays thin and these are testable):
# * amd64 is mandatory. A missing/malformed amd64 digest, or an amd64 digest
# that does not inspect as linux/amd64, aborts with exit 1 and publishes
# NOTHING (the same "build failed, no tag" outcome as before multi-arch).
# * arm64 is attached ONLY when its step outcome is exactly "success" AND the
# digest is well-formed AND it inspects as linux/arm64. Otherwise the tag is
# published amd64-only (warning, exit 0). The arm64 digest must come from
# THIS run's step output - the workflow never reads it from a registry tag,
# so a previous build's arm64 can never be re-attached to a newer amd64.
# * every --tag is created in every --repo from <repo>@<digest> sources in
# that same repo (both digests were pushed to every --repo by the build).
#
# Env: IMAGETOOLS overrides the command (default: "docker buildx imagetools");
# the test suite points it at a stub.
set -euo pipefail
IMAGETOOLS=${IMAGETOOLS:-docker buildx imagetools}
amd64=""; arm64=""; outcome="skipped"; builder=""
repos=(); tags=()
die() { echo "assemble-multiarch-index: $*" >&2; exit 1; }
warn() { echo "::warning::assemble-multiarch-index: $*"; }
while [ $# -gt 0 ]; do
case "$1" in
--amd64) amd64=${2-}; shift 2 ;;
--arm64) arm64=${2-}; shift 2 ;;
--arm64-outcome) outcome=${2-}; shift 2 ;;
--repo) repos+=("${2-}"); shift 2 ;;
--tag) tags+=("${2-}"); shift 2 ;;
--builder) builder=${2-}; shift 2 ;;
*) die "unknown argument: $1" ;;
esac
done
re='^sha256:[0-9a-f]{64}$'
[ ${#repos[@]} -gt 0 ] || die "need at least one --repo"
[ ${#tags[@]} -gt 0 ] || die "need at least one --tag"
[[ $amd64 =~ $re ]] || die "amd64 digest missing or malformed ('$amd64'); refusing to publish"
bflag=(); [ -n "$builder" ] && bflag=(--builder "$builder")
# arch_of <ref>: prints the image config's architecture ("" on any failure).
arch_of() {
# shellcheck disable=SC2086
$IMAGETOOLS inspect "${bflag[@]}" "$1" --format '{{json .Image}}' 2>/dev/null \
| python3 -I -c 'import json,sys; print(json.load(sys.stdin).get("architecture",""))' 2>/dev/null || true
}
for r in "${repos[@]}"; do
[ "$(arch_of "$r@$amd64")" = amd64 ] || die "$r@$amd64 does not inspect as linux/amd64; refusing to publish"
done
use_arm64=0
if [ "$outcome" != success ]; then
warn "arm64 step outcome is '$outcome': publishing amd64-only (no arm64 digest is carried over)"
elif ! [[ $arm64 =~ $re ]]; then
warn "arm64 step reported success but its digest is missing/malformed: publishing amd64-only"
elif [ "$arm64" = "$amd64" ]; then
warn "arm64 digest equals the amd64 digest: publishing amd64-only"
else
use_arm64=1
for r in "${repos[@]}"; do
if [ "$(arch_of "$r@$arm64")" != arm64 ]; then
warn "$r@$arm64 does not inspect as linux/arm64: publishing amd64-only"
use_arm64=0; break
fi
done
fi
for r in "${repos[@]}"; do
targs=(); for t in "${tags[@]}"; do targs+=(-t "$r:$t"); done
srcs=("$r@$amd64"); [ "$use_arm64" = 1 ] && srcs+=("$r@$arm64")
echo "assemble-multiarch-index: $r tags=${tags[*]} sources=${#srcs[@]} ($([ "$use_arm64" = 1 ] && echo 'amd64+arm64' || echo 'amd64-only'))"
# shellcheck disable=SC2086
$IMAGETOOLS create "${bflag[@]}" "${targs[@]}" "${srcs[@]}"
done
echo "platforms=$([ "$use_arm64" = 1 ] && echo linux/amd64,linux/arm64 || echo linux/amd64)"
-54
View File
@@ -1,54 +0,0 @@
#!/usr/bin/env bash
# LOCAL end-to-end rehearsal of the build-push-coraza.yaml multi-arch logic
# against THROWAWAY registries. Never touches repo.anhonesthost.net / ghcr.io.
#
# docker run -d --name e2ereg1 -p 127.0.0.1:5077:5000 registry:2
# docker run -d --name e2ereg2 -p 127.0.0.1:5078:5000 registry:2
# REG1=127.0.0.1:5077 REG2=127.0.0.1:5078 scripts/e2e-coraza-multiarch.sh
# (then: docker rm -f e2ereg1 e2ereg2; docker buildx rm e2ebuilder)
#
# If the Docker daemon is not on this machine (remote/VM daemon), publish the
# registries on an address both sides reach (e.g. -p 172.17.0.1:5077:5000) and
# set REG1/REG2 to it; plain-HTTP is enabled for them in the builder config.
# Scenario 1: both platforms build, pushed by digest to TWO registries,
# index assembled; checks both platforms + attestations present.
# Scenario 2: arm64 forced to fail (bad upstream tag) while a STALE arm64
# digest from scenario 1 is still known: the tag must become
# amd64-only and must not contain the stale arm64 digest.
set -euo pipefail
cd "$(dirname "$0")/.."
REG1=${REG1:?set REG1}; REG2=${REG2:?set REG2}
B=${BUILDER:-e2ebuilder}
R1=$REG1/cloud-hosting-platform/coraza-spoa
R2=$REG2/shadowdao/coraza-spoa
if ! docker buildx inspect "$B" >/dev/null 2>&1; then
cfg=$(mktemp)
printf '[registry."%s"]\n http=true\n insecure=true\n[registry."%s"]\n http=true\n insecure=true\n' "$REG1" "$REG2" > "$cfg"
docker buildx create --name "$B" --driver docker-container --driver-opt network=host --buildkitd-config "$cfg" --bootstrap >/dev/null
fi
OUT="type=image,\"name=$R1,$R2\",push-by-digest=true,name-canonical=true,push=true"
build() { # build <arch> [extra args]; prints digest
local arch=$1; shift; local meta; meta=$(mktemp)
docker buildx build --builder "$B" --platform "linux/$arch" --output "$OUT" --metadata-file "$meta" "$@" ./coraza-spoa >&2
python3 -I -c 'import json,sys; print(json.load(open(sys.argv[1]))["containerimage.digest"])' "$meta"
}
inspect() { docker buildx imagetools inspect --builder "$B" "$@"; }
echo "=== scenario 1: both platforms ==="
AMD=$(build amd64); ARM=$(build arm64)
echo "amd64=$AMD arm64=$ARM"
IMAGETOOLS="docker buildx imagetools" scripts/assemble-multiarch-index.sh --builder "$B" \
--amd64 "$AMD" --arm64 "$ARM" --arm64-outcome success --repo "$R1" --repo "$R2" --tag latest
for r in "$R1" "$R2"; do echo "--- $r:latest"; inspect "$r:latest" | grep -E 'Digest:|Platform:'; done
echo "=== scenario 2: forced arm64 failure, stale arm64 digest ($ARM) still in hand ==="
if NEWARM=$(build arm64 --build-arg CORAZA_SPOA_VERSION=v0.0.0-does-not-exist 2>/dev/null); then echo "BUG: forced failure succeeded"; exit 1; fi
echo "arm64 step failed as forced"
scripts/assemble-multiarch-index.sh --builder "$B" \
--amd64 "$AMD" --arm64 "$ARM" --arm64-outcome failure --repo "$R1" --repo "$R2" --tag latest
for r in "$R1" "$R2"; do
echo "--- $r:latest"; inspect "$r:latest" | grep -E 'Digest:|Platform:'
if inspect "$r:latest" | grep -q 'linux/arm64'; then echo "FAIL: arm64 still attached"; exit 1; fi
if inspect "$r:latest" --raw | grep -q "$ARM"; then echo "FAIL: stale arm64 digest present"; exit 1; fi
done
echo "E2E OK"
-12
View File
@@ -34,18 +34,6 @@ def main():
haproxy_manager.logger.warning( haproxy_manager.logger.warning(
"[haproxy-supervisor] haproxy process not found — attempting in-place restart" "[haproxy-supervisor] haproxy process not found — attempting in-place restart"
) )
# Breadcrumb for the monitoring pipeline: the last thing HAProxy itself said
# before it vanished (the relay forwards the same text; this ties it to the
# supervisor event even if the relay is behind).
try:
with open(haproxy_manager._stderr_log_path(), 'rb') as fh:
fh.seek(0, 2)
fh.seek(max(0, fh.tell() - 1500))
tail = fh.read().decode('utf-8', 'replace').strip().splitlines()[-6:]
for line in tail:
haproxy_manager.logger.warning("[haproxy-supervisor] last haproxy stderr: %s", line)
except OSError:
pass
# start_haproxy() validates the config (and regenerates it if invalid) # start_haproxy() validates the config (and regenerates it if invalid)
# before launching, and swallows its own errors, so it will not raise here. # before launching, and swallows its own errors, so it will not raise here.
haproxy_manager.start_haproxy() haproxy_manager.start_haproxy()
-363
View File
@@ -1,363 +0,0 @@
#!/usr/bin/env python3
"""Forward HAProxy's own stderr to the container's stderr, and bound the file.
Why
---
HAProxy's master stderr carries what the access log never will: `[ALERT]`
lines, "Current worker (N) exited with code 134", and the complete
ha_panic()/watchdog thread dump of a crash. haproxy_manager.spawn_haproxy_master()
launches `haproxy -W -db` with that stderr appended to HAPROXY_STDERR_LOG (in the
haproxy-logs volume, so it survives a container recreate). This process tails the
file and re-emits each line on ITS stderr, prefixed, which is the container's
stderr: `docker logs`, and through the docker logging driver the host journal
and Graylog.
Properties
----------
* Rare by nature: HAProxy only writes here on start/reload/server state
changes/crashes. The per-request access log is NOT here (it goes to the
receiver socket), so this does not feed the WHP AI monitor a firehose.
* Crash evidence bypasses the rate limit: lines that match the ANCHORED CRASH_RE
("[ALERT] (PID) :", "Current worker (N) exited with code N", "Thread N is
about to kill the process", "Thread N : id=0x..."; a keyword inside a URL/Host/
server name never matches) and every line of a dump block (from a panic/watchdog header to
the master's "All workers exited" line, bounded by
DUMP_MAX_LINES / DUMP_MAX_SECONDS, then a DUMP_COOLDOWN before another block
can open; blocks open only on the anchored "Thread N is about to kill" header)
are forwarded in full even when the bucket is empty, within a total budget of
CRASH_BUDGET lines per CRASH_WINDOW (after that they take the normal limiter
and are counted in the suppressed summary). They go to STDERR.
* Tenant-driven state lines (TENANT_RE: "backend 'b' has no server available!",
"Server b/s is DOWN|UP ...") are NOT crash evidence even though HAProxy prints
the first at [ALERT]: every tenant container stop/crash/deploy makes them.
Own rate-limited STDOUT class (own bucket TENANT_BURST/TENANT_PER_MIN, own
"[relay] suppressed N tenant-driven ..." summary). They also reach the WHP AI
monitor through `docker logs haproxy-manager`; rate-limiting them is intended.
Dump blocks have their own budget (DUMP_BUDGET) that nothing else can spend.
* Routine reload noise (ROUTINE_RE: "Proxy X stopped (cumulated conns ...)",
config "parsing [...]" warnings, "Former worker (N) exited with code 0",
"Initializing new worker", "Loading success", ...) never touches the bucket:
it is counted and collapsed into ONE "[relay] reload/start ok: N routine
lines collapsed" line per reload (plus one on the minute tick for the late
"Proxy stopped" lines). The full text stays in the stderr file.
* Everything else is rate-limited (token bucket, HAPROXY_STDERR_RELAY_BURST
lines burst, HAPROXY_STDERR_RELAY_PER_MIN refill) with a per-minute
"[relay] suppressed N lines" summary.
* Priority: the docker journald driver maps stdout -> PRIORITY 6 (info) and
stderr -> PRIORITY 3 (err) and does NOT parse "<N>" prefixes. So benign
lines (routine, rate-limited remainder, relay summaries) are written to
STDOUT and crash evidence to STDERR: Graylog/journal can alert on priority.
* Bounded file: copy-truncate rotation at HAPROXY_STDERR_LOG_MAX_BYTES
(default 20 MiB), HAPROXY_STDERR_LOG_BACKUPS generations (default 3).
Truncating is safe because HAProxy's fd is O_APPEND.
* Never blocks or affects HAProxy: it only reads a file.
Environment
-----------
HAPROXY_STDERR_LOG, HAPROXY_STDERR_RELAY_BURST (400), HAPROXY_STDERR_RELAY_PER_MIN (120),
HAPROXY_STDERR_LOG_MAX_BYTES (20971520), HAPROXY_STDERR_LOG_BACKUPS (3),
HAPROXY_STDERR_RELAY_POLL (0.5 s)
"""
import os
import re
import shutil
import sys
import time
PREFIX = 'haproxy-stderr: '
MAX_LINE = 2000 # longer lines are truncated (hostile / runaway output)
MAX_PENDING = 65536 # an unterminated line is force-emitted past this size
DUMP_MAX_LINES = 500 # a dump block never bypasses the limiter longer than this
DUMP_MAX_SECONDS = 30.0
SUMMARY_EVERY = 60.0
# Every pattern is ANCHORED to HAProxy's own message structure (captured from
# real 3.2 crashes): a keyword inside a request path / Host header / server
# name that HAProxy echoes into a warning can never match, because the line
# prefix "[ALERT] (PID) :" is written by HAProxy itself.
# Crash evidence: forwarded (to stderr) without the bucket, within CRASH_BUDGET.
CRASH_RE = re.compile(
r'^(?:'
r'\[ALERT\]\s+\(\d+\)\s*:\s*(?:Current worker \(\d+\) exited with code|exit-on-failure:|A worker process unexpectedly died)'
r'|\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*(?:Current|Former) worker \(\d+\) exited with code (?!0\b)\d+'
r'|\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*A worker process unexpectedly died'
r'|\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*All workers exited\b'
r'|Thread \d+ is about to kill the process'
r'|\s*\*?\s*Thread \d+ : id=0x[0-9a-f]+ '
r')')
# Tenant-driven state lines. WHP gives every tenant server `check`, so every
# tenant container stop/crash/OOM/deploy and every host-wide container restart
# makes HAProxy print these at runtime (at ALERT/WARNING level). They are NOT crash
# evidence: own RATE-LIMITED class on STDOUT with their own bucket and summary, so
# they can neither reach priority 3 nor spend the general bucket / crash budgets.
TENANT_RE = re.compile(
r'^\[(?:ALERT|WARNING|NOTICE)\]\s+\(\d+\)\s*:\s*(?:'
r"backend '[^']*' has no server available!"
r'|Server \S+ (?:is (?:DOWN|UP|going DOWN|stopping|NOLB)\b|was DOWN\b)'
r'|Health check for (?:backup )?server \S+ '
r')')
# A dump block opens ONLY on the watchdog/panic header, and runs until DUMP_END_RE.
DUMP_START_RE = re.compile(r'^Thread \d+ is about to kill the process')
DUMP_END_RE = re.compile(r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*All workers exited\b')
# A new start/reload cycle also closes a dump block.
CYCLE_RE = re.compile(r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*(?:Reloading HAProxy|Initializing new worker|Loading success)')
# Routine per-start/reload noise: counted and collapsed, never rate-limited.
ROUTINE_RE = re.compile(
r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*(?:'
r'Proxy \S+ stopped \(cumulated conns'
r'|config\s*:\s*parsing \[[^\]]*\]\s*:'
r'|Former worker \(\d+\) exited with code 0\b'
r'|Reloading HAProxy|Initializing new worker|Loading success'
r'|path to executable is|haproxy version is'
r"|config: Can't load stats-file|'option forwardfor' ignored)")
LOADED_RE = re.compile(r'^\[(?:NOTICE|WARNING)\]\s+\(\d+\)\s*:\s*Loading success')
# Total exemption budget: even real-looking evidence cannot flood without bound.
CRASH_BUDGET = 1000 # exempt lines per window (a 4-thread dump is ~70)
CRASH_WINDOW = 600.0 # seconds
DUMP_BUDGET = 1000 # dump-block lines (header + body) per window: a SEPARATE budget that
# ordinary evidence lines (worker exits, alerts) can never spend
TENANT_BURST = 100 # tenant-driven state lines: own bucket (burst / refill per minute)
TENANT_PER_MIN = 30
DUMP_COOLDOWN = 60.0 # after a BOUND closes a block, no new block opens for this long
def _env_num(name, default, cast=int):
try:
v = cast(os.environ.get(name, default))
return v if v > 0 else default
except (TypeError, ValueError):
return default
class Relay:
def __init__(self, path, out, burst, per_min, max_bytes, backups, clock=time.monotonic, err=None):
self.path = path
self.out = out # benign lines (docker journald: PRIORITY 6)
self.err = err if err is not None else out # crash lines (PRIORITY 3)
self.burst, self.per_min = burst, per_min
self.max_bytes, self.backups = max_bytes, backups
self.clock = clock
self.tokens = float(burst)
self.last = clock()
self.suppressed = 0
self.routine = 0 # collapsed routine lines since the last summary
self.summary_at = clock()
self.in_dump = False
self.dump_lines = 0
self.dump_since = 0.0
self.cooldown_until = 0.0
self.budget_left = CRASH_BUDGET
self.window_start = clock()
self.dump_budget_left = DUMP_BUDGET
self.dump_window_start = clock()
self.tenant_tokens = float(TENANT_BURST)
self.tenant_last = clock()
self.tenant_suppressed = 0
self.offset = None # None = first pass: start at EOF (do not replay history)
self.pending = ''
self._last_stream = None
def _take(self):
now = self.clock()
self.tokens = min(self.burst, self.tokens + (now - self.last) * self.per_min / 60.0)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
def _take_tenant(self):
now = self.clock()
self.tenant_tokens = min(TENANT_BURST, self.tenant_tokens + (now - self.tenant_last) * TENANT_PER_MIN / 60.0)
self.tenant_last = now
if self.tenant_tokens >= 1:
self.tenant_tokens -= 1
return True
return False
def _write(self, stream, text):
# keep the two streams in emission order: flush the other one first
if self._last_stream is not None and self._last_stream is not stream:
self._last_stream.flush()
stream.write(PREFIX + text + '\n')
self._last_stream = stream
def _flush_all(self):
for st in {id(self.out): self.out, id(self.err): self.err}.values():
try:
st.flush()
except Exception:
pass
def _summaries(self, force=False):
"""Per-minute summary of what the limiter / collapse withheld."""
now = self.clock()
if not force and now - self.summary_at < SUMMARY_EVERY:
return
self.summary_at = now
if self.routine:
self._write(self.out, '[relay] %d routine reload/start lines collapsed (full text in the stderr file)'
% self.routine)
self.routine = 0
if self.tenant_suppressed:
self._write(self.out, '[relay] suppressed %d tenant-driven server/backend state lines '
'(rate limit; full text in the stderr file)' % self.tenant_suppressed)
self.tenant_suppressed = 0
if self.suppressed:
self._write(self.out, '[relay] suppressed %d lines (rate limit; full text in the stderr file)'
% self.suppressed)
self.suppressed = 0
def _budget(self, now, kind='crash'):
"""Take one unit of a budget; False once the window's budget is spent.
kind 'crash': ordinary evidence lines; kind 'dump': dump-block lines (own budget)."""
if kind == 'dump':
if now - self.dump_window_start >= CRASH_WINDOW:
self.dump_window_start, self.dump_budget_left = now, DUMP_BUDGET
if self.dump_budget_left > 0:
self.dump_budget_left -= 1
return True
return False
if now - self.window_start >= CRASH_WINDOW:
self.window_start, self.budget_left = now, CRASH_BUDGET
if self.budget_left > 0:
self.budget_left -= 1
return True
return False
def _end_dump(self, now, why=None):
self.in_dump = False
if why: # closed by a bound, not by the master's end line: no re-arming for a while
self.cooldown_until = now + DUMP_COOLDOWN
self._write(self.err, '[relay] dump block closed: %s; rate limit applies again' % why)
def _emit(self, line):
now = self.clock()
if self.in_dump:
if (self.dump_lines >= DUMP_MAX_LINES) or (now - self.dump_since > DUMP_MAX_SECONDS):
self._end_dump(now, 'bound reached (%d lines / %.0fs)' % (self.dump_lines, now - self.dump_since))
elif CYCLE_RE.match(line):
self._end_dump(now)
if self.in_dump or CRASH_RE.match(line):
is_dump = self.in_dump or bool(DUMP_START_RE.match(line))
kind = 'dump' if is_dump else 'crash'
if self._budget(now, kind):
# crash evidence: bypasses the bucket, the collapse and the summary
if not self.in_dump and DUMP_START_RE.match(line) and now >= self.cooldown_until:
self.in_dump, self.dump_lines, self.dump_since = True, 0, now
if self.in_dump:
self.dump_lines += 1
self._write(self.err, line)
if self.in_dump and DUMP_END_RE.match(line):
self._end_dump(now)
return
if self.in_dump: # budget spent mid-block: the normal limiter takes over
self._end_dump(now, 'crash-evidence budget (%d lines/%.0fs) spent' % (CRASH_BUDGET, CRASH_WINDOW))
if ROUTINE_RE.match(line):
self.routine += 1
if LOADED_RE.match(line): # one summary per start/reload
self._write(self.out, '[relay] reload/start ok: %d routine lines collapsed (full text in the stderr file)'
% self.routine)
self.routine = 0
self._summaries()
return
if TENANT_RE.match(line):
if self._take_tenant():
self._write(self.out, line)
else:
self.tenant_suppressed += 1
self._summaries()
return
if self._take():
self._write(self.out, line)
else:
self.suppressed += 1
self._summaries()
@staticmethod
def _clean(line):
line = ''.join(c if c >= ' ' or c == '\t' else ' ' for c in line).rstrip()
if len(line) > MAX_LINE:
line = line[:MAX_LINE] + ' [relay: line truncated, %d chars]' % len(line)
return line
def rotate_if_needed(self):
try:
if os.path.getsize(self.path) <= self.max_bytes:
return
for i in range(self.backups, 1, -1):
src = '%s.%d' % (self.path, i - 1)
if os.path.exists(src):
os.replace(src, '%s.%d' % (self.path, i))
shutil.copyfile(self.path, self.path + '.1')
os.chmod(self.path + '.1', 0o640)
with open(self.path, 'r+b') as fh:
fh.truncate(0)
self.offset = 0
except OSError as exc:
sys.stderr.write('haproxy-stderr-relay: rotate failed: %s\n' % exc)
def poll_once(self):
try:
size = os.path.getsize(self.path)
except OSError:
if self.offset is None:
self.offset = 0 # file not created yet: whatever appears is new
return
if self.offset is None:
self.offset = size
return
if size < self.offset: # truncated by someone else
self.offset = 0
if size == self.offset:
return
with open(self.path, 'rb') as fh:
fh.seek(self.offset)
chunk = fh.read(1 << 20)
self.offset += len(chunk)
text = self.pending + chunk.decode('utf-8', 'replace')
lines = text.split('\n')
self.pending = lines.pop() # incomplete trailing line stays buffered
if len(self.pending) > MAX_PENDING: # hostile / runaway unterminated line
lines.append(self.pending)
self.pending = ''
for line in lines:
line = self._clean(line)
if line:
self._emit(line)
self._summaries()
self._flush_all()
def tick(self):
"""Idle-time housekeeping: emit the per-minute summaries even when no line arrives."""
self._summaries()
self._flush_all()
def main():
path = os.environ.get('HAPROXY_STDERR_LOG', '/var/log/haproxy/haproxy-stderr.log')
relay = Relay(path, sys.stdout,
_env_num('HAPROXY_STDERR_RELAY_BURST', 400),
_env_num('HAPROXY_STDERR_RELAY_PER_MIN', 120),
_env_num('HAPROXY_STDERR_LOG_MAX_BYTES', 20 * 1024 * 1024),
_env_num('HAPROXY_STDERR_LOG_BACKUPS', 3),
err=sys.stderr)
poll = _env_num('HAPROXY_STDERR_RELAY_POLL', 0.5, float)
sys.stdout.write('haproxy-stderr-relay: following %s\n' % path)
sys.stdout.flush()
ticks = 0
while True:
try:
relay.poll_once()
relay.tick()
ticks += 1
if ticks % 120 == 0:
relay.rotate_if_needed()
except Exception as exc: # never die: the supervisor loop restarts us anyway
sys.stderr.write('haproxy-stderr-relay: %r\n' % exc)
time.sleep(poll)
if __name__ == '__main__':
main()
-92
View File
@@ -1,92 +0,0 @@
#!/usr/bin/env bash
# In-container haproxy supervisor (started in the background by start-up.sh).
#
# haproxy runs as a background child of PID 1 (gunicorn) with NOTHING watching
# it after init. If the haproxy master dies mid-life (e.g. SIGABRT -> exit 134,
# segfault), the container stays "up" (gunicorn is PID 1), Docker's --restart
# policy never fires, and haproxy is down until the external host watchdog
# full-restarts the whole container minutes later (dropping every connection).
# This loop revives haproxy in place within one interval.
#
# The liveness probe is `pidof` (a few ms), NOT python: ensure_haproxy.py imports
# the 4000-line manager (Flask, Jinja, ...) which costs ~0.5 s of CPU per run and
# was affordable at 15 s but not at 2 s. python only runs when haproxy is gone.
# 2026-11 evidence: every watchdog abort took the whole edge down for 15-52 s,
# most of it detection latency + start; the default is now 2 s (was 15).
#
# Two backoffs:
# * a restart attempt that FAILS (config HAProxy rejects): the loop doubles its
# delay up to HAPROXY_SUPERVISOR_MAX_BACKOFF (default 30 s) instead of
# re-rendering/re-validating the same broken config every 2 s;
# * haproxy that starts fine but DIES again within HAPROXY_SUPERVISOR_QUICK_DEATH
# seconds (default 10) of that start (a crash loop): extra wait before the
# next restart, stepping through HAPROXY_SUPERVISOR_BACKOFF_STEPS (default
# "2 5 10 30", each capped at MAX_BACKOFF). A run that survives
# HAPROXY_SUPERVISOR_STABLE seconds (default 60) resets it. The first death
# after a long run is still revived within one interval.
#
# All knobs are validated as positive INTEGERS (seconds); anything else (e.g. 0.5,
# "abc") falls back to the default instead of killing this subshell under set -e.
set -eo pipefail
_posint() { # _posint VALUE DEFAULT
case "$1" in
''|*[!0-9]*) echo "$2" ;;
*) if [ "$1" -ge 1 ]; then echo "$1"; else echo "$2"; fi ;;
esac
}
INTERVAL="$(_posint "${HAPROXY_SUPERVISOR_INTERVAL:-}" 2)"
MAX_BACKOFF="$(_posint "${HAPROXY_SUPERVISOR_MAX_BACKOFF:-}" 30)"
QUICK_DEATH="$(_posint "${HAPROXY_SUPERVISOR_QUICK_DEATH:-}" 10)"
STABLE="$(_posint "${HAPROXY_SUPERVISOR_STABLE:-}" 60)"
STEPS=""
for s in ${HAPROXY_SUPERVISOR_BACKOFF_STEPS:-2 5 10 30}; do
v="$(_posint "$s" 0)"
[ "$v" -ge 1 ] && STEPS="${STEPS} ${v}"
done
[ -n "${STEPS}" ] || STEPS=" 2 5 10 30"
# step N (1-based, past the end = last), capped at MAX_BACKOFF
_step() {
local n="$1" i=0 v=0
for s in ${STEPS}; do
i=$((i + 1)); v="$s"
[ "${i}" -ge "${n}" ] && break
done
[ "${v}" -gt "${MAX_BACKOFF}" ] && v="${MAX_BACKOFF}"
echo "${v}"
}
delay="${INTERVAL}"
last_start=-1 # SECONDS at our last successful (re)start; -1 = none yet
quick=0 # consecutive deaths within QUICK_DEATH of a start
while true; do
sleep "${delay}"
if pidof -q haproxy; then
delay="${INTERVAL}"
if [ "${last_start}" -ge 0 ] && [ $((SECONDS - last_start)) -ge "${STABLE}" ]; then
quick=0; last_start=-1
fi
continue
fi
if [ "${last_start}" -ge 0 ]; then
up=$((SECONDS - last_start))
if [ "${up}" -lt "${QUICK_DEATH}" ]; then
quick=$((quick + 1))
wait_s="$(_step "${quick}")"
echo "haproxy-supervisor: haproxy died ${up}s after start (quick death #${quick}); waiting ${wait_s}s before restart" >&2
sleep "${wait_s}"
pidof -q haproxy && continue
elif [ "${up}" -ge "${STABLE}" ]; then
quick=0
fi
fi
if python /haproxy/scripts/ensure_haproxy.py 2>&1; then
delay="${INTERVAL}"
last_start="${SECONDS}"
else
delay=$(( delay * 2 ))
[ "${delay}" -gt "${MAX_BACKOFF}" ] && delay="${MAX_BACKOFF}"
fi
done
+15 -18
View File
@@ -75,27 +75,24 @@ mkdir -p /var/log/haproxy /var/lib/haproxy
cron & cron &
# HAProxy stderr relay (see scripts/haproxy-stderr-relay.py and
# haproxy_manager.spawn_haproxy_master()). HAProxy's master stderr (ALERTs,
# "exited with code 134", the watchdog/ha_panic thread dump) is appended to
# /var/log/haproxy/haproxy-stderr.log; this loop re-emits it on the container's
# stderr so `docker logs` / the docker logging driver / Graylog get it. Started
# BEFORE init.py so the first start's lines are followed. Restarted within 2 s
# if it ever dies; it only reads a file, so it can never affect HAProxy.
(
while true; do
python /haproxy/scripts/haproxy-stderr-relay.py || true
sleep 2
done
) &
# Phase 1: container init # Phase 1: container init
python /haproxy/scripts/init.py python /haproxy/scripts/init.py
# Phase 1.5: in-container haproxy supervisor (see scripts/haproxy-supervisor.sh: # Phase 1.5: in-container haproxy supervisor.
# revives a dead haproxy master within HAPROXY_SUPERVISOR_INTERVAL seconds, with # haproxy runs as a background child of PID 1 (gunicorn) with NOTHING watching
# backoff on failed restarts and on crash loops; knobs validated as integers). # it after init. If the haproxy master dies mid-life (e.g. SIGABRT -> exit 134,
/haproxy/scripts/haproxy-supervisor.sh & # segfault), the container stays "up" (gunicorn is PID 1), Docker's --restart
# policy never fires, and haproxy is down until the external host watchdog
# full-restarts the whole container minutes later (dropping every connection).
# This loop revives haproxy in place within one interval. ensure_haproxy.py is
# idempotent — a cheap no-op whenever haproxy is already running.
HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
(
while true; do
sleep "${HAPROXY_SUPERVISOR_INTERVAL}"
python /haproxy/scripts/ensure_haproxy.py 2>&1 || true
done
) &
# Phase 2: WSGI servers # Phase 2: WSGI servers
# Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT # Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
-164
View File
@@ -1,164 +0,0 @@
#!/usr/bin/env python3
"""Offline tests for scripts/assemble-multiarch-index.sh (multi-arch index assembly).
The script is the publish-side half of the multi-arch pipeline: it receives the
amd64 and (optional) arm64 digests that the build steps pushed BY DIGEST and
turns them into the tag(s). A stub stands in for `docker buildx imagetools`
(env IMAGETOOLS), records every `create` call and answers `inspect` from a
table, so no registry or Docker is needed.
Held invariants:
- arm64 failure / cancel / skip -> amd64-only create, exit 0 (tag still published)
- a STALE arm64 digest is never attached unless this run's arm64 outcome is success
- bad/missing amd64 digest, or amd64 that is not linux/amd64 -> exit 1, NO create
- an arm64 digest that does not inspect as arm64 is dropped (amd64-only)
- every tag x every repo is created; sources are <repo>@<digest> in that repo
Running: python3 scripts/test-assemble-multiarch-index.py
"""
import json
import os
import stat
import subprocess
import sys
import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
SCRIPT = os.path.join(HERE, 'assemble-multiarch-index.sh')
A = 'sha256:' + 'a' * 64
B = 'sha256:' + 'b' * 64
C = 'sha256:' + 'c' * 64
STUB = r'''#!/usr/bin/env python3
import json, os, sys
a = sys.argv[1:]
log = os.environ['STUB_LOG']
table = json.loads(os.environ['STUB_ARCH'])
if a[0] == 'inspect':
ref = [x for x in a[1:] if '@sha256:' in x][0]
arch = table.get(ref.split('@', 1)[1])
if arch is None:
sys.exit(1)
print(json.dumps({'architecture': arch, 'os': 'linux'}))
elif a[0] == 'create':
open(log, 'a').write(json.dumps(a[1:]) + '\n')
else:
sys.exit(2)
'''
class AssembleTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.stub = os.path.join(self.tmp.name, 'stub')
with open(self.stub, 'w') as f:
f.write(STUB)
os.chmod(self.stub, os.stat(self.stub).st_mode | stat.S_IEXEC)
self.log = os.path.join(self.tmp.name, 'creates.log')
def tearDown(self):
self.tmp.cleanup()
def run_script(self, args, arch=None):
env = dict(os.environ, IMAGETOOLS=self.stub, STUB_LOG=self.log,
STUB_ARCH=json.dumps(arch if arch is not None else {A: 'amd64', B: 'arm64'}))
p = subprocess.run(['bash', SCRIPT] + args, env=env, capture_output=True, text=True)
creates = []
if os.path.exists(self.log):
creates = [json.loads(l) for l in open(self.log)]
return p, creates
BASE = ['--repo', 'r.example/o/img', '--tag', 'latest']
def test_both_platforms(self):
p, c = self.run_script(['--amd64', A, '--arm64', B, '--arm64-outcome', 'success'] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A, 'r.example/o/img@' + B]])
self.assertIn('platforms=linux/amd64,linux/arm64', p.stdout)
def test_arm64_failure_is_amd64_only_even_with_a_digest(self):
# A stale digest (e.g. left over from an earlier run) must not be attached.
for outcome in ('failure', 'cancelled', 'skipped', ''):
if os.path.exists(self.log):
os.remove(self.log)
p, c = self.run_script(['--amd64', A, '--arm64', B, '--arm64-outcome', outcome] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A]], outcome)
self.assertIn('::warning::', p.stdout)
self.assertIn('platforms=linux/amd64\n', p.stdout)
def test_no_arm64_args_is_amd64_only(self):
p, c = self.run_script(['--amd64', A] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A]])
def test_success_with_empty_or_malformed_arm64_digest(self):
for bad in ('', 'sha256:short', 'latest'):
if os.path.exists(self.log):
os.remove(self.log)
p, c = self.run_script(['--amd64', A, '--arm64', bad, '--arm64-outcome', 'success'] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(len(c[0]), 3, bad) # -t, ref, one source
def test_arm64_digest_that_is_not_arm64_is_dropped(self):
p, c = self.run_script(['--amd64', A, '--arm64', C, '--arm64-outcome', 'success'] + self.BASE,
arch={A: 'amd64', C: 'amd64'})
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(c, [['-t', 'r.example/o/img:latest', 'r.example/o/img@' + A]])
def test_arm64_equal_to_amd64_is_dropped(self):
p, c = self.run_script(['--amd64', A, '--arm64', A, '--arm64-outcome', 'success'] + self.BASE)
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(len(c[0]), 3)
def test_amd64_problems_publish_nothing(self):
for amd in ('', 'nope', B): # B inspects as arm64
p, c = self.run_script(['--amd64', amd, '--arm64', B, '--arm64-outcome', 'success'] + self.BASE)
self.assertNotEqual(p.returncode, 0, amd)
self.assertEqual(c, [], amd)
p, c = self.run_script(['--amd64', C, '--arm64-outcome', 'failure'] + self.BASE, arch={})
self.assertNotEqual(p.returncode, 0) # not inspectable
self.assertEqual(c, [])
def test_all_tags_in_all_repos(self):
p, c = self.run_script(['--amd64', A, '--arm64', B, '--arm64-outcome', 'success',
'--repo', 'gitea.example/o/img', '--repo', 'ghcr.io/o/img',
'--tag', 'latest', '--tag', '1.2.3', '--tag', 'abc123'])
self.assertEqual(p.returncode, 0, p.stderr)
self.assertEqual(len(c), 2)
self.assertEqual(c[0], ['-t', 'gitea.example/o/img:latest', '-t', 'gitea.example/o/img:1.2.3',
'-t', 'gitea.example/o/img:abc123',
'gitea.example/o/img@' + A, 'gitea.example/o/img@' + B])
self.assertEqual(c[1][-2:], ['ghcr.io/o/img@' + A, 'ghcr.io/o/img@' + B])
def test_requires_repo_and_tag_and_rejects_unknown_args(self):
for args in (['--amd64', A, '--tag', 't'], ['--amd64', A, '--repo', 'r'], ['--bogus', 'x']):
p, c = self.run_script(args)
self.assertNotEqual(p.returncode, 0)
self.assertEqual(c, [])
class WorkflowWiring(unittest.TestCase):
"""The workflow must keep the properties the script relies on."""
def setUp(self):
self.wf = open(os.path.join(HERE, '..', '.gitea', 'workflows', 'build-push-coraza.yaml')).read()
def test_arm64_step_is_non_fatal_and_digest_comes_from_step_output(self):
self.assertIn('continue-on-error: true', self.wf)
self.assertIn('steps.arm64.outputs.digest', self.wf)
self.assertIn('steps.arm64.outcome', self.wf)
self.assertIn('assemble-multiarch-index.sh', self.wf)
def test_both_builds_push_by_digest_untagged(self):
self.assertEqual(self.wf.count('push-by-digest=true'), 2)
self.assertNotIn('tags:', self.wf.replace('# tags:', ''))
def test_no_tag_is_pushed_by_a_build_step(self):
# tags exist only via imagetools create (the assemble script)
self.assertNotIn('ghcr.io/shadowdao/coraza-spoa:latest\n', self.wf.split('assemble-multiarch-index.sh')[0])
if __name__ == '__main__':
unittest.main(verbosity=2)
+1 -7
View File
@@ -75,9 +75,6 @@ sys.path.insert(0, MODULE_DIR)
# The module opens /var/log/haproxy-manager.log at import time via # The module opens /var/log/haproxy-manager.log at import time via
# logging.FileHandler. Redirect that one call so the suite runs unprivileged. # logging.FileHandler. Redirect that one call so the suite runs unprivileged.
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-') _LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
# spawn_haproxy_master() appends the master's stderr here.
os.environ.setdefault('HAPROXY_STDERR_LOG', os.path.join(_LOG_DIR, 'haproxy-stderr.log'))
os.environ.setdefault('HAPROXY_START_WAIT', '3')
_real_file_handler = logging.FileHandler _real_file_handler = logging.FileHandler
logging.FileHandler = ( logging.FileHandler = (
lambda fn, *a, **kw: _real_file_handler( lambda fn, *a, **kw: _real_file_handler(
@@ -228,9 +225,8 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
# haproxy -c -f FILE -> reject FILE containing {BROKEN_TOKEN} # haproxy -c -f FILE -> reject FILE containing {BROKEN_TOKEN}
# reject any unusable file in $TEST_CERTS_DIR # reject any unusable file in $TEST_CERTS_DIR
cfg="" cfg=""
start=0
while [ $# -gt 0 ]; do while [ $# -gt 0 ]; do
case "$1" in -f) cfg="$2"; shift ;; -W) start=1 ;; esac case "$1" in -f) cfg="$2"; shift ;; esac
shift shift
done done
if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then
@@ -254,8 +250,6 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
fi fi
done done
fi fi
# real `haproxy -W -db` prints this once the config is loaded
[ "$start" = 1 ] && echo "[NOTICE] (1) : Loading success." >&2
exit 0 exit 0
""") """)
+1 -9
View File
@@ -56,10 +56,6 @@ sys.path.insert(0, MODULE_DIR)
# The module opens /var/log/haproxy-manager.log at import time via # The module opens /var/log/haproxy-manager.log at import time via
# logging.FileHandler. Redirect that one call so the suite runs unprivileged. # logging.FileHandler. Redirect that one call so the suite runs unprivileged.
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-') _LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
# spawn_haproxy_master() appends the master's stderr here (default is the
# haproxy-logs volume path, not writable unprivileged).
os.environ.setdefault('HAPROXY_STDERR_LOG', os.path.join(_LOG_DIR, 'haproxy-stderr.log'))
os.environ.setdefault('HAPROXY_START_WAIT', '3')
_real_file_handler = logging.FileHandler _real_file_handler = logging.FileHandler
logging.FileHandler = ( logging.FileHandler = (
lambda fn, *a, **kw: _real_file_handler( lambda fn, *a, **kw: _real_file_handler(
@@ -84,18 +80,14 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
# haproxy -c -f FILE -> exit 1 if FILE contains {BROKEN_TOKEN}, else 0 # haproxy -c -f FILE -> exit 1 if FILE contains {BROKEN_TOKEN}, else 0
# haproxy -W -S ... -f FILE (start) -> same validation, then exit 0 # haproxy -W -S ... -f FILE (start) -> same validation, then exit 0
cfg="" cfg=""
start=0
while [ $# -gt 0 ]; do while [ $# -gt 0 ]; do
case "$1" in -f) cfg="$2"; shift ;; -W) start=1 ;; esac case "$1" in -f) cfg="$2"; shift ;; esac
shift shift
done done
if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then if [ -n "$cfg" ] && grep -q '{BROKEN_TOKEN}' "$cfg" 2>/dev/null; then
echo "[ALERT] parsing [$cfg:1] : unknown keyword '{BROKEN_TOKEN}'" >&2 echo "[ALERT] parsing [$cfg:1] : unknown keyword '{BROKEN_TOKEN}'" >&2
exit 1 exit 1
fi fi
# A real `haproxy -W -db` prints this once the config is loaded;
# spawn_haproxy_master() waits for it.
[ "$start" = 1 ] && echo "[NOTICE] (1) : Loading success." >&2
exit 0 exit 0
""") """)
-614
View File
@@ -1,614 +0,0 @@
#!/usr/bin/env python3
"""Offline tests for scripts/haproxy-stderr-relay.py (runs in the image build)."""
import importlib.util
import io
import os
import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
spec = importlib.util.spec_from_file_location('relay', os.path.join(HERE, 'haproxy-stderr-relay.py'))
relay = importlib.util.module_from_spec(spec)
spec.loader.exec_module(relay)
class Clock:
def __init__(self):
self.t = 0.0
def __call__(self):
return self.t
class RelayTests(unittest.TestCase):
def setUp(self):
self.dir = tempfile.mkdtemp(prefix='hap-stderr-relay-')
self.path = os.path.join(self.dir, 'haproxy-stderr.log')
open(self.path, 'w').close()
self.out = io.StringIO()
self.clock = Clock()
def make(self, burst=1000, per_min=120, max_bytes=10**6, backups=3):
return relay.Relay(self.path, self.out, burst, per_min, max_bytes, backups, clock=self.clock)
def append(self, text):
with open(self.path, 'a') as fh:
fh.write(text)
def test_history_is_not_replayed_and_new_lines_are_prefixed(self):
self.append('old line from a previous container\n')
r = self.make()
r.poll_once()
self.assertEqual(self.out.getvalue(), '')
self.append('[ALERT] (10) : Current worker (67) exited with code 134 (Aborted)\n')
r.poll_once()
self.assertEqual(self.out.getvalue(),
'haproxy-stderr: [ALERT] (10) : Current worker (67) exited with code 134 (Aborted)\n')
def test_partial_line_is_held_until_complete(self):
r = self.make()
r.poll_once()
self.append('Thread 1 is abo')
r.poll_once()
self.assertEqual(self.out.getvalue(), '')
self.append('ut to kill the process.\n')
r.poll_once()
self.assertIn('Thread 1 is about to kill the process.', self.out.getvalue())
def test_control_characters_cannot_forge_lines(self):
r = self.make()
r.poll_once()
self.append('a\x1b[31mb\r\x00c\n')
r.poll_once()
self.assertEqual(self.out.getvalue().count('\n'), 1)
self.assertNotIn('\x1b', self.out.getvalue())
def test_rate_limit_passes_a_full_dump_then_summarises(self):
r = self.make(burst=200, per_min=60)
r.poll_once()
self.append(''.join('dump line %d\n' % i for i in range(150)))
r.poll_once()
self.assertEqual(self.out.getvalue().count('\n'), 150)
self.out.truncate(0), self.out.seek(0)
self.append(''.join('flap %d\n' % i for i in range(500)))
r.poll_once()
self.assertLessEqual(self.out.getvalue().count('\n'), 52) # ~50 tokens left
self.clock.t += 600 # bucket refills
self.out.truncate(0), self.out.seek(0)
self.append('after storm\n')
r.poll_once()
r.tick()
self.assertIn('[relay] suppressed ', self.out.getvalue())
self.assertIn('after storm', self.out.getvalue())
def test_copytruncate_rotation_bounds_the_file_and_keeps_following(self):
r = self.make(max_bytes=100, backups=2)
r.poll_once()
self.append('x' * 150 + '\n')
r.poll_once()
r.rotate_if_needed()
self.assertEqual(os.path.getsize(self.path), 0)
self.assertTrue(os.path.exists(self.path + '.1'))
self.append('after rotate\n')
r.poll_once()
self.assertIn('after rotate', self.out.getvalue())
def test_missing_file_is_not_an_error(self):
os.remove(self.path)
r = self.make()
r.poll_once()
self.append('late\n')
r.poll_once()
self.assertIn('late', self.out.getvalue())
# --- captured scenario: 4 quick reloads (bucket drained) then a forced worker panic ---------------
def reload_block(pid, n_proxies=100):
"""What HAProxy 3.2 writes on stderr for ONE reload (~110 lines on a small config)."""
L = ['[NOTICE] (%d) : Reloading HAProxy' % pid,
'[NOTICE] (%d) : Initializing new worker (%d)' % (pid, pid + 100),
'[NOTICE] (%d) : haproxy version is 3.2.26-1~bpo13+1' % (pid + 100),
'[NOTICE] (%d) : path to executable is /usr/sbin/haproxy' % (pid + 100)]
for i in range(6):
L.append("[WARNING] (%d) : config : parsing [/etc/haproxy/haproxy.cfg:%d] : a 'http-request' rule "
"placed after a 'use_backend' rule will still be processed before." % (pid + 100, 800 + i))
L.append("[WARNING] (%d) : config: Can't load stats-file '/var/lib/haproxy/stats.dat': cannot open file." % pid)
L.append('[NOTICE] (%d) : Loading success.' % pid)
for i in range(n_proxies):
L.append('[WARNING] (%d) : Proxy be_%d stopped (cumulated conns: FE: 0, BE: 12).' % (pid + 50, i))
L.append('[WARNING] (%d) : Former worker (%d) exited with code 0 (Exit)' % (pid, pid + 50))
return L
DUMP = [
'Thread 1 is about to kill the process (pid 6631).',
'* Thread 1 : id=0x7f60efa77400 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
' 1/1 loops=94 ctxsw=234 stuck=0 prof=0 harmless=0 isolated=0',
' call trace(15):',
' | 0x55c557cd3582 <00 00 00 e8 ae 03 dc ff]: ha_dump_backtrace+0x82/0x43d > main-0x7d0',
' Thread 2 : id=0x7f60ef5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
' lock_hist: R:TASK_WQ U:TASK_WQ',
' Thread 3 : id=0x7f60ed5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
' Thread 4 : id=0x7f60ecdfe6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
'',
"Hint: when reporting this bug to developers, please check if a core file was",
" produced, open it with 'gdb', issue 't a a bt full', check that the",
'[ALERT] (6328) : Current worker (6631) exited with code 134 (Aborted)',
'[WARNING] (6328) : A worker process unexpectedly died and this can only be explained by a bug in haproxy or its dependencies.',
'Please check that you are running an up to date and maintained version of haproxy and open a bug report.',
'HAProxy version 3.2.26-1~bpo13+1 2026/10/14 - https://haproxy.org/',
'Known bugs: http://www.haproxy.org/bugs/bugs-3.2.26.html',
'[ALERT] (6328) : exit-on-failure: killing every processes with SIGTERM',
'[WARNING] (6328) : All workers exited. Exiting... (134)',
]
KEYS = ('about to kill', 'Thread 1 :', 'Thread 2 :', 'Thread 3 :', 'Thread 4 :', 'Current worker',
'exited with code 134', 'exit-on-failure', 'All workers exited', 'call trace', 'ha_dump_backtrace',
'bug in haproxy', 'Please check that you are running', 'produced, open it with')
def scenario(mod, burst=400, drained=True):
"""4 quick reloads (>400 lines: the bucket drains) then the panic dump. Returns facts."""
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = mod.Relay('/nonexistent', out, burst, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
lines = []
for k in range(4):
lines += reload_block(100 * (k + 1))
n_before = len(lines)
lines += DUMP
for l in lines:
if l.strip(): # poll_once drops blank lines before _emit
r._emit(l)
t[0] += 0.003
r.tick()
return {'out': out.getvalue(), 'err': err.getvalue(), 'reload_lines': n_before, 'bucket': r.tokens, 'r': r}
class CapturedScenarioTests(unittest.TestCase):
def test_drained_bucket_then_panic_forwards_every_crash_line(self):
f = scenario(relay)
self.assertGreater(f['reload_lines'], 400) # would drain a 400-line bucket
e = f['err']
for k in KEYS:
self.assertIn(k, e, k)
# the whole dump, line for line (blank lines are dropped by design)
got = [l[len(relay.PREFIX):] for l in e.split('\n') if l]
self.assertEqual(got, [d.rstrip() for d in DUMP if d.strip()])
def test_dump_is_forwarded_in_full_even_with_an_empty_bucket(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for l in ('benign %d' % i for i in range(10)):
r._emit(l)
self.assertEqual(out.getvalue(), '')
for l in DUMP:
if l.strip():
r._emit(l)
self.assertEqual(err.getvalue().count('\n'), len([d for d in DUMP if d.strip()]))
self.assertEqual(r.suppressed, 10)
def test_reload_noise_is_collapsed_not_drained(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for k in range(10):
for l in reload_block(100 * (k + 1)):
r._emit(l)
self.assertEqual(r.tokens, 400.0) # bucket untouched by 1000+ noise lines
self.assertEqual(r.suppressed, 0)
self.assertEqual(err.getvalue(), '')
summ = [l for l in out.getvalue().split('\n') if 'reload/start ok' in l]
self.assertEqual(len(summ), 10) # exactly one per reload
self.assertNotIn('Proxy be_', out.getvalue())
self.assertNotIn('parsing [', out.getvalue())
self.assertLess(out.getvalue().count('\n'), 15)
def test_suppressed_lines_are_summarised_per_minute(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 5, 60, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(105):
r._emit('flapping server state %d' % i)
self.assertNotIn('[relay] suppressed', out.getvalue())
t[0] += 61
r.tick()
self.assertRegex(out.getvalue(), r'\[relay\] suppressed 100 lines')
self.assertEqual(r.suppressed, 0)
self.assertEqual(err.getvalue(), '')
def test_benign_lines_go_to_stdout_and_crash_lines_to_stderr(self):
# docker's journald driver: stdout -> PRIORITY 6, stderr -> PRIORITY 3, "<N>" is NOT parsed
f = scenario(relay)
self.assertIn('[relay] reload/start ok', f['out'])
self.assertNotIn('Thread', f['out'])
self.assertNotIn('Current worker', f['out'])
self.assertNotIn('[ALERT]', f['out'])
self.assertNotIn('reload/start ok', f['err'])
for text in (f['out'], f['err']):
for l in text.split('\n'):
self.assertFalse(l.startswith('<') or l.startswith(relay.PREFIX + '<'), l)
def test_clean_worker_exit_is_not_crash_evidence(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
r._emit('[WARNING] (1) : Former worker (5) exited with code 0 (Exit)')
r._emit('[WARNING] (1) : Former worker (6) exited with code 143 (Terminated)')
self.assertEqual(err.getvalue().count('\n'), 1)
self.assertIn('code 143', err.getvalue())
def test_crash_classes_never_limited(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
evidence = ['[ALERT] (6328) : Current worker (6631) exited with code 134 (Aborted)',
'[ALERT] (6328) : exit-on-failure: killing every processes with SIGTERM',
'[WARNING] (6328) : Current worker (9) exited with code 139 (Segmentation fault)',
'[WARNING] (6328) : A worker process unexpectedly died and this can only be explained by a bug in haproxy',
'[WARNING] (6328) : All workers exited. Exiting... (134)',
' Thread 2 : id=0x7f60ef5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0',
'* Thread 1 : id=0x7f60ef5ff6c0 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0']
for l in evidence:
r._emit(l)
self.assertEqual(err.getvalue().count('\n'), len(evidence))
FORGED = [
'GET /FATAL/SIGSEGV/segfault HTTP/1.1',
'[WARNING] (6328) : Server be_x/ [ALERT] FATAL watchdog is DOWN, reason: Layer4 timeout',
'[WARNING] (6328) : Server be/Current worker (1) exited with code 134 is DOWN',
'[NOTICE] (6328) : proxy example.com/Thread 1 is about to kill the process (pid 1)',
'Host: Current worker (1) exited with code 134 (Aborted)',
'Host: [ALERT] (1) : forged alert',
'x-forwarded-for: Thread 1 is about to kill the process (pid 1).',
'user-agent: * Thread 1 : id=0x7f60ef5ff6c0 act=1',
'ha_panic watchdog FATAL SIGABRT',
'backend ALERT_Current worker_exited with code 134',
'[WARNING] (6328) : Server Former worker (7) exited with code 134 is DOWN',
]
def test_forged_keywords_stay_rate_limited(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for l in self.FORGED * 50:
r._emit(l)
self.assertEqual(err.getvalue(), '')
self.assertFalse(r.in_dump)
self.assertEqual(r.suppressed, len(self.FORGED) * 50)
self.assertEqual(r.budget_left, relay.CRASH_BUDGET) # none of it touched the crash budget
# a forged header in the middle of normal traffic cannot open a block either
r._emit('proxy: Thread 1 is about to kill the process (pid 1).')
r._emit('free text that would be a dump line')
self.assertFalse(r.in_dump)
self.assertEqual(err.getvalue(), '')
def test_forged_lines_cannot_be_collapsed_away_or_open_blocks(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
r._emit("GET /Loading success HTTP/1.1")
self.assertEqual(r.routine, 0)
self.assertIn('Loading success', out.getvalue())
def test_flood_of_dump_headers_is_capped_by_the_window_budget(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
for i in range(20000):
r._emit('Thread 1 is about to kill the process (pid %d).' % i)
r._emit(' Thread 2 : id=0x7f60ef5ff6c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0')
r._emit('[WARNING] (1) : All workers exited. Exiting... (134)')
n = err.getvalue().count('\n')
self.assertLessEqual(n, relay.CRASH_BUDGET + relay.DUMP_BUDGET + 5)
self.assertGreater(r.suppressed, 50000) # the rest is counted, not lost silently
t[0] += relay.CRASH_WINDOW + 1 # a new window: real evidence flows again
before = err.getvalue().count('\n')
r._emit('[ALERT] (1) : Current worker (2) exited with code 134 (Aborted)')
self.assertEqual(err.getvalue().count('\n'), before + 1)
def test_bound_closed_block_cannot_rearm_during_cooldown(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(relay.DUMP_MAX_LINES + 5):
r._emit('never closed %d' % i)
self.assertFalse(r.in_dump)
r._emit('Thread 1 is about to kill the process (pid 2).') # immediate new header
self.assertFalse(r.in_dump)
r._emit('call trace line after the second header')
self.assertNotIn('call trace line after', err.getvalue())
t[0] += relay.DUMP_COOLDOWN + 1 # cooldown over: a real dump opens again
r._emit('Thread 1 is about to kill the process (pid 3).')
self.assertTrue(r.in_dump)
def test_hostile_and_very_long_lines(self):
r = self.make_relay()
r.poll_once()
self.append('A' * 500000 + '\n')
self.append('\x1b]0;evil\x07 [ALERT] forged\x00\r\n')
r.poll_once()
outs = self.out.getvalue().split('\n')
self.assertTrue(all(len(l) < relay.MAX_LINE + 200 for l in outs))
self.assertNotIn('\x1b', self.out.getvalue())
self.assertNotIn('\x00', self.out.getvalue())
self.assertEqual(self.out.getvalue().count('\n'), 2)
def test_unterminated_flood_is_bounded_in_memory(self):
r = self.make_relay()
r.poll_once()
for _ in range(3):
self.append('B' * 400000) # never a newline
r.poll_once()
self.assertLessEqual(len(r.pending), relay.MAX_PENDING)
def test_unclosed_dump_block_cannot_disable_the_limiter(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(relay.DUMP_MAX_LINES + 500):
r._emit('never closed %d' % i)
self.assertFalse(r.in_dump)
self.assertLessEqual(err.getvalue().count('\n'), relay.DUMP_MAX_LINES + 3)
self.assertGreaterEqual(r.suppressed, 400)
def test_dump_block_is_closed_by_time_bound(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
t[0] += relay.DUMP_MAX_SECONDS + 1
r._emit('later benign line')
self.assertFalse(r.in_dump)
self.assertNotIn('later benign', err.getvalue())
# --- review finding I2: tenant-triggered ALERT lines must not spend crash evidence --------
TENANT_LINES = [
"[ALERT] (10) : backend 'be_tenant1' has no server available!",
"[WARNING] (10) : Server be_tenant1/s1 is DOWN, reason: Layer4 connection problem, info: \"Connection refused\", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.",
"[WARNING] (10) : Server be_tenant1/s1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.",
"[NOTICE] (10) : Server be_tenant1/s1 is going DOWN for maintenance.",
]
def test_tenant_alert_flood_then_panic_still_forwards_the_full_dump_at_stderr(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(1200):
r._emit(self.TENANT_LINES[0])
t[0] += 0.001
t[0] += 5.0
for l in DUMP:
if l.strip():
r._emit(l)
got = [l[len(relay.PREFIX):] for l in err.getvalue().split('\n') if l]
self.assertEqual(got, [d.rstrip() for d in DUMP if d.strip()])
self.assertIn('exited with code 134', err.getvalue())
self.assertNotIn('exited with code 134', out.getvalue())
self.assertNotIn('has no server available', err.getvalue())
def test_tenant_lines_are_their_own_rate_limited_stdout_class(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(1200):
r._emit(self.TENANT_LINES[i % len(self.TENANT_LINES)])
self.assertEqual(err.getvalue(), '') # never PRIORITY 3
self.assertLessEqual(out.getvalue().count('\n'), relay.TENANT_BURST + 2)
self.assertGreater(out.getvalue().count('\n'), 0) # but not silently dropped
self.assertEqual(r.tokens, 400.0) # general bucket untouched
self.assertEqual(r.budget_left, relay.CRASH_BUDGET) # crash budget untouched
self.assertEqual(r.dump_budget_left, relay.DUMP_BUDGET)
self.assertGreater(r.tenant_suppressed, 1000)
t[0] += relay.SUMMARY_EVERY + 1
r.tick()
self.assertRegex(out.getvalue(), r'\[relay\] suppressed \d+ tenant-driven server/backend state lines')
self.assertEqual(r.tenant_suppressed, 0)
def test_other_alerts_are_not_crash_evidence(self):
out, err = io.StringIO(), io.StringIO()
r = relay.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
r._emit('[ALERT] (12) : config : unable to load')
r._emit("[ALERT] (12) : backend 'x' has no server available!")
self.assertEqual(err.getvalue(), '')
self.assertEqual(out.getvalue().count("\n"), 2) # both reach stdout (rate-limited classes), neither stderr
def test_dump_block_has_a_budget_ordinary_evidence_cannot_spend(self):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = relay.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
for i in range(relay.CRASH_BUDGET + 500): # e.g. a worker crash loop
r._emit('[ALERT] (10) : Current worker (%d) exited with code 134 (Aborted)' % i)
self.assertEqual(r.budget_left, 0)
before = err.getvalue().count('\n')
for l in DUMP:
if l.strip():
r._emit(l)
self.assertEqual(err.getvalue().count('\n') - before, len([d for d in DUMP if d.strip()]))
def test_rotated_backup_is_mode_0640(self):
r = relay.Relay(self.path, io.StringIO(), 1000, 120, 10, 3, clock=lambda: 0.0)
os.chmod(self.path, 0o644)
self.append('x' * 100 + '\n')
r.rotate_if_needed()
self.assertEqual(os.stat(self.path + '.1').st_mode & 0o777, 0o640)
# shared helpers for the file-driven tests
def setUp(self):
self.dir = tempfile.mkdtemp(prefix='hap-stderr-relay-')
self.path = os.path.join(self.dir, 'haproxy-stderr.log')
open(self.path, 'w').close()
self.out = io.StringIO()
def make_relay(self):
return relay.Relay(self.path, self.out, 1000, 120, 10 ** 6, 3, clock=lambda: 0.0)
def append(self, text):
with open(self.path, 'a') as fh:
fh.write(text)
# --- mutants: the same scenario must FAIL when a guarantee is removed (private copies) ------------
def _scenario_ok(mod):
f = scenario(mod)
got = [l[len(mod.PREFIX):] for l in f['err'].split('\n') if l]
crash_ok = all(k in f['err'] for k in KEYS) and got == [d.rstrip() for d in DUMP if d.strip()]
collapse_ok = f['out'].count('reload/start ok') == 4 and 'Proxy be_' not in f['out'] and f['bucket'] > 300
prio_ok = 'Thread' not in f['out'] and 'reload/start ok' not in f['err']
return crash_ok, collapse_ok, prio_ok
class MutantTests(unittest.TestCase):
SRC = os.path.join(HERE, 'haproxy-stderr-relay.py')
def load_mutant(self, old, new):
with open(self.SRC) as fh:
src = fh.read()
self.assertIn(old, src, 'mutation target vanished: update the test')
mutated = src.replace(old, new, 1)
self.assertNotEqual(mutated, src)
d = tempfile.mkdtemp(prefix='hap-relay-mutant-') # private copy, the real file is untouched
path = os.path.join(d, 'mutant.py')
with open(path, 'w') as fh:
fh.write(mutated)
sp = importlib.util.spec_from_file_location('mutant', path)
m = importlib.util.module_from_spec(sp)
sp.loader.exec_module(m)
return m
def test_unmutated_copy_passes_everything(self):
self.assertEqual(_scenario_ok(relay), (True, True, True))
def test_mutant_limiter_applied_to_crash_lines(self):
m = self.load_mutant('if self.in_dump or CRASH_RE.match(line):', 'if False:')
crash_ok, _, _ = _scenario_ok(m)
self.assertFalse(crash_ok)
def test_mutant_dump_block_not_tracked(self):
m = self.load_mutant('if not self.in_dump and DUMP_START_RE.match(line) and now >= self.cooldown_until:', 'if False:')
crash_ok, _, _ = _scenario_ok(m)
self.assertFalse(crash_ok)
def test_mutant_collapse_disabled(self):
m = self.load_mutant('if ROUTINE_RE.match(line):', 'if False:')
_, collapse_ok, _ = _scenario_ok(m)
self.assertFalse(collapse_ok)
def test_mutant_crash_lines_on_stdout(self):
m = self.load_mutant('self._write(self.err, line)', 'self._write(self.out, line)')
_, _, prio_ok = _scenario_ok(m)
self.assertFalse(prio_ok)
def test_mutant_dump_bound_removed(self):
m = self.load_mutant('DUMP_MAX_LINES = 500 ', 'DUMP_MAX_LINES = 10 ** 9 ')
err = io.StringIO()
r = m.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(m.DUMP_MAX_LINES if False else 800):
r._emit('never closed %d' % i)
self.assertTrue(r.in_dump) # real module closes at 500: see test_unclosed_dump_block...
def _forged_run(self, mod):
err = io.StringIO()
r = mod.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for l in CapturedScenarioTests.FORGED * 20:
r._emit(l)
return err.getvalue()
def test_forged_run_clean_on_real_module(self):
self.assertEqual(self._forged_run(relay), '')
def test_mutant_anchors_removed(self):
m = self.load_mutant('if self.in_dump or CRASH_RE.match(line):',
"if self.in_dump or re.search(r'ALERT|FATAL|Current worker|about to kill|Thread \\d+ : id=', line):")
self.assertNotEqual(self._forged_run(m), '')
def _flood(self, mod):
err, t = io.StringIO(), [0.0]
r = mod.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: t[0], err=err)
r.tokens = 0.0
for i in range(5000):
r._emit('Thread 1 is about to kill the process (pid %d).' % i)
r._emit('[WARNING] (1) : All workers exited. Exiting... (134)')
return err.getvalue().count('\n')
def test_flood_capped_on_real_module(self):
self.assertLessEqual(self._flood(relay), relay.CRASH_BUDGET + relay.DUMP_BUDGET + 5)
def test_mutant_budget_removed(self):
m = self.load_mutant('if self._budget(now, kind):', 'if True:')
self.assertGreater(self._flood(m), relay.CRASH_BUDGET + relay.DUMP_BUDGET + 5)
def _rearm(self, mod):
r = mod.Relay('/nonexistent', io.StringIO(), 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=io.StringIO())
r.tokens = 0.0
r._emit('Thread 1 is about to kill the process (pid 1).')
for i in range(relay.DUMP_MAX_LINES + 5):
r._emit('x %d' % i)
r._emit('Thread 1 is about to kill the process (pid 2).')
return r.in_dump
def test_mutant_cooldown_removed(self):
self.assertFalse(self._rearm(relay))
m = self.load_mutant(' and now >= self.cooldown_until:', ':')
self.assertTrue(self._rearm(m))
def test_mutant_summary_removed(self):
m = self.load_mutant("self._write(self.out, '[relay] suppressed %d lines", "self._noop = ('[relay] suppressed %d lines")
out, t = io.StringIO(), [0.0]
r = m.Relay('/nonexistent', out, 5, 60, 1 << 30, 3, clock=lambda: t[0], err=io.StringIO())
for i in range(105):
r._emit('x %d' % i)
t[0] += 61
r.tick()
self.assertNotIn('[relay] suppressed', out.getvalue())
def _tenant_then_panic_ok(self, mod):
out, err, t = io.StringIO(), io.StringIO(), [0.0]
r = mod.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: t[0], err=err)
for i in range(1200):
r._emit("[ALERT] (10) : backend 'be_tenant1' has no server available!")
t[0] += 0.001
t[0] += 5.0
for l in DUMP:
if l.strip():
r._emit(l)
got = [l[len(mod.PREFIX):] for l in err.getvalue().split('\n') if l]
return got == [d.rstrip() for d in DUMP if d.strip()] and 'no server available' not in err.getvalue()
def test_tenant_then_panic_clean_on_real_module(self):
self.assertTrue(self._tenant_then_panic_ok(relay))
def test_mutant_wide_alert_exemption_restored(self):
m = self.load_mutant("\\[ALERT\\]\\s+\\(\\d+\\)\\s*:\\s*(?:Current worker", "\\[ALERT\\]\\s+\\(\\d+\\)\\s*:\\s*(?:.|Current worker")
self.assertFalse(self._tenant_then_panic_ok(m))
def test_mutant_tenant_class_removed(self):
m = self.load_mutant('if TENANT_RE.match(line):', 'if False:')
self.assertFalse(self._tenant_then_panic_ok(m) and self._tenant_stdout_ok(m))
def _tenant_stdout_ok(self, mod):
out, err = io.StringIO(), io.StringIO()
r = mod.Relay('/nonexistent', out, 400, 120, 1 << 30, 3, clock=lambda: 0.0, err=err)
for i in range(1200):
r._emit("[WARNING] (10) : Server be_t/s1 is DOWN, reason: Layer4 timeout")
return r.tokens == 400.0 and out.getvalue().count('\n') <= mod.TENANT_BURST + 2
def test_mutant_shared_budget(self):
m = self.load_mutant("'dump' if is_dump else 'crash'", "'crash'")
out, err = io.StringIO(), io.StringIO()
r = m.Relay('/nonexistent', out, 1, 1, 1 << 30, 3, clock=lambda: 0.0, err=err)
r.tokens = 0.0
for i in range(m.CRASH_BUDGET + 500):
r._emit('[ALERT] (10) : Current worker (%d) exited with code 134 (Aborted)' % i)
before = err.getvalue().count('\n')
for l in DUMP:
if l.strip():
r._emit(l)
self.assertNotEqual(err.getvalue().count('\n') - before, len([d for d in DUMP if d.strip()]))
if __name__ == '__main__':
unittest.main()
-97
View File
@@ -1,97 +0,0 @@
#!/usr/bin/env python3
"""Offline tests for scripts/haproxy-supervisor.sh (pidof / python are stubbed on PATH)."""
import os
import stat
import subprocess
import tempfile
import time
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
SCRIPT = os.path.join(HERE, 'haproxy-supervisor.sh')
PIDOF = """#!/bin/sh
# "haproxy" is alive iff the start stamp exists and is younger than $LIFE seconds
[ -f "$STATE/started" ] || exit 1
start=$(cat "$STATE/started")
now=$(date +%s.%N)
awk -v s="$start" -v n="$now" -v l="$LIFE" 'BEGIN{exit !(n - s < l)}'
"""
PYTHON = """#!/bin/sh
# stand-in for ensure_haproxy.py: stamp the start, record it
date +%s.%N > "$STATE/started"
date +%s.%N >> "$STATE/starts"
exit ${ENSURE_RC:-0}
"""
class SupervisorTests(unittest.TestCase):
def setUp(self):
self.d = tempfile.mkdtemp(prefix='hap-sup-')
self.state = os.path.join(self.d, 'state')
os.mkdir(self.state)
for name, body in (('pidof', PIDOF), ('python', PYTHON)):
p = os.path.join(self.d, name)
with open(p, 'w') as fh:
fh.write(body)
os.chmod(p, os.stat(p).st_mode | stat.S_IXUSR)
def run_sup(self, seconds, **env):
e = dict(os.environ, PATH=self.d + ':' + os.environ['PATH'], STATE=self.state)
e.setdefault('LIFE', '0')
e.update(env)
try:
subprocess.run(['bash', SCRIPT], env=e, timeout=seconds, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return 'exited'
except subprocess.TimeoutExpired:
return 'running' # still looping when the clock ran out = alive
def starts(self):
try:
with open(os.path.join(self.state, 'starts')) as fh:
return [float(x) for x in fh.read().split()]
except OSError:
return []
def test_non_integer_interval_does_not_kill_the_supervisor(self):
self.assertEqual(self.run_sup(8, HAPROXY_SUPERVISOR_INTERVAL='0.5', ENSURE_RC='1'), 'running')
self.assertGreaterEqual(len(self.starts()), 2) # fell back to the default interval and kept trying
def test_garbage_interval_and_max_fall_back(self):
self.assertEqual(self.run_sup(4, HAPROXY_SUPERVISOR_INTERVAL='abc', HAPROXY_SUPERVISOR_MAX_BACKOFF='x'), 'running')
self.assertGreaterEqual(len(self.starts()), 1)
def test_crash_loop_backs_off_2_5_10_capped(self):
# haproxy dies immediately after every successful start (LIFE=0) -> quick deaths
self.run_sup(14, HAPROXY_SUPERVISOR_INTERVAL='1', HAPROXY_SUPERVISOR_BACKOFF_STEPS='1 2 3',
HAPROXY_SUPERVISOR_QUICK_DEATH='5', HAPROXY_SUPERVISOR_STABLE='8',
HAPROXY_SUPERVISOR_MAX_BACKOFF='3')
s = self.starts()
gaps = [round(b - a, 1) for a, b in zip(s, s[1:])]
self.assertGreaterEqual(len(s), 4, gaps)
# gap = interval(1) + backoff step: 2, 3, 4, then capped at 1+3 = 4
self.assertLess(gaps[0], gaps[1] + 0.3, gaps)
self.assertGreaterEqual(gaps[1], 3.0 - 0.3, gaps)
self.assertGreaterEqual(gaps[2], 4.0 - 0.3, gaps)
self.assertLessEqual(max(gaps), 4.0 + 0.8, gaps)
def test_stable_haproxy_resets_the_backoff(self):
# alive for 2s after each start; QUICK_DEATH=1 so a 2s life is NOT quick: no backoff growth
self.run_sup(9, HAPROXY_SUPERVISOR_INTERVAL='1', LIFE='2', HAPROXY_SUPERVISOR_BACKOFF_STEPS='3 3 3',
HAPROXY_SUPERVISOR_QUICK_DEATH='1', HAPROXY_SUPERVISOR_STABLE='8',
HAPROXY_SUPERVISOR_MAX_BACKOFF='3')
s = self.starts()
gaps = [b - a for a, b in zip(s, s[1:])]
self.assertGreaterEqual(len(s), 3, gaps)
self.assertLess(max(gaps), 3.6, gaps) # never the 3 s backoff on top
def test_running_haproxy_is_left_alone(self):
with open(os.path.join(self.state, 'started'), 'w') as fh:
fh.write(str(time.time()))
self.assertEqual(self.run_sup(3, HAPROXY_SUPERVISOR_INTERVAL='1', LIFE='1000'), 'running')
self.assertEqual(self.starts(), [])
if __name__ == '__main__':
unittest.main()