Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  shadowdaoandClaude Sonnet 5
|
cbcaea8abb | ||
|
shadowdaoandClaude Sonnet 5
|
b025dec55e | ||
|
shadowdaoandClaude Opus 5
|
edddb7a9aa |
No files matched your search
@@ -0,0 +1 @@
|
||||
cloud-hosting-platform/haproxy-manager-base
|
||||
@@ -8,11 +8,6 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
- **API Testing**: `./scripts/test-api.sh` - Tests all API endpoints with optional authentication
|
||||
- **Certificate Request Testing**: `./scripts/test-certificate-request.sh` - Tests certificate generation endpoints
|
||||
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
|
||||
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
|
||||
- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
|
||||
- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
|
||||
- **WAF body inspection**: `python3 scripts/test-waf-body-inspection.py` - offline (runs in the image build); pins `option http-buffer-request`, the `X-WHP-WAF-Body` complete/partial stamp (strip -> stamp -> SPOE -> strip), `tune.bufsize` against the 100 ms SPOE fail-open budget and the 65535-byte SPOP frame cap, and Coraza's body limits in `coraza-spoa/config.yaml`. Run it after touching the Coraza block, `tune.bufsize`, or the SPOE engine.
|
||||
- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
|
||||
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
|
||||
|
||||
### Reading stick tables (and why it is easy to get silently wrong)
|
||||
@@ -36,29 +31,6 @@ reported "Scan Count"/"BLOCKED" figures parsed from `gpc0`/`gpc1` — fields no
|
||||
stick table has ever stored — for their entire existence. See the header of
|
||||
`haproxy_tarpit_config.txt` and the contract test.
|
||||
|
||||
### Changing a runtime map (`add map` / `del map`)
|
||||
|
||||
Same socket, two more ways to fail silently — and both were live in
|
||||
`add_ip_to_runtime_map()`/`remove_ip_from_runtime_map()` for their whole
|
||||
existence:
|
||||
|
||||
* **Reference the map by FILE PATH, never `#<id>`.** Ids are assigned at
|
||||
config-parse time and move on every config regeneration (on whp01
|
||||
`blocked_ips.map` is 37, `trusted_ips.map` is 10 — there is no id 0). Use
|
||||
`add map /etc/haproxy/blocked_ips.map <ip> 1`.
|
||||
* **A mutation answers NOTHING on success**, so an empty body is the only
|
||||
success — any output at all is a rejection. Worse, `@1 add map #0 <ip> 1`
|
||||
*also* answers nothing and adds nothing, so the body cannot prove an add
|
||||
worked. **Read it back** with `@1 get map <path> <key>`.
|
||||
* Entries must carry the value `1`; haproxy.cfg matches with
|
||||
`map_ip(...,0) -m int gt 0`, so a valueless entry does not block.
|
||||
|
||||
In Python use `haproxy_cli(cmd, worker=True, expect_empty=True)` for mutations
|
||||
and `runtime_map_lookup()` / `runtime_map_keys()` to verify. The runtime map is
|
||||
only a fast path: `/etc/haproxy/blocked_ips.map` is authoritative and HAProxy
|
||||
re-reads it on reload, so a failed runtime command must degrade to
|
||||
"enforced on reload" and be reported, never swallowed.
|
||||
|
||||
### Running the Application
|
||||
- **Docker Build**: `docker build -t haproxy-manager .`
|
||||
- **Local Development**: `python haproxy_manager.py` (requires HAProxy, certbot, and dependencies installed)
|
||||
@@ -169,8 +141,6 @@ re-reads it on reload, so a failed runtime command must degrade to
|
||||
- Exposes ports 80 (HTTP), 443 (HTTPS), and 8000 (management API/UI)
|
||||
- Stats page on port 8404 (localhost only inside container)
|
||||
- Management interface on port 8000 should be firewall-protected in production
|
||||
- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
|
||||
- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
|
||||
- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
|
||||
- Dockerfile HEALTHCHECK verifies both port 8000 (Flask API) and port 80 (HAProxy), with `start-period=60s` and `timeout=10s`
|
||||
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
|
||||
- HAProxy is version 3.0.11
|
||||
+2
-10
@@ -44,9 +44,6 @@ WORKDIR /haproxy
|
||||
COPY ./templates /haproxy/templates
|
||||
COPY requirements.txt /haproxy/
|
||||
COPY haproxy_manager.py /haproxy/
|
||||
COPY haproxy_gunicorn_worker.py /haproxy/
|
||||
# Read by GET /api/capabilities (WHP gates whole-site re-bundling on it).
|
||||
COPY VERSION /haproxy/VERSION
|
||||
COPY scripts /haproxy/scripts
|
||||
COPY trusted_ips.list /etc/haproxy/trusted_ips.list
|
||||
COPY trusted_ips.map /etc/haproxy/trusted_ips.map
|
||||
@@ -95,8 +92,6 @@ RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
|
||||
&& python3 /haproxy/scripts/test-config-rollback.py \
|
||||
&& python3 /haproxy/scripts/test-cert-write-safety.py \
|
||||
&& python3 /haproxy/scripts/test-cert-scripts.py \
|
||||
&& python3 /haproxy/scripts/test-waf-body-inspection.py \
|
||||
&& python3 /haproxy/scripts/test-gunicorn-socket-timeouts.py \
|
||||
&& python3 /haproxy/scripts/validate-rendered-config.py
|
||||
# Create log directories
|
||||
RUN mkdir -p /var/log && touch /var/log/haproxy-manager.log /var/log/haproxy-manager-errors.log
|
||||
@@ -110,10 +105,7 @@ RUN mkdir -p /var/spool/cron/crontabs && \
|
||||
# 443/udp carries HTTP/3 (QUIC). EXPOSE is documentation only — the container
|
||||
# must still be run with `-p 443:443/udp` for the UDP listener to be reachable.
|
||||
EXPOSE 80 443 443/udp 8000
|
||||
# Health check: management API (:8000, short timeout) AND HAProxy (:80).
|
||||
# See scripts/healthcheck.sh. NOTE: WHP's lib-edge-container.sh passes its own
|
||||
# --health-cmd, which REPLACES this one on deployed hosts; it must be pointed
|
||||
# at /haproxy/scripts/healthcheck.sh for the fleet to get the :8000 probe.
|
||||
# Add health check
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
||||
CMD /haproxy/scripts/healthcheck.sh
|
||||
CMD curl -sf --max-time 5 http://localhost:8000/health && curl -s --max-time 5 -o /dev/null http://localhost/ || exit 1
|
||||
CMD ["/haproxy/scripts/start-up.sh"]
|
||||
+5
-30
@@ -508,45 +508,20 @@ curl -X POST http://localhost:8000/api/blocked-ips/sync \
|
||||
|
||||
For advanced users, you can interact directly with HAProxy's runtime API:
|
||||
|
||||
Three things about these commands are easy to get wrong, and each one fails
|
||||
**silently** (socat exits 0 either way — the rejection, if any, is only in the
|
||||
response body):
|
||||
|
||||
* `/tmp/haproxy-cli` is HAProxy's **master** CLI socket. Map commands are
|
||||
worker commands and need the `@1` prefix. Without it the reply is
|
||||
`Unknown command: 'add', ...`.
|
||||
* Reference the map by its **file path**, never by `#<id>`. Ids are assigned at
|
||||
config-parse time and move on every config regeneration (on a live edge,
|
||||
`blocked_ips.map` is id 37, `trusted_ips.map` is 10 — there is no id 0).
|
||||
Worse, `@1 add map #0 <ip> 1` returns an **empty** reply and adds nothing.
|
||||
* Entries must carry the value `1`. `haproxy.cfg` matches with
|
||||
`map_ip(...,0) -m int gt 0`, so a valueless entry does not block. (`add map`
|
||||
with no value is rejected: `'add map' expects three parameters ...`.)
|
||||
|
||||
```bash
|
||||
MAP=/etc/haproxy/blocked_ips.map
|
||||
|
||||
# Add IP to runtime (immediate effect)
|
||||
echo "@1 add map $MAP 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
|
||||
echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
|
||||
|
||||
# Remove IP from runtime
|
||||
echo "@1 del map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
|
||||
|
||||
# Confirm what actually happened (do not trust the exit status)
|
||||
echo "@1 get map $MAP 192.168.1.100" | socat stdio /tmp/haproxy-cli
|
||||
echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
|
||||
|
||||
# Clear all blocked IPs from runtime
|
||||
echo "@1 clear map $MAP" | socat stdio /tmp/haproxy-cli
|
||||
echo "clear map #0" | socat stdio /var/run/haproxy.sock
|
||||
|
||||
# Show all runtime map entries, and the map ids currently in use
|
||||
echo "@1 show map $MAP" | socat stdio /tmp/haproxy-cli
|
||||
echo "@1 show map" | socat stdio /tmp/haproxy-cli
|
||||
# Show all runtime map entries
|
||||
echo "show map #0" | socat stdio /var/run/haproxy.sock
|
||||
```
|
||||
|
||||
The runtime map is a **fast path only**. `/etc/haproxy/blocked_ips.map` is
|
||||
authoritative: HAProxy re-reads it on reload, so a failed runtime command
|
||||
delays a block until the next reload rather than losing it.
|
||||
|
||||
## Migration from ACL Method
|
||||
|
||||
If you're upgrading from the old ACL-based method:
|
||||
|
||||
+2
-10
@@ -50,22 +50,14 @@ http-request deny status 403 if { src -f /etc/haproxy/blocked_ips.map }
|
||||
- **Graceful error handling**
|
||||
|
||||
### 2. Runtime IP Management
|
||||
Map commands go to a **worker** (`@1`), reference the map by **file path**
|
||||
(ids move between config regenerations, and `#0` silently adds nothing), and
|
||||
carry the value `1` that `map_ip(...,0) -m int gt 0` matches on:
|
||||
|
||||
```bash
|
||||
# Add IP without reload (immediate effect)
|
||||
echo "@1 add map /etc/haproxy/blocked_ips.map 192.168.1.100 1" | socat stdio /tmp/haproxy-cli
|
||||
echo "add map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
|
||||
|
||||
# Remove IP without reload
|
||||
echo "@1 del map /etc/haproxy/blocked_ips.map 192.168.1.100" | socat stdio /tmp/haproxy-cli
|
||||
echo "del map #0 192.168.1.100" | socat stdio /var/run/haproxy.sock
|
||||
```
|
||||
|
||||
socat exits 0 even when HAProxy rejects the command, so read the response body
|
||||
(or read the entry back with `@1 get map ...`) rather than the exit status.
|
||||
See IP_BLOCKING_API.md for the full set.
|
||||
|
||||
### 3. New API Endpoints
|
||||
|
||||
#### Safe Config Reload
|
||||
|
||||
@@ -26,37 +26,6 @@ applications:
|
||||
# CRS-bundled defaults: recommended Coraza settings + CRS setup +
|
||||
# the rule pack itself (~16 MB of rules embedded in the binary).
|
||||
Include @coraza.conf-recommended
|
||||
|
||||
# Request-body limits, set explicitly instead of inheriting the
|
||||
# recommended file's 12.5 MB / Reject. HAProxy only ever hands us one
|
||||
# buffer of body (tune.bufsize 16384 in haproxy-manager's
|
||||
# hap_header.tpl), and the SPOP library refuses frames over 65535
|
||||
# bytes, so no body we receive can exceed 64 KB. The invariant is:
|
||||
# tune.bufsize (16 KB) < SPOP frame cap (65535) < SecRequestBodyLimit
|
||||
# Breaking it is NOT harmless: SecRequestBodyLimitAction Reject
|
||||
# interrupts with 413 even under DetectionOnly (Coraza
|
||||
# setAndReturnBodyLimitInterruption ignores the engine mode), which the
|
||||
# edge turns into a 403 on every large upload. ProcessPartial is the
|
||||
# belt to that brace. InMemoryLimit == BodyLimit keeps the body buffer
|
||||
# off disk (this image is distroless). Pinned by
|
||||
# haproxy-manager-base/scripts/test-waf-body-inspection.py.
|
||||
SecRequestBodyLimit 131072
|
||||
SecRequestBodyInMemoryLimit 131072
|
||||
SecRequestBodyLimitAction ProcessPartial
|
||||
|
||||
# JSON nesting limit (the recommended file sets 1024). Coraza's JSON
|
||||
# flattener is quadratic in nesting: a 601-byte `[[[...]]]` of depth 300
|
||||
# cost 297 ms and depth 1000 (2 KB) 3.1 s of CPU -- past the 100 ms SPOE
|
||||
# timeout, so the request passed UNINSPECTED (fail-open) while the WAF
|
||||
# burned CPU. Over the limit the body fails to parse (REQBODY_ERROR,
|
||||
# 0.1 ms), which WHP's enforce-mode body-integrity rule denies on a
|
||||
# complete body. Legitimate depth, measured on the JSON shipped in
|
||||
# WordPress 6.x core, Gutenberg, WooCommerce and Elementor (theme.json,
|
||||
# styles/*.json, block.json, checkout schemas): max 8, so a global-styles
|
||||
# save is ~9; 64 leaves ~7x headroom for Divi/Elementor-style module
|
||||
# trees that could not be sampled. Depth 64 itself costs ~21 ms.
|
||||
SecRequestBodyJsonDepthLimit 64
|
||||
|
||||
Include @crs-setup.conf.example
|
||||
|
||||
# Runtime-managed PRE-CRS exclusions written by WHP UI. Empty by default.
|
||||
|
||||
@@ -47,17 +47,7 @@ SecRuleUpdateActionById 933170-933200 "ctl:ruleEngine=On"
|
||||
# vulnerable, but blocking at the edge keeps logs clean and protects
|
||||
# any future Java workloads.
|
||||
# ---------------------------------------------------------------------------
|
||||
# 944110 and 944120 are NOT promoted (changed 2026-10-09, 2026.10.2). Both are
|
||||
# CHAINS whose first link matches bare words anywhere in the request --
|
||||
# `runtime|processbuilder` and `getproperty|forclosure|filewriter|...` -- and
|
||||
# Coraza runs an appended ctl:ruleEngine=On on that first link even when the
|
||||
# chain then fails. Measured (Coraza v3.7.0 + CRS 4.25.0): in detect_only,
|
||||
# `?q=runtime&id=1' or '1'='1` was 403 while the SQLi alone was 200. Same
|
||||
# chain-starter defect as 920440. They still add to the anomaly score.
|
||||
# Log4Shell (944150/944151/944152, ...) stays promoted. WHP mirrors this split
|
||||
# in coraza_baked_promotion_ranges() (web-files/libs/coraza_blocking_set.php).
|
||||
SecRuleUpdateActionById 944100 "ctl:ruleEngine=On"
|
||||
SecRuleUpdateActionById 944130-944300 "ctl:ruleEngine=On"
|
||||
SecRuleUpdateActionById 944100-944300 "ctl:ruleEngine=On"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 920440 — URL file extension restricted by policy
|
||||
@@ -69,26 +59,7 @@ SecRuleUpdateActionById 944130-944300 "ctl:ruleEngine=On"
|
||||
# single GCP-hosted scanner. Zero false positives observed; standard
|
||||
# WP/WooCommerce/Divi/HPR URLs do not end in these extensions.
|
||||
# ---------------------------------------------------------------------------
|
||||
# HOW IT IS PROMOTED (changed 2026-10-09, haproxy-manager-base 2026.10.2).
|
||||
# This used to be `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"`, which
|
||||
# was WRONG: 920440 is a CHAIN whose first link is
|
||||
# `REQUEST_BASENAME "@rx \.([^.]+)$"` -- it matches EVERY URL with a file
|
||||
# extension -- and Coraza runs actions appended to a chain starter when that
|
||||
# first link matches, even though the chain then fails. Measured (Coraza
|
||||
# v3.7.0 + CRS 4.25.0, in-process and live): every *.php / *.css / ...
|
||||
# request ran with the engine On, so the whole CRS anomaly score (949110) and
|
||||
# 200002 enforced there -- in detect_only mode too, since SecRuleEngine
|
||||
# DetectionOnly cannot override a runtime ctl.
|
||||
#
|
||||
# Now: 920440 itself is untouched (it still sets tx.extension in its first
|
||||
# link and adds the anomaly score in its second), and this separate phase-1
|
||||
# rule, which runs after it, flips the engine only when the extension really
|
||||
# is in tx.restricted_extensions -- the original intent. Removing 920440
|
||||
# (break-glass, a per-host exception) also removes tx.extension, so this rule
|
||||
# goes quiet with it. Id 992920440 is outside WHP's generated range
|
||||
# (990000000-991999999).
|
||||
SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \
|
||||
"id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"
|
||||
SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 930130 — Restricted File Access Attempt
|
||||
|
||||
@@ -1,502 +0,0 @@
|
||||
"""gunicorn gthread worker that cannot be wedged by silent or slow clients.
|
||||
|
||||
Incident 2026-10-09 (sdbees), and very likely the 2026-07-07 whp01 one too
|
||||
-------------------------------------------------------------------------
|
||||
Both :8000 API workers had 7 of 8 pool threads parked in the kernel in
|
||||
``tcp_recvmsg -> sk_wait_data``: a blocking socket read with no timeout.
|
||||
/health and /api/blocked-ips hung for 40s+, the container stayed "healthy".
|
||||
|
||||
haproxy_manager.py makes NO in-process outbound network reads (no requests,
|
||||
urllib, http.client, raw socket or ssl client anywhere; every ACME/HAProxy
|
||||
interaction is a subprocess, already bounded since 2026.07.3). The reads that
|
||||
blocked were gunicorn's own reads of the CLIENT socket:
|
||||
|
||||
gunicorn/workers/gthread.py TConn.init(): self.sock.setblocking(True)
|
||||
|
||||
Once the main loop sees a connection become readable it hands it to a pool
|
||||
thread and turns the socket fully blocking -- timeout None -- before parsing
|
||||
the request. Any peer that sends a few bytes and then waits (a TLS ClientHello
|
||||
sent to this plain-HTTP port, a port scanner, a half-written request) pins
|
||||
that pool thread forever. :8000 was published on 0.0.0.0 by the deploy
|
||||
(`-p 8000:8000`, which bypasses firewalld) and internet scanners were measured
|
||||
connecting to it. 4 such connections exhaust a worker's pool.
|
||||
|
||||
gunicorn's --timeout does not catch this: for gthread it only checks that the
|
||||
worker's MAIN thread keeps heart-beating, and the main thread keeps polling
|
||||
happily while every pool thread is stuck. --max-requests does not help either:
|
||||
a worker whose threads never finish a request never reaches the limit.
|
||||
socket.setdefaulttimeout() does not help: setblocking(True) explicitly resets
|
||||
the timeout to None, overriding any default.
|
||||
|
||||
Stock gthread has two more exhaustion paths on the same port (review F1/F2):
|
||||
|
||||
* A connection that NEVER sends a byte never becomes readable, so it is never
|
||||
handed to a thread and never gets any deadline -- but it holds one of the
|
||||
worker's ``worker_connections`` (1000) slots. At 1000 the run loop stops
|
||||
polling altogether and only waits on futures; none exist, so even the
|
||||
attacker's FINs are never seen and the worker is wedged PERMANENTLY, still
|
||||
heart-beating, with no request in flight for a watchdog to notice.
|
||||
* Slowloris: one header byte every <timeout seconds never trips a per-recv
|
||||
timeout, so a socket timeout alone does not bound header-read time.
|
||||
|
||||
What this worker changes
|
||||
------------------------
|
||||
1. CLIENT_TIMEOUT (HAPROXY_MGR_CLIENT_TIMEOUT, default 20s): every client
|
||||
socket gets an explicit socket timeout at the exact point gthread makes it
|
||||
blocking (re-applied on every keep-alive reuse). Python applies it to each
|
||||
recv() call and to each WHOLE sendall() call -- gunicorn writes a response
|
||||
chunk with one sendall, so any single chunk that takes longer than this to
|
||||
transmit is cut off. Responses here are KB-sized JSON/PEMs to localhost, so
|
||||
that never happens in practice. It does NOT bound how long the app takes to
|
||||
handle a request (a long certbot run is a subprocess with its own timeout
|
||||
and touches the client socket only once it is done).
|
||||
|
||||
2. FIRST_BYTE_TIMEOUT (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, default 10s): an
|
||||
accepted connection that has not become readable within this is closed by
|
||||
the main loop, freeing its slot (review F1).
|
||||
|
||||
3. HEADER_DEADLINE (HAPROXY_MGR_HEADER_DEADLINE, default 10s): total time a
|
||||
client may take to deliver the request line + headers, counted from when
|
||||
the connection was ENQUEUED for a thread, with a floor of
|
||||
HAPROXY_MGR_HEADER_PICKUP_GRACE (1s) after a thread picks it up:
|
||||
deadline = max(enqueue + 10s, pickup + 1s). Disarmed in handle_request()
|
||||
(which gunicorn calls only once the parser has returned a complete
|
||||
request). A 1s monitor loop shuts down (SHUT_RDWR, not close -- no fd-reuse
|
||||
race with the blocked recv) the socket of any connection past it; the
|
||||
blocked recv returns EOF and the thread is freed (review F2). A legit
|
||||
request queued behind busy threads (certbot runs) already has its headers
|
||||
in the kernel buffer and parses in milliseconds inside the 1s grace; a
|
||||
slow client that sat in the queue gets only the grace, so reconnecting
|
||||
drippers cannot starve the pool by queueing (re-review). Bodies are only
|
||||
read by routes behind require_api_key, so the unauthenticated surface is
|
||||
exactly this phase.
|
||||
|
||||
4. Wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS, default 3600s; 0 disables). If
|
||||
EVERY pool thread has been inside the same request for longer than that the
|
||||
worker logs every thread's Python stack to the error log and exits; the
|
||||
gunicorn master respawns it in about a second, without touching HAProxy or
|
||||
the edge. 3600s because a multi-domain /api/certificates/request runs one
|
||||
certbot (up to 180s) per domain; with (1)-(3) closing the known client-side
|
||||
sources, this is only a backstop for blocking sources not yet found.
|
||||
|
||||
5. Peer filter (re-review R1). haproxy-manager shares the client-net bridge
|
||||
with every tenant container, so tenant PHP could reach haproxy-manager:8000
|
||||
directly; the 127.0.0.1 publish only removes the internet path. On the
|
||||
ports in HAPROXY_MGR_PEER_FILTER_PORTS (default "8000" -- NOT the :8080
|
||||
default-page server, which HAProxy feeds over loopback) accept() closes
|
||||
any connection whose peer is not in HAPROXY_MGR_API_ALLOWED_PEERS (default
|
||||
"127.0.0.1,::1,gateway"; IPs/CIDRs; "gateway" = this container's default
|
||||
gateway, which is where host callers arrive from through docker-proxy).
|
||||
Refused before a thread or connection slot is used; logged at most once a
|
||||
minute as a per-peer summary. Effective only once the edge publishes
|
||||
127.0.0.1:8000 -- with a 0.0.0.0 publish a tenant can hairpin through the
|
||||
host and arrive from the gateway too.
|
||||
|
||||
6. pool_status() exposes busy/total threads and the oldest in-flight age so
|
||||
/health can report saturation.
|
||||
|
||||
This overrides gthread internals (enqueue_req, accept,
|
||||
on_client_socket_readable, murder_keepalived, handle, handle_request) as they
|
||||
are in gunicorn 23.0.0, which requirements.txt pins. A warning is logged if a
|
||||
different version is loaded; scripts/test-gunicorn-socket-timeouts.py runs
|
||||
real gunicorn in the Docker build gate and fails on behavioural drift.
|
||||
"""
|
||||
|
||||
import errno
|
||||
import ipaddress
|
||||
import logging
|
||||
import os
|
||||
import selectors
|
||||
import socket
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
from collections import deque
|
||||
from functools import partial
|
||||
|
||||
import gunicorn
|
||||
from gunicorn.workers.gthread import TConn, ThreadWorker
|
||||
|
||||
|
||||
def _env_float(name, default):
|
||||
try:
|
||||
return float(os.environ.get(name, default))
|
||||
except (TypeError, ValueError):
|
||||
return float(default)
|
||||
|
||||
|
||||
CLIENT_TIMEOUT = _env_float('HAPROXY_MGR_CLIENT_TIMEOUT', 20)
|
||||
FIRST_BYTE_TIMEOUT = _env_float('HAPROXY_MGR_FIRST_BYTE_TIMEOUT', 10)
|
||||
HEADER_DEADLINE = _env_float('HAPROXY_MGR_HEADER_DEADLINE', 10)
|
||||
WEDGE_SECONDS = _env_float('HAPROXY_MGR_WEDGE_SECONDS', 3600)
|
||||
WATCHDOG_INTERVAL = _env_float('HAPROXY_MGR_WATCHDOG_INTERVAL', 15)
|
||||
MONITOR_TICK = 1.0
|
||||
# Once a thread picks a queued connection up it gets at least this long to
|
||||
# finish its headers, even if its enqueue-based deadline has already passed.
|
||||
HEADER_PICKUP_GRACE = _env_float('HAPROXY_MGR_HEADER_PICKUP_GRACE', 1)
|
||||
|
||||
# Peer filter (review R1). Listener ports it applies to (comma-separated;
|
||||
# empty disables) and the peers allowed through. "gateway" means this
|
||||
# container's default gateway on its bridge network -- the address host-side
|
||||
# callers arrive from via the 127.0.0.1:8000 publish (docker-proxy) and via a
|
||||
# direct host->container-IP connect. Tenant containers on the same bridge
|
||||
# (client-net) arrive from their OWN addresses and are refused.
|
||||
PEER_FILTER_PORTS = os.environ.get('HAPROXY_MGR_PEER_FILTER_PORTS', '8000')
|
||||
ALLOWED_PEERS_SPEC = os.environ.get('HAPROXY_MGR_API_ALLOWED_PEERS', '127.0.0.1,::1,gateway')
|
||||
PEER_REJECT_LOG_INTERVAL = 60.0
|
||||
|
||||
|
||||
def _default_gateways(route_file='/proc/net/route'):
|
||||
"""IPv4 default gateway(s) from the kernel routing table."""
|
||||
gws = []
|
||||
try:
|
||||
with open(route_file) as fh:
|
||||
next(fh, None)
|
||||
for line in fh:
|
||||
f = line.split()
|
||||
if len(f) >= 4 and f[1] == '00000000' and int(f[3], 16) & 0x2:
|
||||
gws.append(str(ipaddress.IPv4Address(int(f[2], 16).to_bytes(4, 'little'))))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
return gws
|
||||
|
||||
|
||||
def parse_allowed_peers(spec, gateways=None):
|
||||
"""Return (networks, problems). Tokens: IPs, CIDRs, or the word
|
||||
'gateway'. Unparseable tokens are reported, never silently widened."""
|
||||
nets, problems = [], []
|
||||
for tok in (t.strip() for t in (spec or '').split(',')):
|
||||
if not tok:
|
||||
continue
|
||||
if tok.lower() == 'gateway':
|
||||
gws = _default_gateways() if gateways is None else gateways
|
||||
if not gws:
|
||||
problems.append('gateway: no IPv4 default route found')
|
||||
for g in gws:
|
||||
nets.append(ipaddress.ip_network(g + '/32'))
|
||||
continue
|
||||
try:
|
||||
nets.append(ipaddress.ip_network(tok, strict=False))
|
||||
except ValueError:
|
||||
problems.append('unparseable entry %r' % tok)
|
||||
return nets, problems
|
||||
|
||||
|
||||
def peer_allowed(addr, nets):
|
||||
try:
|
||||
ip = ipaddress.ip_address(addr.split('%', 1)[0])
|
||||
except (ValueError, AttributeError):
|
||||
return False
|
||||
if ip.version == 6 and ip.ipv4_mapped is not None:
|
||||
ip = ip.ipv4_mapped
|
||||
return any(ip.version == n.version and ip in n for n in nets)
|
||||
|
||||
TESTED_GUNICORN_VERSION = '23.0.0'
|
||||
|
||||
# Exit status for a watchdog-killed worker. Must NOT be 3 (WORKER_BOOT_ERROR)
|
||||
# or 4 (APP_LOAD_ERROR): on those the gunicorn master halts the whole server
|
||||
# instead of respawning the worker.
|
||||
WEDGED_EXIT_CODE = 1
|
||||
|
||||
_current_worker = None
|
||||
|
||||
|
||||
def pool_status():
|
||||
"""Busy/total threads of the worker serving the caller, or None when the
|
||||
app is not running under this worker class (e.g. tests, dev server)."""
|
||||
w = _current_worker
|
||||
if w is None:
|
||||
return None
|
||||
return w.pool_status()
|
||||
|
||||
|
||||
class _QuietClientTimeout(logging.Filter):
|
||||
"""gthread logs any OSError without an errno it recognises as
|
||||
log.exception("Socket error processing request.") -- a full traceback.
|
||||
A client timeout is now an expected, bounded event (scanners hit this port
|
||||
constantly), so reduce it to one line instead of flooding the error log."""
|
||||
|
||||
def filter(self, record):
|
||||
exc = record.exc_info[1] if record.exc_info else None
|
||||
if isinstance(exc, TimeoutError) and \
|
||||
str(record.msg).startswith('Socket error processing request'):
|
||||
record.msg = ('Closed client connection idle for more than %ss '
|
||||
'(HAPROXY_MGR_CLIENT_TIMEOUT)' % CLIENT_TIMEOUT)
|
||||
record.args = ()
|
||||
record.exc_info = None
|
||||
record.exc_text = None
|
||||
record.levelno = logging.INFO
|
||||
record.levelname = 'INFO'
|
||||
return True
|
||||
|
||||
|
||||
class _InFlight:
|
||||
__slots__ = ('conn', 'start', 'header_deadline')
|
||||
|
||||
def __init__(self, conn, start, header_deadline):
|
||||
self.conn = conn
|
||||
self.start = start
|
||||
self.header_deadline = header_deadline
|
||||
|
||||
|
||||
class TimeoutThreadWorker(ThreadWorker):
|
||||
"""gthread worker with client-socket timeouts, a first-byte deadline, a
|
||||
total header deadline and a wedge watchdog. See module docstring."""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
self._inflight = {} # thread ident -> _InFlight
|
||||
self._inflight_lock = threading.Lock()
|
||||
self._pending = deque() # accepted, not yet readable (FIFO by deadline)
|
||||
self._filter_ports = set()
|
||||
for tok in PEER_FILTER_PORTS.split(','):
|
||||
tok = tok.strip()
|
||||
if tok.isdigit():
|
||||
self._filter_ports.add(int(tok))
|
||||
self._allowed_nets = []
|
||||
self._rejected = {} # peer ip -> count since last log line
|
||||
self._rejected_lock = threading.Lock()
|
||||
self._last_reject_log = 0.0
|
||||
|
||||
# -- (1) client socket timeout ------------------------------------------
|
||||
def enqueue_req(self, conn):
|
||||
# Same three steps as ThreadWorker.enqueue_req, with the timeout set
|
||||
# right after conn.init() -- init() is what calls setblocking(True)
|
||||
# (== settimeout(None)), and it runs again on every keep-alive reuse,
|
||||
# so the timeout must be re-applied here each time, not at accept().
|
||||
conn.init()
|
||||
if CLIENT_TIMEOUT > 0:
|
||||
conn.sock.settimeout(CLIENT_TIMEOUT)
|
||||
conn.whp_enqueued = time.monotonic() # header deadline counts from here
|
||||
fs = self.tpool.submit(self.handle, conn)
|
||||
self._wrap_future(fs, conn)
|
||||
|
||||
# -- (2) first-byte deadline for never-readable connections ---------------
|
||||
def accept(self, server, listener):
|
||||
# ThreadWorker.accept (23.0.0) plus: a deadline, and the conn is
|
||||
# tracked in self._pending so murder_keepalived() can reap it.
|
||||
try:
|
||||
sock, client = listener.accept()
|
||||
if self._filter_ports and self._server_port(server) in self._filter_ports \
|
||||
and not peer_allowed(client[0] if isinstance(client, tuple) else '',
|
||||
self._allowed_nets):
|
||||
# Refused before it costs a TConn, a connection slot, a poller
|
||||
# registration or a thread.
|
||||
sock.close()
|
||||
self._note_rejected(client[0] if isinstance(client, tuple) else str(client))
|
||||
return
|
||||
conn = TConn(self.cfg, sock, client, server)
|
||||
conn.timeout = time.time() + FIRST_BYTE_TIMEOUT
|
||||
|
||||
self.nr_conns += 1
|
||||
with self._lock:
|
||||
self._pending.append(conn)
|
||||
self.poller.register(conn.sock, selectors.EVENT_READ,
|
||||
partial(self.on_client_socket_readable, conn))
|
||||
except OSError as e:
|
||||
if e.errno not in (errno.EAGAIN, errno.ECONNABORTED,
|
||||
errno.EWOULDBLOCK):
|
||||
raise
|
||||
|
||||
@staticmethod
|
||||
def _server_port(server):
|
||||
try:
|
||||
return int(server[1])
|
||||
except (TypeError, IndexError, ValueError):
|
||||
return None
|
||||
|
||||
def _note_rejected(self, peer):
|
||||
with self._rejected_lock:
|
||||
self._rejected[peer] = self._rejected.get(peer, 0) + 1
|
||||
if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
|
||||
self._flush_rejected()
|
||||
|
||||
def _flush_rejected(self):
|
||||
with self._rejected_lock:
|
||||
if not self._rejected:
|
||||
return
|
||||
items, self._rejected = self._rejected, {}
|
||||
self._last_reject_log = time.monotonic()
|
||||
total = sum(items.values())
|
||||
top = sorted(items.items(), key=lambda kv: -kv[1])[:5]
|
||||
self.log.warning(
|
||||
'Refused %d connection(s) to the management API from %d non-allowed peer(s) '
|
||||
'(HAPROXY_MGR_API_ALLOWED_PEERS=%s): %s%s', total, len(items), ALLOWED_PEERS_SPEC,
|
||||
', '.join('%s x%d' % kv for kv in top), ' ...' if len(items) > 5 else '')
|
||||
|
||||
def on_client_socket_readable(self, conn, client):
|
||||
# ThreadWorker version only knows about self._keep; a first request
|
||||
# comes off self._pending instead.
|
||||
with self._lock:
|
||||
self.poller.unregister(client)
|
||||
queue = self._keep if conn.initialized else self._pending
|
||||
try:
|
||||
queue.remove(conn)
|
||||
except ValueError:
|
||||
# already reaped (race with murder_keepalived)
|
||||
return
|
||||
self.enqueue_req(conn)
|
||||
|
||||
def murder_keepalived(self):
|
||||
# Runs on every pass of ThreadWorker.run() -- including the branch
|
||||
# taken when nr_conns >= worker_connections, where the poller is not
|
||||
# serviced. Reaping here is what breaks that permanent deadlock.
|
||||
super().murder_keepalived()
|
||||
if FIRST_BYTE_TIMEOUT <= 0:
|
||||
return
|
||||
now = time.time()
|
||||
while True:
|
||||
with self._lock:
|
||||
if not self._pending or self._pending[0].timeout > now:
|
||||
break
|
||||
conn = self._pending.popleft()
|
||||
try:
|
||||
self.poller.unregister(conn.sock)
|
||||
except (OSError, KeyError, ValueError):
|
||||
pass
|
||||
self.nr_conns -= 1
|
||||
conn.close()
|
||||
|
||||
# -- (3) header deadline + busy tracking ----------------------------------
|
||||
def handle(self, conn):
|
||||
ident = threading.get_ident()
|
||||
now = time.monotonic()
|
||||
deadline = None
|
||||
if HEADER_DEADLINE > 0:
|
||||
# max(enqueue + D, pickup + grace): a legit request that waited in
|
||||
# the queue behind busy threads already has its headers in the
|
||||
# kernel buffer and parses in milliseconds; a slow client that
|
||||
# waited gets only the grace, not a fresh full deadline (review:
|
||||
# 40 reconnecting drippers otherwise starved legit requests).
|
||||
enq = getattr(conn, 'whp_enqueued', now)
|
||||
deadline = max(enq + HEADER_DEADLINE, now + HEADER_PICKUP_GRACE)
|
||||
entry = _InFlight(conn, now, deadline)
|
||||
with self._inflight_lock:
|
||||
self._inflight[ident] = entry
|
||||
try:
|
||||
return super().handle(conn)
|
||||
finally:
|
||||
with self._inflight_lock:
|
||||
self._inflight.pop(ident, None)
|
||||
|
||||
def handle_request(self, req, conn):
|
||||
# The parser has produced a complete request line + headers.
|
||||
entry = self._inflight.get(threading.get_ident())
|
||||
if entry is not None:
|
||||
entry.header_deadline = None
|
||||
return super().handle_request(req, conn)
|
||||
|
||||
def _sweep_header_deadlines(self):
|
||||
now = time.monotonic()
|
||||
with self._inflight_lock:
|
||||
late = [e for e in self._inflight.values()
|
||||
if e.header_deadline is not None and e.header_deadline < now]
|
||||
for e in late:
|
||||
e.header_deadline = None # one shutdown per connection
|
||||
try:
|
||||
# shutdown, not close: wakes the blocked recv() with EOF
|
||||
# without releasing the fd while another thread is in it.
|
||||
e.conn.sock.shutdown(socket.SHUT_RDWR)
|
||||
except OSError:
|
||||
pass
|
||||
if late:
|
||||
self.log.info('Cut %d client connection(s) still sending request headers '
|
||||
'after %ss (HAPROXY_MGR_HEADER_DEADLINE)', len(late), HEADER_DEADLINE)
|
||||
|
||||
def pool_status(self):
|
||||
now = time.monotonic()
|
||||
with self._inflight_lock:
|
||||
starts = [e.start for e in self._inflight.values()]
|
||||
return {
|
||||
'pid': os.getpid(),
|
||||
'threads': self.cfg.threads,
|
||||
'busy': len(starts),
|
||||
'oldest_busy_seconds': round(now - min(starts), 1) if starts else 0.0,
|
||||
'client_timeout_seconds': CLIENT_TIMEOUT,
|
||||
}
|
||||
|
||||
# -- (4) watchdog ---------------------------------------------------------
|
||||
def _wedged_for(self):
|
||||
"""Seconds the pool has been completely wedged (every thread busy in
|
||||
its current request), measured from the YOUNGEST in-flight request.
|
||||
0 when at least one thread is free."""
|
||||
with self._inflight_lock:
|
||||
starts = [e.start for e in self._inflight.values()]
|
||||
if len(starts) < self.cfg.threads:
|
||||
return 0.0
|
||||
return time.monotonic() - max(starts)
|
||||
|
||||
def _dump_stacks(self):
|
||||
frames = sys._current_frames()
|
||||
names = {t.ident: t.name for t in threading.enumerate()}
|
||||
out = []
|
||||
for ident, frame in frames.items():
|
||||
out.append('--- thread %s (%s)\n%s' % (
|
||||
ident, names.get(ident, '?'), ''.join(traceback.format_stack(frame))))
|
||||
return '\n'.join(out)
|
||||
|
||||
def _check_wedged(self):
|
||||
wedged = self._wedged_for()
|
||||
if WEDGE_SECONDS > 0 and wedged > WEDGE_SECONDS:
|
||||
try:
|
||||
self.log.critical(
|
||||
'WEDGED WORKER (pid %s): all %s threads busy in the same request '
|
||||
'for %.0fs > HAPROXY_MGR_WEDGE_SECONDS=%s. Exiting so the master '
|
||||
'respawns it. Thread stacks:\n%s',
|
||||
os.getpid(), self.cfg.threads, wedged, WEDGE_SECONDS,
|
||||
self._dump_stacks())
|
||||
for h in logging.getLogger('gunicorn.error').handlers:
|
||||
try:
|
||||
h.flush()
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
# os._exit, not SIGTERM: a graceful exit joins the (stuck,
|
||||
# non-daemon) pool threads at interpreter shutdown and would
|
||||
# hang until the master's --timeout escalates to SIGKILL.
|
||||
os._exit(WEDGED_EXIT_CODE)
|
||||
|
||||
def _monitor(self):
|
||||
"""One daemon thread: header-deadline sweep every second, wedge check
|
||||
every WATCHDOG_INTERVAL. Never allowed to die silently."""
|
||||
last_wedge_check = time.monotonic()
|
||||
while self.alive:
|
||||
time.sleep(MONITOR_TICK)
|
||||
try:
|
||||
if HEADER_DEADLINE > 0:
|
||||
self._sweep_header_deadlines()
|
||||
if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
|
||||
self._flush_rejected()
|
||||
if time.monotonic() - last_wedge_check >= WATCHDOG_INTERVAL:
|
||||
last_wedge_check = time.monotonic()
|
||||
self._check_wedged()
|
||||
except Exception:
|
||||
try:
|
||||
self.log.exception('haproxy_gunicorn_worker monitor error (continuing)')
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def init_process(self):
|
||||
global _current_worker
|
||||
_current_worker = self
|
||||
try:
|
||||
self.log.error_log.addFilter(_QuietClientTimeout())
|
||||
except AttributeError:
|
||||
pass
|
||||
if gunicorn.__version__ != TESTED_GUNICORN_VERSION:
|
||||
self.log.warning(
|
||||
'haproxy_gunicorn_worker overrides gthread internals as of gunicorn %s; '
|
||||
'running %s -- re-run scripts/test-gunicorn-socket-timeouts.py',
|
||||
TESTED_GUNICORN_VERSION, gunicorn.__version__)
|
||||
if self._filter_ports:
|
||||
self._allowed_nets, problems = parse_allowed_peers(ALLOWED_PEERS_SPEC)
|
||||
for prob in problems:
|
||||
self.log.error('HAPROXY_MGR_API_ALLOWED_PEERS: %s -- that entry allows nothing '
|
||||
'(fail closed)', prob)
|
||||
self.log.info('Management API peer filter on port(s) %s: allowing %s',
|
||||
','.join(str(p) for p in sorted(self._filter_ports)),
|
||||
', '.join(str(n) for n in self._allowed_nets) or 'NOTHING')
|
||||
threading.Thread(target=self._monitor, name='worker-monitor',
|
||||
daemon=True).start()
|
||||
super().init_process() # runs the event loop; does not return
|
||||
+198
-895
File diff suppressed because it is too large.
Load diff
@@ -203,82 +203,6 @@ cert_bundle_valid() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# cert_is_expired CERT_FILE
|
||||
#
|
||||
# Returns 0 only when openssl positively reports that the first certificate in
|
||||
# CERT_FILE has expired. Anything else (not expired, unreadable, no openssl)
|
||||
# returns 1; unreadable material is rejected with a better message by
|
||||
# cert_bundle_valid().
|
||||
#
|
||||
# Why: HAProxy loads the crt directory wholesale and serves the FIRST file that
|
||||
# claims an SNI name. An expired bundle in there is never useful and can only
|
||||
# shadow a valid one - on 2026-10-07 renew-certificates.sh kept re-publishing
|
||||
# the expired brain-jar.com lineage every 12 hours, and HAProxy served it for
|
||||
# brainjar.net instead of the valid brainjar.net bundle next to it.
|
||||
cert_is_expired() {
|
||||
local file="$1" out
|
||||
[ -f "$file" ] || return 1
|
||||
command -v openssl >/dev/null 2>&1 || return 1
|
||||
out="$(openssl x509 -checkend 0 -noout -in "$file" 2>/dev/null </dev/null)"
|
||||
[ $? -eq 1 ] && [[ "$out" == *"will expire"* ]]
|
||||
}
|
||||
|
||||
# cert_not_after CERT_FILE - the notAfter date string, or empty.
|
||||
cert_not_after() {
|
||||
openssl x509 -enddate -noout -in "$1" 2>/dev/null </dev/null | sed 's/^notAfter=//'
|
||||
}
|
||||
|
||||
cert_retired_dir() {
|
||||
if [ -n "${CERT_RETIRED_DIR:-}" ]; then
|
||||
echo "$CERT_RETIRED_DIR"
|
||||
else
|
||||
echo "$(dirname "${SSL_CERTS_DIR:-/etc/haproxy/certs}")/cert-retired"
|
||||
fi
|
||||
}
|
||||
|
||||
# cert_retire_expired_pems
|
||||
#
|
||||
# Move every EXPIRED *.pem out of $SSL_CERTS_DIR into cert-retired/ (a sibling,
|
||||
# never inside the crt dir). Sets CERT_RETIRED_EXPIRED to the number moved.
|
||||
#
|
||||
# Not publishing expired lineages stops the re-publish loop, but a pem that was
|
||||
# published while valid and has expired since stays in the crt dir and keeps
|
||||
# shadowing - that was exactly brain-jar.com.pem (review M2). An expired
|
||||
# certificate is never a valid certificate for anyone, so it can always go: the
|
||||
# only-valid-cert safety rule cannot be violated by moving it. Two exceptions:
|
||||
# * default_self_signed_cert.pem - HAProxy's fallback, regenerated separately;
|
||||
# * the LAST .pem in the directory - a crt directory with no certificate at
|
||||
# all fails the whole :443 bind, which is worse than serving an expired one.
|
||||
# Nothing is deleted; reverse with mv.
|
||||
cert_retire_expired_pems() {
|
||||
local dir="${SSL_CERTS_DIR:-/etc/haproxy/certs}" retired f base dest remaining
|
||||
CERT_RETIRED_EXPIRED=0
|
||||
retired="$(cert_retired_dir)"
|
||||
for f in "$dir"/*.pem; do
|
||||
[ -f "$f" ] || continue
|
||||
base="$(basename "$f")"
|
||||
[ "$base" = "default_self_signed_cert.pem" ] && continue
|
||||
cert_is_expired "$f" || continue
|
||||
remaining="$(find "$dir" -maxdepth 1 -type f -name '*.pem' | wc -l)"
|
||||
if [ "$remaining" -le 1 ]; then
|
||||
log_error "Expired certificate $f is the only certificate left in $dir - NOT retiring it (an empty crt directory fails the :443 bind)"
|
||||
continue
|
||||
fi
|
||||
if ! mkdir -p "$retired"; then
|
||||
log_error "cannot create $retired - expired $f left in place"
|
||||
continue
|
||||
fi
|
||||
dest="$retired/$base"
|
||||
[ -e "$dest" ] && dest="$dest.$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
if mv -- "$f" "$dest"; then
|
||||
log_error "Retired EXPIRED certificate $base (notAfter $(cert_not_after "$dest")) from $dir to $dest - it could only shadow valid certificates"
|
||||
CERT_RETIRED_EXPIRED=$((CERT_RETIRED_EXPIRED + 1))
|
||||
else
|
||||
log_error "failed to move expired $f to $dest"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# cert_publish CERT_FILE KEY_FILE DEST_FILE
|
||||
#
|
||||
# Assemble CERT_FILE + KEY_FILE into DEST_FILE without ever exposing a
|
||||
@@ -303,11 +227,6 @@ cert_publish() {
|
||||
log_error "cert_publish: private key $key_file is missing or empty"
|
||||
return 1
|
||||
fi
|
||||
# (a2) never publish an expired certificate (see cert_is_expired).
|
||||
if cert_is_expired "$cert_file"; then
|
||||
log_error "cert_publish: certificate $cert_file expired ($(cert_not_after "$cert_file")) - refusing to publish $dest_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# (b) assemble in the staging dir - NOT in the certs dir, which HAProxy
|
||||
# scans wholesale.
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Container liveness probe: BOTH planes must answer quickly.
|
||||
#
|
||||
# 1. Management API (:8000) -- GET /health with a short --max-time. On
|
||||
# 2026-10-09 (sdbees) the API workers' thread pools were wedged for long
|
||||
# enough that /health hung 40s+, while the container reported "healthy"
|
||||
# because the deployed --health-cmd only probed :80. A hung API is a dead
|
||||
# control plane (every WHP site update times out), so it must turn the
|
||||
# container unhealthy.
|
||||
# 2. HAProxy data plane (:80) -- any HTTP answer at all (the default page,
|
||||
# a redirect, a 4xx/5xx) proves HAProxy accepted and answered. Only a
|
||||
# connect/timeout failure fails it, which is why there is no -f here.
|
||||
#
|
||||
# 127.0.0.1, not localhost: the API binds IPv4 only, and resolving localhost
|
||||
# to ::1 first costs a failed connect on every probe.
|
||||
#
|
||||
# WHP's lib-edge-container.sh overrides the image HEALTHCHECK with its own
|
||||
# --health-cmd; point that at this script so the fleet gets the same probe.
|
||||
set -u
|
||||
API_TIMEOUT="${HAPROXY_MGR_HEALTH_API_TIMEOUT:-4}"
|
||||
EDGE_TIMEOUT="${HAPROXY_MGR_HEALTH_EDGE_TIMEOUT:-5}"
|
||||
|
||||
if ! curl -sf -o /dev/null --max-time "$API_TIMEOUT" http://127.0.0.1:8000/health; then
|
||||
echo "management API :8000 /health failed or took longer than ${API_TIMEOUT}s"
|
||||
exit 1
|
||||
fi
|
||||
if ! curl -s -o /dev/null --max-time "$EDGE_TIMEOUT" http://127.0.0.1/; then
|
||||
echo "HAProxy :80 did not answer within ${EDGE_TIMEOUT}s"
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
@@ -65,22 +65,13 @@ fi
|
||||
# Copy certificates for each domain
|
||||
UPDATED=0
|
||||
FAILED=0
|
||||
SKIPPED_EXPIRED=0
|
||||
|
||||
while read -r domain; do
|
||||
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
|
||||
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
|
||||
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
|
||||
|
||||
if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
|
||||
# Never publish an expired lineage. It cannot help any client and,
|
||||
# because HAProxy serves the first file claiming an SNI name, it can
|
||||
# shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
|
||||
# Counted separately from FAILED: a dead lineage that keeps expiring is
|
||||
# reported on every run (ERROR log) without making every run "fail".
|
||||
log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
|
||||
SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
|
||||
elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
|
||||
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
|
||||
# Assemble in a staging dir and rename into place. NEVER redirect into
|
||||
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and
|
||||
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
|
||||
@@ -98,15 +89,10 @@ while read -r domain; do
|
||||
fi
|
||||
done <<< "$DOMAINS"
|
||||
|
||||
# Already-published pems that have expired since are moved out too (they
|
||||
# can only shadow; see cert_retire_expired_pems).
|
||||
cert_retire_expired_pems
|
||||
RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
|
||||
|
||||
log_info "Certificate update completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
|
||||
log_info "Certificate update completed: $UPDATED updated, $FAILED failed"
|
||||
|
||||
# Reload HAProxy if any certificates were updated
|
||||
if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
|
||||
if [ $UPDATED -gt 0 ]; then
|
||||
# Never reload onto unvalidated material: a reload that fails to load the
|
||||
# certs directory drops HTTPS for every site on this host.
|
||||
if ! haproxy_config_ok; then
|
||||
|
||||
+3
-18
@@ -77,7 +77,7 @@ HAPROXY_SUPERVISOR_INTERVAL="${HAPROXY_SUPERVISOR_INTERVAL:-15}"
|
||||
) &
|
||||
|
||||
# Phase 2: WSGI servers
|
||||
# Tunable via env: HAPROXY_MGR_API_WORKERS (default 2), HAPROXY_MGR_API_TIMEOUT
|
||||
# Tunable via env: HAPROXY_MGR_API_WORKERS (default 1), HAPROXY_MGR_API_TIMEOUT
|
||||
# (default 120 — API can do slow ACME calls), HAPROXY_MGR_MAX_REQUESTS (default
|
||||
# 1000 — worker recycle frequency).
|
||||
#
|
||||
@@ -91,27 +91,12 @@ API_TIMEOUT="${HAPROXY_MGR_API_TIMEOUT:-120}"
|
||||
MAX_REQ="${HAPROXY_MGR_MAX_REQUESTS:-1000}"
|
||||
MAX_REQ_JITTER="${HAPROXY_MGR_MAX_REQUESTS_JITTER:-100}"
|
||||
|
||||
# Worker class (incident 2026-10-09): stock gthread makes every client socket
|
||||
# fully blocking with NO timeout, so any peer that sends a few bytes and goes
|
||||
# quiet (TLS hello on this plain-HTTP port, scanners -- :8000 is published on
|
||||
# 0.0.0.0) pins a pool thread forever; 4 of them wedge a worker. Stock gthread
|
||||
# also never reaps connections that send nothing (2x1000 of them wedge it
|
||||
# permanently) and has no total header-read bound (slowloris). This subclass
|
||||
# adds: a per-connection socket timeout (HAPROXY_MGR_CLIENT_TIMEOUT, 20s), a
|
||||
# first-byte deadline (HAPROXY_MGR_FIRST_BYTE_TIMEOUT, 10s), a total
|
||||
# request-line+headers deadline (HAPROXY_MGR_HEADER_DEADLINE, 10s), and a
|
||||
# watchdog that exits a worker whose threads have ALL been stuck for
|
||||
# HAPROXY_MGR_WEDGE_SECONDS (3600) so the master respawns it.
|
||||
# See haproxy_gunicorn_worker.py. --timeout below does NOT cover any of these:
|
||||
# for gthread it only watches the main thread's heartbeat.
|
||||
WORKER_CLASS="haproxy_gunicorn_worker.TimeoutThreadWorker"
|
||||
|
||||
# Default page server on :8080. Stays in the background.
|
||||
# --threads 4 lets one worker handle bursts of blocked-IP/default-page hits
|
||||
# without forking. --max-requests recycles the worker to bound memory drift.
|
||||
gunicorn \
|
||||
--bind 0.0.0.0:8080 \
|
||||
--workers 1 --threads 4 --worker-class "${WORKER_CLASS}" \
|
||||
--workers 1 --threads 4 --worker-class gthread \
|
||||
--max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
|
||||
--timeout 30 \
|
||||
--access-logfile - --error-logfile - --log-level info \
|
||||
@@ -123,7 +108,7 @@ gunicorn \
|
||||
# up). Longer --timeout because cert issuance hits ACME and can take a while.
|
||||
exec gunicorn \
|
||||
--bind 0.0.0.0:8000 \
|
||||
--workers "${API_WORKERS}" --threads 4 --worker-class "${WORKER_CLASS}" \
|
||||
--workers "${API_WORKERS}" --threads 4 --worker-class gthread \
|
||||
--max-requests "${MAX_REQ}" --max-requests-jitter "${MAX_REQ_JITTER}" \
|
||||
--timeout "${API_TIMEOUT}" \
|
||||
--access-logfile - --error-logfile - --log-level info \
|
||||
|
||||
@@ -47,22 +47,13 @@ fi
|
||||
# Copy certificates for each domain
|
||||
UPDATED=0
|
||||
FAILED=0
|
||||
SKIPPED_EXPIRED=0
|
||||
|
||||
while read -r domain; do
|
||||
CERT_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/fullchain.pem"
|
||||
KEY_FILE="${LETSENCRYPT_LIVE_DIR}/${domain}/privkey.pem"
|
||||
COMBINED_FILE="${SSL_CERTS_DIR}/${domain}.pem"
|
||||
|
||||
if [ -f "$CERT_FILE" ] && cert_is_expired "$CERT_FILE"; then
|
||||
# Never publish an expired lineage. It cannot help any client and,
|
||||
# because HAProxy serves the first file claiming an SNI name, it can
|
||||
# shadow a valid bundle for a name it shares (brainjar.net, 2026-10-07).
|
||||
# Counted separately from FAILED: a dead lineage that keeps expiring is
|
||||
# reported on every run (ERROR log) without making every run "fail".
|
||||
log_error "Skipping EXPIRED certificate lineage $domain (notAfter $(cert_not_after "$CERT_FILE")) - not publishing it"
|
||||
SKIPPED_EXPIRED=$((SKIPPED_EXPIRED + 1))
|
||||
elif [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
|
||||
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
|
||||
# Assemble in a staging dir and rename into place. NEVER redirect into
|
||||
# $COMBINED_FILE: the shell truncates the live pem before cat runs, and
|
||||
# HAProxy loads $SSL_CERTS_DIR as a directory, so one bad file there
|
||||
@@ -80,15 +71,10 @@ while read -r domain; do
|
||||
fi
|
||||
done <<< "$DOMAINS"
|
||||
|
||||
# Already-published pems that have expired since are moved out too (they
|
||||
# can only shadow; see cert_retire_expired_pems).
|
||||
cert_retire_expired_pems
|
||||
RETIRED_EXPIRED=$CERT_RETIRED_EXPIRED
|
||||
|
||||
log_info "Certificate sync completed: $UPDATED updated, $FAILED failed, $SKIPPED_EXPIRED expired lineage(s) skipped, $RETIRED_EXPIRED expired pem(s) retired"
|
||||
log_info "Certificate sync completed: $UPDATED updated, $FAILED failed"
|
||||
|
||||
# Reload HAProxy if any certificates were updated
|
||||
if [ $UPDATED -gt 0 ] || [ "$RETIRED_EXPIRED" -gt 0 ]; then
|
||||
if [ $UPDATED -gt 0 ]; then
|
||||
# Never reload onto unvalidated material: a reload that fails to load the
|
||||
# certs directory drops HTTPS for every site on this host.
|
||||
if ! haproxy_config_ok; then
|
||||
|
||||
@@ -73,15 +73,6 @@ LIB = os.path.join(SCRIPTS_DIR, 'cert-publish-lib.sh')
|
||||
BROKEN_TOKEN = '__BROKEN__'
|
||||
DOMAIN = 'test.example.com'
|
||||
|
||||
# Real SAN certificates (see scripts/test-fixtures/san-certs/). expired_old is
|
||||
# the brain-jar.com shape: a multi-SAN certificate that expired in 2020.
|
||||
SAN_FIXTURES = os.path.join(SCRIPTS_DIR, 'test-fixtures', 'san-certs')
|
||||
|
||||
|
||||
def san_fixture(name):
|
||||
with open(os.path.join(SAN_FIXTURES, name)) as fh:
|
||||
return fh.read()
|
||||
|
||||
# --- test key material -------------------------------------------------------
|
||||
# openssl req -x509 -newkey rsa:2048 -keyout key1 -out cert1 -days 3650 -nodes \
|
||||
# -subj /CN=test.example.com
|
||||
@@ -627,72 +618,6 @@ class CertScriptBehaviour:
|
||||
self.assertFalse(self.reload_attempted())
|
||||
|
||||
|
||||
# -- never publish an expired certificate (brainjar.net, 2026-10-07) ----
|
||||
def test_expired_lineage_is_skipped_and_reported(self):
|
||||
"""renew-certificates.sh used to re-publish EVERY live/ lineage every 12h,
|
||||
expired or not; the expired brain-jar.com bundle then shadowed the valid
|
||||
brainjar.net bundle for the names they shared."""
|
||||
dead = os.path.join(self.le_live, 'brain-jar.com')
|
||||
os.makedirs(dead)
|
||||
write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
|
||||
write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
|
||||
|
||||
result = self.run_it()
|
||||
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
self.assertFalse(
|
||||
os.path.exists(os.path.join(self.certs_dir, 'brain-jar.com.pem')),
|
||||
'an expired lineage was published into the crt directory')
|
||||
# The healthy lineage next to it is still published.
|
||||
self.assertEqual(read(self.live_pem), NEW_BUNDLE)
|
||||
self.assertIn('Skipping EXPIRED certificate lineage brain-jar.com', self.logs())
|
||||
self.assertIn('1 updated, 0 failed, 1 expired lineage(s) skipped', self.logs())
|
||||
self.assertIn('brain-jar.com', read(self.error_log),
|
||||
'the skip must land in the monitored error log')
|
||||
|
||||
def test_expired_lineage_does_not_replace_an_existing_pem(self):
|
||||
dead = os.path.join(self.le_live, 'brain-jar.com')
|
||||
os.makedirs(dead)
|
||||
write(os.path.join(dead, 'fullchain.pem'), san_fixture('expired_old.crt'))
|
||||
write(os.path.join(dead, 'privkey.pem'), san_fixture('expired_old.key'))
|
||||
existing = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'), OTHER_BUNDLE)
|
||||
|
||||
self.run_it()
|
||||
|
||||
self.assertEqual(read(existing), OTHER_BUNDLE,
|
||||
'an expired lineage overwrote the file in the crt dir')
|
||||
|
||||
def test_already_published_expired_pem_is_retired(self):
|
||||
"""Review M2: skipping publication is not enough - an expired pem that
|
||||
is ALREADY in the crt dir (brain-jar.com.pem) keeps shadowing."""
|
||||
expired = write(os.path.join(self.certs_dir, 'brain-jar.com.pem'),
|
||||
san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
|
||||
result = self.run_it()
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
self.assertFalse(os.path.exists(expired), 'expired pem still in the crt dir')
|
||||
retired = os.path.join(self.haproxy_dir, 'cert-retired', 'brain-jar.com.pem')
|
||||
self.assertTrue(os.path.exists(retired), 'expired pem was not moved to cert-retired/ (never delete)')
|
||||
self.assertIn('Retired EXPIRED certificate brain-jar.com.pem', self.logs())
|
||||
self.assertIn('1 expired pem(s) retired', self.logs())
|
||||
self.assertTrue(self.reload_attempted())
|
||||
self.assertEqual(read(self.live_pem), NEW_BUNDLE, 'valid neighbours untouched')
|
||||
self.assert_certs_dir_is_clean()
|
||||
|
||||
def test_retiring_an_expired_pem_alone_triggers_a_reload(self):
|
||||
self.seed_previous_bundle(NEW_BUNDLE)
|
||||
shutil.rmtree(self.domain_dir)
|
||||
other = os.path.join(self.le_live, 'other.example')
|
||||
os.makedirs(other)
|
||||
write(os.path.join(other, 'fullchain.pem'), san_fixture('expired_old.crt'))
|
||||
write(os.path.join(other, 'privkey.pem'), san_fixture('expired_old.key'))
|
||||
write(os.path.join(self.certs_dir, 'other.example.pem'),
|
||||
san_fixture('expired_old.crt') + san_fixture('expired_old.key'))
|
||||
result = self.run_it()
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
self.assertIn('0 updated', self.logs())
|
||||
self.assertTrue(self.reload_attempted(), 'HAProxy must reload to drop the retired pem')
|
||||
|
||||
|
||||
class TestRenewCertificates(CertScriptBehaviour, CertScriptFixture):
|
||||
SCRIPT = 'renew-certificates.sh'
|
||||
|
||||
@@ -713,52 +638,6 @@ class TestCertPublishLibrary(CertScriptFixture):
|
||||
path = write(os.path.join(self.tmp, 'ok.pem'), NEW_BUNDLE)
|
||||
self.assertEqual(self.call('cert_bundle_valid "$1"', path).returncode, 0)
|
||||
|
||||
def test_cert_is_expired_reports_both_directions(self):
|
||||
expired = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
|
||||
valid = write(os.path.join(self.tmp, 'valid.crt'), san_fixture('new_bundle.crt'))
|
||||
self.assertEqual(0, self.call('cert_is_expired "$1"', expired).returncode,
|
||||
'an expired certificate was not recognised as expired')
|
||||
self.assertEqual(1, self.call('cert_is_expired "$1"', valid).returncode,
|
||||
'a valid certificate was reported as expired')
|
||||
self.assertEqual(1, self.call('cert_is_expired "$1"',
|
||||
os.path.join(self.tmp, 'missing')).returncode)
|
||||
|
||||
def test_expired_sweep_keeps_the_last_pem_and_the_default_cert(self):
|
||||
exp = san_fixture('expired_old.crt') + san_fixture('expired_old.key')
|
||||
only = write(os.path.join(self.certs_dir, 'only.pem'), exp)
|
||||
r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
|
||||
self.assertIn('n=0', r.stdout)
|
||||
self.assertTrue(os.path.exists(only), 'emptied the crt dir: that fails the whole :443 bind')
|
||||
dflt = write(os.path.join(self.certs_dir, 'default_self_signed_cert.pem'), exp)
|
||||
r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
|
||||
self.assertIn('n=1', r.stdout)
|
||||
self.assertTrue(os.path.exists(dflt), 'the default self-signed cert is not ours to retire')
|
||||
self.assertFalse(os.path.exists(only))
|
||||
|
||||
def test_expired_sweep_never_touches_a_valid_pem(self):
|
||||
valid = write(os.path.join(self.certs_dir, 'a.pem'), NEW_BUNDLE)
|
||||
write(os.path.join(self.certs_dir, 'b.pem'), OTHER_BUNDLE)
|
||||
r = self.call('cert_retire_expired_pems; echo "n=$CERT_RETIRED_EXPIRED"')
|
||||
self.assertIn('n=0', r.stdout)
|
||||
self.assertEqual(read(valid), NEW_BUNDLE)
|
||||
|
||||
def test_cert_publish_refuses_an_expired_certificate(self):
|
||||
cert = write(os.path.join(self.tmp, 'expired.crt'), san_fixture('expired_old.crt'))
|
||||
key = write(os.path.join(self.tmp, 'expired.key'), san_fixture('expired_old.key'))
|
||||
dest = os.path.join(self.certs_dir, 'brain-jar.com.pem')
|
||||
|
||||
result = self.call('cert_publish "$1" "$2" "$3"', cert, key, dest)
|
||||
|
||||
self.assert_rejected(result, because='expired')
|
||||
self.assertFalse(os.path.exists(dest), 'expired certificate was published')
|
||||
# And the same material with a valid certificate IS publishable, so the
|
||||
# refusal above is about expiry, not about the fixture.
|
||||
good_cert = write(os.path.join(self.tmp, 'good.crt'), san_fixture('new_bundle.crt'))
|
||||
good_key = write(os.path.join(self.tmp, 'good.key'), san_fixture('new_bundle.key'))
|
||||
ok = self.call('cert_publish "$1" "$2" "$3"', good_cert, good_key, dest)
|
||||
self.assertEqual(0, ok.returncode, ok.stdout + ok.stderr)
|
||||
self.assertTrue(os.path.exists(dest))
|
||||
|
||||
def test_empty_and_missing_bundles_rejected(self):
|
||||
empty = write(os.path.join(self.tmp, 'empty.pem'), '')
|
||||
self.assert_rejected(self.call('cert_bundle_valid "$1"', empty),
|
||||
|
||||
@@ -27,8 +27,6 @@ These tests pin the invariants:
|
||||
* nothing is published that is not a complete, validated cert+key pair;
|
||||
* no old certificate file is removed and no lineage deleted until the
|
||||
replacement is validated, in place, and actually loaded by HAProxy;
|
||||
* removing ONE domain never unlinks a .pem, or deletes a certbot lineage,
|
||||
that other still-configured domains are being served from;
|
||||
* only final .pem files ever exist in the crt directory.
|
||||
|
||||
Running
|
||||
@@ -57,7 +55,6 @@ import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import time
|
||||
import unittest
|
||||
|
||||
BROKEN_TOKEN = '__BROKEN__'
|
||||
@@ -255,9 +252,6 @@ FAKE_HAPROXY = textwrap.dedent(f"""\
|
||||
|
||||
# certbot stub: succeeds, announces a renewal, and records every invocation so
|
||||
# tests can assert that `certbot delete` did or did not run.
|
||||
# TEST_CERTBOT_INSTALL_SRC/_DIR: on certonly, copy fullchain.pem + privkey.pem
|
||||
# from SRC into DIR (a real issuance changes the lineage's certificate).
|
||||
# TEST_CERTBOT_NOT_DUE: on certonly, print certbot's "not yet due" wording.
|
||||
FAKE_CERTBOT = textwrap.dedent("""\
|
||||
#!/bin/sh
|
||||
if [ -n "$TEST_CERTBOT_LOG" ]; then
|
||||
@@ -266,14 +260,6 @@ FAKE_CERTBOT = textwrap.dedent("""\
|
||||
case "$1" in
|
||||
renew) echo "Congratulations, all renewals succeeded" ;;
|
||||
delete) [ -n "$TEST_CERTBOT_DELETE_FAILS" ] && exit 1 ;;
|
||||
certonly)
|
||||
if [ -n "$TEST_CERTBOT_INSTALL_SRC" ]; then
|
||||
mkdir -p "$TEST_CERTBOT_INSTALL_DIR"
|
||||
cp "$TEST_CERTBOT_INSTALL_SRC/fullchain.pem" "$TEST_CERTBOT_INSTALL_SRC/privkey.pem" "$TEST_CERTBOT_INSTALL_DIR/"
|
||||
fi
|
||||
if [ -n "$TEST_CERTBOT_NOT_DUE" ]; then
|
||||
echo "Certificate not yet due for renewal; no action taken."
|
||||
fi ;;
|
||||
esac
|
||||
exit 0
|
||||
""")
|
||||
@@ -343,7 +329,6 @@ class CertPublishTestCase(unittest.TestCase):
|
||||
'CORAZA_SPOE_BACKUP_PATH': os.path.join(self.etc, 'coraza-spoe.cfg.backup'),
|
||||
'CLUSTER_SECRET_PATH': os.path.join(self.etc, 'cluster-secret'),
|
||||
'SSL_CERTS_DIR': self.certs,
|
||||
'LETSENCRYPT_DIR': os.path.join(self.tmp, 'letsencrypt'),
|
||||
'HAPROXY_SOCKET_PATH': os.path.join(self.etc, 'haproxy.sock'),
|
||||
'API_KEY': None,
|
||||
}
|
||||
@@ -587,7 +572,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
|
||||
|
||||
@FIX_ONLY
|
||||
@NEEDS_OPENSSL
|
||||
def test_lineage_is_untouched_when_haproxy_does_not_load_the_bundle(self):
|
||||
def test_lineage_is_deleted_only_after_haproxy_loads_the_bundle(self):
|
||||
"""A reload failure must leave the lineage intact and the file recoverable."""
|
||||
old_path = self._setup_supersede(broken=False)
|
||||
# Make generate_config() produce a config the validator rejects, so the
|
||||
@@ -604,7 +589,7 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
|
||||
[], self.certbot_deletes(),
|
||||
'the lineage must not be deleted when HAProxy did not reload')
|
||||
self.assertFalse(os.path.exists(old_path))
|
||||
quarantined = os.path.join(hm.cert_retired_dir(),
|
||||
quarantined = os.path.join(hm.cert_backup_dir(),
|
||||
os.path.basename(old_path))
|
||||
self.assertTrue(os.path.exists(quarantined),
|
||||
'the superseded file must be recoverable by hand')
|
||||
@@ -612,9 +597,8 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
|
||||
|
||||
@FIX_ONLY
|
||||
@NEEDS_OPENSSL
|
||||
def test_successful_bundle_still_supersedes_without_deleting(self):
|
||||
"""The cleanup must still do its job on the happy path - by moving, never
|
||||
by `certbot delete` (irreversible; recovery is a rate-limited ACME order)."""
|
||||
def test_successful_bundle_still_supersedes_and_deletes(self):
|
||||
"""The cleanup must still do its job on the happy path."""
|
||||
old_path = self._setup_supersede(broken=False)
|
||||
|
||||
resp = self.client.post('/api/ssl/bundle', json={
|
||||
@@ -626,381 +610,12 @@ class TestOldCertificateIsNotDestroyedFirst(CertPublishTestCase):
|
||||
self.assertFalse(os.path.exists(old_path),
|
||||
'the superseded file must leave the crt directory or '
|
||||
'it keeps shadowing the new bundle')
|
||||
self.assertTrue(os.path.exists(os.path.join(
|
||||
hm.cert_retired_dir(), os.path.basename(old_path))),
|
||||
'the superseded file must be moved aside, not destroyed')
|
||||
self.assertEqual([], self.certbot_deletes(),
|
||||
'retirement must never `certbot delete` anything')
|
||||
self.assertEqual(['delete --cert-name www.test.example.com -n'],
|
||||
self.certbot_deletes())
|
||||
self.assertTrue(self.edge_would_start())
|
||||
self.assert_only_final_pems_in_certs_dir()
|
||||
|
||||
|
||||
|
||||
# --- SAN-overlap retirement (brainjar.net, 2026-10-07) -----------------------
|
||||
SAN_FIXTURES = os.path.join(MODULE_DIR, 'scripts', 'test-fixtures', 'san-certs')
|
||||
|
||||
|
||||
def san(name):
|
||||
with open(os.path.join(SAN_FIXTURES, name)) as fh:
|
||||
return fh.read()
|
||||
|
||||
|
||||
def make_certbot_lineage(le_dir, lineage, cert, key):
|
||||
"""archive/<x>/{cert,chain,fullchain,privkey}1.pem + live/<x>/*.pem
|
||||
symlinks (../../archive/<x>/...1.pem) + renewal/<x>.conf, like certbot."""
|
||||
archive = os.path.join(le_dir, 'archive', lineage)
|
||||
live = os.path.join(le_dir, 'live', lineage)
|
||||
os.makedirs(archive, exist_ok=True)
|
||||
os.makedirs(live, exist_ok=True)
|
||||
for kind, body in (('cert', cert), ('chain', cert), ('fullchain', cert), ('privkey', key)):
|
||||
with open(os.path.join(archive, f'{kind}1.pem'), 'w') as fh:
|
||||
fh.write(body)
|
||||
link = os.path.join(live, f'{kind}.pem')
|
||||
if os.path.lexists(link):
|
||||
os.remove(link)
|
||||
os.symlink(f'../../archive/{lineage}/{kind}1.pem', link)
|
||||
os.makedirs(os.path.join(le_dir, 'renewal'), exist_ok=True)
|
||||
with open(os.path.join(le_dir, 'renewal', f'{lineage}.conf'), 'w') as fh:
|
||||
fh.write(f'archive_dir = {archive}\ncert = {live}/cert.pem\n')
|
||||
return live
|
||||
|
||||
|
||||
def retired_path(le_dir, part, name):
|
||||
hits = sorted(__import__('glob').glob(os.path.join(le_dir, 'retired', '*', part, name)))
|
||||
return hits[-1] if hits else os.path.join(le_dir, 'retired', '<none>', part, name)
|
||||
|
||||
|
||||
FIX_ONLY_RETIRE = unittest.skipIf(
|
||||
TESTING_FOREIGN_TREE and not hasattr(hm, 'retirement_blockers'),
|
||||
'HAPROXY_MANAGER_DIR points at a tree without SAN-overlap retirement')
|
||||
|
||||
|
||||
class TestRetirementRules(unittest.TestCase):
|
||||
"""retirement_blockers() is the shared-PEM safety rule, tested directly."""
|
||||
|
||||
NOW = 2_000_000_000
|
||||
|
||||
def cert(self, names, days_left, fp):
|
||||
return {'names': set(names), 'fingerprint': fp,
|
||||
'not_after': self.NOW + days_left * 86400, 'cn': sorted(names)[0]}
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_expired_candidate_never_blocks(self):
|
||||
old = self.cert(['brain-jar.com', 'brainjar.net'], -2, 'A')
|
||||
self.assertEqual([], hm.retirement_blockers(
|
||||
old, [], {'brainjar.net'}, {'brain-jar.com', 'brainjar.net'}, [], now=self.NOW))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_valid_candidate_that_is_last_cert_for_a_served_name_is_kept(self):
|
||||
old = self.cert(['shared.example', 'brainjar.net', 'other.example'], 40, 'A')
|
||||
self.assertEqual(['other.example'], hm.retirement_blockers(
|
||||
old, [], {'brainjar.net'}, {'brainjar.net', 'other.example'}, [], now=self.NOW))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_unreachable_names_do_not_block(self):
|
||||
old = self.cert(['dead.example', 'brainjar.net'], 40, 'A')
|
||||
self.assertEqual([], hm.retirement_blockers(
|
||||
old, [], {'brainjar.net'}, {'brainjar.net', 'dead.example'},
|
||||
['dead.example'], now=self.NOW))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_name_covered_by_another_valid_cert_does_not_block(self):
|
||||
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
|
||||
other = self.cert(['other.example'], 60, 'B')
|
||||
self.assertEqual([], hm.retirement_blockers(
|
||||
old, [other], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_cover_by_an_expired_or_identical_cert_does_not_count(self):
|
||||
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
|
||||
expired = self.cert(['other.example'], -1, 'B')
|
||||
same = self.cert(['other.example'], 60, 'A')
|
||||
self.assertEqual(['other.example'], hm.retirement_blockers(
|
||||
old, [expired, same], {'brainjar.net'}, {'other.example'}, [], now=self.NOW))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_unknown_served_set_keeps_every_valid_name(self):
|
||||
old = self.cert(['brainjar.net', 'other.example'], 40, 'A')
|
||||
self.assertEqual(['other.example'], hm.retirement_blockers(
|
||||
old, [], {'brainjar.net'}, None, [], now=self.NOW))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_wildcard_coverage(self):
|
||||
self.assertTrue(hm.cert_name_covers('*.example.com', 'a.example.com'))
|
||||
self.assertFalse(hm.cert_name_covers('*.example.com', 'a.b.example.com'))
|
||||
self.assertFalse(hm.cert_name_covers('*.example.com', 'example.com'))
|
||||
old = self.cert(['*.example.com', 'brainjar.net'], 40, 'A')
|
||||
self.assertEqual(['shop.example.com'], hm.retirement_blockers(
|
||||
old, [], {'brainjar.net'}, {'shop.example.com'}, [], now=self.NOW))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_read_cert_info_reads_sans_and_expiry(self):
|
||||
tmp = tempfile.mkdtemp(prefix='cert-info-')
|
||||
self.addCleanup(shutil.rmtree, tmp, True)
|
||||
path = os.path.join(tmp, 'c.pem')
|
||||
with open(path, 'w') as fh:
|
||||
fh.write(san('expired_old.crt'))
|
||||
info = hm.read_cert_info(path)
|
||||
self.assertEqual({'brain-jar.com', 'www.brain-jar.com', 'brainjar.net',
|
||||
'www.brainjar.net'}, info['names'])
|
||||
self.assertEqual('brain-jar.com', info['cn'])
|
||||
self.assertEqual(1585699200, info['not_after']) # 2020-04-01T00:00:00Z
|
||||
self.assertTrue(info['fingerprint'])
|
||||
|
||||
|
||||
class TestSanOverlapRetirement(CertPublishTestCase):
|
||||
"""The new bundle must not stay shadowed by an older cert sharing a SAN."""
|
||||
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.le = os.path.join(self.tmp, 'letsencrypt')
|
||||
self.new_src = os.path.join(self.tmp, 'issued', 'brainjar.net')
|
||||
self.write(os.path.join(self.new_src, 'fullchain.pem'), san('new_bundle.crt'))
|
||||
self.write(os.path.join(self.new_src, 'privkey.pem'), san('new_bundle.key'))
|
||||
# certbot "issues" brainjar.net into live/ when called.
|
||||
os.environ['TEST_CERTBOT_INSTALL_SRC'] = self.new_src
|
||||
os.environ['TEST_CERTBOT_INSTALL_DIR'] = os.path.join(self.le_live, 'brainjar.net')
|
||||
for name in ('brainjar.net', 'www.brainjar.net'):
|
||||
self.add_domain(name, name.replace('.', '_'))
|
||||
|
||||
def old_lineage(self, lineage, fixture, publish=True):
|
||||
"""A lineage laid out the way certbot does it: real files in
|
||||
archive/<x>/*1.pem, relative symlinks in live/<x>/, a renewal conf."""
|
||||
live = make_certbot_lineage(self.le, lineage, san(f'{fixture}.crt'),
|
||||
san(f'{fixture}.key'))
|
||||
conf = os.path.join(self.le, 'renewal', f'{lineage}.conf')
|
||||
if not publish:
|
||||
return live, conf, None
|
||||
pem = self.write(os.path.join(self.certs, f'{lineage}.pem'),
|
||||
san(f'{fixture}.crt') + san(f'{fixture}.key'))
|
||||
return live, conf, pem
|
||||
|
||||
def bundle(self, **extra):
|
||||
body = {'primary': 'brainjar.net', 'sans': ['www.brainjar.net']}
|
||||
body.update(extra)
|
||||
resp = self.client.post('/api/ssl/bundle', json=body)
|
||||
self.assertEqual(200, resp.status_code, resp.get_data(as_text=True))
|
||||
return resp.get_json()
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_expired_overlapping_cert_with_a_different_cn_is_retired(self):
|
||||
"""The incident: CN brain-jar.com, SANs include brainjar.net."""
|
||||
self.add_domain('brain-jar.com', 'bjc')
|
||||
self.add_domain('www.brain-jar.com', 'wbjc')
|
||||
live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
|
||||
|
||||
data = self.bundle(unreachable_names=['brain-jar.com', 'www.brain-jar.com'])
|
||||
|
||||
self.assertFalse(os.path.exists(pem),
|
||||
'the expired SAN-overlapping pem is still shadowing the bundle')
|
||||
retired = [e['file'] for e in data['cleanup']['removed']]
|
||||
self.assertEqual(['brain-jar.com.pem'], retired)
|
||||
self.assertTrue(os.path.exists(os.path.join(hm.cert_retired_dir(), 'brain-jar.com.pem')))
|
||||
# Lineage retired, not deleted: renew-certificates.sh can no longer
|
||||
# re-publish it and certbot can no longer try to renew it.
|
||||
self.assertFalse(os.path.exists(live))
|
||||
self.assertFalse(os.path.exists(conf))
|
||||
# live, archive AND renewal leave together (review H2), into one tree.
|
||||
self.assertFalse(os.path.exists(os.path.join(self.le, 'archive', 'brain-jar.com')))
|
||||
self.assertTrue(os.path.isdir(retired_path(self.le, 'live', 'brain-jar.com')))
|
||||
self.assertTrue(os.path.isdir(retired_path(self.le, 'archive', 'brain-jar.com')))
|
||||
self.assertTrue(os.path.exists(retired_path(self.le, 'renewal', 'brain-jar.com.conf')))
|
||||
# Relative symlinks still resolve in the retired tree (restore = mv back).
|
||||
self.assertIn('BEGIN CERTIFICATE', self.read(
|
||||
os.path.join(retired_path(self.le, 'live', 'brain-jar.com'), 'cert.pem')))
|
||||
self.assertEqual([], self.certbot_deletes())
|
||||
self.assertTrue(os.path.exists(os.path.join(self.certs, 'brainjar.net.pem')))
|
||||
self.assertTrue(self.edge_would_start())
|
||||
self.assert_only_final_pems_in_certs_dir()
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_expired_overlap_is_retired_even_without_the_unreachable_hint(self):
|
||||
self.add_domain('brain-jar.com', 'bjc')
|
||||
live, conf, pem = self.old_lineage('brain-jar.com', 'expired_old')
|
||||
self.bundle()
|
||||
self.assertFalse(os.path.exists(pem))
|
||||
self.assertFalse(os.path.exists(live))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_shared_pem_still_serving_another_site_is_kept(self):
|
||||
"""The shared-PEM hazard: never retire the only valid cert for a served name."""
|
||||
self.add_domain('other-site.example', 'other_site')
|
||||
live, conf, pem = self.old_lineage('shared-old.example', 'shared_old')
|
||||
|
||||
data = self.bundle()
|
||||
|
||||
self.assertTrue(os.path.exists(pem),
|
||||
'retired the only valid certificate for other-site.example')
|
||||
self.assertTrue(os.path.exists(live))
|
||||
self.assertTrue(os.path.exists(conf))
|
||||
kept = {e['file']: e['blockers'] for e in data['cleanup']['kept']}
|
||||
self.assertEqual({'shared-old.example.pem': ['other-site.example']}, kept)
|
||||
self.assertEqual(['other-site.example'],
|
||||
data['cleanup']['lineages']['kept'][0]['blockers'])
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_valid_overlap_whose_other_names_are_unreachable_is_retired(self):
|
||||
self.add_domain('dead.example', 'dead')
|
||||
live, conf, pem = self.old_lineage('dead.example', 'dead_old')
|
||||
|
||||
self.bundle(unreachable_names=['dead.example'])
|
||||
|
||||
self.assertFalse(os.path.exists(pem))
|
||||
self.assertFalse(os.path.exists(live))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_valid_overlap_still_served_without_the_hint_is_kept(self):
|
||||
"""Same files as above, but nobody said dead.example is gone: keep it."""
|
||||
self.add_domain('dead.example', 'dead')
|
||||
live, conf, pem = self.old_lineage('dead.example', 'dead_old')
|
||||
|
||||
self.bundle()
|
||||
|
||||
self.assertTrue(os.path.exists(pem))
|
||||
self.assertTrue(os.path.exists(live))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_non_overlapping_certificates_are_untouched(self):
|
||||
live, conf, pem = self.old_lineage('unrelated.example', 'unrelated')
|
||||
self.bundle()
|
||||
self.assertTrue(os.path.exists(pem), 'retired a cert sharing no name with the bundle')
|
||||
self.assertTrue(os.path.exists(live))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_response_distinguishes_issuance_from_not_yet_due(self):
|
||||
issued = self.bundle()
|
||||
self.assertTrue(issued['renewed'], 'a fresh issuance was not reported as one')
|
||||
self.assertEqual(4922899200, issued['not_after']) # 2126-01-01T00:00:00Z
|
||||
|
||||
# Lineage now exists with the same names; certbot keeps it.
|
||||
del os.environ['TEST_CERTBOT_INSTALL_SRC']
|
||||
os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
|
||||
noop = self.bundle()
|
||||
self.assertFalse(noop['renewed'], 'a certbot no-op was reported as a renewal')
|
||||
self.assertTrue(noop['certbot_not_due'])
|
||||
self.assertEqual(4922899200, noop['not_after'])
|
||||
self.assertIn('--keep-until-expiring', self.certbot_invocations()[-1])
|
||||
|
||||
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_not_due_noop_never_retires_its_own_cert_name_lineage(self):
|
||||
"""Review H1: find_certbot_live_dir() returns the NEWEST of brainjar.net
|
||||
and brainjar.net-0001. After a certbot no-op that was the stale -0001:
|
||||
it got published as brainjar.net.pem and the real brainjar.net lineage
|
||||
was retired as "superseded"."""
|
||||
del os.environ['TEST_CERTBOT_INSTALL_SRC']
|
||||
os.environ['TEST_CERTBOT_NOT_DUE'] = '1'
|
||||
real = make_certbot_lineage(self.le, 'brainjar.net', san('new_bundle.crt'), san('new_bundle.key'))
|
||||
stale = make_certbot_lineage(self.le, 'brainjar.net-0001', san('dead_old.crt'), san('dead_old.key'))
|
||||
later = time.time() + 60
|
||||
os.utime(os.path.join(self.le, 'archive', 'brainjar.net-0001', 'fullchain1.pem'), (later, later))
|
||||
# Production behaviour of find_certbot_live_dir(): newest wins.
|
||||
hm.find_certbot_live_dir = lambda base: stale if base == 'brainjar.net' else None
|
||||
|
||||
data = self.bundle()
|
||||
|
||||
self.assertTrue(os.path.isdir(real), 'the --cert-name lineage itself was retired')
|
||||
self.assertTrue(os.path.exists(os.path.join(self.le, 'renewal', 'brainjar.net.conf')))
|
||||
self.assertNotIn('brainjar.net', [e['lineage'] for e in data['cleanup']['lineages']['retired']])
|
||||
published = self.read(os.path.join(self.certs, 'brainjar.net.pem'))
|
||||
self.assertIn(san('new_bundle.crt').strip().splitlines()[1], published,
|
||||
'published the stale -0001 certificate instead of live/brainjar.net')
|
||||
self.assertFalse(data['renewed'])
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_bundle_reports_capabilities(self):
|
||||
data = self.bundle()
|
||||
self.assertIn('san_overlap_retirement', data.get('capabilities', []))
|
||||
caps = self.client.get('/api/capabilities').get_json()
|
||||
self.assertIn('unreachable_names', caps['capabilities'])
|
||||
|
||||
|
||||
class TestRetiredLineageCanBeReissued(CertPublishTestCase):
|
||||
"""Review H2: retirement must leave certbot able to issue the name again."""
|
||||
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.le = hm.LETSENCRYPT_DIR
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_retirement_removes_every_part_certbot_checks(self):
|
||||
make_certbot_lineage(self.le, 'brain-jar.com', san('expired_old.crt'), san('expired_old.key'))
|
||||
out = hm.retire_lineage_tree('brain-jar.com')
|
||||
# certbot new_lineage() refuses if any of these exist for the name.
|
||||
for part in (('live', 'brain-jar.com'), ('archive', 'brain-jar.com'),
|
||||
('renewal', 'brain-jar.com.conf')):
|
||||
self.assertFalse(os.path.lexists(os.path.join(self.le, *part)), part)
|
||||
self.assertTrue(os.path.lexists(os.path.join(out['retired_to'], *part)), part)
|
||||
self.assertIn('BEGIN CERTIFICATE',
|
||||
self.read(os.path.join(out['retired_to'], 'live', 'brain-jar.com', 'fullchain.pem')))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_second_retirement_of_the_same_name_does_not_collide(self):
|
||||
make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
|
||||
a = hm.retire_lineage_tree('x.example', stamp='S')
|
||||
make_certbot_lineage(self.le, 'x.example', san('new_bundle.crt'), san('new_bundle.key'))
|
||||
b = hm.retire_lineage_tree('x.example', stamp='S')
|
||||
self.assertNotEqual(a['retired_to'], b['retired_to'])
|
||||
self.assertTrue(os.path.isdir(os.path.join(a['retired_to'], 'archive', 'x.example')))
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
def test_certbot_can_issue_the_name_again_after_retirement(self):
|
||||
"""Against certbot's own storage code (skipped where certbot is not
|
||||
importable, e.g. the image's app python)."""
|
||||
# Import AND initialisation are both guarded: a half-installed or
|
||||
# version-skewed certbot can fail with far more than ImportError (a
|
||||
# missing transitive dependency, a pkg_resources/entry-point error, a
|
||||
# moved internal). Any of those means "cannot run this check here",
|
||||
# not "retirement is broken". new_lineage() itself is NOT inside the
|
||||
# guard - that is the assertion.
|
||||
try:
|
||||
import types
|
||||
from certbot._internal import storage
|
||||
real_rv = storage.relevant_values
|
||||
new_lineage = storage.RenewableCert.new_lineage
|
||||
cfg = types.SimpleNamespace(
|
||||
config_dir=self.le, renewal_configs_dir=os.path.join(self.le, 'renewal'),
|
||||
default_archive_dir=os.path.join(self.le, 'archive'),
|
||||
live_dir=os.path.join(self.le, 'live'))
|
||||
except Exception as e: # noqa: BLE001 - deliberately broad, see above
|
||||
self.skipTest(f'certbot unusable from this python ({type(e).__name__}: {e})')
|
||||
storage.relevant_values = lambda c: {'server': 'https://acme.invalid/directory'}
|
||||
self.addCleanup(setattr, storage, 'relevant_values', real_rv)
|
||||
cert, key = san('new_bundle.crt').encode(), san('new_bundle.key').encode()
|
||||
|
||||
first = new_lineage('brain-jar.com', cert, key, cert, cfg)
|
||||
self.assertEqual('brain-jar.com', first.lineagename)
|
||||
hm.retire_lineage_tree('brain-jar.com')
|
||||
again = new_lineage('brain-jar.com', cert, key, cert, cfg)
|
||||
self.assertEqual('brain-jar.com', again.lineagename,
|
||||
'certbot could not re-create the lineage under the same name')
|
||||
self.assertTrue(os.path.getsize(os.path.join(self.le, 'renewal', 'brain-jar.com.conf')) > 0)
|
||||
|
||||
class TestExpiredCertificateIsNeverPublished(CertPublishTestCase):
|
||||
|
||||
@FIX_ONLY_RETIRE
|
||||
@NEEDS_OPENSSL
|
||||
def test_publisher_refuses_an_expired_leaf(self):
|
||||
src = os.path.join(self.tmp, 'expired')
|
||||
cert = self.write(os.path.join(src, 'fullchain.pem'), san('expired_old.crt'))
|
||||
key = self.write(os.path.join(src, 'privkey.pem'), san('expired_old.key'))
|
||||
live = self.publish_live_bundle('brain-jar.com')
|
||||
with self.assertRaises(hm.CertificatePublishError) as ctx:
|
||||
hm.publish_pem_bundle(live, [cert, key])
|
||||
self.assertIn('expired', str(ctx.exception))
|
||||
self.assertEqual(PREVIOUS_BUNDLE, self.read(live))
|
||||
|
||||
|
||||
class TestBundleValidation(CertPublishTestCase):
|
||||
"""What may and may not be published."""
|
||||
|
||||
@@ -1279,184 +894,6 @@ class TestBundleValidation(CertPublishTestCase):
|
||||
'a private key must never be group/world writable')
|
||||
|
||||
|
||||
class TestSharedCertificateSurvivesDomainRemoval(CertPublishTestCase):
|
||||
"""Bug 5: DELETE /api/domain unlinked a PEM other live sites were served from.
|
||||
|
||||
`domains.domain` is UNIQUE; `domains.ssl_cert_path` is not, and nothing
|
||||
ever made it so. request_ssl_bundle() issues one SAN certificate and points
|
||||
every included name's row at the same /etc/haproxy/certs/<primary>.pem, so
|
||||
sharing is not an edge case - it is the normal shape of the table. Measured
|
||||
on production the day this was written: 39 of 71 distinct cert paths on one
|
||||
host were referenced by more than one domain row (118 of 150 SSL-enabled
|
||||
rows), and one shared path was
|
||||
/etc/haproxy/certs/threeworldsoneheart.org.pem, referenced by the live
|
||||
apex, its www, and a mail.* alias.
|
||||
|
||||
remove_domain() did, unconditionally:
|
||||
|
||||
os.remove(ssl_cert_path)
|
||||
certbot delete --cert-name <domain>
|
||||
|
||||
Removing the mail.* alias would therefore have deleted the PEM the apex was
|
||||
serving on, and (had the alias been the bundle primary) the lineage behind
|
||||
it - HTTPS down for every other name in the bundle, with no local copy and
|
||||
only a fresh, rate-limited ACME order to recover from.
|
||||
|
||||
These tests assert file-system and certbot-invocation outcomes, not which
|
||||
branch was taken.
|
||||
"""
|
||||
|
||||
BUNDLE = '/etc/haproxy/certs' # documentation only; SSL_CERTS_DIR is stubbed
|
||||
|
||||
def _cert_for(self, primary):
|
||||
"""Publish a bundle for `primary` and return its path."""
|
||||
return self.publish_live_bundle(primary)
|
||||
|
||||
def remove(self, domain):
|
||||
return self.client.delete('/api/domain', json={'domain': domain})
|
||||
|
||||
# -- last reference: the cleanup must still happen --------------------
|
||||
|
||||
def test_last_reference_removal_unlinks_the_pem(self):
|
||||
cert = self._cert_for('example.com')
|
||||
self.add_domain('example.com', 'be_example', ssl_cert_path=cert)
|
||||
|
||||
resp = self.remove('example.com')
|
||||
|
||||
self.assertEqual(200, resp.status_code, resp.data)
|
||||
self.assertFalse(
|
||||
os.path.exists(cert),
|
||||
'nothing else referenced this bundle - it must be cleaned up, or '
|
||||
'the crt directory accumulates certs for domains that are gone')
|
||||
|
||||
def test_last_reference_removal_deletes_the_lineage(self):
|
||||
cert = self._cert_for('example.com')
|
||||
self.add_domain('example.com', 'be_example', ssl_cert_path=cert)
|
||||
|
||||
self.remove('example.com')
|
||||
|
||||
self.assertEqual(
|
||||
['delete --cert-name example.com --non-interactive'],
|
||||
self.certbot_deletes(),
|
||||
'the last name on a lineage went away - the lineage should go too')
|
||||
|
||||
def test_lineage_deleted_is_the_bundles_not_the_domains_own_name(self):
|
||||
"""Removing a SAN member must target the lineage that issued the file."""
|
||||
cert = self._cert_for('example.com')
|
||||
self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
|
||||
|
||||
self.remove('www.example.com')
|
||||
|
||||
self.assertEqual(
|
||||
['delete --cert-name example.com --non-interactive'],
|
||||
self.certbot_deletes(),
|
||||
'the lineage is named after the bundle primary (--cert-name), not '
|
||||
'after whichever SAN happened to be removed last')
|
||||
|
||||
# -- shared reference: nothing may be destroyed -----------------------
|
||||
|
||||
def test_shared_pem_survives_removal_of_one_name(self):
|
||||
cert = self._cert_for('example.com')
|
||||
before = self.read(cert)
|
||||
self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
|
||||
self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
|
||||
|
||||
resp = self.remove('www.example.com')
|
||||
|
||||
self.assertEqual(200, resp.status_code, resp.data)
|
||||
self.assertTrue(
|
||||
os.path.exists(cert),
|
||||
'example.com is still configured and still served from this file')
|
||||
self.assertEqual(before, self.read(cert),
|
||||
'the surviving bundle must be byte-for-byte intact')
|
||||
self.assertTrue(self.edge_would_start(),
|
||||
'the edge must still load the crt directory')
|
||||
|
||||
def test_shared_pem_survives_removal_of_the_bundle_primary(self):
|
||||
"""The worst shape: the name being removed IS the lineage/file name."""
|
||||
cert = self._cert_for('example.com')
|
||||
before = self.read(cert)
|
||||
self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
|
||||
self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
|
||||
|
||||
self.remove('example.com')
|
||||
|
||||
self.assertTrue(
|
||||
os.path.exists(cert),
|
||||
'www.example.com is still configured and is served from this exact '
|
||||
'file - removing the apex must not unlink it')
|
||||
self.assertEqual(before, self.read(cert))
|
||||
self.assertTrue(self.edge_would_start())
|
||||
|
||||
def test_shared_lineage_is_not_certbot_deleted(self):
|
||||
cert = self._cert_for('example.com')
|
||||
self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
|
||||
self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
|
||||
|
||||
self.remove('example.com')
|
||||
|
||||
self.assertEqual(
|
||||
[], self.certbot_deletes(),
|
||||
'certbot delete destroys archive, live and renewal config for a '
|
||||
'lineage that is still renewing the certificate www.example.com '
|
||||
'is served with')
|
||||
|
||||
def test_production_shape_mail_alias_removal(self):
|
||||
"""The exact row that stopped a production cleanup.
|
||||
|
||||
mail.threeworldsoneheart.org carried
|
||||
ssl_cert_path=/etc/haproxy/certs/threeworldsoneheart.org.pem - the live
|
||||
PEM of a different, serving site.
|
||||
"""
|
||||
cert = self._cert_for('threeworldsoneheart.org')
|
||||
before = self.read(cert)
|
||||
self.add_domain('threeworldsoneheart.org', 'be_apex', ssl_cert_path=cert)
|
||||
self.add_domain('www.threeworldsoneheart.org', 'be_www', ssl_cert_path=cert)
|
||||
self.add_domain('mail.threeworldsoneheart.org', 'be_mail', ssl_cert_path=cert)
|
||||
|
||||
self.remove('mail.threeworldsoneheart.org')
|
||||
|
||||
self.assertTrue(os.path.exists(cert),
|
||||
'two sites are still served from this bundle')
|
||||
self.assertEqual(before, self.read(cert))
|
||||
self.assertEqual([], self.certbot_deletes())
|
||||
self.assertTrue(self.edge_would_start())
|
||||
|
||||
def test_removing_every_name_eventually_cleans_up(self):
|
||||
"""The guard defers cleanup, it does not cancel it."""
|
||||
cert = self._cert_for('example.com')
|
||||
self.add_domain('example.com', 'be_apex', ssl_cert_path=cert)
|
||||
self.add_domain('www.example.com', 'be_www', ssl_cert_path=cert)
|
||||
|
||||
self.remove('www.example.com')
|
||||
self.assertTrue(os.path.exists(cert), 'apex still needs it')
|
||||
|
||||
self.remove('example.com')
|
||||
self.assertFalse(os.path.exists(cert),
|
||||
'the last reference is gone - now it may be removed')
|
||||
self.assertEqual(
|
||||
['delete --cert-name example.com --non-interactive'],
|
||||
self.certbot_deletes())
|
||||
|
||||
def test_unrelated_domains_certificate_is_untouched(self):
|
||||
"""Sharing is by exact path; different bundles stay independent."""
|
||||
mine = self._cert_for('example.com')
|
||||
theirs = self._cert_for('other.example')
|
||||
theirs_before = self.read(theirs)
|
||||
self.add_domain('example.com', 'be_mine', ssl_cert_path=mine)
|
||||
self.add_domain('other.example', 'be_theirs', ssl_cert_path=theirs)
|
||||
|
||||
self.remove('example.com')
|
||||
|
||||
self.assertFalse(os.path.exists(mine))
|
||||
self.assertTrue(os.path.exists(theirs),
|
||||
'a different bundle must not be collateral damage')
|
||||
self.assertEqual(theirs_before, self.read(theirs))
|
||||
self.assertEqual(
|
||||
['delete --cert-name example.com --non-interactive'],
|
||||
self.certbot_deletes())
|
||||
|
||||
|
||||
@unittest.skipIf(TESTING_FOREIGN_TREE,
|
||||
'HAPROXY_MANAGER_DIR points at another tree')
|
||||
class TestPublisherApiIsPresent(unittest.TestCase):
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIC9DCCAdygAwIBAgICEAMwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMZGVh
|
||||
ZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
|
||||
EwYDVQQDDAxkZWFkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
|
||||
AoIBAQCumL3T27a3D6fZbIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnO
|
||||
abvnQNxEVqyYtTJOn9taitVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BU
|
||||
SpW96rK40KVftzzDEUnH+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0
|
||||
seOIdyP/n3T2cX1wsCa/F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9b
|
||||
PBd2cBaBwTe/3TICu0RnvvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2
|
||||
wzG4zlKU31fdLvQOzMGH1W5H91sBAgMBAAGjSDBGMCUGA1UdEQQeMByCDGRlYWQu
|
||||
ZXhhbXBsZYIMYnJhaW5qYXIubmV0MB0GA1UdDgQWBBSPDJiv0Z2YpuloexOfvpIQ
|
||||
UrWufTANBgkqhkiG9w0BAQsFAAOCAQEAJwBtN3Qw41XvYuSa8Mjn869QOTKEgkaR
|
||||
EuNQ8WXVBOTRJdxaLHDunGO4ExjHHswCWzXkefYOoIClAU9qLnk9M667BSNxOcz0
|
||||
VcG/DUMd0tO8X4Zm0zOMjt4VC7whgj7MdYtYVg2+Jhb4sMSx6OqmCUAD4X6ULSTn
|
||||
lZK2holEC/H3bKt8Q6cecgeubgWojP1bWmWK7jQL5NRIzxQ0KzFxgAf3t2Z05nX8
|
||||
loABkBiR7vDthkQseUW6RQMayDyzhVqORLt+gZ9D5T3hT1M7VaDc4Twr7UbWPFcR
|
||||
GuQ7RwKplPIIbPe7+c91pm+agdSDjnV5HmKcAMCwf7bWGQ9/Xc0BFg==
|
||||
-----END CERTIFICATE-----
|
||||
@@ -1,28 +0,0 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCumL3T27a3D6fZ
|
||||
bIgwovdH6SfvKAkz3NwBiDLTtIn+Zia3l5FynrGXWZnOabvnQNxEVqyYtTJOn9ta
|
||||
itVc30wLVpk31BQdLab5rdVa4IADnd3Mel5Cd0M+P3BUSpW96rK40KVftzzDEUnH
|
||||
+3ncuzPQ7sKXom0JYqBXT42s8LF5UKtXI1oF1iW/7uz0seOIdyP/n3T2cX1wsCa/
|
||||
F0v2MAyze5FhdtWij+2IK1FRbIbSSr6zUCAeAWdqiM9bPBd2cBaBwTe/3TICu0Rn
|
||||
vvRBvlMNrgbEBm+wnDf2qlxAO2pSER1bipi4C9gu1nm2wzG4zlKU31fdLvQOzMGH
|
||||
1W5H91sBAgMBAAECggEADUvYBFLn6gPWSZj55nxOqR+2Lmtwh04omgbu2fpWhPTE
|
||||
ujgKd/23srnnMZXnR6qZQQIPGwJiOPnJuRjiu/PMZSj8j54bbwfA6XmbH9SCIzm0
|
||||
la50UGBSkw0T50jBI76vICMm6PBkf0rT1oKN9VOd/qlkJoCUiFcGCh177Fbt07oY
|
||||
R3iAKu7/+yEANX0iEcxbO330TPW3oI0ah5OryfKVoNvxpF8NUfV9pWY66APkOYMJ
|
||||
zsYvIMvzV/KIgfrtyxJjSjgd7Ud8uictYG1BrZBXpIuGxuxyKUJ7OXOvMMVNnEME
|
||||
2uzk/aL75UjpurJJU0LCywdUOGwRqS2q4PYQiTeDLQKBgQDjdywH7RODoYeH05Rd
|
||||
sa2MkNx0Day1i3A4IBds55mzn8jxpT55HlqvHWhNA30kEw01H8+6sB7iMX04xQtS
|
||||
u3N2opR1VlCvMJhGyRUYUtfu4NnAi/HSu7mCOFGAwaLclI7BAI2or2UrclrlUhGb
|
||||
CYIBKfEr6Tn6b6AffWaOruCbtQKBgQDEf70R922fnJK+//QuiYoaESwJd3/k10yJ
|
||||
nH0ZAgbnFcNLT9YVFxHvlt4RHGIotaBjfN+FKMn045jeekgeVdX+uF8+7FwEazAH
|
||||
zOfxswOeKd8EOooqCD7n7cW3Ma6ev8oWmXwJsz2coRW7Z5zCGQWSzCEDB4yFU9Mt
|
||||
0z6aN9SJnQKBgQCvBZSIqL+6KgD53zArA7sRDKnEVn1M13TKis7bij02/DyGyuoO
|
||||
EfAfDBakI4oXxt0Dwm15LWkmDIRYuS0kRH3si96FhpvgoRiaZsDnjV7/PF3wjP3O
|
||||
c4mjDCiSVOw2o8xP71gQspy62GveRc6sJ52zvY0Hh5aH/3oKoNfX8/F5fQKBgALQ
|
||||
vbnDLMvrp2KbMjl0eQkUk+fy0V9LAmq6VkSEderbsfejRjL5KKWhKL9gA7wHliu2
|
||||
v1EeG36pwCf34o6ktjM3Gw5AvrTFgGOsOOqKoROkD/D0kcQ2aAhUrM+KI7FcmB2o
|
||||
x1EwQ0K6QCB2RJW3XBAVYqezdPz4DX+LlksJX781AoGBAISGOer3s4+oRjO/2VuT
|
||||
hnMQrET3A0GTol8cgYL2N9wfx7jy5bhSfiJIX/dkh5IrPGYoBdWM87gcUgNQ8AZ7
|
||||
il5dVeG80oc7KFU/QltZJu9VS32vwXnNsnrXupQ5Ee0ztTLciTMWU5jxvB0ZJHtK
|
||||
StgYepr4IPDUaObqrPx4Jvfm
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -1,19 +0,0 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIDGjCCAgKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwGDEWMBQGA1UEAwwNYnJh
|
||||
aW4tamFyLmNvbTAeFw0yMDAxMDEwMDAwMDBaFw0yMDA0MDEwMDAwMDBaMBgxFjAU
|
||||
BgNVBAMMDWJyYWluLWphci5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
|
||||
AoIBAQC9SJUMsK5HhVEATu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9s
|
||||
i5KzgYp0K2z899wQAKuLzZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ
|
||||
7jax2QWFMQVhyy6M5QTCHUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyN
|
||||
N8eolukhvFxQNU3E7oo1jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSy
|
||||
UEc+Gje6dJLZ9CdJtSQSNGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8
|
||||
FHC7atJOe5EoaekaAXw5Tq+yB+ODAgMBAAGjbjBsMEsGA1UdEQREMEKCDWJyYWlu
|
||||
LWphci5jb22CEXd3dy5icmFpbi1qYXIuY29tggxicmFpbmphci5uZXSCEHd3dy5i
|
||||
cmFpbmphci5uZXQwHQYDVR0OBBYEFAWqbBh2PDUvqQzZoyh960cpuhcOMA0GCSqG
|
||||
SIb3DQEBCwUAA4IBAQCaovtrXl56YcFlXWMrW/XZBe2/fXI14860qyiR5rb07lS6
|
||||
CbKOsnRyQdxZ3cdfRwTN1UMmz5IPOZ6fhdo3IPaS2Wae1OdeupRaRWr1SAJtpVZ3
|
||||
WWugDaumsWV5eWVxVTf93yDEkMERWtTGMviCKBeeHjzHIFiD/EjZznE87V5DYwuf
|
||||
guOd1F4wXCMRFu8q7sBdPXjQ8rZMlQT+dQXzMST9Ee5PXiInmCAA5nT8vMInAwd2
|
||||
rP6FxuHJB75ekYzs57CJ82BKKboOCESjUbm54wmVV5pBL305/EjxGf+VovNlRktg
|
||||
t1znolntIIofE25UCbVNPdrtuDkZgKT3Rv8tCdTP
|
||||
-----END CERTIFICATE-----
|
||||
@@ -1,28 +0,0 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC9SJUMsK5HhVEA
|
||||
Tu/FWwJsvaK7lkigh7EhlkxTNkLFNzlgkLcoKMhD2F9si5KzgYp0K2z899wQAKuL
|
||||
zZcde6BUnw6RHzheCRsIqLZlbY8B3Ayd6VhSMYqhWoXJ7jax2QWFMQVhyy6M5QTC
|
||||
HUVJACMpega+KrsYuNkNAxLc2uGhgHLPnnOoDhGXosyNN8eolukhvFxQNU3E7oo1
|
||||
jBzzzSN+FFPCZ8oWab6ydW7YSxZLlxRWoQ+2mL9I+SSyUEc+Gje6dJLZ9CdJtSQS
|
||||
NGnFoqOUGIipNpS2rrBGctVWijM9y459skTFKbamLPy8FHC7atJOe5EoaekaAXw5
|
||||
Tq+yB+ODAgMBAAECggEABAeOCL71eYH4jKRMOBbXu98483u1CxnTEa8klnw9ZVES
|
||||
JctJjY96B/EMwv+g/V+gxhgAkiK9VWfg6bEplOVDWgR9fRBoP57mVkOv3QPn5fcR
|
||||
xlt9LQQZ+pULQRZoUAFEkVEkV1sfIBJmDLMufOnbAHCKa9+hVlrlCqu0lJgM8put
|
||||
2bkpSAqwroHVfmD0SmKFAZggBhFwXFJzwmUf3WL+N25f4RZpJsX17FOJ216mlhQ5
|
||||
edje6dcDpmko2VQtGJXYr8Ge02ZYeRyKGcTsgUZlarB4xUZq1pcnUUSy0Ez7LVbk
|
||||
tKjPcUpZLlnAA/a2pDgHmwN1+fv/X+Wwv/vZaQMBTQKBgQDuZFyfDRSHCssHtQ4+
|
||||
Ezxg4WEjwXkq58WxL0zTtLTcquZfQZ1VXgLwAylVwver0I4/UgCfW/AYY+2eLrba
|
||||
FhuYOosd/3rsUQqf9y+TkSkhe1uVS2/1zjzC1G/5coleLddjzd+YBpEtVxfxDRbA
|
||||
rVePY3GxkJiTrjAg5OETk7GYpwKBgQDLQ6bPhaocJPQmDi3hKyW8zPJ8wAJNyI0c
|
||||
RWt9gjeidjmMvwCKtJbBDcV5V0oQ5SB2M5E49mT58B62totAD7FS18c4a4133haV
|
||||
a1pLBX0U524eQgDzMJ6YA6N9bHdXOH4FZd+Pw90bHetbT1HcKpOdXv4SF/XITju5
|
||||
Ro5xiymdxQKBgBB4fq2+iZyK8lwl9kXzrkhDSiLeSzDbspz7V6OI6LYWWLRcJDKx
|
||||
vCI5FbIHNnV4g1t8nu5ju8LzpIc5UwjeXim2YcZA0I1jTAihW3gmn2Dg1QMLo0Co
|
||||
bypG4zWo5DTNlgQfRUOLsJwhWFYfQWMuls4hYMmT4/bQIiMjrSldGvghAoGALpaB
|
||||
7DyZobvCjdpgH6TREHX0+izUGlnmJd2vZmPrNnxccmrwx9neV789TLN6EWdCynTF
|
||||
S0YGHNcxYvVx8yvlm7RDQCY+lYO8r2CcoGdQpPLqZVsBdrvlDbh/x2aN79rN35Ta
|
||||
6cudqPJxmEsU2x6TGNXF7AW72r47qBqI2XNpBXUCgYAFQdHRRxa2E5rgV0WkL7PF
|
||||
vXok3isAzpnKAsb5oxPZbhIdLo5EPUGCslsTE1Na8NBdUSoyjhgkMELEDx/ceGL7
|
||||
IIlNrc/4j71+IahljzaX/D5b5qaW+RnoTVAlbMRR7L/ecbkoovhQR9d37QdoiD/t
|
||||
B9si92+XB4HYDAjXFxTK3w==
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -1,18 +0,0 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIC+DCCAeCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwFzEVMBMGA1UEAwwMYnJh
|
||||
aW5qYXIubmV0MCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAwWjAXMRUw
|
||||
EwYDVQQDDAxicmFpbmphci5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
|
||||
AoIBAQCyhL5YcFhPEekc/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdb
|
||||
l4twhlkcyvyaEbcHIgLEgz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgU
|
||||
QcAAvdAyP77jDF6lX11oCjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zz
|
||||
XRVp4l2WyqEUJmKxcpppAnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU
|
||||
0v78QfI+ueCw261UWZ+iXwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQ
|
||||
dFOfKjH0QGInyByji9eAaXlmvCKtAgMBAAGjTDBKMCkGA1UdEQQiMCCCDGJyYWlu
|
||||
amFyLm5ldIIQd3d3LmJyYWluamFyLm5ldDAdBgNVHQ4EFgQUdDxfZKX96j61mW7O
|
||||
NiQpN4T+sZ4wDQYJKoZIhvcNAQELBQADggEBAJE1lP62caByZT7GkKtJLfOj84Ih
|
||||
MeP751w2gdU8ZfiO8GxXXiR3ZjYzbJgsEi9LiNT9jqHeBf3pv/mJLjPVZ+U557ws
|
||||
Gq3V5A3uKlRKGcejXlHaWnExhWqLyIaLTeLKstbJyiB/ja4yeTL2TYcOvvlu6weh
|
||||
ERqwx0p6r7LSQ3nRfTdjEhNq3FVUmqQ8neBP/vXLp2hKr2ACZndGyQOiv0oZWzud
|
||||
part6RMd8gRNq1Tctrl0YqRYoLRIddBrnTtUtac4FLqxLSBNc03/I9ZgF56Yf+Sl
|
||||
M1ikmyDJ2ZYJFU6v42Og4/8o3fpq0ttP6pKPi8fInGaFQ+KXO1NWFf1ogD0=
|
||||
-----END CERTIFICATE-----
|
||||
@@ -1,28 +0,0 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCyhL5YcFhPEekc
|
||||
/xNMaEHO0hjoNRd5yNhG3BF+pbChYJhkjw5McdH89fdbl4twhlkcyvyaEbcHIgLE
|
||||
gz5caANrlGnzQBTOBTILhA0leTF9hBCQX1yKEotQFdgUQcAAvdAyP77jDF6lX11o
|
||||
CjJwzoT0YeLcFG3XyfDQ7hRSJNJFDqvbOgCc9kZ+B3zzXRVp4l2WyqEUJmKxcppp
|
||||
AnfbVhokwy2417CumYp8wIzYJ57mRk6RrRRNbENw3wGU0v78QfI+ueCw261UWZ+i
|
||||
XwlZt/icvzWtcuoPVxid3uPeRPGV40RUbPGuetvIjLNQdFOfKjH0QGInyByji9eA
|
||||
aXlmvCKtAgMBAAECggEAEQi8lvjFXGojmco3H7PVD7SZftLOFidVh9Fo8/JAe+2Z
|
||||
LQM/hIjZ1i7sNverbQXMzAQMfXESS2Jx7Clv0mZ0aRgE0hq7wXrabyHfFUPOtqDa
|
||||
6cPHsz8YvwAOmcQ6W3fBPYI6pzSfdV03rv1GGeCC7ETI3mIupwgUbANDuiDcVY1q
|
||||
+VuK/hDzPJ5KslDHUNR42MgWacIftq99fV9jN0zQQw1EcyO8QwpXzTSEbdyiAkT+
|
||||
n/tR9He2MYhgRhjyWPMaxmY6XQpv1h5ASq1yT1kI2oNTaRT3xvd40sLkbfwpZnxi
|
||||
HnqzCkPptEpWeCLrme+r71+Hxu/+Bph7tPtcuK7jgQKBgQD8XCoPRiWbgCl6kM2G
|
||||
E3GeM6fftTieTIs7sFVte7wutIKlsec4vizc94ipGaoT7yGEG0T8g9+5hAMbebc2
|
||||
zzhxvJMB2s18Z/He+Ujv1L1AhBBNVY3zCAxfdq8/n5MGXppEdVRo9QOQndoBfuAV
|
||||
Qw2Vp4bcy728n4T7sBugW0Ad7QKBgQC1F+utuiF8XI2yx0mX0+Fbty6o+odyM0eb
|
||||
G/QQDlcGpY0K7V3WAFMESNfytig8OhidzRfWn8lNoqhd80SJfCuiYjiGJ8tjBzcp
|
||||
+Vqrd/KAxJySJ3NJ2MwyAp7W3l3HAEZLGVYVCankoa4O3qR8YqaIZwc2nk5+8KnG
|
||||
nBYFlSF/wQKBgQCqaRYh0E2O7wGTeXq/ohukJ9+hqf88rhWLTeyyDM8rfsuGVlb8
|
||||
MzeaUzEed20pHt60FbIFDowh+yoaAJwez9mevCdNJlKCQrLKze1Zvn+5tDRJ9I7x
|
||||
YSrxv/Rhc0FWGMX1moFQirtIib9wKRvqX6gpucHREWfjmpizWN2e8F+glQKBgAOp
|
||||
/jy/nVnUzZEmcxi9Urv2Ut1cZE0tOG+s+x0wDRa3Vsk30EQ6MXPOvX/Nq7a8Pi9D
|
||||
LQ5MoVgOyHVOLL7TYlagwiAEwhq41KbmwqzhemickcZrQevmlQsV/ny+dwtFuCZg
|
||||
1tsKHQPhOZJXcuCJZnEwFHdIcLIk0GqSUee/ArSBAoGALOD+Opk+0j4ObNcqtrpP
|
||||
UcEYqhOuAscbxLoUv53wnOvc7XzMG8JCK1y5LthB2Xanr1JSlheMRrvGxC+WlDmc
|
||||
5v0Nz71cq5QQ1p8AXnosLRjuy/+edMvGIxxwFMEJBO2ra2Jizpdwd34bjpQ42+CQ
|
||||
6pVwFQX0oet8SI5xWUWx7NU=
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -1,19 +0,0 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIDGjCCAgKgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwHTEbMBkGA1UEAwwSc2hh
|
||||
cmVkLW9sZC5leGFtcGxlMCAXDTI2MDEwMTAwMDAwMFoYDzIxMjYwMTAxMDAwMDAw
|
||||
WjAdMRswGQYDVQQDDBJzaGFyZWQtb2xkLmV4YW1wbGUwggEiMA0GCSqGSIb3DQEB
|
||||
AQUAA4IBDwAwggEKAoIBAQCBg7iYuvjhIG7nlFeja5Lyv2t2fdtNvxzUib7ygejC
|
||||
+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpjo5/ym2KFZPwiSZoqKUugi9lCoKuL
|
||||
2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMHYFwURjARRAku3a/GJUHQR0C572mW
|
||||
mt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64fGeYSOaYOHEOp/VqIRlzsy7eGQh3
|
||||
xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2JbqFxvijCLnBz5WsrS/S/VWOELNEfGr
|
||||
84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLfWuSgp5hRAgMBAAGjYjBgMD8GA1Ud
|
||||
EQQ4MDaCEnNoYXJlZC1vbGQuZXhhbXBsZYIMYnJhaW5qYXIubmV0ghJvdGhlci1z
|
||||
aXRlLmV4YW1wbGUwHQYDVR0OBBYEFP+AdZ9MkBmFjAQTgLnWZxbJtnvkMA0GCSqG
|
||||
SIb3DQEBCwUAA4IBAQA32Xl/1IrOyESd2UQcWp/fuHvkivLgamJrheKvKP0Y0vMV
|
||||
atRu0D0GhqMOr79OT60H/H1FFW7bWQ+UPcZpnuXec40/PcrHxwaSoiLoIUVWhB4t
|
||||
BOhtTDReX29gEjnB7OqBCJu4ub3HnpFTflUZd7n5A4EcetBsowc5EZ9UAtfvkgGD
|
||||
nK9ONvNVGurk/V8xgtMJ877ZK1fn+UtgvIWp/F1Gc/taLhe+oCZOPkOqvvqUnS8q
|
||||
QyIhqmMWRUkBP+jwRA4TP4pj7OH38VY/Ae408Xe4Y97GF7NbOjAL0bkX/bHkLnSG
|
||||
WzoGO0TK7pCemeFQ+Ghic8Yl0eDynoP7rZIhCKS0
|
||||
-----END CERTIFICATE-----
|
||||
@@ -1,28 +0,0 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCBg7iYuvjhIG7n
|
||||
lFeja5Lyv2t2fdtNvxzUib7ygejC+WQvNdH4k12q8vCXNYuadbGAceSejG4bazpj
|
||||
o5/ym2KFZPwiSZoqKUugi9lCoKuL2EUmCHoTsr+CLPuZXyjxu/x250m5llJOhgMH
|
||||
YFwURjARRAku3a/GJUHQR0C572mWmt/WtWueHlqEFoI7jCuBAUmww1p1YNV0cl64
|
||||
fGeYSOaYOHEOp/VqIRlzsy7eGQh3xLQzEMyXzrjD54PZ6NuJeT7heZHVO3if2Jbq
|
||||
FxvijCLnBz5WsrS/S/VWOELNEfGr84/Jcwze9ZV8e0y4tOwh9qT82adbl1nyAgLf
|
||||
WuSgp5hRAgMBAAECggEACB1tvTwciKvp5wliGF45feDv5c5q+vXE462h0OYXjyfl
|
||||
1sNuMuoLjhYyFvdcQO2rlTBjB8Ald1hXoeidsVYlF9XvqmJRjjS6MAazMpEzTnba
|
||||
66V6U1j5gOQ6pi3SjfBJGFYo8L+cy/NqUIBWz0+6hp/BsjW3uW2aMGer2Y/zs2Oi
|
||||
s0kUNqqUWQpGpEDxbDnUAzto4OOcTOO4A5NB5Oe2ifzFiXCWtTNqFIC0ae9tiqGy
|
||||
m0sggF2hzmlH1WC54rbv7xujYT6tOAEi1OKp/Cpj8XVLbjaqju5avRl1t2nDRQri
|
||||
9vnOI9P4bnJKbIYzwJVyy8B1M5iK+W2K7wUQ4ZMetQKBgQC2zwtywQ4jUduLWf4f
|
||||
6HLUIvmcHdJI0YBHS1+4fKc7tsakRo/0FHxNdUlBhqkYTyp57h6/R1T2gG70Kla1
|
||||
MarGIgyPIlUXsRojlUcNyKXqSVs+kejftIClb+VDXWrpKtGtyxuESsMjaJBsBBAN
|
||||
hg3yUx2NL2ct4nnlpl1iq3pH9wKBgQC1XkrjntuzjVYlFeg2ih5va3PUD2qmH9Ej
|
||||
leNXTGhzZL1eQWz4GaRCgzJH8q41pXBxcXQ8hMXZCWdW5GHBYfVqjTRKMUPdI6FI
|
||||
ypwMbbTBfQfw1kz5VHfLH3romxikNQXeorZkIZqv7/aLPsxLY78XdW8nSDIsT9ZP
|
||||
mh7It+Tf9wKBgHT7VSYf6D4t2cH42qfshd6+auC+LJywg9gboatbCWEq+z6QcPoO
|
||||
SeT/Utt9oqONtW7FNCPPL812LopMtaPJnrCfhU3q+GzqxgFTO17UsWeVfbAyisTs
|
||||
qw4y5pEPLKT846TFPlyPwwPbm9+jUep1iGMzUMvSM8KnFL4ikcvcU6nNAoGBAJQI
|
||||
MVHV0HadzokYd19EnpDMgE6+goqy/21vIo6XY82DlPjAp+NccnCDSt/kmQJaVQuS
|
||||
0Y+vwxhhj4IhiBRiEwBz3hEMYtq1ASg7juCqDlW66mdNUqhSpDAbMqN2K8ejK3ZL
|
||||
EllqACrHzlBa4j5KXKcWB7JlP+ZhB4gTQDsNtXaDAoGBAJCktIdyM3b6UtM5OofB
|
||||
lzn6/2uT5u46ECBRybk/J+Y2gfcS2cFzkH179SUB4WgfR0h21FlLmjns5ZvHVk0p
|
||||
/U4aY3/J42JU6lCYrsyV5BBH+O0stSmrUG/4JdKRe6Mq/pB321RsWD06LVJrBDpi
|
||||
6NUPW03tNLc+Qe+cnQu0c7y+
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -1,18 +0,0 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIC8zCCAdugAwIBAgICEAQwDQYJKoZIhvcNAQELBQAwHDEaMBgGA1UEAwwRdW5y
|
||||
ZWxhdGVkLmV4YW1wbGUwHhcNMjAwMTAxMDAwMDAwWhcNMjAwNDAxMDAwMDAwWjAc
|
||||
MRowGAYDVQQDDBF1bnJlbGF0ZWQuZXhhbXBsZTCCASIwDQYJKoZIhvcNAQEBBQAD
|
||||
ggEPADCCAQoCggEBALEpfqdRPBSrbzixAxD2WhsO1AOTUd/Gguf3zjDZ3TUmWJKb
|
||||
OhlFthtU6PJVhMdAqOk+hiz9Coo/UngWjykyNMOj1YIpq3M4/P8wBTNxkJd/z2x0
|
||||
iAQBKvWlB6hxf9c1kenu/ea1iMIThA/uudtX3g8k39Asj/37+o070cdBJExWGnbG
|
||||
lz103FCO6kBzlJPGIVQwzKH6OKthTzi7Uxy5Fxc/ymS9gc43F0tyY6892Wyl8Y62
|
||||
a3r1PpZoWR0GDFxPIyCS7NP44T15DSL/2RBi4YnWXOC8QMvPnHbSKjoQ6GLQtGkO
|
||||
1NO4Uy3XbwDFanYXv6Wxv0oUmZMuY7CRTOMdxCMCAwEAAaM/MD0wHAYDVR0RBBUw
|
||||
E4IRdW5yZWxhdGVkLmV4YW1wbGUwHQYDVR0OBBYEFBh0L/5U5f94TylibBCgS9pD
|
||||
/VXkMA0GCSqGSIb3DQEBCwUAA4IBAQB5MLqNjTqIdyztjJsKPdYUkA27NpfdqqY6
|
||||
Yprv5N7U0slsGUqFIDrSuJ1mMMwTpt02IAFyCOWMUtze+/0PqM9k9TqxJH2cTODp
|
||||
0ET34Qh8NTo0B534EQUWhny+EzgNAzWK02wbpgm88+Mp/t//sHzhWzOm7uk1dlkq
|
||||
+5ty/8OveWwAtqAWlDtcsRhkEjzynZKUCAF1extwUSuapFSm20mJ4ivt7Ff5MkTI
|
||||
fq8P+3XAxWTaBW8DzBIzYd1mwFHK2pcklLMRRTW3hukPxEcTivBgJdwsHMskM3UO
|
||||
/ejy5EHem5ohCMbwMybvUqLS4xdoqXV36Sj2I1zv0IjgO0wgdK1r
|
||||
-----END CERTIFICATE-----
|
||||
@@ -1,28 +0,0 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCxKX6nUTwUq284
|
||||
sQMQ9lobDtQDk1HfxoLn984w2d01JliSmzoZRbYbVOjyVYTHQKjpPoYs/QqKP1J4
|
||||
Fo8pMjTDo9WCKatzOPz/MAUzcZCXf89sdIgEASr1pQeocX/XNZHp7v3mtYjCE4QP
|
||||
7rnbV94PJN/QLI/9+/qNO9HHQSRMVhp2xpc9dNxQjupAc5STxiFUMMyh+jirYU84
|
||||
u1McuRcXP8pkvYHONxdLcmOvPdlspfGOtmt69T6WaFkdBgxcTyMgkuzT+OE9eQ0i
|
||||
/9kQYuGJ1lzgvEDLz5x20io6EOhi0LRpDtTTuFMt128AxWp2F7+lsb9KFJmTLmOw
|
||||
kUzjHcQjAgMBAAECggEAJjBKo+iH6fC2beMCRTLFY5XDr+2etX0GFNi+8prT6zQT
|
||||
3a5mTLJROf0Ipcw3bfgLMRQQOjxlfD2AGnDKdVfkNdPoH0G6OoErUBra1CcDwxSC
|
||||
8t7/vtU9nGvxhabJputeQ64HvyacjzFDw2dzWsvLCzqooCgq9dXOQYmGg8wP9gAx
|
||||
pxfKnd84h41C3pXERdufi6nJHPjZPRW+dKsQ1NMLmtgO5YQnTt3LfVOkZaRa7Izq
|
||||
ZFWZCBfdsNHHuXCYj3FtqYrqSxELuZAxJ62hxpizrfQdKixguGTObHhC4muIo9fR
|
||||
eIMCvrAzrs4Fs8nM/K/IPyDU9wPwINK6Z+sxJkhM+QKBgQDaL3Yt3T/LE6bTYyb0
|
||||
EjVcXoybnuAjFQAenCTIsBRJv21Yv2ddqjq6rOTD64hFuFWPMEOi1TJxIPsFFDMw
|
||||
NeAwF41fiSrtU8awXs0hTKRQeHjElSBckumOl1DrHu2oKuwWXLniViD62vMsWmpZ
|
||||
89KT2rgFQuMPvsD3pps8l6W2HwKBgQDP3eTwdVJTdRYlZonqd0+LHG7h715TT1OE
|
||||
QJOdN9vK5g/9GOCOGHoUxB3IxJVmvz57PrfWaD2rSZ6JMs4TcAY5a0PbJEpgqoJY
|
||||
kIm/EDnSrFxu2L8DNsAzuThfzXrwWc7PzajYJctYtEfvxCyfo/GdZahg2xaEU0Le
|
||||
dp+7fxtJfQKBgGVdLWtLYarH+3Yzl1JG71F1f5+A5SKHTCHsas5tt083xuY3Lh9+
|
||||
yYAWeMVotEunwuAeDR4mLCQQrW4Dn3BjZz8QghaDVFUj2afwsx4oTCIyhfDWU02O
|
||||
10DunaJOIR1Cq6IL/DPhzbqdY+IUM1qNAyS7tUiEdZbICUBEv9Qbo9oDAoGAORnY
|
||||
cSGZaumoGsAPNMRvqmQ2JYSO4c3RwGvI4RBeJV+LlthFn2Mj7h5tnCA8dkwkytBQ
|
||||
yeo6vpnrnf7B018bppxLuKBAekt8oP2biUvYxoGopp6N4kEhDVRdIxwcIHhbMoRp
|
||||
VG8VbBmiBUBbjcMIdVdeqLSme9UvclMdz/C7uzUCgYEAzD2dvoJSu9zfo3fACw1X
|
||||
te53Rvr6Kq6DTw0SS4UrAnxlhX9TwD9gymIrR9rxnC+UPwfMwWMdpYqY+Da4sF1G
|
||||
uwd+nyZ4iFNq+lbLii9rmJ58h03+uo4lUXWTrAik6qOF1ztIfYKWFrOyBQmxjWwo
|
||||
HnUinudalENFJYZRwSUAtXo=
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -1,631 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regression tests: a silent peer must not be able to wedge the API's threads.
|
||||
|
||||
Why this file exists
|
||||
--------------------
|
||||
On 2026-10-09 (sdbees) both :8000 API workers had 7 of 8 gthread pool threads
|
||||
blocked in the kernel in tcp_recvmsg -> sk_wait_data, /health hung 40s+, and the
|
||||
container stayed "healthy". The app makes no in-process outbound socket reads;
|
||||
the blocked reads were gunicorn gthread reading the CLIENT socket, which it
|
||||
makes fully blocking with no timeout (TConn.init -> setblocking(True)).
|
||||
|
||||
These tests run REAL gunicorn processes (no mocks) against tiny WSGI apps and
|
||||
pin:
|
||||
|
||||
* the failure mode itself, on the stock gthread worker (control);
|
||||
* that haproxy_gunicorn_worker.TimeoutThreadWorker frees the thread when a
|
||||
client stalls before / during the request line, mid-body, or on a reused
|
||||
keep-alive connection;
|
||||
* that a legitimately SLOW request (handling time > client timeout) is not
|
||||
cut off -- the timeout bounds socket I/O, not request duration;
|
||||
* the watchdog: a worker whose every thread is stuck is replaced;
|
||||
* review round 1, at production shape (2 workers x 4 threads): 2050
|
||||
connections that never send a byte are reaped (F1, stock gthread wedges
|
||||
permanently), and slowloris drip connections are cut by the total header
|
||||
deadline while /health stays responsive (F2);
|
||||
* the timeout is logged as one line, not a traceback per scanner;
|
||||
* haproxy_manager's socket.setdefaulttimeout: an in-process client call to a
|
||||
stalled upstream (accepts, never replies) times out instead of hanging;
|
||||
* why that default alone is not enough (setblocking(True) clears it).
|
||||
|
||||
Running
|
||||
-------
|
||||
python3 scripts/test-gunicorn-socket-timeouts.py
|
||||
|
||||
stdlib unittest, like the other scripts/test-*.py; needs only the app's own
|
||||
requirements (gunicorn, Flask...). Runs in the Docker build gate. ~70s.
|
||||
"""
|
||||
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
import urllib.request
|
||||
|
||||
REPO_DIR = os.path.abspath(
|
||||
os.environ.get('HAPROXY_MANAGER_DIR',
|
||||
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')))
|
||||
|
||||
try:
|
||||
import gunicorn # noqa: F401
|
||||
except ImportError:
|
||||
sys.stderr.write('SKIP: gunicorn not installed (pip install -r requirements.txt)\n')
|
||||
raise SystemExit(77)
|
||||
|
||||
WSGI_APP = textwrap.dedent('''\
|
||||
import os, time
|
||||
|
||||
def app(environ, start_response):
|
||||
path = environ.get('PATH_INFO', '/')
|
||||
if path == '/sleep':
|
||||
time.sleep(float(environ.get('QUERY_STRING') or 3))
|
||||
elif path == '/hang':
|
||||
time.sleep(3600)
|
||||
elif path == '/body':
|
||||
n = int(environ.get('CONTENT_LENGTH') or 0)
|
||||
environ['wsgi.input'].read(n)
|
||||
body = ('pid=%d' % os.getpid()).encode()
|
||||
start_response('200 OK', [('Content-Type', 'text/plain'),
|
||||
('Content-Length', str(len(body)))])
|
||||
return [body]
|
||||
''')
|
||||
|
||||
|
||||
def free_port():
|
||||
s = socket.socket()
|
||||
s.bind(('127.0.0.1', 0))
|
||||
port = s.getsockname()[1]
|
||||
s.close()
|
||||
return port
|
||||
|
||||
|
||||
def http_get(port, path='/', timeout=5):
|
||||
with urllib.request.urlopen('http://127.0.0.1:%d%s' % (port, path),
|
||||
timeout=timeout) as r:
|
||||
return r.read().decode()
|
||||
|
||||
|
||||
class Gunicorn:
|
||||
"""A real gunicorn master on a free port, with its output captured."""
|
||||
|
||||
def __init__(self, worker_class, threads=1, env=None, workers=1, extra_ports=()):
|
||||
self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
|
||||
with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
|
||||
fh.write(WSGI_APP)
|
||||
self.port = free_port()
|
||||
self.log_path = os.path.join(self.tmp, 'gunicorn.log')
|
||||
e = dict(os.environ)
|
||||
e.update(env or {})
|
||||
self.log = open(self.log_path, 'w')
|
||||
self.proc = subprocess.Popen(
|
||||
[sys.executable, '-m', 'gunicorn',
|
||||
'--bind', '127.0.0.1:%d' % self.port,
|
||||
*sum((['--bind', '127.0.0.1:%d' % p] for p in extra_ports), []),
|
||||
'--workers', str(workers), '--threads', str(threads),
|
||||
'--worker-class', worker_class,
|
||||
'--timeout', '120', '--graceful-timeout', '2',
|
||||
'--log-level', 'info', '--error-logfile', '-',
|
||||
'--pythonpath', '%s,%s' % (REPO_DIR, self.tmp),
|
||||
'tapp:app'],
|
||||
stdout=self.log, stderr=subprocess.STDOUT, env=e)
|
||||
deadline = time.time() + 15
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
http_get(self.port, '/', timeout=1)
|
||||
return
|
||||
except Exception:
|
||||
time.sleep(0.2)
|
||||
self.stop()
|
||||
raise RuntimeError('gunicorn did not come up:\n' + self.output())
|
||||
|
||||
def output(self):
|
||||
self.log.flush()
|
||||
with open(self.log_path) as fh:
|
||||
return fh.read()
|
||||
|
||||
def stop(self):
|
||||
if self.proc.poll() is None:
|
||||
self.proc.terminate()
|
||||
try:
|
||||
self.proc.wait(5)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.proc.kill()
|
||||
self.proc.wait()
|
||||
self.log.close()
|
||||
|
||||
|
||||
def stall_client(port, payload=b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03'):
|
||||
"""Connect and send a few bytes that are not a complete HTTP request (by
|
||||
default the start of a TLS ClientHello -- what an HTTPS scanner sends to
|
||||
this plain-HTTP port), then go silent and keep the connection open."""
|
||||
s = socket.create_connection(('127.0.0.1', port))
|
||||
if payload:
|
||||
s.sendall(payload)
|
||||
return s
|
||||
|
||||
|
||||
def wait_closed_by_server(sock, within):
|
||||
"""True if the server closed (EOF/RST) the connection within `within` s."""
|
||||
sock.settimeout(within)
|
||||
try:
|
||||
while True:
|
||||
data = sock.recv(4096)
|
||||
if not data:
|
||||
return True
|
||||
except (ConnectionResetError, BrokenPipeError):
|
||||
return True
|
||||
except socket.timeout:
|
||||
return False
|
||||
|
||||
|
||||
STOCK = 'gthread'
|
||||
FIXED = 'haproxy_gunicorn_worker.TimeoutThreadWorker'
|
||||
FAST = {'HAPROXY_MGR_CLIENT_TIMEOUT': '1'}
|
||||
|
||||
|
||||
class StockGthreadControl(unittest.TestCase):
|
||||
"""Proves the failure mode is real, so the fixed-worker tests mean something."""
|
||||
|
||||
def test_stock_gthread_is_wedged_by_one_silent_client(self):
|
||||
g = Gunicorn(STOCK, threads=1)
|
||||
self.addCleanup(g.stop)
|
||||
s = stall_client(g.port)
|
||||
self.addCleanup(s.close)
|
||||
time.sleep(0.3)
|
||||
with self.assertRaises(Exception, msg='stock gthread should be wedged'):
|
||||
http_get(g.port, '/', timeout=3)
|
||||
|
||||
|
||||
class TimeoutWorkerTests(unittest.TestCase):
|
||||
|
||||
def start(self, threads=1, env=None):
|
||||
g = Gunicorn(FIXED, threads=threads, env=env or FAST)
|
||||
self.addCleanup(g.stop)
|
||||
return g
|
||||
|
||||
def test_silent_client_is_dropped_and_thread_freed(self):
|
||||
g = self.start()
|
||||
s = stall_client(g.port)
|
||||
self.addCleanup(s.close)
|
||||
t0 = time.time()
|
||||
# Only one thread: this request can only be served once the stalled
|
||||
# connection has released it.
|
||||
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
|
||||
self.assertLess(time.time() - t0, 4)
|
||||
self.assertTrue(wait_closed_by_server(s, 3),
|
||||
'server should have closed the stalled connection')
|
||||
|
||||
def test_connect_and_send_nothing_does_not_hold_a_thread(self):
|
||||
# Never readable -> never handed to a pool thread (main loop only).
|
||||
g = self.start()
|
||||
s = stall_client(g.port, payload=b'')
|
||||
self.addCleanup(s.close)
|
||||
self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
|
||||
|
||||
def test_half_written_headers(self):
|
||||
g = self.start()
|
||||
s = stall_client(g.port, payload=b'GET /health HTTP/1.1\r\nHost: x\r\n')
|
||||
self.addCleanup(s.close)
|
||||
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
|
||||
self.assertTrue(wait_closed_by_server(s, 3))
|
||||
|
||||
def test_body_shorter_than_content_length(self):
|
||||
g = self.start()
|
||||
s = stall_client(g.port, payload=(
|
||||
b'POST /body HTTP/1.1\r\nHost: x\r\nContent-Length: 100\r\n\r\n0123456789'))
|
||||
self.addCleanup(s.close)
|
||||
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
|
||||
|
||||
def test_timeout_reapplied_on_keepalive_reuse(self):
|
||||
g = self.start()
|
||||
s = socket.create_connection(('127.0.0.1', g.port))
|
||||
self.addCleanup(s.close)
|
||||
s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
|
||||
s.settimeout(3)
|
||||
self.assertIn(b'200 OK', s.recv(4096))
|
||||
# Same connection, now stall mid-request. gthread calls conn.init()
|
||||
# (setblocking(True)) again for the reused connection.
|
||||
s.sendall(b'GET / HTT')
|
||||
time.sleep(0.3)
|
||||
self.assertTrue(http_get(g.port, '/', timeout=5).startswith('pid='))
|
||||
self.assertTrue(wait_closed_by_server(s, 3))
|
||||
|
||||
def test_slow_legit_request_is_not_cut_off(self):
|
||||
# Handling takes 2.5s, client timeout is 1s: must still succeed.
|
||||
g = self.start()
|
||||
self.assertTrue(http_get(g.port, '/sleep?2.5', timeout=10).startswith('pid='))
|
||||
|
||||
def test_timeout_logged_as_one_line_not_traceback(self):
|
||||
g = self.start()
|
||||
s = stall_client(g.port)
|
||||
self.addCleanup(s.close)
|
||||
http_get(g.port, '/', timeout=5)
|
||||
time.sleep(0.3)
|
||||
out = g.output()
|
||||
self.assertIn('Closed client connection idle', out)
|
||||
self.assertNotIn('Traceback', out)
|
||||
|
||||
def test_watchdog_replaces_fully_wedged_worker(self):
|
||||
g = self.start(env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
|
||||
'HAPROXY_MGR_WEDGE_SECONDS': '2',
|
||||
'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.5'})
|
||||
pid_before = http_get(g.port, '/')
|
||||
# Wedge the only thread inside the app (not a socket read).
|
||||
hung = threading.Thread(target=lambda: self._swallow(g.port), daemon=True)
|
||||
hung.start()
|
||||
time.sleep(0.5)
|
||||
deadline = time.time() + 12
|
||||
pid_after = None
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
pid_after = http_get(g.port, '/', timeout=2)
|
||||
if pid_after != pid_before:
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(0.3)
|
||||
self.assertIsNotNone(pid_after)
|
||||
self.assertNotEqual(pid_before, pid_after, 'worker was not replaced')
|
||||
out = g.output()
|
||||
self.assertIn('WEDGED WORKER', out)
|
||||
self.assertIn('tapp.py', out) # the stuck thread's stack was dumped
|
||||
self.assertIsNone(g.proc.poll(), 'master must keep running')
|
||||
|
||||
def test_watchdog_leaves_partially_busy_pool_alone(self):
|
||||
# 2 threads, one hung forever: the pool still serves, so no restart.
|
||||
g = self.start(threads=2, env={'HAPROXY_MGR_CLIENT_TIMEOUT': '1',
|
||||
'HAPROXY_MGR_WEDGE_SECONDS': '1',
|
||||
'HAPROXY_MGR_WATCHDOG_INTERVAL': '0.3'})
|
||||
pid_before = http_get(g.port, '/')
|
||||
threading.Thread(target=lambda: self._swallow(g.port), daemon=True).start()
|
||||
time.sleep(3)
|
||||
self.assertEqual(pid_before, http_get(g.port, '/', timeout=3))
|
||||
self.assertNotIn('WEDGED WORKER', g.output())
|
||||
|
||||
@staticmethod
|
||||
def _swallow(port):
|
||||
try:
|
||||
http_get(port, '/hang', timeout=60)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def raise_nofile(n):
|
||||
try:
|
||||
import resource
|
||||
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
|
||||
if soft < n:
|
||||
resource.setrlimit(resource.RLIMIT_NOFILE, (min(n, hard), hard))
|
||||
return resource.getrlimit(resource.RLIMIT_NOFILE)[0]
|
||||
except (ImportError, ValueError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
class ProductionShapeExhaustion(unittest.TestCase):
|
||||
"""Review round 1 (F1/F2), at the production shape: 2 workers x 4 threads,
|
||||
default worker_connections=1000."""
|
||||
|
||||
def start(self, env):
|
||||
g = Gunicorn(FIXED, threads=4, workers=2, env=env)
|
||||
self.addCleanup(g.stop)
|
||||
return g
|
||||
|
||||
def test_f1_2050_idle_conns_reaped_and_health_recovers(self):
|
||||
# Stock gthread: 2x1000 never-readable conns fill every slot, the run
|
||||
# loop stops polling, and the worker stays dead even after the
|
||||
# attacker closes everything. Here they must be reaped after
|
||||
# FIRST_BYTE_TIMEOUT -- even while the attacker still holds them open.
|
||||
if (raise_nofile(4500) or 0) < 4300:
|
||||
self.skipTest('RLIMIT_NOFILE too low for 2050 client sockets')
|
||||
g = self.start({'HAPROXY_MGR_FIRST_BYTE_TIMEOUT': '3'})
|
||||
socks = []
|
||||
self.addCleanup(lambda: [x.close() for x in socks])
|
||||
for _ in range(2050):
|
||||
s = socket.socket()
|
||||
s.settimeout(2)
|
||||
s.connect(('127.0.0.1', g.port))
|
||||
socks.append(s)
|
||||
# Attacker keeps every socket open. Within FIRST_BYTE_TIMEOUT plus the
|
||||
# time to drain the backlog, the API must answer again.
|
||||
deadline = time.time() + 20
|
||||
ok = None
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
ok = http_get(g.port, '/', timeout=3)
|
||||
break
|
||||
except Exception:
|
||||
time.sleep(0.5)
|
||||
self.assertIsNotNone(ok, '/health never recovered while idle conns were held')
|
||||
held = time.time()
|
||||
# The reaped sockets were closed by the server.
|
||||
closed = sum(1 for x in socks[:1500] if wait_closed_by_server(x, 0.01))
|
||||
self.assertGreater(closed, 1000, 'server did not close idle conns (%d)' % closed)
|
||||
for x in socks:
|
||||
x.close()
|
||||
socks.clear()
|
||||
time.sleep(1)
|
||||
for _ in range(5):
|
||||
self.assertTrue(http_get(g.port, '/', timeout=3).startswith('pid='))
|
||||
self.assertLess(held, time.time())
|
||||
|
||||
def test_f2_drip_connections_cut_and_health_responsive_throughout(self):
|
||||
# 8 connections (= every thread at production shape) each drip one
|
||||
# header byte per second -- far under the 20s per-recv timeout, so only
|
||||
# the total HEADER_DEADLINE can cut them.
|
||||
g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '3'}) # CLIENT_TIMEOUT stays 20
|
||||
stop = threading.Event()
|
||||
cut_after = []
|
||||
|
||||
def drip():
|
||||
s = socket.create_connection(('127.0.0.1', g.port))
|
||||
t0 = time.time()
|
||||
req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 1000
|
||||
try:
|
||||
for i in range(len(req)):
|
||||
if stop.is_set():
|
||||
return
|
||||
s.send(req[i:i + 1])
|
||||
time.sleep(1.0)
|
||||
s.settimeout(0.01)
|
||||
try:
|
||||
if s.recv(1) == b'':
|
||||
raise ConnectionResetError
|
||||
except socket.timeout:
|
||||
pass
|
||||
except OSError:
|
||||
cut_after.append(time.time() - t0)
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
drippers = [threading.Thread(target=drip, daemon=True) for _ in range(8)]
|
||||
for t in drippers:
|
||||
t.start()
|
||||
self.addCleanup(stop.set)
|
||||
time.sleep(0.5)
|
||||
worst = 0.0
|
||||
t_end = time.time() + 12
|
||||
while time.time() < t_end:
|
||||
t0 = time.time()
|
||||
self.assertTrue(http_get(g.port, '/', timeout=6).startswith('pid='))
|
||||
worst = max(worst, time.time() - t0)
|
||||
time.sleep(0.5)
|
||||
stop.set()
|
||||
self.assertEqual(len(cut_after), 8, 'not every drip connection was cut: %r' % cut_after)
|
||||
# The deadline is armed when a THREAD picks the connection up, not at
|
||||
# accept: a request queued behind busy threads (e.g. legit certbot
|
||||
# runs) must not be cut for waiting. So a drip conn that was queued
|
||||
# (the 2 workers rarely split 8 conns 4/4) is cut at up to about
|
||||
# 2 x deadline; one that got a thread at once within ~deadline + 1s.
|
||||
self.assertLessEqual(min(cut_after), 3 + 2, cut_after)
|
||||
self.assertTrue(all(c < 2 * 3 + 2 for c in cut_after), cut_after)
|
||||
self.assertLess(worst, 5.0)
|
||||
self.assertIn('HAPROXY_MGR_HEADER_DEADLINE', g.output())
|
||||
|
||||
def test_f2_slow_but_complete_headers_within_deadline_are_served(self):
|
||||
g = self.start({'HAPROXY_MGR_HEADER_DEADLINE': '4'})
|
||||
s = socket.create_connection(('127.0.0.1', g.port))
|
||||
self.addCleanup(s.close)
|
||||
for part in (b'GET / HTTP/1.1\r\n', b'Host: x\r\n', b'\r\n'):
|
||||
s.sendall(part)
|
||||
time.sleep(1)
|
||||
s.settimeout(5)
|
||||
self.assertIn(b'200 OK', s.recv(4096))
|
||||
|
||||
|
||||
def refused(port, wait=2.0):
|
||||
"""True if the server closes a fresh connection without answering a
|
||||
complete request (peer filter)."""
|
||||
s = socket.create_connection(('127.0.0.1', port), timeout=wait)
|
||||
try:
|
||||
try:
|
||||
s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
|
||||
except OSError:
|
||||
return True
|
||||
s.settimeout(wait)
|
||||
try:
|
||||
return s.recv(4096) == b''
|
||||
except (ConnectionResetError, BrokenPipeError):
|
||||
return True
|
||||
except socket.timeout:
|
||||
return False
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
class PeerFilter(unittest.TestCase):
|
||||
"""Re-review R1: tenant containers share client-net with haproxy-manager,
|
||||
so the API port must refuse peers that are not loopback / the host path."""
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
sys.path.insert(0, REPO_DIR)
|
||||
import haproxy_gunicorn_worker as w
|
||||
cls.w = w
|
||||
|
||||
def test_allowlist_parsing_and_matching(self):
|
||||
nets, problems = self.w.parse_allowed_peers('127.0.0.1,::1,gateway', gateways=['172.18.0.1'])
|
||||
self.assertEqual(problems, [])
|
||||
ok = self.w.peer_allowed
|
||||
self.assertTrue(ok('127.0.0.1', nets))
|
||||
self.assertTrue(ok('::1', nets))
|
||||
self.assertTrue(ok('::ffff:127.0.0.1', nets))
|
||||
self.assertTrue(ok('172.18.0.1', nets)) # docker-proxy / host
|
||||
self.assertFalse(ok('172.18.0.7', nets)) # a tenant container
|
||||
self.assertFalse(ok('127.0.0.2', nets))
|
||||
self.assertFalse(ok('', nets))
|
||||
self.assertFalse(ok('garbage', nets))
|
||||
|
||||
def test_bad_entries_fail_closed_and_are_reported(self):
|
||||
nets, problems = self.w.parse_allowed_peers('nonsense,gateway', gateways=[])
|
||||
self.assertEqual(nets, [])
|
||||
self.assertEqual(len(problems), 2, problems)
|
||||
nets, _ = self.w.parse_allowed_peers('10.0.0.0/8')
|
||||
self.assertTrue(self.w.peer_allowed('10.1.2.3', nets))
|
||||
|
||||
def test_default_gateway_parsed_from_route_table(self):
|
||||
fn = os.path.join(tempfile.mkdtemp(), 'route')
|
||||
with open(fn, 'w') as fh:
|
||||
fh.write('Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\n'
|
||||
'eth0\t00000000\t010012AC\t0003\t0\t0\t0\t00000000\n'
|
||||
'eth0\t000012AC\t00000000\t0001\t0\t0\t0\t0000FFFF\n')
|
||||
self.assertEqual(self.w._default_gateways(fn), ['172.18.0.1'])
|
||||
|
||||
def test_filtered_port_refuses_unlisted_peer_other_port_unaffected(self):
|
||||
other = free_port()
|
||||
g = Gunicorn(FIXED, threads=1, extra_ports=(other,), env={
|
||||
'HAPROXY_MGR_PEER_FILTER_PORTS': str(other),
|
||||
'HAPROXY_MGR_API_ALLOWED_PEERS': '10.255.255.1'})
|
||||
self.addCleanup(g.stop)
|
||||
# g.port (unfiltered) works; `other` refuses 127.0.0.1 (not listed).
|
||||
self.assertTrue(http_get(g.port, '/').startswith('pid='))
|
||||
for _ in range(5):
|
||||
self.assertTrue(refused(other), 'filtered port answered an unlisted peer')
|
||||
self.assertTrue(http_get(g.port, '/').startswith('pid='))
|
||||
out = g.output()
|
||||
self.assertIn('Refused 1 connection(s) to the management API', out) # first one, immediately
|
||||
self.assertEqual(out.count('Refused '), 1, 'reject logging is not rate-limited')
|
||||
|
||||
def test_filtered_port_serves_listed_peer(self):
|
||||
port = free_port()
|
||||
g = Gunicorn(FIXED, threads=1, extra_ports=(port,), env={
|
||||
'HAPROXY_MGR_PEER_FILTER_PORTS': str(port),
|
||||
'HAPROXY_MGR_API_ALLOWED_PEERS': '127.0.0.1,::1'})
|
||||
self.addCleanup(g.stop)
|
||||
self.assertTrue(http_get(port, '/').startswith('pid='))
|
||||
self.assertNotIn('Refused', g.output())
|
||||
|
||||
|
||||
class QueuedDripStarvation(unittest.TestCase):
|
||||
"""Re-review: with the deadline armed at thread pickup, 40 reconnecting
|
||||
drippers queued behind each other starved legit requests (every probe
|
||||
>30s). Armed as max(enqueue + D, pickup + 1s) they must not."""
|
||||
|
||||
def test_40_reconnecting_drippers_do_not_starve_legit_requests(self):
|
||||
g = Gunicorn(FIXED, threads=4, workers=2, env={}) # production defaults
|
||||
self.addCleanup(g.stop)
|
||||
stop = threading.Event()
|
||||
self.addCleanup(stop.set)
|
||||
|
||||
def drip():
|
||||
while not stop.is_set():
|
||||
try:
|
||||
s = socket.create_connection(('127.0.0.1', g.port))
|
||||
except OSError:
|
||||
time.sleep(0.2)
|
||||
continue
|
||||
try:
|
||||
req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 5000
|
||||
for i in range(len(req)):
|
||||
if stop.is_set():
|
||||
break
|
||||
s.send(req[i:i + 1])
|
||||
time.sleep(1.0)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
for _ in range(40):
|
||||
threading.Thread(target=drip, daemon=True).start()
|
||||
time.sleep(2)
|
||||
lat = []
|
||||
t_end = time.time() + 25
|
||||
while time.time() < t_end:
|
||||
t0 = time.time()
|
||||
try:
|
||||
http_get(g.port, '/', timeout=30)
|
||||
lat.append(time.time() - t0)
|
||||
except Exception:
|
||||
lat.append(99.0)
|
||||
time.sleep(1)
|
||||
stop.set()
|
||||
lat.sort()
|
||||
median = lat[len(lat) // 2]
|
||||
sys.stderr.write('\n 40 drippers: probes=%d median=%.2fs worst=%.2fs\n'
|
||||
% (len(lat), median, lat[-1]))
|
||||
# Armed at pickup, every probe here exceeded 30s (total starvation).
|
||||
# With max(enqueue + D, pickup + 1s) queued drippers get ~1s of thread
|
||||
# time, so legit requests keep getting through. They still queue: the
|
||||
# equilibrium wait is ~(drippers/threads) x hold time, about 9s at 40
|
||||
# drippers on 8 threads -- measured, see the round-2 report. The
|
||||
# remaining fix for that is not admitting partial requests to a thread
|
||||
# at all; the peer filter makes it a local-only scenario.
|
||||
# The property under test is "no starvation" (every probe answered
|
||||
# inside 30s). The latency bounds depend on host load: a shared CI
|
||||
# runner measured median 14.97s on 2 probes, so they are only enforced
|
||||
# when HAPROXY_MGR_STRICT_TIMING=1 (quiet local box).
|
||||
self.assertNotIn(99.0, lat, 'a legit request was starved for 30s+')
|
||||
if os.environ.get('HAPROXY_MGR_STRICT_TIMING') == '1':
|
||||
self.assertLess(median, 12.0, lat)
|
||||
self.assertLess(lat[-1], 20.0, lat)
|
||||
|
||||
|
||||
class StalledUpstreamDefaultTimeout(unittest.TestCase):
|
||||
"""haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
|
||||
client call to an upstream that accepts and never replies must time out."""
|
||||
|
||||
def test_in_process_client_to_stalled_upstream_times_out(self):
|
||||
srv = socket.socket()
|
||||
srv.bind(('127.0.0.1', 0))
|
||||
srv.listen(5)
|
||||
self.addCleanup(srv.close)
|
||||
port = srv.getsockname()[1]
|
||||
accepted = []
|
||||
threading.Thread(target=lambda: accepted.append(srv.accept()),
|
||||
daemon=True).start()
|
||||
# Import the real module in a child (it changes process-global socket
|
||||
# state and opens its log files at import), then make a client call
|
||||
# with NO explicit timeout -- what a careless future handler would do.
|
||||
probe = textwrap.dedent('''\
|
||||
import logging, os, sys, tempfile, time, urllib.request
|
||||
d = tempfile.mkdtemp()
|
||||
real = logging.FileHandler
|
||||
logging.FileHandler = lambda fn, *a, **k: real(os.path.join(d, os.path.basename(fn)), *a, **k)
|
||||
os.chdir(sys.argv[1]); sys.path.insert(0, sys.argv[1])
|
||||
import haproxy_manager
|
||||
logging.FileHandler = real
|
||||
import socket
|
||||
print('default', socket.getdefaulttimeout())
|
||||
t0 = time.time()
|
||||
try:
|
||||
urllib.request.urlopen('http://127.0.0.1:%s/' % sys.argv[2]).read()
|
||||
print('result returned')
|
||||
except Exception as e:
|
||||
print('result', type(e).__name__, round(time.time() - t0, 1))
|
||||
''')
|
||||
env = dict(os.environ, HAPROXY_MGR_SOCKET_DEFAULT_TIMEOUT='1.5')
|
||||
r = subprocess.run([sys.executable, '-c', probe, REPO_DIR, str(port)],
|
||||
capture_output=True, text=True, timeout=30, env=env)
|
||||
if 'No module named' in r.stderr and 'flask' in r.stderr.lower():
|
||||
self.skipTest('Flask not installed')
|
||||
self.assertIn('default 1.5', r.stdout, r.stderr)
|
||||
line = [l for l in r.stdout.splitlines() if l.startswith('result')][0]
|
||||
self.assertTrue(line.startswith('result TimeoutError '), line)
|
||||
elapsed = float(line.split()[-1])
|
||||
self.assertLess(elapsed, 5)
|
||||
self.assertTrue(accepted, 'upstream never saw the connection')
|
||||
accepted[0][0].close()
|
||||
|
||||
def test_default_timeout_alone_does_not_cover_gthread_client_sockets(self):
|
||||
# Documents why the custom worker exists: gthread's setblocking(True)
|
||||
# resets any timeout -- including the process default -- to None.
|
||||
old = socket.getdefaulttimeout()
|
||||
socket.setdefaulttimeout(60)
|
||||
try:
|
||||
a, b = socket.socketpair()
|
||||
finally:
|
||||
socket.setdefaulttimeout(old)
|
||||
self.addCleanup(a.close)
|
||||
self.addCleanup(b.close)
|
||||
self.assertEqual(a.gettimeout(), 60)
|
||||
a.setblocking(True)
|
||||
self.assertIsNone(a.gettimeout())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
@@ -1,397 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Contract test: the runtime-map fast path (blocked IPs).
|
||||
|
||||
Why this file exists
|
||||
--------------------
|
||||
`add_ip_to_runtime_map()` and `remove_ip_from_runtime_map()` spent their whole
|
||||
existence sending
|
||||
|
||||
add map #0 <ip> 1
|
||||
del map #0 <ip>
|
||||
|
||||
to `/tmp/haproxy-cli` and returning True whenever socat exited 0. Neither
|
||||
command has ever worked. Two independent defects:
|
||||
|
||||
* **No `@1` prefix.** `/tmp/haproxy-cli` is HAProxy's MASTER CLI socket; map
|
||||
commands are worker commands. The master answers `Unknown command: 'add',
|
||||
but maybe one of the following ones is a better match: ...` -- and **socat
|
||||
still exits 0**, so `result.returncode == 0` was true and the function
|
||||
logged "Added IP x to runtime map".
|
||||
* **`#0` is not a valid map id.** Ids are assigned at config-parse time and
|
||||
move on every config regeneration; on the live edge `blocked_ips.map` is
|
||||
id 37 and `trusted_ips.map` is 10. There is no id 0. Any hardcoded number
|
||||
is wrong -- the map must be referenced by its FILE PATH, which is what
|
||||
haproxy.cfg itself names in `map_ip(/etc/haproxy/blocked_ips.map,0)`.
|
||||
|
||||
And a third silence that makes a response-body check alone insufficient:
|
||||
`@1 add map #0 <ip> 1` returns an **empty body**, exit 0, and adds nothing
|
||||
anywhere -- while `@1 del map #0 <ip>` answers `Unknown map identifier.`. The
|
||||
add path can therefore only be trusted after reading the entry back.
|
||||
|
||||
IP blocking still worked, because `update_blocked_ips_map()` rewrites
|
||||
`/etc/haproxy/blocked_ips.map` and the callers reload HAProxy, which re-reads
|
||||
it. The FILE is authoritative; this is the no-reload fast path, and it has
|
||||
never once run while reporting that it did.
|
||||
|
||||
What it enforces
|
||||
----------------
|
||||
1. The command strings actually sent: `@1` prefix first, map referenced by
|
||||
PATH and never by `#<id>`, and the value `1` that
|
||||
`map_ip(...,0) -m int gt 0` requires.
|
||||
2. Every captured rejection is classified as FAILURE (returns False), not
|
||||
success -- including the two that carry no error text at all.
|
||||
3. Success is only reported when the entry reads back in the state asked
|
||||
for. Not the exit status, not an empty reply.
|
||||
4. No source in this repo builds a map command with a `#<id>` reference.
|
||||
Comments may describe the old form; code may not use it.
|
||||
|
||||
Runs fully offline: `_cli_send()` is replaced, so no socket, no socat, no
|
||||
HAProxy, no network.
|
||||
|
||||
Running
|
||||
-------
|
||||
python3 scripts/test-runtime-map-contract.py
|
||||
"""
|
||||
|
||||
import ast
|
||||
import io
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import glob
|
||||
import logging
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
MODULE_DIR = os.path.abspath(
|
||||
os.environ.get('HAPROXY_MANAGER_DIR',
|
||||
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
|
||||
)
|
||||
os.chdir(MODULE_DIR)
|
||||
sys.path.insert(0, MODULE_DIR)
|
||||
|
||||
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
|
||||
_real_file_handler = logging.FileHandler
|
||||
logging.FileHandler = (
|
||||
lambda filename, *a, **kw: _real_file_handler(
|
||||
os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
|
||||
)
|
||||
|
||||
import haproxy_manager # noqa: E402
|
||||
|
||||
logging.getLogger().setLevel(logging.CRITICAL)
|
||||
|
||||
MAP = haproxy_manager.BLOCKED_IPS_MAP_PATH
|
||||
IP = '192.0.2.77'
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Responses captured VERBATIM from the haproxy-manager container on the live
|
||||
# edge (HAProxy 3.0.11, 2026-08-22). socat exited 0 for every single one of
|
||||
# them, which is the entire reason none of this is decided on exit status.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# What the MASTER socket answers to an unprefixed `add map ...` -- i.e. the
|
||||
# reply the old code read as success.
|
||||
MASTER_REJECTS_ADD = """\
|
||||
Unknown command: 'add', but maybe one of the following ones is a better match:
|
||||
@!<pid> : send a command to the <pid> process
|
||||
@master : send a command to the master process
|
||||
hard-reload : achieve a hard-reload (-st) of haproxy
|
||||
reload : achieve a soft-reload (-sf) of haproxy
|
||||
user : lower the level of the current CLI session to user
|
||||
help [<command>] : list matching or all commands
|
||||
prompt [timed] : toggle interactive mode with prompt
|
||||
quit : disconnect
|
||||
"""
|
||||
|
||||
MASTER_REJECTS_DEL = MASTER_REJECTS_ADD.replace("'add'", "'del'")
|
||||
|
||||
# `@1 del map #0 <ip>` / `@1 get map /etc/haproxy/nope.map <ip>`.
|
||||
UNKNOWN_MAP_IDENTIFIER = 'Unknown map identifier. Please use #<id> or <file>.\n'
|
||||
|
||||
# `@1 add map <map> <ip>` with the value omitted (as the old docs showed).
|
||||
ADD_MAP_MISSING_VALUE = (
|
||||
"'add map' expects three parameters (map identifier, key and value) or one "
|
||||
"parameter (map identifier) and a payload\n")
|
||||
|
||||
# `@1 del map <map> <ip>` for a key the runtime map does not hold.
|
||||
KEY_NOT_FOUND = 'Key not found.\n'
|
||||
|
||||
# A successful mutation. THIS IS THE WHOLE PROBLEM: it is byte-for-byte what
|
||||
# `@1 add map #0 <ip> 1` also returns while adding nothing at all.
|
||||
MUTATION_OK = ''
|
||||
|
||||
GET_FOUND = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
|
||||
'value="1", type="str"\n' % IP)
|
||||
GET_NOT_FOUND = 'type=ip, case=sensitive, found=no\n'
|
||||
# `@1 get map #0 <ip>` -- "found", but with no value. haproxy.cfg matches with
|
||||
# `-m int gt 0`, so an entry like this does NOT block.
|
||||
GET_FOUND_NO_VALUE = ('type=ip, case=sensitive, found=yes, idx=tree, key="%s", '
|
||||
'value=none\n' % IP)
|
||||
|
||||
SHOW_MAP_SAMPLE = (
|
||||
'0x7f6e5a788700 101.36.109.130 1\n'
|
||||
'0x7f6e5a788780 101.47.140.218 1\n'
|
||||
'0x7f6e5a1fd500 %s 1\n' % IP)
|
||||
|
||||
|
||||
class FakeSocket(object):
|
||||
"""Replaces _cli_send(). Records every command, answers from a script.
|
||||
|
||||
`script` is a list of (substring, response) pairs, consulted in order; the
|
||||
first whose substring appears in the command wins. Anything unmatched is
|
||||
an explicit test bug, not a silent default.
|
||||
"""
|
||||
|
||||
def __init__(self, script):
|
||||
self.script = script
|
||||
self.sent = []
|
||||
|
||||
def __call__(self, command, socket_path, timeout):
|
||||
self.sent.append(command)
|
||||
for needle, response in self.script:
|
||||
if needle in command:
|
||||
return response
|
||||
raise AssertionError('test script has no response for %r' % command)
|
||||
|
||||
|
||||
def with_socket(script):
|
||||
"""Install a FakeSocket for the duration of a `with` block."""
|
||||
class _Ctx(object):
|
||||
def __enter__(self):
|
||||
self.fake = FakeSocket(script)
|
||||
self._real = haproxy_manager._cli_send
|
||||
haproxy_manager._cli_send = self.fake
|
||||
return self.fake
|
||||
|
||||
def __exit__(self, *exc):
|
||||
haproxy_manager._cli_send = self._real
|
||||
return False
|
||||
return _Ctx()
|
||||
|
||||
|
||||
# Every command the happy path needs, in the order the code issues them.
|
||||
HAPPY_ADD = [('get map', GET_FOUND), ('add map', MUTATION_OK)]
|
||||
HAPPY_DEL = [('get map', GET_NOT_FOUND), ('del map', MUTATION_OK)]
|
||||
|
||||
|
||||
class CommandsAreWellFormed(unittest.TestCase):
|
||||
"""Guard 1: the exact bytes on the wire.
|
||||
|
||||
Both original defects are visible here and nowhere else -- a `#0` map
|
||||
reference and a missing `@1` are perfectly ordinary-looking Python.
|
||||
"""
|
||||
|
||||
def test_add_sends_worker_prefixed_path_referenced_command(self):
|
||||
with with_socket(HAPPY_ADD) as fake:
|
||||
self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
self.assertEqual(fake.sent[0], '@1 add map %s %s 1' % (MAP, IP))
|
||||
|
||||
def test_del_sends_worker_prefixed_path_referenced_command(self):
|
||||
with with_socket(HAPPY_DEL) as fake:
|
||||
self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
|
||||
self.assertEqual(fake.sent[0], '@1 del map %s %s' % (MAP, IP))
|
||||
|
||||
def test_every_command_is_tried_on_the_worker_first(self):
|
||||
"""`/tmp/haproxy-cli` is the MASTER socket; bare map commands 404."""
|
||||
for run in (lambda: haproxy_manager.add_ip_to_runtime_map(IP),
|
||||
lambda: haproxy_manager.remove_ip_from_runtime_map(IP)):
|
||||
with with_socket(HAPPY_ADD + HAPPY_DEL) as fake:
|
||||
run()
|
||||
for command in fake.sent:
|
||||
self.assertTrue(command.startswith('@1 '),
|
||||
'%r is missing the @1 worker prefix' % command)
|
||||
|
||||
def test_no_command_references_a_map_by_id(self):
|
||||
"""Map ids move on every config regeneration. Path, always."""
|
||||
script = HAPPY_ADD + HAPPY_DEL + [('show map', SHOW_MAP_SAMPLE)]
|
||||
with with_socket(script) as fake:
|
||||
haproxy_manager.add_ip_to_runtime_map(IP)
|
||||
haproxy_manager.remove_ip_from_runtime_map(IP)
|
||||
haproxy_manager.runtime_map_keys(MAP)
|
||||
for command in fake.sent:
|
||||
self.assertNotRegex(
|
||||
command, r'\bmap\s+#',
|
||||
'%r references a map by id; ids are not stable' % command)
|
||||
self.assertIn(MAP, command,
|
||||
'%r does not name the map file' % command)
|
||||
|
||||
def test_add_carries_the_value_the_config_matches_on(self):
|
||||
"""`map_ip(...,0) -m int gt 0`: a valueless entry does not block."""
|
||||
self.assertEqual(haproxy_manager.BLOCKED_IPS_MAP_VALUE, '1')
|
||||
with with_socket(HAPPY_ADD) as fake:
|
||||
haproxy_manager.add_ip_to_runtime_map(IP)
|
||||
self.assertTrue(fake.sent[0].endswith(' %s 1' % IP),
|
||||
'%r has no value; HAProxy rejects it' % fake.sent[0])
|
||||
|
||||
|
||||
class RejectionIsFailure(unittest.TestCase):
|
||||
"""Guard 2+3: nothing may report success unless the map really changed.
|
||||
|
||||
Every response below was returned by the live socket with **exit code 0**.
|
||||
The old code returned True for all of them.
|
||||
"""
|
||||
|
||||
def test_master_socket_rejection_of_add_is_failure(self):
|
||||
with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
|
||||
self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
|
||||
def test_master_socket_rejection_of_del_is_failure(self):
|
||||
with with_socket([('get map', GET_FOUND), ('del map', MASTER_REJECTS_DEL)]):
|
||||
self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
|
||||
|
||||
def test_unknown_map_identifier_is_failure(self):
|
||||
with with_socket([('get map', GET_NOT_FOUND), ('add map', UNKNOWN_MAP_IDENTIFIER)]):
|
||||
self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
with with_socket([('get map', GET_FOUND), ('del map', UNKNOWN_MAP_IDENTIFIER)]):
|
||||
self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
|
||||
|
||||
def test_missing_value_rejection_is_failure(self):
|
||||
"""Carries no marker word at all -- caught by 'a mutation says nothing'."""
|
||||
with with_socket([('get map', GET_NOT_FOUND), ('add map', ADD_MAP_MISSING_VALUE)]):
|
||||
self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
|
||||
def test_silent_noop_add_is_failure(self):
|
||||
"""The `#0` failure mode: accepted, empty reply, nothing added.
|
||||
|
||||
Nothing in the response distinguishes this from success. Only the
|
||||
read-back does -- which is why the read-back is not optional.
|
||||
"""
|
||||
with with_socket([('get map', GET_NOT_FOUND), ('add map', MUTATION_OK)]):
|
||||
self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
|
||||
def test_add_that_lands_without_a_value_is_failure(self):
|
||||
with with_socket([('get map', GET_FOUND_NO_VALUE), ('add map', MUTATION_OK)]):
|
||||
self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
|
||||
def test_del_that_leaves_the_key_behind_is_failure(self):
|
||||
with with_socket([('get map', GET_FOUND), ('del map', MUTATION_OK)]):
|
||||
self.assertFalse(haproxy_manager.remove_ip_from_runtime_map(IP))
|
||||
|
||||
def test_key_not_found_on_del_is_the_requested_end_state(self):
|
||||
"""Not a failure: the runtime map already lacks the key."""
|
||||
with with_socket([('get map', GET_NOT_FOUND), ('del map', KEY_NOT_FOUND)]):
|
||||
self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
|
||||
|
||||
def test_verified_success_is_reported_as_success(self):
|
||||
with with_socket(HAPPY_ADD):
|
||||
self.assertTrue(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
with with_socket(HAPPY_DEL):
|
||||
self.assertTrue(haproxy_manager.remove_ip_from_runtime_map(IP))
|
||||
|
||||
def test_a_runtime_failure_never_raises_into_the_request_handler(self):
|
||||
"""The map file + reload still enforces the block; degrade, don't 500."""
|
||||
with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]):
|
||||
self.assertIs(haproxy_manager.add_ip_to_runtime_map(IP), False)
|
||||
|
||||
def test_mutations_are_checked_for_an_empty_body_not_a_marker_list(self):
|
||||
"""_HAPROXY_CLI_ERROR_MARKERS can only know rejections already seen."""
|
||||
with self.assertRaises(haproxy_manager.HaproxyCliError):
|
||||
with with_socket([('add map', 'something nobody has ever seen\n')]):
|
||||
haproxy_manager.haproxy_cli('add map %s x 1' % MAP,
|
||||
worker=True, expect_empty=True)
|
||||
|
||||
def test_the_new_error_markers_are_recognised(self):
|
||||
for response in (UNKNOWN_MAP_IDENTIFIER, KEY_NOT_FOUND,
|
||||
MASTER_REJECTS_ADD):
|
||||
self.assertTrue(haproxy_manager._cli_response_is_error(response),
|
||||
'%r must be classified as an error' % response[:40])
|
||||
for response in (GET_FOUND, GET_NOT_FOUND, SHOW_MAP_SAMPLE):
|
||||
self.assertFalse(haproxy_manager._cli_response_is_error(response),
|
||||
'%r is data, not an error' % response[:40])
|
||||
|
||||
def test_socat_exit_zero_carries_no_information(self):
|
||||
"""FakeSocket never signals failure any other way, and neither did socat."""
|
||||
with with_socket([('get map', GET_NOT_FOUND), ('add map', MASTER_REJECTS_ADD)]) as fake:
|
||||
self.assertFalse(haproxy_manager.add_ip_to_runtime_map(IP))
|
||||
self.assertTrue(fake.sent, 'the command was sent and "succeeded" at the '
|
||||
'process level; only the body says otherwise')
|
||||
|
||||
|
||||
class ReadBack(unittest.TestCase):
|
||||
"""The read-back primitives the guarantees above rest on."""
|
||||
|
||||
def test_lookup_reports_found_with_value(self):
|
||||
with with_socket([('get map', GET_FOUND)]):
|
||||
self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
|
||||
(True, '1'))
|
||||
|
||||
def test_lookup_reports_not_found(self):
|
||||
with with_socket([('get map', GET_NOT_FOUND)]):
|
||||
self.assertEqual(haproxy_manager.runtime_map_lookup(MAP, IP),
|
||||
(False, None))
|
||||
|
||||
def test_lookup_raises_on_a_rejected_reference(self):
|
||||
with with_socket([('get map', UNKNOWN_MAP_IDENTIFIER)]):
|
||||
with self.assertRaises(haproxy_manager.HaproxyCliError):
|
||||
haproxy_manager.runtime_map_lookup(MAP, IP)
|
||||
|
||||
def test_keys_parses_show_map_output(self):
|
||||
with with_socket([('show map', SHOW_MAP_SAMPLE)]):
|
||||
self.assertEqual(
|
||||
haproxy_manager.runtime_map_keys(MAP),
|
||||
{'101.36.109.130', '101.47.140.218', IP})
|
||||
|
||||
def test_empty_map_is_not_a_rejection(self):
|
||||
with with_socket([('show map', '')]):
|
||||
self.assertEqual(haproxy_manager.runtime_map_keys(MAP), set())
|
||||
|
||||
def test_rejected_show_map_still_raises(self):
|
||||
with with_socket([('show map', UNKNOWN_MAP_IDENTIFIER)]):
|
||||
with self.assertRaises(haproxy_manager.HaproxyCliError):
|
||||
haproxy_manager.runtime_map_keys(MAP)
|
||||
|
||||
|
||||
MAP_BY_ID_RE = re.compile(r'\b(?:add|del|clear|show|get)\s+map\s+#')
|
||||
|
||||
|
||||
class NoSourceBuildsAMapIdCommand(unittest.TestCase):
|
||||
"""Guard 4: `map #<id>` may be described in comments, never executed.
|
||||
|
||||
Scanning string literals rather than raw text is deliberate -- the whole
|
||||
reason this bug is documented at length in the source is so the next reader
|
||||
does not reintroduce it, and a plain grep would fail on those comments.
|
||||
"""
|
||||
|
||||
def test_no_python_string_literal_builds_a_map_id_command(self):
|
||||
tree = ast.parse(io.open('haproxy_manager.py', encoding='utf-8').read())
|
||||
offenders = [
|
||||
node.value for node in ast.walk(tree)
|
||||
if isinstance(node, ast.Constant) and isinstance(node.value, str)
|
||||
and MAP_BY_ID_RE.search(node.value)
|
||||
and not (ast.get_docstring(tree) == node.value)
|
||||
]
|
||||
# Docstrings are string literals too; exclude any literal that is a
|
||||
# docstring of a module/class/function.
|
||||
docstrings = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef,
|
||||
ast.AsyncFunctionDef)):
|
||||
doc = ast.get_docstring(node, clean=False)
|
||||
if doc:
|
||||
docstrings.add(doc)
|
||||
offenders = [o for o in offenders if o not in docstrings]
|
||||
self.assertEqual(offenders, [],
|
||||
'these string literals build a map command with an '
|
||||
'unstable #<id> reference')
|
||||
|
||||
def test_no_shell_or_template_code_line_uses_a_map_id(self):
|
||||
targets = (glob.glob('scripts/*.sh') + glob.glob('templates/*.tpl')
|
||||
+ ['Dockerfile'])
|
||||
offenders = []
|
||||
for path in targets:
|
||||
if not os.path.exists(path):
|
||||
continue
|
||||
for lineno, line in enumerate(
|
||||
io.open(path, encoding='utf-8').read().splitlines(), 1):
|
||||
if line.lstrip().startswith('#'):
|
||||
continue # a comment describing the old form is fine
|
||||
if MAP_BY_ID_RE.search(line):
|
||||
offenders.append('%s:%d: %s' % (path, lineno, line.strip()))
|
||||
self.assertEqual(offenders, [],
|
||||
'map ids are assigned at config-parse time and move; '
|
||||
'reference the map file by path')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
@@ -1,382 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regression tests for WAF request-body inspection (Coraza via SPOE).
|
||||
|
||||
Why this file exists
|
||||
--------------------
|
||||
Measured 2026-10-09 on haproxy 3.0.11 + coraza-spoa v0.7.1 + CRS 4.25.0
|
||||
(report: waf-body-hardening-report.md):
|
||||
|
||||
* Without `option http-buffer-request`, send-spoe-group fires at header
|
||||
time and `req.body` holds only what happened to arrive with the headers.
|
||||
A 14-byte `content=$a($b)` POST (CRS 933180, an ENFORCED rule) passed
|
||||
uninspected when its body was written in a second send(); a body sent 2s
|
||||
late, or after `Expect: 100-continue`, was never inspected at all.
|
||||
* Coraza+CRS costs ~3.3 ms per KB of body. The SPOE `timeout processing`
|
||||
is 100 ms and FAILS OPEN, so raising tune.bufsize to 32 KB turned a
|
||||
payload at byte 0 of a 30 KB base64 body into a bypass. bufsize is a
|
||||
security limit, not a performance knob.
|
||||
* coraza-spoa's SPOP library refuses frames > 65535 bytes, and Coraza's
|
||||
SecRequestBodyLimitAction Reject 413s even under DetectionOnly, so the
|
||||
three limits must stay ordered.
|
||||
* The X-WHP-WAF-Body stamp drives enforcement of body-parse errors
|
||||
(200002/200003) in WHP's generated Coraza rules. A client must never be
|
||||
able to set it, and a backend must never see it.
|
||||
|
||||
These tests pin the rendered config. That HAProxy ACCEPTS it is
|
||||
validate-rendered-config.py's job; that it BEHAVES is the live measurement.
|
||||
|
||||
Running
|
||||
-------
|
||||
python3 scripts/test-waf-body-inspection.py
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
MODULE_DIR = os.path.abspath(
|
||||
os.environ.get('HAPROXY_MANAGER_DIR',
|
||||
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..'))
|
||||
)
|
||||
os.chdir(MODULE_DIR)
|
||||
sys.path.insert(0, MODULE_DIR)
|
||||
|
||||
_LOG_DIR = tempfile.mkdtemp(prefix='haproxy-mgr-test-logs-')
|
||||
_real_file_handler = logging.FileHandler
|
||||
logging.FileHandler = (
|
||||
lambda filename, *a, **kw: _real_file_handler(
|
||||
os.path.join(_LOG_DIR, os.path.basename(filename)), *a, **kw)
|
||||
)
|
||||
|
||||
import haproxy_manager # noqa: E402
|
||||
|
||||
SPOA_FRAME_CAP = 65535 # dropmorepackets/haproxy-go v0.0.8 maxFrameSize
|
||||
CORAZA_CONFIG = os.path.join(MODULE_DIR, 'coraza-spoa', 'config.yaml')
|
||||
|
||||
|
||||
def render(template, coraza):
|
||||
kw = dict(crt_path='/etc/haproxy/certs', suspension_enabled=False,
|
||||
coraza_spoe_backend='coraza-spoa:9000' if coraza else None,
|
||||
cluster_secret=None, syslog_target='172.18.0.1:514')
|
||||
return haproxy_manager.template_env.get_template(template).render(**kw)
|
||||
|
||||
|
||||
def active_lines(text):
|
||||
"""Config lines with comments and blanks stripped (comments mention the
|
||||
directives by name, so a raw substring check would pass on prose)."""
|
||||
out = []
|
||||
for line in text.splitlines():
|
||||
s = line.strip()
|
||||
if s and not s.startswith('#'):
|
||||
out.append(s)
|
||||
return out
|
||||
|
||||
|
||||
def index_of(lines, pattern):
|
||||
for i, line in enumerate(lines):
|
||||
if re.search(pattern, line):
|
||||
return i
|
||||
return -1
|
||||
|
||||
|
||||
class ListenerWithCoraza(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
self.lines = active_lines(render('hap_listener.tpl', coraza=True))
|
||||
|
||||
def test_body_is_buffered_before_inspection(self):
|
||||
self.assertIn('option http-buffer-request', self.lines)
|
||||
|
||||
def test_stamp_order_strip_set_inspect_strip(self):
|
||||
strip1 = index_of(self.lines, r'^http-request del-header X-WHP-WAF-Body$')
|
||||
size = index_of(self.lines, r'^http-request set-var\(txn\.waf_body_size\) req\.body_size if \{ req\.hdr_cnt\(content-length\) eq 1 \}$')
|
||||
complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
|
||||
partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if \{ var\(txn\.waf_body_size\) -m found \} !\{ req\.body_len,sub\(txn\.waf_body_size\) eq 0 \}$')
|
||||
hdrs_len = index_of(self.lines, r'^http-request set-var\(txn\.waf_hdrs_len\) req\.hdrs,length if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \}$')
|
||||
nocl_complete = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(complete\) if \{ var\(txn\.waf_hdrs_len\) -m found \} \{ req\.body_len,add\(txn\.waf_hdrs_len\) lt (\d+) \}$')
|
||||
nocl_partial = index_of(self.lines, r'^http-request set-var\(txn\.waf_body\) str\(partial\) if !\{ var\(txn\.waf_body_size\) -m found \} \{ req\.body_len gt 0 \} !\{ var\(txn\.waf_body\) -m found \}$')
|
||||
stamp = index_of(self.lines, r'^http-request set-header X-WHP-WAF-Body %\[var\(txn\.waf_body\)\] if \{ var\(txn\.waf_body\) -m found \}$')
|
||||
spoe = index_of(self.lines, r'^http-request send-spoe-group coraza coraza-req$')
|
||||
deny = index_of(self.lines, r'^http-request return status 403 .*var\(txn\.coraza\.action\) -m str deny')
|
||||
dup_ct = index_of(self.lines, r'^http-request deny deny_status 400 if \{ req\.fhdr_cnt\(content-type\) gt 1 \}$')
|
||||
self.assertGreaterEqual(dup_ct, 0, 'duplicate Content-Type must be rejected at the edge')
|
||||
self.assertLess(dup_ct, spoe, 'and before Coraza is asked')
|
||||
for name, idx in (('client-copy strip', strip1), ('size var', size), ('complete', complete),
|
||||
('partial', partial), ('no-CL header length', hdrs_len),
|
||||
('no-CL complete', nocl_complete),
|
||||
('no-CL partial', nocl_partial), ('stamp', stamp),
|
||||
('send-spoe-group', spoe), ('deny', deny)):
|
||||
self.assertGreaterEqual(idx, 0, f'{name} rule missing from the rendered listener')
|
||||
# Client copies stripped BEFORE we stamp, stamp BEFORE the SPOE call.
|
||||
self.assertLess(strip1, size)
|
||||
self.assertLess(size, complete)
|
||||
# The no-CL `partial` fallback relies on "-m found" of an earlier
|
||||
# `complete`: order is load-bearing.
|
||||
self.assertLess(size, hdrs_len)
|
||||
self.assertLess(hdrs_len, nocl_complete)
|
||||
self.assertLess(nocl_complete, nocl_partial)
|
||||
self.assertLess(max(complete, partial, nocl_partial), stamp)
|
||||
self.assertLess(stamp, spoe)
|
||||
# Stripped again AFTER the SPOE call so no backend ever sees it.
|
||||
strip2 = [i for i, l in enumerate(self.lines)
|
||||
if l == 'http-request del-header X-WHP-WAF-Body' and i > spoe]
|
||||
self.assertTrue(strip2, 'X-WHP-WAF-Body must be deleted again after send-spoe-group')
|
||||
self.assertLess(strip2[0], deny)
|
||||
|
||||
def test_no_content_length_complete_cannot_have_filled_the_buffer(self):
|
||||
"""Without a Content-Length, `complete` is inferred from "the buffer
|
||||
cannot have been full". body + headers thresholds must sit well below
|
||||
bufsize - maxrewrite (1024), or a truncated chunked body would be
|
||||
stamped complete and a parse error on it would 403."""
|
||||
line = [l for l in self.lines if 'str(complete) if { var(txn.waf_hdrs_len)' in l][0]
|
||||
total_lt = int(re.search(r'add\(txn\.waf_hdrs_len\) lt (\d+)', line).group(1))
|
||||
header = active_lines(render('hap_header.tpl', coraza=True))
|
||||
bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
|
||||
self.assertLessEqual(total_lt, (bufsize - 1024) * 0.85)
|
||||
|
||||
def test_complete_and_partial_are_mutually_exclusive_conditions(self):
|
||||
"""With a Content-Length, `complete` and `partial` must be the same
|
||||
condition, negated -- an inverted or loosened `complete` would enforce
|
||||
parse errors on truncated bodies (403 on large legitimate posts)."""
|
||||
c = [l for l in self.lines if 'str(complete) if { var(txn.waf_body_size)' in l]
|
||||
p = [l for l in self.lines if 'str(partial) if { var(txn.waf_body_size)' in l]
|
||||
self.assertEqual((len(c), len(p)), (1, 1))
|
||||
c, p = c[0], p[0]
|
||||
self.assertEqual(c.split(' if ', 1)[1].replace('!{', '{'),
|
||||
p.split(' if ', 1)[1].replace('!{', '{'))
|
||||
self.assertIn('!{ req.body_len', p)
|
||||
self.assertNotIn('!{', c)
|
||||
|
||||
def test_stamp_never_set_from_client_input(self):
|
||||
"""The only set-header for the stamp is from our own var."""
|
||||
sets = [l for l in self.lines if re.search(r'set-header\s+X-WHP-WAF-Body', l, re.I)]
|
||||
self.assertEqual(len(sets), 1)
|
||||
self.assertIn('%[var(txn.waf_body)]', sets[0])
|
||||
self.assertFalse([l for l in self.lines if re.search(r'add-header\s+X-WHP-WAF-Body', l, re.I)])
|
||||
|
||||
def test_access_log_carries_coverage_fields(self):
|
||||
lf = [l for l in self.lines if l.startswith('log-format ')]
|
||||
self.assertEqual(len(lf), 1)
|
||||
self.assertIn('wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)]', lf[0])
|
||||
|
||||
def test_access_log_carries_waf_verdict_fields(self):
|
||||
"""wrule/wrules/wscore are appended LAST so any consumer anchored on
|
||||
the earlier fields keeps working."""
|
||||
lf = [l for l in self.lines if l.startswith('log-format ')]
|
||||
self.assertEqual(len(lf), 1)
|
||||
self.assertTrue(lf[0].endswith(
|
||||
'wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] '
|
||||
'wscore=%[var(txn.coraza.anomaly_score)]"'))
|
||||
|
||||
def test_fail_open_preserved(self):
|
||||
"""Nothing may deny on var(txn.coraza.error): an SPOA outage must not
|
||||
503 customer sites."""
|
||||
self.assertFalse([l for l in self.lines if 'txn.coraza.error' in l and
|
||||
re.search(r'\b(deny|reject|return|silent-drop|tarpit)\b', l)])
|
||||
|
||||
|
||||
class ListenerWithoutCoraza(unittest.TestCase):
|
||||
|
||||
def test_waf_off_config_has_no_body_buffering(self):
|
||||
lines = active_lines(render('hap_listener.tpl', coraza=False))
|
||||
self.assertNotIn('option http-buffer-request', lines)
|
||||
self.assertFalse([l for l in lines if 'X-WHP-WAF-Body' in l])
|
||||
self.assertFalse([l for l in lines if 'send-spoe-group' in l])
|
||||
|
||||
def test_waf_off_header_keeps_default_bufsize(self):
|
||||
lines = active_lines(render('hap_header.tpl', coraza=False))
|
||||
self.assertFalse([l for l in lines if l.startswith('tune.bufsize')])
|
||||
|
||||
|
||||
class BufsizeCeilings(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
lines = active_lines(render('hap_header.tpl', coraza=True))
|
||||
found = [l for l in lines if l.startswith('tune.bufsize')]
|
||||
self.assertEqual(len(found), 1, 'tune.bufsize must be pinned when Coraza is on')
|
||||
self.bufsize = int(found[0].split()[1])
|
||||
spoe = active_lines(haproxy_manager.template_env.get_template(
|
||||
'hap_coraza_spoe_engine.tpl').render())
|
||||
m = [re.match(r'timeout\s+processing\s+(\d+)ms$', l) for l in spoe]
|
||||
m = [x for x in m if x]
|
||||
self.assertEqual(len(m), 1)
|
||||
self.processing_ms = int(m[0].group(1))
|
||||
|
||||
def test_bufsize_fits_the_spoe_time_budget(self):
|
||||
"""~3.3 ms/KB measured; 31 KB took 104 ms against a 100 ms timeout and
|
||||
FAILED OPEN. Keep the worst case at about half the budget."""
|
||||
worst_ms = (self.bufsize / 1024.0) * 3.3
|
||||
self.assertLessEqual(worst_ms, self.processing_ms * 0.6,
|
||||
f'tune.bufsize {self.bufsize} => ~{worst_ms:.0f} ms of Coraza time '
|
||||
f'against a {self.processing_ms} ms fail-open timeout')
|
||||
|
||||
def test_bufsize_below_spop_frame_cap(self):
|
||||
self.assertLess(self.bufsize, SPOA_FRAME_CAP)
|
||||
|
||||
def test_body_arg_is_the_full_buffered_body(self):
|
||||
"""The message must keep sending req.body unsliced: the `complete`
|
||||
stamp compares req.body_len to Content-Length, so a slice would make
|
||||
a truncated body look complete to Coraza."""
|
||||
spoe = haproxy_manager.template_env.get_template('hap_coraza_spoe_engine.tpl').render()
|
||||
args = [l.strip() for l in spoe.splitlines() if l.strip().startswith('args ')]
|
||||
self.assertEqual(len(args), 1)
|
||||
self.assertTrue(args[0].endswith(' body=req.body'))
|
||||
|
||||
|
||||
class CorazaTransactionIdIsTheRequestReference(unittest.TestCase):
|
||||
"""Coraza's transaction ID must be HAProxy's unique-id, so the reference a
|
||||
blocked customer quotes (403-waf.html / X-Request-Reference / `id=` in
|
||||
haproxy.log) is the `unique_id` in /var/log/coraza/audit.log and
|
||||
spoa.log. Without `id=` coraza-spoa v0.7.1 invents a random 16-letter ID
|
||||
(internal/application.go HandleRequest) that joins to nothing."""
|
||||
|
||||
def setUp(self):
|
||||
spoe = haproxy_manager.template_env.get_template(
|
||||
'hap_coraza_spoe_engine.tpl').render()
|
||||
lines = active_lines(spoe)
|
||||
self.messages = {}
|
||||
current = None
|
||||
for line in lines:
|
||||
m = re.match(r'^spoe-message\s+(\S+)$', line)
|
||||
if m:
|
||||
current = m.group(1)
|
||||
continue
|
||||
if line.startswith('spoe-') or line.startswith('['):
|
||||
current = None
|
||||
elif current and line.startswith('args '):
|
||||
self.messages[current] = line.split()[1:]
|
||||
|
||||
def args_of(self, message):
|
||||
return dict(a.split('=', 1) for a in self.messages[message])
|
||||
|
||||
def test_request_message_sends_unique_id(self):
|
||||
self.assertIn('coraza-req', self.messages)
|
||||
self.assertEqual(self.args_of('coraza-req').get('id'), 'unique-id')
|
||||
|
||||
def test_request_message_exports_rule_ids(self):
|
||||
"""Without exportRuleIDs coraza-spoa never sets txn.coraza.rule_ids
|
||||
and the access log's wrules= is always '-'."""
|
||||
self.assertEqual(self.args_of('coraza-req').get('exportRuleIDs'), 'bool(true)')
|
||||
|
||||
def test_app_is_still_the_first_arg(self):
|
||||
"""agent.go HandleSPOE Panics unless the FIRST kv entry is `app`."""
|
||||
for name, args in self.messages.items():
|
||||
self.assertTrue(args[0].startswith('app='),
|
||||
f'{name}: first arg must be app=, got {args[0]}')
|
||||
|
||||
def test_every_response_message_carries_the_same_id(self):
|
||||
"""HandleResponse finds the request transaction by `id`; a coraza-res
|
||||
without it errors ("response id is empty") on every response."""
|
||||
for name in self.messages:
|
||||
if name != 'coraza-req':
|
||||
self.assertEqual(self.args_of(name).get('id'), 'unique-id', name)
|
||||
|
||||
def test_unique_id_is_the_header_reference(self):
|
||||
"""The fetch is only meaningful while unique-id-format is the UUID that
|
||||
is also emitted as X-Request-Reference and on the 403 page."""
|
||||
header = active_lines(render('hap_header.tpl', coraza=True))
|
||||
self.assertIn('unique-id-format %[uuid()]', [re.sub(r'\s+', ' ', l) for l in header])
|
||||
self.assertIn('unique-id-header X-Request-Reference', [re.sub(r'\s+', ' ', l) for l in header])
|
||||
listener = render('hap_listener.tpl', coraza=True)
|
||||
self.assertIn('x-request-reference "%[unique-id]"', listener)
|
||||
|
||||
|
||||
CORAZA_OVERRIDES = os.path.join(MODULE_DIR, 'coraza-spoa', 'overrides.conf')
|
||||
|
||||
|
||||
@unittest.skipUnless(os.path.exists(CORAZA_OVERRIDES),
|
||||
'coraza-spoa/overrides.conf is not shipped in this image; '
|
||||
'this class runs from a repo checkout')
|
||||
class BakedPromotions(unittest.TestCase):
|
||||
"""920440 is a CHAIN whose first link matches every URL with a file
|
||||
extension. `SecRuleUpdateActionById 920440 "ctl:ruleEngine=On"` ran on that
|
||||
first link, so every *.php / *.css request was evaluated with the engine
|
||||
On (full CRS anomaly enforcement, in detect_only mode too). Measured
|
||||
in-process and live, 2026-10-09; fixed by keying the flip on the chain's
|
||||
real condition."""
|
||||
|
||||
def setUp(self):
|
||||
with open(CORAZA_OVERRIDES) as fh:
|
||||
self.lines = active_lines(fh.read())
|
||||
|
||||
def test_920440_is_not_promoted_on_its_chain_starter(self):
|
||||
self.assertFalse([l for l in self.lines
|
||||
if re.match(r'SecRuleUpdateActionById\s+(\S*-)?920440\b', l)],
|
||||
'920440 promoted via SecRuleUpdateActionById again')
|
||||
|
||||
def test_no_chain_starter_is_promoted(self):
|
||||
"""Chain-starter rules in the promoted ranges (CRS 4.25.0): 920440,
|
||||
944110, 944120. None may be covered by a SecRuleUpdateActionById."""
|
||||
chain_starters = (920440, 944110, 944120)
|
||||
promoted = []
|
||||
for l in self.lines:
|
||||
m = re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l)
|
||||
if m:
|
||||
lo = int(m.group(1))
|
||||
promoted.append((lo, int(m.group(2) or lo)))
|
||||
self.assertTrue(promoted, 'no promotions parsed -- the test is blind')
|
||||
for rid in chain_starters:
|
||||
self.assertFalse([r for r in promoted if r[0] <= rid <= r[1]],
|
||||
f'{rid} is a CRS chain starter and must not be promoted')
|
||||
|
||||
def test_log4shell_range_still_promoted(self):
|
||||
for rid in (944100, 944130, 944150, 944151, 944152, 944300):
|
||||
self.assertTrue([1 for l in self.lines
|
||||
if (m := re.match(r'SecRuleUpdateActionById\s+(\d+)(?:-(\d+))?\s', l))
|
||||
and int(m.group(1)) <= rid <= int(m.group(2) or m.group(1))],
|
||||
f'{rid} lost its promotion')
|
||||
|
||||
def test_920440_flip_keys_on_the_restricted_extension(self):
|
||||
i = self.lines.index('SecRule TX:EXTENSION "@within %{tx.restricted_extensions}" \\')
|
||||
self.assertEqual(self.lines[i + 1],
|
||||
'"id:992920440,phase:1,pass,nolog,t:none,t:lowercase,ctl:ruleEngine=On"')
|
||||
|
||||
|
||||
@unittest.skipUnless(os.path.exists(CORAZA_CONFIG),
|
||||
'coraza-spoa/config.yaml is not shipped in this image; '
|
||||
'this class runs from a repo checkout')
|
||||
class CorazaBodyLimits(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
with open(CORAZA_CONFIG) as fh:
|
||||
self.cfg = active_lines(fh.read())
|
||||
header = active_lines(render('hap_header.tpl', coraza=True))
|
||||
self.bufsize = int([l for l in header if l.startswith('tune.bufsize')][0].split()[1])
|
||||
|
||||
def directive(self, name):
|
||||
vals = [l.split(None, 1)[1] for l in self.cfg if l.split()[0] == name]
|
||||
self.assertEqual(len(vals), 1, f'{name} must be set exactly once in config.yaml')
|
||||
return vals[0]
|
||||
|
||||
def test_limit_strictly_above_anything_haproxy_can_send(self):
|
||||
limit = int(self.directive('SecRequestBodyLimit'))
|
||||
self.assertGreater(limit, SPOA_FRAME_CAP)
|
||||
self.assertGreater(limit, self.bufsize)
|
||||
|
||||
def test_over_limit_never_rejects(self):
|
||||
"""Reject interrupts even under DetectionOnly -> 403 on uploads."""
|
||||
self.assertEqual(self.directive('SecRequestBodyLimitAction'), 'ProcessPartial')
|
||||
|
||||
def test_body_buffer_stays_in_memory(self):
|
||||
self.assertGreaterEqual(int(self.directive('SecRequestBodyInMemoryLimit')),
|
||||
int(self.directive('SecRequestBodyLimit')))
|
||||
|
||||
def test_json_depth_limit_pinned(self):
|
||||
"""Quadratic JSON flattening: depth 300 = 297 ms, past the fail-open
|
||||
timeout. Legitimate WordPress JSON measured at depth <= 9."""
|
||||
depth = int(self.directive('SecRequestBodyJsonDepthLimit'))
|
||||
self.assertGreaterEqual(depth, 32, 'too tight for real block/theme payloads')
|
||||
self.assertLessEqual(depth, 64, 'depth 100 already costs ~40 ms')
|
||||
|
||||
def test_limits_are_set_after_the_recommended_file(self):
|
||||
inc = self.cfg.index('Include @coraza.conf-recommended')
|
||||
for name in ('SecRequestBodyLimit ', 'SecRequestBodyJsonDepthLimit '):
|
||||
lim = [i for i, l in enumerate(self.cfg) if l.startswith(name)][0]
|
||||
self.assertLess(inc, lim, f'the recommended file would override {name.strip()}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
@@ -10,9 +10,9 @@
|
||||
# config.yaml.
|
||||
#
|
||||
# Reference: this config follows the shape from coraza-spoa's upstream
|
||||
# example/haproxy/coraza.cfg (v0.7.1). Arg NAMES are a contract with
|
||||
# coraza-spoa — DO NOT rename without coordinating with the agent. Ordering:
|
||||
# only `app` is positional (it must be FIRST); the rest are matched by name.
|
||||
# example/haproxy/coraza.cfg (v0.7.1). Arg names + ordering are required by
|
||||
# Coraza-SPOA exactly as specified — DO NOT reorder or rename without
|
||||
# coordinating with the agent.
|
||||
|
||||
[coraza]
|
||||
|
||||
@@ -56,41 +56,9 @@ spoe-agent coraza
|
||||
|
||||
# Per-request inspection message. No `event` directive — fires only when
|
||||
# explicitly invoked from haproxy.cfg via `http-request send-spoe-group`.
|
||||
#
|
||||
# How coraza-spoa v0.7.1 reads these args (measured from source, not inferred):
|
||||
# * `app` MUST be the first arg. internal/agent.go HandleSPOE reads exactly
|
||||
# one KV entry and Panics if its name is not "app".
|
||||
# * Every other arg is matched BY NAME in a switch (internal/application.go
|
||||
# HandleRequest); order is irrelevant and unknown names are ignored with a
|
||||
# debug log. So a typo'd name silently drops that input — renames still
|
||||
# break inspection, they just don't error.
|
||||
# `app=str(haproxy)` is the literal application name from coraza-spoa's
|
||||
# config.yaml `applications:` block.
|
||||
#
|
||||
# id=unique-id makes Coraza's transaction ID the SAME UUID HAProxy generates
|
||||
# via `unique-id-format %[uuid()]` (hap_header.tpl) — the value shown on the
|
||||
# 403-waf.html page, sent as X-Request-Reference, and logged as `id=` in
|
||||
# /var/log/haproxy.log. coraza-spoa passes it to waf.NewTransactionWithID(),
|
||||
# so it becomes `transaction.id` in /var/log/coraza/audit.log and
|
||||
# `match.unique_id` in /var/log/coraza/spoa.log: a customer-quoted reference
|
||||
# greps straight to the rule that fired. Without it coraza-spoa invents a
|
||||
# random 16-letter A-Z ID that appears nowhere on the HAProxy side. If the
|
||||
# fetch ever came back empty the agent falls back to that random ID, so this
|
||||
# cannot break inspection.
|
||||
#
|
||||
# exportRuleIDs=bool(true) makes coraza-spoa (application.go exportWAFMetrics)
|
||||
# set var(txn.coraza.rule_ids) = comma list of every matched CRS attack rule
|
||||
# (910000-959999 + 190000-199999; control rules such as our 99xxxxxxx host
|
||||
# exceptions and 2000xx body rules are NOT listed), alongside rules_hit and
|
||||
# anomaly_score which it sets regardless. var(txn.coraza.ruleid) is set on
|
||||
# every interruption (agent.go) -- for anomaly blocks that is 949110, so
|
||||
# rule_ids is what actually says WHY. All three are logged in the access
|
||||
# log (hap_listener.tpl log-format: wrule= wrules= wscore=).
|
||||
#
|
||||
# Response inspection (a `coraza-res` message) is NOT configured:
|
||||
# config.yaml has response_check: false. If it is ever enabled, coraza-res
|
||||
# MUST also send `id=unique-id` -- HandleResponse
|
||||
# looks the request transaction up by that ID and errors without it.
|
||||
# Arg order/names are mandatory: Coraza-SPOA parses positionally and renames
|
||||
# break the agent. `app=str(haproxy)` is the literal application name from
|
||||
# coraza-spoa's config.yaml `applications:` block.
|
||||
#
|
||||
# src-ip uses var(txn.real_ip) — HAProxy resolves the real client IP at the
|
||||
# top of the frontend (CF-Connecting-IP > X-Real-IP > X-Forwarded-For > src,
|
||||
@@ -98,7 +66,7 @@ spoe-agent coraza
|
||||
# when no proxy headers are present. This is what shows up as `client_ip`
|
||||
# in /var/log/coraza/audit.log and the rule manager's view-matches panel.
|
||||
spoe-message coraza-req
|
||||
args app=str(haproxy) id=unique-id exportRuleIDs=bool(true) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
|
||||
args app=str(haproxy) src-ip=var(txn.real_ip) src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
|
||||
|
||||
# Group binding for send-spoe-group invocation in the frontend. One group,
|
||||
# one message; could add more in the future (e.g. coraza-res for response
|
||||
|
||||
@@ -106,41 +106,6 @@ global
|
||||
|
||||
# Stats persistence for zero-downtime reloads
|
||||
stats-file /var/lib/haproxy/stats.dat
|
||||
{%- if coraza_spoe_backend %}
|
||||
|
||||
# WAF BODY INSPECTION LIMIT. Only emitted when Coraza is wired in, so a
|
||||
# WAF-off / standalone config is unchanged byte-for-byte. 16384 is also
|
||||
# HAProxy's default: this line changes no behaviour or memory, it pins
|
||||
# the value so nobody raises it without reading this.
|
||||
#
|
||||
# The request body Coraza sees is whatever `req.body` holds when
|
||||
# send-spoe-group fires, and with `option http-buffer-request`
|
||||
# (hap_listener.tpl) that is "the whole body, or as much as fits in ONE
|
||||
# buffer". So bufsize IS the inspection limit: a body up to roughly
|
||||
# bufsize - maxrewrite (1024) - the request headers (~15 KB) is inspected
|
||||
# whole and stamped `complete`; anything larger is forwarded intact, its
|
||||
# first ~15 KB inspected, and stamped `partial`. Large uploads are never
|
||||
# rejected or held for it.
|
||||
#
|
||||
# DO NOT RAISE THIS WITHOUT RE-MEASURING -- three ceilings, all measured:
|
||||
# * TIME. Coraza+CRS 4.25 costs ~3.3 ms per KB of body (linear; 31 KB
|
||||
# base64 = 104 ms, 15 KB = ~50 ms on a 32-core box). The SPOE
|
||||
# `timeout processing` is 100 ms (hap_coraza_spoe_engine.tpl) and a
|
||||
# timeout FAILS OPEN: with tune.bufsize 32768 a payload at byte 0 of a
|
||||
# 30 KB base64 multipart body passed 933180 uninspected. A larger
|
||||
# buffer is a bypass, not more coverage, unless the timeout (and CPU
|
||||
# budget) grows with it.
|
||||
# * FRAME. coraza-spoa's SPOP library (dropmorepackets/haproxy-go
|
||||
# v0.0.8) refuses a HELLO whose max-frame-size exceeds 65535; HAProxy
|
||||
# derives its offer from bufsize. bufsize >= 64 KB = every SPOE
|
||||
# handshake fails = WAF silently off for all traffic.
|
||||
# * MEMORY. maxconn 4000 x 2 buffers x bufsize, inside the 1 GB edge
|
||||
# container.
|
||||
# Coraza's SecRequestBodyLimit (coraza-spoa/config.yaml, 131072) must stay
|
||||
# above anything this can deliver. Pinned by
|
||||
# scripts/test-waf-body-inspection.py.
|
||||
tune.bufsize 16384
|
||||
{%- endif %}
|
||||
|
||||
#---------------------------------------------------------------------
|
||||
# DNS resolver for Docker container name resolution
|
||||
|
||||
@@ -63,18 +63,7 @@ frontend web
|
||||
# host=/ua= identify the vhost and client; %ST/%B/%tsc give status,
|
||||
# bytes and the termination state that distinguishes a rate-limit
|
||||
# deny (PR--) from a tarpit (PT--) from a normal close.
|
||||
# wafb=/wafe= WAF body coverage: wafb is complete|partial|- (did Coraza
|
||||
# get the whole request body, see the Coraza block below);
|
||||
# wafe is var(txn.coraza.error), set when the SPOE call failed or
|
||||
# timed out and the request went through UNINSPECTED (fail-open).
|
||||
# Both are "-" when the WAF is off or the request had no body.
|
||||
# wrule=/wrules=/wscore= WHY Coraza acted, without opening audit.log:
|
||||
# wrule is the interrupting rule (949110 for anomaly blocks),
|
||||
# wrules every matched CRS attack rule (e.g. 920440,949110),
|
||||
# wscore the inbound anomaly score. Set by coraza-spoa via the
|
||||
# exportRuleIDs SPOE arg (hap_coraza_spoe_engine.tpl). All "-"
|
||||
# when the WAF is off; wrule is "-" unless Coraza interrupted.
|
||||
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major] wafb=%[var(txn.waf_body)] wafe=%[var(txn.coraza.error)] wrule=%[var(txn.coraza.ruleid)] wrules=%[var(txn.coraza.rule_ids)] wscore=%[var(txn.coraza.anomaly_score)]"
|
||||
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r cip=%[var(txn.real_ip)] id=%ID host=%[capture.req.hdr(0)] ua=%[capture.req.hdr(1)] sni=%[ssl_fc_sni] hv=%[fc_http_major]"
|
||||
|
||||
# --- URI normalisation (MUST be the first path-touching block here) ---
|
||||
# Every path-based control in this frontend (the ACME health-check bypass,
|
||||
@@ -593,9 +582,7 @@ frontend web
|
||||
|
||||
# IP blocking using map file (manual blocks only)
|
||||
# Map file format: /etc/haproxy/blocked_ips.map contains "<ip_or_cidr> 1" per line
|
||||
# Runtime updates (worker command, map referenced by PATH -- "#<id>" ids
|
||||
# move on every config regeneration and "#0" silently adds nothing):
|
||||
# echo "@1 add map /etc/haproxy/blocked_ips.map IP_ADDRESS 1" | socat stdio /tmp/haproxy-cli
|
||||
# Runtime updates: echo "add map #0 IP_ADDRESS 1" | socat stdio /var/run/haproxy.sock
|
||||
# Checks the real client IP (from headers if present, otherwise src)
|
||||
# map_ip() converter supports both single IPs and CIDR ranges (e.g., 192.168.1.0/24)
|
||||
acl is_blocked_ip var(txn.real_ip),map_ip(/etc/haproxy/blocked_ips.map,0) -m int gt 0
|
||||
@@ -622,74 +609,7 @@ frontend web
|
||||
# guards (no point asking the WAF about requests we're already dropping)
|
||||
# and AFTER the real-client-IP resolution (so Coraza sees the right src).
|
||||
filter spoe engine coraza config /etc/haproxy/coraza-spoe.cfg
|
||||
|
||||
# --- Make Coraza actually see the request body -------------------------
|
||||
# send-spoe-group sends `req.body` = whatever body bytes are in the buffer
|
||||
# AT THAT MOMENT. Without buffering that is usually NOTHING: measured on
|
||||
# haproxy 3.0.11 + coraza-spoa v0.7.1, a 14-byte `content=$a($b)` POST
|
||||
# whose body was written in a second send() right after the headers
|
||||
# passed 933180 (an enforced rule) uninspected, while the identical bytes
|
||||
# in one send() were blocked. A body sent 2s late, or after an
|
||||
# `Expect: 100-continue`, was never inspected at all.
|
||||
#
|
||||
# `option http-buffer-request` makes HAProxy wait, before evaluating the
|
||||
# http-request rules, until the WHOLE body is in or the buffer is full,
|
||||
# bounded by `timeout http-request` (30s, hap_header.tpl). It answers
|
||||
# `Expect: 100-continue` itself so the client actually sends the body.
|
||||
# It does NOT hold uploads: once one buffer (tune.bufsize, 16 KB) is full
|
||||
# the request is forwarded and the rest streams through as before -- a
|
||||
# 50 MB upload is not stored at the edge.
|
||||
option http-buffer-request
|
||||
|
||||
# Exactly one Content-Type, or 400. With two Content-Type lines Coraza and
|
||||
# the backend disagree about what the body IS: Coraza evaluates every value
|
||||
# (`text/xml` on either line selects its XML processor, which accepts a
|
||||
# form body without error), while Apache merges the lines with ", " and
|
||||
# PHP cuts the media type at the first ',' / ';'. Measured (Coraza v3.7.0
|
||||
# + PHP 8.3): `application/x-www-form-urlencoded` + `text/xml` puts the
|
||||
# payload in $_POST with nothing inspected. req.fhdr_cnt counts header
|
||||
# LINES, so a single comma-joined value (which browsers produce when a
|
||||
# script calls setRequestHeader twice) is not affected here; that form is
|
||||
# parsed consistently enough by both sides (see the report's matrix).
|
||||
# Browsers, curl, PHP-curl and the common HTTP libraries send one line.
|
||||
http-request deny deny_status 400 if { req.fhdr_cnt(content-type) gt 1 }
|
||||
|
||||
# Tell Coraza whether it got the whole body. Coraza cannot tell on its
|
||||
# own (REQUEST_BODY_LENGTH is only filled by the urlencoded/raw
|
||||
# processors), and the difference matters: a TRUNCATED JSON body, or a
|
||||
# multipart body cut inside a boundary/part header, fails to parse by
|
||||
# construction, so the body-parse-error rules (200002/200003) may only
|
||||
# deny a COMPLETE body. WHP's generated Coraza rules
|
||||
# (regenerate-coraza-overrides.sh) key on this header:
|
||||
# complete -> 200002/200003 deny on multipart/JSON (enforce mode)
|
||||
# partial -> 200002/200003 removed for this request (every mode)
|
||||
#
|
||||
# With a Content-Length: complete iff req.body_len (buffered) equals
|
||||
# req.body_size (= Content-Length).
|
||||
# Without one (HTTP/1.1 chunked, HTTP/2-3 without content-length) HAProxy
|
||||
# 3.0 has no "end of message" fetch and req.body_size is just what is
|
||||
# buffered. But http-buffer-request only releases the request early when
|
||||
# the body ENDED or the buffer FILLED (a stalled body gets a 408 instead),
|
||||
# and body + headers well below one buffer (12 KB of the ~15 KB usable)
|
||||
# cannot have filled it. So that is complete, anything else partial.
|
||||
# Without this branch an attacker only had to add
|
||||
# `Transfer-Encoding: chunked` to escape enforcement (measured).
|
||||
#
|
||||
# Client-supplied copies are deleted FIRST so it cannot be forged, and it
|
||||
# is deleted again after the SPOE call so backends never see it.
|
||||
# txn.waf_body is also logged (`wafb=` in the log-format above) so the
|
||||
# complete/partial mix is measurable per host.
|
||||
http-request del-header X-WHP-WAF-Body
|
||||
http-request set-var(txn.waf_body_size) req.body_size if { req.hdr_cnt(content-length) eq 1 }
|
||||
http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_body_size) -m found } { req.body_len,sub(txn.waf_body_size) eq 0 }
|
||||
http-request set-var(txn.waf_body) str(partial) if { var(txn.waf_body_size) -m found } !{ req.body_len,sub(txn.waf_body_size) eq 0 }
|
||||
http-request set-var(txn.waf_hdrs_len) req.hdrs,length if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 }
|
||||
http-request set-var(txn.waf_body) str(complete) if { var(txn.waf_hdrs_len) -m found } { req.body_len,add(txn.waf_hdrs_len) lt 12288 }
|
||||
http-request set-var(txn.waf_body) str(partial) if !{ var(txn.waf_body_size) -m found } { req.body_len gt 0 } !{ var(txn.waf_body) -m found }
|
||||
http-request set-header X-WHP-WAF-Body %[var(txn.waf_body)] if { var(txn.waf_body) -m found }
|
||||
|
||||
http-request send-spoe-group coraza coraza-req
|
||||
http-request del-header X-WHP-WAF-Body
|
||||
|
||||
# Enforce Coraza's verdict. The SPOA sets var(txn.coraza.action) to
|
||||
# "deny" / "drop" / "redirect" when a rule with the corresponding
|
||||
|
||||
Reference in new issue
Block a user