fix(api): refuse non-local peers on :8000; count header deadline from enqueue
Re-review R1: haproxy-manager shares client-net with every tenant container, so tenant PHP could reach haproxy-manager:8000 directly (the 127.0.0.1 publish only removed the internet path). On the ports in HAPROXY_MGR_PEER_FILTER_PORTS (default 8000; NOT :8080, fed by HAProxy over loopback) the overridden accept() closes any peer not in HAPROXY_MGR_API_ALLOWED_PEERS (default 127.0.0.1,::1,gateway), before a TConn, slot, poller registration or thread is used. "gateway" is read from /proc/net/route: measured, host callers via the 127.0.0.1 publish arrive from the bridge gateway (docker-proxy). Bad entries fail closed and are logged; refusals are logged at most once a minute as a per-peer summary. Header deadline is now max(enqueue + 10s, pickup + 1s) (HAPROXY_MGR_HEADER_PICKUP_GRACE): 40 reconnecting drippers no longer starve legit requests (before: every probe >30s; after: median ~9s). Tests: 21 (was 15): allowlist parsing/matching incl. v4-mapped, fail closed, /proc/net/route parsing, real-gunicorn filtered vs unfiltered port with rate-limited logging, listed peer served, 40 reconnecting drippers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fd06b30d78
commit
957fba29d9
3 files changed
+299
-15
No files matched your search
@@ -10,7 +10,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|||||||
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
|
- **Stick-table contract**: `python3 scripts/test-stick-table-contract.py` - offline; holds the templates' `store` clauses, `STICK_TABLE_FIELD_CONTRACT`, and every consumer to each other. Run it after touching any `stick-table` line.
|
||||||
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
|
- **Runtime-map contract**: `python3 scripts/test-runtime-map-contract.py` - offline; asserts the runtime map commands are `@1`-prefixed, reference the map by FILE PATH (never `#<id>`), carry the value `1`, and that every captured rejection is classified as a failure. Run it after touching any `add map`/`del map`/`clear map` path.
|
||||||
- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
|
- **Certificate destruction safety**: `python3 scripts/test-cert-write-safety.py` - offline; asserts a live `.pem` is never truncated, removed, or its certbot lineage deleted while any configured domain still references it (one bundle serves many names, so `ssl_cert_path` is routinely shared). Run it after touching any `os.remove`/`certbot delete`/PEM-write path.
|
||||||
- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`), slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
|
- **API thread-pool safety**: `python3 scripts/test-gunicorn-socket-timeouts.py` - runs REAL gunicorn (~60s); asserts a silent/half-written client cannot pin a gthread pool thread (per-connection socket timeout, `HAPROXY_MGR_CLIENT_TIMEOUT`), 2050 never-sending connections are reaped (`HAPROXY_MGR_FIRST_BYTE_TIMEOUT`), slowloris header drips are cut (`HAPROXY_MGR_HEADER_DEADLINE`, counted from enqueue), 40 reconnecting drippers cannot starve legit requests, the :8000 peer filter (`HAPROXY_MGR_API_ALLOWED_PEERS`) refuses non-allowed peers, slow legit requests are not cut off, the wedge watchdog (`HAPROXY_MGR_WEDGE_SECONDS`) replaces a fully stuck worker, and an in-process call to a stalled upstream times out. Run it after touching `haproxy_gunicorn_worker.py`, start-up.sh's gunicorn lines, or upgrading gunicorn (the worker overrides gthread internals: `enqueue_req`, `accept`, `on_client_socket_readable`, `murder_keepalived`, `handle`, `handle_request`).
|
||||||
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
|
- **Manual Testing**: Run `curl` commands against `http://localhost:8000` endpoints as shown in README.md
|
||||||
|
|
||||||
### Reading stick tables (and why it is easy to get silently wrong)
|
### Reading stick tables (and why it is easy to get silently wrong)
|
||||||
@@ -168,6 +168,7 @@ re-reads it on reload, so a failed runtime command must degrade to
|
|||||||
- Stats page on port 8404 (localhost only inside container)
|
- Stats page on port 8404 (localhost only inside container)
|
||||||
- Management interface on port 8000 should be firewall-protected in production
|
- Management interface on port 8000 should be firewall-protected in production
|
||||||
- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
|
- Dockerfile HEALTHCHECK runs `scripts/healthcheck.sh`: port 8000 `/health` (4s max) AND port 80 (HAProxy). WHP's `lib-edge-container.sh` passes its own `--health-cmd`, which REPLACES the image's; it must call `/haproxy/scripts/healthcheck.sh` or the fleet never probes :8000
|
||||||
|
- **The :8000 API refuses peers other than 127.0.0.1/::1 and the container's default gateway** (`HAPROXY_MGR_API_ALLOWED_PEERS`, default `127.0.0.1,::1,gateway`; ports via `HAPROXY_MGR_PEER_FILTER_PORTS`, default `8000`). haproxy-manager shares `client-net` with every tenant container; host callers arrive from the gateway via docker-proxy. Only effective with the WHP `-p 127.0.0.1:8000:8000` publish (a 0.0.0.0 publish lets a tenant hairpin through the host and arrive from the gateway). A sidecar that must call :8000 directly needs its IP added
|
||||||
- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
|
- gunicorn uses `haproxy_gunicorn_worker.TimeoutThreadWorker` (gthread + client socket timeout + first-byte deadline + total header deadline + wedge watchdog). Stock gthread leaves client sockets blocking with NO timeout; `--timeout` only watches the main thread's heartbeat
|
||||||
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
|
- Supports deployment on servers with git directory at `/root/whp` and web file sync via rsync to `/docker/whp/web/`
|
||||||
- HAProxy is version 3.0.11
|
- HAProxy is version 3.0.11
|
||||||
+152
-13
@@ -56,17 +56,20 @@ What this worker changes
|
|||||||
the main loop, freeing its slot (review F1).
|
the main loop, freeing its slot (review F1).
|
||||||
|
|
||||||
3. HEADER_DEADLINE (HAPROXY_MGR_HEADER_DEADLINE, default 10s): total time a
|
3. HEADER_DEADLINE (HAPROXY_MGR_HEADER_DEADLINE, default 10s): total time a
|
||||||
pool thread may spend reading the request line + headers. Armed when the
|
client may take to deliver the request line + headers, counted from when
|
||||||
thread picks the connection up, disarmed in handle_request() (which gunicorn
|
the connection was ENQUEUED for a thread, with a floor of
|
||||||
calls only once the parser has returned a complete request). A 1s monitor
|
HAPROXY_MGR_HEADER_PICKUP_GRACE (1s) after a thread picks it up:
|
||||||
loop shuts down (SHUT_RDWR, not close -- no fd-reuse race with the blocked
|
deadline = max(enqueue + 10s, pickup + 1s). Disarmed in handle_request()
|
||||||
recv) the socket of any connection past it; the blocked recv returns EOF and
|
(which gunicorn calls only once the parser has returned a complete
|
||||||
the thread is freed (review F2). Armed in handle(), NOT at accept/enqueue,
|
request). A 1s monitor loop shuts down (SHUT_RDWR, not close -- no fd-reuse
|
||||||
on purpose: a legit request queued behind busy threads (certbot runs) must
|
race with the blocked recv) the socket of any connection past it; the
|
||||||
not be cut for waiting. A slow client queued behind other slow clients is
|
blocked recv returns EOF and the thread is freed (review F2). A legit
|
||||||
therefore bounded by (wait for a thread) + HEADER_DEADLINE. Bodies are
|
request queued behind busy threads (certbot runs) already has its headers
|
||||||
only read by routes behind require_api_key, so the unauthenticated surface
|
in the kernel buffer and parses in milliseconds inside the 1s grace; a
|
||||||
is exactly this phase.
|
slow client that sat in the queue gets only the grace, so reconnecting
|
||||||
|
drippers cannot starve the pool by queueing (re-review). Bodies are only
|
||||||
|
read by routes behind require_api_key, so the unauthenticated surface is
|
||||||
|
exactly this phase.
|
||||||
|
|
||||||
4. Wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS, default 3600s; 0 disables). If
|
4. Wedge watchdog (HAPROXY_MGR_WEDGE_SECONDS, default 3600s; 0 disables). If
|
||||||
EVERY pool thread has been inside the same request for longer than that the
|
EVERY pool thread has been inside the same request for longer than that the
|
||||||
@@ -76,7 +79,20 @@ What this worker changes
|
|||||||
certbot (up to 180s) per domain; with (1)-(3) closing the known client-side
|
certbot (up to 180s) per domain; with (1)-(3) closing the known client-side
|
||||||
sources, this is only a backstop for blocking sources not yet found.
|
sources, this is only a backstop for blocking sources not yet found.
|
||||||
|
|
||||||
5. pool_status() exposes busy/total threads and the oldest in-flight age so
|
5. Peer filter (re-review R1). haproxy-manager shares the client-net bridge
|
||||||
|
with every tenant container, so tenant PHP could reach haproxy-manager:8000
|
||||||
|
directly; the 127.0.0.1 publish only removes the internet path. On the
|
||||||
|
ports in HAPROXY_MGR_PEER_FILTER_PORTS (default "8000" -- NOT the :8080
|
||||||
|
default-page server, which HAProxy feeds over loopback) accept() closes
|
||||||
|
any connection whose peer is not in HAPROXY_MGR_API_ALLOWED_PEERS (default
|
||||||
|
"127.0.0.1,::1,gateway"; IPs/CIDRs; "gateway" = this container's default
|
||||||
|
gateway, which is where host callers arrive from through docker-proxy).
|
||||||
|
Refused before a thread or connection slot is used; logged at most once a
|
||||||
|
minute as a per-peer summary. Effective only once the edge publishes
|
||||||
|
127.0.0.1:8000 -- with a 0.0.0.0 publish a tenant can hairpin through the
|
||||||
|
host and arrive from the gateway too.
|
||||||
|
|
||||||
|
6. pool_status() exposes busy/total threads and the oldest in-flight age so
|
||||||
/health can report saturation.
|
/health can report saturation.
|
||||||
|
|
||||||
This overrides gthread internals (enqueue_req, accept,
|
This overrides gthread internals (enqueue_req, accept,
|
||||||
@@ -87,6 +103,7 @@ real gunicorn in the Docker build gate and fails on behavioural drift.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import errno
|
import errno
|
||||||
|
import ipaddress
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import selectors
|
import selectors
|
||||||
@@ -115,6 +132,65 @@ HEADER_DEADLINE = _env_float('HAPROXY_MGR_HEADER_DEADLINE', 10)
|
|||||||
WEDGE_SECONDS = _env_float('HAPROXY_MGR_WEDGE_SECONDS', 3600)
|
WEDGE_SECONDS = _env_float('HAPROXY_MGR_WEDGE_SECONDS', 3600)
|
||||||
WATCHDOG_INTERVAL = _env_float('HAPROXY_MGR_WATCHDOG_INTERVAL', 15)
|
WATCHDOG_INTERVAL = _env_float('HAPROXY_MGR_WATCHDOG_INTERVAL', 15)
|
||||||
MONITOR_TICK = 1.0
|
MONITOR_TICK = 1.0
|
||||||
|
# Once a thread picks a queued connection up it gets at least this long to
|
||||||
|
# finish its headers, even if its enqueue-based deadline has already passed.
|
||||||
|
HEADER_PICKUP_GRACE = _env_float('HAPROXY_MGR_HEADER_PICKUP_GRACE', 1)
|
||||||
|
|
||||||
|
# Peer filter (review R1). Listener ports it applies to (comma-separated;
|
||||||
|
# empty disables) and the peers allowed through. "gateway" means this
|
||||||
|
# container's default gateway on its bridge network -- the address host-side
|
||||||
|
# callers arrive from via the 127.0.0.1:8000 publish (docker-proxy) and via a
|
||||||
|
# direct host->container-IP connect. Tenant containers on the same bridge
|
||||||
|
# (client-net) arrive from their OWN addresses and are refused.
|
||||||
|
PEER_FILTER_PORTS = os.environ.get('HAPROXY_MGR_PEER_FILTER_PORTS', '8000')
|
||||||
|
ALLOWED_PEERS_SPEC = os.environ.get('HAPROXY_MGR_API_ALLOWED_PEERS', '127.0.0.1,::1,gateway')
|
||||||
|
PEER_REJECT_LOG_INTERVAL = 60.0
|
||||||
|
|
||||||
|
|
||||||
|
def _default_gateways(route_file='/proc/net/route'):
|
||||||
|
"""IPv4 default gateway(s) from the kernel routing table."""
|
||||||
|
gws = []
|
||||||
|
try:
|
||||||
|
with open(route_file) as fh:
|
||||||
|
next(fh, None)
|
||||||
|
for line in fh:
|
||||||
|
f = line.split()
|
||||||
|
if len(f) >= 4 and f[1] == '00000000' and int(f[3], 16) & 0x2:
|
||||||
|
gws.append(str(ipaddress.IPv4Address(int(f[2], 16).to_bytes(4, 'little'))))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
pass
|
||||||
|
return gws
|
||||||
|
|
||||||
|
|
||||||
|
def parse_allowed_peers(spec, gateways=None):
|
||||||
|
"""Return (networks, problems). Tokens: IPs, CIDRs, or the word
|
||||||
|
'gateway'. Unparseable tokens are reported, never silently widened."""
|
||||||
|
nets, problems = [], []
|
||||||
|
for tok in (t.strip() for t in (spec or '').split(',')):
|
||||||
|
if not tok:
|
||||||
|
continue
|
||||||
|
if tok.lower() == 'gateway':
|
||||||
|
gws = _default_gateways() if gateways is None else gateways
|
||||||
|
if not gws:
|
||||||
|
problems.append('gateway: no IPv4 default route found')
|
||||||
|
for g in gws:
|
||||||
|
nets.append(ipaddress.ip_network(g + '/32'))
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
nets.append(ipaddress.ip_network(tok, strict=False))
|
||||||
|
except ValueError:
|
||||||
|
problems.append('unparseable entry %r' % tok)
|
||||||
|
return nets, problems
|
||||||
|
|
||||||
|
|
||||||
|
def peer_allowed(addr, nets):
|
||||||
|
try:
|
||||||
|
ip = ipaddress.ip_address(addr.split('%', 1)[0])
|
||||||
|
except (ValueError, AttributeError):
|
||||||
|
return False
|
||||||
|
if ip.version == 6 and ip.ipv4_mapped is not None:
|
||||||
|
ip = ip.ipv4_mapped
|
||||||
|
return any(ip.version == n.version and ip in n for n in nets)
|
||||||
|
|
||||||
TESTED_GUNICORN_VERSION = '23.0.0'
|
TESTED_GUNICORN_VERSION = '23.0.0'
|
||||||
|
|
||||||
@@ -173,6 +249,15 @@ class TimeoutThreadWorker(ThreadWorker):
|
|||||||
self._inflight = {} # thread ident -> _InFlight
|
self._inflight = {} # thread ident -> _InFlight
|
||||||
self._inflight_lock = threading.Lock()
|
self._inflight_lock = threading.Lock()
|
||||||
self._pending = deque() # accepted, not yet readable (FIFO by deadline)
|
self._pending = deque() # accepted, not yet readable (FIFO by deadline)
|
||||||
|
self._filter_ports = set()
|
||||||
|
for tok in PEER_FILTER_PORTS.split(','):
|
||||||
|
tok = tok.strip()
|
||||||
|
if tok.isdigit():
|
||||||
|
self._filter_ports.add(int(tok))
|
||||||
|
self._allowed_nets = []
|
||||||
|
self._rejected = {} # peer ip -> count since last log line
|
||||||
|
self._rejected_lock = threading.Lock()
|
||||||
|
self._last_reject_log = 0.0
|
||||||
|
|
||||||
# -- (1) client socket timeout ------------------------------------------
|
# -- (1) client socket timeout ------------------------------------------
|
||||||
def enqueue_req(self, conn):
|
def enqueue_req(self, conn):
|
||||||
@@ -183,6 +268,7 @@ class TimeoutThreadWorker(ThreadWorker):
|
|||||||
conn.init()
|
conn.init()
|
||||||
if CLIENT_TIMEOUT > 0:
|
if CLIENT_TIMEOUT > 0:
|
||||||
conn.sock.settimeout(CLIENT_TIMEOUT)
|
conn.sock.settimeout(CLIENT_TIMEOUT)
|
||||||
|
conn.whp_enqueued = time.monotonic() # header deadline counts from here
|
||||||
fs = self.tpool.submit(self.handle, conn)
|
fs = self.tpool.submit(self.handle, conn)
|
||||||
self._wrap_future(fs, conn)
|
self._wrap_future(fs, conn)
|
||||||
|
|
||||||
@@ -192,6 +278,14 @@ class TimeoutThreadWorker(ThreadWorker):
|
|||||||
# tracked in self._pending so murder_keepalived() can reap it.
|
# tracked in self._pending so murder_keepalived() can reap it.
|
||||||
try:
|
try:
|
||||||
sock, client = listener.accept()
|
sock, client = listener.accept()
|
||||||
|
if self._filter_ports and self._server_port(server) in self._filter_ports \
|
||||||
|
and not peer_allowed(client[0] if isinstance(client, tuple) else '',
|
||||||
|
self._allowed_nets):
|
||||||
|
# Refused before it costs a TConn, a connection slot, a poller
|
||||||
|
# registration or a thread.
|
||||||
|
sock.close()
|
||||||
|
self._note_rejected(client[0] if isinstance(client, tuple) else str(client))
|
||||||
|
return
|
||||||
conn = TConn(self.cfg, sock, client, server)
|
conn = TConn(self.cfg, sock, client, server)
|
||||||
conn.timeout = time.time() + FIRST_BYTE_TIMEOUT
|
conn.timeout = time.time() + FIRST_BYTE_TIMEOUT
|
||||||
|
|
||||||
@@ -205,6 +299,32 @@ class TimeoutThreadWorker(ThreadWorker):
|
|||||||
errno.EWOULDBLOCK):
|
errno.EWOULDBLOCK):
|
||||||
raise
|
raise
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _server_port(server):
|
||||||
|
try:
|
||||||
|
return int(server[1])
|
||||||
|
except (TypeError, IndexError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _note_rejected(self, peer):
|
||||||
|
with self._rejected_lock:
|
||||||
|
self._rejected[peer] = self._rejected.get(peer, 0) + 1
|
||||||
|
if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
|
||||||
|
self._flush_rejected()
|
||||||
|
|
||||||
|
def _flush_rejected(self):
|
||||||
|
with self._rejected_lock:
|
||||||
|
if not self._rejected:
|
||||||
|
return
|
||||||
|
items, self._rejected = self._rejected, {}
|
||||||
|
self._last_reject_log = time.monotonic()
|
||||||
|
total = sum(items.values())
|
||||||
|
top = sorted(items.items(), key=lambda kv: -kv[1])[:5]
|
||||||
|
self.log.warning(
|
||||||
|
'Refused %d connection(s) to the management API from %d non-allowed peer(s) '
|
||||||
|
'(HAPROXY_MGR_API_ALLOWED_PEERS=%s): %s%s', total, len(items), ALLOWED_PEERS_SPEC,
|
||||||
|
', '.join('%s x%d' % kv for kv in top), ' ...' if len(items) > 5 else '')
|
||||||
|
|
||||||
def on_client_socket_readable(self, conn, client):
|
def on_client_socket_readable(self, conn, client):
|
||||||
# ThreadWorker version only knows about self._keep; a first request
|
# ThreadWorker version only knows about self._keep; a first request
|
||||||
# comes off self._pending instead.
|
# comes off self._pending instead.
|
||||||
@@ -242,7 +362,16 @@ class TimeoutThreadWorker(ThreadWorker):
|
|||||||
def handle(self, conn):
|
def handle(self, conn):
|
||||||
ident = threading.get_ident()
|
ident = threading.get_ident()
|
||||||
now = time.monotonic()
|
now = time.monotonic()
|
||||||
entry = _InFlight(conn, now, now + HEADER_DEADLINE if HEADER_DEADLINE > 0 else None)
|
deadline = None
|
||||||
|
if HEADER_DEADLINE > 0:
|
||||||
|
# max(enqueue + D, pickup + grace): a legit request that waited in
|
||||||
|
# the queue behind busy threads already has its headers in the
|
||||||
|
# kernel buffer and parses in milliseconds; a slow client that
|
||||||
|
# waited gets only the grace, not a fresh full deadline (review:
|
||||||
|
# 40 reconnecting drippers otherwise starved legit requests).
|
||||||
|
enq = getattr(conn, 'whp_enqueued', now)
|
||||||
|
deadline = max(enq + HEADER_DEADLINE, now + HEADER_PICKUP_GRACE)
|
||||||
|
entry = _InFlight(conn, now, deadline)
|
||||||
with self._inflight_lock:
|
with self._inflight_lock:
|
||||||
self._inflight[ident] = entry
|
self._inflight[ident] = entry
|
||||||
try:
|
try:
|
||||||
@@ -337,6 +466,8 @@ class TimeoutThreadWorker(ThreadWorker):
|
|||||||
try:
|
try:
|
||||||
if HEADER_DEADLINE > 0:
|
if HEADER_DEADLINE > 0:
|
||||||
self._sweep_header_deadlines()
|
self._sweep_header_deadlines()
|
||||||
|
if time.monotonic() - self._last_reject_log >= PEER_REJECT_LOG_INTERVAL:
|
||||||
|
self._flush_rejected()
|
||||||
if time.monotonic() - last_wedge_check >= WATCHDOG_INTERVAL:
|
if time.monotonic() - last_wedge_check >= WATCHDOG_INTERVAL:
|
||||||
last_wedge_check = time.monotonic()
|
last_wedge_check = time.monotonic()
|
||||||
self._check_wedged()
|
self._check_wedged()
|
||||||
@@ -358,6 +489,14 @@ class TimeoutThreadWorker(ThreadWorker):
|
|||||||
'haproxy_gunicorn_worker overrides gthread internals as of gunicorn %s; '
|
'haproxy_gunicorn_worker overrides gthread internals as of gunicorn %s; '
|
||||||
'running %s -- re-run scripts/test-gunicorn-socket-timeouts.py',
|
'running %s -- re-run scripts/test-gunicorn-socket-timeouts.py',
|
||||||
TESTED_GUNICORN_VERSION, gunicorn.__version__)
|
TESTED_GUNICORN_VERSION, gunicorn.__version__)
|
||||||
|
if self._filter_ports:
|
||||||
|
self._allowed_nets, problems = parse_allowed_peers(ALLOWED_PEERS_SPEC)
|
||||||
|
for prob in problems:
|
||||||
|
self.log.error('HAPROXY_MGR_API_ALLOWED_PEERS: %s -- that entry allows nothing '
|
||||||
|
'(fail closed)', prob)
|
||||||
|
self.log.info('Management API peer filter on port(s) %s: allowing %s',
|
||||||
|
','.join(str(p) for p in sorted(self._filter_ports)),
|
||||||
|
', '.join(str(n) for n in self._allowed_nets) or 'NOTHING')
|
||||||
threading.Thread(target=self._monitor, name='worker-monitor',
|
threading.Thread(target=self._monitor, name='worker-monitor',
|
||||||
daemon=True).start()
|
daemon=True).start()
|
||||||
super().init_process() # runs the event loop; does not return
|
super().init_process() # runs the event loop; does not return
|
||||||
@@ -93,7 +93,7 @@ def http_get(port, path='/', timeout=5):
|
|||||||
class Gunicorn:
|
class Gunicorn:
|
||||||
"""A real gunicorn master on a free port, with its output captured."""
|
"""A real gunicorn master on a free port, with its output captured."""
|
||||||
|
|
||||||
def __init__(self, worker_class, threads=1, env=None, workers=1):
|
def __init__(self, worker_class, threads=1, env=None, workers=1, extra_ports=()):
|
||||||
self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
|
self.tmp = tempfile.mkdtemp(prefix='gunicorn-timeout-test-')
|
||||||
with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
|
with open(os.path.join(self.tmp, 'tapp.py'), 'w') as fh:
|
||||||
fh.write(WSGI_APP)
|
fh.write(WSGI_APP)
|
||||||
@@ -105,6 +105,7 @@ class Gunicorn:
|
|||||||
self.proc = subprocess.Popen(
|
self.proc = subprocess.Popen(
|
||||||
[sys.executable, '-m', 'gunicorn',
|
[sys.executable, '-m', 'gunicorn',
|
||||||
'--bind', '127.0.0.1:%d' % self.port,
|
'--bind', '127.0.0.1:%d' % self.port,
|
||||||
|
*sum((['--bind', '127.0.0.1:%d' % p] for p in extra_ports), []),
|
||||||
'--workers', str(workers), '--threads', str(threads),
|
'--workers', str(workers), '--threads', str(threads),
|
||||||
'--worker-class', worker_class,
|
'--worker-class', worker_class,
|
||||||
'--timeout', '120', '--graceful-timeout', '2',
|
'--timeout', '120', '--graceful-timeout', '2',
|
||||||
@@ -416,6 +417,149 @@ class ProductionShapeExhaustion(unittest.TestCase):
|
|||||||
self.assertIn(b'200 OK', s.recv(4096))
|
self.assertIn(b'200 OK', s.recv(4096))
|
||||||
|
|
||||||
|
|
||||||
|
def refused(port, wait=2.0):
|
||||||
|
"""True if the server closes a fresh connection without answering a
|
||||||
|
complete request (peer filter)."""
|
||||||
|
s = socket.create_connection(('127.0.0.1', port), timeout=wait)
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
s.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
|
||||||
|
except OSError:
|
||||||
|
return True
|
||||||
|
s.settimeout(wait)
|
||||||
|
try:
|
||||||
|
return s.recv(4096) == b''
|
||||||
|
except (ConnectionResetError, BrokenPipeError):
|
||||||
|
return True
|
||||||
|
except socket.timeout:
|
||||||
|
return False
|
||||||
|
finally:
|
||||||
|
s.close()
|
||||||
|
|
||||||
|
|
||||||
|
class PeerFilter(unittest.TestCase):
|
||||||
|
"""Re-review R1: tenant containers share client-net with haproxy-manager,
|
||||||
|
so the API port must refuse peers that are not loopback / the host path."""
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def setUpClass(cls):
|
||||||
|
sys.path.insert(0, REPO_DIR)
|
||||||
|
import haproxy_gunicorn_worker as w
|
||||||
|
cls.w = w
|
||||||
|
|
||||||
|
def test_allowlist_parsing_and_matching(self):
|
||||||
|
nets, problems = self.w.parse_allowed_peers('127.0.0.1,::1,gateway', gateways=['172.18.0.1'])
|
||||||
|
self.assertEqual(problems, [])
|
||||||
|
ok = self.w.peer_allowed
|
||||||
|
self.assertTrue(ok('127.0.0.1', nets))
|
||||||
|
self.assertTrue(ok('::1', nets))
|
||||||
|
self.assertTrue(ok('::ffff:127.0.0.1', nets))
|
||||||
|
self.assertTrue(ok('172.18.0.1', nets)) # docker-proxy / host
|
||||||
|
self.assertFalse(ok('172.18.0.7', nets)) # a tenant container
|
||||||
|
self.assertFalse(ok('127.0.0.2', nets))
|
||||||
|
self.assertFalse(ok('', nets))
|
||||||
|
self.assertFalse(ok('garbage', nets))
|
||||||
|
|
||||||
|
def test_bad_entries_fail_closed_and_are_reported(self):
|
||||||
|
nets, problems = self.w.parse_allowed_peers('nonsense,gateway', gateways=[])
|
||||||
|
self.assertEqual(nets, [])
|
||||||
|
self.assertEqual(len(problems), 2, problems)
|
||||||
|
nets, _ = self.w.parse_allowed_peers('10.0.0.0/8')
|
||||||
|
self.assertTrue(self.w.peer_allowed('10.1.2.3', nets))
|
||||||
|
|
||||||
|
def test_default_gateway_parsed_from_route_table(self):
|
||||||
|
fn = os.path.join(tempfile.mkdtemp(), 'route')
|
||||||
|
with open(fn, 'w') as fh:
|
||||||
|
fh.write('Iface\tDestination\tGateway \tFlags\tRefCnt\tUse\tMetric\tMask\n'
|
||||||
|
'eth0\t00000000\t010012AC\t0003\t0\t0\t0\t00000000\n'
|
||||||
|
'eth0\t000012AC\t00000000\t0001\t0\t0\t0\t0000FFFF\n')
|
||||||
|
self.assertEqual(self.w._default_gateways(fn), ['172.18.0.1'])
|
||||||
|
|
||||||
|
def test_filtered_port_refuses_unlisted_peer_other_port_unaffected(self):
|
||||||
|
other = free_port()
|
||||||
|
g = Gunicorn(FIXED, threads=1, extra_ports=(other,), env={
|
||||||
|
'HAPROXY_MGR_PEER_FILTER_PORTS': str(other),
|
||||||
|
'HAPROXY_MGR_API_ALLOWED_PEERS': '10.255.255.1'})
|
||||||
|
self.addCleanup(g.stop)
|
||||||
|
# g.port (unfiltered) works; `other` refuses 127.0.0.1 (not listed).
|
||||||
|
self.assertTrue(http_get(g.port, '/').startswith('pid='))
|
||||||
|
for _ in range(5):
|
||||||
|
self.assertTrue(refused(other), 'filtered port answered an unlisted peer')
|
||||||
|
self.assertTrue(http_get(g.port, '/').startswith('pid='))
|
||||||
|
out = g.output()
|
||||||
|
self.assertIn('Refused 1 connection(s) to the management API', out) # first one, immediately
|
||||||
|
self.assertEqual(out.count('Refused '), 1, 'reject logging is not rate-limited')
|
||||||
|
|
||||||
|
def test_filtered_port_serves_listed_peer(self):
|
||||||
|
port = free_port()
|
||||||
|
g = Gunicorn(FIXED, threads=1, extra_ports=(port,), env={
|
||||||
|
'HAPROXY_MGR_PEER_FILTER_PORTS': str(port),
|
||||||
|
'HAPROXY_MGR_API_ALLOWED_PEERS': '127.0.0.1,::1'})
|
||||||
|
self.addCleanup(g.stop)
|
||||||
|
self.assertTrue(http_get(port, '/').startswith('pid='))
|
||||||
|
self.assertNotIn('Refused', g.output())
|
||||||
|
|
||||||
|
|
||||||
|
class QueuedDripStarvation(unittest.TestCase):
|
||||||
|
"""Re-review: with the deadline armed at thread pickup, 40 reconnecting
|
||||||
|
drippers queued behind each other starved legit requests (every probe
|
||||||
|
>30s). Armed as max(enqueue + D, pickup + 1s) they must not."""
|
||||||
|
|
||||||
|
def test_40_reconnecting_drippers_do_not_starve_legit_requests(self):
|
||||||
|
g = Gunicorn(FIXED, threads=4, workers=2, env={}) # production defaults
|
||||||
|
self.addCleanup(g.stop)
|
||||||
|
stop = threading.Event()
|
||||||
|
self.addCleanup(stop.set)
|
||||||
|
|
||||||
|
def drip():
|
||||||
|
while not stop.is_set():
|
||||||
|
try:
|
||||||
|
s = socket.create_connection(('127.0.0.1', g.port))
|
||||||
|
except OSError:
|
||||||
|
time.sleep(0.2)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
req = b'GET / HTTP/1.1\r\nX-a: ' + b'a' * 5000
|
||||||
|
for i in range(len(req)):
|
||||||
|
if stop.is_set():
|
||||||
|
break
|
||||||
|
s.send(req[i:i + 1])
|
||||||
|
time.sleep(1.0)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
s.close()
|
||||||
|
|
||||||
|
for _ in range(40):
|
||||||
|
threading.Thread(target=drip, daemon=True).start()
|
||||||
|
time.sleep(2)
|
||||||
|
lat = []
|
||||||
|
t_end = time.time() + 25
|
||||||
|
while time.time() < t_end:
|
||||||
|
t0 = time.time()
|
||||||
|
try:
|
||||||
|
http_get(g.port, '/', timeout=30)
|
||||||
|
lat.append(time.time() - t0)
|
||||||
|
except Exception:
|
||||||
|
lat.append(99.0)
|
||||||
|
time.sleep(1)
|
||||||
|
stop.set()
|
||||||
|
lat.sort()
|
||||||
|
median = lat[len(lat) // 2]
|
||||||
|
sys.stderr.write('\n 40 drippers: probes=%d median=%.2fs worst=%.2fs\n'
|
||||||
|
% (len(lat), median, lat[-1]))
|
||||||
|
# Armed at pickup, every probe here exceeded 30s (total starvation).
|
||||||
|
# With max(enqueue + D, pickup + 1s) queued drippers get ~1s of thread
|
||||||
|
# time, so legit requests keep getting through. They still queue: the
|
||||||
|
# equilibrium wait is ~(drippers/threads) x hold time, about 9s at 40
|
||||||
|
# drippers on 8 threads -- measured, see the round-2 report. The
|
||||||
|
# remaining fix for that is not admitting partial requests to a thread
|
||||||
|
# at all; the peer filter makes it a local-only scenario.
|
||||||
|
self.assertNotIn(99.0, lat, 'a legit request was starved for 30s+')
|
||||||
|
self.assertLess(median, 12.0, lat)
|
||||||
|
self.assertLess(lat[-1], 20.0, lat)
|
||||||
|
|
||||||
|
|
||||||
class StalledUpstreamDefaultTimeout(unittest.TestCase):
|
class StalledUpstreamDefaultTimeout(unittest.TestCase):
|
||||||
"""haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
|
"""haproxy_manager sets socket.setdefaulttimeout() at import. An in-process
|
||||||
client call to an upstream that accepts and never replies must time out."""
|
client call to an upstream that accepts and never replies must time out."""
|
||||||
|
|||||||
Reference in new issue
Block a user