feat: HAPROXY_NBTHREAD per-host thread-count knob (global nbthread, only when set)
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 22m41s
HAProxy Manager Build and Push / Build-and-Push (push) Successful in 22m41s
2026-11-19 sdbees: with nbthread 4 a thread spun on a server lock held by a TLS health-check thread stalled in a page fault under memory pressure -> watchdog abort of the whole edge. nbthread 1 removes the failure class. HAPROXY_NBTHREAD (container env, integer 1-64) renders `nbthread N` in the global section ONLY when set. Unset/invalid = not rendered: every other host's config is byte-identical to before (verified against main's template, with and without Coraza). Invalid values are ignored with a warning, never rendered. Tests: scripts/test-nbthread-knob.py (parse, render, unset byte-identity vs the template minus the block, no cpu-map/thread-groups), run in the image build; real haproxy -c of nbthread 1 (+SPOE+QUIC) and of a bad value in validate-rendered-config.py. VERSION 2026.11.8 (with the arm64 base guard). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185ukXi4YX5FxJGqhi6qab6
This commit is contained in:
1 parent
56404f4777
commit
8ab6fdcef0
7 files changed
+130
-2
No files matched your search
@@ -15,6 +15,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
- **Superseded-cert retirement**: covered by the same file (`TestRetirementRules`, `TestSanOverlapRetirement`). A new bundle retires any OTHER pem/lineage sharing ANY name (CN or SAN) with it - the pem moved to `cert-retired/`, the lineage's live+archive+renewal moved TOGETHER to `/etc/letsencrypt/retired/<stamp>/` (leaving archive/ behind makes certbot refuse the name forever), never deleted; the bundle's own `--cert-name` lineage is `live/<primary>` exactly and is never retired - unless that cert is still the only valid one for a name this edge serves (`retirement_blockers()`). `GET /api/capabilities` advertises this to WHP, which must not use whole-site re-bundling without it. Expired pems already in the crt dir are moved to `cert-retired/` by `cert_retire_expired_pems` (never the last pem, never `default_self_signed_cert.pem`). Expired lineages are never published (`cert_is_expired` in `cert-publish-lib.sh`, `leaf_is_expired()` in Python). Fixtures: `scripts/test-fixtures/san-certs/`.
|
||||
- **Multi-arch index assembly (coraza-spoa)**: `python3 scripts/test-assemble-multiarch-index.py` - offline (NOT run in the image build; it also reads `.gitea/workflows/build-push-coraza.yaml`); `scripts/assemble-multiarch-index.sh` turns the by-digest amd64 + arm64 pushes into the tag(s): arm64 is attached only if THIS run's arm64 step outcome is `success` (failure => amd64-only tag, never a stale arm64), a bad amd64 digest publishes nothing. `scripts/e2e-coraza-multiarch.sh` rehearses the whole thing against throwaway local registries (see coraza-spoa/README.md "Multi-arch").
|
||||
- **Publish verify gate**: `python3 scripts/test-verify-published-tags.py` - offline (needs `jq`; NOT run in the image build); `scripts/verify-published-tags.sh` is the last build-push.yaml step: every tag in every registry must resolve, carry the assembled platforms, and share ONE index digest. The digest is the sha256 of `imagetools inspect --raw` cross-checked against the text `Digest:` line / bare `--format` output (buildx 0.28 ignores `--format` and prints full text; CI run 5381).
|
||||
- **Per-host thread count knob**: `python3 scripts/test-nbthread-knob.py` - runs in the image build; `HAPROXY_NBTHREAD` (container env, 1-64; invalid/unset = ignored) renders `nbthread N` in `global` ONLY when set, so every other host's config is byte-identical. Real `haproxy -c` of `nbthread 1` is the `nbthread-1` scenario in `scripts/validate-rendered-config.py`. WHP side: `/etc/whp/edge-nbthread` -> container env at recreate.
|
||||
- **Default splash**: `python3 scripts/test-default-splash.py` - offline (runs in the image build); unrouted hosts (server hostname, bare IP, unknown names) are answered by `http-request return ... file /haproxy/errors/default-splash.html` BEFORE the Coraza SPOE call, never for a routed host (`routed_hosts.list`/`routed_wildcards.list`, written by `generate_config()` from the routing rows and in the rollback set), never for ACME/blocked/suspended. Run it after touching the splash block, the routing ACL templates, or the domain loops in `generate_config()`. See README "Default Splash".
|
||||
- **gunicorn access logging**: `python3 scripts/test-gunicorn-access-logging.py` - real gunicorn; :8080 has no access log, :8000 access goes to `/var/log/haproxy/api-access.log` (rotated by `scripts/gunicorn-access-rotate.py`), nothing per-request on stdout (journald rate-limit/Graylog flood), gunicorn errors stay on stderr. See docs/logging.md. Run after touching start-up.sh's gunicorn lines.
|
||||
- **Access log receiver**: `python3 scripts/test-access-log-receiver.py` - offline (runs in the image build); one datagram = one line (PRI stripped, control chars neutralised), size cap + gzip generations, write errors drop without raising, real UNIX-socket path. Run it after touching `haproxy-log-receiver.py` or the `log` line in `hap_header.tpl`.
|
||||
|
||||
@@ -162,6 +162,7 @@ RUN python3 /haproxy/scripts/test-wpadmin-gate.py \
|
||||
&& python3 /haproxy/scripts/test-trusted-proxy-gate.py \
|
||||
&& python3 /haproxy/scripts/test-xmlrpc-rate-limit.py \
|
||||
&& python3 /haproxy/scripts/test-default-splash.py \
|
||||
&& python3 /haproxy/scripts/test-nbthread-knob.py \
|
||||
&& python3 /haproxy/scripts/test-access-log-receiver.py \
|
||||
&& python3 /haproxy/scripts/test-haproxy-stderr-relay.py \
|
||||
&& python3 /haproxy/scripts/test-haproxy-supervisor.py \
|
||||
|
||||
@@ -3153,6 +3153,21 @@ def get_or_create_cluster_secret():
|
||||
logger.error("Failed to get/create cluster-secret: %s", e)
|
||||
return ''
|
||||
|
||||
|
||||
def parse_nbthread(raw):
|
||||
"""HAPROXY_NBTHREAD -> int 1..64, or None (= do not render `nbthread`).
|
||||
|
||||
Fail-safe: anything that is not a plain positive integer in range is
|
||||
IGNORED with a warning, never rendered, so a typo in a host file cannot
|
||||
produce an invalid haproxy.cfg (which would be a total edge outage)."""
|
||||
raw = (raw or '').strip()
|
||||
if not raw:
|
||||
return None
|
||||
if not (raw.isascii() and raw.isdigit()) or not 1 <= int(raw) <= 64:
|
||||
logger.warning(f"Ignoring HAPROXY_NBTHREAD={raw!r}: expected an integer 1-64")
|
||||
return None
|
||||
return int(raw)
|
||||
|
||||
def generate_config():
|
||||
try:
|
||||
conn = sqlite3.connect(DB_FILE)
|
||||
@@ -3246,10 +3261,15 @@ def generate_config():
|
||||
splash_raw = os.environ.get('HAPROXY_DEFAULT_SPLASH', '').strip().lower()
|
||||
default_splash = splash_raw not in ('0', 'false', 'no', 'off')
|
||||
|
||||
# Optional per-host thread count (global `nbthread`). Unset/invalid =
|
||||
# not rendered = HAProxy's automatic choice, byte-identical config.
|
||||
nbthread = parse_nbthread(os.environ.get('HAPROXY_NBTHREAD'))
|
||||
|
||||
# Add Haproxy Default Headers
|
||||
default_headers = template_env.get_template('hap_header.tpl').render(
|
||||
cluster_secret = get_or_create_cluster_secret(),
|
||||
syslog_target = syslog_target,
|
||||
nbthread = nbthread,
|
||||
coraza_spoe_backend = coraza_spoe_backend,
|
||||
)
|
||||
config_parts.append(default_headers)
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""HAPROXY_NBTHREAD knob: parsing, rendering, and the unset-path invariant.
|
||||
|
||||
The invariant that matters: a host that does not set HAPROXY_NBTHREAD gets a
|
||||
`global` section BYTE-IDENTICAL to what the image rendered before the knob
|
||||
existed (HAProxy keeps choosing its own thread count). The real `haproxy -c`
|
||||
of the `nbthread 1` shape is in scripts/validate-rendered-config.py.
|
||||
|
||||
python3 scripts/test-nbthread-knob.py
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
MODULE_DIR = os.path.abspath(
|
||||
os.environ.get('HAPROXY_MANAGER_DIR',
|
||||
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')))
|
||||
sys.path.insert(0, MODULE_DIR)
|
||||
import haproxy_manager as hm # noqa: E402
|
||||
|
||||
|
||||
def render(**kw):
|
||||
return hm.template_env.get_template('hap_header.tpl').render(
|
||||
cluster_secret='s', syslog_target='/log.sock',
|
||||
coraza_spoe_backend=kw.pop('coraza_spoe_backend', None), **kw)
|
||||
|
||||
|
||||
class ParseNbthread(unittest.TestCase):
|
||||
def test_unset_and_blank(self):
|
||||
for raw in (None, '', ' '):
|
||||
self.assertIsNone(hm.parse_nbthread(raw))
|
||||
|
||||
def test_valid(self):
|
||||
self.assertEqual(hm.parse_nbthread('1'), 1)
|
||||
self.assertEqual(hm.parse_nbthread(' 4\n'), 4)
|
||||
self.assertEqual(hm.parse_nbthread('64'), 64)
|
||||
|
||||
def test_invalid_is_ignored_never_rendered(self):
|
||||
for raw in ('0', '65', '-1', 'abc', '1.5', '1 2', '1;x', '١', '0x4', '+2'):
|
||||
self.assertIsNone(hm.parse_nbthread(raw), raw)
|
||||
|
||||
|
||||
class Render(unittest.TestCase):
|
||||
def test_unset_has_no_nbthread_at_all(self):
|
||||
for kw in ({}, {'nbthread': None}, {'nbthread': 0}, {'coraza_spoe_backend': 'x:1'}):
|
||||
self.assertNotIn('nbthread', render(**kw).replace('{{', ''), kw)
|
||||
|
||||
def test_unset_is_byte_identical_to_template_without_the_block(self):
|
||||
src_path = os.path.join(MODULE_DIR, 'templates', 'hap_header.tpl')
|
||||
with open(src_path) as fh:
|
||||
src = fh.read()
|
||||
stripped, n = re.subn(r'\{%- if nbthread %\}.*?\{%- endif %\}\n', '', src, flags=re.S)
|
||||
self.assertEqual(n, 1, 'expected exactly one nbthread block in hap_header.tpl')
|
||||
legacy = hm.template_env.from_string(stripped)
|
||||
for coraza in (None, 'x:1'):
|
||||
want = legacy.render(cluster_secret='s', syslog_target='/log.sock',
|
||||
coraza_spoe_backend=coraza)
|
||||
self.assertEqual(render(coraza_spoe_backend=coraza), want)
|
||||
self.assertEqual(render(coraza_spoe_backend=coraza, nbthread=None), want)
|
||||
|
||||
def test_set_renders_one_directive_in_global_only(self):
|
||||
out = render(nbthread=1)
|
||||
self.assertEqual(len(re.findall(r'^\s*nbthread 1$', out, re.M)), 1)
|
||||
glob = out.split('\nresolvers ', 1)[0]
|
||||
self.assertIn('\n nbthread 1\n', glob)
|
||||
self.assertLess(out.index('nbthread 1'), out.index('tune.ssl.default-dh-param'))
|
||||
# nothing else changed: removing the directive restores the unset render
|
||||
self.assertEqual(out.replace('\n nbthread 1', '', 1).count('nbthread 1'), 0)
|
||||
|
||||
def test_no_cpu_map_or_thread_groups_rendered(self):
|
||||
for kw in ({}, {'nbthread': 1}):
|
||||
out = '\n'.join(l for l in render(**kw).splitlines()
|
||||
if not l.lstrip().startswith('#'))
|
||||
for bad in ('cpu-map', 'thread-groups', 'nbtgroups'):
|
||||
self.assertNotIn(bad, out)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
@@ -146,6 +146,17 @@ SCENARIOS = (
|
||||
'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
|
||||
'HAPROXY_DEFAULT_SPLASH': 'off',
|
||||
}),
|
||||
# Per-host thread count knob: `nbthread 1` must parse next to everything
|
||||
# else rendered (limited-quic, QUIC bind, SPOE, stats-file). A bad value
|
||||
# must be IGNORED (not rendered), never reach haproxy.
|
||||
('nbthread-1', {
|
||||
'HAPROXY_SUSPENSION_ENABLED': 'true',
|
||||
'HAPROXY_CORAZA_SPOE_BACKEND': '127.0.0.1:9000',
|
||||
'HAPROXY_NBTHREAD': '1',
|
||||
}),
|
||||
('nbthread-bad', {
|
||||
'HAPROXY_NBTHREAD': 'abc',
|
||||
}),
|
||||
)
|
||||
|
||||
|
||||
@@ -308,7 +319,7 @@ def render(scenario_name, env_overrides, workdir):
|
||||
|
||||
saved_env = {}
|
||||
for key in ('HAPROXY_SUSPENSION_ENABLED', 'HAPROXY_CORAZA_SPOE_BACKEND',
|
||||
'HAPROXY_DEFAULT_SPLASH'):
|
||||
'HAPROXY_DEFAULT_SPLASH', 'HAPROXY_NBTHREAD'):
|
||||
saved_env[key] = os.environ.pop(key, None)
|
||||
os.environ.update(env_overrides)
|
||||
|
||||
@@ -342,6 +353,8 @@ def assert_scenario_branches(scenario_name, config_path, env_overrides):
|
||||
'HAPROXY_CORAZA_SPOE_BACKEND' in env_overrides),
|
||||
'splash': ('file /haproxy/errors/default-splash.html',
|
||||
env_overrides.get('HAPROXY_DEFAULT_SPLASH') != 'off'),
|
||||
'nbthread': ('\n nbthread 1\n',
|
||||
env_overrides.get('HAPROXY_NBTHREAD') == '1'),
|
||||
}
|
||||
for label, (needle, should_be_present) in expected.items():
|
||||
present = needle in text
|
||||
|
||||
@@ -40,6 +40,19 @@ global
|
||||
user haproxy
|
||||
group haproxy
|
||||
daemon
|
||||
{%- if nbthread %}
|
||||
|
||||
# PER-HOST THREAD COUNT. Only emitted when HAPROXY_NBTHREAD is set on the
|
||||
# container (WHP: /etc/whp/edge-nbthread -> container-manager.sh -> env),
|
||||
# so a host that does not set it renders byte-identically to before and
|
||||
# HAProxy keeps choosing automatically (all CPUs in its affinity mask).
|
||||
# Why a host would set 1: 2026-11-19 sdbees, thread 1 spun on a server
|
||||
# lock held by a TLS health-check thread stalled in a page fault under
|
||||
# memory pressure -> watchdog abort (whole edge down). With one thread
|
||||
# there is no other lock holder, so the failure class cannot occur.
|
||||
# Do NOT combine with cpu-map/thread-groups here: neither is rendered.
|
||||
nbthread {{ nbthread }}
|
||||
{%- endif %}
|
||||
|
||||
# SSL and Performance
|
||||
tune.ssl.default-dh-param 2048
|
||||
|
||||
Reference in new issue
Block a user