Follow-up to 9d16151 (take the rollback backup BEFORE writing the new config).
Review findings, none of them blockers, plus two corrections to the record.
blocked_ips.map was still written with a plain open(path,'w'). `haproxy -c`
LOADS that file - hap_listener.tpl matches on
map_ip(/etc/haproxy/blocked_ips.map,0) - and a half-written final line is a
FATAL config error, not one dropped entry. Verified against HAProxy 2.8: a
truncated "198.51.10" gives "is not a valid IPv4 or IPv6 address at line 2 of
file ..." and the whole configuration is rejected. That is precisely the
failure shape the backup-ordering fix exists to prevent, on a different file,
reachable from all five /api/blocked-ips routes. It now goes through the same
write_config_atomically() as haproxy.cfg and coraza-spoe.cfg. (A truncation
that happens to land on a line boundary is not fatal - it silently drops
blocks - which is its own reason to write the file atomically.)
The previous commit message claimed the fast path adds "no `haproxy -c`
latency to customer-facing API calls". That was measured on an idle box and is
false in this fleet's normal pattern: update_blocked_ips_map() is called from
those five routes OUTSIDE generate_config(), so the live map drifts from its
backup and the NEXT config change misses create_backup()'s fast path. Measured
`haproxy -c` runs per domain add: 1 steady state, 2 after an IP block. This
fleet blocks IPs automatically, so the 2x recurred on the customer-facing call
indefinitely. update_blocked_ips_map() now promotes the map it just wrote to
its backup, restoring the steady state - guarded twice: nothing is promoted
unless a config backup set already exists (never fabricate a rollback target),
and not unless the map parses as IPs/CIDRs, so promotion cannot leave a
"rollback target" HAProxy would refuse to load. generate_config() passes
promote_backup=False: it took the snapshot moments earlier and the map is part
of the not-yet-validated change, so refreshing the backup there would be the
original bug again. The remaining 2 is the first generation after this upgrade
(coraza-spoe.cfg has no backup yet); that is once per host, by construction.
Two claims in 9d16151's message are wrong and are corrected here rather than by
rewriting a pushed commit:
* "12 of the 17 fail against the previous code" - it is 14 of 17 (6 failures +
8 errors). 12 was measured before two fast-path tests were added and never
re-measured.
* "a missing haproxy binary is not read as a bad config" - true of
create_backup() only. validate_haproxy_config() collapses both 'invalid' and
'unavailable' to False, so in the reload path a missing validator still
triggers a full rollback labelled "Config validation failed". The behaviour
is right (without a working validator we cannot claim the new config is
safe, and the reload path is where guessing wrong takes the edge down); the
sentence was broader than the code. Now documented on the function.
Tests: 26 (was 17), all green; 22 fail against main. New coverage closes the
review's mutation survivors:
* the "Refusing to regenerate config" guard, previously entirely uncovered;
* the invalid/unavailable split, previously zero coverage - both the verdict
and the consequence create_backup() draws from it;
* the byte-compare loop, with a same-size-different-content config, which is
the exact case the "not filecmp.cmp" rationale exists for. Building that
fixture found a bug in the test itself: sizing the drifted config with
len(str) instead of bytes made it pass for the wrong reason, because the
rendered config contains non-ASCII.
* test_failed_write_leaves_the_previous_file_intact was vacuous: its bare
assertRaises(Exception) swallowed the AttributeError from
write_config_atomically not existing, so it passed against main and would
have kept passing if the function were deleted. Narrowed to TypeError -
proven by deleting the function and watching it go red.
* test_backup_set_covers_every_file_generate_config_writes restated the three
files it expected, so it could never have noticed a fourth. It now derives
the set - observed on disk for the branches the fixture can execute, read
out of generate_config()'s source for the env-gated one that writes a
hardcoded /etc/haproxy path - and requires anything unbacked-up to be on a
documented exclusion list (suspended_domains.list, cluster-secret, each with
its reason). Proven by adding a fourth written file and watching it fail.
Every change above was mutation-proved: 11 mutations, 0 survivors, each
reddening only the tests that cover it. Two review items were confirmed
untestable in-process and are deliberately skipped: the fsync (M11) and a
log-line-only mutation (M15).
Left alone deliberately, all pre-existing on main and unchanged here: the
outer `except Exception` in reload_haproxy_safely() does not roll back (narrow
window, now commented at the site); stale .tmp files after SIGKILL are never
swept (verified inert - nothing globs /etc/haproxy, and the only
directory-wide load is `crt /etc/haproxy/certs`, which nothing here writes to);
a partial backup-copy failure can leave a mixed-vintage backup set (very
narrow, and generate_config() correctly refuses to write).
VERSION stays 2026.08.1. NOTE: fix/cert-write-safety, which is stacked on this
branch, carries the same 2026.08.1. If both land on main as separate commits,
CI pushes :2026.08.1 twice with different content. Either that branch moves to
2026.08.2 or the two land as a single merge - not decided here.
No template, QUIC or HTTP/3 changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
generate_config() wrote /etc/haproxy/haproxy.cfg and only then called
reload_haproxy_safely(), which called create_backup(). The "backup" was
therefore a copy of the config that had just been written, so on a validation
failure restore_backup() restored the identical broken bytes: the advertised
rollback was a no-op and a fatal haproxy.cfg stayed on disk, where
start_haproxy() refuses to launch. Same shape as the June 2026 incident where
a missing template produced a fatal config and took an edge down.
Reproduced end to end before the fix (invalid config generated -> "Backups
created successfully" -> "Backups restored successfully" -> haproxy.cfg on
disk still invalid, `haproxy -c` rc=1).
Changes:
* create_backup() is now called by generate_config() BEFORE the first write,
which also covers blocked_ips.map (rewritten early in generate_config) and
coraza-spoe.cfg - both previously written before the backup and, for the
SPOE file, never backed up at all even though `haproxy -c` parses it.
* create_backup() refuses to promote a config HAProxy already rejects, so a
broken file on disk cannot overwrite a known-good backup ("rollback" must
not mean "restore a different broken config"). It returns (ok, status) so
the caller knows whether a rollback target exists.
* promote_current_config_to_backup() records the config as known-good only
after it has validated AND loaded, so a box whose first generation succeeded
has a rollback target immediately, and a config that never loaded is never
promoted.
* restore_backup() returns (restored, message) and distinguishes "no backup
available" from "restored". Every caller now surfaces the difference; a
failed rollback is logged CRITICAL and reported as ROLLBACK FAILED in the
API error message instead of silently looking like a successful recovery.
* reload_haproxy_safely(backup_status=...) no longer takes its own backup - it
runs after the write, where a backup is meaningless. Called without a status
it logs the contract violation rather than overwriting a good backup.
* validate_config_file() separates "config is invalid" from "validator could
not run" so a missing haproxy binary is not read as a bad config.
* Config writes are atomic (temp file + fsync + os.replace, mode preserved);
a truncated haproxy.cfg is as fatal as an invalid one. Removes the dead
temp_config_path variable whose comment claimed this already happened.
* Fast path: if the live config set is already byte-identical to the backup
(the normal case after a successful reload), skip the re-validation and the
copy, so this adds no `haproxy -c` latency to customer-facing API calls.
Tests: scripts/test-config-rollback.py - 17 self-contained stdlib-unittest
tests, no new dependencies (the repo has no Python test framework; the
existing scripts/test-*.sh are curl integration scripts). A stub `haproxy`
binary stands in for the validator. 12 of the 17 fail against the previous
code; every assertion was mutation-proven (9 mutations, each reddening only
the tests that cover it).
No template, QUIC or HTTP/3 changes.
shadowdaoandClaude Opus 5