David Snelling
1e046aa115
Two guards for every gate lane, born from the 2026-08-13 lost-day ledger. gate-preflight.sh refuses a lane on a machine that cannot be trusted to produce honest numbers — co-tenant processes named by pid and command, load average, CPU governor, disk floors — one FATAL line per violation so the operator can act from the message alone. vitest-verdict-check.sh refuses a suite log that cannot be trusted as a verdict — missing or mismatched summary counts, files that never executed (a truncated run once read as green from three files of ninety-nine), and worker-pool death signatures. Both verified live: the preflight correctly refuses this workstation naming its actual offenders; the verdict guard passes/fails five fixture shapes (clean, wrong-count, truncated, worker-death, no-summary) and both CLI modes. Wire-up into the CI lanes rides the runner program.
85 lines
3.7 KiB
Markdown
85 lines
3.7 KiB
Markdown
# Gate Guards
|
|
|
|
Two standalone scripts that stand between a test/build gate and a false
|
|
verdict: one refuses to let the gate start on a noisy machine, the other
|
|
refuses to let a truncated or crashed vitest run be read as green.
|
|
|
|
## Why these exist
|
|
|
|
Both guards exist because of the 2026-08-13 lost-day ledger: a gate ran on
|
|
a machine under load, and separately a vitest worker pool died mid-suite
|
|
while still printing a plausible-looking summary line, and in both cases
|
|
the bad result was trusted and acted on for the better part of a day before
|
|
anyone noticed. Neither failure mode announces itself — a loaded machine
|
|
still finishes and reports numbers, and a truncated test run still prints a
|
|
`Test Files` / `Tests` line — so both guards check the evidence explicitly
|
|
rather than trusting that a gate finishing means the gate was valid.
|
|
|
|
## gate-preflight.sh
|
|
|
|
Run before any gate lane starts. Exits 1 the moment the machine isn't
|
|
gate-clean, with one `FATAL:` line per violation naming the exact offender
|
|
(the pid and command, the path, the measured value). Prints one `OK:` line
|
|
per check that passes. `WARNING:` lines mark checks that were skipped, not
|
|
failures.
|
|
|
|
Checks:
|
|
|
|
| # | Check | Default threshold | Override |
|
|
|---|-------|--------------------|----------|
|
|
| a | 1-minute load average | `nproc / 2` | `GATE_MAX_LOAD` |
|
|
| b | any non-allowlisted process over 50% of one core | 50% | `GATE_ALLOW_REGEX` (extra pattern matched against the process's args) |
|
|
| c | cpu0 scaling governor must be `performance` | — | none (warns and skips if the sysfs path is absent) |
|
|
| d | free space on `/` and `/tmp` | 10G each | `GATE_SKIP_DISK_CHECK=1` to skip entirely |
|
|
|
|
The allowlist for check (b) is always: this script's own process tree
|
|
(its ancestors and its direct child processes), `sshd`, `systemd`, and
|
|
kernel threads (recognizable by args wrapped in brackets, e.g.
|
|
`[kworker/0:1]`). `GATE_ALLOW_REGEX` extends it — it does not replace it.
|
|
|
|
## vitest-verdict-check.sh
|
|
|
|
Run after every vitest lane, against that lane's captured log. Fails
|
|
loudly, quoting the exact line or string that tripped it, when the log's
|
|
own summary can't be trusted:
|
|
|
|
- no `Test Files` (or, in `--count-tests` mode, `Tests`) summary line is
|
|
present at all
|
|
- the parenthesized total in that line doesn't match what was expected
|
|
- fewer files/tests are accounted for (passed + failed + skipped) than the
|
|
total claims — a truncated run
|
|
- the log contains `Unhandled Error` or `Timeout calling` anywhere — a dead
|
|
worker pool, regardless of what the summary line claims
|
|
|
|
```
|
|
vitest-verdict-check.sh <log-file> <expected-file-count>
|
|
vitest-verdict-check.sh --count-tests <log-file> <minimum-test-count>
|
|
```
|
|
|
|
The first form checks `Test Files` for an exact match. The second checks
|
|
`Tests` for a minimum (a floor, not an exact count, since the total number
|
|
of individual tests moves more often than the number of test files).
|
|
|
|
## Wiring into a CI lane
|
|
|
|
```sh
|
|
# Before any lane that will report a verdict:
|
|
scripts/gate/gate-preflight.sh || exit 1
|
|
|
|
# Run the suite, capturing its output:
|
|
npx vitest run tests/unit 2>&1 | tee /tmp/unit.log
|
|
|
|
# After every vitest lane, check the log against the actual file count:
|
|
EXPECTED_FILES=$(ls tests/unit/**/*.test.ts | wc -l)
|
|
scripts/gate/vitest-verdict-check.sh /tmp/unit.log "$EXPECTED_FILES" || exit 1
|
|
```
|
|
|
|
## Exit-code contract
|
|
|
|
| Script | Exit 0 | Exit 1 |
|
|
|--------|--------|--------|
|
|
| `gate-preflight.sh` | machine is gate-clean | one or more `FATAL:` violations printed |
|
|
| `vitest-verdict-check.sh` | log's summary is trustworthy and matches | usage error, missing/unreadable log, or one or more `FATAL:` violations printed |
|
|
|
|
Non-zero from either script means: do not trust the gate that was about to
|
|
run, or the result of the one that just ran.
|