ci(gate): the machine-health preflight and the truncation verdict guard

Two guards for every gate lane, born from the 2026-08-13 lost-day ledger.
gate-preflight.sh refuses a lane on a machine that cannot be trusted to
produce honest numbers — co-tenant processes named by pid and command, load
average, CPU governor, disk floors — one FATAL line per violation so the
operator can act from the message alone. vitest-verdict-check.sh refuses a
suite log that cannot be trusted as a verdict — missing or mismatched
summary counts, files that never executed (a truncated run once read as
green from three files of ninety-nine), and worker-pool death signatures.

Both verified live: the preflight correctly refuses this workstation naming
its actual offenders; the verdict guard passes/fails five fixture shapes
(clean, wrong-count, truncated, worker-death, no-summary) and both CLI
modes. Wire-up into the CI lanes rides the runner program.

scripts/gate/README.md

@@ -0,0 +1,85 @@
+# Gate Guards
+
+Two standalone scripts that stand between a test/build gate and a false
+verdict: one refuses to let the gate start on a noisy machine, the other
+refuses to let a truncated or crashed vitest run be read as green.
+
+## Why these exist
+
+Both guards exist because of the 2026-08-13 lost-day ledger: a gate ran on
+a machine under load, and separately a vitest worker pool died mid-suite
+while still printing a plausible-looking summary line, and in both cases
+the bad result was trusted and acted on for the better part of a day before
+anyone noticed. Neither failure mode announces itself — a loaded machine
+still finishes and reports numbers, and a truncated test run still prints a
+`Test Files` / `Tests` line — so both guards check the evidence explicitly
+rather than trusting that a gate finishing means the gate was valid.
+
+## gate-preflight.sh
+
+Run before any gate lane starts. Exits 1 the moment the machine isn't
+gate-clean, with one `FATAL:` line per violation naming the exact offender
+(the pid and command, the path, the measured value). Prints one `OK:` line
+per check that passes. `WARNING:` lines mark checks that were skipped, not
+failures.
+
+Checks:
+
+| # | Check | Default threshold | Override |
+|---|-------|--------------------|----------|
+| a | 1-minute load average | `nproc / 2` | `GATE_MAX_LOAD` |
+| b | any non-allowlisted process over 50% of one core | 50% | `GATE_ALLOW_REGEX` (extra pattern matched against the process's args) |
+| c | cpu0 scaling governor must be `performance` | — | none (warns and skips if the sysfs path is absent) |
+| d | free space on `/` and `/tmp` | 10G each | `GATE_SKIP_DISK_CHECK=1` to skip entirely |
+
+The allowlist for check (b) is always: this script's own process tree
+(its ancestors and its direct child processes), `sshd`, `systemd`, and
+kernel threads (recognizable by args wrapped in brackets, e.g.
+`[kworker/0:1]`). `GATE_ALLOW_REGEX` extends it — it does not replace it.
+
+## vitest-verdict-check.sh
+
+Run after every vitest lane, against that lane's captured log. Fails
+loudly, quoting the exact line or string that tripped it, when the log's
+own summary can't be trusted:
+
+- no `Test Files` (or, in `--count-tests` mode, `Tests`) summary line is
+  present at all
+- the parenthesized total in that line doesn't match what was expected
+- fewer files/tests are accounted for (passed + failed + skipped) than the
+  total claims — a truncated run
+- the log contains `Unhandled Error` or `Timeout calling` anywhere — a dead
+  worker pool, regardless of what the summary line claims
+
+```
+vitest-verdict-check.sh <log-file> <expected-file-count>
+vitest-verdict-check.sh --count-tests <log-file> <minimum-test-count>
+```
+
+The first form checks `Test Files` for an exact match. The second checks
+`Tests` for a minimum (a floor, not an exact count, since the total number
+of individual tests moves more often than the number of test files).
+
+## Wiring into a CI lane
+
+```sh
+# Before any lane that will report a verdict:
+scripts/gate/gate-preflight.sh || exit 1
+
+# Run the suite, capturing its output:
+npx vitest run tests/unit 2>&1 | tee /tmp/unit.log
+
+# After every vitest lane, check the log against the actual file count:
+EXPECTED_FILES=$(ls tests/unit/**/*.test.ts | wc -l)
+scripts/gate/vitest-verdict-check.sh /tmp/unit.log "$EXPECTED_FILES" || exit 1
+```
+
+## Exit-code contract
+
+| Script | Exit 0 | Exit 1 |
+|--------|--------|--------|
+| `gate-preflight.sh` | machine is gate-clean | one or more `FATAL:` violations printed |
+| `vitest-verdict-check.sh` | log's summary is trustworthy and matches | usage error, missing/unreadable log, or one or more `FATAL:` violations printed |
+
+Non-zero from either script means: do not trust the gate that was about to
+run, or the result of the one that just ran.

scripts/gate/gate-preflight.sh

@@ -0,0 +1,206 @@
+#!/bin/bash
+set -euo pipefail
+
+# Brainy Gate Preflight
+# Refuses to let a test/build gate run on a machine that isn't clean enough
+# to trust the numbers it produces. See scripts/gate/README.md for why (the
+# 2026-08-13 lost-day ledger).
+#
+# Checks: 1-minute load average, any non-allowlisted process pinning a core,
+# the cpu0 scaling governor, and free space on / and /tmp.
+#
+# Exit 0 and print one OK line per passing check when the machine is clean.
+# Exit 1 and print one FATAL line per violation, naming the offender, when
+# it is not.
+#
+# Known trap: a helper function whose last executed statement is a `while`
+# (or any command whose own exit status happens to be nonzero) hands that
+# status back as the function's return value. Called as a plain statement,
+# that silently kills this script under `set -e`. Every helper below ends
+# on an explicit `return 0` as its own statement, never on a loop or test.
+#
+# The same failure mode hides in plainer-looking lines too: `var=$(cmd)` is
+# a bare assignment, so `set -e` DOES treat a nonzero `cmd` (or, under
+# `pipefail`, a nonzero stage anywhere in `cmd`'s pipeline) as a failure of
+# that statement and kills the script right there — even mid-loop, even
+# when the "failure" is routine (a process that exited before a second
+# lookup, a path that doesn't exist). Every such assignment below is paired
+# with an explicit `|| var=""` fallback so a routine miss degrades to an
+# empty value instead of an exit.
+
+VIOLATIONS=0
+ANCESTOR_PIDS=""
+
+fatal() {
+  echo "FATAL: $1"
+  VIOLATIONS=$((VIOLATIONS + 1))
+}
+
+ok() {
+  echo "OK: $1"
+}
+
+# Walks this process's parent chain up to pid 1, then takes one snapshot of
+# its direct children (the ps/read pipeline in check_processes), and
+# records both in ANCESTOR_PIDS — so the process-scan below can recognize
+# its own tree (the shell/terminal/session that launched it, plus its own
+# helper commands) instead of flagging it. Children are captured once, up
+# front, rather than re-queried per row later, so a helper command that has
+# already exited by the time it's looked up can't be mistaken for a miss.
+build_ancestor_pids() {
+  local pid="$$"
+  local ppid child
+  ANCESTOR_PIDS=" $pid "
+  while [ "$pid" != "1" ]; do
+    ppid=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d ' ') || ppid=""
+    if [ -z "$ppid" ]; then
+      break
+    fi
+    ANCESTOR_PIDS="${ANCESTOR_PIDS}${ppid} "
+    pid="$ppid"
+  done
+
+  while IFS= read -r child; do
+    [ -z "$child" ] && continue
+    ANCESTOR_PIDS="${ANCESTOR_PIDS}${child} "
+  done < <(ps --ppid "$$" -o pid= 2>/dev/null || true)
+
+  return 0
+}
+
+# (a) 1-minute load average vs. threshold (default: nproc / 2).
+check_load() {
+  local max_load="${GATE_MAX_LOAD:-}"
+  if [ -z "$max_load" ]; then
+    max_load=$(( $(nproc) / 2 ))
+    if [ "$max_load" -lt 1 ]; then
+      max_load=1
+    fi
+  fi
+
+  local load_1m
+  load_1m=$(cut -d' ' -f1 /proc/loadavg)
+
+  if awk -v l="$load_1m" -v m="$max_load" 'BEGIN { exit !(l > m) }'; then
+    fatal "1-minute load average ${load_1m} exceeds threshold ${max_load} (GATE_MAX_LOAD=${max_load})"
+  else
+    ok "1-minute load average ${load_1m} is within threshold ${max_load}"
+  fi
+  return 0
+}
+
+# (b) any process outside the allowlist pinning more than half a core.
+# Parsed with `read` into named fields, not an awk/cut chain — a fixed-column
+# awk/cut split on `ps` output duplicated fields the first time this was
+# tried, because process args vary in word count. `read` with a fixed list
+# of variables dumps everything left over into the last one (args), which
+# handles that correctly.
+check_processes() {
+  local max_pcpu=50
+  local extra_regex="${GATE_ALLOW_REGEX:-}"
+  local violation_found=0
+  local line pcpu pid args pcpu_int
+
+  while IFS= read -r line; do
+    [ -z "$line" ] && continue
+    read -r pcpu pid args <<< "$line"
+
+    # Kernel threads report their comm in brackets, e.g. "[kworker/0:1]".
+    case "$args" in
+      \[*\]) continue ;;
+    esac
+
+    # This script's own tree: its ancestors (shell, terminal, session) and
+    # its direct children, both captured once by build_ancestor_pids.
+    case " $ANCESTOR_PIDS " in
+      *" $pid "*) continue ;;
+    esac
+
+    case "$args" in
+      *sshd*|*systemd*) continue ;;
+    esac
+
+    if [ -n "$extra_regex" ] && [[ "$args" =~ $extra_regex ]]; then
+      continue
+    fi
+
+    pcpu_int="${pcpu%.*}"
+    if [ -z "$pcpu_int" ]; then
+      pcpu_int=0
+    fi
+    if [ "$pcpu_int" -gt "$max_pcpu" ]; then
+      fatal "pid ${pid} ('${args}') is using ${pcpu}% of one core"
+      violation_found=1
+    fi
+  done < <(ps -eo pcpu,pid,args --sort=-pcpu | tail -n +2)
+
+  if [ "$violation_found" -eq 0 ]; then
+    ok "no process outside the allowlist exceeds ${max_pcpu}% of one core"
+  fi
+  return 0
+}
+
+# (c) cpu0 scaling governor must be "performance". Skipped with a warning
+# (not a violation) when the sysfs path doesn't exist on this machine.
+check_governor() {
+  local gov_path="/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor"
+  if [ ! -r "$gov_path" ]; then
+    echo "WARNING: ${gov_path} not present; skipping governor check"
+    return 0
+  fi
+
+  local governor
+  governor=$(cat "$gov_path" 2>/dev/null) || governor=""
+  if [ "$governor" != "performance" ]; then
+    fatal "cpu0 governor is '${governor}', not 'performance'"
+  else
+    ok "cpu0 governor is 'performance'"
+  fi
+  return 0
+}
+
+# (d) free-space floors on / and /tmp (default 10G each). Skip entirely via
+# GATE_SKIP_DISK_CHECK=1.
+check_disk() {
+  if [ "${GATE_SKIP_DISK_CHECK:-0}" = "1" ]; then
+    echo "WARNING: disk free-space check skipped (GATE_SKIP_DISK_CHECK=1)"
+    return 0
+  fi
+
+  local floor_gb=10
+  local floor_bytes=$((floor_gb * 1024 * 1024 * 1024))
+  local path avail_bytes avail_gb
+
+  for path in / /tmp; do
+    avail_bytes=$(df --output=avail -B1 "$path" 2>/dev/null | tail -n 1 | tr -d ' ') || avail_bytes=""
+    if [ -z "$avail_bytes" ]; then
+      echo "WARNING: could not determine free space on ${path}; skipping"
+      continue
+    fi
+    if [ "$avail_bytes" -lt "$floor_bytes" ]; then
+      avail_gb=$((avail_bytes / 1024 / 1024 / 1024))
+      fatal "${path} has only ${avail_gb}G free, below the ${floor_gb}G floor"
+    else
+      ok "${path} has enough free space (floor ${floor_gb}G)"
+    fi
+  done
+  return 0
+}
+
+echo "Brainy gate preflight"
+echo "----------------------"
+
+build_ancestor_pids
+check_load
+check_processes
+check_governor
+check_disk
+
+echo "----------------------"
+if [ "$VIOLATIONS" -gt 0 ]; then
+  echo "FATAL: gate preflight failed with ${VIOLATIONS} violation(s) — machine is not gate-clean"
+  exit 1
+fi
+
+echo "gate preflight passed — machine is gate-clean"
+exit 0

scripts/gate/vitest-verdict-check.sh

@@ -0,0 +1,158 @@
+#!/bin/bash
+set -euo pipefail
+
+# Brainy Vitest Verdict Check
+# Confirms a vitest run's own summary line is trustworthy before anything
+# downstream treats a green run as green. See scripts/gate/README.md for why
+# (the 2026-08-13 lost-day ledger).
+#
+# Usage:
+#   vitest-verdict-check.sh <log-file> <expected-file-count>
+#   vitest-verdict-check.sh --count-tests <log-file> <minimum-test-count>
+#
+# The first form checks the "Test Files" summary line's total against an
+# exact expected count. The second checks the "Tests" summary line's total
+# against a minimum. Both also fail on any sign the worker pool died
+# mid-run, whether or not a summary line still made it into the log.
+#
+# Exit 0 and print one OK line per passing check when the log is clean.
+# Exit 1 and print one FATAL line per violation, quoting the exact line or
+# string that tripped it, when it is not.
+#
+# Known trap (shared with gate-preflight.sh): every helper below ends on an
+# explicit `return 0` as its own statement, never on a loop or test, so a
+# helper's last command can never hand its own exit status back as the
+# function's under `set -e`. The same applies to `var=$(cmd)` assignments
+# mid-helper: a bare assignment IS checked by `set -e`, so a `grep` that
+# legitimately finds nothing (exit 1) would otherwise kill the script
+# instead of just leaving the variable empty — every such assignment below
+# is paired with an explicit `|| true` inside the substitution.
+
+usage() {
+  echo "Usage: $0 <log-file> <expected-file-count>"
+  echo "       $0 --count-tests <log-file> <minimum-test-count>"
+  exit 1
+}
+
+MODE="files"
+if [ "${1:-}" = "--count-tests" ]; then
+  MODE="tests"
+  shift
+fi
+
+LOG_FILE="${1:-}"
+THRESHOLD="${2:-}"
+
+if [ -z "$LOG_FILE" ] || [ -z "$THRESHOLD" ]; then
+  usage
+fi
+
+if [ ! -f "$LOG_FILE" ]; then
+  echo "FATAL: log file '${LOG_FILE}' does not exist"
+  exit 1
+fi
+
+if ! [[ "$THRESHOLD" =~ ^[0-9]+$ ]]; then
+  echo "FATAL: threshold '${THRESHOLD}' is not a non-negative integer"
+  exit 1
+fi
+
+VIOLATIONS=0
+
+fatal() {
+  echo "FATAL: $1"
+  VIOLATIONS=$((VIOLATIONS + 1))
+}
+
+ok() {
+  echo "OK: $1"
+}
+
+# Vitest colorizes its summary with ANSI escapes; strip them before parsing
+# anything, or the color codes end up embedded in the fields we grep for.
+CLEAN_LOG="$(sed 's/\x1b\[[0-9;]*m//g' "$LOG_FILE")"
+
+# Worker-pool death: if either string appears, the run's own summary line —
+# even if present and even if its numbers look fine — cannot be trusted,
+# because the process died mid-suite and vitest's own accounting is what
+# died with it.
+check_worker_death() {
+  if echo "$CLEAN_LOG" | grep -q "Unhandled Error"; then
+    fatal "log contains 'Unhandled Error' — worker pool died mid-run"
+  fi
+  if echo "$CLEAN_LOG" | grep -q "Timeout calling"; then
+    fatal "log contains 'Timeout calling' — worker pool died mid-run"
+  fi
+  return 0
+}
+
+# Shared shape between the "Test Files" and "Tests" summary lines:
+#   <label>  <n> passed | <n> failed | <n> skipped (<total>)
+# `compare` is "eq" (total must equal threshold) or "min" (total must be at
+# least threshold).
+check_summary_line() {
+  local label="$1"
+  local threshold="$2"
+  local compare="$3"
+  local summary_line total accounted n
+
+  summary_line=$(echo "$CLEAN_LOG" | grep -E "^[[:space:]]*${label}[[:space:]]+" | tail -n 1 || true)
+
+  if [ -z "$summary_line" ]; then
+    fatal "no '${label}' summary line found in ${LOG_FILE}"
+    return 0
+  fi
+
+  total=$(echo "$summary_line" | grep -oE '\([0-9]+\)' | tr -d '()' | tail -n 1 || true)
+  if [ -z "$total" ]; then
+    fatal "'${label}' summary line has no parenthesized total: \"${summary_line}\""
+    return 0
+  fi
+
+  if [ "$compare" = "eq" ]; then
+    if [ "$total" -ne "$threshold" ]; then
+      fatal "'${label}' total is ${total}, expected ${threshold}: \"${summary_line}\""
+    else
+      ok "'${label}' total matches expected ${threshold}"
+    fi
+  else
+    if [ "$total" -lt "$threshold" ]; then
+      fatal "'${label}' total is ${total}, below minimum ${threshold}: \"${summary_line}\""
+    else
+      ok "'${label}' total ${total} meets minimum ${threshold}"
+    fi
+  fi
+
+  accounted=0
+  for n in $(echo "$summary_line" | grep -oE '[0-9]+ (passed|failed|skipped)' | grep -oE '^[0-9]+'); do
+    accounted=$((accounted + n))
+  done
+
+  if [ "$accounted" -lt "$total" ]; then
+    fatal "'${label}' line accounts for only ${accounted} of ${total} — truncated run: \"${summary_line}\""
+  else
+    ok "'${label}' line accounts for all ${total}"
+  fi
+
+  return 0
+}
+
+echo "Brainy vitest verdict check: ${LOG_FILE}"
+echo "----------------------"
+
+check_worker_death
+
+if [ "$MODE" = "files" ]; then
+  check_summary_line "Test Files" "$THRESHOLD" "eq"
+else
+  check_summary_line "Tests" "$THRESHOLD" "min"
+fi
+
+echo "----------------------"
+if [ "$VIOLATIONS" -gt 0 ]; then
+  echo "FATAL: vitest verdict check failed with ${VIOLATIONS} violation(s) for ${LOG_FILE}"
+  exit 1
+fi
+
+echo "vitest verdict check passed for ${LOG_FILE}"
+exit 0