David Snelling
ebe06cdf33
DP6/DP8 of the Path Registry (BRAINY-PROD-LATENCY-TRIAD, the proven flicker mechanism): update paths staged RemoveFromVectorIndex then AddToVectorIndex as two separately-awaited transaction ops — between them a live row was in NEITHER index (dark to semantic recall, fine in metadata list). The native pair widened that window to seconds in production before their side's visibility-commit fix; the structural cure lands here: - hnswIndex.updateItem: absent → add; SAME vector → pure no-op (the production shape — a type-only update re-indexed an unchanged vector, remove+add did pure damage); changed vector → the node NEVER leaves the index: synchronous vector swap first (every query from that instant sees correct distances), then unlink/relink at the node's existing level via shared internals (linkNode/unlinkNodeEdges refactored out of add/remove; entry point and maxLevel provably unchanged). - ReplaceInVectorIndexOperation: ONE transaction leg; feature-detects provider updateItem (native seam flagged — their side ships updateItem, then the adjacent remove+add fallback is dead code). Both update staging sites swapped; delete sites untouched. - LAZY-OPEN GATE (fleet adoption find, SELF-ENGINE-PAIR-STANDARD): under disableAutoRebuild, ensureIndexesLoaded assessed ONLY the vector index — a not-ready native METADATA provider never blocked the completion latch and every find() silently returned [] on a populated store. All three providers now vote; any not-ready report falls through to the rebuild. - docs/path-registry.md: brainy's twin table for the 32 shared path IDs — service class, budgets, lifecycle, narration, and the cited pin per row; owed rows named (LC4 doors-open migration, MT4 yielding heals, LC7 downgrade contract) per the lifecycle-sprint choreography. Pins: update-item-atomic 9/9 (visibility-atomic swap, reverse-index parity vs fresh rebuild, entry-point invariants) · lazy-notready-honor 2/2. Gates: unit 1928/1928 (148 files) · integration 760 · conformance 27/27.
85 lines
9.5 KiB
Markdown
85 lines
9.5 KiB
Markdown
# The Path Registry — brainy's twin table
|
||
|
||
The brainy half of the cross-engine Path Registry (the native accelerator
|
||
maintains the master list; IDs are shared and stable — `LC3`, `DP7`, … are
|
||
citable in commits, board rounds, release notes, and pins). Every row owes
|
||
five things: **service class** (INDEX-SERVED | BOUNDED-FALLBACK, announced |
|
||
TYPED REFUSAL), **latency budget** at 1k/10k/100k/1M (design bar: billions),
|
||
**lifecycle behavior**, **failure narration**, and a **test pin**. A path not
|
||
in this registry does not ship; an unregistered path is a red gate in the
|
||
scan audit.
|
||
|
||
**The availability bar governing every row: user-visible downtime is
|
||
seconds, at restart only.** Migration, heal, compaction, embedding, and
|
||
retention run behind the doors — yielding, budget-capped, narrated. No path
|
||
may hold the doors while it does housekeeping.
|
||
|
||
Status legend: ✅ contracted + pinned (test cited) · 🟡 partial (what holds
|
||
and what's missing, stated) · 🔴 owed (named, never silent).
|
||
|
||
## LC — Lifecycle
|
||
|
||
| ID | Brainy row | Status |
|
||
|----|-----------|--------|
|
||
| LC1 | Same-version reopen adopts everything: brain-format epoch match → zero rebuilds; aggregation state adopts by stamp; persisted indexes load. | ✅ `tests/unit/brainy/brain-format-handshake` + `migration-deference` (no-drift reopen never rebuilds) |
|
||
| LC2 | New empty brain: doors immediate. | ✅ exercised by every suite's setup |
|
||
| LC3 | Upgrade, same epoch: as LC1 — new code on unchanged formats owes nothing at open. | ✅ same pins as LC1 (epoch equality is the gate) |
|
||
| LC4 | Upgrade with epoch migration: TODAY brainy's epoch rebuild runs at open before doors. | 🔴 **owed — the sev's lockout row.** The doors-open-serving-old-structures design (yielding installments + atomic swap) lands measured-and-gated behind the service-class pair, per the lifecycle-sprint choreography. Acceptance case: the 9,184-row hours-lockout. |
|
||
| LC5 | Crash recovery: bounded, resumable, narrated. Aggregation leg ✅ (behind-stamp → incremental catch-up off the fact log + time-travel reconciliation, capped at 5,000 affected before an ANNOUNCED rescan). Vector/metadata legs ride epoch machinery (rebuild-from-canonical, narrated). | 🟡 aggregation pinned (`tests/integration/aggregation-lifecycle-catchup`); the rebuild legs are narrated but not yet installment-yielding (couples to LC4) |
|
||
| LC6 | Shutdown under load: close() drains the background flush flight, tears down cadence timers, runs ONE time-bounded compaction pass (~5s budget, resumable). | 🟡 pinned for flush/compaction (8.9.0 suites); SIGTERM drain budget not yet declared |
|
||
| LC7 | Rollback/downgrade: an N−1 build opening an N brain. | 🔴 owed — no declared read-compat window or typed refusal today (epoch mismatch triggers a rebuild, not a refusal; v2 nested-bag records read as a phantom user field on pre-law builds). Needs the declared-window contract. |
|
||
| LC8 | Relocatable brain directory: no absolute paths in artifacts; persist()/load() round-trips. | 🟡 persist/load pinned; byte-for-byte relocation depot cases are the pair gate's (shared corpora) |
|
||
| LC9 | Double-open: second writer gets a typed lock refusal (PID-liveness + heartbeat stale detection; `force` escape hatch logs loudly). | ✅ writer-lock suites (8.7.1) |
|
||
|
||
## DP — Data plane
|
||
|
||
| ID | Brainy row | Status |
|
||
|----|-----------|--------|
|
||
| DP1 | `get()` by id: direct storage read + hydrate. INDEX-SERVED (id-mapped). Milliseconds at every scale. | ✅ exercised everywhere; budget rides the pair speed table |
|
||
| DP2 | `find({query})`: embed + vector search. The embed dominates (native side owns the budget); JS HNSW serves the search leg. | 🟡 300ms-class p95 is the pair speed-table row; brainy-alone budget declared there |
|
||
| DP3 | Filtered/sorted list: column top-K when the field is columnized (INDEX-SERVED, zero canonical reads on the sorted page — value pairs come from ONE batched metadata-record pass); no-column fallback is BOUNDED-ANNOUNCED (one batch pass, announces once per field past 500 rows); unknown field → TYPED REFUSAL naming both candidate spellings. | ✅ `tests/unit/utils/metadataIndex-sort-callshape` (zero per-row reads, batch-only — latency-blind) + `metadataIndex-nested-orderby` (dotted keys serve-or-refuse) + `tests/integration/orderby-sort-bug` |
|
||
| DP4 | Aggregation/stats: ALWAYS answers. Write-time incremental; behind-stamp reconciles incrementally; genuine rebuilds go through the native parallel door or the paged JS walk; nothing ever latches off; before-image-less deletes flag a LOUD rescan, never a silent skip. | ✅ `tests/integration/aggregation-lifecycle-catchup` + `tests/unit/aggregation/aggregation-provider-rebuild` |
|
||
| DP5 | Graph traversal: `related()` paged via adjacency; whole-graph analytics carry declared cost. | 🟡 paged reads pinned; analytics cost-class declaration owed (rides VENUE-GRAPH-TRUST audit tool) |
|
||
| DP6 | Single write: ack at the canonical commit; visibility committed at ack (the atomic vector update kills the remove→add dark window); maintenance NEVER holds the ack (background flush cadence — THE ACK LAW pin: a hung flush cannot block a write). | 🟡 ack law pinned (`tests/unit/brainy/persistence-policy`); atomic-update pin lands with the flicker fix in this train |
|
||
| DP7 | Bulk ingest: sustained rate holds flat — per-write maintenance taxes must not grow with brain size (A4 removed caller-flush convoys; deferred embedding removes the per-write embed tax where opted). | 🟡 the decay-curve row is a pair speed-table RED GATE; brainy-alone sustained-rate run rides the same corpora |
|
||
| DP8 | Read under write pressure: no flicker window — a row that exists is never invisible to recall, even transiently (same-vector re-index is a no-op; changed-vector swaps in place, node never leaves the index). | 🟡 lands in this train (atomic `updateItem` + `ReplaceInVectorIndexOperation`); symmetry suite + sentinels are the B4 program |
|
||
| — | **The lazy-open gate honors EVERY provider's not-ready report** (a not-ready metadata provider can no longer latch the silent-empty state under `disableAutoRebuild`). | ✅ `tests/unit/brainy/lazy-notready-honor` |
|
||
|
||
## MT — Maintenance (never in the door path)
|
||
|
||
| ID | Brainy row | Status |
|
||
|----|-----------|--------|
|
||
| MT1 | Flush/checkpoint: ENGINE-OWNED cadence (write-count/interval/idle triggers, single-flight, background, loud on failure; callers never flush in hot paths; `flush()` stays as an awaitable barrier). | ✅ `tests/unit/brainy/persistence-policy` |
|
||
| MT2 | Compaction: never on flush (durability-only law, 8.9.0); close-time pass time-budgeted + resumable; explicit `compactHistory({timeBudgetMs})`. | ✅ 8.9.0 suites |
|
||
| MT3 | Index upkeep (mapper folds, delta promotion): native-side machinery; brainy's JS legs are small and synchronous-cheap. | 🟡 declared; yield audit rides the pair |
|
||
| MT4 | Heal/rebuild walks (`repairIndex`, backfill walks): paged; failure latches with cooldown; NOT yet yield-to-foreground installments. | 🔴 owed — the priority-isolation clause (couples to LC4; same choreography) |
|
||
| MT5 | Deferred embedding worker: ack at durability, durable pending markers, crash-recovered at open, single-flight batches. | 🔴 lands as A3 in this train (design frozen on the incident thread) |
|
||
| MT6 | Retention/archival walks: retention `'all'` does nothing by design; bounded-retention reclaim is close-time/explicit only. | 🟡 8.9.0 behavior pinned; archival profile is the co-frozen D1+D3 unit |
|
||
|
||
## FM — Failure modes
|
||
|
||
| ID | Brainy row | Status |
|
||
|----|-----------|--------|
|
||
| FM1 | Disk full / IO error mid-op: transaction rollback + typed error; failed rollback → StoreInconsistentError quarantines writes until repairIndex(). | 🟡 rollback paths pinned; explicit disk-full depot case owed |
|
||
| FM2 | Memory pressure: query limits + reserved-memory config; unified cache eviction. | 🟡 declared budgets; cascade pin owed |
|
||
| FM3 | Torn/corrupt file on open: malformed brain-format marker → safe rebuild (never trusting a bad epoch); corrupt records surface loudly. | 🟡 marker pin ✅ (`brain-format-handshake`); broader quarantine is native-side |
|
||
| FM4 | Native module unavailable: plugin load failure is LOUD (version-coupling law throws on range mismatch — never silently version-drifted); JS engine serves with its own declared budgets, named as the active backend in op names. | ✅ `tests/unit/plugin-version-coupling` + op-name stamping |
|
||
|
||
## FL — Fleet
|
||
|
||
| ID | Brainy row | Status |
|
||
|----|-----------|--------|
|
||
| FL1 | Cold open on demand: LC1's adopt-everything open; warm() available for eager paths. | 🟡 open cost pinned at LC1; millisecond budget rides the speed table |
|
||
| FL2–FL4 | Boot storm / upgrade wave / isolation: fleet-layer policies over LC1/LC4 — engine leg = budgeted opens + LC4's behind-doors migration. | 🔴 owed with LC4 |
|
||
| FL5 | Brain as product object: create instant (LC2) · erase = `clear()` explicit + complete · export = portable-graph, canon-complete mode available. | ✅ clear-persistence + portable-graph + canonical-enumeration suites |
|
||
|
||
## Status summary
|
||
|
||
Contracted + pinned this train: **DP3, DP4, MT1, LC5(aggregation), the
|
||
lazy-open not-ready gate, LC1/LC3/LC9, FM4, FL5** — each with the cited
|
||
test. Landing in this train: **DP6/DP8 (atomic vector update), MT5 (A3
|
||
deferred embedding)**. Owed, in production-risk order, all coupled to the
|
||
priority-isolation program the lifecycle sev opened: **LC4 (doors-open
|
||
migration), MT4 (yielding heals), LC7 (downgrade contract), LC6 (SIGTERM
|
||
budget), FL2–FL4, FM1/FM2 depot cases.** Rows move from owed to contracted
|
||
only with a cited test — none lands by prose.
|