Sweeping the vestigial hnsw sharding machinery surfaced two real defects on
the counts-recovery path (the code that rebuilds totalNounCount/totalVerbCount
when counts.json is lost or corrupted — container restarts, partial copies):
- initializeCountsFromDisk counted files in entities/*/hnsw/ — directories the
8.0 write path never populates — so an established store recovered to ZERO
counts on real data: wrong getNounCount()/stats, a "New installation"-style
boot log, and a mis-sized rebuild-strategy decision at open. The scan now
walks the canonical entities/<kind>/<shard>/<id>/ tree (one entity per id
directory), with the same sampled type-distribution estimate as before.
- The sampler called getNounMetadata — whose ensureInitialized() re-enters
init() — from INSIDE init(), a latent deadlock reachable the moment the scan
found anything to sample. The sampled metadata files are now read directly
with fs (gz-transparent), no guarded accessors inside init.
The dead machinery itself (verified zero callers by reachability analysis
before deletion): the nine 7.x hnsw-layout entity/edge CRUD methods, the
sharding-depth prober + its depth-migration engine (unreachable — the probe
always returned null on 8.0 stores), an orphaned streaming verb paginator,
their path/scan helpers, and the now-unused type aliases and fields. The init
block keeps the truthful new-vs-established boot log introduced in 8.0.13,
now decided directly from the canonical layout. Net -1,138 lines; no public
API touched.
Adds a regression test: delete counts.json from a populated store, reopen,
counts recover exactly from the canonical tree.
The fast-follow to the ifRev CAS fix: the sweep for the same check-then-act
class found two more instances, both now closed with the same discipline —
the decision runs at the serialization point that guards the apply.
add({ifAbsent}) / add({upsert}): the absence check ran before the commit
mutex, so N concurrent same-id creates could all pass it and all write — the
second silently overwriting the first, violating ifAbsent's "return the
existing id WITHOUT writing" contract and upsert's merge-never-clobber
contract. The insert leg now carries a must-be-absent commit precondition
(the same conditional-commit primitive ifRev uses), verified under the commit
mutex against the authoritative before-image. A losing caller takes its
documented resolution instead of overwriting: ifAbsent returns the existing
id with zero writes; upsert merges into the now-existing entity via update()
(shared param mapping in upsertMergeParams so the planning-time branch and
the conflict retry can never drift), with a bounded retry if a concurrent
delete lands between the conflict and the merge. The planning-time checks
remain as fast-fails. transact()'s batch-level ifAbsent/upsert keep
planning-time semantics (documented, converges to a valid entity).
BlobStorage: write()'s dedup decision (exists → increment refCount, absent →
create at 1) and delete()'s decrement-then-remove were unserialized
read-modify-writes over blob-meta:<hash>. Concurrent writes of identical
content could lose references, so a later delete removed bytes another file
still referenced (data loss), or leaked unreferenced blobs. All
reference-count-bearing mutations now serialize through a per-hash
InMemoryMutex — distinct content never contends; incrementRefCount/
decrementRefCount are documented lock-assumed internals.
Both fixes are complete by construction in-process: storage enforces
single-writer-per-directory, so the process is the whole concurrency domain.
Tests (tests/integration/ifabsent-upsert-blob-concurrency.test.ts): 8-way
ifAbsent storm advances the store by exactly one generation with _rev 1;
8-way upsert storm on an absent id yields one create + seven merges
(_rev === 8); sequential ifAbsent/upsert semantics pinned unchanged; N
identical concurrent blob writes → refCount === N; the blob survives until
the true last reference drops; interleaved write/delete storm never loses a
landed reference.
N concurrent update({ ifRev }) calls carrying the same expected revision all
fulfilled (zero RevisionConflictErrors, last-writer-wins) — the check ran
before the commit mutex, so interleaved callers all passed it before any apply
landed. Sequential calls conflicted correctly, which hid the race. A production
cutover rehearsal caught it: 8 parallel ifRev-guarded ledger writes all
"succeeded" and 7 were silently lost. Advisory locks built on exactly-one-winner
semantics could hand the same lock to two workers.
Fix: conditional commit. commitSingleOp and commitTransaction accept an
optional precommit(beforeImages) precondition, invoked under the commit mutex
against the just-read authoritative before-images and before anything is staged
or applied — the per-record analogue of ifAtGeneration, which always ran there.
A throw aborts the commit atomically (generation reservation returned, zero
staging I/O in the transaction path). The store stays domain-ignorant; update()
and transact() supply the ifRev predicate:
- update(): the planning-time check remains as a fast-fail (avoids embedding
cost on an obviously stale expectation); the authoritative check re-verifies
the before-image's _rev and re-stamps the update's _rev from it, so the
counter is monotonic even for concurrent non-CAS updates (N plain updates
advance _rev by N). CAS against a concurrently-removed entity now throws
EntityNotFoundError instead of silently resurrecting it; plain updates keep
their last-writer-wins re-create semantics.
- transact(): per-op ifRev expectations registered during planning
(PlannedTransact.casUpdates) are re-verified in the batch's precommit; any
conflict rejects the whole batch before staging. Multiple updates to one
entity sequence through a running rev; ids the batch itself adds
(PlannedTransact.createdNouns — add resets the rev baseline even on
overwrite) keep their exact plan-time sequencing.
Regression suite (tests/integration/ifrev-concurrent-cas.test.ts): 8 parallel
same-rev updates → exactly 1 winner + 7 conflicts; same for transact batches;
_rev monotonicity; the documented read→CAS→retry ledger loop converging exactly
(8 workers, 8 decrements, 0 lost); sequential behavior unchanged; forward-ref
add+update in one batch unchanged.
The one-time 7→8 layout migration rescues the head branch's entities
(branches/<head>/entities/* → entities/*), then drained the entire
branches/<head>/ directory. Any durable state a 7.x engine wrote under the head
branch OUTSIDE entities/ — a branch-scoped index, blob area, or field registry —
would be silently deleted by that drain, the same failure class as the VFS
content blobs a 7.x store kept in _cow/.
Guard it: after the entity move, list what remains under branches/<head>/ and
exclude entities/. If anything survives, it is non-entity durable state, so
PRESERVE the branch (skip removeRawPrefix) and warn loudly with the leftover
keys — recoverable, not lost. A clean branch (only the moved entities) still
drains exactly as before. Adds a PARITY GUARD test to the 7→8 migration suite.
Every persisted 8.0 store logged "📁 New installation: using depth 1 sharding"
on every open — even brains holding thousands of entities — which is alarming to
read during a restart or incident. 8.0 stores nouns in the canonical
`entities/nouns/<shard>/<id>/vectors.json` layout (the path `saveNoun`/`getNouns`
use), but the legacy sharding probe `detectExistingShardingDepth()` inspects
`entities/nouns/hnsw` — a 7.x directory the 8.0 write path never populates (its
only writer, `saveNode`, is dead code with zero callers). So the probe returned
null for every 8.0 store and concluded "new".
Drive the new-vs-existing log from the layout the database actually reads and
writes: a new `hasCanonicalEntities()` checks for a real 2-hex shard directory
under `entities/nouns/`, and the known noun count short-circuits it. An
established store now logs "Using depth 1 sharding (N entities)"; only a genuinely
empty store reports a new installation. Behavior is otherwise unchanged — the
probe only ever set a log line, never triggered a rebuild or migration.
The 7.x branch system stored Virtual Filesystem content as blobs in its
copy-on-write area (`_cow/`). 8.0 removed that system and stores content blobs
in the content-addressed store (`_cas/`), but the one-time layout migration only
moves entities — it never adopted the `_cow/` content blobs. A store that used
the VFS was left with every VFS-backed read throwing "Blob metadata not found"
and its pages 500ing, with no first-party recovery and the pre-upgrade backup
already removed on entity-migration "success".
Add an on-open recovery pass (`autoAdoptLegacyVfsBlobsIfNeeded`) that runs right
after the layout migration and adopts every orphaned `_cow/` blob into `_cas/` —
copying both the bytes and the metadata via the raw object primitives,
idempotently and non-destructively (the `_cow/` originals are never deleted). It
is a separate phase, not folded into the migration, so it also heals a store
already upgraded by an earlier 8.0.x that stranded the blobs (whose layout-
migration marker is stamped): it is gated on the presence of `_cow/` and its own
`_system/vfs-blob-adoption.json` marker. Filesystem-only; native-8.0 and non-VFS
stores no-op on a cheap existence check.
- `BaseStorage.adoptLegacyCowBlobs()` — the scan/copy primitive, returning
`{ cowBlobs, adopted, alreadyPresent, incomplete }`; skips (does not half-adopt)
a blob missing its bytes or metadata.
- `brain.vfs.adoptOrphanedBlobs()` — the explicit force path; self-initializes.
- Backup retention: the automatic pre-upgrade backup is now kept if any blob
can't be fully adopted (`incomplete > 0`), instead of being removed on
entity-migration success alone — a blob-parity gate the migration signal
cannot provide.
Adds an end-to-end integration test (storage-level adopt with idempotency and
incomplete handling; self-heal on reopen; the explicit API) and an upgrade guide.
`find({ where })` had two gaps that could silently return nothing. The in-memory
matcher (`matchesQuery`, used for egress re-validation and historical reads) was
missing several documented operators — `in`, `greaterThanOrEqual`,
`lessThanOrEqual` — so it disagreed with the index path, which does handle them.
And an unknown operator key (a typo, or `notIn` written where `not: { in }` was
meant) fell through to a nested-object-field interpretation and matched nothing.
- Align `matchesQuery` to the full documented operator set (add the missing
aliases; the default branch now throws instead of treating an unknown key as a
nested field — dotted paths remain the supported nested form).
- Validate the `where` filter up front (`validateWhereFilter`), throwing a typed
`BrainyError('INVALID_QUERY')` that names the unknown operator, recursing into
`allOf` / `anyOf` / `not`.
- Stop excluding a user field literally named `level` from the metadata index
(it collided with an internal never-index key), so numeric/exact filters on it
resolve instead of returning 0.
Adds unit coverage for the operator set and the throw-on-unknown behavior.
- data-storage-architecture.md described transact() as 'the unit of history'
(rc.2 era). Single-op retention has been the model since 8.0 — every write
gets its own generation; transact() only groups several into one. Corrected.
- RELEASE-GUIDE.md now records the hard ordering constraint: no embedding-model
change ships before vector model-version stamping + hard-error-on-mismatch
lands.
A production deployment measured ~48 seconds on EVERY reopen of an
11k-entity brain. Root cause: brainy's rebuild gate decided from in-memory
size()/count, which read 0 for a durable-but-not-resident index, so it
re-read every entity file to rebuild from scratch. At GA we gave only the
GRAPH provider a readiness contract (init() eager cold-load + isReady()
honest signal) so it would never eat that spurious rebuild; the vector and
metadata providers never got it, and brainy never even eager-inited the
vector provider.
Complete the contract symmetrically:
- plugin.ts: VectorIndexProvider gains optional init()+isReady();
MetadataIndexProvider gains isReady() — mirroring GraphIndexProvider.
Additive and optional; a provider that exposes nothing keeps today's
behavior.
- brainy.ts: eager-init every provider that exposes init() (after metadata
init() so the id-mapper is hydrated first), then decide per leg in
precedence order — migrating (skip) -> epoch drift (rebuild) -> isReady()
-> a per-leg empty fallback. The old instant fast-path keyed off
this.index.size()>0, a dishonest proxy that skipped the metadata/graph
checks whenever the vector was warm and never fired on a real cold process
anyway; removed.
The per-leg fallbacks differ because "empty" means different things: the JS
vector's rebuild() IS its load, so size()===0 correctly triggers it; the
id-mapper backs metadata, so totalEntries===0 (past the empty-store return)
is a real load failure; but entities do not imply edges, so a graph
size()===0 is a valid empty state, not a load failure.
- The JS graph now COLD-LOADS its durable LSM instead of re-deriving from a
full canonical verb scan on every boot (baseStorage._initializeGraphIndex
loads the persisted SSTables via a new GraphAdjacencyIndex.init(); it
self-heals from canonical only when the durable state is genuinely missing).
This removes an O(E)-per-open cost every filesystem consumer paid.
- LSMTree.loadManifest loads its SSTables BEFORE publishing the relationship
count, and resets to an honest-empty state on load failure — a tree can no
longer claim persisted relationships while holding none (the silent-empty
cold-load class the query-time guards exist to prevent).
Verified end-to-end against a built brain: a warm reopen (with edges and
edgeless) reloads only the JS vector; the graph and metadata cold-load with
no rebuild, and queries return correct results. New tests in
cold-open-rebuild-gate.test.ts pin the contract (isReady() defers, self-heal
still fires); migration-deference updated to drive size-based deference
through the vector, the leg where empty->rebuild remains correct.
Pairs with the native provider's isReady()/init() implementation — brainy's
gate defers only to a signal the provider exposes.
A consumer's clean-room verification of the 8.0.10 fix found relate() still
hanging their scripts. Root-causing the CLASS instead of the repro found two
mechanisms:
1. The 'beforeExit' auto-flush hook looped forever on any script that never
reaches close(): Node re-emits beforeExit after every event-loop drain, and
the async flush schedules new work — flush, drain, flush, forever. The
listener now self-deregisters BEFORE its one flush, so the next drain exits.
(Empirically: process.on('beforeExit', async () => await anything) alone
never exits — this was the deepest root of the whole hang class.)
2. Every background-maintenance interval is now unref'd at creation — graph
auto-flush, LSM compaction, metadata write-buffer flush, VFS cache
maintenance, PathResolver cache maintenance, statistics debounce (the
writer-lock heartbeat, flush watcher, and cache monitors already were).
Durability is owned by close() and the beforeExit flush, both deterministic;
a best-effort interval must never keep the host process alive.
Proof: the reported shape (add x2 + relate, filesystem) exits cleanly BOTH
with close() (~0.5 s) and with NO teardown at all — and in the no-teardown
case the beforeExit flush still lands the data (verified by reopen: both
nouns + the edge present). New per-op-class sweep test asserts no ref'd
timer survives close() for add / relate / graph find / metadata update /
vfs — turning this bug class off permanently instead of per-repro.
A consumer report on the GA pair: any minimal add/find/close script hangs
forever. Root cause was FOUR ref'd keep-alives brainy never released:
1+2. The global SIGTERM/SIGINT shutdown hooks (registered once at init) were
anonymous and never removed — a process.on signal listener holds a ref'd
signal handle. They are now named statics, deregistered when the LAST live
instance closes (Brainy.instances also stopped leaking closed instances);
a later init re-registers them.
3. UnifiedCache.startFairnessMonitor() created an interval it never stored,
cleared, or unref'd — unclearable by construction. Now stored + unref'd
(same posture as the existing memory-pressure timer): a cache monitor must
never keep the host process alive.
4. brainy.close() never called VirtualFileSystem.close(), stranding the VFS
background-maintenance interval and the PathResolver maintenance interval.
close() now shuts the VFS down.
Proof: a bare script (init/add/close, no process.exit) exits 1 ms after
close() returns; before the fix it hung until killed. 3 new lifecycle tests
pin the hook semantics (once globally, survive while other instances live,
removed on last close, re-register after).
With plugins unset (the default), init() probes for the first-party
accelerator: not installed means plain brainy with zero noise; installed and
healthy means it loads and announces itself; installed but broken (import
failure, invalid shape, failed activation, version mismatch) makes init()
THROW. An installed accelerator never silently vanishes behind the JS engines
- the anti-drift posture of the explicit list, applied to detection.
plugins: []/false stays a true opt-out (no probe); an explicit list keeps its
required-and-loud semantics. The import runs through an importPluginPackage
seam (variable specifier - bundlers cannot static-resolve the optional
package; tests simulate all outcomes without it installed).
Also: when a plugin activates but registers zero native providers (e.g. a
licensing gate declining to engage), the provider summary now warns loudly
instead of leaving every query silently on the JS engines.
Supersedes 8.0.8's explicit-opt-in wording; README/PLUGINS/types now document
the guarded contract. 7 new tests (tests/unit/plugin-autodetect.test.ts).
A clean-room install smoke of the published GA pair caught the README telling
scale-up users the native provider is auto-detected. It is not, by design: the
loader (loadPlugins) does no auto-detection — undefined means no plugins, and
only an explicit plugins: ['@soulcraft/cor'] loads the native engine (loud
failure if it can't). The stale comment on the plugins config field in the
public types claimed auto-detection and is corrected to match the loader; one
phrase in the plugin-author guide likewise.
Rewrites the README around the feature-showcase structure: write->index->query
table, runnable quick start (subtype-correct on the 8.0 strict default, real
VerbTypes), feature tour with per-pillar code, the scale-up path to
@soulcraft/cor, and measured-only performance claims with benchmark citations.
GA retargets to 8.0.1: npm permanently retired the 8.0.0 version number after
a January development-cycle publish/unpublish, so the first stable 8.x release
is 8.0.1. RELEASES.md documents this; the phantom 8.0.0 CHANGELOG entry is
removed (the release run regenerates it as 8.0.1).
The 3.0 native engine ships as @soulcraft/cor (brainy 8.x <-> cor 3.x are a
version-matched pair); the old @soulcraft/cortex package stays on the 2.x/7.x
line. Public docs and .d.ts-visible comments now name the correct package.
Historical 2.x contract references (e.g. the 2.3.1 read-side fallback) keep
the old name deliberately.
Brings the u64-id core to main as a merge of feat/8.0-u64-ids. main's tree now
matches the gated 8.0 line exactly; the 7.31.8-7.33.5 hotfix history is retained
as the merge's first parent (every one of those fixes is already in the 8.0 line).
cor confirmed the migration write-new / tmp+rename invariant the hard-link backup
relies on — with one exception: the native shared mmap `_id_mapper/*` is mutated
IN PLACE (msync + a rebuild truncate+re-inject), the same category as the tx-log.
A hard-linked snapshot of it would be corrupted by a live in-place mutation
reaching through the shared inode, so the backup would not be a faithful
pre-upgrade copy.
Add a SNAPSHOT_BYTE_COPY_DIRS set (`_id_mapper`) so snapshotToDirectory byte-copies
every file under it (the directory analogue of SNAPSHOT_BYTE_COPY_PATHS); the id
map is bounded, so the copy cost is small. Everything else (SSTables / .dkann /
manifests / objects, all tmp+rename) still hard-links. Test: `_id_mapper/*` gets
an independent inode + faithful content; a regular file stays hard-linked.
Companion to 7.33.4's graph cold-read guard, now for the metadata field index. A
downstream deployment reported find({where}) returning a silent [] on a
freshly-opened brain (a native metadata provider that reports data but hasn't
loaded its field postings), blanking filtered pages after every restart.
verifyMetadataLive() — one-shot per brain on the first filtered find(): probe a
known persisted field value; if served, live (one O(1) probe on a warm brain). If
not, rebuild from canonical + re-probe; if still unserved, throw the new exported
MetadataIndexNotReadyError rather than let a silent [] pass. Inconclusive cases
(empty store, no plain field, shared-store foreign entity) resolve to live, no
false rebuild. Also exports BrainyError + GraphIndexNotReadyError (were internal).
The JS index cold-loads correctly; this guards the native path (durable cure is
cortex-side, same shape as the graph 2.7.8 cure). 4 unit tests. tsc 0, guard 4/4,
full unit 1538 pass (+1 pre-existing flaky VFS perf test, green in isolation).
A downstream deployment reported find({where}) returning a silent [] on a
freshly-opened brain (a native metadata provider that reports data but hasn't
loaded its field postings), blanking filtered pages after every restart. Brainy
had a cold-read guard for GRAPH reads (verifyGraphAdjacencyLive → self-heal or a
loud GraphIndexNotReadyError) but no equivalent for metadata `where`.
Add verifyMetadataLive() — the field-index counterpart, one-shot per brain on the
first filtered find(): probe a known persisted entity's plain field value; if the
index serves it, live (the only cost on a warm brain — one O(1) probe). If not,
the postings didn't load: rebuild from canonical + re-probe; if still unserved,
throw the new exported MetadataIndexNotReadyError rather than let a silent [] pass.
Inconclusive cases (empty store, no plain field, shared-store foreign entity,
migrating provider) resolve to live — never a false rebuild.
The 8.0 open-core JS index already cold-loads correctly (verified: cold where 3/3,
cold related 2/2) — this guards the NATIVE path, where the durable cold-load cure
is cortex-side (same shape as the graph 2.7.8 cure). 4 unit tests (warm no-rebuild,
self-heal, loud-fail, no-filter-no-probe). Gates: typecheck 0, build 0, test:unit
1757/1757.
David's ask: back up the brain before the one-time upgrade, drop it after. On
open, when the on-disk format is stale (a 7.x→8.0 rebuild will run) and the store
holds data, brainy hard-link-snapshots the brain dir to a sibling
`<brainDir>.migration-backup` BEFORE any provider rebuilds — near-zero cost/space
(shared inodes; the store is immutable-by-rename), instant even at scale. Removed
automatically once the upgrade verifies + stamps the marker; retained on failure
for rollback. Default-on; opt out with `migrationBackup: false`.
- Reuses the existing snapshotToDirectory() hard-link farm via new optional
adapter methods createMigrationBackup()/removeMigrationBackup() (filesystem
only — feature-detected; memory + non-fs are a graceful no-op).
- Taken before the native provider inits, so it captures true pre-migration
state; a prior failed upgrade's backup is reused, not overwritten.
- Best-effort: a backup failure never blocks the upgrade (the migration is itself
safe — reads canonical, reconstructable, self-heals). Catastrophe-insurance
against a migration bug, not a data-loss guard or an off-device backup.
4 lifecycle tests (adapter hard-link / reuse / remove-without-touching-live /
empty→null; stale-epoch reopen create+remove; opt-out; memory no-op). Gates:
typecheck 0, build 0, test:unit 1753/1753, layout-migration suite 5/5.
The Phase-0 CI was red on all three runtimes: `src/embeddings/wasm/pkg/` (the
wasm-pack output the dynamic `import('./pkg/candle_embeddings.js')` resolves) was
gitignored + untracked, so tsc failed to resolve it on a fresh clone (TS2307), and
the Bun job ran test:bun without building dist first.
Fixes:
- Track the 3.1 MB prebuilt pkg (candle_embeddings.js/.d.ts + _bg.wasm/.d.ts). It
ships in the npm tarball anyway; versioning it lets consumers + CI build without
a Rust/wasm-pack toolchain and makes the shipped artifact reproducible (closes
the "publish ships whatever the maintainer last built" supply-chain gap). The
top-level .gitignore intended this (`!*.wasm` etc.) but excluded the whole dir,
so the re-includes were dead, and a nested wasm-pack `.gitignore *` also masked
it — force-added past both.
- CI Bun job: `npm run build` before `test:bun` (it imports the built dist/).
Verified against a tracked-files-only tree (fresh-clone sim): typecheck 0, build 0,
test:bun 8/8.
Make the compiler config honestly describe a Node 22 / Bun 1.1+ / Deno library:
target ES2020 -> ES2023, lib [DOM, ESNext, DOM.Asynciterable] -> [ES2023], drop
the now-inert downlevelIteration. 8.0 dropped the browser path, so the DOM lib
was misleading (it implied web APIs the package no longer supports). No
ES2021-2023 syntax is used anywhere, so .js emit is effectively unchanged — this
is hygiene, not a perf/behavior change.
The one real consequence of dropping DOM: the Cloudflare-Workers edge probe read
globalThis.caches from the DOM lib — declared `caches?: unknown` locally on the
runtime-globals intersection alongside Deno/Bun/HTMLRewriter.
.d.ts is shape-stable (TS-5.x-parseable preserved — no cor coordination). Gates:
typecheck 0, cli-tsconfig 0, typecheck-tsconfig (the .d.ts contract surface) 0,
build 0, test:unit 1753/1753, test:bun 8/8.
Rewrite the four open-core distance functions (cosine / euclidean / manhattan /
dot-product) from object-accumulating `reduce` to single-pass allocation-free
indexed loops. cosine's per-element `{dotProduct,normA,normB}` object was the
hot-path GC lever.
MEASURED (tests/benchmarks/distance-microbench.mjs, dim=384, N=20000, median of
41): cosine 44.3ms -> 7.4ms (~6x), euclidean 9.2ms -> 6.6ms (~1.4x); the built
cosineDistance drops ~44ms -> ~9ms. Numerically identical (same ops, same order)
so recall is unchanged; full suite green (1753/1753).
Also drop the unfounded perf JSDoc ("faster than GPU", "Node.js 23.11+") and the
`new Function(distanceFn.toString())` eval in calculateDistancesBatch — with the
functions now tight loops, the batch is a thin JIT-inlined map (no worker, no
stringify/reconstruct).
Evidence-revised scope: the Float32Array resident-storage half of the original
Fork X is DROPPED. The same microbench shows Float32Array is ~1.7x SLOWER for
this compute (V8 widens f32 -> f64 on every element read), so it would regress
the hot path for a RAM win the open-core JS path does not need — billion-scale
vector RAM is the native provider's SIMD/mmap/quantized domain. The resident
representation stays number[]; no type-chain or cache changes.
Replaces rc.8's no-freeze deference with an automatic, observable, coordinated
migration LOCK (David's reversal: "unknown/halfway states are more dangerous
than blocking"). While a native provider runs its one-time 7.x→8.0
rebuild-from-canonical (`isMigrating()===true`), brainy blocks/queues data-plane
reads AND writes so no operation ever touches a half-built index — closing the
three seam-map gaps (lost mid-rebuild writes, degraded reads, read-time rebuild).
Mechanism (one choke point):
- awaitMigrationLock() at ensureInitialized() covers all ~40 data-plane methods;
a non-migrating brain pays one boolean check. Polls isMigrating() at 250ms;
after migrationWaitTimeoutMs (default 30s) throws a retryable, exported
MigrationInProgressError. The timeout bounds the CALLER'S WAIT, not the
migration — the rebuild is unbounded and never interrupted.
- init() awaits the lock before VFS bootstrap + before serving ("not ready until
upgraded"); a rebuild past the budget surfaces MigrationInProgressError at init
(raise the budget, or run the offline migrator).
Observability (readiness-probe correct — never gated):
- getIndexStatus() gains `migrating` + `migration` (MigrationProgress); a probe
maps migrating→HTTP 503+Retry-After, not 500. health()/checkHealth() lock-exempt
(health() reports a `warn` migration check). stampBrainFormat()/close() are
ungated so cor can clear the lock — no deadlock.
- Optional provider migrationStatus() is relayed verbatim for a live %.
verifyGraphAdjacencyLive() honors the lock (no self-rebuild mid-migration); the
stamp is still withheld while any provider migrates (cor stamps after verify).
Adversarially verified: fixed a critical init/VFS-bootstrap deadlock, a mixed-
provider busy-spin (dropped the event-driven signal → pure poll), a not-yet-
initialized getIndexStatus crash, and once-per-window log/clock resets. 10 lock
tests (incl. the init-during-migration regression). Gates: typecheck 0, build 0,
test:unit 1753/1753.
Consumer-invisible modernization pass — no public API or runtime-behavior
change; dist for the override-only files is byte-identical.
Toolchain:
- CI: GitHub Actions matrix — Node 22/24 (test:unit) + Bun latest (test:bun).
- engines: node ">=22" (was "22.x"), bun ">=1.1.0".
- tsconfig: isolatedModules + noImplicitOverride; add the 33 `override`
modifiers the flag requires across storage/integrations/vfs/transaction.
- deps: @types/node ^22; add prettier; drop dead standard-version,
@rollup/plugin-* and the redundant embedded eslintConfig (flat
eslint.config.js is the active config — verified identical lint output).
paramValidation: replace the top-level `await import('node:os'/'node:fs')`
with static ESM imports. The top-level-await form poisoned the module graph;
static imports also drop the browser/edge fallback branches no supported
runtime reaches (8.0 is Node/Bun/Deno-only).
Bun positioning: recommend Bun as a runtime (`bun add` / `bun run`), which is
green (test:bun 8/8). Drop single-binary `bun build --compile` as a target —
native addons cannot embed into it, and Bun 1.3.10 has a `--compile` codegen
regression around top-level await. Rename the Bun test to bun-runtime-test.ts
and correct docs that overclaimed single-binary support.
Gates: typecheck 0, build 0, test:unit 1743/1743, test:bun 8/8.
