Reconciles the vector-index rename to the ruled three-layer naming: the
provider contract's identity field is now a REQUIRED readonly name (was
optional providerId), self-reported and truthful, rendered as
[vector-index:<name>] where the index identifies itself and stamped into
the op-name strings journals already parse (AddToVectorIndex(<name>)).
The built-in JS engine names itself hnsw-js. A runtime provider instance
compiled against the previous optional contract is tolerated - never
crashed on, never silently mislabeled: it stamps unknown-provider and
emits one loud warning naming the missing field. Graph index operations
keep their static names (they never interpolate provider identity), and
no public API exports a provider-routed hnsw-carrying name, so no
deprecation shim is required.
Three pieces addressing the cold-restart-write incident where a production
deployment's first writes after every restart (33-35s each on a cold page
cache) blew the op-count-scaled transact budget mid-batch: every write is
itself a multi-op transaction, so one cold operation consumed the whole
budget, the gate before the next operation tripped, and the write rolled
back atomically - refused, retried, and refused again until the page cache
warmed passively.
- The budget's start-gating contract is now explicit and pinned: it gates
STARTING the next operation, never rolling back completed work for
elapsed time (the shipped schedule since 8.7.0, now stated in contract
JSDoc, guarded by code for operation 0, and enforced by regression
tests). The 30s floor is configurable via transactionBudgetFloorMs for
stores whose cold operations legitimately run long.
- New brain.warm() eagerly loads the vector index, metadata index, and
graph adjacency so first operations after a cold restart run at
steady-state cost. Returns a WarmReport with an honest per-surface
outcome (warmed / probed / unavailable) - never reports a probe as a
warm. warmOnOpen: true runs it during init(). New optional provider hook
warm() on the vector and graph plugin contracts.
- Vector-index transaction op classes renamed from the backend-specific
AddToHNSWOperation / RemoveFromHNSWOperation to backend-neutral
AddToVectorIndexOperation / RemoveFromVectorIndexOperation, stamping the
active backend into the emitted op-name string (AddToVectorIndex(js-hnsw)
vs a native provider's own identity) so journals never misdirect an
operator toward an index that isn't running.
transact() promises atomic forward references — add an entity and relate to
it in one batch — but the planner resolved relationship endpoint ints at
PLAN time, before the batch's add operations had applied. Asking the id
mapper about an entity that does not exist yet made the (correctly strict)
native mapper throw in EntityIdMapper.getOrAssign, so
transact([{op:'add', id:X}, {op:'relate', to:X}]) failed on native
deployments; the permissive JS mapper masked the bug and silently leaked an
id assignment whenever a batch was later rejected by a commit precondition
(normal control flow since the conditional-commit CAS landed).
Make endpoint resolution lazy: AddToGraphIndexOperation and
RemoveFromGraphIndexOperation take VerbEndpointInts — an eager
{sourceInt, targetInt} or a thunk evaluated when the operation EXECUTES,
mirroring the constructor's already-lazy generationFn. The four
transact-planner sites (relate, its bidirectional reverse edge, remove's
relationship cascade, unrelate) pass thunks, so resolution happens inside
the commit after same-batch adds have applied; the seven single-operation
sites keep eager objects (their endpoints provably pre-exist — relate
validates both, unrelate/updateRelation/remove read the existing verb).
The remove operation's rollback captures the ints resolved at execute so
its re-add uses the same mappings.
Byproduct fix: a rejected batch no longer pollutes the id mapper — the
regression suite pins this (mapper has no assignment for the phantom
entity after a precondition-rejected forward-ref batch), alongside the
exact reported shape, both-endpoints-in-batch, bidirectional,
add+relate+remove in one batch, and the split-transact control
(tests/integration/transact-forward-ref-graph.test.ts).
Graph time-travel needs an edge's existence recorded per generation so
db.asOf(g) hops resolve historically correct endpoints. The metadata layer
already threads brainy's commit generation per write; the graph write path did
not, leaving a versioned verb-endpoint store unable to answer "which edges
existed at generation g".
- GraphIndexProvider.addVerb/removeVerb gain a `generation: bigint` parameter
(the same watermark the storage layer stamps onto the record). A provider with
a per-generation edge chain stamps the edge at that generation; the JS baseline
has no such chain and accepts-and-ignores it — graph time-travel is a
native-provider capability, and the open-core path serves edges as-of-now (the
one documented graph time-travel limitation).
- The two graph transaction operations resolve the generation via a thunk at
EXECUTE time: the generation store assigns the batch generation only once the
commit begins executing, after the operations are planned. The same generation
is reused for an operation's rollback half.
- All graph-write call sites pass the in-flight generation.
Adds a spy-provider test proving the threading, execute-time resolution, and
forward/rollback generation reuse. The JS index ignores the value, so behaviour
is unchanged: unit 1402/1402, db-mvcc 25/25, bigint-contract relate/unrelate 10/10.
The COW version-control surface (fork, branches, checkout, commit,
getHistory/streamHistory, asOfCommit, brain.versions) is gone, along with
its subsystems: src/versioning/, the COW object store (CommitLog,
CommitObject, RefManager, TreeObject), HistoricalStorageAdapter, and the
TypeAwareHNSWIndex path it kept alive. The Db API (now/transact/asOf/
with/persist/restore) is the one versioning model in 8.0.
Survivors and replacements:
- BlobStorage survives (the VFS stores file content through it), relocated
to src/storage/blobStorage.ts with binaryDataCodec.ts; its adapter
interface is now BlobStoreAdapter, slimmed to the consumed surface
(write/read/has/delete/getMetadata + MIME-aware compression policy).
- brain.migrate() backup branches are replaced by persist-before-migrate:
MigrateOptions.backupTo persists a hard-link snapshot of the current
generation before any transform runs; MigrationResult.backupPath reports
it, and brain.restore(path) brings it back wholesale.
- CLI: cow.ts (fork/branch/checkout/history/migrate) is replaced by
snapshot.ts — snapshot <path>, restore <path>, history (tx-log),
generation.
- New public read API: brain.transactionLog({limit}) exposes the reified
tx-log (generation/timestamp/meta, newest first) that backs the CLI
history command; TxLogEntry is exported.
Tests: superseded suites deleted; fork/commit blocks excised from shared
suites; BlobStorage tests relocated + reworked against the slimmed store;
migration tests now prove the backupTo snapshot/restore round trip; new
transactionLog coverage in db-mvcc.
GraphIndexProvider now speaks BigInt at the boundary (D.2 mirror):
- getNeighbors/getVerbIdsBySource/getVerbIdsByTarget take entity ints and
return entity/verb ints as bigint[]
- new REQUIRED verbIntsToIds(bigint[]) batch reverse resolver (L.7
identity-fingerprint design — verb ids are UUIDs by contract, so the
provider-side interning is losslessly reversible)
- addVerb(verb, sourceInt, targetInt) returns the interned verb int;
removeVerb(verbId) joins the contract
Coordinator (brainy.ts) owns ALL UUID <-> int conversion: getOrAssign on
writes, getInt on reads (unmapped UUID -> empty result without calling the
provider), getUuid / verbIntsToIds on returns, plus a bounded ~100k-entry
insertion-order warm cache for verb-int -> verb-id pairs fed by addVerb
returns and resolver results. GraphVerb gains derived sourceInt/targetInt
(populated at add time, never persisted). findConnectedSubtype gains a
native fast path that routes single-type single-subtype outgoing BFS
through the provider when available.
JS GraphAdjacencyIndex satisfies the contract while staying string/u32-keyed
internally: entity ints resolve through the shared entity-id mapper (threaded
in by the coordinator on init/fork/checkout), verb ints come from an
in-process append-only interning map re-derived from storage on
rebuild/cold-start. ColumnStoreProvider widens the same way: addEntity/
removeEntity take bigint, sortTopK/filteredSortTopK return bigint[].
relate() now rejects a caller-supplied id with a teaching error — verb ids
are brainy-generated UUIDs by contract in 8.0 (previously a passed id was
silently ignored). No Roaring64 provider-boundary decode site exists yet;
the JS-internal column store stays Roaring32 and the Treemap decoder lands
with the first consumer of provider-returned filter buffers.
Public brain API unchanged. 1413 tests green (+10 new BigInt contract tests).
Step 3 of the brainy 8.0 rename scaffolding. The class name now matches
its role: the JS HNSW implementation of the VectorIndexProvider contract.
Per planning § 2.3: this is NOT cosmetic — leaving HNSWIndex when the
public contract is VectorIndexProvider creates a confusing read at the
implementation layer.
CHANGES
Repo-wide mechanical rename: HNSWIndex → JsHnswVectorIndex across 52
references in src/ + 5 references in tests/ via sed. Class declaration,
imports, JSDoc, comments, and identifiers all updated.
src/index.ts
- Primary public export: JsHnswVectorIndex (the new name).
- Backwards-compat alias: `export const HNSWIndex = JsHnswVectorIndex`
with @deprecated note. Removed in the final 8.0 cleanup commit.
NO-OP scope
No behavioural change. The class is exactly the same; only the name
moved. Tests + build green.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
Shutdown/close/flush now properly flushes all 4 components in parallel:
metadataIndex, graphIndex, HNSW dirty nodes, and storage counts. Previously
only counts were flushed, causing native provider data loss on restart.
Also:
- Wire roaring, msgpack, entityIdMapper provider consumption from plugins
- Fix allOf filter O(n²) intersection → O(n) Set-based
- Fix ne/exists negation filter to use Set-based exclusion
- Add setMsgpackImplementation() swap in SSTable for native msgpack
- Add setRoaringImplementation() swap for native CRoaring bitmaps
- Add getAllIntIds() to EntityIdMapper for bitmap operations
- Remove TypeAwareHNSWIndex from default index creation path
- Export memory detection utilities from internals
- Clean up 26 permanently-skipped dead tests
