Write/index-spine hardening, first batch of Pass 1. Each fix restores an
invariant the surrounding code already intended; every one has a
fail-before/pass-after test.
- Pattern C, finding 5 (baseStorage): delete now decrements the user-facing
scalar total symmetrically — deleteNounMetadata was decrementing only the
per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so
getNounCount()/getVerbCount() inflated permanently (the stale scalar wins
pagination via Math.max and is persisted). Invariant now holds:
scalar total === Σ per-type across add/update/delete and reopen.
- Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no
longer publishes the manifest's full relationship count as healthy. Any
per-SSTable load failure throws after the batch, which resets to honest-empty
and lets the existing size()===0 self-heal rebuild run — size()/isHealthy()
can no longer lie about a partial load.
- Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears
dirty nodes whose connections failed to persist — failed nodes stay in the
retry set, and flush() throws HnswFlushError instead of returning a lying
node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so
addItem() rejects rather than returning an id for a rootless index.
- Pattern B, finding 11 (part — storage reads): new shared isAbsentError()
helper (utils/errorClassification, ENOENT-only absence) applied to
loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE)
now propagates loudly instead of masquerading as "absent", which had driven
needless rebuilds / empty reads (loadBinaryBlob feeds the native provider).
Regression: 78 green across the 3 new suites + db-mvcc, generationStore,
temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before
the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb
legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded),
finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
Transaction.execute() checked its time budget at the top of the operation
loop and threw TransactionTimeoutError from OUTSIDE the per-operation
try/catch, so a mid-flight timeout bypassed rollback entirely — only
per-operation failures rolled back. A bulk transact that crossed its 30s
budget mid-flight left the operations already applied to canonical storage
in place while the generation was never stamped: torn, generation-less
state. The generation-store commit path's abort cleanup explicitly assumes a
throw from execute() already restored the applied operations byte-identically
(it only discards the uncommitted staging directory), so the missing
rollback broke that invariant.
Give execute() a single rollback point: the operation loop is the sole
rollback-guarded region, and any error escaping it — an operation failure OR
a mid-flight timeout — rolls back every applied operation in reverse order,
then surfaces the original error (a rollback failure still supersedes it via
TransactionRollbackError). The per-exit-path rollback that let the timeout
throw slip past is gone; atomicity now holds by construction for every error
type. An aborted transaction leaves generation() unchanged and storage
byte-identical to its pre-transaction state.
Regression (tests/unit/transaction/timeout-rollback.test.ts): a mid-flight
timeout leaves an in-memory canonical store byte-identical with the tx in the
rolled_back terminal state; the operation-failure and rollback-failure paths
through the same single rollback point; and a clean transaction still commits.
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
Sweeping the vestigial hnsw sharding machinery surfaced two real defects on
the counts-recovery path (the code that rebuilds totalNounCount/totalVerbCount
when counts.json is lost or corrupted — container restarts, partial copies):
- initializeCountsFromDisk counted files in entities/*/hnsw/ — directories the
8.0 write path never populates — so an established store recovered to ZERO
counts on real data: wrong getNounCount()/stats, a "New installation"-style
boot log, and a mis-sized rebuild-strategy decision at open. The scan now
walks the canonical entities/<kind>/<shard>/<id>/ tree (one entity per id
directory), with the same sampled type-distribution estimate as before.
- The sampler called getNounMetadata — whose ensureInitialized() re-enters
init() — from INSIDE init(), a latent deadlock reachable the moment the scan
found anything to sample. The sampled metadata files are now read directly
with fs (gz-transparent), no guarded accessors inside init.
The dead machinery itself (verified zero callers by reachability analysis
before deletion): the nine 7.x hnsw-layout entity/edge CRUD methods, the
sharding-depth prober + its depth-migration engine (unreachable — the probe
always returned null on 8.0 stores), an orphaned streaming verb paginator,
their path/scan helpers, and the now-unused type aliases and fields. The init
block keeps the truthful new-vs-established boot log introduced in 8.0.13,
now decided directly from the canonical layout. Net -1,138 lines; no public
API touched.
Adds a regression test: delete counts.json from a populated store, reopen,
counts recover exactly from the canonical tree.
Every persisted 8.0 store logged "📁 New installation: using depth 1 sharding"
on every open — even brains holding thousands of entities — which is alarming to
read during a restart or incident. 8.0 stores nouns in the canonical
`entities/nouns/<shard>/<id>/vectors.json` layout (the path `saveNoun`/`getNouns`
use), but the legacy sharding probe `detectExistingShardingDepth()` inspects
`entities/nouns/hnsw` — a 7.x directory the 8.0 write path never populates (its
only writer, `saveNode`, is dead code with zero callers). So the probe returned
null for every 8.0 store and concluded "new".
Drive the new-vs-existing log from the layout the database actually reads and
writes: a new `hasCanonicalEntities()` checks for a real 2-hex shard directory
under `entities/nouns/`, and the known noun count short-circuits it. An
established store now logs "Using depth 1 sharding (N entities)"; only a genuinely
empty store reports a new installation. Behavior is otherwise unchanged — the
probe only ever set a log line, never triggered a rebuild or migration.
`find({ where })` had two gaps that could silently return nothing. The in-memory
matcher (`matchesQuery`, used for egress re-validation and historical reads) was
missing several documented operators — `in`, `greaterThanOrEqual`,
`lessThanOrEqual` — so it disagreed with the index path, which does handle them.
And an unknown operator key (a typo, or `notIn` written where `not: { in }` was
meant) fell through to a nested-object-field interpretation and matched nothing.
- Align `matchesQuery` to the full documented operator set (add the missing
aliases; the default branch now throws instead of treating an unknown key as a
nested field — dotted paths remain the supported nested form).
- Validate the `where` filter up front (`validateWhereFilter`), throwing a typed
`BrainyError('INVALID_QUERY')` that names the unknown operator, recursing into
`allOf` / `anyOf` / `not`.
- Stop excluding a user field literally named `level` from the metadata index
(it collided with an internal never-index key), so numeric/exact filters on it
resolve instead of returning 0.
Adds unit coverage for the operator set and the throw-on-unknown behavior.
A production deployment measured ~48 seconds on EVERY reopen of an
11k-entity brain. Root cause: brainy's rebuild gate decided from in-memory
size()/count, which read 0 for a durable-but-not-resident index, so it
re-read every entity file to rebuild from scratch. At GA we gave only the
GRAPH provider a readiness contract (init() eager cold-load + isReady()
honest signal) so it would never eat that spurious rebuild; the vector and
metadata providers never got it, and brainy never even eager-inited the
vector provider.
Complete the contract symmetrically:
- plugin.ts: VectorIndexProvider gains optional init()+isReady();
MetadataIndexProvider gains isReady() — mirroring GraphIndexProvider.
Additive and optional; a provider that exposes nothing keeps today's
behavior.
- brainy.ts: eager-init every provider that exposes init() (after metadata
init() so the id-mapper is hydrated first), then decide per leg in
precedence order — migrating (skip) -> epoch drift (rebuild) -> isReady()
-> a per-leg empty fallback. The old instant fast-path keyed off
this.index.size()>0, a dishonest proxy that skipped the metadata/graph
checks whenever the vector was warm and never fired on a real cold process
anyway; removed.
The per-leg fallbacks differ because "empty" means different things: the JS
vector's rebuild() IS its load, so size()===0 correctly triggers it; the
id-mapper backs metadata, so totalEntries===0 (past the empty-store return)
is a real load failure; but entities do not imply edges, so a graph
size()===0 is a valid empty state, not a load failure.
- The JS graph now COLD-LOADS its durable LSM instead of re-deriving from a
full canonical verb scan on every boot (baseStorage._initializeGraphIndex
loads the persisted SSTables via a new GraphAdjacencyIndex.init(); it
self-heals from canonical only when the durable state is genuinely missing).
This removes an O(E)-per-open cost every filesystem consumer paid.
- LSMTree.loadManifest loads its SSTables BEFORE publishing the relationship
count, and resets to an honest-empty state on load failure — a tree can no
longer claim persisted relationships while holding none (the silent-empty
cold-load class the query-time guards exist to prevent).
Verified end-to-end against a built brain: a warm reopen (with edges and
edgeless) reloads only the JS vector; the graph and metadata cold-load with
no rebuild, and queries return correct results. New tests in
cold-open-rebuild-gate.test.ts pin the contract (isReady() defers, self-heal
still fires); migration-deference updated to drive size-based deference
through the vector, the leg where empty->rebuild remains correct.
Pairs with the native provider's isReady()/init() implementation — brainy's
gate defers only to a signal the provider exposes.
A consumer's clean-room verification of the 8.0.10 fix found relate() still
hanging their scripts. Root-causing the CLASS instead of the repro found two
mechanisms:
1. The 'beforeExit' auto-flush hook looped forever on any script that never
reaches close(): Node re-emits beforeExit after every event-loop drain, and
the async flush schedules new work — flush, drain, flush, forever. The
listener now self-deregisters BEFORE its one flush, so the next drain exits.
(Empirically: process.on('beforeExit', async () => await anything) alone
never exits — this was the deepest root of the whole hang class.)
2. Every background-maintenance interval is now unref'd at creation — graph
auto-flush, LSM compaction, metadata write-buffer flush, VFS cache
maintenance, PathResolver cache maintenance, statistics debounce (the
writer-lock heartbeat, flush watcher, and cache monitors already were).
Durability is owned by close() and the beforeExit flush, both deterministic;
a best-effort interval must never keep the host process alive.
Proof: the reported shape (add x2 + relate, filesystem) exits cleanly BOTH
with close() (~0.5 s) and with NO teardown at all — and in the no-teardown
case the beforeExit flush still lands the data (verified by reopen: both
nouns + the edge present). New per-op-class sweep test asserts no ref'd
timer survives close() for add / relate / graph find / metadata update /
vfs — turning this bug class off permanently instead of per-repro.
A consumer report on the GA pair: any minimal add/find/close script hangs
forever. Root cause was FOUR ref'd keep-alives brainy never released:
1+2. The global SIGTERM/SIGINT shutdown hooks (registered once at init) were
anonymous and never removed — a process.on signal listener holds a ref'd
signal handle. They are now named statics, deregistered when the LAST live
instance closes (Brainy.instances also stopped leaking closed instances);
a later init re-registers them.
3. UnifiedCache.startFairnessMonitor() created an interval it never stored,
cleared, or unref'd — unclearable by construction. Now stored + unref'd
(same posture as the existing memory-pressure timer): a cache monitor must
never keep the host process alive.
4. brainy.close() never called VirtualFileSystem.close(), stranding the VFS
background-maintenance interval and the PathResolver maintenance interval.
close() now shuts the VFS down.
Proof: a bare script (init/add/close, no process.exit) exits 1 ms after
close() returns; before the fix it hung until killed. 3 new lifecycle tests
pin the hook semantics (once globally, survive while other instances live,
removed on last close, re-register after).
With plugins unset (the default), init() probes for the first-party
accelerator: not installed means plain brainy with zero noise; installed and
healthy means it loads and announces itself; installed but broken (import
failure, invalid shape, failed activation, version mismatch) makes init()
THROW. An installed accelerator never silently vanishes behind the JS engines
- the anti-drift posture of the explicit list, applied to detection.
plugins: []/false stays a true opt-out (no probe); an explicit list keeps its
required-and-loud semantics. The import runs through an importPluginPackage
seam (variable specifier - bundlers cannot static-resolve the optional
package; tests simulate all outcomes without it installed).
Also: when a plugin activates but registers zero native providers (e.g. a
licensing gate declining to engage), the provider summary now warns loudly
instead of leaving every query silently on the JS engines.
Supersedes 8.0.8's explicit-opt-in wording; README/PLUGINS/types now document
the guarded contract. 7 new tests (tests/unit/plugin-autodetect.test.ts).
A downstream deployment reported find({where}) returning a silent [] on a
freshly-opened brain (a native metadata provider that reports data but hasn't
loaded its field postings), blanking filtered pages after every restart. Brainy
had a cold-read guard for GRAPH reads (verifyGraphAdjacencyLive → self-heal or a
loud GraphIndexNotReadyError) but no equivalent for metadata `where`.
Add verifyMetadataLive() — the field-index counterpart, one-shot per brain on the
first filtered find(): probe a known persisted entity's plain field value; if the
index serves it, live (the only cost on a warm brain — one O(1) probe). If not,
the postings didn't load: rebuild from canonical + re-probe; if still unserved,
throw the new exported MetadataIndexNotReadyError rather than let a silent [] pass.
Inconclusive cases (empty store, no plain field, shared-store foreign entity,
migrating provider) resolve to live — never a false rebuild.
The 8.0 open-core JS index already cold-loads correctly (verified: cold where 3/3,
cold related 2/2) — this guards the NATIVE path, where the durable cold-load cure
is cortex-side (same shape as the graph 2.7.8 cure). 4 unit tests (warm no-rebuild,
self-heal, loud-fail, no-filter-no-probe). Gates: typecheck 0, build 0, test:unit
1757/1757.
Replaces rc.8's no-freeze deference with an automatic, observable, coordinated
migration LOCK (David's reversal: "unknown/halfway states are more dangerous
than blocking"). While a native provider runs its one-time 7.x→8.0
rebuild-from-canonical (`isMigrating()===true`), brainy blocks/queues data-plane
reads AND writes so no operation ever touches a half-built index — closing the
three seam-map gaps (lost mid-rebuild writes, degraded reads, read-time rebuild).
Mechanism (one choke point):
- awaitMigrationLock() at ensureInitialized() covers all ~40 data-plane methods;
a non-migrating brain pays one boolean check. Polls isMigrating() at 250ms;
after migrationWaitTimeoutMs (default 30s) throws a retryable, exported
MigrationInProgressError. The timeout bounds the CALLER'S WAIT, not the
migration — the rebuild is unbounded and never interrupted.
- init() awaits the lock before VFS bootstrap + before serving ("not ready until
upgraded"); a rebuild past the budget surfaces MigrationInProgressError at init
(raise the budget, or run the offline migrator).
Observability (readiness-probe correct — never gated):
- getIndexStatus() gains `migrating` + `migration` (MigrationProgress); a probe
maps migrating→HTTP 503+Retry-After, not 500. health()/checkHealth() lock-exempt
(health() reports a `warn` migration check). stampBrainFormat()/close() are
ungated so cor can clear the lock — no deadlock.
- Optional provider migrationStatus() is relayed verbatim for a live %.
verifyGraphAdjacencyLive() honors the lock (no self-rebuild mid-migration); the
stamp is still withheld while any provider migrates (cor stamps after verify).
Adversarially verified: fixed a critical init/VFS-bootstrap deadlock, a mixed-
provider busy-spin (dropped the event-driven signal → pure poll), a not-yet-
initialized getIndexStatus crash, and once-per-window log/clock resets. 10 lock
tests (incl. the init-during-migration regression). Gates: typecheck 0, build 0,
test:unit 1753/1753.
Three hooks so a native provider can run a non-blocking, online, background
build-new→verify→swap index migration on a large-brain upgrade while brainy stays
out of the way — no minutes-long blocking rebuild-on-open/first-query.
- isMigrating?(): boolean — OPTIONAL on MetadataIndexProvider / GraphIndexProvider /
VectorIndexProvider (mirrors isReady?()/init?(), feature-detected). While a
provider reports migrating, brainy SKIPS its rebuild for that index — per-index,
so a non-migrating sibling still rebuilds; skipped even under epoch-drift or
size()===0 — in both rebuildIndexesIfNeeded and the lazy first-query force path.
The provider serves correct reads from canonical until it verifies-and-swaps.
- brain.stampBrainFormat() — public; the provider calls it once its background swap
verifies, so brainy authors dataFormat (the provider never does). brainy withholds
its own marker stamp while any provider is migrating, so the shared epoch is never
advanced ahead of a deferred index.
- ./brain-format export — the marker module's EXPECTED_INDEX_EPOCH / CURRENT_DATA_FORMAT
are importable so a native provider shares the constant (no duplicate = no
lockstep-drift). 8-case test; unit 1743 green.
The MVCC time-travel layer kept nounChains/verbChains as unbounded
Map<id, number[]> — one resident chain per id ever touched across retained
history (O(N) RAM, defeating billion-scale time travel). Replace with a hot-tail
window + bounded cold LRU + a mutex-free bulk resolver, keeping resolveAt exactly
correct.
The most-recent W generations stay resident as full chains (the common recent-pin
read is O(log), zero scan); deeper pins reconstruct one id's chain on demand into
an L-bounded LRU. Reconstruction is LOCK-LIGHT — it holds no commit mutex, so a
historical read never stalls writers; correctness holds because a live pin's
answer is always > minPinnedGeneration, which compaction can never reclaim, and a
concurrently-reclaimed gen below that is skipped. materializeAtGeneration routes
through a new mutex-free resolveManyAt (one forward pass, O(R + |ids|)) — without
it a deep-pin materialize both regresses to O(N*R) and deadlocks on snapshotWith's
mutex. The cold cache is invalidated on re-touch to stay coherent. Resident RAM is
O(W*d + L), independent of N. 11-case test (oracle-vs-bruteforce, held-Db across
eviction + concurrent compaction, no-deadlock, write-path I/O-free); unit 1735 +
integration 613 (db-mvcc/db-temporal/db-asof green).
Add _system/brain-format.json { dataFormat, indexEpoch } + a sync brain.formatInfo()
accessor + a compiled EXPECTED_INDEX_EPOCH, the surface a native provider reads at
init() to drive whole-brain auto-upgrade, and brainy's own derived-index
rebuild-on-format-drift trigger (closing the gap where JS indexes rebuilt only on
size()===0, never on a format-version change).
indexEpoch is shared and lockstep-bumped with the native provider on any coordinated
release whose on-disk derived-index format changes; dataFormat is brainy-owned. On open,
the marker is read in the store-open phase BEFORE provider construction (so formatInfo()
is synchronously available at the provider's init); a drifted or absent epoch rebuilds
the derived indexes from canonical records, and the marker is stamped only AFTER the
rebuild verifies (build-new -> verify -> stamp; a crash before the stamp re-triggers the
idempotent rebuild). brainFormat.ts is the single source of the shared constants.
6-case test; 1724 unit green.
Both shapes confirmed with cor for the lockstep (cor builds the native side
against these; the JS index is a no-op / unchanged):
- probeConsistency?(): Promise<boolean> — OPTIONAL O(1) cold-open consistency
sampler on MetadataIndexProvider. brainy calls it ONCE per brain on the first
read; on `false` it self-heals via detectAndRepairCorruption() — the metadata
counterpart of the 7.33.2 graph cold-load guard, completing the phantom triad's
detect-and-repair-on-open. Best-effort: a probe failure never breaks a read
(the one-shot guard resets so a transient failure retries). The JS index omits
the method, so it is simply never probed.
- getIdsForFilter(filter, opts?: { limit?; offset? }) — brainy passes a page
bound on the UNSORTED find({ type, where, limit }) path so a native provider can
early-stop and return only the [0, limit) prefix, killing the O(N) FFI marshal
at billion scale (pairs with cor #75). brainy passes offset:0 and ALWAYS
re-windows the result itself (visibility filter + slice); the JS index ignores
opts and returns all matches (behaviour unchanged).
New unit tests cover brainy's call behaviour for both (probe→repair on false,
healthy→no-repair, failure-is-best-effort, once-per-brain; opts passed with
offset 0 on the unsorted path). Full gate green: unit 1718, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
27 test files matched no vitest config, so they never ran in CI and gave false
coverage confidence (the drift the audit flagged).
- Re-homed 10 functional suites into the gate (renamed to *.unit.test.ts /
*.integration.test.ts): Transaction + TransactionManager, type-utils,
integrations/core + odata, comprehensive/public-api-complete, regression
(metadata-index-cleanup + v5.7.0-deadlock), vfs/tree-operations +
vfs-bulkwrite-race. ~192 previously-dark tests now run and pass.
- Deleted 8 bit-rotted/redundant files that fail against 8.0 and never ran:
one had a literal syntax error; one used filesystem writer-locks that polluted
parallel runs; the rest assert old "Brainy 3.0/v3.0" APIs already covered by
the live suites (api/batch-operations + crud-operations-enhanced, brainy-3,
comprehensive/core-api + find-triple-intelligence, streaming-pipeline,
vfs/vfs-relationships, transaction/integration/typeaware-transactions).
- Added a coverage guard (tests/unit/test-suite-coverage-guard.test.ts) that
FAILS CI if any *.test.ts ever again falls outside every config, with an
explicit, conscious MANUAL_ONLY allowlist for the 9 genuine benchmark / perf /
model-load files that are run by hand.
Gate green: unit 1712, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The JS metadata index served `ne`, `exists:false`, and `missing:true` by
materializing the ENTIRE id universe as an array of UUID strings, building a Set
of the excluded ids, and filtering — O(N) heap and two full passes on the
open-core path, on the most common shape (`deleted !== true`).
Compute the complement as a roaring-bitmap difference over the int-id universe
instead (`complementIds(excludeInts)` = universe \ exclude), converting only the
result back to UUIDs. The exclude set is small (the matching value), so this
avoids the full-corpus string materialization entirely. Behaviour is unchanged,
including the load-bearing soft-delete semantic that `field !== value` includes
entities with no such field. New focused tests pin `ne`/`exists:false`/
`missing:true`/`exists:true` result sets and that the complement reflects deletes.
Full gate green: unit 1520, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
removeItem() scanned the ENTIRE corpus on every delete to find nodes that
referenced the removed id (and to repair edges left asymmetric by pruning), so a
delete was O(N) and a bulk delete O(N²) — on the open-core JS vector path that
serves when no native provider is registered.
Maintain a reverse-adjacency index (`target → level → set of nodes that link to
target`), so removeItem touches only the removed node's actual in-neighbors:
O(in-degree) per delete, O(N·degree) for a bulk delete. The index is lazily
built, maintained incrementally at every forward-edge mutation (add-link and
prune), and invalidated (rebuilt on next use) by the bulk paths (cold-load
restore + clear). New tests assert the three invariants that matter for a
reverse index: no dangling references to deleted nodes, the incrementally-
maintained index exactly equals a fresh rebuild from the live adjacency, and
search still returns the survivors' true nearest neighbours (exact vs brute force).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- find() search-mode: collapsed the two overlapping options to one canonical
`searchMode: SearchMode` (the redundant `mode` alias was silently ignored on
the primary find() path while honored on the historical path — a footgun) and
removed the unwired `explain?` FindParams field (never read by find()).
- Documentation accuracy on the public type surface: entity ids documented as
UUID v7 (auto) / v5 (natural key), not v4; dropped the stale "Brainy 3.0"
banners and "backward compatibility" hedging on the fresh-8.0 Result type;
documented the via/type alias; removed the dead GraphConstraints.bidirectional;
fixed the no-op `EmbeddedGraphVerb = Omit<GraphVerb,'source'>` to omit the real
sourceId; corrected the GraphIndexProvider.isInitialized JSDoc; documented
removeMany's per-chunk generation granularity; JSDoc'd the public VFS surface.
- Honest perf comments in source: removed unmeasured billion-scale figures from
the LSMTree / graphAdjacencyIndex headers (the JS fallback is in-memory after
open; native is the scale path).
- Robustness: getVerbMetadata propagates read errors symmetrically with
getNounMetadata; the find() egress integrity-guard keeps a row when the JS
matcher doesn't implement an operator the provider already matched on; hardened
the boundary-no-native CI guard to catch side-effect imports + re-exports.
- Tests: de-theatricalized a relateMany test to assert the real contract; fixed
a stale Model-B header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Version-coupling cold-init: getBrainyVersion() returned a stale build-time
default ('3.14.0') on the FIRST synchronous call — the one loadPlugins() makes
to build context.version — because the package.json read was async. A native
provider declaring a realistic `>=8.0.0` range was therefore rejected on cold
init. version.ts now reads package.json synchronously (8.0 targets Node-like
runtimes only); added a cold-init coupling regression with a `^8.0.0` plugin.
- No-silent-failures on the highest-fan-in read path: getNouns()/getVerbs()
converted a storage read failure into a success-shaped empty page, which the
cold-start rebuild then read as "store empty" and skipped the rebuild — booting
a permanently-empty index with no signal (the same silent-failure class as the
phantom bug). Both now re-throw a named BrainyError; the rebuild path re-throws
read failures (fail loud) and records a queryable degraded state, surfaced via
checkHealth(), for non-fatal rebuild hiccups.
- LSM SSTable leak: graph compaction wrote a merged SSTable but only dropped the
old ones from the manifest, orphaning their payloads forever ("In production
we'd add a cleanup mechanism"). Added StorageAdapter.deleteMetadata() and now
reclaim each compacted-away SSTable.
- Documented the two headline methods add() and find() (the only undocumented
public methods on the class).
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
find() trusted the metadata index's returned ids: it loaded each id's entity
and returned it, checking only that the entity existed — never that it actually
matched the query. The indexes are acceleration structures; the loaded entity
is ground truth. A stale or cross-bucket index posting (e.g. an id left in a
field-value bucket by a delete or an `update({ field: undefined })`) therefore
surfaced an entity matching NEITHER the requested type NOR the where filter — a
production report saw a timeslot (NounType.Event) returned for
`find({ type: Person, where: { entityType: 'staff' } })`.
Add a single egress chokepoint after the result IIFE that re-validates every
result with `entityMatchesFind` (type/subtype/where/service/excludeVFS) — the
live-path mirror of the historical path's per-candidate check in db.ts. It
covers every find() branch (metadata, vector, text, proximity, graph) and
`similar()` (which delegates to find) in one place. A no-op on a healthy index;
on a corrupted one it drops the bad row instead of returning a phantom. Full
unit + integration gate green confirms the where re-validation is consistent
with the index (no valid rows dropped).
This is the brainy-side safety net. The durable corruption itself lives in the
metadata index that owns the query (the native provider when present) and is
addressed separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The optional native graph-acceleration provider exposes `isInitialized`,
which is false during its cold-start / rebuild window (engine + cursor not
yet loaded). The accessor cached the resolved provider INSTANCE but never
re-checked readiness, so `brain.graph.*` could dispatch to a not-ready
provider and throw instead of answering.
Resolve and cache the instance once, but check `isInitialized` LIVE on every
call: route to the pure-TS analytics path while the provider is not ready,
and re-engage the native path automatically the moment it flips true. This
gates every native route at once — subgraph, export, rank, communities,
path, and the query→expand fusion. Adds a test asserting the provider is
never called (and the TS fallback returns real answers) while not ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-GA dead-code sweep. A deterministic import-reachability walk from the
package's real entry points (the exports map, the CLI bin, the conversation
surface) found 34 source modules that nothing reachable imports — they
compile and ship as dist output that no consumer or internal path can ever
reach. All were superseded duplicates, abandoned parallel implementations,
or built-but-never-wired features:
- superseded duplicates of live modules: an older "unified" entry, a
standalone neural-import variant, a static NLP processor + its matcher,
a parallel API-types module, a duplicate progress-types module
- an abandoned import path (orchestrator + entity deduplicator + barrels)
left behind when ingestion moved to the coordinator
- unwired feature modules (instance pool, import presets, cached
embeddings, relationship-confidence scorer) reachable only from tests
- dead leaf utilities (write buffer, deleted-items index, bounded
registry, two crypto shims, a cache manager, a structured logger, a
stale v5 type-migration helper, a browser-only FS type shim) and
several dead re-export barrels
- a CLI catalog command wired into no command
Also removed two tests that only exercised deleted modules, repointed two
VFS tests off a deleted barrel onto the implementation module, and deleted
one example that demoed removed features.
Verified: clean build (both tsc passes), unit 1505 + integration 607 green,
and a re-run of the reachability walk reports zero remaining orphans.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered + ordered find() materialized the FULL sorted match set before slicing
the page — passing k = filteredIds.length to the column store's filteredSortTopK.
A broad filter + orderBy returning 20 rows at billion scale produced hundreds of
millions of sorted ids to discard all but the page (O(matches) heap).
Thread the page bound (offset + limit + the hidden-tier over-fetch) into
getSortedIdsForFilter → filteredSortTopK / sortTopK / the sparse-path slice, so
the sort produces only the page. Ordering and pagination are unchanged.
From cor's 2026-06-24 co-release scaling audit (item 1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The subgraph seed selector now accepts not just entity id(s) but a find() result
or a FindParams query — brain.graph.subgraph({ where: { team: 'platform' } },
{ depth: 1 }) runs the query and expands the neighborhood of every match in one
call. Existing id-seeded calls are unchanged.
The win is the native query→expand path: for a metadata-only query, the matched
universe is forwarded to traverse() as an OpaqueIdSet (a roaring Buffer) with NO
id materialization in TypeScript — the find() result never leaves the engine's
representation (the O(1)-crossing the cor 3.0 contract is built around). The
pure-JS path (or any query carrying vector/text/proximity criteria) materializes
the matched ids via find() and seeds the traversal from them, capped at a bounded
default when the caller pins no limit.
- New public `SubgraphSelector<T>` union (id | id[] | Result[] | FindParams).
- find()'s metadata filter-builder extracted to a shared `buildMetadataFilter`
(reused by the opaque universe producer); behavior unchanged.
- graphSubgraphNative generalized to accept `bigint[] | OpaqueIdSet` seeds.
Tested: JS query→expand + Result[] selector + empty-match in graph-subgraph, and
the native opaque pass-through (Buffer reaches traverse unmaterialized) vs the
find()-materialized bigint[] seeds via the mock provider in graph-native-routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Filtered semantic search now keeps its recall. A find({ query, where }) that
pairs vector search with a metadata filter restricts the HNSW beam walk to the
matching candidates INSIDE the traversal (walk-all, collect-allowed) rather than
filtering the top-k afterward — so a query whose nearest vectors are all filtered
out still returns the best matches that DO pass the filter, instead of empty.
- JS HNSW search() honors the 8.0 `allowedIds: OpaqueIdSet | ReadonlySet<string>`
contract param (ANDs with candidateIds; ignores the opaque Buffer form it can't
decode — only a native provider consumes that).
- MetadataIndexProvider gains an OPTIONAL `getIdSetForFilter(filter): OpaqueIdSet`
producer — the native (cor) metadata index returns its roaring filter result as
a serialized buffer; the JS index does not implement it.
- find() forwards the matched universe to the vector walk: the opaque buffer
(zero id materialization) when the native producer is present, alongside the
materialized visibility-precise candidateIds for the JS index. Visibility stays
correct via the existing post-search hard filter.
Tested: JS recall/restriction/AND/opaque-ignore on the index directly, plus
find() forwarding the opaque universe through to the beam walk via a stubbed
native producer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds three intent-level graph reads to the `brain.graph` namespace, each
native-dispatched to the optional `@soulcraft/cor` 3.0 graph engine when present
and served from pure-TS kernels otherwise (identical public shapes, default
visibility filter respected on both paths):
- `rank(opts?)` → `{ id, score }[]` descending — importance / centrality.
TS fallback: PageRank power-iteration with dangling-mass redistribution.
- `communities(opts?)` → `{ groups, count }` — connected grouping. TS fallback:
union-find weakly-connected components, or iterative Tarjan SCC when
`{ directed: true }`.
- `path(from, to, opts?)` → `{ nodes, relationships, cost } | null` — best route.
TS fallback: BFS for fewest hops, Dijkstra (min-heap) for least summed edge
weight (`by: 'weight'`); on-demand frontier expansion so short paths terminate
early. `direction` / `type` / `maxDepth` filters apply.
These are intent contracts, not algorithm contracts — the question is the
promise, the algorithm is the engine's choice.
Pure kernels live in src/graph/analyticsFallback.ts (PageRank, connected
components, Tarjan SCC, MinHeap) — unit-tested in isolation. The full surface is
tested end-to-end through the TS fallback, and the native dispatch + int↔uuid
hydration paths are covered by a mock provider in graph-native-routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the two coverage gaps a temporal-completeness audit surfaced (the code
was already correct, just untested):
- changedBetween / generationsTouching include un-flushed PENDING generations
and stay identical across the flush — the building blocks of since/diff/history
see single-op writes with no forced flush.
- setRetentionBudget() drives adaptive auto-compaction on flush() down to the
byte budget: history is reclaimed, the live record is untouched (the cor #65
retention-consume wiring, proven end-to-end).
65 generation/temporal tests green; 1528 unit green.
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
Historical reads (asOf/get) scanned the global committedGens list linearly,
making them O(database-age): a read of an unchanged entity at an old pin scaled
~12x for 10x history depth. Add per-id inverted history chains (nounChains/
verbChains) so resolveAt binary-searches the id's own generation chain instead —
O(log) and flat with depth. Chains build lazily under the commit mutex,
maintain incrementally on commit, and invalidate on compaction.
Also bound deltaCache (LRU cap 4096; getDelta re-reads evicted deltas) so a
long-lived high-write process's heap is O(cap) not O(generations) on the
disk-backed path, and binary-search commitTimestampAtOrBefore (O(log)).
Verified: 373 unit tests green; new tests/unit/db/generation-chain.test.ts
covers chain resolution, eviction re-read, and chain rebuild after compaction.
The public brain.graph.* surface and the GraphAccelerationProvider seam must name
operations by INTENT, not by the algorithm that happens to implement them — so a
plugin can swap algorithms without the contract lying. The JS fallback computes
communities via connected-components and rank via PageRank, but a native provider
(cor) is free to use Louvain/Leiden for communities and personalized-PageRank /
eigenvector-centrality for rank: same intent, different algorithm, same name.
Renamed the (not-yet-implemented) Phase-C analytics methods + their option/result
types on GraphAccelerationProvider:
- pageRank -> rank (PageRankOptions -> RankOptions; dropped the
pagerank-internal damping/maxIterations/tolerance
knobs from the contract — each impl tunes itself)
- connectedComponents -> communities (ComponentsOptions -> CommunitiesOptions,
mode:'weak'|'strong' -> directed?:boolean;
GraphComponents -> GraphCommunities, componentIds/
Count -> communityIds/Count)
- shortestPath -> path (ShortestPathOptions -> PathOptions; weight -> by:'hops'|'weight')
- neighborhoodSample -> sample (NeighborhoodSampleOptions -> SampleOptions)
- topByDegree -> mostConnected (TopByDegreeOptions -> MostConnectedOptions)
traverse / edgesForNode / graphCursorOpen/Next/Close + GraphScores/GraphPath/
OpaqueIdSet/Subgraph are UNCHANGED — everything cor has already built is untouched;
this is a pure pre-implementation rename (cor coordinated). JSDoc reworded to state
the algorithm is the provider's choice. index.ts exports + the native-seam test
mock updated. tsc/unit 1517/integration 599 green.
The brain.graph.subgraph/export NATIVE routing (graphSubgraphNative /
graphExportNative / hydrateNativeSubgraph + provider resolution) had ZERO brainy
CI coverage — in production it's exercised only cross-layer against cor's engine,
so a columnar return-shape or hydration-alignment drift would pass brainy CI
silently. This registers a faithful MOCK GraphAccelerationProvider returning a
columnar Subgraph built from the brain's REAL ints, locking the seam:
- subgraph() routes native (traverse called) and hydrates node int->id, the
nodeDepth column aligned to the node column, node type via batchGet, and edge
verb-int->Relation via verbIntsToIds + getVerbsBatchCached.
- node<->depth alignment is preserved when a node int does NOT resolve
(deleted/unknown) — the unresolvable int is dropped, not collapsed (which would
shift every later depth). This is the exact hydration risk the release audit flagged.
- export() routes to the graph cursor, hydrates chunks, and ALWAYS closes the handle.
Also fixes a latent contract bug found writing the test: graphAccelerationProvider()
only accepted a ready instance, so a provider registered as a (storage)=>provider
FACTORY (the convention graphIndex/metadataIndex/vector use) would fail the duck-test
and the native path would silently never engage. Now resolves instance OR factory,
cached. Covered by the factory-registration test.
brain.graph.export() streams the WHOLE graph in one O(N+E) pass — node chunks
then edge chunks, async-iterable — the right primitive for visualizing all data
(vs. paging per node). Native snapshot-consistent graphCursor when present, else
a TS cursor walk over nouns + verbs. Building it surfaced two latent noun bugs,
both fixed here (each a correctness win beyond export):
- getNounsWithPagination IGNORED its cursor ('offset-based, cursor planned') —
the noun mirror of the verb bug fixed in the cursor-pagination commit. Latent
because the only multi-page consumer (aggregate backfill) uses one big page; a
small chunkSize needed page 2 and, trusting the returned-but-ignored nextCursor,
re-fetched page 0 forever (infinite loop). Ported the proven verb cursor:
opaque cn1:<shard>:<id> token, stable within-shard id order, resume-after,
O(N) at any chunk size. (Generalized verbIdFromVectorPath → idFromVectorPath.)
- hydrateNounWithMetadata DROPPED 'visibility' (noun mirror of the Fix#1 verb
hydration bug) — so getNouns-fed visibility filters saw nothing and leaked
system (VFS root) / internal nodes. Now hydrated.
- New: GraphApi.export + GraphExportOptions (chunkSize, includeInternal/System,
includeNodes/Edges). hydrateNativeSubgraph extracted + shared by subgraph+export.
Test: graph-export.test.ts — full-graph completeness incl. isolated nodes,
default-hides-internal, node/edge include toggles, chunkSize chunking (the
case that exposed the cursor hang). Full gate green.
Introduces the brain.graph namespace and its first op, subgraph() — bounded
multi-hop neighborhood extraction returning a hydrated GraphView (nodes with hop
depth + type/subtype, edges as Relations, a truncated flag). The one-call answer
to 'show me everything around this node, N hops out' the graph-viz path needs.
- brain.graph.subgraph(seeds, { depth, direction, type, subtype, includeInternal,
includeSystem, maxNodes, maxEdges, hydrateNodes }). seeds: id or id[].
- Feature-detect routing: when a GraphAccelerationProvider is registered
(isGraphAccelerationProvider on the 'graphAcceleration' provider) it routes to
native traverse() and hydrates the columnar Subgraph (node ints↔ids paired with
depth position-preserving, edge verb ints → Relations); otherwise a pure-TS BFS
fallback expands each frontier through the O(degree) related() adjacency.
- Both paths produce the identical GraphView, so CI exercises the shape via the
fallback; the native path is cross-layer-tested against cor's provider.
- New public types: GraphView, GraphNode, SubgraphOptions, GraphApi (the surface
grows: export/rank/path/communities follow). isGraphAccelerationProvider guard
added to plugin.ts.
Test: graph-subgraph.test.ts — depth tracking, direction, type filter, default-
hides-internal, node hydration on/off, maxNodes truncation, multi-seed.
related({ from }) / related({ to }) each return one direction; related({ node })
unions both (every edge where the node is source OR target), deduped, in a single
O(degree) call — the indexed equivalent of merging two related() calls by hand.
The 'all edges on a noun' query the graph-viz path needs.
- RelatedParams.node: mutually exclusive with from/to (throws if combined);
composes with type/subtype/visibility filters; natural-key ids resolved.
- Implemented as a parallel out-edge (sourceId) + in-edge (targetId) fetch through
the O(degree) graph-index fast paths, merged + deduped by id (a self-loop appears
once), sorted for a stable page. Full incidence is the intended O(degree) cost;
deep offset paging of a very-high-degree node is best-effort (use the native
edgesForNode / a graph cursor for exhaustive paging).
- db.related() parity: node resolved + honored in relationMatchesParams (matches an
edge incident in either direction) for the historical/overlay paths.
Test: related-node-bidirectional.test.ts — both directions, equals from∪to, self-loop
dedupe, type-filter composition, default-hides-internal, node+from/to throws.
Defines the optional native graph-acceleration provider (cor 3.0) that
brainy feature-detects and routes brain.graph.* / related({node}) /
find({connected}) to, falling back to pure-TS adjacency when absent. This is
the published seam cor wires its native build against (mirrors the existing
VersionedIndexProvider pattern: defined in plugin.ts, exported from index.ts,
implemented externally, feature-detected at runtime).
Contract (locked cross-team in the design thread):
- GraphAccelerationProvider: traverse, edgesForNode, graphCursorOpen/Next/Close,
pageRank, connectedComponents, shortestPath, neighborhoodSample, topByDegree.
Every read op is generation-aware (optional trailing generation?: bigint) so
db.asOf(g).graph.* resolves historically; omitted = now.
- Subgraph: columnar wire format (BigInt64Array node/edge columns, Uint16Array
edgeTypes, optional Uint8Array nodeDepth, truncated flag) — parallel typed
arrays, never array-of-objects; brainy maps u64<->UUID lazily for rendered rows.
- OpaqueIdSet: a find() universe forwarded opaquely into traverse/search for the
zero-crossing query->expand fusion (brainy never inspects it; the provider
version-tags + validates the envelope).
- VectorIndexProvider.search gains allowedIds?: OpaqueIdSet | ReadonlySet<string>
(predicate pushdown — recovers filtered recall lost to post-filtering).
- EntityIdMapperProvider gains entityIntsToUuids(BigInt64Array) — the bigint batch
reverse resolver for Subgraph.nodes; implemented on the JS mapper (TS fallback).
All new public types exported from index.ts. Test: entityIntsToUuids round-trip
+ order-preservation + empty-int sentinel.
getVerbsWithPagination was offset-only: it re-scanned from shard 0 on every page
and early-terminated at offset+limit, so a full edge walk was O(N²) (a consumer
measured ~19k edges → 27s paging at 900/page). It now accepts an opaque resume
token (cv1:<shard>:<verbId>) that resumes the shard walk immediately AFTER the
last returned verb — O(N) total at any page size, no scale cliff.
- Stable within-shard order (sort by verb id); ids come from the path so verbs
skipped by the cursor are never read.
- Cursor supersedes offset when present; offset path preserved for back-compat,
and offset callers now get a usable nextCursor so they can switch to cursor
paging. Foreign/malformed tokens decode to null → offset fallback (no throw).
- The relationships stream generator now uses cursor (was offset → O(N²)).
- Parity with the noun side (getNounsWithPagination already cursored).
Test: tests/unit/storage/verb-cursor-pagination.test.ts — a full cursored walk
visits every edge exactly once (no dup, no skip), matches the offset walk's set,
and a foreign cursor falls back gracefully.
related({from/to}) routes through storage.getVerbs, which has O(degree)
GraphAdjacencyIndex fast paths (getVerbsBySource/Target/Type_internal). But
those fast paths were disqualified whenever excludeVisibility was set —
and every default related() sets it, because visibility defaults to excluding
internal+system. So the common per-node edge lookup fell through to a full O(E)
shard scan (multi-second per node on large graphs; a consumer measured
1.4-4.6s/node and an O(N^2) whole-graph walk).
Root cause two-parter, both fixed here:
- hydrateVerbWithMetadata dropped 'visibility' (mapped subtype but not the
reserved visibility field), so fast-path results couldn't be visibility-filtered
AND related({includeInternal}) couldn't even report which edges were internal
(latent correctness bug — verbsToRelations already mapped it). Now hydrated.
- getVerbs disqualified the fast paths on subtype/excludeVisibility. Now the
fast paths apply both filters on their already-hydrated O(degree) candidate set
via applyVerbMetadataFilters() — matching the full-scan fallback's semantics —
so default related() stays on the index instead of scanning the whole graph.
Filtering semantics are unchanged (same result set as the scan); only the path
that computes it changes. Test: related-visibility-fast-path.test.ts (default
excludes internal both directions, includeInternal surfaces + reports visibility,
subtype filter on the fast path).
Three additive ergonomics from the API-simplification audit (no behavior change
to existing call sites):
- AddParams.upsert: create-or-update in one call. With a custom id, an existing
entity is MERGED via the update path (merges metadata, re-embeds changed data,
bumps _rev, PRESERVES createdAt) instead of the destructive full overwrite a
plain add() does. Mutually exclusive with ifAbsent (throws if both set);
ignored when no id is supplied. Wired into add(), addMany (per-item flag
propagation), and the transact add op (routes to planTxUpdate). Kills the
get()-then-add() round-trip for idempotent writes.
- FindParams.includeVectors: mirror of GetOptions.includeVectors — find() returns
stored vectors when set; default stays empty (the perf contract is preserved).
Honored on both the query and metadata-only where paths, and in db.find().
- removeMany adaptive chunking: replaced the hardcoded chunkSize=10 with
params.chunkSize ?? storageConfig.maxBatchSize, matching addMany/relateMany —
one storage-adaptive batch policy across all *Many methods (no thrash on
high-latency backends).
Tests: tests/unit/brainy/upsert.test.ts (insert/merge/createdAt-preserved/
re-embed/ifAbsent-conflict/no-id/addMany/transact), find-include-vectors.test.ts
(true/default/where-path), batch-operations.test.ts (removeMany >10 items).
An untyped (JS) caller that smuggles a Brainy-reserved field (confidence,
weight, subtype, visibility, service, createdBy, noun/verb, data, createdAt,
updatedAt, _rev) inside a write-path metadata bag previously got a silent
remap-or-drop — a class of bug where confidence-evolution writes no-oped for
weeks before being caught on read-back. 8.0 closes this with no silent failures.
- New BrainyConfig.reservedFieldPolicy: 'throw' | 'warn' | 'remap' (default 'throw').
'throw' rejects the write naming every offending key + its correct write path;
'warn' remaps with a one-shot per-key warning; 'remap' is the legacy silent path.
- Central enforceReservedPolicy gate wired into all four remap methods (add,
update, relate, updateRelation) so live calls AND their transact()/with()
mirrors honor it. Single-source reservedWritePath guidance shared by throw+warn.
- 'warn' now warns for EVERY reserved key (closes the gap where only
system-managed fields warned). Dead warnDropped* helpers removed.
- Import pipeline migrated to route reserved values (confidence/weight/subtype)
through dedicated params and strip reserved keys from extractor/customMetadata
bags via the canonical split*MetadataRecord helpers — imports no longer trip
the default throw.
- Tests: new reservedFieldPolicy matrix (throw/warn/remap across every write
path + transact); remap-correctness suite reframed as opt-in 'remap'; shared
test-factory no longer emits reserved keys in custom metadata.
- id-normalization (#18): `brain.newId()` (UUID v7) + v7 default ids; non-UUID string ids are
transparently normalized to a stable UUID v5 on creation AND every lookup — get/update/remove,
relate(from,to), related, find({connected}), getMany, removeMany, the transact op-handler, and
db.with() overlays — with the caller's original key preserved under `_originalId`. The engine only
ever sees a UUID; real UUIDs pass through untouched. universal/uuid.ts gains v5/v7/isUUID +
BRAINY_ID_NAMESPACE; new utils/idNormalization.ts (coerceNewEntityId / resolveEntityId).
- aggregation min/max delete-safety: `queryAggregate` no longer leaves a stale min/max after
deleting the current extreme — min/max now track a value multiset and recompute in-memory (no
entity scan; that scan was the prior delete-then-hang a consumer reported). Other metrics were
already exact across deletes. Regression test proves resolve-after-delete + correct new min/max.
- release.sh RC-safe: explicit version arg, prerelease detection (npm `--tag rc` + GitHub
`--prerelease`), full `test:ci` gate (unit + integration), current-branch push, npm-access
verification after publish.
- native-engine references point at `@soulcraft/cor` (8.0's partner) in user-facing strings/docs.
Unit 1461/0 · integration 599/0 · tsc clean.
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
Two independent restart/multi-instance correctness bugs.
VFS path cache (A.3): PathResolver keyed the PROCESS-GLOBAL path cache on
`vfs:path:<path>` with no instance scope. Multiple Brainys per process (a
supported pattern) over DIFFERENT storage collided — instance A's `/x → id_A`
satisfied instance B's `stat('/x')` against unrelated storage → stale id →
"Entity not found" (surfaced by metadata-only-comprehensive's VFS stat() once
another VFS test had seeded the cache). Scope every `vfs:path:` key by a
monotonic per-process instance token (the VFS root id is a fixed sentinel, so it
can't disambiguate; the global cache is itself process-scoped so the token
suffices). Scoped the invalidate/prefix-delete paths too, so a branch-switch /
clear / fork on one brain no longer wipes another brain's cache. Cross-instance
sharing was only an optimization and was the collision itself; each instance
keeps its own local pathCache.
Verb totalCount (verb mirror of b2005ff): getVerbsWithPagination returned
`collectedVerbs.length` (the peeked page size, bounded by offset+limit+1) as
totalCount, so getVerbs({limit:1}).totalCount read 1/2 for any non-empty brain
on the unfiltered path. Now returns the authoritative O(1) totalVerbCount
(isNew-gated, visibility-filtered, rehydrated on init); filtered scans keep the
collected length. Regression: tests/unit/storage/getVerbs-totalCount.test.ts
(warm + cold reopen, mirrors the noun test).
metadata-only-comprehensive + vfs-api-wiring integration green; 1471 unit green.
getIdsForRange computed includeMin/includeMax correctly for every operator
(gt→false/true, lt→true/false, between→true/true) but DROPPED them when the
column store served the query — it called columnStore.rangeQuery(field,min,max)
with no flags, and the column-store path was inclusive-only. So whenever a field
lived in the column store, strict lessThan/greaterThan silently behaved as
lte/gte. The sparse-fallback path already forwarded the flags, so this was
column-store-only.
Thread includeMin/includeMax end to end:
- ColumnSegmentCursor: add textbook lowerBound/upperBound helpers and express
binarySearchRange in terms of them. Inclusive/inclusive is byte-identical to
the old impl (lowerBound(lo) == old binarySearchValue(lo).index; upperBound(hi)
== old "first position after hi"). Exclusive lower advances past ALL duplicates
of the boundary value; exclusive upper stops before them.
- ColumnStore.rangeQuery: accept the flags, apply them in the segment cursor AND
the tail-buffer linear scan (strict > / < when exclusive). A bound taken from a
segment's own min/max stays inclusive — it is a real stored value.
- VectorIndex/types interface + getIdsForRange call site updated.
Surfaced by brainy-complete dual-bound test (year/popularity exclusive both ends
→ ['Express','Vue.js']; React@95 and Angular@75 correctly excluded). Added 5
column-store unit tests: exclusive lower, exclusive upper, both, duplicate
boundary values, and the unflushed tail-buffer path. 1469 unit green.
