The async group-commit flush that persists single-op generation history
swallowed persist failures as a bare warn: writes kept succeeding while their
before-images piled up in memory, never durable and unbounded, with no signal.
The generation store now accounts for every failed flush at one place
(flushPendingSingleOps), tolerates a transient blip (retry with capped
exponential backoff), and after PENDING_FLUSH_FAILURE_THRESHOLD consecutive
failures LATCHES a durability failure and refuses further single-op and transact
writes with a typed, exported PendingFlushDurabilityError — rather than promise a
durability it cannot deliver. Live canonical data is untouched; only the
immutable history is stuck. The latch self-heals: the moment a flush succeeds
(a retry, or an explicit flush()/close()) it lifts and writes resume.
Loud errors, never quiet losses.
clear() removed entities, indexes, system and _cas but left three top-level
trees on disk: _blobs (raw HNSW/LSM segment bytes and the native dkann index),
_id_mapper (the native shared mmap id-mapper), and _column_index (column-store
manifests). A cleared brain therefore re-read stale native state.
Worse, the column store splits its state across two of those trees — manifests
under _column_index/ and their segment bytes under _blobs/_column_index/ — so
removing one without the other stranded a manifest listing segments that no
longer exist, which the hardened segment-load path now (correctly) refuses with
ColumnSegmentLoadError. The three trees now fall together as a set, exactly as
_cas already does. locks/ is deliberately preserved: it is live coordination
state, not data.
ColumnStore silently skipped a manifest-listed segment it could not load: a
corrupt or missing segment dropped every one of its entities out of
filter/rangeQuery/sortTopK with no error, so an inconsistent index read as a
merely short result. It now throws a typed ColumnSegmentLoadError when a listed
segment yields undecodable or no bytes, and lets a genuine storage IO fault
propagate verbatim. Only a field with no manifest at all stays benign (nothing
was ever written for it).
baseStorage.getNoun_internal / getVerb_internal likewise caught every error and
returned null, reporting a present-but-unreadable entity as "not found". They
now return null only for genuine ENOENT-class absence (via isAbsentError) and
rethrow real faults and deserialize errors.
Pattern-B (blind catch) hardening: absence -> null, fault -> loud.
Write/index-spine hardening, first batch of Pass 1. Each fix restores an
invariant the surrounding code already intended; every one has a
fail-before/pass-after test.
- Pattern C, finding 5 (baseStorage): delete now decrements the user-facing
scalar total symmetrically — deleteNounMetadata was decrementing only the
per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so
getNounCount()/getVerbCount() inflated permanently (the stale scalar wins
pagination via Math.max and is persisted). Invariant now holds:
scalar total === Σ per-type across add/update/delete and reopen.
- Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no
longer publishes the manifest's full relationship count as healthy. Any
per-SSTable load failure throws after the batch, which resets to honest-empty
and lets the existing size()===0 self-heal rebuild run — size()/isHealthy()
can no longer lie about a partial load.
- Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears
dirty nodes whose connections failed to persist — failed nodes stay in the
retry set, and flush() throws HnswFlushError instead of returning a lying
node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so
addItem() rejects rather than returning an id for a rootless index.
- Pattern B, finding 11 (part — storage reads): new shared isAbsentError()
helper (utils/errorClassification, ENOENT-only absence) applied to
loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE)
now propagates loudly instead of masquerading as "absent", which had driven
needless rebuilds / empty reads (loadBinaryBlob feeds the native provider).
Regression: 78 green across the 3 new suites + db-mvcc, generationStore,
temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before
the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb
legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded),
finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
A returned write must be immediately readable by id, metadata filter, vector
search, and graph traversal — no await, delay, or retry between the write
returning and the read. This holds by construction because every projection
(canonical + HNSW + metadata index + graph index) commits inside the write's
transaction; the generation counter is the {seq} a caller can pin. The test
locks that guarantee so index maintenance can never quietly move off the commit
path (which would turn a returned write into a not-yet-queryable one).
First brainy chapter of the write/index-spine hardening program.
When a transaction failed and its rollback ALSO failed to undo a canonical
write (retries exhausted), the old path logged, continued, threw
TransactionRollbackError, and set state='rolled_back' — while the record it
could not undo stayed durable on disk. The caller got an error implying the
write was undone; a read-back showed the record. A failed rollback had no
truthful response (the post-commit response lie).
Give a failed rollback a two-branch honest contract, decided by observation:
both commit paths already hold byte-identical before-images, so after a failed
rollback the store compares current canonical state to them and classifies each
touched record as reconciled, an additive orphan (present when it should be
gone), or a restorative loss (gone/wrong when it should have been restored).
- Adopt-forward: a single-op write whose only damage is a durably-present
orphan — the record the caller asked for — is committed forward (its
generation buffered) and returns success with a loud warning that the derived
index may be incomplete for that id until the next rebuild/repairIndex(). No
error, no double-write; the record is durable and get-able immediately.
- Fail loud + quarantine: a multi-op batch, or ANY restorative loss, throws the
new StoreInconsistentError naming every unreconciled record and its
disposition, and puts the brain into write-quarantine (reads keep working,
writes refused via assertWritable) until repairIndex() reconciles the derived
indexes against canonical and lifts it. The counter is not advanced, and
Transaction.rollback now reports state 'inconsistent' instead of the
'rolled_back' lie when any undo failed.
New: StoreInconsistentError + UnreconciledRecord exported from the root;
repairIndex() forces a rebuild and clears the quarantine. Regression
tests/integration/rollback-trapdoor.test.ts injects index-add-throws +
canonical-delete-undo-fails and pins adopt-forward (durable, get-able, not
quarantined), fail-loud (StoreInconsistentError + quarantine + reads work +
repairIndex lifts it), and the error's record naming.
restore() removed the entire live brain directory and then fs.cp'd the
snapshot in, so any copy failure left the store destroyed with only a partial
copy. The sharpest edge: fs.cp materialized the holes of sparse mmap blob
files, so a snapshot that fits on disk could balloon and ENOSPC mid-copy — the
recovery tool destroying the brain it was asked to recover.
Rewrite restoreFromDirectory as stage → verify → atomic swap:
- Copy the snapshot into a _restore_staging area BEFORE touching live data,
sparse-aware (copyTreeSparse/copyFileSparse skip all-zero 4 MiB chunks, so a
mostly-hole store restores at its true allocated size, not its apparent size).
- On any copy failure (ENOSPC included) remove only the half-written staging
area and throw — the live store is left exactly as it was.
- Only after the copy succeeds, fsync a completion marker naming the staged
entries, then swapStagedRestoreIn(): an idempotent per-entry
rm-old + rename-staged-in (same-filesystem renames that cannot ENOSPC),
removing stale live entries the snapshot lacks.
- completeInterruptedRestore(), wired into init() right after the root dir is
ensured, finishes a crash mid-swap FORWARD (resume a committed swap) or
discards an uncommitted staging area (live still authoritative).
_restore_staging is excluded from snapshots. persist() (hard-link snapshot)
was already safe and is unchanged; no public API change.
Regression (tests/integration/restore-nondestructive.test.ts): a forced copy
failure leaves live data fully intact and cleans staging; a normal restore
round-trips to the snapshot; an interrupted-but-committed restore completes on
reopen; an uncommitted staging area is discarded on open; the sparse copy is
byte-identical with allocated blocks far below apparent size.
A committed transact() reported success while its canonical entity writes were
still only in the OS page cache (tmp+rename, not fsync'd), even though the
generation counter and manifest were fsync'd. A hard kill in that window could
leave the durable counter ahead of the persisted entity bytes — phantom
progress for any generation-based consumer resuming from the counter. Reported
from a downstream migration's crash-lifecycle forensics.
Add an optional transaction durability barrier to GenerationStorage
(beginWriteBarrier / flushWriteBarrier). FileSystemStorage implements it:
writeObjectToPath records each successful canonical write and
deleteObjectFromPath records each delete's parent dir, between begin and flush;
flush fsyncs every recorded write (contents + rename dir entries, via
syncRawObjects) and the parent dir of every delete. commitTransaction opens the
barrier before running the planned operations and flushes it after, BEFORE
persisting the counter and manifest — so the batch's entire canonical footprint
is durable before the generation stamp advances. The barrier is optional: the
generation store calls it through optional chaining, so in-memory and
durable-per-call (cloud object-PUT) adapters treat it as a no-op.
The single-op group-commit path is unchanged (deferred durability is the
Model-B design that avoids a 3-5x per-write fsync regression), but its
durability contract is now documented explicitly on commitSingleOp: transact =
durable on return; single-op = durable at the next flush()/close(), with the
counter buffered alongside the data so a crash loses both together (never a
torn counter-ahead-of-state store).
Regression (tests/integration/transact-durability-barrier.test.ts): the entity
writes fsync in an earlier syncRawObjects batch than the manifest for single-op
and multi-op (add+relate) transactions; a precommit-rejected batch opens no
barrier and advances nothing; MemoryStorage exposes no barrier (optional-chain
no-op).
Transaction.execute() checked its time budget at the top of the operation
loop and threw TransactionTimeoutError from OUTSIDE the per-operation
try/catch, so a mid-flight timeout bypassed rollback entirely — only
per-operation failures rolled back. A bulk transact that crossed its 30s
budget mid-flight left the operations already applied to canonical storage
in place while the generation was never stamped: torn, generation-less
state. The generation-store commit path's abort cleanup explicitly assumes a
throw from execute() already restored the applied operations byte-identically
(it only discards the uncommitted staging directory), so the missing
rollback broke that invariant.
Give execute() a single rollback point: the operation loop is the sole
rollback-guarded region, and any error escaping it — an operation failure OR
a mid-flight timeout — rolls back every applied operation in reverse order,
then surfaces the original error (a rollback failure still supersedes it via
TransactionRollbackError). The per-exit-path rollback that let the timeout
throw slip past is gone; atomicity now holds by construction for every error
type. An aborted transaction leaves generation() unchanged and storage
byte-identical to its pre-transaction state.
Regression (tests/unit/transaction/timeout-rollback.test.ts): a mid-flight
timeout leaves an in-memory canonical store byte-identical with the tx in the
rolled_back terminal state; the operation-failure and rollback-failure paths
through the same single rollback point; and a clean transaction still commits.
transact() promises atomic forward references — add an entity and relate to
it in one batch — but the planner resolved relationship endpoint ints at
PLAN time, before the batch's add operations had applied. Asking the id
mapper about an entity that does not exist yet made the (correctly strict)
native mapper throw in EntityIdMapper.getOrAssign, so
transact([{op:'add', id:X}, {op:'relate', to:X}]) failed on native
deployments; the permissive JS mapper masked the bug and silently leaked an
id assignment whenever a batch was later rejected by a commit precondition
(normal control flow since the conditional-commit CAS landed).
Make endpoint resolution lazy: AddToGraphIndexOperation and
RemoveFromGraphIndexOperation take VerbEndpointInts — an eager
{sourceInt, targetInt} or a thunk evaluated when the operation EXECUTES,
mirroring the constructor's already-lazy generationFn. The four
transact-planner sites (relate, its bidirectional reverse edge, remove's
relationship cascade, unrelate) pass thunks, so resolution happens inside
the commit after same-batch adds have applied; the seven single-operation
sites keep eager objects (their endpoints provably pre-exist — relate
validates both, unrelate/updateRelation/remove read the existing verb).
The remove operation's rollback captures the ints resolved at execute so
its re-add uses the same mappings.
Byproduct fix: a rejected batch no longer pollutes the id mapper — the
regression suite pins this (mapper has no assignment for the phantom
entity after a precondition-rejected forward-ref batch), alongside the
exact reported shape, both-endpoints-in-batch, bidirectional,
add+relate+remove in one batch, and the split-transact control
(tests/integration/transact-forward-ref-graph.test.ts).
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
Subscribe once and receive one post-commit event per affected record for
EVERY canonical write, regardless of origin: direct calls, batch methods,
transact(), imports, and Virtual Filesystem writes all funnel through the
same commit points the feed is emitted from. This is the authoritative
in-process signal for live UIs, cache invalidation, and realtime sync layers
that forward it over their own transports.
Architecture — the post-commit dual of the ifRev precommit hook: mutation
methods hand lightweight event descriptors to the commit seam
(persistSingleOp / transact's plan), which stamps the committed
{generation, timestamp}, enriches entity deletes with the record's LAST
committed state from the commit's own before-images (free — Model B reads
them anyway; removeMany's id-only deletes gain full payloads this way), and
emits only after the commit succeeds — a losing CAS or rejected batch never
announces anything. Dispatch is a microtask FIFO after the mutex releases:
commit-ordered, a slow listener never delays a write, a throwing listener is
logged and isolated, and with no subscribers the write path constructs no
events at all. Single-point emission was chosen over per-method hooks
because the aggregation-hook pattern demonstrably drifted (relation ops and
removeMany were silently missing from it).
Coverage: add/update/remove (+ cascade unrelate per deleted relationship),
relate (both edges when bidirectional)/unrelate/updateRelation, per-item
events for addMany/updateMany/relateMany/removeMany, per-item events sharing
one generation for transact(), and transitively imports + VFS. clear() and
restore() — wholesale raw-state operations outside the per-record commit
path — emit a single store-level event meaning "refetch everything".
brain.close() drops all listeners.
New module src/events/changeFeed.ts (BrainyChangeEvent + ChangeFeed,
exported from the package root); guide docs/guides/reacting-to-changes.md.
Integration suite pins the contract: per-op payload fidelity, delete
last-state payloads, batch per-item emission, one-generation transact
batches, VFS-origin events, CAS-loser silence, ordering, listener isolation,
unsubscribe, and store-level events.
Sweeping the vestigial hnsw sharding machinery surfaced two real defects on
the counts-recovery path (the code that rebuilds totalNounCount/totalVerbCount
when counts.json is lost or corrupted — container restarts, partial copies):
- initializeCountsFromDisk counted files in entities/*/hnsw/ — directories the
8.0 write path never populates — so an established store recovered to ZERO
counts on real data: wrong getNounCount()/stats, a "New installation"-style
boot log, and a mis-sized rebuild-strategy decision at open. The scan now
walks the canonical entities/<kind>/<shard>/<id>/ tree (one entity per id
directory), with the same sampled type-distribution estimate as before.
- The sampler called getNounMetadata — whose ensureInitialized() re-enters
init() — from INSIDE init(), a latent deadlock reachable the moment the scan
found anything to sample. The sampled metadata files are now read directly
with fs (gz-transparent), no guarded accessors inside init.
The dead machinery itself (verified zero callers by reachability analysis
before deletion): the nine 7.x hnsw-layout entity/edge CRUD methods, the
sharding-depth prober + its depth-migration engine (unreachable — the probe
always returned null on 8.0 stores), an orphaned streaming verb paginator,
their path/scan helpers, and the now-unused type aliases and fields. The init
block keeps the truthful new-vs-established boot log introduced in 8.0.13,
now decided directly from the canonical layout. Net -1,138 lines; no public
API touched.
Adds a regression test: delete counts.json from a populated store, reopen,
counts recover exactly from the canonical tree.
The fast-follow to the ifRev CAS fix: the sweep for the same check-then-act
class found two more instances, both now closed with the same discipline —
the decision runs at the serialization point that guards the apply.
add({ifAbsent}) / add({upsert}): the absence check ran before the commit
mutex, so N concurrent same-id creates could all pass it and all write — the
second silently overwriting the first, violating ifAbsent's "return the
existing id WITHOUT writing" contract and upsert's merge-never-clobber
contract. The insert leg now carries a must-be-absent commit precondition
(the same conditional-commit primitive ifRev uses), verified under the commit
mutex against the authoritative before-image. A losing caller takes its
documented resolution instead of overwriting: ifAbsent returns the existing
id with zero writes; upsert merges into the now-existing entity via update()
(shared param mapping in upsertMergeParams so the planning-time branch and
the conflict retry can never drift), with a bounded retry if a concurrent
delete lands between the conflict and the merge. The planning-time checks
remain as fast-fails. transact()'s batch-level ifAbsent/upsert keep
planning-time semantics (documented, converges to a valid entity).
BlobStorage: write()'s dedup decision (exists → increment refCount, absent →
create at 1) and delete()'s decrement-then-remove were unserialized
read-modify-writes over blob-meta:<hash>. Concurrent writes of identical
content could lose references, so a later delete removed bytes another file
still referenced (data loss), or leaked unreferenced blobs. All
reference-count-bearing mutations now serialize through a per-hash
InMemoryMutex — distinct content never contends; incrementRefCount/
decrementRefCount are documented lock-assumed internals.
Both fixes are complete by construction in-process: storage enforces
single-writer-per-directory, so the process is the whole concurrency domain.
Tests (tests/integration/ifabsent-upsert-blob-concurrency.test.ts): 8-way
ifAbsent storm advances the store by exactly one generation with _rev 1;
8-way upsert storm on an absent id yields one create + seven merges
(_rev === 8); sequential ifAbsent/upsert semantics pinned unchanged; N
identical concurrent blob writes → refCount === N; the blob survives until
the true last reference drops; interleaved write/delete storm never loses a
landed reference.
N concurrent update({ ifRev }) calls carrying the same expected revision all
fulfilled (zero RevisionConflictErrors, last-writer-wins) — the check ran
before the commit mutex, so interleaved callers all passed it before any apply
landed. Sequential calls conflicted correctly, which hid the race. A production
cutover rehearsal caught it: 8 parallel ifRev-guarded ledger writes all
"succeeded" and 7 were silently lost. Advisory locks built on exactly-one-winner
semantics could hand the same lock to two workers.
Fix: conditional commit. commitSingleOp and commitTransaction accept an
optional precommit(beforeImages) precondition, invoked under the commit mutex
against the just-read authoritative before-images and before anything is staged
or applied — the per-record analogue of ifAtGeneration, which always ran there.
A throw aborts the commit atomically (generation reservation returned, zero
staging I/O in the transaction path). The store stays domain-ignorant; update()
and transact() supply the ifRev predicate:
- update(): the planning-time check remains as a fast-fail (avoids embedding
cost on an obviously stale expectation); the authoritative check re-verifies
the before-image's _rev and re-stamps the update's _rev from it, so the
counter is monotonic even for concurrent non-CAS updates (N plain updates
advance _rev by N). CAS against a concurrently-removed entity now throws
EntityNotFoundError instead of silently resurrecting it; plain updates keep
their last-writer-wins re-create semantics.
- transact(): per-op ifRev expectations registered during planning
(PlannedTransact.casUpdates) are re-verified in the batch's precommit; any
conflict rejects the whole batch before staging. Multiple updates to one
entity sequence through a running rev; ids the batch itself adds
(PlannedTransact.createdNouns — add resets the rev baseline even on
overwrite) keep their exact plan-time sequencing.
Regression suite (tests/integration/ifrev-concurrent-cas.test.ts): 8 parallel
same-rev updates → exactly 1 winner + 7 conflicts; same for transact batches;
_rev monotonicity; the documented read→CAS→retry ledger loop converging exactly
(8 workers, 8 decrements, 0 lost); sequential behavior unchanged; forward-ref
add+update in one batch unchanged.
The one-time 7→8 layout migration rescues the head branch's entities
(branches/<head>/entities/* → entities/*), then drained the entire
branches/<head>/ directory. Any durable state a 7.x engine wrote under the head
branch OUTSIDE entities/ — a branch-scoped index, blob area, or field registry —
would be silently deleted by that drain, the same failure class as the VFS
content blobs a 7.x store kept in _cow/.
Guard it: after the entity move, list what remains under branches/<head>/ and
exclude entities/. If anything survives, it is non-entity durable state, so
PRESERVE the branch (skip removeRawPrefix) and warn loudly with the leftover
keys — recoverable, not lost. A clean branch (only the moved entities) still
drains exactly as before. Adds a PARITY GUARD test to the 7→8 migration suite.
Every persisted 8.0 store logged "📁 New installation: using depth 1 sharding"
on every open — even brains holding thousands of entities — which is alarming to
read during a restart or incident. 8.0 stores nouns in the canonical
`entities/nouns/<shard>/<id>/vectors.json` layout (the path `saveNoun`/`getNouns`
use), but the legacy sharding probe `detectExistingShardingDepth()` inspects
`entities/nouns/hnsw` — a 7.x directory the 8.0 write path never populates (its
only writer, `saveNode`, is dead code with zero callers). So the probe returned
null for every 8.0 store and concluded "new".
Drive the new-vs-existing log from the layout the database actually reads and
writes: a new `hasCanonicalEntities()` checks for a real 2-hex shard directory
under `entities/nouns/`, and the known noun count short-circuits it. An
established store now logs "Using depth 1 sharding (N entities)"; only a genuinely
empty store reports a new installation. Behavior is otherwise unchanged — the
probe only ever set a log line, never triggered a rebuild or migration.
The 7.x branch system stored Virtual Filesystem content as blobs in its
copy-on-write area (`_cow/`). 8.0 removed that system and stores content blobs
in the content-addressed store (`_cas/`), but the one-time layout migration only
moves entities — it never adopted the `_cow/` content blobs. A store that used
the VFS was left with every VFS-backed read throwing "Blob metadata not found"
and its pages 500ing, with no first-party recovery and the pre-upgrade backup
already removed on entity-migration "success".
Add an on-open recovery pass (`autoAdoptLegacyVfsBlobsIfNeeded`) that runs right
after the layout migration and adopts every orphaned `_cow/` blob into `_cas/` —
copying both the bytes and the metadata via the raw object primitives,
idempotently and non-destructively (the `_cow/` originals are never deleted). It
is a separate phase, not folded into the migration, so it also heals a store
already upgraded by an earlier 8.0.x that stranded the blobs (whose layout-
migration marker is stamped): it is gated on the presence of `_cow/` and its own
`_system/vfs-blob-adoption.json` marker. Filesystem-only; native-8.0 and non-VFS
stores no-op on a cheap existence check.
- `BaseStorage.adoptLegacyCowBlobs()` — the scan/copy primitive, returning
`{ cowBlobs, adopted, alreadyPresent, incomplete }`; skips (does not half-adopt)
a blob missing its bytes or metadata.
- `brain.vfs.adoptOrphanedBlobs()` — the explicit force path; self-initializes.
- Backup retention: the automatic pre-upgrade backup is now kept if any blob
can't be fully adopted (`incomplete > 0`), instead of being removed on
entity-migration success alone — a blob-parity gate the migration signal
cannot provide.
Adds an end-to-end integration test (storage-level adopt with idempotency and
incomplete handling; self-heal on reopen; the explicit API) and an upgrade guide.
`find({ where })` had two gaps that could silently return nothing. The in-memory
matcher (`matchesQuery`, used for egress re-validation and historical reads) was
missing several documented operators — `in`, `greaterThanOrEqual`,
`lessThanOrEqual` — so it disagreed with the index path, which does handle them.
And an unknown operator key (a typo, or `notIn` written where `not: { in }` was
meant) fell through to a nested-object-field interpretation and matched nothing.
- Align `matchesQuery` to the full documented operator set (add the missing
aliases; the default branch now throws instead of treating an unknown key as a
nested field — dotted paths remain the supported nested form).
- Validate the `where` filter up front (`validateWhereFilter`), throwing a typed
`BrainyError('INVALID_QUERY')` that names the unknown operator, recursing into
`allOf` / `anyOf` / `not`.
- Stop excluding a user field literally named `level` from the metadata index
(it collided with an internal never-index key), so numeric/exact filters on it
resolve instead of returning 0.
Adds unit coverage for the operator set and the throw-on-unknown behavior.
A production deployment measured ~48 seconds on EVERY reopen of an
11k-entity brain. Root cause: brainy's rebuild gate decided from in-memory
size()/count, which read 0 for a durable-but-not-resident index, so it
re-read every entity file to rebuild from scratch. At GA we gave only the
GRAPH provider a readiness contract (init() eager cold-load + isReady()
honest signal) so it would never eat that spurious rebuild; the vector and
metadata providers never got it, and brainy never even eager-inited the
vector provider.
Complete the contract symmetrically:
- plugin.ts: VectorIndexProvider gains optional init()+isReady();
MetadataIndexProvider gains isReady() — mirroring GraphIndexProvider.
Additive and optional; a provider that exposes nothing keeps today's
behavior.
- brainy.ts: eager-init every provider that exposes init() (after metadata
init() so the id-mapper is hydrated first), then decide per leg in
precedence order — migrating (skip) -> epoch drift (rebuild) -> isReady()
-> a per-leg empty fallback. The old instant fast-path keyed off
this.index.size()>0, a dishonest proxy that skipped the metadata/graph
checks whenever the vector was warm and never fired on a real cold process
anyway; removed.
The per-leg fallbacks differ because "empty" means different things: the JS
vector's rebuild() IS its load, so size()===0 correctly triggers it; the
id-mapper backs metadata, so totalEntries===0 (past the empty-store return)
is a real load failure; but entities do not imply edges, so a graph
size()===0 is a valid empty state, not a load failure.
- The JS graph now COLD-LOADS its durable LSM instead of re-deriving from a
full canonical verb scan on every boot (baseStorage._initializeGraphIndex
loads the persisted SSTables via a new GraphAdjacencyIndex.init(); it
self-heals from canonical only when the durable state is genuinely missing).
This removes an O(E)-per-open cost every filesystem consumer paid.
- LSMTree.loadManifest loads its SSTables BEFORE publishing the relationship
count, and resets to an honest-empty state on load failure — a tree can no
longer claim persisted relationships while holding none (the silent-empty
cold-load class the query-time guards exist to prevent).
Verified end-to-end against a built brain: a warm reopen (with edges and
edgeless) reloads only the JS vector; the graph and metadata cold-load with
no rebuild, and queries return correct results. New tests in
cold-open-rebuild-gate.test.ts pin the contract (isReady() defers, self-heal
still fires); migration-deference updated to drive size-based deference
through the vector, the leg where empty->rebuild remains correct.
Pairs with the native provider's isReady()/init() implementation — brainy's
gate defers only to a signal the provider exposes.
A consumer's clean-room verification of the 8.0.10 fix found relate() still
hanging their scripts. Root-causing the CLASS instead of the repro found two
mechanisms:
1. The 'beforeExit' auto-flush hook looped forever on any script that never
reaches close(): Node re-emits beforeExit after every event-loop drain, and
the async flush schedules new work — flush, drain, flush, forever. The
listener now self-deregisters BEFORE its one flush, so the next drain exits.
(Empirically: process.on('beforeExit', async () => await anything) alone
never exits — this was the deepest root of the whole hang class.)
2. Every background-maintenance interval is now unref'd at creation — graph
auto-flush, LSM compaction, metadata write-buffer flush, VFS cache
maintenance, PathResolver cache maintenance, statistics debounce (the
writer-lock heartbeat, flush watcher, and cache monitors already were).
Durability is owned by close() and the beforeExit flush, both deterministic;
a best-effort interval must never keep the host process alive.
Proof: the reported shape (add x2 + relate, filesystem) exits cleanly BOTH
with close() (~0.5 s) and with NO teardown at all — and in the no-teardown
case the beforeExit flush still lands the data (verified by reopen: both
nouns + the edge present). New per-op-class sweep test asserts no ref'd
timer survives close() for add / relate / graph find / metadata update /
vfs — turning this bug class off permanently instead of per-repro.
A consumer report on the GA pair: any minimal add/find/close script hangs
forever. Root cause was FOUR ref'd keep-alives brainy never released:
1+2. The global SIGTERM/SIGINT shutdown hooks (registered once at init) were
anonymous and never removed — a process.on signal listener holds a ref'd
signal handle. They are now named statics, deregistered when the LAST live
instance closes (Brainy.instances also stopped leaking closed instances);
a later init re-registers them.
3. UnifiedCache.startFairnessMonitor() created an interval it never stored,
cleared, or unref'd — unclearable by construction. Now stored + unref'd
(same posture as the existing memory-pressure timer): a cache monitor must
never keep the host process alive.
4. brainy.close() never called VirtualFileSystem.close(), stranding the VFS
background-maintenance interval and the PathResolver maintenance interval.
close() now shuts the VFS down.
Proof: a bare script (init/add/close, no process.exit) exits 1 ms after
close() returns; before the fix it hung until killed. 3 new lifecycle tests
pin the hook semantics (once globally, survive while other instances live,
removed on last close, re-register after).
With plugins unset (the default), init() probes for the first-party
accelerator: not installed means plain brainy with zero noise; installed and
healthy means it loads and announces itself; installed but broken (import
failure, invalid shape, failed activation, version mismatch) makes init()
THROW. An installed accelerator never silently vanishes behind the JS engines
- the anti-drift posture of the explicit list, applied to detection.
plugins: []/false stays a true opt-out (no probe); an explicit list keeps its
required-and-loud semantics. The import runs through an importPluginPackage
seam (variable specifier - bundlers cannot static-resolve the optional
package; tests simulate all outcomes without it installed).
Also: when a plugin activates but registers zero native providers (e.g. a
licensing gate declining to engage), the provider summary now warns loudly
instead of leaving every query silently on the JS engines.
Supersedes 8.0.8's explicit-opt-in wording; README/PLUGINS/types now document
the guarded contract. 7 new tests (tests/unit/plugin-autodetect.test.ts).
cor confirmed the migration write-new / tmp+rename invariant the hard-link backup
relies on — with one exception: the native shared mmap `_id_mapper/*` is mutated
IN PLACE (msync + a rebuild truncate+re-inject), the same category as the tx-log.
A hard-linked snapshot of it would be corrupted by a live in-place mutation
reaching through the shared inode, so the backup would not be a faithful
pre-upgrade copy.
Add a SNAPSHOT_BYTE_COPY_DIRS set (`_id_mapper`) so snapshotToDirectory byte-copies
every file under it (the directory analogue of SNAPSHOT_BYTE_COPY_PATHS); the id
map is bounded, so the copy cost is small. Everything else (SSTables / .dkann /
manifests / objects, all tmp+rename) still hard-links. Test: `_id_mapper/*` gets
an independent inode + faithful content; a regular file stays hard-linked.
A downstream deployment reported find({where}) returning a silent [] on a
freshly-opened brain (a native metadata provider that reports data but hasn't
loaded its field postings), blanking filtered pages after every restart. Brainy
had a cold-read guard for GRAPH reads (verifyGraphAdjacencyLive → self-heal or a
loud GraphIndexNotReadyError) but no equivalent for metadata `where`.
Add verifyMetadataLive() — the field-index counterpart, one-shot per brain on the
first filtered find(): probe a known persisted entity's plain field value; if the
index serves it, live (the only cost on a warm brain — one O(1) probe). If not,
the postings didn't load: rebuild from canonical + re-probe; if still unserved,
throw the new exported MetadataIndexNotReadyError rather than let a silent [] pass.
Inconclusive cases (empty store, no plain field, shared-store foreign entity,
migrating provider) resolve to live — never a false rebuild.
The 8.0 open-core JS index already cold-loads correctly (verified: cold where 3/3,
cold related 2/2) — this guards the NATIVE path, where the durable cold-load cure
is cortex-side (same shape as the graph 2.7.8 cure). 4 unit tests (warm no-rebuild,
self-heal, loud-fail, no-filter-no-probe). Gates: typecheck 0, build 0, test:unit
1757/1757.
David's ask: back up the brain before the one-time upgrade, drop it after. On
open, when the on-disk format is stale (a 7.x→8.0 rebuild will run) and the store
holds data, brainy hard-link-snapshots the brain dir to a sibling
`<brainDir>.migration-backup` BEFORE any provider rebuilds — near-zero cost/space
(shared inodes; the store is immutable-by-rename), instant even at scale. Removed
automatically once the upgrade verifies + stamps the marker; retained on failure
for rollback. Default-on; opt out with `migrationBackup: false`.
- Reuses the existing snapshotToDirectory() hard-link farm via new optional
adapter methods createMigrationBackup()/removeMigrationBackup() (filesystem
only — feature-detected; memory + non-fs are a graceful no-op).
- Taken before the native provider inits, so it captures true pre-migration
state; a prior failed upgrade's backup is reused, not overwritten.
- Best-effort: a backup failure never blocks the upgrade (the migration is itself
safe — reads canonical, reconstructable, self-heals). Catastrophe-insurance
against a migration bug, not a data-loss guard or an off-device backup.
4 lifecycle tests (adapter hard-link / reuse / remove-without-touching-live /
empty→null; stale-epoch reopen create+remove; opt-out; memory no-op). Gates:
typecheck 0, build 0, test:unit 1753/1753, layout-migration suite 5/5.
Rewrite the four open-core distance functions (cosine / euclidean / manhattan /
dot-product) from object-accumulating `reduce` to single-pass allocation-free
indexed loops. cosine's per-element `{dotProduct,normA,normB}` object was the
hot-path GC lever.
MEASURED (tests/benchmarks/distance-microbench.mjs, dim=384, N=20000, median of
41): cosine 44.3ms -> 7.4ms (~6x), euclidean 9.2ms -> 6.6ms (~1.4x); the built
cosineDistance drops ~44ms -> ~9ms. Numerically identical (same ops, same order)
so recall is unchanged; full suite green (1753/1753).
Also drop the unfounded perf JSDoc ("faster than GPU", "Node.js 23.11+") and the
`new Function(distanceFn.toString())` eval in calculateDistancesBatch — with the
functions now tight loops, the batch is a thin JIT-inlined map (no worker, no
stringify/reconstruct).
Evidence-revised scope: the Float32Array resident-storage half of the original
Fork X is DROPPED. The same microbench shows Float32Array is ~1.7x SLOWER for
this compute (V8 widens f32 -> f64 on every element read), so it would regress
the hot path for a RAM win the open-core JS path does not need — billion-scale
vector RAM is the native provider's SIMD/mmap/quantized domain. The resident
representation stays number[]; no type-chain or cache changes.
Replaces rc.8's no-freeze deference with an automatic, observable, coordinated
migration LOCK (David's reversal: "unknown/halfway states are more dangerous
than blocking"). While a native provider runs its one-time 7.x→8.0
rebuild-from-canonical (`isMigrating()===true`), brainy blocks/queues data-plane
reads AND writes so no operation ever touches a half-built index — closing the
three seam-map gaps (lost mid-rebuild writes, degraded reads, read-time rebuild).
Mechanism (one choke point):
- awaitMigrationLock() at ensureInitialized() covers all ~40 data-plane methods;
a non-migrating brain pays one boolean check. Polls isMigrating() at 250ms;
after migrationWaitTimeoutMs (default 30s) throws a retryable, exported
MigrationInProgressError. The timeout bounds the CALLER'S WAIT, not the
migration — the rebuild is unbounded and never interrupted.
- init() awaits the lock before VFS bootstrap + before serving ("not ready until
upgraded"); a rebuild past the budget surfaces MigrationInProgressError at init
(raise the budget, or run the offline migrator).
Observability (readiness-probe correct — never gated):
- getIndexStatus() gains `migrating` + `migration` (MigrationProgress); a probe
maps migrating→HTTP 503+Retry-After, not 500. health()/checkHealth() lock-exempt
(health() reports a `warn` migration check). stampBrainFormat()/close() are
ungated so cor can clear the lock — no deadlock.
- Optional provider migrationStatus() is relayed verbatim for a live %.
verifyGraphAdjacencyLive() honors the lock (no self-rebuild mid-migration); the
stamp is still withheld while any provider migrates (cor stamps after verify).
Adversarially verified: fixed a critical init/VFS-bootstrap deadlock, a mixed-
provider busy-spin (dropped the event-driven signal → pure poll), a not-yet-
initialized getIndexStatus crash, and once-per-window log/clock resets. 10 lock
tests (incl. the init-during-migration regression). Gates: typecheck 0, build 0,
test:unit 1753/1753.
Consumer-invisible modernization pass — no public API or runtime-behavior
change; dist for the override-only files is byte-identical.
Toolchain:
- CI: GitHub Actions matrix — Node 22/24 (test:unit) + Bun latest (test:bun).
- engines: node ">=22" (was "22.x"), bun ">=1.1.0".
- tsconfig: isolatedModules + noImplicitOverride; add the 33 `override`
modifiers the flag requires across storage/integrations/vfs/transaction.
- deps: @types/node ^22; add prettier; drop dead standard-version,
@rollup/plugin-* and the redundant embedded eslintConfig (flat
eslint.config.js is the active config — verified identical lint output).
paramValidation: replace the top-level `await import('node:os'/'node:fs')`
with static ESM imports. The top-level-await form poisoned the module graph;
static imports also drop the browser/edge fallback branches no supported
runtime reaches (8.0 is Node/Bun/Deno-only).
Bun positioning: recommend Bun as a runtime (`bun add` / `bun run`), which is
green (test:bun 8/8). Drop single-binary `bun build --compile` as a target —
native addons cannot embed into it, and Bun 1.3.10 has a `--compile` codegen
regression around top-level await. Rename the Bun test to bun-runtime-test.ts
and correct docs that overclaimed single-binary support.
Gates: typecheck 0, build 0, test:unit 1743/1743, test:bun 8/8.
Three hooks so a native provider can run a non-blocking, online, background
build-new→verify→swap index migration on a large-brain upgrade while brainy stays
out of the way — no minutes-long blocking rebuild-on-open/first-query.
- isMigrating?(): boolean — OPTIONAL on MetadataIndexProvider / GraphIndexProvider /
VectorIndexProvider (mirrors isReady?()/init?(), feature-detected). While a
provider reports migrating, brainy SKIPS its rebuild for that index — per-index,
so a non-migrating sibling still rebuilds; skipped even under epoch-drift or
size()===0 — in both rebuildIndexesIfNeeded and the lazy first-query force path.
The provider serves correct reads from canonical until it verifies-and-swaps.
- brain.stampBrainFormat() — public; the provider calls it once its background swap
verifies, so brainy authors dataFormat (the provider never does). brainy withholds
its own marker stamp while any provider is migrating, so the shared epoch is never
advanced ahead of a deferred index.
- ./brain-format export — the marker module's EXPECTED_INDEX_EPOCH / CURRENT_DATA_FORMAT
are importable so a native provider shares the constant (no duplicate = no
lockstep-drift). 8-case test; unit 1743 green.
The MVCC time-travel layer kept nounChains/verbChains as unbounded
Map<id, number[]> — one resident chain per id ever touched across retained
history (O(N) RAM, defeating billion-scale time travel). Replace with a hot-tail
window + bounded cold LRU + a mutex-free bulk resolver, keeping resolveAt exactly
correct.
The most-recent W generations stay resident as full chains (the common recent-pin
read is O(log), zero scan); deeper pins reconstruct one id's chain on demand into
an L-bounded LRU. Reconstruction is LOCK-LIGHT — it holds no commit mutex, so a
historical read never stalls writers; correctness holds because a live pin's
answer is always > minPinnedGeneration, which compaction can never reclaim, and a
concurrently-reclaimed gen below that is skipped. materializeAtGeneration routes
through a new mutex-free resolveManyAt (one forward pass, O(R + |ids|)) — without
it a deep-pin materialize both regresses to O(N*R) and deadlocks on snapshotWith's
mutex. The cold cache is invalidated on re-touch to stay coherent. Resident RAM is
O(W*d + L), independent of N. 11-case test (oracle-vs-bruteforce, held-Db across
eviction + concurrent compaction, no-deadlock, write-path I/O-free); unit 1735 +
integration 613 (db-mvcc/db-temporal/db-asof green).
Add _system/brain-format.json { dataFormat, indexEpoch } + a sync brain.formatInfo()
accessor + a compiled EXPECTED_INDEX_EPOCH, the surface a native provider reads at
init() to drive whole-brain auto-upgrade, and brainy's own derived-index
rebuild-on-format-drift trigger (closing the gap where JS indexes rebuilt only on
size()===0, never on a format-version change).
indexEpoch is shared and lockstep-bumped with the native provider on any coordinated
release whose on-disk derived-index format changes; dataFormat is brainy-owned. On open,
the marker is read in the store-open phase BEFORE provider construction (so formatInfo()
is synchronously available at the provider's init); a drifted or absent epoch rebuilds
the derived indexes from canonical records, and the marker is stamped only AFTER the
rebuild verifies (build-new -> verify -> stamp; a crash before the stamp re-triggers the
idempotent rebuild). brainFormat.ts is the single source of the shared constants.
6-case test; 1724 unit green.
8.0 shipped with no cold-graph guard (the 7.33.4 fix was never ported), so on a
cold open of a large brain where the native adjacency reports membership but its
source->target edges did not load, find({connected})/neighbors()/related() could
silently return [] for persisted edges.
Add the converged isReady() contract: GraphIndexProvider.isReady?(): boolean is
the honest cold-load readiness signal (true ONLY when edges are loaded), so brainy
gates on it instead of the lying membership-size() proxy. verifyGraphAdjacencyLive()
checks it — Strategy 1: false -> hydrate the id-mapper, rebuild from storage, re-check,
throw GraphIndexNotReadyError if still not ready; providers without isReady() fall
back to a global known-edge-sample probe (Strategy 2, not the queried anchor, so a
genuinely edgeless node still returns []). rebuildIndexesIfNeeded gates the graph
rebuild on it too, with the id-mapper hydrated before the adjacency rebuild on the
lazy cold-open path (the CTX-BR-RESTORE-REBUILD ordering, shared with restore()).
executeGraphSearch re-verifies before trusting an empty connected result and
re-collects on a heal. 6-case integration test + 1718 unit green.
Both shapes confirmed with cor for the lockstep (cor builds the native side
against these; the JS index is a no-op / unchanged):
- probeConsistency?(): Promise<boolean> — OPTIONAL O(1) cold-open consistency
sampler on MetadataIndexProvider. brainy calls it ONCE per brain on the first
read; on `false` it self-heals via detectAndRepairCorruption() — the metadata
counterpart of the 7.33.2 graph cold-load guard, completing the phantom triad's
detect-and-repair-on-open. Best-effort: a probe failure never breaks a read
(the one-shot guard resets so a transient failure retries). The JS index omits
the method, so it is simply never probed.
- getIdsForFilter(filter, opts?: { limit?; offset? }) — brainy passes a page
bound on the UNSORTED find({ type, where, limit }) path so a native provider can
early-stop and return only the [0, limit) prefix, killing the O(N) FFI marshal
at billion scale (pairs with cor #75). brainy passes offset:0 and ALWAYS
re-windows the result itself (visibility filter + slice); the JS index ignores
opts and returns all matches (behaviour unchanged).
New unit tests cover brainy's call behaviour for both (probe→repair on false,
healthy→no-repair, failure-is-best-effort, once-per-brain; opts passed with
offset 0 on the unsorted path). Full gate green: unit 1718, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
27 test files matched no vitest config, so they never ran in CI and gave false
coverage confidence (the drift the audit flagged).
- Re-homed 10 functional suites into the gate (renamed to *.unit.test.ts /
*.integration.test.ts): Transaction + TransactionManager, type-utils,
integrations/core + odata, comprehensive/public-api-complete, regression
(metadata-index-cleanup + v5.7.0-deadlock), vfs/tree-operations +
vfs-bulkwrite-race. ~192 previously-dark tests now run and pass.
- Deleted 8 bit-rotted/redundant files that fail against 8.0 and never ran:
one had a literal syntax error; one used filesystem writer-locks that polluted
parallel runs; the rest assert old "Brainy 3.0/v3.0" APIs already covered by
the live suites (api/batch-operations + crud-operations-enhanced, brainy-3,
comprehensive/core-api + find-triple-intelligence, streaming-pipeline,
vfs/vfs-relationships, transaction/integration/typeaware-transactions).
- Added a coverage guard (tests/unit/test-suite-coverage-guard.test.ts) that
FAILS CI if any *.test.ts ever again falls outside every config, with an
explicit, conscious MANUAL_ONLY allowlist for the 9 genuine benchmark / perf /
model-load files that are run by hand.
Gate green: unit 1712, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The JS metadata index served `ne`, `exists:false`, and `missing:true` by
materializing the ENTIRE id universe as an array of UUID strings, building a Set
of the excluded ids, and filtering — O(N) heap and two full passes on the
open-core path, on the most common shape (`deleted !== true`).
Compute the complement as a roaring-bitmap difference over the int-id universe
instead (`complementIds(excludeInts)` = universe \ exclude), converting only the
result back to UUIDs. The exclude set is small (the matching value), so this
avoids the full-corpus string materialization entirely. Behaviour is unchanged,
including the load-bearing soft-delete semantic that `field !== value` includes
entities with no such field. New focused tests pin `ne`/`exists:false`/
`missing:true`/`exists:true` result sets and that the complement reflects deletes.
Full gate green: unit 1520, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
removeItem() scanned the ENTIRE corpus on every delete to find nodes that
referenced the removed id (and to repair edges left asymmetric by pruning), so a
delete was O(N) and a bulk delete O(N²) — on the open-core JS vector path that
serves when no native provider is registered.
Maintain a reverse-adjacency index (`target → level → set of nodes that link to
target`), so removeItem touches only the removed node's actual in-neighbors:
O(in-degree) per delete, O(N·degree) for a bulk delete. The index is lazily
built, maintained incrementally at every forward-edge mutation (add-link and
prune), and invalidated (rebuilt on next use) by the bulk paths (cold-load
restore + clear). New tests assert the three invariants that matter for a
reverse index: no dangling references to deleted nodes, the incrementally-
maintained index exactly equals a fresh rebuild from the live adjacency, and
search still returns the survivors' true nearest neighbours (exact vs brute force).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- find() search-mode: collapsed the two overlapping options to one canonical
`searchMode: SearchMode` (the redundant `mode` alias was silently ignored on
the primary find() path while honored on the historical path — a footgun) and
removed the unwired `explain?` FindParams field (never read by find()).
- Documentation accuracy on the public type surface: entity ids documented as
UUID v7 (auto) / v5 (natural key), not v4; dropped the stale "Brainy 3.0"
banners and "backward compatibility" hedging on the fresh-8.0 Result type;
documented the via/type alias; removed the dead GraphConstraints.bidirectional;
fixed the no-op `EmbeddedGraphVerb = Omit<GraphVerb,'source'>` to omit the real
sourceId; corrected the GraphIndexProvider.isInitialized JSDoc; documented
removeMany's per-chunk generation granularity; JSDoc'd the public VFS surface.
- Honest perf comments in source: removed unmeasured billion-scale figures from
the LSMTree / graphAdjacencyIndex headers (the JS fallback is in-memory after
open; native is the scale path).
- Robustness: getVerbMetadata propagates read errors symmetrically with
getNounMetadata; the find() egress integrity-guard keeps a row when the JS
matcher doesn't implement an operator the provider already matched on; hardened
the boundary-no-native CI guard to catch side-effect imports + re-exports.
- Tests: de-theatricalized a relateMany test to assert the real contract; fixed
a stale Model-B header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Version-coupling cold-init: getBrainyVersion() returned a stale build-time
default ('3.14.0') on the FIRST synchronous call — the one loadPlugins() makes
to build context.version — because the package.json read was async. A native
provider declaring a realistic `>=8.0.0` range was therefore rejected on cold
init. version.ts now reads package.json synchronously (8.0 targets Node-like
runtimes only); added a cold-init coupling regression with a `^8.0.0` plugin.
- No-silent-failures on the highest-fan-in read path: getNouns()/getVerbs()
converted a storage read failure into a success-shaped empty page, which the
cold-start rebuild then read as "store empty" and skipped the rebuild — booting
a permanently-empty index with no signal (the same silent-failure class as the
phantom bug). Both now re-throw a named BrainyError; the rebuild path re-throws
read failures (fail loud) and records a queryable degraded state, surfaced via
checkHealth(), for non-fatal rebuild hiccups.
- LSM SSTable leak: graph compaction wrote a merged SSTable but only dropped the
old ones from the manifest, orphaning their payloads forever ("In production
we'd add a cleanup mechanism"). Added StorageAdapter.deleteMetadata() and now
reclaim each compacted-away SSTable.
- Documented the two headline methods add() and find() (the only undocumented
public methods on the class).
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GA-readiness audit found the public docs had drifted from the shipped
surface and presented uncited performance numbers as measured fact.
- quick-start: `FindResult`→`Result`, `VerbType.BuiltOn`→`DependsOn` (the
canonical getting-started example now compiles).
- noun-verb-taxonomy: rewrote every sample off removed/fictional APIs
(`augment`/`connectModel`/`getVerbs`/two-arg `add`/`like`/`$gte`) onto the
real single-object `add`/`find`/`relate`/`related`; replaced the stale
31-noun/40-verb catalogs with accurate, complete tables (42 nouns, 127 verbs).
- triple-intelligence: `like:`→`query:`, dollar-operators→bare operators, and
several other fictional keys swept to the real `FindParams`.
- FIND_SYSTEM / PERFORMANCE / index-architecture / BATCHING: replaced
fabricated, mutually-inconsistent latency tables and uncited speedup
multipliers with Big-O characterizations, qualitative mechanism descriptions,
and the one genuinely-measured benchmark (graph O(1) neighbor lookup), per the
evidence-based-claims rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
find() trusted the metadata index's returned ids: it loaded each id's entity
and returned it, checking only that the entity existed — never that it actually
matched the query. The indexes are acceleration structures; the loaded entity
is ground truth. A stale or cross-bucket index posting (e.g. an id left in a
field-value bucket by a delete or an `update({ field: undefined })`) therefore
surfaced an entity matching NEITHER the requested type NOR the where filter — a
production report saw a timeslot (NounType.Event) returned for
`find({ type: Person, where: { entityType: 'staff' } })`.
Add a single egress chokepoint after the result IIFE that re-validates every
result with `entityMatchesFind` (type/subtype/where/service/excludeVFS) — the
live-path mirror of the historical path's per-candidate check in db.ts. It
covers every find() branch (metadata, vector, text, proximity, graph) and
`similar()` (which delegates to find) in one place. A no-op on a healthy index;
on a corrupted one it drops the bad row instead of returning a phantom. Full
unit + integration gate green confirms the where re-validation is consistent
with the index (no valid rows dropped).
This is the brainy-side safety net. The durable corruption itself lives in the
metadata index that owns the query (the native provider when present) and is
addressed separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The optional native graph-acceleration provider exposes `isInitialized`,
which is false during its cold-start / rebuild window (engine + cursor not
yet loaded). The accessor cached the resolved provider INSTANCE but never
re-checked readiness, so `brain.graph.*` could dispatch to a not-ready
provider and throw instead of answering.
Resolve and cache the instance once, but check `isInitialized` LIVE on every
call: route to the pure-TS analytics path while the provider is not ready,
and re-engage the native path automatically the moment it flips true. This
gates every native route at once — subgraph, export, rank, communities,
path, and the query→expand fusion. Adds a test asserting the provider is
never called (and the TS fallback returns real answers) while not ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-GA dead-code sweep. A deterministic import-reachability walk from the
package's real entry points (the exports map, the CLI bin, the conversation
surface) found 34 source modules that nothing reachable imports — they
compile and ship as dist output that no consumer or internal path can ever
reach. All were superseded duplicates, abandoned parallel implementations,
or built-but-never-wired features:
- superseded duplicates of live modules: an older "unified" entry, a
standalone neural-import variant, a static NLP processor + its matcher,
a parallel API-types module, a duplicate progress-types module
- an abandoned import path (orchestrator + entity deduplicator + barrels)
left behind when ingestion moved to the coordinator
- unwired feature modules (instance pool, import presets, cached
embeddings, relationship-confidence scorer) reachable only from tests
- dead leaf utilities (write buffer, deleted-items index, bounded
registry, two crypto shims, a cache manager, a structured logger, a
stale v5 type-migration helper, a browser-only FS type shim) and
several dead re-export barrels
- a CLI catalog command wired into no command
Also removed two tests that only exercised deleted modules, repointed two
VFS tests off a deleted barrel onto the implementation module, and deleted
one example that demoed removed features.
Verified: clean build (both tsc passes), unit 1505 + integration 607 green,
and a re-run of the reachability walk reports zero remaining orphans.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes brainy's #35 side. When the at-gen vector gate serves a filtered
semantic read, Brainy now ships the historically-correct candidate vectors so the
native provider reranks against them with zero per-vector FFI crossing — the
provider need not duplicate the per-generation retention Brainy already does.
- New `AtGenerationVectors { ids: BigInt64Array; vectors: Float32Array; dim }` on
the VectorIndexProvider.search options bag (row-major: vectors[i*dim..] == ids[i];
invariant vectors.length === ids.length*dim; ids = the candidate set; dim must
match the index dim). The built-in JS index ignores it.
- buildAtGenerationVectors resolves each universe id's vector AS OF the generation
(the record before-image, or live getNoun when untouched since the pin — not
readNounRaw, whose canonical path is empty under lazy-vector eviction), interns
the id via the shared mapper, and packs row-major. Absent/vectorless/wrong-dim
ids are dropped (not vector-rankable). Bounded by the filtered universe.
Shape + the four clarifications (row-major, ids-are-the-candidate-set, dim-asserted,
filtered-path-only) confirmed with cor in the handoff #35 thread. Inert until cor's
native at-gen rerank advertises isGenerationVisible (honesty guard).
Tested: the mock versioned provider now asserts it receives atGenerationVectors with
the row-major invariant, correct dim, and ids resolving back to the at-gen universe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`should cache paths for fast repeated access` compared a single cold vs warm
readFile with Date.now() (ms resolution). Both reads are sub-millisecond, so the
warm read intermittently measured 1ms vs the cold 0ms and the `time2 <= time1`
assertion failed on rounding noise — a non-deterministic gate that blocked a clean
release. Replace it with the average of 100 warm reads via performance.now()
against a generous absolute bound (cached path reads are sub-ms), plus a content
round-trip check. Same intent (cached access is fast), no single-shot ms flake.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered + ordered find() materialized the FULL sorted match set before slicing
the page — passing k = filteredIds.length to the column store's filteredSortTopK.
A broad filter + orderBy returning 20 rows at billion scale produced hundreds of
millions of sorted ids to discard all but the page (O(matches) heap).
Thread the page bound (offset + limit + the hidden-tier over-fetch) into
getSortedIdsForFilter → filteredSortTopK / sortTopK / the sparse-path slice, so
the sort produces only the page. Ordering and pagination are unchanged.
From cor's 2026-06-24 co-release scaling audit (item 1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The subgraph seed selector now accepts not just entity id(s) but a find() result
or a FindParams query — brain.graph.subgraph({ where: { team: 'platform' } },
{ depth: 1 }) runs the query and expands the neighborhood of every match in one
call. Existing id-seeded calls are unchanged.
The win is the native query→expand path: for a metadata-only query, the matched
universe is forwarded to traverse() as an OpaqueIdSet (a roaring Buffer) with NO
id materialization in TypeScript — the find() result never leaves the engine's
representation (the O(1)-crossing the cor 3.0 contract is built around). The
pure-JS path (or any query carrying vector/text/proximity criteria) materializes
the matched ids via find() and seeds the traversal from them, capped at a bounded
default when the caller pins no limit.
- New public `SubgraphSelector<T>` union (id | id[] | Result[] | FindParams).
- find()'s metadata filter-builder extracted to a shared `buildMetadataFilter`
(reused by the opaque universe producer); behavior unchanged.
- graphSubgraphNative generalized to accept `bigint[] | OpaqueIdSet` seeds.
Tested: JS query→expand + Result[] selector + empty-match in graph-subgraph, and
the native opaque pass-through (Buffer reaches traverse unmaterialized) vs the
find()-materialized bigint[] seeds via the mock provider in graph-native-routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
