Four fixes from a consumer conformance report, one root disease — two
field-resolution regimes where there must be one:
- The delete/update aggregation hooks fed the engine a partial entity
view (type/service/data/metadata only), so a reserved-field groupBy
(subtype, visibility, ...) resolved to a nonexistent group on the way
down: counts drifted upward forever after deletes, and updates moving
an entity between reserved-field groups double-counted. The hooks now
pass the full-fidelity view via entityForAggFromRawRecord (every
reserved field top-level, mirroring the add path); the update sites
pass the full get() view instead of a hand-rolled subset.
- Aggregation source.where resolved fields only against the custom
metadata bag, so where on a reserved field silently matched nothing.
The matcher now resolves each filtered field through
resolveEntityField — the same single source of truth groupBy uses.
- removeMany() with no usable selector (bare array passed positionally,
empty params, ids: []) resolved successfully having deleted nothing.
All three now throw; the two legacy tests that pinned the silent
no-op as 'graceful' now pin the refusal.
- find() where keys accept both spellings: a metadata.-prefixed key
falls back to its flattened spelling when the prefixed one is not
indexed (metadata is flattened at index time). A literal nested custom
key named metadata still wins when indexed as spelled.
Five regression pins in aggregate-reserved-fields.test.ts (4 of 5 vary
red on the unfixed code).
Under default adaptive retention, every flush() recomputed total history
bytes by walking EVERY committed generation's delta — O(all generations)
with disk re-reads past the 4096-entry delta-cache bound. On a production
brain with 70,000+ accumulated generations this turned every write into a
full-tail scan (60-100s writes, escalating with history growth), even
though the free-RAM budget never tripped and nothing was ever reclaimed
(SELF-GENERATIONS-GROWTH).
historyBytes() now maintains a running total: seeded by one walk on first
use, then updated incrementally at both commit paths (+bytes) and the
compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The
adaptive retention check on every flush is O(1). Invariant regression-
pinned: running total ≡ fresh walk through transact commits, single-op
group commits, and compaction.
New brain.historyStats() (exported HistoryStats): read-only generation
count / bytes / generation+timestamp range / horizon / retention mode /
effective budget — the one-call per-brain fleet audit for retention
exposure.
- New src/utils/osLimits.ts: reads RLIMIT_NOFILE (soft/hard, from
/proc/self/limits) and vm.max_map_count at open — once per process,
Linux-only, measurement-only — and warns loudly when either sits below the
pool-scale floors (soft NOFILE < 65536, max_map_count < 262144), with the
exact raise commands. On stock defaults the failure otherwise arrives as
EMFILE or a failed mmap deep inside an index open, long after the cause
stopped being visible. An unreadable limit produces NO warning — no
measurement, no claim — so non-Linux platforms stay silent.
- Exported for ops doors: checkOsLimits() returns the full OsLimitsReport;
floors exported as constants. Wired fire-and-forget in performInit after
storage init; the check can never affect open.
- Unit tests pin the parser (incl. 'unlimited'), the floor thresholds, the
null-never-warns rule, and the off-Linux silent path.
- Aggregation backfill walks build into a STAGING map and swap in atomically
on completion. A mid-walk failure drops the staging map — the previous live
state keeps serving, the aggregate stays flagged pending, and the storage
error surfaces to the failing query. Previously the walk wiped live state
before a scan that could throw, never cleared the pending flag on failure,
and re-ran a full walk on every subsequent query: a silent wipe/walk/throw
loop at the caller's retry rate.
- Failed walks are latched: retries within a 30s cooldown rethrow the recorded
error instantly instead of re-walking, so a tight caller-side retry loop
costs one loud error per query, never a full store walk per query.
- Persisted aggregation state is stamped with the store's committed generation
at flush; reopen adoption requires stamp equality. Stale state (unclean
shutdown) or over-counting state (a log truncation on a copied store pulled
the watermark back) triggers exactly one loud rescan, never a silent adopt.
- The backfill/adoption path narrates: adoption decisions, walk start/finish
with counts and duration, and failures all log by default.
- getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a
loud error instead of silently restarting the walk at offset 0 (which
re-served page 1 forever to any while(hasMore) caller).
- The graph cold-load verb walk aborts loudly on a missing or non-advancing
cursor with hasMore=true.
- A versioned index provider whose generation is AHEAD of the committed
watermark (torn copy / crash-recovery truncation) is now named loudly at
open, alongside the existing behind-direction message.
- AggregationIndex: defineAggregate before init no longer forces a backfill.
init reconciles instead of clobbering: the app definition wins, persisted
state is adopted on hash match, a write landing pre-adoption forces an exact
rescan, and a successful state load clears the backfill flag. New ready()
settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
store walk and every pending aggregate fills from that same walk; the old
behavior let each concurrent query wipe the others' partial state and start
its own full walk, so a store under steady aggregate traffic never converged.
queryAggregate also waits for persisted definitions before deciding an
aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
at 1000 - below the documented 10000 auto floor, reported under a stale
basis label) is removed; the cap comes from its construction-time basis or
explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
semantics, RELEASES.md 8.5.1 entry.
Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.
- Wire format: positional msgpack facts [generation, timestamp, ops,
meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
32-byte segment header (magic, formatVersion, firstGeneration,
zeroed+verified reserved); length+crc32c frame per fact; zero-padded
segment names so lexicographic order == generation order; JSON
manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
the existing durability window, so a crash can only leave the log
AHEAD of committed truth — open() truncates back (torn tails detected
by CRC). Absent generation = never committed; a scan can never see an
uncommitted fact. transact() facts are durable-on-return; single-op
facts ride the group-commit flush exactly like buffered history. A
fact-append failure fails the write, loudly — a silent gap would be a
lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
telemetry: head/segments/approx up front, per-batch generation range
+ bytes + segment id, loud abort on gaps, summary cross-check) and
brain.factSegmentPaths() (immutable sealed segments for zero-copy
consumers; the mutable tail excluded). Exported types CommitFact,
FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
feature-detected; filesystem + memory adapters implement them; an
adapter without them hosts no fact log. Fact segments are byte-copied
(never hard-linked) into snapshots. The _generations/facts/ namespace
is registered as a protected family (rebuildable: false): no sweeper
or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
Two production-reported spine fixes:
- Canonical delete is a FULL removal: remove()/removeMany() deleted the
metadata leg but never the canonical vectors.json leg or the <id>/
directory, leaving ghost rows that inflated enumerated counts forever,
read as damage scars, and caused duplicate readdir entries for
re-created VFS paths (the stale-unpost path). The delete operation now
routes through storage.deleteNoun — both legs + the entity container —
with a full two-leg before-image rollback; deleteVerb symmetric; the
blind 'no metadata file' catch that masked real faults is gone. The
generation log still holds the delete's before-image, so asOf() history
is unchanged. repairIndex() gains a conservative, loud orphan-prune
sweep (containers with no metadata content leg) + count recompute for
stores damaged by earlier versions.
- Family-scoped migration gate: every read blocked on the whole-brain
migration lock even when the migrating index was irrelevant. The gate
now scopes to the index families a read actually consults — canonical
reads never wait; find() waits only on its query shape's families;
graph traversals wait only on the graph family. Writes keep the
conservative whole-brain wait. A read that needs the migrating family
still blocks (bounded) with the retryable MigrationInProgressError.
The canonical enumeration walk (getNounsWithPagination) hydrated each entity's
vector + metadata ONE-AT-A-TIME inside the shard loop — every enumeration paid
N x per-op-latency serially, and every index heal enumerates canonical, so this
was the dominant multiplier in the measured multi-minute heals (cortex heal-cost
decomposition).
- Hydration is now 16-way bounded-concurrency (matching the wave cor's native
rebuild uses): heal wall-clock becomes ~2xN/16 x per-op instead of N x per-op.
Order, cursor resume (skipped nouns still never read), filters, peek/hasMore
and totalCount are all preserved — pages are byte-identical to the serial walk.
- New getNounIdsWithPagination(): the id-only opt-out for callers that own their
IO schedule (an index heal). Unfiltered = ids straight from the shard paths,
ZERO per-entity reads; filtered hydrates metadata only (16-way). Same
cursor/offset/nextCursor contract, so it is page-compatible with the hydrating
walk.
Regression: pagination-parallel-hydration.test.ts pins paged==big-page identity,
ids==items order, zero-read id-only, and filter parity. 103 storage tests green.
The class-killer for the lost-main.dkann incident (ADR-004 §7). A provider can
declare a derived-index blob FAMILY (a set of members that are load-bearing
together, e.g. vector-base = main.dkann + main.slotmap + main.slotrev). Once
declared:
- deleteBinaryBlob / removeRawPrefix REFUSE to remove a declared member, throwing
the new ProtectedArtifactError — an in-process GC / sweeper is now INCAPABLE of
deleting a load-bearing index file (COLD != DEAD). Intentional retirement is an
explicit unregisterDerivedFamily(name) first.
- The declaration persists to _system/derived-artifacts.json, so protection
survives a reopen; clear() resets it with the rest of the derived footprint.
- checkDerivedFamiliesPresent() names any member missing on open (the catch for an
EXTERNAL deleter that bypasses the in-process refusal) → rebuild from canonical.
- Transients (*.tmp.*, *.rebuild-tmp, *.rotate-tmp) are never protected; a
namespace family protects a growing prefix (seg-*).
New StorageAdapter surface (optional): registerDerivedFamily / unregisterDerivedFamily
/ listDerivedFamilies + DerivedFamilyDeclaration; new exported errors
ProtectedArtifactError / DerivedArtifactMissingError. Enforcement is inert until a
provider declares a family (no regression). Cor declares its 6 families + does the
atomic set-swap in 3.0.15 (M2); brainy builds the contract now. 8 tests.
validateIndexConsistency() was blind to native providers — it only ran the JS
metadata index's own check, so a native provider whose manifest/segments/counts
had diverged still read as "healthy". Per ADR-004 §6 it now feature-detects and
aggregates each provider's optional validateInvariants() (a never-throwing,
<50ms self-report of its own cross-layer invariants), names any failing
invariant with its numbers in the recommendation, and exposes the per-provider
reports. A provider that violates the never-throw contract is surfaced as
unhealthy, never swallowed.
repairIndex() now reconciles NATIVE derived state from canonical too: it consults
each provider's invariants and calls rebuild() on any whose failing invariant
asks for heal:'rebuild' — the native counterpart of detectAndRepairCorruption().
New provider surface: optional validateInvariants(); new exported types
ProviderInvariantReport / InvariantResult / InvariantHeal. Additive, no break.
Cor implements the hook in 3.0.15 (M2); brainy builds against the shape now
(feature-detected, inert until a provider exposes it). 5 tests.
Closes the "dishonest readiness proxy" anti-pattern (Pattern A): size()>0 /
isInitialized were treated as "this index serves queries", but a cold native
index can load its COUNT before its SERVING structure, so a query returned a
silent [] indistinguishable from "no such data". A shared assessIndexReadiness()
now reads only the provider's honest isReady() signal (never size()), applied at
every site:
- Vector: a one-shot verifyVectorLive() guard on the semantic/proximity search
path (a pure semantic find({query}) has no filter, so nothing guarded it). It
prefers isReady(), else a known-vector self-match probe; self-heals via rebuild
or throws the new VectorIndexNotReadyError instead of a silent [].
- Graph: getVerbsBySource/ByTarget skip the fast path when the provider reports
not-ready (falling to the canonical shard scan), plus a one-shot probe that
self-heals a no-isReady provider whose adjacency did not cold-load.
- getIndexStatus(): folds in per-index honest `ready` (making `populated`
honest) + rebuildFailed/rebuildError/degradedIds, so a readiness probe never
200s a brain that is still warming up or degraded.
Unblocked by the native providers now reporting serving-truth (graph via
SSTable-residency readiness, vector via durableBaseLoadFailed). New export:
VectorIndexNotReadyError. getIndexStatus gains additive fields. No breaking API.
13 new tests; existing readiness guards green.
The Pass-1 hardening that makes loadBinaryBlob distinguish genuine absence
(ENOENT -> null) from a real fault (EIO/EACCES/... -> throw) was held out of
8.2.6 because the native accelerator's two column-store read sites still relied
on null-on-error. That accelerator release has now hardened those sites to
handle the throw (a faulted segment read marks the field unavailable and throws
a named error), so the fault-propagating form is restored and ships lockstep.
A present-but-unreadable index blob no longer masquerades as absent (which drove
needless rebuilds / empty reads). Adds the loadBinaryBlob leg to the blob
durability suite (absent -> null; present -> bytes; real fault -> throws).
The debounced materialization caught its failure with an empty `catch (() => {})`,
silently leaving the materialized Measurement entity stale; the aggregation-index
init() state-load failure was swallowed the same way, leaving aggregates reading
empty with no signal. Both are non-fatal (values rebuild via backfill-on-query),
but a silent stale/empty read violates loud-errors-never-quiet-losses. Both now
emit a loud warning naming the affected group / cause.
Two known-degraded states were recorded but never consulted by the read paths, so
a partial result looked authoritative:
- commitSingleOp returns `degraded` ids on an adopt-forward failed-rollback
recovery (the canonical record is durable but its derived-index entry may be
incomplete). persistSingleOp dropped that list on the floor — no health flag,
no read signal. It now records them in a queryable degraded set.
- A non-fatal index-rebuild failure at init (_indexRebuildFailed) was folded into
checkHealth()/validateIndexConsistency() but no read consulted it.
find() and get() now emit ONE loud warning per degraded window (reads still
return — canonical is the source of truth — but the caller is told results may be
partial and to run repairIndex()). Both degraded sources fold into the two health
surfaces, and repairIndex() reconciles from canonical and clears them.
With the unique per-writer temp suffix, a rename ENOENT can no longer mean "a
concurrent idempotent writer already renamed it" — nobody else holds this
writer's temp. It means our just-written temp vanished before the rename, so the
bytes did NOT land. The old code returned success on that ENOENT, acknowledging
a write that persisted nothing; the native provider mmaps these blobs, so a
phantom-acked blob is a silent-loss.
saveBinaryBlob now retries once with a fresh temp (self-healing a transient
external-sweeper/crash-cleanup race), and if the temp vanishes again it throws
loud rather than acknowledging a store-nothing write. Non-ENOENT rename faults
propagate verbatim as before. The unique-temp suffix already eliminated the
concurrent same-key collision that originally motivated the ENOENT shortcut.
The async group-commit flush that persists single-op generation history
swallowed persist failures as a bare warn: writes kept succeeding while their
before-images piled up in memory, never durable and unbounded, with no signal.
The generation store now accounts for every failed flush at one place
(flushPendingSingleOps), tolerates a transient blip (retry with capped
exponential backoff), and after PENDING_FLUSH_FAILURE_THRESHOLD consecutive
failures LATCHES a durability failure and refuses further single-op and transact
writes with a typed, exported PendingFlushDurabilityError — rather than promise a
durability it cannot deliver. Live canonical data is untouched; only the
immutable history is stuck. The latch self-heals: the moment a flush succeeds
(a retry, or an explicit flush()/close()) it lifts and writes resume.
Loud errors, never quiet losses.
clear() removed entities, indexes, system and _cas but left three top-level
trees on disk: _blobs (raw HNSW/LSM segment bytes and the native dkann index),
_id_mapper (the native shared mmap id-mapper), and _column_index (column-store
manifests). A cleared brain therefore re-read stale native state.
Worse, the column store splits its state across two of those trees — manifests
under _column_index/ and their segment bytes under _blobs/_column_index/ — so
removing one without the other stranded a manifest listing segments that no
longer exist, which the hardened segment-load path now (correctly) refuses with
ColumnSegmentLoadError. The three trees now fall together as a set, exactly as
_cas already does. locks/ is deliberately preserved: it is live coordination
state, not data.
ColumnStore silently skipped a manifest-listed segment it could not load: a
corrupt or missing segment dropped every one of its entities out of
filter/rangeQuery/sortTopK with no error, so an inconsistent index read as a
merely short result. It now throws a typed ColumnSegmentLoadError when a listed
segment yields undecodable or no bytes, and lets a genuine storage IO fault
propagate verbatim. Only a field with no manifest at all stays benign (nothing
was ever written for it).
baseStorage.getNoun_internal / getVerb_internal likewise caught every error and
returned null, reporting a present-but-unreadable entity as "not found". They
now return null only for genuine ENOENT-class absence (via isAbsentError) and
rethrow real faults and deserialize errors.
Pattern-B (blind catch) hardening: absence -> null, fault -> loud.
Write/index-spine hardening, first batch of Pass 1. Each fix restores an
invariant the surrounding code already intended; every one has a
fail-before/pass-after test.
- Pattern C, finding 5 (baseStorage): delete now decrements the user-facing
scalar total symmetrically — deleteNounMetadata was decrementing only the
per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so
getNounCount()/getVerbCount() inflated permanently (the stale scalar wins
pagination via Math.max and is persisted). Invariant now holds:
scalar total === Σ per-type across add/update/delete and reopen.
- Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no
longer publishes the manifest's full relationship count as healthy. Any
per-SSTable load failure throws after the batch, which resets to honest-empty
and lets the existing size()===0 self-heal rebuild run — size()/isHealthy()
can no longer lie about a partial load.
- Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears
dirty nodes whose connections failed to persist — failed nodes stay in the
retry set, and flush() throws HnswFlushError instead of returning a lying
node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so
addItem() rejects rather than returning an id for a rootless index.
- Pattern B, finding 11 (part — storage reads): new shared isAbsentError()
helper (utils/errorClassification, ENOENT-only absence) applied to
loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE)
now propagates loudly instead of masquerading as "absent", which had driven
needless rebuilds / empty reads (loadBinaryBlob feeds the native provider).
Regression: 78 green across the 3 new suites + db-mvcc, generationStore,
temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before
the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb
legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded),
finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
Transaction.execute() checked its time budget at the top of the operation
loop and threw TransactionTimeoutError from OUTSIDE the per-operation
try/catch, so a mid-flight timeout bypassed rollback entirely — only
per-operation failures rolled back. A bulk transact that crossed its 30s
budget mid-flight left the operations already applied to canonical storage
in place while the generation was never stamped: torn, generation-less
state. The generation-store commit path's abort cleanup explicitly assumes a
throw from execute() already restored the applied operations byte-identically
(it only discards the uncommitted staging directory), so the missing
rollback broke that invariant.
Give execute() a single rollback point: the operation loop is the sole
rollback-guarded region, and any error escaping it — an operation failure OR
a mid-flight timeout — rolls back every applied operation in reverse order,
then surfaces the original error (a rollback failure still supersedes it via
TransactionRollbackError). The per-exit-path rollback that let the timeout
throw slip past is gone; atomicity now holds by construction for every error
type. An aborted transaction leaves generation() unchanged and storage
byte-identical to its pre-transaction state.
Regression (tests/unit/transaction/timeout-rollback.test.ts): a mid-flight
timeout leaves an in-memory canonical store byte-identical with the tx in the
rolled_back terminal state; the operation-failure and rollback-failure paths
through the same single rollback point; and a clean transaction still commits.
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
Sweeping the vestigial hnsw sharding machinery surfaced two real defects on
the counts-recovery path (the code that rebuilds totalNounCount/totalVerbCount
when counts.json is lost or corrupted — container restarts, partial copies):
- initializeCountsFromDisk counted files in entities/*/hnsw/ — directories the
8.0 write path never populates — so an established store recovered to ZERO
counts on real data: wrong getNounCount()/stats, a "New installation"-style
boot log, and a mis-sized rebuild-strategy decision at open. The scan now
walks the canonical entities/<kind>/<shard>/<id>/ tree (one entity per id
directory), with the same sampled type-distribution estimate as before.
- The sampler called getNounMetadata — whose ensureInitialized() re-enters
init() — from INSIDE init(), a latent deadlock reachable the moment the scan
found anything to sample. The sampled metadata files are now read directly
with fs (gz-transparent), no guarded accessors inside init.
The dead machinery itself (verified zero callers by reachability analysis
before deletion): the nine 7.x hnsw-layout entity/edge CRUD methods, the
sharding-depth prober + its depth-migration engine (unreachable — the probe
always returned null on 8.0 stores), an orphaned streaming verb paginator,
their path/scan helpers, and the now-unused type aliases and fields. The init
block keeps the truthful new-vs-established boot log introduced in 8.0.13,
now decided directly from the canonical layout. Net -1,138 lines; no public
API touched.
Adds a regression test: delete counts.json from a populated store, reopen,
counts recover exactly from the canonical tree.
Every persisted 8.0 store logged "📁 New installation: using depth 1 sharding"
on every open — even brains holding thousands of entities — which is alarming to
read during a restart or incident. 8.0 stores nouns in the canonical
`entities/nouns/<shard>/<id>/vectors.json` layout (the path `saveNoun`/`getNouns`
use), but the legacy sharding probe `detectExistingShardingDepth()` inspects
`entities/nouns/hnsw` — a 7.x directory the 8.0 write path never populates (its
only writer, `saveNode`, is dead code with zero callers). So the probe returned
null for every 8.0 store and concluded "new".
Drive the new-vs-existing log from the layout the database actually reads and
writes: a new `hasCanonicalEntities()` checks for a real 2-hex shard directory
under `entities/nouns/`, and the known noun count short-circuits it. An
established store now logs "Using depth 1 sharding (N entities)"; only a genuinely
empty store reports a new installation. Behavior is otherwise unchanged — the
probe only ever set a log line, never triggered a rebuild or migration.
`find({ where })` had two gaps that could silently return nothing. The in-memory
matcher (`matchesQuery`, used for egress re-validation and historical reads) was
missing several documented operators — `in`, `greaterThanOrEqual`,
`lessThanOrEqual` — so it disagreed with the index path, which does handle them.
And an unknown operator key (a typo, or `notIn` written where `not: { in }` was
meant) fell through to a nested-object-field interpretation and matched nothing.
- Align `matchesQuery` to the full documented operator set (add the missing
aliases; the default branch now throws instead of treating an unknown key as a
nested field — dotted paths remain the supported nested form).
- Validate the `where` filter up front (`validateWhereFilter`), throwing a typed
`BrainyError('INVALID_QUERY')` that names the unknown operator, recursing into
`allOf` / `anyOf` / `not`.
- Stop excluding a user field literally named `level` from the metadata index
(it collided with an internal never-index key), so numeric/exact filters on it
resolve instead of returning 0.
Adds unit coverage for the operator set and the throw-on-unknown behavior.
A production deployment measured ~48 seconds on EVERY reopen of an
11k-entity brain. Root cause: brainy's rebuild gate decided from in-memory
size()/count, which read 0 for a durable-but-not-resident index, so it
re-read every entity file to rebuild from scratch. At GA we gave only the
GRAPH provider a readiness contract (init() eager cold-load + isReady()
honest signal) so it would never eat that spurious rebuild; the vector and
metadata providers never got it, and brainy never even eager-inited the
vector provider.
Complete the contract symmetrically:
- plugin.ts: VectorIndexProvider gains optional init()+isReady();
MetadataIndexProvider gains isReady() — mirroring GraphIndexProvider.
Additive and optional; a provider that exposes nothing keeps today's
behavior.
- brainy.ts: eager-init every provider that exposes init() (after metadata
init() so the id-mapper is hydrated first), then decide per leg in
precedence order — migrating (skip) -> epoch drift (rebuild) -> isReady()
-> a per-leg empty fallback. The old instant fast-path keyed off
this.index.size()>0, a dishonest proxy that skipped the metadata/graph
checks whenever the vector was warm and never fired on a real cold process
anyway; removed.
The per-leg fallbacks differ because "empty" means different things: the JS
vector's rebuild() IS its load, so size()===0 correctly triggers it; the
id-mapper backs metadata, so totalEntries===0 (past the empty-store return)
is a real load failure; but entities do not imply edges, so a graph
size()===0 is a valid empty state, not a load failure.
- The JS graph now COLD-LOADS its durable LSM instead of re-deriving from a
full canonical verb scan on every boot (baseStorage._initializeGraphIndex
loads the persisted SSTables via a new GraphAdjacencyIndex.init(); it
self-heals from canonical only when the durable state is genuinely missing).
This removes an O(E)-per-open cost every filesystem consumer paid.
- LSMTree.loadManifest loads its SSTables BEFORE publishing the relationship
count, and resets to an honest-empty state on load failure — a tree can no
longer claim persisted relationships while holding none (the silent-empty
cold-load class the query-time guards exist to prevent).
Verified end-to-end against a built brain: a warm reopen (with edges and
edgeless) reloads only the JS vector; the graph and metadata cold-load with
no rebuild, and queries return correct results. New tests in
cold-open-rebuild-gate.test.ts pin the contract (isReady() defers, self-heal
still fires); migration-deference updated to drive size-based deference
through the vector, the leg where empty->rebuild remains correct.
Pairs with the native provider's isReady()/init() implementation — brainy's
gate defers only to a signal the provider exposes.
A consumer's clean-room verification of the 8.0.10 fix found relate() still
hanging their scripts. Root-causing the CLASS instead of the repro found two
mechanisms:
1. The 'beforeExit' auto-flush hook looped forever on any script that never
reaches close(): Node re-emits beforeExit after every event-loop drain, and
the async flush schedules new work — flush, drain, flush, forever. The
listener now self-deregisters BEFORE its one flush, so the next drain exits.
(Empirically: process.on('beforeExit', async () => await anything) alone
never exits — this was the deepest root of the whole hang class.)
2. Every background-maintenance interval is now unref'd at creation — graph
auto-flush, LSM compaction, metadata write-buffer flush, VFS cache
maintenance, PathResolver cache maintenance, statistics debounce (the
writer-lock heartbeat, flush watcher, and cache monitors already were).
Durability is owned by close() and the beforeExit flush, both deterministic;
a best-effort interval must never keep the host process alive.
Proof: the reported shape (add x2 + relate, filesystem) exits cleanly BOTH
with close() (~0.5 s) and with NO teardown at all — and in the no-teardown
case the beforeExit flush still lands the data (verified by reopen: both
nouns + the edge present). New per-op-class sweep test asserts no ref'd
timer survives close() for add / relate / graph find / metadata update /
vfs — turning this bug class off permanently instead of per-repro.
A consumer report on the GA pair: any minimal add/find/close script hangs
forever. Root cause was FOUR ref'd keep-alives brainy never released:
1+2. The global SIGTERM/SIGINT shutdown hooks (registered once at init) were
anonymous and never removed — a process.on signal listener holds a ref'd
signal handle. They are now named statics, deregistered when the LAST live
instance closes (Brainy.instances also stopped leaking closed instances);
a later init re-registers them.
3. UnifiedCache.startFairnessMonitor() created an interval it never stored,
cleared, or unref'd — unclearable by construction. Now stored + unref'd
(same posture as the existing memory-pressure timer): a cache monitor must
never keep the host process alive.
4. brainy.close() never called VirtualFileSystem.close(), stranding the VFS
background-maintenance interval and the PathResolver maintenance interval.
close() now shuts the VFS down.
Proof: a bare script (init/add/close, no process.exit) exits 1 ms after
close() returns; before the fix it hung until killed. 3 new lifecycle tests
pin the hook semantics (once globally, survive while other instances live,
removed on last close, re-register after).
With plugins unset (the default), init() probes for the first-party
accelerator: not installed means plain brainy with zero noise; installed and
healthy means it loads and announces itself; installed but broken (import
failure, invalid shape, failed activation, version mismatch) makes init()
THROW. An installed accelerator never silently vanishes behind the JS engines
- the anti-drift posture of the explicit list, applied to detection.
plugins: []/false stays a true opt-out (no probe); an explicit list keeps its
required-and-loud semantics. The import runs through an importPluginPackage
seam (variable specifier - bundlers cannot static-resolve the optional
package; tests simulate all outcomes without it installed).
Also: when a plugin activates but registers zero native providers (e.g. a
licensing gate declining to engage), the provider summary now warns loudly
instead of leaving every query silently on the JS engines.
Supersedes 8.0.8's explicit-opt-in wording; README/PLUGINS/types now document
the guarded contract. 7 new tests (tests/unit/plugin-autodetect.test.ts).
A downstream deployment reported find({where}) returning a silent [] on a
freshly-opened brain (a native metadata provider that reports data but hasn't
loaded its field postings), blanking filtered pages after every restart. Brainy
had a cold-read guard for GRAPH reads (verifyGraphAdjacencyLive → self-heal or a
loud GraphIndexNotReadyError) but no equivalent for metadata `where`.
Add verifyMetadataLive() — the field-index counterpart, one-shot per brain on the
first filtered find(): probe a known persisted entity's plain field value; if the
index serves it, live (the only cost on a warm brain — one O(1) probe). If not,
the postings didn't load: rebuild from canonical + re-probe; if still unserved,
throw the new exported MetadataIndexNotReadyError rather than let a silent [] pass.
Inconclusive cases (empty store, no plain field, shared-store foreign entity,
migrating provider) resolve to live — never a false rebuild.
The 8.0 open-core JS index already cold-loads correctly (verified: cold where 3/3,
cold related 2/2) — this guards the NATIVE path, where the durable cold-load cure
is cortex-side (same shape as the graph 2.7.8 cure). 4 unit tests (warm no-rebuild,
self-heal, loud-fail, no-filter-no-probe). Gates: typecheck 0, build 0, test:unit
1757/1757.
Replaces rc.8's no-freeze deference with an automatic, observable, coordinated
migration LOCK (David's reversal: "unknown/halfway states are more dangerous
than blocking"). While a native provider runs its one-time 7.x→8.0
rebuild-from-canonical (`isMigrating()===true`), brainy blocks/queues data-plane
reads AND writes so no operation ever touches a half-built index — closing the
three seam-map gaps (lost mid-rebuild writes, degraded reads, read-time rebuild).
Mechanism (one choke point):
- awaitMigrationLock() at ensureInitialized() covers all ~40 data-plane methods;
a non-migrating brain pays one boolean check. Polls isMigrating() at 250ms;
after migrationWaitTimeoutMs (default 30s) throws a retryable, exported
MigrationInProgressError. The timeout bounds the CALLER'S WAIT, not the
migration — the rebuild is unbounded and never interrupted.
- init() awaits the lock before VFS bootstrap + before serving ("not ready until
upgraded"); a rebuild past the budget surfaces MigrationInProgressError at init
(raise the budget, or run the offline migrator).
Observability (readiness-probe correct — never gated):
- getIndexStatus() gains `migrating` + `migration` (MigrationProgress); a probe
maps migrating→HTTP 503+Retry-After, not 500. health()/checkHealth() lock-exempt
(health() reports a `warn` migration check). stampBrainFormat()/close() are
ungated so cor can clear the lock — no deadlock.
- Optional provider migrationStatus() is relayed verbatim for a live %.
verifyGraphAdjacencyLive() honors the lock (no self-rebuild mid-migration); the
stamp is still withheld while any provider migrates (cor stamps after verify).
Adversarially verified: fixed a critical init/VFS-bootstrap deadlock, a mixed-
provider busy-spin (dropped the event-driven signal → pure poll), a not-yet-
initialized getIndexStatus crash, and once-per-window log/clock resets. 10 lock
tests (incl. the init-during-migration regression). Gates: typecheck 0, build 0,
test:unit 1753/1753.
Three hooks so a native provider can run a non-blocking, online, background
build-new→verify→swap index migration on a large-brain upgrade while brainy stays
out of the way — no minutes-long blocking rebuild-on-open/first-query.
- isMigrating?(): boolean — OPTIONAL on MetadataIndexProvider / GraphIndexProvider /
VectorIndexProvider (mirrors isReady?()/init?(), feature-detected). While a
provider reports migrating, brainy SKIPS its rebuild for that index — per-index,
so a non-migrating sibling still rebuilds; skipped even under epoch-drift or
size()===0 — in both rebuildIndexesIfNeeded and the lazy first-query force path.
The provider serves correct reads from canonical until it verifies-and-swaps.
- brain.stampBrainFormat() — public; the provider calls it once its background swap
verifies, so brainy authors dataFormat (the provider never does). brainy withholds
its own marker stamp while any provider is migrating, so the shared epoch is never
advanced ahead of a deferred index.
- ./brain-format export — the marker module's EXPECTED_INDEX_EPOCH / CURRENT_DATA_FORMAT
are importable so a native provider shares the constant (no duplicate = no
lockstep-drift). 8-case test; unit 1743 green.
The MVCC time-travel layer kept nounChains/verbChains as unbounded
Map<id, number[]> — one resident chain per id ever touched across retained
history (O(N) RAM, defeating billion-scale time travel). Replace with a hot-tail
window + bounded cold LRU + a mutex-free bulk resolver, keeping resolveAt exactly
correct.
The most-recent W generations stay resident as full chains (the common recent-pin
read is O(log), zero scan); deeper pins reconstruct one id's chain on demand into
an L-bounded LRU. Reconstruction is LOCK-LIGHT — it holds no commit mutex, so a
historical read never stalls writers; correctness holds because a live pin's
answer is always > minPinnedGeneration, which compaction can never reclaim, and a
concurrently-reclaimed gen below that is skipped. materializeAtGeneration routes
through a new mutex-free resolveManyAt (one forward pass, O(R + |ids|)) — without
it a deep-pin materialize both regresses to O(N*R) and deadlocks on snapshotWith's
mutex. The cold cache is invalidated on re-touch to stay coherent. Resident RAM is
O(W*d + L), independent of N. 11-case test (oracle-vs-bruteforce, held-Db across
eviction + concurrent compaction, no-deadlock, write-path I/O-free); unit 1735 +
integration 613 (db-mvcc/db-temporal/db-asof green).
Add _system/brain-format.json { dataFormat, indexEpoch } + a sync brain.formatInfo()
accessor + a compiled EXPECTED_INDEX_EPOCH, the surface a native provider reads at
init() to drive whole-brain auto-upgrade, and brainy's own derived-index
rebuild-on-format-drift trigger (closing the gap where JS indexes rebuilt only on
size()===0, never on a format-version change).
indexEpoch is shared and lockstep-bumped with the native provider on any coordinated
release whose on-disk derived-index format changes; dataFormat is brainy-owned. On open,
the marker is read in the store-open phase BEFORE provider construction (so formatInfo()
is synchronously available at the provider's init); a drifted or absent epoch rebuilds
the derived indexes from canonical records, and the marker is stamped only AFTER the
rebuild verifies (build-new -> verify -> stamp; a crash before the stamp re-triggers the
idempotent rebuild). brainFormat.ts is the single source of the shared constants.
6-case test; 1724 unit green.
Both shapes confirmed with cor for the lockstep (cor builds the native side
against these; the JS index is a no-op / unchanged):
- probeConsistency?(): Promise<boolean> — OPTIONAL O(1) cold-open consistency
sampler on MetadataIndexProvider. brainy calls it ONCE per brain on the first
read; on `false` it self-heals via detectAndRepairCorruption() — the metadata
counterpart of the 7.33.2 graph cold-load guard, completing the phantom triad's
detect-and-repair-on-open. Best-effort: a probe failure never breaks a read
(the one-shot guard resets so a transient failure retries). The JS index omits
the method, so it is simply never probed.
- getIdsForFilter(filter, opts?: { limit?; offset? }) — brainy passes a page
bound on the UNSORTED find({ type, where, limit }) path so a native provider can
early-stop and return only the [0, limit) prefix, killing the O(N) FFI marshal
at billion scale (pairs with cor #75). brainy passes offset:0 and ALWAYS
re-windows the result itself (visibility filter + slice); the JS index ignores
opts and returns all matches (behaviour unchanged).
New unit tests cover brainy's call behaviour for both (probe→repair on false,
healthy→no-repair, failure-is-best-effort, once-per-brain; opts passed with
offset 0 on the unsorted path). Full gate green: unit 1718, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
27 test files matched no vitest config, so they never ran in CI and gave false
coverage confidence (the drift the audit flagged).
- Re-homed 10 functional suites into the gate (renamed to *.unit.test.ts /
*.integration.test.ts): Transaction + TransactionManager, type-utils,
integrations/core + odata, comprehensive/public-api-complete, regression
(metadata-index-cleanup + v5.7.0-deadlock), vfs/tree-operations +
vfs-bulkwrite-race. ~192 previously-dark tests now run and pass.
- Deleted 8 bit-rotted/redundant files that fail against 8.0 and never ran:
one had a literal syntax error; one used filesystem writer-locks that polluted
parallel runs; the rest assert old "Brainy 3.0/v3.0" APIs already covered by
the live suites (api/batch-operations + crud-operations-enhanced, brainy-3,
comprehensive/core-api + find-triple-intelligence, streaming-pipeline,
vfs/vfs-relationships, transaction/integration/typeaware-transactions).
- Added a coverage guard (tests/unit/test-suite-coverage-guard.test.ts) that
FAILS CI if any *.test.ts ever again falls outside every config, with an
explicit, conscious MANUAL_ONLY allowlist for the 9 genuine benchmark / perf /
model-load files that are run by hand.
Gate green: unit 1712, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The JS metadata index served `ne`, `exists:false`, and `missing:true` by
materializing the ENTIRE id universe as an array of UUID strings, building a Set
of the excluded ids, and filtering — O(N) heap and two full passes on the
open-core path, on the most common shape (`deleted !== true`).
Compute the complement as a roaring-bitmap difference over the int-id universe
instead (`complementIds(excludeInts)` = universe \ exclude), converting only the
result back to UUIDs. The exclude set is small (the matching value), so this
avoids the full-corpus string materialization entirely. Behaviour is unchanged,
including the load-bearing soft-delete semantic that `field !== value` includes
entities with no such field. New focused tests pin `ne`/`exists:false`/
`missing:true`/`exists:true` result sets and that the complement reflects deletes.
Full gate green: unit 1520, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
removeItem() scanned the ENTIRE corpus on every delete to find nodes that
referenced the removed id (and to repair edges left asymmetric by pruning), so a
delete was O(N) and a bulk delete O(N²) — on the open-core JS vector path that
serves when no native provider is registered.
Maintain a reverse-adjacency index (`target → level → set of nodes that link to
target`), so removeItem touches only the removed node's actual in-neighbors:
O(in-degree) per delete, O(N·degree) for a bulk delete. The index is lazily
built, maintained incrementally at every forward-edge mutation (add-link and
prune), and invalidated (rebuilt on next use) by the bulk paths (cold-load
restore + clear). New tests assert the three invariants that matter for a
reverse index: no dangling references to deleted nodes, the incrementally-
maintained index exactly equals a fresh rebuild from the live adjacency, and
search still returns the survivors' true nearest neighbours (exact vs brute force).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- find() search-mode: collapsed the two overlapping options to one canonical
`searchMode: SearchMode` (the redundant `mode` alias was silently ignored on
the primary find() path while honored on the historical path — a footgun) and
removed the unwired `explain?` FindParams field (never read by find()).
- Documentation accuracy on the public type surface: entity ids documented as
UUID v7 (auto) / v5 (natural key), not v4; dropped the stale "Brainy 3.0"
banners and "backward compatibility" hedging on the fresh-8.0 Result type;
documented the via/type alias; removed the dead GraphConstraints.bidirectional;
fixed the no-op `EmbeddedGraphVerb = Omit<GraphVerb,'source'>` to omit the real
sourceId; corrected the GraphIndexProvider.isInitialized JSDoc; documented
removeMany's per-chunk generation granularity; JSDoc'd the public VFS surface.
- Honest perf comments in source: removed unmeasured billion-scale figures from
the LSMTree / graphAdjacencyIndex headers (the JS fallback is in-memory after
open; native is the scale path).
- Robustness: getVerbMetadata propagates read errors symmetrically with
getNounMetadata; the find() egress integrity-guard keeps a row when the JS
matcher doesn't implement an operator the provider already matched on; hardened
the boundary-no-native CI guard to catch side-effect imports + re-exports.
- Tests: de-theatricalized a relateMany test to assert the real contract; fixed
a stale Model-B header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Version-coupling cold-init: getBrainyVersion() returned a stale build-time
default ('3.14.0') on the FIRST synchronous call — the one loadPlugins() makes
to build context.version — because the package.json read was async. A native
provider declaring a realistic `>=8.0.0` range was therefore rejected on cold
init. version.ts now reads package.json synchronously (8.0 targets Node-like
runtimes only); added a cold-init coupling regression with a `^8.0.0` plugin.
- No-silent-failures on the highest-fan-in read path: getNouns()/getVerbs()
converted a storage read failure into a success-shaped empty page, which the
cold-start rebuild then read as "store empty" and skipped the rebuild — booting
a permanently-empty index with no signal (the same silent-failure class as the
phantom bug). Both now re-throw a named BrainyError; the rebuild path re-throws
read failures (fail loud) and records a queryable degraded state, surfaced via
checkHealth(), for non-fatal rebuild hiccups.
- LSM SSTable leak: graph compaction wrote a merged SSTable but only dropped the
old ones from the manifest, orphaning their payloads forever ("In production
we'd add a cleanup mechanism"). Added StorageAdapter.deleteMetadata() and now
reclaim each compacted-away SSTable.
- Documented the two headline methods add() and find() (the only undocumented
public methods on the class).
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
find() trusted the metadata index's returned ids: it loaded each id's entity
and returned it, checking only that the entity existed — never that it actually
matched the query. The indexes are acceleration structures; the loaded entity
is ground truth. A stale or cross-bucket index posting (e.g. an id left in a
field-value bucket by a delete or an `update({ field: undefined })`) therefore
surfaced an entity matching NEITHER the requested type NOR the where filter — a
production report saw a timeslot (NounType.Event) returned for
`find({ type: Person, where: { entityType: 'staff' } })`.
Add a single egress chokepoint after the result IIFE that re-validates every
result with `entityMatchesFind` (type/subtype/where/service/excludeVFS) — the
live-path mirror of the historical path's per-candidate check in db.ts. It
covers every find() branch (metadata, vector, text, proximity, graph) and
`similar()` (which delegates to find) in one place. A no-op on a healthy index;
on a corrupted one it drops the bad row instead of returning a phantom. Full
unit + integration gate green confirms the where re-validation is consistent
with the index (no valid rows dropped).
This is the brainy-side safety net. The durable corruption itself lives in the
metadata index that owns the query (the native provider when present) and is
addressed separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The optional native graph-acceleration provider exposes `isInitialized`,
which is false during its cold-start / rebuild window (engine + cursor not
yet loaded). The accessor cached the resolved provider INSTANCE but never
re-checked readiness, so `brain.graph.*` could dispatch to a not-ready
provider and throw instead of answering.
Resolve and cache the instance once, but check `isInitialized` LIVE on every
call: route to the pure-TS analytics path while the provider is not ready,
and re-engage the native path automatically the moment it flips true. This
gates every native route at once — subgraph, export, rank, communities,
path, and the query→expand fusion. Adds a test asserting the provider is
never called (and the TS fallback returns real answers) while not ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-GA dead-code sweep. A deterministic import-reachability walk from the
package's real entry points (the exports map, the CLI bin, the conversation
surface) found 34 source modules that nothing reachable imports — they
compile and ship as dist output that no consumer or internal path can ever
reach. All were superseded duplicates, abandoned parallel implementations,
or built-but-never-wired features:
- superseded duplicates of live modules: an older "unified" entry, a
standalone neural-import variant, a static NLP processor + its matcher,
a parallel API-types module, a duplicate progress-types module
- an abandoned import path (orchestrator + entity deduplicator + barrels)
left behind when ingestion moved to the coordinator
- unwired feature modules (instance pool, import presets, cached
embeddings, relationship-confidence scorer) reachable only from tests
- dead leaf utilities (write buffer, deleted-items index, bounded
registry, two crypto shims, a cache manager, a structured logger, a
stale v5 type-migration helper, a browser-only FS type shim) and
several dead re-export barrels
- a CLI catalog command wired into no command
Also removed two tests that only exercised deleted modules, repointed two
VFS tests off a deleted barrel onto the implementation module, and deleted
one example that demoed removed features.
Verified: clean build (both tsc passes), unit 1505 + integration 607 green,
and a re-run of the reachability walk reports zero remaining orphans.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered + ordered find() materialized the FULL sorted match set before slicing
the page — passing k = filteredIds.length to the column store's filteredSortTopK.
A broad filter + orderBy returning 20 rows at billion scale produced hundreds of
millions of sorted ids to discard all but the page (O(matches) heap).
Thread the page bound (offset + limit + the hidden-tier over-fetch) into
getSortedIdsForFilter → filteredSortTopK / sortTopK / the sparse-path slice, so
the sort produces only the page. Ordering and pagination are unchanged.
From cor's 2026-06-24 co-release scaling audit (item 1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The subgraph seed selector now accepts not just entity id(s) but a find() result
or a FindParams query — brain.graph.subgraph({ where: { team: 'platform' } },
{ depth: 1 }) runs the query and expands the neighborhood of every match in one
call. Existing id-seeded calls are unchanged.
The win is the native query→expand path: for a metadata-only query, the matched
universe is forwarded to traverse() as an OpaqueIdSet (a roaring Buffer) with NO
id materialization in TypeScript — the find() result never leaves the engine's
representation (the O(1)-crossing the cor 3.0 contract is built around). The
pure-JS path (or any query carrying vector/text/proximity criteria) materializes
the matched ids via find() and seeds the traversal from them, capped at a bounded
default when the caller pins no limit.
- New public `SubgraphSelector<T>` union (id | id[] | Result[] | FindParams).
- find()'s metadata filter-builder extracted to a shared `buildMetadataFilter`
(reused by the opaque universe producer); behavior unchanged.
- graphSubgraphNative generalized to accept `bigint[] | OpaqueIdSet` seeds.
Tested: JS query→expand + Result[] selector + empty-match in graph-subgraph, and
the native opaque pass-through (Buffer reaches traverse unmaterialized) vs the
find()-materialized bigint[] seeds via the mock provider in graph-native-routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Filtered semantic search now keeps its recall. A find({ query, where }) that
pairs vector search with a metadata filter restricts the HNSW beam walk to the
matching candidates INSIDE the traversal (walk-all, collect-allowed) rather than
filtering the top-k afterward — so a query whose nearest vectors are all filtered
out still returns the best matches that DO pass the filter, instead of empty.
- JS HNSW search() honors the 8.0 `allowedIds: OpaqueIdSet | ReadonlySet<string>`
contract param (ANDs with candidateIds; ignores the opaque Buffer form it can't
decode — only a native provider consumes that).
- MetadataIndexProvider gains an OPTIONAL `getIdSetForFilter(filter): OpaqueIdSet`
producer — the native (cor) metadata index returns its roaring filter result as
a serialized buffer; the JS index does not implement it.
- find() forwards the matched universe to the vector walk: the opaque buffer
(zero id materialization) when the native producer is present, alongside the
materialized visibility-precise candidateIds for the JS index. Visibility stays
correct via the existing post-search hard filter.
Tested: JS recall/restriction/AND/opaque-ignore on the index directly, plus
find() forwarding the opaque universe through to the beam walk via a stubbed
native producer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds three intent-level graph reads to the `brain.graph` namespace, each
native-dispatched to the optional `@soulcraft/cor` 3.0 graph engine when present
and served from pure-TS kernels otherwise (identical public shapes, default
visibility filter respected on both paths):
- `rank(opts?)` → `{ id, score }[]` descending — importance / centrality.
TS fallback: PageRank power-iteration with dangling-mass redistribution.
- `communities(opts?)` → `{ groups, count }` — connected grouping. TS fallback:
union-find weakly-connected components, or iterative Tarjan SCC when
`{ directed: true }`.
- `path(from, to, opts?)` → `{ nodes, relationships, cost } | null` — best route.
TS fallback: BFS for fewest hops, Dijkstra (min-heap) for least summed edge
weight (`by: 'weight'`); on-demand frontier expansion so short paths terminate
early. `direction` / `type` / `maxDepth` filters apply.
These are intent contracts, not algorithm contracts — the question is the
promise, the algorithm is the engine's choice.
Pure kernels live in src/graph/analyticsFallback.ts (PageRank, connected
components, Tarjan SCC, MinHeap) — unit-tested in isolation. The full surface is
tested end-to-end through the TS fallback, and the native dispatch + int↔uuid
hydration paths are covered by a mock provider in graph-native-routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
